★ wanayoo — archive 1999 https://github.com/nodejs/node/pull/21952Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Makefile: Add lint-python which uses flake8 #21952

Merged
merged 1 commit into from Oct 24, 2018

Conversation

Projects
None yet
8 participants
@cclauss
Copy link
Contributor

cclauss commented Jul 24, 2018

Add a lint-python section to Makefile which pip installs flake8 and then executes:

  • flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

Based on the experimentation done at https://travis-ci.com/nodejs/node/builds/79706150 - #21942

These tests currently exclude several directories and files:

  • ./deps/npm/node_modules/node-gyp -- this repo currently has no automated testing
  • ./deps/v8 -- contains several Python 3 only features such as async def
  • ./src/noperfctr_macros.py -- contains "macros" which Python AST views as syntax errors
  • ./src/notrace_macros.py -- contains "macros" which Python AST views as syntax errors
  • ./tools -- contains "macros" which Python AST views as syntax errors

Hopefully this list can be reduced in future PRs with code modifications or the use of # noqa

E901,E999,F821,F822,F823 are the "showstopper" flake8 issues that can halt the runtime with a SyntaxError, NameError, etc. Most other flake8 issues are merely "style violations" -- useful for readability but they do not effect runtime safety.

  • F821: undefined name name
  • F822: undefined name name in __all__
  • F823: local variable name referenced before assignment
  • E901: SyntaxError or IndentationError
  • E999: SyntaxError -- failed to compile a file into an Abstract Syntax Tree
Checklist
  • make -j4 test (UNIX), or vcbuild test (Windows) passes
  • tests and/or benchmarks are included
  • documentation is changed or added
  • commit message follows commit guidelines
@mscdex

This comment has been minimized.

Copy link
Contributor

mscdex commented Jul 24, 2018

Is it not possible to bundle the linting library instead, like we do for eslint? That way we don't need to make system-wide changes (e.g. implicitly upgrading pip)?

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Jul 24, 2018

We could remove pip install —upgrade pip.

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Jul 24, 2018

@mscdex Does that last commit cover what you wanted or did I misunderstand your request?

Is the automated testing actually doing a "make lint"? If so, how do I view the output?

https://github.com/nodejs/node/runs/8932739 does not seem to contain the words "lint-python" or "flake8".

@cclauss cclauss referenced this pull request Jul 24, 2018

Closed

Travis CI: Add flake8 jobs for Python 2.7 and Python 3.7 #21953

1 of 4 tasks complete
Makefile Outdated
# Flag the build if there are Python syntax errors or undefined names
lint-python: lint-python
@echo "Running Python linter on $(shell $(PYTHON) --version)..."
@$(PYTHON) -m pip install --upgrade flake8

This comment has been minimized.

Copy link
@targos

targos Jul 24, 2018

Member

This doesn't work on my system when not run as root:

OSError: [Errno 13] Permission denied: '/usr/lib/python2.7/site-packages/enum34-1.1.6.dist-info'
make[1]: *** [Makefile:1212: lint-python] Error 2

This comment has been minimized.

Copy link
@cclauss

cclauss Jul 24, 2018

Author Contributor

Can you please try to add --user to the command and let me know if that works?

This comment has been minimized.

Copy link
@targos

targos Jul 24, 2018

Member

👌 It works with --user

@targos

This comment has been minimized.

Copy link
Member

targos commented Jul 24, 2018

Exclusions don't seem to work. I see many errors coming from deps/v8 in the output

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Jul 24, 2018

You see these errors where? Locally or in server-based build log?

@targos

This comment has been minimized.

Copy link
Member

targos commented Jul 24, 2018

Locally, with make lint

@refack

This comment has been minimized.

Copy link
Member

refack commented Jul 24, 2018

@refack

refack approved these changes Jul 24, 2018

Copy link
Member

refack left a comment

  1. Should split build code to a lint-python-build target, and add conditions (maype if $(shell python -m flake8 --version)
  2. remove @ for traceability
  3. IMHO setting should go into tox.ini and not stored in the Makefile
@refack
Copy link
Member

refack left a comment

see comments

@refack

This comment has been minimized.

Copy link
Member

refack commented Jul 24, 2018

Hello @cclauss and thank you for the contribution 🥇
I made a few comments about the current implementation, but a part from that I'm not so sure this should run as part of the lint target (probably just the lint-ci target). We have a goal of optimizing the build time, for developers running locally, and separately on the CI cluster. The churn of the python files in the project is very low, so IMHO we could skip running this for every change...
Also we should simply exclude the whole of the /deps/ dir since those files are vendored in from 3rd party repos (almost everything) same for /tools/gyp/, as well as the /src/ and /lib/ dirs since they should not contain any python files.

@refack refack self-assigned this Jul 24, 2018

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Jul 24, 2018

I implemented all suggestions except:

  1. the if statement to jump from lint-python to lint-python-build (I could not get it right)
  2. tox.ini because I am not a fan of another level of indirection and it robs the power to create a second pass thru flake8 with no --select but with --exit-zero on to find PEP8 violations.
  3. Slight changes on the --excludes and a lint-python-all-files that has no --exclude
Makefile Outdated
.PHONY: lint-python
# Lints the Python code with flake8.
# Flag the build if there are Python syntax errors or undefined names
lint-python: lint-python

This comment has been minimized.

Copy link
@richardlau

richardlau Jul 24, 2018

Member
lint-python: lint-python-build

?

@mscdex

This comment has been minimized.

Copy link
Contributor

mscdex commented Jul 24, 2018

Does that last commit cover what you wanted or did I misunderstand your request?

What I meant was include the 'flake8' module in the repo, like we do with eslint for js files, to avoid having to install it system/user-wide (which I would find to be unexpected behavior).

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Sep 10, 2018

What's the status on this one?

@jasnell jasnell added the stalled label Sep 10, 2018

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Oct 21, 2018

@mscdex it is expected behavior for Python package. "Vendoring in" is quite rare in the Python world.

@refack Could you please advise here. It has been three months...

@refack refack force-pushed the cclauss:add-flake8-to-testing branch from a4f0ae4 to 4458162 Oct 21, 2018

@refack

refack approved these changes Oct 21, 2018

@refack

refack approved these changes Oct 21, 2018

@refack

refack approved these changes Oct 21, 2018

@refack

refack approved these changes Oct 22, 2018

@refack

refack approved these changes Oct 22, 2018

@refack

refack approved these changes Oct 22, 2018

@refack

refack approved these changes Oct 22, 2018

@refack

refack approved these changes Oct 22, 2018

@cclauss

This comment has been minimized.

Copy link
Contributor Author

cclauss commented Oct 22, 2018

@refack refack removed the stalled label Oct 22, 2018

@refack

This comment has been minimized.

@refack refack force-pushed the cclauss:add-flake8-to-testing branch from 4458162 to a1b6029 Oct 24, 2018

refack added a commit to cclauss/node that referenced this pull request Oct 24, 2018

build: add lint-py which uses flake8
PR-URL: nodejs#21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@refack refack force-pushed the cclauss:add-flake8-to-testing branch from a1b6029 to 604890b Oct 24, 2018

refack added a commit to cclauss/node that referenced this pull request Oct 24, 2018

build: add lint-py which uses flake8
PR-URL: nodejs#21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@refack refack force-pushed the cclauss:add-flake8-to-testing branch from 604890b to 9a690c3 Oct 24, 2018

refack added a commit to cclauss/node that referenced this pull request Oct 24, 2018

build: add lint-py which uses flake8
PR-URL: nodejs#21952
Reviewed-By: Refael Ackermann <refack@gmail.com>
build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@refack refack force-pushed the cclauss:add-flake8-to-testing branch from 9a690c3 to 0c39290 Oct 24, 2018

@refack refack merged commit 0c39290 into nodejs:master Oct 24, 2018

@refack

This comment has been minimized.

Copy link
Member

refack commented Oct 24, 2018

Congratulations 🎉
Landed in 0c39290
@cclauss, thank you very much for the contribution.

@cclauss cclauss deleted the cclauss:add-flake8-to-testing branch Oct 24, 2018

@refack refack removed their assignment Oct 24, 2018

targos added a commit that referenced this pull request Oct 26, 2018

build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@targos targos referenced this pull request Oct 27, 2018

Merged

Release proposal: v11.1.0 #23922

targos added a commit that referenced this pull request Nov 1, 2018

build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

MylesBorins added a commit that referenced this pull request Nov 26, 2018

build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@codebytere codebytere referenced this pull request Nov 27, 2018

Closed

v10.13.1 proposal #24675

rvagg added a commit that referenced this pull request Nov 28, 2018

build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

MylesBorins added a commit that referenced this pull request Nov 29, 2018

build: add lint-py which uses flake8
PR-URL: #21952
Reviewed-By: Refael Ackermann <refack@gmail.com>

@codebytere codebytere referenced this pull request Nov 29, 2018

Merged

v10.14.2 proposal #24727

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.