Hypermodern Python Chapter 3: Linting

Read this article on Medium

In this third installment of the Hypermodern Python series, I’m going to discuss how to add linting, code formatting, and static analysis to your project.1 Previously, we discussed Automated Testing. (If you start reading here, you can also download the code for the previous chapter.)

Here are the topics covered in this chapter on Linting in Python:

Here is a full list of the articles in this series:

This guide has a companion repository: cjolowicz/hypermodern-python. Each article in the guide corresponds to a set of commits in the GitHub repository:

Linting with Flake8

Linters analyze source code to flag programming errors, bugs, stylistic errors, and suspicious constructs. The most common ones for Python are pylint and the linter aggregators flake8, pylama, and prospector. There are also multi-language linter frameworks such as pre-commit and coala. In this chapter, we use Flake8 and pre-commit.

Add a Nox session to run Flake8 on your codebase:

# noxfile.py
locations = "src", "tests", "noxfile.py"


@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    session.install("flake8")
    session.run("flake8", *args)

By default, we run Flake8 on three locations: the package source tree, the test suite, and noxfile.py itself. You can override this by passing specific source files, separated from Nox’s own options by --. The session.install method installs Flake8 into the virtual environment via pip.

Flake8 glues together several tools. The messages produced by these tools are assigned error codes, prefixed by one or more letters. The prefixes group the errors into so-called violation classes:

  • F are errors reported by pyflakes, a tool which parses source files and finds invalid Python code.
  • W and E are warnings and errors reported by pycodestyle, which checks your Python code against some of the style conventions in PEP 8.
  • C are violations reported by mccabe, which checks the code complexity of your Python package against a configured limit.

Configure Flake8 using the .flake8 configuration file, enabling all the built-in violation classes and setting the complexity limit:

# .flake8
[flake8]
select = C,E,F,W
max-complexity = 10

By default, Nox runs all sessions defined in noxfile.py. Use the --session (-s) option to restrict it to a specific session:

nox -rs lint

There are many awesome Flake8 extensions. Some of these will be presented in later sections.

Code formatting with Black

The next addition to our toolbox is Black, the uncompromising Python code formatter. One of its greatest features is its lack of configurability. Blackened code looks the same regardless of the project you’re reading.

Adding Black as a Nox session is straightforward:

# noxfile.py
@nox.session(python="3.8")
def black(session):
    args = session.posargs or locations
    session.install("black")
    session.run("black", *args)

With the Nox session in place, you can reformat your code like this:

$ nox -rs black

nox > Running session black
nox > Creating virtual environment (virtualenv) using python3.8 in .nox/black
nox > pip install black
nox > black src tests noxfile.py
All done! ✨ 🍰 ✨
5 files left unchanged.
nox > Session black was successful.

Invoking nox without arguments triggers all the sessions, including Black. It would be better to only validate the coding style without modifying the conflicting files. Exclude Black from the sessions run by default, by setting nox.options.sessions at the top:

# noxfile.py
nox.options.sessions = "lint", "tests"

Instead, check adherence to the Black code style inside the linter session. The flake8-black plugin generates warnings if it detects that Black would reformat a source file:

# noxfile.py
@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    session.install("flake8", "flake8-black")
    session.run("flake8", *args)

Configure Flake8 to enable the flake8-black warnings, which are prefixed by BLK. Also, some built-in warnings do not align well with Black. You need to ignore warnings E203 (Whitespace before ‘:'), and W503 (Line break before binary operator), and set the maximum line length to a more permissive value:

# .flake8
[flake8]
select = BLK,C,E,F,W
ignore = E203,W503
max-line-length = 88

Checking imports with flake8-import-order

The flake8-import-order plugin checks that import statements are grouped and ordered in a consistent and PEP 8-compliant way. Imports should be arranged in three groups, like this:

# standard library
import time

# third-party packages
import click

# local packages
from hypermodern_python import wikipedia

Install the plugin in the linter session:

# noxfile.py
@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    session.install("flake8", "flake8-black", "flake8-import-order")
    session.run("flake8", *args)

Enable the warnings emitted by the plugin (I like import).

# .flake8
[flake8]
select = BLK,C,E,F,I,W

Inform the plugin about package names which are considered local:

# .flake8
[flake8]
application-import-names = hypermodern_python,tests

Adopt the Google styleguide with respect to the grouping and ordering details:

# .flake8
[flake8]
import-order-style = google

Recommending an import linter in 2020 is not an easy task, as there is currently a lot of movement in this area. The excellent plugin recommended in this section has been placed in maintenance mode. An alternative is isort, which comes with Flake8 integration via flake8-isort and additionally supports rewriting files. isort enjoys widespread popularity, but has also attracted much criticism (which its author intends to address in the upcoming major release). If you are looking for a tool to rewrite imports, you should also have a look at asottile/reorder-python-imports and sqlalchemyorg/zimports.

Finding more bugs with flake8-bugbear

The flake8-bugbear plugin helps you find various bugs and design problems in your programs. Add the plugin to the linter session in your noxfile.py:

# noxfile.py
@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    session.install("flake8", "flake8-black", "flake8-bugbear", "flake8-import-order")
    session.run("flake8", *args)

Enable the plugin warnings in Flake8’s configuration file (B like bugbear):

# .flake8
[flake8]
select = B,B9,BLK,C,E,F,I,W

B9 is required for Bugbear’s more opinionated warnings, which are disabled by default. In particular, B950 checks the maximum line length like the built-in E501, but with a tolerance margin of 10%. Ignore the built-in error E501 and set the maximum line length to a sane value:

# .flake8
[flake8]
ignore = E203,E501,W503
max-line-length = 80

Identifying security issues with Bandit

Bandit is a tool designed to find common security issues in Python code. Install it via the flake8-bandit plugin:

# noxfile.py
@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    session.install(
        "flake8",
        "flake8-bandit",
        "flake8-black",
        "flake8-bugbear",
        "flake8-import-order",
    )
    session.run("flake8", *args)

Enable the plugin warnings in Flake8’s configuration file (S like security):

# .flake8
[flake8]
select = B,B9,BLK,C,E,F,I,S,W
...

Bandit flags uses of assert to enforce interface constraints because assertions are removed when compiling to optimized byte code. You should disable this warning for your test suite, as Pytest uses assertions to verify expectations in tests:

# .flake8
[flake8]
per-file-ignores = tests/*:S101
...

Bandit finds known issues that can be detected via static file checking. If you are very concerned with security, you should consider using additional tools, for example a fuzzing tool such as python-afl.

Finding security vulnerabilities in dependencies with Safety

Safety checks the dependencies of your project for known security vulnerabilities, using a curated database of insecure Python packages. Add the following Nox session to run Safety on your project:

import tempfile


@nox.session(python="3.8")
def safety(session):
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        session.install("safety")
        session.run("safety", "check", f"--file={requirements.name}", "--full-report")

The session uses the poetry export command to convert Poetry’s lock file to a requirements file, for consumption by Safety. The standard tempfile module is used to create a temporary file for the requirements.

Include Safety in the default Nox sessions by adding it to nox.options.sessions:

# noxfile.py
nox.options.sessions = "lint", "safety", "tests"

To see how Safety works, install the infamous insecure-package:

poetry add insecure-package

Here’s what Safety has to say about this:

$ nox -rs safety

nox > Running session safety
nox > Re-using existing virtual environment at .nox/safety.
nox > poetry export --dev --format=requirements.txt --without-hashes --output=/var/folders/13/g258r36n3fd7rj0jrgln5gd04dw8n3/T/tmpkgcb549m
nox > poetry export --dev --format=requirements.txt --output=/var/folders/13/g258r36n3fd7rj0jrgln5gd04dw8n3/T/tmpyejztram
nox > pip install --constraint=/var/folders/13/g258r36n3fd7rj0jrgln5gd04dw8n3/T/tmpyejztram safety
nox > safety check --file=/var/folders/13/g258r36n3fd7rj0jrgln5gd04dw8n3/T/tmpkgcb549m --full-report
╒══════════════════════════════════════════════════════════════════════════════╕
│                                                                              │
│                               /$$$$$$            /$$                         │
│                              /$$__  $$          | $$                         │
│           /$$$$$$$  /$$$$$$ | $$  \__//$$$$$$  /$$$$$$   /$$   /$$           │
│          /$$_____/ |____  $$| $$$$   /$$__  $$|_  $$_/  | $$  | $$           │
│         |  $$$$$$   /$$$$$$$| $$_/  | $$$$$$$$  | $$    | $$  | $$           │
│          \____  $$ /$$__  $$| $$    | $$_____/  | $$ /$$| $$  | $$           │
│          /$$$$$$$/|  $$$$$$$| $$    |  $$$$$$$  |  $$$$/|  $$$$$$$           │
│         |_______/  \_______/|__/     \_______/   \___/   \____  $$           │
│                                                          /$$  | $$           │
│                                                         |  $$$$$$/           │
│  by pyup.io                                              \______/            │
│                                                                              │
╞══════════════════════════════════════════════════════════════════════════════╡
│ REPORT                                                                       │
│ checked 48 packages, using default DB                                        │
╞════════════════════════════╤═══════════╤══════════════════════════╤══════════╡
│ package                    │ installed │ affected                 │ ID       │
╞════════════════════════════╧═══════════╧══════════════════════════╧══════════╡
│ insecure-package           │ 0.1.0     │ <0.2.0                   │ 25853    │
╞══════════════════════════════════════════════════════════════════════════════╡
│ This is an insecure package with lots of exploitable security                │
│ vulnerabilities.                                                             │
╘══════════════════════════════════════════════════════════════════════════════╛
nox > Command safety check --file=/var/folders/13/g258r36n3fd7rj0jrgln5gd04dw8n3/T/tmpkgcb549m --full-report failed with exit code 255
nox > Session safety failed.

Don’t forget to uninstall this monster (just kidding, it’s an empty package flagged by Safety DB for testing purposes):

poetry remove insecure-package

Feel free to re-run Safety via Nox.

Managing dependencies in Nox sessions with Poetry

In this section, I describe how to use Poetry to manage development dependencies in your Nox sessions, and how to make your Nox sessions more reproducible.

In the first chapter, we saw that Poetry writes the exact version of each package dependency to a file named poetry.lock. The same is done for development dependencies like pytest. This is known as pinning, and it allows you to build and test your package in a predictable and deterministic way.

By contrast, this is how we have been installing packages into Nox sessions so far:

session.install("flake8")

No version specified! Nox will install whatever pip considers the latest version when the session is run. The checks may succeed when you run them on your local machine, but suddenly break on another developer’s machine or on a Continuous Integration server, due to a change to Flake8 or one of its dependencies. These things happen all the time, and the problem accumulates quickly as the dependencies of your project grow.

You could pin Flake8 using something like the following:

session.install("flake8==3.7.9")

This approach improves the situation, but it has some limitations:

  • We’re back to handling requirements manually, rather than using Poetry’s rich support for dependency management.
  • The check is still not deterministic, because dependencies of dependencies remain unpinned. (Flake8 is a good example for this: At its core, it aggregates several more specialized tools. While Flake8 protects you from breaking changes to these tools, their exact versions are still left to chance.)

How about we declare Flake8 as a development dependency of our project, like we did with Pytest in the previous chapter? Then we can benefit from Poetry as a dependency manager, and record the versions of Flake8 and its dependencies in its lock file. – Well, there is a catch. Look how we installed development dependencies in the Nox session for testing:

session.run("poetry", "install", external=True)

This command installs a bunch of things our linting session does not need:

  • the package under development
  • the package dependencies
  • unrelated development dependencies (e.g. Pytest)

A major difference between testing and linting is that you need to install your package to be able to run the test suite, but you don’t need to install your package to run linters on it. Linters are static analysis tools, they don’t need to run your program.

Wouldn’t it be great if you could install individual packages with session.install, but somehow use Poetry’s lock file to constrain their versions? Fortunately, there is a pip feature that let’s you do exactly this: constraints files. If you have used a requirements.txt file before, the format is exactly the same. And Poetry has a command to export its lock file to requirements format. So we have all the building blocks for a solution.

The function install_with_constraints below is a wrapper for session.install. It generates a constraints file by running poetry export, and passes that file to pip using its --constraint option. The function uses the standard tempfile module to create a temporary file for the constraints.

# noxfile.py
def install_with_constraints(session, *args, **kwargs):
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)

Change the Nox sessions to call the install_with_constraints wrapper instead of invoking session.install directly:

@nox.session(python="3.8")
def black(session):
    args = session.posargs or locations
    install_with_constraints(session, "black")
    session.run("black", *args)


@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    install_with_constraints(
        session,
        "flake8",
        "flake8-bandit",
        "flake8-black",
        "flake8-bugbear",
        "flake8-import-order",
    )
    session.run("flake8", *args)


@nox.session(python="3.8")
def safety(session):
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        install_with_constraints(session, "safety")
        session.run("safety", "check", f"--file={requirements.name}", "--full-report")

You can now use Poetry to manage Black, Flake8, and the other tools as development dependencies:

poetry add --dev \
    black \
    flake8 \
    flake8-bandit \
    flake8-black \
    flake8-bugbear \
    flake8-import-order \
    safety

You should also adapt the testing session. That session only needs packages required for running the test suite, and should not be cluttered by anything else. Instead of simply invoking poetry install, pass the --no-dev option. This excludes development dependencies, and installs only the package itself and its dependencies. Then install the test requirements explicitly using install_with_constraints. Here is the rewritten Nox session:

@nox.session(python=["3.8", "3.7"])
def tests(session):
    args = session.posargs or ["--cov", "-m", "not e2e"]
    session.run("poetry", "install", "--no-dev", external=True)
    install_with_constraints(
        session, "coverage[toml]", "pytest", "pytest-cov", "pytest-mock"
    )
    session.run("pytest", *args)

Your linter checks are now deterministic, and your Nox sessions benefit from Poetry’s convenient and reliable dependency management. ✌

Managing Git hooks with pre-commit

Git provides hooks which allow you to run custom commands when important actions occur, such as a commit or push. You can leverage this to run automated checks when you commit changes. pre-commit is a framework for managing and maintaining such hooks. Use it to integrate the best industry standard linters into your workflow, even those written in a language other than Python.

Install pre-commit via pip or pipx:

pip install --user --upgrade pre-commit

Configure pre-commit using the .pre-commit-config.yaml configuration file, in the top-level directory of your repository. Let’s start with the following sample configuration:

# .pre-commit-config.yaml
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: https://github.com/psf/black
    rev: 19.3b0
    hooks:
    -   id: black

Install the hooks by running the following command:

pre-commit install

The hooks run automatically every time you invoke git commit, applying checks to any newly created or modified files. When you add new hooks (like just now), you can trigger them manually for all files using the following command:

$ pre-commit run --all-files

[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Check Yaml....................................................Passed
Fix End of Files..............................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing LICENSE

Trim Trailing Whitespace......................................Passed
black.........................................................Passed

As you can see from the output, the end-of-file-fixer hook failed because the license file was missing a final newline. The hook already appended the missing newline to the file, so you can simply commit the file:

git commit --message="Fix missing newline at end of LICENSE" LICENSE

There is a problem though: The sample configuration pins Black to a specific version, and so does Poetry’s lock file. This setup requires you to keep the versions aligned manually, and can result in failed checks when the environments managed by pre-commit, Poetry, and Nox drift apart.

Let’s replace the Black entry using a repository-local hook, and run Black in the development environment created by Poetry:

# .pre-commit-config.yaml
repos:
-   repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v2.3.0
    hooks:
    -   id: check-yaml
    -   id: end-of-file-fixer
    -   id: trailing-whitespace
-   repo: local
    hooks:
    -   id: black
        name: black
        entry: poetry run black
        language: system
        types: [python]

This method allows you to rely on Poetry to manage development dependencies, without worrying about version mismatch caused by other tools.

Use the same technique to run Flake8 from the pre-commit hook:

# .pre-commit-config.yaml
-   repo: local
    hooks:
    -   id: black
        ...
    -   id: flake8
        name: flake8
        entry: poetry run flake8
        language: system
        types: [python]

The checks run somewhat faster than the corresponding Nox sessions, for two reasons:

  • They only run on files changed by the commit in question.
  • They assume that the tools are already installed.

Thanks for reading!

The next chapter is about adding type annotations and static type checking to your project. It will be published on January 22, 2020.

Continue to the next chapter


  1. The images in this chapter come from a series of futuristic pictures by Jean-Marc Côté and other artists issued in France in 1899, 1900, 1901 and 1910 (source: Wikimedia Commons via The Public Domain Review) ↩︎