Hypermodern Python Chapter 4: Typing

Read this article on Medium

In this fourth installment of the Hypermodern Python series, I’m going to discuss how to add type annotations and type checking to your project.¹ Previously, we discussed how to add linting, static analysis, and code formatting. (If you start reading here, you can also download the code for the previous chapter.)

Here are the topics covered in this chapter on Typing in Python:

• Type annotations and type checkers
• Static type checking with mypy
• Static type checking with pytype
• Adding type annotations to the package
• Data validation using Desert and Marshmallow
• Runtime type checking with Typeguard
• Increasing type coverage with flake8-annotations
• Adding type annotations to Nox sessions
• Adding type annotations to the test suite

Here is a full list of the articles in this series:

• Chapter 1: Setup
• Chapter 2: Testing
• Chapter 3: Linting
• Chapter 4: Typing (this article)
• Chapter 5: Documentation
• Chapter 6: CI/CD

This guide has a companion repository: cjolowicz/hypermodern-python. Each article in the guide corresponds to a set of commits in the GitHub repository:

• View changes
• Download code

Type annotations and type checkers

Type annotations, first introduced in Python 3.5, are a way to annotate functions and variables with types. Combined with tooling that understands them, they can make your programs easier to understand, debug, and maintain. Here are two simple examples of type annotations:

# This is a variable holding an integer.
answer: int = 42


# This is a function which accepts and returns an integer.
def increment(number: int) -> int:
    return number + 1

The Python runtime does not enforce type annotations. Python is a dynamically typed language: it only verifies the types of your program at runtime, and uses duck typing to do so (“if it walks and quacks like a duck, it is a duck”). A static type checker, by contrast, can use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed.

The introduction of type annotations has paved the way for an entire generation of static type checkers: mypy can be considered the pioneer and de facto reference implementation of Python type checking; several core Python developers are involved in its development. Google, Facebook, and Microsoft have each announced their own static type checkers for Python: pytype, pyre, and pyright, respectively. JetBrain’s Python IDE PyCharm also ships with a static type checker. In this chapter, we introduce mypy and pytype. We also take a look at Typeguard, a runtime type checker.

Static type checking with mypy

Add mypy as a development dependency:

poetry add --dev mypy

Add the following Nox session to type-check your source code with mypy:

@nox.session(python=["3.8", "3.7"])
def mypy(session):
    args = session.posargs or locations
    install_with_constraints(session, "mypy")
    session.run("mypy", *args)

Include mypy in the default Nox sessions:

nox.options.sessions = "lint", "mypy", "tests"

Run the Nox session using the following command:

nox -rs mypy

Mypy raises an error if it cannot find any type definitions for a Python package used by your program. Unless you are going to write these type definitions yourself, you should disable the error using the mypy.ini configuration file:

# mypy.ini
[mypy]

[mypy-nox.*,pytest]
ignore_missing_imports = True

Specifying the packages explicitly helps you keep track of dependencies that are currently out of scope of the type checker. You may soon be able to cut down on this list, as many projects are actively working on typing support.

Static type checking with pytype

Add pytype as a development dependency, for Python 3.7 only:

poetry add --dev --python=3.7 pytype

Add the following Nox session to run pytype:

# noxfile.py
@nox.session(python="3.7")
def pytype(session):
    """Run the static type checker."""
    args = session.posargs or ["--disable=import-error", *locations]
    install_with_constraints(session, "pytype")
    session.run("pytype", *args)

In this session, we use the command-line option --disable=import-error because pytype, like mypy, reports import errors for third-party packages without type annotations.

Run the Nox session using the following command:

nox -rs pytype

Update nox.options.session to include static type checking with pytype by default:

nox.options.sessions = "lint", "mypy", "pytype", "tests"

Adding type annotations to the package

Let’s add some type annotations to the package, starting with console.main. Don’t be distracted by the decorators applied to it: This is a simple function accepting a str, and returning None by “falling off its end”:

# src/hypermodern_python/console.py
def main(language: str) -> None: ...

In the wikipedia module, the API_URL constant is a string:

API_URL: str = "https://{language}.wikipedia.org/api/rest_v1/page/random/summary"

The wikipedia.random_page function accepts an optional parameter of type str:

# src/hypermodern_python/wikipedia.py
def random_page(language: str = "en"): ...

The wikipedia.random_page function returns the JSON object received from the Wikipedia API. JSON objects are represented in Python using built-in types such as dict, list, str, and int. Due to the recursive nature of JSON objects, their type is still hard to express in Python, and is usually given as Any:

# src/hypermodern_python/wikipedia.py
from typing import Any


def random_page(language: str = "en") -> Any: ...

You can think of the enigmatic Any type as a box which can hold any type on the inside, but behaves like all of the types on the outside. It is the most permissive kind of type you can apply to a variable, parameter, or return type in your program. Contrast this with object, which can also hold values of any type, but only supports the minimal interface that is common to all of them.

Data validation using Desert and Marshmallow

Returning Any is unsatisfactory, because we know quite precisely which JSON structures we can expect from the Wikipedia API. An API contract is not a type guarantee, but you can turn it into one by validating the data you receive. This will also demonstrate some great ways to use type annotations at runtime.

The first step is to define the target type for the validation. Currently, the function should return a dictionary with several keys, of which we are only interested in title and extract. But your program can do better than operating on a dictionary: Using dataclasses from the standard library, you can define a fully-featured data type in a concise and straightforward manner. Let’s define a wikipedia.Page type for our application:

# src/hypermodern_python/wikipedia.py
from dataclasses import dataclass


@dataclass
class Page:
    title: str
    extract: str

We are going to return this data type from wikipedia.random_page:

# src/hypermodern_python/wikipedia.py
def random_page(language: str = "en") -> Page: ...

Data types have a beneficial influence on the structure of code bases. You can see this by adapting console.main to use wikipedia.Page instead of the raw dictionary. The body turns into a clear and concise three-liner:

# src/hypermodern_python/console.py
def main(language: str) -> None:
    """The hypermodern Python project."""
    page = wikipedia.random_page(language=language)

    click.secho(page.title, fg="green")
    click.echo(textwrap.fill(page.extract))

Of course, we are still missing the actual code to create wikipedia.Page objects. Let’s start with a test case, performing a simple runtime type check:

# tests/test_wikipedia.py
def test_random_page_returns_page(mock_requests_get):
    page = wikipedia.random_page()
    assert isinstance(page, wikipedia.Page)

How are we going to turn JSON into wikipedia.Page objects? Enter Marshmallow: This library allows you to define schemas to serialize, deserialize and validate data. Used by countless applications, Marshmallow has also spawned an ecosystem of tools and libraries built on top of it. One of these libraries, Desert, uses the type annotations of dataclasses to generate serialization schemas for them. (Another great data validation library using type annotations is typical.)

Add Desert and Marshmallow to your dependencies:

poetry add desert marshmallow

You need to add the libraries to mypy.ini to avoid import errors:

# mypy.ini
[mypy-desert,marshmallow,nox.*,pytest]
ignore_missing_imports = True

The basic usage of Desert is shown in the following example:

# Generate a schema from a dataclass.
schema = desert.schema(Page)

# Use the schema to load data.
page = schema.load(data)

Our data type represents the Wikipedia page resource only partially. Marshmallow errs on the side of safety and raises a validation error when encountering unknown fields. However, you can tell it to ignore unknown fields via the meta keyword. Add the schema as a module-level variable:

# src/hypermodern_python/wikipedia.py
import desert
import marshmallow


schema = desert.schema(Page, meta={"unknown": marshmallow.EXCLUDE})

Using the schema, we are ready to implement wikipedia.random_page:

# src/hypermodern_python/wikipedia.py
def random_page(language: str = "en") -> Page:
    ...
    with requests.get(url) as response:
        response.raise_for_status()
        data = response.json()
        return schema.load(data)

Let’s also handle validation errors gracefully by converting them to ClickException, as we did in chapter 2 for request errors. For example, suppose that Wikipedia responds with a body of “null” instead of an actual resource, due to a fictitious bug.

We can turn this scenario into a test case by configuring the requests.get mock to produce None as the JSON object, and using pytest.raises to check for the correct exception:

# tests/test_wikipedia.py
def test_random_page_handles_validation_errors(mock_requests_get: Mock) -> None:
    mock_requests_get.return_value.__enter__.return_value.json.return_value = None
    with pytest.raises(click.ClickException):
        wikipedia.random_page()

Getting this to pass is a simple matter of adding marshmallow.ValidationError to the except clause:

# src/hypermodern_python/wikipedia.py
except (requests.RequestException, marshmallow.ValidationError) as error:

This completes the implementation. Here is the wikipedia module with type annotations and validation:

# src/hypermodern_python/wikipedia.py
from dataclasses import dataclass

import click
import desert
import marshmallow
import requests


API_URL: str = "https://{language}.wikipedia.org/api/rest_v1/page/random/summary"


@dataclass
class Page:
    title: str
    extract: str


schema = desert.schema(Page, meta={"unknown": marshmallow.EXCLUDE})


def random_page(language: str = "en") -> Page:
    url = API_URL.format(language=language)

    try:
        with requests.get(url) as response:
            response.raise_for_status()
            data = response.json()
            return schema.load(data)
    except (requests.RequestException, marshmallow.ValidationError) as error:
        message = str(error)
        raise click.ClickException(message)

Runtime type checking with Typeguard

Typeguard is a runtime type checker for Python: It checks that arguments match parameter types of annotated functions as your program is being executed (and similarly for return values). Runtime type checking can be a valuable tool when it is impossible or impractical to strictly type an entire code path, for example when crossing system boundaries or interfacing with other libraries.

Here is an example of a type-related bug which can be caught by a runtime type checker, but is not detected by mypy or pytype because the incorrectly typed argument is loaded from JSON. (Do not add this test function to your test suite permanently; it’s for demonstration purposes only.)

# tests/test_wikipedia.py
def test_trigger_typeguard(mock_requests_get):
    import json

    data = json.loads('{ "language": 1 }')
    wikipedia.random_page(language=data["language"])

Add Typeguard to your development dependencies:

poetry add --dev typeguard

Typeguard comes with a Pytest plugin which instruments your package for type checking while you run the test suite. You can enable it by passing the --typeguard-packages option with the name of your package. Add a Nox session to run the test suite with runtime type checking:

# noxfile.py
package = "hypermodern_python"


@nox.session(python=["3.8", "3.7"])
def typeguard(session):
    args = session.posargs or ["-m", "not e2e"]
    session.run("poetry", "install", "--no-dev", external=True)
    install_with_constraints(session, "pytest", "pytest-mock", "typeguard")
    session.run("pytest", f"--typeguard-packages={package}", *args)

Run the Nox session:

nox -rs typeguard

Typeguard catches the bug we introduced at the start of this section. You will also notice a warning about missing type annotations for a Click object. This is due to the fact that console.main is wrapped by a decorator, and its type annotations only apply to the inner function, not the resulting object as seen by the test suite.

Increasing type coverage with flake8-annotations

flake8-annotations is a Flake8 plugin that detects the absence of type annotations for functions, helping you keep track of unannotated code.

Add the plugin to your development dependencies:

poetry add --dev flake8-annotations

Install the plugin into the linting session:

# noxfile.py
@nox.session(python=["3.8", "3.7"])
def lint(session):
    args = session.posargs or locations
    install_with_constraints(
        session,
        "flake8",
        "flake8-annotations",
        "flake8-bandit",
        "flake8-black",
        "flake8-bugbear",
        "flake8-import-order",
    )
    session.run("flake8", *args)

Configure Flake8 to switch on the warnings generated by the plugin (ANN like annotations):

# .flake8
[flake8]
select = ANN,B,B9,BLK,C,E,F,I,S,W

Run the lint session:

nox -rs lint

The plugin spews out a multitude of warnings about missing type annotations in the Nox sessions and the test suite. It is possible to disable warnings for these locations using Flake8’s per-file-ignores option:

# .flake8
[flake8]
per-file-ignores =
    tests/*:S101,ANN
    noxfile.py:ANN

Adding type annotations to Nox sessions

In this section, I show you how to add type annotations to Nox sessions. If you disabled type coverage warnings (ANN) for Nox sessions, re-enable them for the purposes of this section.

The central type for Nox sessions is nox.sessions.Session, which is also the first and only argument of your session functions. The return value of these functions is None—the implicit return value of every Python function that does not explicitly return anything. Annotate your session functions like this:

# noxfile.py
from nox.sessions import Session


def black(session: Session) -> None: ...

def lint(session: Session) -> None: ...

def safety(session: Session) -> None: ...

def mypy(session: Session) -> None: ...

def pytype(session: Session) -> None: ...

def tests(session: Session) -> None: ...

def typeguard(session: Session) -> None: ...

Our helper function install_with_constraints is a wrapper for Session.install. The positional arguments of this function are the command-line arguments for pip, so their type is str. The keyword arguments are the same as for Session.run. Instead of listing their types individually, we’ll be pragmatic and type them as Any:

# noxfile.py
def install_with_constraints(session: Session, *args: str, **kwargs: Any) -> None: ...

Adding type annotations to the test suite

In this section, I show you how to add type annotations to the test suite. If you disabled type coverage warnings (ANN) for the test suite, re-enable them for the purposes of this section.

Test functions in Pytest use parameters to declare fixtures they use. Typing them is simpler than it may seem: You don’t specify the type of the fixture itself, but the type of the value that the fixture supplies to the test function. For example, the mock_requests_get fixture returns a standard mock object of type unittest.mock.Mock. (The actual type is unittest.mock.MagicMock, but you can use the more general type to annotate your test functions.)

Let’s start by annotating the test functions in the wikipedia test module:

# tests/test_wikipedia.py
from unittest.mock import Mock


def test_random_page_uses_given_language(mock_requests_get: Mock) -> None: ...

def test_random_page_returns_page(mock_requests_get: Mock) -> None: ...

def test_random_page_handles_validation_errors(mock_requests_get: Mock) -> None: ...

Next, let’s annotate mock_requests_get itself. The return type of this function is the same unittest.mock.Mock. The function takes a single argument, the mocker fixture from pytest-mock, which is of type pytest_mock.MockFixture:

# tests/conftest.py
from unittest.mock import Mock

from pytest_mock import MockFixture


def mock_requests_get(mocker: MockFixture) -> Mock: ...

Configure mypy to ignore the missing import for pytest_mock:

# mypy.ini
[mypy-desert,marshmallow,nox.*,pytest,pytest_mock]
ignore_missing_imports = True

Use the same type signature for the mock fixture in the console test module:

# tests/test_console.py
from unittest.mock import Mock

from pytest_mock import MockFixture


def mock_wikipedia_random_page(mocker: MockFixture) -> Mock: ...

The test module for console also defines a simple fixture returning a click.testing.CliRunner:

# tests/test_console.py
from click.testing import CliRunner


def runner() -> CliRunner: ...

Typing the test functions for console is now straightforward:

# tests/test_console.py
from unittest.mock import Mock

from click.testing import CliRunner
from pytest_mock import MockFixture


def test_main_succeeds(runner: CliRunner, mock_requests_get: Mock) -> None: ...

def test_main_succeeds_in_production_env(runner: CliRunner) -> None: ...

def test_main_prints_title(runner: CliRunner, mock_requests_get: Mock) -> None: ...

def test_main_invokes_requests_get(runner: CliRunner, mock_requests_get: Mock) -> None: ...

def test_main_uses_en_wikipedia_org(runner: CliRunner, mock_requests_get: Mock) -> None: ...

def test_main_uses_specified_language(runner: CliRunner, mock_wikipedia_random_page: Mock) -> None: ...

def test_main_fails_on_request_error(runner: CliRunner, mock_requests_get: Mock) -> None: ...

def test_main_prints_message_on_request_error(runner: CliRunner, mock_requests_get: Mock) -> None: ...

The missing piece to achieve full type coverage (as far as the Flake8 annotations plugin is concerned) is the pytest_configure hook. The function takes a Pytest configuration object as its single parameter. Unfortunately, the type of this object is not (yet) part of Pytest’s public interface. You have the choice of using Any or reaching into Pytest’s internals to import _pytest.config.Config. Let’s opt for the latter:

# tests/conftest.py
from _pytest.config import Config


def pytest_configure(config: Config) -> None: ...

You also need to tell mypy to ignore missing imports for _pytest.*:

# mypy.ini
[mypy-desert,marshmallow,nox.*,pytest,pytest_mock,_pytest.*]
ignore_missing_imports = True

This concludes our foray into the Python type system. Type annotations make your programs easier to understand, debug, and maintain. Static type checkers use type annotations and type inference to verify the type correctness of your program without executing it, helping you discover many bugs that would otherwise go unnoticed. An increasing number of libraries leverage the power of type annotations at runtime, for example to validate data. Happy typing!

Thanks for reading!

The next chapter is about adding documentation for your project.