An open source Python project CI pipeline

Lately I've been iterating on a reusable set of open-source continuous integration (CI) tools for my Python projects. Here's an overview, along with a sample GitHub Actions workflow file that runs these checks.

The checks below cover unit testing, code formatting, code quality, security and coverage. They are all free and open source tools with large numbers of contributors. Where there is configuration involved the configuration can live in your pyproject.toml file so that it's shared between your local and CI environments.

You can copy the pipeline file at the bottom of this post directly into your repository to get started. Each of these workflow steps is intended to be built upon as your project evolves.

Some of these workflow jobs have re-usable GitHub Actions available, and I'll include links to those if they exist. However, my preference is generally to be explicit about how I'm running these jobs. This allows developers to look at the CI configuration and run the same steps locally to execute the checks themselves.

If you're not familiar with Github Actions and the Workflows syntax I recommend checking out the official guide.

When everything is green the workflow that we are working towards will end up looking like this:

Unit tests

The original purpose of a CI pipeline is to build and test your project. Since my Python projects generally don't have a build step, the initial workflow job focuses on installing dependencies, then executing the test cases against supported Python versions. Creating this workflow early in your project's lifespan, even if there are no unit tests yet, helps to establish how you will set up the project on a non-development machine.

GitHub Actions has a handy matrix feature that allows us to run the same job against multiple versions of Python. Let's take advantage of that to execute the tests with the last four major releases.

test:
  runs-on: ubuntu-latest
  strategy:
    matrix:
      python-version: ["3.8", "3.9", "3.10", "3.11"]

  steps:
  - uses: actions/checkout@v3

  - name: Set up Python ${{ matrix.python-version }}
    uses: actions/setup-python@v4
    with:
      python-version: ${{ matrix.python-version }}

  - name: Install test dependencies
    run: |
      pip install -e '.[test]'

  - name: Test with unittest
    run: |
      python -m unittest discover

The checkout and setup-python steps are going to be standard across most types of projects. The "Install test dependencies" and "Test with unittest" steps might vary based on how your project is setup. You just need to make sure that you have everything required for your tests to execute.

Code formatting: black

Maintained by the Python Software Foundation black has become the standard code formatter for Python projects.

A code formatter operates differently to a traditional linter in that it actively modifies your code to meet the style guide. This can prevent arguments and bikeshedding about formatting rules and helps to keep your code feeling consistent.

Your black settings, including files to exclude and your preferred line length should live in a shared pyproject.yaml file. Adding the settings there instead of using command line flags allows everyone on the team to use the same configuration as your CI server.

For example, to set a longer default line limit you can use:

[tool.black]
line-length = 120

Part of the benefit of using black is that you shouldn't need much configuration. Accepting the defaults is part of the advantage of using a tool like this on your project!

Locally developers can run black in a way that updates their code before committing from the root of your project:

black .

On the CI server we can use --check to make sure that the code was correctly formatting. The complete job ends up looking like this:

black:
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install black
      run: |
        python -m pip install black

    - name: Run black
      run: |
        black --check .

Black provides an official Github Actions integration (psf/black@stable).

Python imports: isort

Managed by the PyCQA, isort is a utility that sorts imports within your Python files. While it's not strictly mandatory to have your imports sorted in a specific way, keeping them tidy is good code hygiene and makes it clear to other developers where new imports should be placed. isort's algorithm will also collapse imports from the same module into a single import helping to ensure that you don't have duplicate items.

Once installed, you can run isort in the root of your project to organise your imports:

isort .

Developers should be running this before committing new code and our CI pipeline needs to check that this was done. isort can be flipped into "check only" mode with the -c flag.

Our action ends up looking like this:

isort:
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install isort
      run: |
        python -m pip install isort

    - name: Run isort
      run: |
        isort --check .

There's an extra step that we need to take for isort. To ensure that isort and black don't continually overwrite files changed by each other we need to use the Black compatibility profile that comes with isort.

Update your pyproject.toml with this configuration:

[tool.isort]
profile = "black"

The isort developers have an official Github Action available that runs isort against a project (isort/isort-action@v1).

Code quality: ruff

Ruff is an extremely fast Python linter written in Rust (but installable with pip) that implements much of the functionality of the popular Flake8 tool. You could call it "the new kid on the block" but it's already seen strong adoption within a wide range of open source projects.

While a formatter like black is great for ensuring that your code is consistent, ruff starts to help to make sure that it's error free. It picks up things like unused variables and imports, overly complex functions, ambiguously named variables and incorrect comparisons.

On top of the Flake8 rule set, Ruff has implemented a whole host of the popular Flake8 plugins and other checks natively. You can optionally add those checks with the select configuration option in your pyproject.toml. While you're in the configuration file, it's important to make sure that you have the same line-length that you are using with black and isort.

[tool.ruff]
# Enable Pyflakes and pycodestyle rules.
select = ["E", "F"]
line-length=120

Ruff maintains compatibility with black and isort so you shouldn't have any issues when running all three.

You can run ruff locally with the optional --fix flag to fix some of the issues flagged in place:

ruff --fix .

Ruff provides a handy formatting option for GitHub Actions that you should take advantage of in your workflow job:

ruff:
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install ruff
      run: |
        python -m pip install ruff

    - name: Run ruff
      run: |
        ruff --format=github .

Security checks: bandit

Another PyCQA project, bandit is a tool that helps to discover common security issues in your project. It produces a report of potential issues by both severity and confidence.

While it's not a replacement for security focussed code reviews, audits or penetration tests, it does provide a great first line of defence that can be completely automated.

Locally, you can run bandit from the root of your project with the -r flag to recursively find Python files:

bandit -r .

Like the other tools we're using, bandit uses your pyproject.toml file for configuration when you use the -c flag. In most cases you won't need to add any excludes. For larger projects you should take advantage of the option of creating a baseline report and comparing your results to that.

In our workflow job, we run things the exact same way we would locally:

bandit:
  runs-on: ubuntu-latest

  steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: 3

    - name: Install bandit
      run: |
        python -m pip install bandit[toml]

    - name: Run bandit scan
      run: |
        bandit -c pyproject.toml -r .

Code coverage: coverage.py

Test coverage and, in particular, test coverage targets, is a contentious issue. You might not want to fail your build because of a specific test coverage outcome, but I strongly believe it's important to track project coverage so that you can see how things are changing.

Coverage.py provides a way to measure coverage with minimal overhead and configuration. Instead of running your test suite directly, you run it with coverage run. In our case, we are using unittest, so running locally with coverage looks like this:

coverage run -m unittest discover

You can then run the report command to get a command-line output of the coverage statistics. The -m flag shows you which line numbers are missing coverage:

coverage report -m

Running in our CI environment is just a slight modification of our test job above. We won't need to run the coverage checks against multiple Python versions, so the matrix part can be removed. Since there's not much point having a CI step that never fails, we'll use --fail-under=95 to fail.

coverage:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3

    - name: Set up Python
      uses: actions/setup-python@v4
      with:
        python-version: "3.11"

    - name: Install test dependencies
      run: |
        pip install -e '.[test]'

    - name: Install coverage.py
      run: |
        pip install coverage

    - name: Test with unittest
      run: |
        coverage run -m unittest discover
        coverage report -m --fail-under=95

The final workflow

With these steps, we're able to achieve the following:

• Running tests in our CI environment and failing the run if there are any failures
• Checking that our code is formatted to our team's coding standards
• Ensuring that imports are correctly ordered and grouped
• Linting our code against an ever-growing set of rules that helps check for errors
• Picking up potential security issues early and breaking the pipeline if they are detected
• Keeping the test coverage of our project high and checking it before merging new code

All configuration for our checks lives in our pyproject.toml file, which helps make our final workflow reusable. Here's the complete GitHub Action yaml file (place this in .github/workflows/checks.yml or similar):

name: Python Checks

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]

    steps:
    - uses: actions/checkout@v3

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install test dependencies
      run: |
        pip install -e '.[test]'

    - name: Test with unittest
      run: |
        python -m unittest discover

  black:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"

      - name: Install black
        run: |
          python -m pip install black

      - name: Run black
        run: |
          black --check .

  isort:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"

      - name: Install isort
        run: |
          python -m pip install isort

      - name: Run isort
        run: |
          isort --check .

  ruff:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"

      - name: Install ruff
        run: |
          python -m pip install ruff

      - name: Run ruff
        run: |
          ruff --format=github .

  bandit:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: 3

      - name: Install bandit
        run: |
          python -m pip install bandit[toml]

      - name: Run bandit scan
        run: |
          bandit -c pyproject.toml -r .

  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.11"

      - name: Install test dependencies
        run: |
          pip install -e '.[test]'

      - name: Install coverage.py
        run: |
          pip install coverage

      - name: Test with unittest
        run: |
          coverage run -m unittest discover
          coverage report -m --fail-under=95

If you're interested, you can see this complete pipeline in action on my thttp project.

Discussion

This article was shared elsewhere, here are the discussion threads:

• Lobste.rs
• /r/Python

(Feel free to share links to other discussion threads)