An open source Python project CI pipeline
Lately I've been iterating on a reusable set of open-source continuous integration (CI) tools for my Python projects. Here's an overview, along with a sample GitHub Actions workflow file that runs these checks.
The checks below cover unit testing, code formatting, code quality, security and coverage.
They are all free and open source tools with large numbers of contributors.
Where there is configuration involved the configuration can live in your pyproject.toml
file so that it's shared between your local and CI environments.
You can copy the pipeline file at the bottom of this post directly into your repository to get started. Each of these workflow steps is intended to be built upon as your project evolves.
Some of these workflow jobs have re-usable GitHub Actions available, and I'll include links to those if they exist. However, my preference is generally to be explicit about how I'm running these jobs. This allows developers to look at the CI configuration and run the same steps locally to execute the checks themselves.
If you're not familiar with Github Actions and the Workflows syntax I recommend checking out the official guide.
When everything is green the workflow that we are working towards will end up looking like this:
Unit tests
The original purpose of a CI pipeline is to build and test your project. Since my Python projects generally don't have a build step, the initial workflow job focuses on installing dependencies, then executing the test cases against supported Python versions. Creating this workflow early in your project's lifespan, even if there are no unit tests yet, helps to establish how you will set up the project on a non-development machine.
GitHub Actions has a handy matrix
feature that allows us to run the same job against multiple versions of Python.
Let's take advantage of that to execute the tests with the last four major releases.
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install test dependencies
run: |
pip install -e '.[test]'
- name: Test with unittest
run: |
python -m unittest discover
The checkout
and setup-python
steps are going to be standard across most types of projects.
The "Install test dependencies" and "Test with unittest" steps might vary based on how your project is setup.
You just need to make sure that you have everything required for your tests to execute.
Code formatting: black
Maintained by the Python Software Foundation black
has become the standard code formatter for Python projects.
A code formatter operates differently to a traditional linter in that it actively modifies your code to meet the style guide. This can prevent arguments and bikeshedding about formatting rules and helps to keep your code feeling consistent.
Your black
settings, including files to exclude and your preferred line length should live in a shared pyproject.yaml
file.
Adding the settings there instead of using command line flags allows everyone on the team to use the same configuration as your CI server.
For example, to set a longer default line limit you can use:
[tool.black]
line-length = 120
Part of the benefit of using black
is that you shouldn't need much configuration.
Accepting the defaults is part of the advantage of using a tool like this on your project!
Locally developers can run black in a way that updates their code before committing from the root of your project:
black .
On the CI server we can use --check
to make sure that the code was correctly formatting.
The complete job ends up looking like this:
black:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install black
run: |
python -m pip install black
- name: Run black
run: |
black --check .
Black provides an official Github Actions integration (
psf/black@stable
).
Python imports: isort
Managed by the PyCQA, isort
is a utility that sorts imports within your Python files.
While it's not strictly mandatory to have your imports sorted in a specific way, keeping them tidy is good code hygiene and makes it clear to other developers where new imports should be placed.
isort
's algorithm will also collapse imports from the same module into a single import helping to ensure that you don't have duplicate items.
Once installed, you can run isort
in the root of your project to organise your imports:
isort .
Developers should be running this before committing new code and our CI pipeline needs to check that this was done.
isort
can be flipped into "check only" mode with the -c
flag.
Our action ends up looking like this:
isort:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install isort
run: |
python -m pip install isort
- name: Run isort
run: |
isort --check .
There's an extra step that we need to take for isort
.
To ensure that isort
and black
don't continually overwrite files changed by each other we need to use the Black compatibility profile that comes with isort
.
Update your pyproject.toml
with this configuration:
[tool.isort]
profile = "black"
The
isort
developers have an official Github Action available that runs isort against a project (isort/isort-action@v1
).
Code quality: ruff
Ruff is an extremely fast Python linter written in Rust (but installable with pip
) that implements much of the functionality of the popular Flake8 tool.
You could call it "the new kid on the block" but it's already seen strong adoption within a wide range of open source projects.
While a formatter like black
is great for ensuring that your code is consistent, ruff
starts to help to make sure that it's error free. It picks up things like unused variables and imports, overly complex functions, ambiguously named variables and incorrect comparisons.
On top of the Flake8 rule set, Ruff has implemented a whole host of the popular Flake8 plugins and other checks natively. You can optionally add those checks with the select
configuration option in your pyproject.toml
. While you're in the configuration file, it's important to make sure that you have the same line-length
that you are using with black
and isort
.
[tool.ruff]
# Enable Pyflakes and pycodestyle rules.
select = ["E", "F"]
line-length=120
Ruff maintains compatibility with black
and isort
so you shouldn't have any issues when running all three.
You can run ruff
locally with the optional --fix
flag to fix some of the issues flagged in place:
ruff --fix .
Ruff provides a handy formatting option for GitHub Actions that you should take advantage of in your workflow job:
ruff:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install ruff
run: |
python -m pip install ruff
- name: Run ruff
run: |
ruff --format=github .
Security checks: bandit
Another PyCQA project, bandit
is a tool that helps to discover common security issues in your project. It produces a report of potential issues by both severity and confidence.
While it's not a replacement for security focussed code reviews, audits or penetration tests, it does provide a great first line of defence that can be completely automated.
Locally, you can run bandit
from the root of your project with the -r
flag to recursively find Python files:
bandit -r .
Like the other tools we're using, bandit
uses your pyproject.toml
file for configuration when you use the -c
flag. In most cases you won't need to add any excludes. For larger projects you should take advantage of the option of creating a baseline report and comparing your results to that.
In our workflow job, we run things the exact same way we would locally:
bandit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3
- name: Install bandit
run: |
python -m pip install bandit[toml]
- name: Run bandit scan
run: |
bandit -c pyproject.toml -r .
Code coverage: coverage.py
Test coverage and, in particular, test coverage targets, is a contentious issue. You might not want to fail your build because of a specific test coverage outcome, but I strongly believe it's important to track project coverage so that you can see how things are changing.
Coverage.py provides a way to measure coverage with minimal overhead and configuration.
Instead of running your test suite directly, you run it with coverage run
.
In our case, we are using unittest
, so running locally with coverage looks like this:
coverage run -m unittest discover
You can then run the report
command to get a command-line output of the coverage statistics. The -m
flag shows you which line numbers are missing coverage:
coverage report -m
Running in our CI environment is just a slight modification of our test
job above. We won't need to run the coverage checks against multiple Python versions, so the matrix
part can be removed. Since there's not much point having a CI step that never fails, we'll use --fail-under=95
to fail.
coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install test dependencies
run: |
pip install -e '.[test]'
- name: Install coverage.py
run: |
pip install coverage
- name: Test with unittest
run: |
coverage run -m unittest discover
coverage report -m --fail-under=95
The final workflow
With these steps, we're able to achieve the following:
- Running tests in our CI environment and failing the run if there are any failures
- Checking that our code is formatted to our team's coding standards
- Ensuring that imports are correctly ordered and grouped
- Linting our code against an ever-growing set of rules that helps check for errors
- Picking up potential security issues early and breaking the pipeline if they are detected
- Keeping the test coverage of our project high and checking it before merging new code
All configuration for our checks lives in our pyproject.toml
file, which helps make our final workflow reusable.
Here's the complete GitHub Action yaml file (place this in .github/workflows/checks.yml
or similar):
name: Python Checks
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
workflow_dispatch:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Install test dependencies
run: |
pip install -e '.[test]'
- name: Test with unittest
run: |
python -m unittest discover
black:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install black
run: |
python -m pip install black
- name: Run black
run: |
black --check .
isort:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install isort
run: |
python -m pip install isort
- name: Run isort
run: |
isort --check .
ruff:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install ruff
run: |
python -m pip install ruff
- name: Run ruff
run: |
ruff --format=github .
bandit:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: 3
- name: Install bandit
run: |
python -m pip install bandit[toml]
- name: Run bandit scan
run: |
bandit -c pyproject.toml -r .
coverage:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install test dependencies
run: |
pip install -e '.[test]'
- name: Install coverage.py
run: |
pip install coverage
- name: Test with unittest
run: |
coverage run -m unittest discover
coverage report -m --fail-under=95
If you're interested, you can see this complete pipeline in action on my thttp project.
Discussion
This article was shared elsewhere, here are the discussion threads:
(Feel free to share links to other discussion threads)