Python

Programming

Data Science

Software Development

Machine Learning

Python Project Setup: A Step-by-Step Guide to Industry Best Practices

Learn how to properly setup and kickstart your python project. And all the nitty-gritty details of which tools to use and how to configure them together, through building a simple CLI.

Armand Sauzay

16 min readJun 5, 2023

As you start working on your python project, you’ll likely need to set it up in a consistent and collaboration-friendly way. In this article, I’ll describe a setup that works great for our projects at Turo, and also my personal ones. It includes many industry best practices (semantic versioning, pre-commit, linting, or how to release code to mention but a few).

In short, we’ll cover how to properly install python, structure your code, run automated checks both on local with pre-commit and the server side with GitHub Actions (aka GHA).

This article covers quite a few topics. To ease the reading, each section will usually be split in two parts:

📚 for the theory part
🛠️ for the practical part (i.e. the commands you need to run to reproduce).

Sometimes a 💡 section will also indicate a coding tip or trick.

Install a Python version manage (pyenv)
Choose an environment manager (poetry)
Otherwise, use Docker to isolate dependencies
Add Some Code
Write Some Tests
Linting our code
Automate checks on local with pre-commit
Automate checks on remote with GitHub Actions
Automate our release with GitHub Actions
Enjoy the benefits of your new code practices

1. Install a Python version manager (pyenv)

📚 The first thing we’ll need to do is to install a python version manager which will enable us to have multiple versions of python installed on our machine and switch between them easily.

To illustrate this, let’s say we have a project that requires python 3.6 and another project that requires python 3.7. If we only had python 3.7 installed on our machine, we would have had to uninstall it and install python 3.6 every time we want to work on the first project… This is where a python version manager comes in handy.

The most commonly used python version manager is pyenv (follow the installation instructions).

🛠️ We can start by installing pyenv with the following command:

# default installation
> curl https://pyenv.run | bash

If we want more control over the installation, on MacOS:

#install pyenv using brew 
brew install pyenv

#add pyenv to your $PATH
#if your terminal is using bash
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc 
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc 
echo 'eval "$(pyenv init -)"' >> ~/.bashrc

#if your terminal is using zshell
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc 
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc 
echo 'eval "$(pyenv init -)"' >> ~/.zshrc

💡 If you’re not sure which shell you’re using, you can run the following command:

> echo $SHELL
/bin/zsh

2. Choose an environment manager (poetry)

📚 Once we have an python version manager (PVM), next step is to choose our environment manager (EM). EMs enable us to create isolated environments for our projects and work hands in hands with our PVMs, which provide the right python version. In python, as you might already know, pip is the default package installer and venv is the default environment manager. While pip will be used to install packages, EMs (like venv or poetry) enable us to isolate our package dependencies and thus allow others to reproduce/re-run our code.

To illustrate this, let’s say we have a project that requires pandas and matplotlib and another project that requires tensorflow to do some ML training. Not only is it considered a good practice to create a separate environment for each project but it will also enable us to reduce to a minimum the set of dependencies (and reduce the image size to a minimum in case our project ends up in “production”). In short, a dependency manager will isolate the packages needed for each project and enable us to keep track of our the environment needed to run our code.

There are many environment managers (pipenv, conda, virtualenv, python built-in venv, etc…). Currently there preferences diverge depending on the programmers, but poetry definitely checks a few boxes. I use it both at Turo and for personal projects.

In general, our environment manager will be installed within our specific python version. For instance, if we use pyenv, we can install poetry by running the following command:

> pyenv install 3.10.6
> pyenv global 3.10.6
> pip install poetry

In this case, poetry will live in our specific python environment (say 3.10.4). This means that we can have a different version of poetry for each python version we have installed on our machine.

🛠️ Pip installing poetry in our python environment with pip is one way to do it but we can also follow the steps below:

# Use curl to install poetry
> curl -sSL https://install.python-poetry.org | python3 -

# add poetry to your path. On MacOS, poetry is added at 
# ~/Library/Application Support/pypoetry/venv/bin/poetry. 
# So we can add it to our PATH by adding the following line 
# to our ~/.bashrc or ~/.zshrc file:
> export PATH="$HOME/Library/Application Support/pypoetry/venv/bin:$PATH"

If you are using a different shell, you can find the path to poetry by running the following command:

> poetry config --list --local | grep virtualenvs.path

And then add it to $PATH.

3. Otherwise, use Docker to isolate dependencies

📚 If you don’t want to install a python version manager and an environment manager, or want to abstract this for your team, you can use Docker instead. Creating a docker image with the right libraries or using VSCode and its devcontainers are the most popular option. I’ll keep this for a subsequent article since it is quite a topic on its own. But if you’ve already seen a .devcontainer file on a github repository and you’re not sure what it is, you can read the documentation above.

4. Add Some Code

💡 You can find the code for this section at armand-sauzay/hello-world-cli

📚 Now that we’ve seen how we can install a python version manager to manage different python versions and a package manager to manage dependencies, let’s get our hands on actual coding. For illustration purposes we’ll create a simple CLI that prints “Hello <name>" when we use it and give how many repositories <name> has on GitHub.

But before going into the code, let’s talk a bit about the code structure of the project:

.github folder: this is a folder for the GitHub actions for our project. We’ll cover this section in depth in the Github Action section below for Linting and Releasing our code.

hello_world_cli folder 💡: by default, poetry wants us to create a package with a folder named as our git repository (with the only difference that it wants underscores instead of dashes). So if our git repository is called python-project-template, poetry will create or expects a folder called python_project_template.

Image by author — Script to rule them all

script folder 💡. Follows the Scripts To Rule Them All best practice: https://github.com/github/scripts-to-rule-them-all. The idea is to have all the scripts that one needs to get up to speed with our project. We can think of it as a sort of standardized Makefile but without having to get the additional make dependency. And using shell instead of make, which probably everyone with hands on coding has some notions of. Note that this folder comes from the template repository: armand-sauzay/python-template.

test folder: it is a best practice to put tests in a tests folder. It makes the interaction with pytest easy as well. We’ll cover this in depth in the testing section below

Lint and Release configuration files:

.commitlintrc.yaml: extends @commitlint/config-conventional meaning it takes the default commitlint configuiration (with “fix: …” and “feat: …” commit messages
.flake8: configuration file for flake8, a python linter, see below for more details on linting.
.isort.cfg: configuration file for isort.
.pre-commit-config.yaml: list of pre-commit-hooks to run when invoking pre-commit
.releaserc.json: configuration for plugins on how to release our code. See Release section below for more details.

.python-version : pyenv uses shims so if we type which python in a terminal, we’ll see /Users/<your-username>/.pyenv/shims/python . And what shims does in short is to look at the current directory. And see if it can find a .python-version file. And if it does it uses the version that exists inside it.

pyproject.toml describes the dependencies of our project. You can think of it as a requirements.txt file.
poetry.lock keeps the exact version of the packages installed. You can think of it as a pip freeze.

OK… A bit of information, right? Don’t worry, by the end of the article all of the files above should start to make sense. Without further ado, let’s start creating the repo and setting it up.

🛠️

# Use gh to create a new repository called hello-world-cli and clone it 
# to our machine
> gh repo create <your-github-username>/hello-world-cli --template armand-sauzay/python-template --public

# clone the repo on local
> git clone https://github.com/<your-username>/hello-world-cli.git
Cloning into 'hello-world-cli'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (29/29), done.
remote: Total 38 (delta 6), reused 28 (delta 3), pack-reused 0
Receiving objects: 100% (38/38), 19.01 KiB | 299.00 KiB/s, done.
Resolving deltas: 100% (6/6), done.

# go to the folder
> cd hello-world-cli

# run the ./script/bootstrap that comes from the template repo
# to initialize the repo
> ./script/bootstrap

# add the requests package to the project
> poetry add requests

💡armand-sauzay/python-template as its name suggests is a template for kickstarting a python project. It will put all the files discussed above (.pre-commmit-config.yaml, .releaserc.json, .flake8, .isort.cfg etc) in the right place so you can focus on creating some code and not on the tedious part of setting up your linters or your release process.

💡 If we don’t have gh installed we can brew install it by running: brew install gh

So, in case you’re a bit lost, so far, we ran the following commands:

> pyenv install 3.10.6 # install python 3.10.6
> pyenv global 3.10.6 # (optional) set python 3.10.6 as the default python version
> pip install poetry # install poetry as a python package in your python 3.10.6 environment
<...commands in the last codeblock to create and clone the repository ...>
> poetry add requests # add the requests package to your project

Then, in the hello_world_cli folder, we can create a file called main.py and add the following code to it:

"""CLI for the hello_world_cli package."""
import argparse

import requests


def main(argv=None) -> int:
    """
    Parse name and print greeting.

    Args:
        argv: The arguments to parse. Expects a name to greet.
    """
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--name", help="The name to greet and count repos for", default="world"
    )
    args = parser.parse_args(argv)
    if args.name == "":
        print("Username cannot be empty")
        return 1

    print(f"Hello {args.name}!")

    repos = requests.get(f"https://api.github.com/users/{args.name}/repos")
    if repos.status_code != 200:
        print(f"Failed to fetch repos for {args.name}")
        return 1
    repos = repos.json()
    print(f"You have {len(repos)} repos.")  # type: ignore
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

There are a few tricks here that are worth mentioning:

We use argparse to parse the arguments passed to our CLI. This is a built-in python package that allows us to parse arguments passed to our CLI.
We use argv to pass the arguments to our main function 💡. This is useful for testing purposes. With argv, we can pass a list of arguments to our main function and test it without having to run the CLI. While this is an interesting trick, it is quite an intermediate to advanced subject so do not worry to much if this does not fully sink in.
We use SystemExit to exit our program. This is a built-in python exception that allows us to exit our program with a specific exit code (0 for success, 1 for failure). Also quite an intermediate/advanced topic but if you’re familiar with how 0s are success codes and 1s are error code in programming you should get the gist of it.

5. Write Some Tests

📚 Now that we have a few basic methods, we can start writing some tests. Tests make sure our code behaves as expected. As you might be aware. there are many types of tests (unit tests, integration tests, end-to-end tests, etc…) but in this article, we will give an example on unit testing. Unit tests are tests that check the smallest unit of your code (i.e. a function or a method).

🛠️ Create a file called test_main.py and add the following code to it:

"""Test the command line interface."""
from hello_world_cli.cli import main


def test_main(capsys):
    """
    Test the main function with a valid name.

    Args:
        capsys: pytest fixture to capture stdout and stderr.
    """
    assert main(["--name", "test"]) == 0
    out, err = capsys.readouterr()
    # test is a user which does not have any contributions since 2010.
    # Hopefully this will not change in the future and the test will not break.
    assert out == "Hello test!\nYou have 5 repos.\n"
    assert err == ""


def test_main_empty_name(capsys):
    """
    Test the main function with an empty name.

    Args:
        capsys: pytest fixture to capture stdout and stderr.
    """
    assert main(["--name", ""]) == 1
    out, err = capsys.readouterr()
    assert out == "Username cannot be empty\n"
    assert err == ""

We use capsys to capture the output of our CLI. This is a pytest built-in that allows us to capture the output of our CLI (stdin, stdout, stderr).

Now, once we have written the test file above, running tests becomes easy:

# add pytest to your test dependencies if it is not already there 
> poetry add --group test pytest

# run test using pytest
> poetry run pytest
==================================================================
platform darwin -- Python 3.10.4, pytest-7.2.2, pluggy-1.0.0
rootdir: /Users/armandsavzay/code/perso/hello-world-cli
plugins: anyio-3.6.2
collected 2 items

tests/cli_test.py ..      [100%]

======================== 2 passed in 0.45s =======================

6. Linting our code

📚 Now that we have some tests, the next logical step is to start linting our code. Linting is the process of checking our code for potential errors and/or missformating. There are many linters out there (pylint, flake8, black, etc…). For my projects (both at Turo and personally), I use the following flake8, black, isort and mypy. Ruff is also becoming quite popular and can be used to replace flake8 and isort.

🛠️

# install flake8, black, isort and mypy, add them to lint dependencies
poetry add --group lint flake8 black isort mypy

💡 Note that we add this to the lint group meaning that we won’t mix our application dependencies with our linting/dev dependencies. Our production code does not really need to know anything about flake8, black, isort or mypy to run correctly. But our team, and basically anyone collaborating on the code, needs to.

A few examples of linting your code include:

# Lint using mypy:
> poetry run mypy hello_world_cli/cli.py

# Lint using flake8
poetry run flake8 hello_world_cli/cli.py

# Lint using black 
poetry run black hello_world_cli/cli.py

But we would not really want to run these commands manually every time we change our code, would we?

This is where pre-commit comes in handy, to both make sure we don’t commit wrongly formatted code and to not have to run these commands manually.

Pre-commit runs certain commands (known as hooks) right before making a commit, or when invoked through pre-commit run --all-files

Also, do note that some of these linters have small incompatible differences. For example, black will complain if we have a line longer than 88 characters, but flake8 will complain if we have a line longer than 79 characters. We need to configure black, isort and flake8 to work correctly together:

we should create a .flake8file to ignore E203 (space before “:” to be PEP8 compliant) and to increase the line length from 79 to 88 (to be the same as the black line length). Note that this file is already created if you used armand-sauzay/python-template as a template for your repository.

#file: hello-world-cli/.flake8 
[flake8]
max-line-length = 88
extend-ignore = E203

we should create a .isort.cfg file with the following content to make isort compatible with black. Note that this file is already created if you used armand-sauzay/python-template as a template for your repository.

# file: hello-world-cli/.isort.cfg
[settings]
profile = black

You can read more on the black documentation specifically how to make black work with other tools (like flake8 or isort here).

7. Automate checks on local with pre-commit

📚 Now that we have an understanding of linting and testing, we can start automating our checks. The goal of automating our checks is to make sure our code is always in a good state. pre-commit is an excellent alternative as it basically prevents us to commit wrong or wrongly formatted code (as its name suggests).

🛠️ To install pre-commit, we can run the following command:

# install pre-commit
> brew install pre-commit

# run pre-commit on all files. 
# Actual checks to run come from the file .pre-commit-config.yaml 
> pre-commit run --all-files
check yaml...............................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
black....................................................................Passed
flake8...................................................................Passed
isort....................................................................Passed
prettier.................................................................Passed
Lint GitHub Actions workflow file Docker.................................Passed
Test shell scripts with shellcheck.......................................Passed
shfmt....................................................................Passed

Image by author — running pre-commit run -a

We can see on the image above that pre-commit is running the checks that we defined in the .pre-commit-config.yaml file. Note that this file is coming from the template above in step 4.

8. Automate checks on remote with GitHub Actions

📚 Now that we have some tests and some linters, we can start automating our checks on the server side. While there are many tools out there that can help us automate our checks (GitHub Actions, Travis CI, Circle CI, etc…), I personally use and recommend Github Actions (aka GHA).

💡 Note: I am using armand-sauzay/actions-python which standardizes actions for python code. This project is forked from open-turo/actions-python where open-turo is our open-source GitHub organization for releasing OSS.

🛠️ To set up GitHub Actions, we can create a file called .github/workflows/ci.yaml and add the following code to it:

name: CI

on:
  workflow_dispatch:
  push:
jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/lint@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/test@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          test-flags: --version

Note that this file should already be created if you used the template mentioned above (see gh command in step 4).

A few things worth noting in the specific GHA file above:

If you’re not familiar with the syntax of GitHub Actions, you can read the documentation.
Here we call two jobs: lint and test. Each job runs on ubuntu-latest (a Linux machine).
The lint job uses the armand-sauzay/actions-python/lint action to run pre-commit.
The test job uses the armand-sauzay/actions-python/test action to run pytest. Here we pass the flag — — version to pytest to just have a dummy test. We can remove this flag to run our actual tests on our CI environment.

Image by author — Running CI checks on GitHub

9. Automate our release with GitHub Actions

📚 We have briefly talked about linting — using black, flake8, isort and mypy — and testing — using pytest. Release is an equally important concept that allows us to give versions to specific snapshots of our code. Conceptually it maps a commit sha with a number (v1.2.3).

🛠️ To set up GitHub Actions, we can create a file called .github/workflows/release.yaml and add the following code to it:

name: Release

on:
  push:
    branches:
      - main
  workflow_dispatch: # enable manual release
jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/lint@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - uses: armand-sauzay/actions-python/test@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          test-flags: --version
  release:
    name: Release
    needs: [lint, test]
    runs-on: ubuntu-latest
    outputs:
      new-release-published: ${{ steps.release.outputs.new-release-published }}
      new-release-version: ${{ steps.release.outputs.new-release-version }}
    steps:
      - uses: armand-sauzay/actions-python/release@v1
        id: release
        with:
          github-token: ${{ secrets.ADMIN_TOKEN || secrets.GITHUB_TOKEN }}