Skip to content

Structuring the codebase

Basic structure

Keep your rust, wrapped-rust and python code separate

You want a top level directory structure like this - with directories for rust, python and tests. Within the rust directory you want separate directories for your main logic and the wrapped exports for python.

FizzBuzz
- rust
  - fizzbuzz
    - src
      ... core rust code goes here
    - tests
      ... tests for core rust code go here
  - fizzbuzzo3
    - src
      ... pyo3 code goes here
- python
  - fizzbuzz
    ... additional python code goes here
- tests
  ... python tests go here

Warning: Import errors

Having any top-level directories with names that match your package leads to all kinds of fun import errors. Depending on the exact context, python can decide that you have implicit namespace packages which collide with your explicit package namespaces.

I ran into problems twice:

  • firstly, I had a time where my tests ran fine but I couldn't import my code in repl;
  • later, the final wheels were unusable but sdist was fine.

There are also reports of end-users being the first to run into trouble

This is also the reason for keeping your final set of python tests in a separate top-level directory: you can be sure they are using the right import logic.

Considerations

  1. When you distribute the final python library as source the primary audience are python-coders, make it understandable for them without a lot of extra explanation by putting your rust code in a clearly marked location
  2. You will want to test your code at each stage of integration: core rust, wrapped rust, final python result; so that you can easily track down bugs
  3. Context switching between two languages is hard, even if you know both parts of the codebase well. Keeping it structured by language boundary helps when coding. For much larger projects you may want to provide a higher-level structure by domain and then structure each domain as above ... but that's outside the scope of a simple starter how-to 😉
  4. Your underlying code probably does something useful - so you could also publish it to the rust eco-system as a dedicated crate for others to use directly in rust. Those users don't want the extra code wrapping your function for python!

Configuration files (pyproject.toml & Cargo.toml)

./pyproject.toml

We'll be using setuptools as the main build agent and pyproject.toml to manage all the config and tooling - that way we keep the number of files to a minimum and stick to as few languages as possible.

The important section is:

[tool.setuptools.packages.find]
  where = ["python"]

More info on pyproject.toml is available from both pypa - the python packaging authority and pip - python's package manager

./Cargo.toml

You need to set up a cargo workspace

[workspace]

  members = [
      "rust/*"
  ]

  resolver = "2"

rust/fizzbuzzo3/Cargo.toml

In the Cargo.toml for your wrapped code you need to specify the core rust code as a path dependency:

...
[dependencies]
  fizzbuzz = { path = "../fizzbuzz" }
...

Full details on Cargo.toml in the Cargo book

Avoiding namespace, plug-in and publishing issues

  1. Point setuptools explicitly to look in python - this helps avoid implicit/explicit namespace errors later
  2. Use a cargo workspace to avoid occasional strange errors from rust-analyzer in your IDE and to make running tests, lints etc. easier from the root directory
  3. Use a path dependency to your core code in the wrapped-code Cargo.toml so you are always using the latest changes. Without a version tag, any attempts to upload the wrapped code as a crate to fail - but you don't want to do that anyway and this is another safety measure to make sure you never do.
  4. You don't need anything special in your core-code Cargo.toml but don't forget to add one!

Overview of all config files

The root has a pyproject.toml and Cargo.toml, each folder under rust also has a Cargo.toml

FizzBuzz
- rust
  - fizzbuzz
    - src
      ... core rust code goes here
    - tests
      ... tests for core rust code go here
    - Cargo.toml
  - fizzbuzzo3
    - src
      ... pyo3 code goes here
    - Cargo.toml
- python
  - fizzbuzz
    ... additional python code goes here
- tests
  ... python tests go here
- Cargo.toml
- pyproject.toml
./pyproject.toml - full source 
[project]
    name = "fizzbuzz"
    description = "An implementation of the traditional fizzbuzz kata"
    # readme = "README.md"
    license = { text = "MIT" }
    authors = [{ name = "Mike Foster" }]
    dynamic = ["version"]
    requires-python = ">=3.8"
    classifiers = [
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
        "Topic :: Software Development :: Libraries :: Python Modules",
        "Intended Audience :: Developers",
        "Natural Language :: English",
    ]

[project.urls]
    # Homepage = ""
    # Documentation = ""
    Repository = "https://github.com/MusicalNinjaDad/fizzbuzz"
    Issues = "https://github.com/MusicalNinjaDad/fizzbuzz/issues"
    # Changelog = ""

[tool.setuptools.dynamic]
    version = { file = "__pyversion__" }


# ===========================
#       Build, package
# ===========================

[build-system]
    requires = ["setuptools", "setuptools-rust"]
    build-backend = "setuptools.build_meta"

[tool.setuptools.packages.find]
    where = ["python"]

[[tool.setuptools-rust.ext-modules]]
    # The last part of the name (e.g. "_lib") has to match lib.name in Cargo.toml,
    # but you can add a prefix to nest it inside of a Python package.
    target = "fizzbuzz.fizzbuzzo3"
    path = "rust/fizzbuzzo3/Cargo.toml"
    binding = "PyO3"
    features = ["pyo3/extension-module"] # IMPORTANT!!!
    debug = false # Adds `--release` to `pip install -e .` builds, necessary for performance testing

[tool.cibuildwheel]
    skip = [
        "*-musllinux_i686",    # No musllinux-i686 rust compiler available.
        "*-musllinux_ppc64le", # no rust target available.
        "*-musllinux_s390x",   # no rust target available.
    ]
    # Use prebuilt copies of pypa images with rust and just ready compiled (saves up to 25 mins on emulated archs)
    manylinux-x86_64-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_x86_64-rust"
    manylinux-i686-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_i686-rust"
    manylinux-aarch64-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_aarch64-rust"
    manylinux-ppc64le-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_ppc64le-rust"
    manylinux-s390x-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_s390x-rust"
    manylinux-pypy_x86_64-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_x86_64-rust"
    manylinux-pypy_i686-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_i686-rust"
    manylinux-pypy_aarch64-image = "ghcr.io/musicalninjas/quay.io/pypa/manylinux2014_aarch64-rust"
    musllinux-x86_64-image = "ghcr.io/musicalninjas/quay.io/pypa/musllinux_1_2_x86_64-rust"
    musllinux-aarch64-image = "ghcr.io/musicalninjas/quay.io/pypa/musllinux_1_2_aarch64-rust"

    # `test-command` will FAIL if only passed `pytest {package}`
    # as this tries to resolve imports based on source files (where there is no `.so` for rust libraries), not installed wheel
    test-command = "pytest {package}/tests"
    test-extras = "test"

    [tool.cibuildwheel.linux]
        before-all = "just clean"
        archs = "all"

    [tool.cibuildwheel.macos]
        before-all = "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile minimal && cargo install just && just clean"

    [tool.cibuildwheel.windows]
        before-all = "rustup target add aarch64-pc-windows-msvc i586-pc-windows-msvc i686-pc-windows-msvc x86_64-pc-windows-msvc && cargo install just && just clean"
        before-build = "pip install delvewheel"
        repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}"


# ===========================
#  Test, lint, documentation
# ===========================

[project.optional-dependencies]
    stubs = ["pyo3-stubgen"]
    lint = ["ruff"]
    test = ["pytest", "pytest-doctest-mkdocstrings"]
    cov = ["fizzbuzz[test]", "pytest-cov"]
    doc = ["black", "mkdocs", "mkdocstrings[python]", "mkdocs-material", "fizzbuzz[stubs]"]
    dev = ["fizzbuzz[stubs,lint,test,cov,doc]"]

[tool.pytest.ini_options]
    xfail_strict = true
    addopts = [
        "--doctest-modules",
        "--doctest-mdcodeblocks",
        "--doctest-glob=*.pyi",
        "--doctest-glob=*.md",
    ]
    testpaths = ["tests", "python"]

[tool.coverage.run]
    branch = true
    omit = ["tests/*"]
    dynamic_context = "test_function"

[tool.ruff]
    line-length = 120
    format.skip-magic-trailing-comma = false
    format.quote-style = "double"

[tool.ruff.lint]
    select = ["ALL"]
    flake8-pytest-style.fixture-parentheses = false
    flake8-annotations.mypy-init-return = true
    extend-ignore = [
        "E701",   # One-line ifs are not great, one-liners with suppression are worse
        "ANN101", # Type annotation for `self`
        "ANN202", # Return type annotation for private functions
        "ANN401", # Using Any often enough that supressing individually is pointless
        "W291",   # Double space at EOL is linebreak in md-docstring
        "W292",   # Too much typing to add newline at end of each file
        "W293",   # Whitespace only lines are OK for us
        "D401",   # First line of docstring should be in imperative mood ("Returns ..." is OK, for example)
        "D203",   # 1 blank line required before class docstring (No thank you, only after the class docstring)
        "D212",   # Multi-line docstring summary should start at the first line (We start on new line, D209)
        "D215",   # Section underline is over-indented (No underlines)
        "D400",   # First line should end with a period (We allow any punctuation, D415)
        "D406",   # Section name should end with a newline (We use a colon, D416)
        "D407",   # Missing dashed underline after section (No underlines)
        "D408",   # Section underline should be in the line following the section's name (No underlines)
        "D409",   # Section underline should match the length of its name (No underlines)
        "D413",   # Missing blank line after last section (Not required)
    ]

[tool.ruff.lint.per-file-ignores]
    # Additional ignores for tests 
    "**/test_*.py" = [
        "INP001",  # Missing __init__.py
        "ANN",     # Missing type annotations
        "S101",    # Use of `assert`
        "PLR2004", # Magic number comparisons are OK in tests
        "D1",      # Don't REQUIRE docstrings for tests - but they are nice
    ]

    "**/__init__.py" = [
        "D104", # Don't require module docstring in __init__.py
        "F401", # Unused imports are fine: using __init__.py to expose them with implicit __ALL__ 
    ]
./Cargo.toml - full source 
[workspace]

members = [
    "rust/*"
]

resolver = "2"
rust/fizzbuzz/Cargo.toml - full source 
[package]
name = "fizzbuzz"
version = "4.0.0"
edition = "2021"

[lib]
name = "fizzbuzz"
path = "src/lib.rs"
crate-type = ["rlib"]  # cdylib required for python import, rlib required for rust tests.

[dependencies]
rayon = "1.10.0"

[dev-dependencies]
googletest = "0.11.0"
criterion = { version = "0.5", features = ["html_reports"] }

[[bench]]
name = "bench_fizzbuzz"
harness = false

[[bench]]
name = "bench_sizes"
harness = false


# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
rust/fizzbuzzo3/Cargo.toml - full source 
[package]
name = "fizzbuzzo3"
version = "2.1.0" # Tracks python version
edition = "2021"

[lib]
name = "fizzbuzzo3"
path = "src/lib.rs"
crate-type = ["cdylib"]  # cdylib required for python import, rlib required for rust tests.

[dependencies]
fizzbuzz = { path = "../fizzbuzz" }
pyo3 = "0.21.2"
rayon = "1.10.0"

[dev-dependencies]
pyo3-testing = "0.3.4"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html