rules_python_external

Bazel rules to transitively fetch and install Python dependencies from a requirements.txt file.

Features

The rules address most of the top packaging issues in bazelbuild/rules_python. This means the rules support common packages such as tensorflow and google.cloud natively.

• Transitive dependency resolution: #35, #102
• Minimal runtime dependencies: #184
• Support for spreading purelibs: #71
• Support for namespace packages: #14, #55, #65, #93, #189
• Fetches pip packages only for building Python targets: #96
• Reproducible builds: #154, #176

Usage

Prerequisites

The rules support Python >= 3.5 (the oldest maintained release).

Setup WORKSPACE

rules_python_external_version = "{COMMIT_SHA}"

http_archive(
    name = "rules_python_external",
    sha256 = "", # Fill in with correct sha256 of your COMMIT_SHA version
    strip_prefix = "rules_python_external-{version}".format(version = rules_python_external_version),
    url = "https://github.com/dillon-giacoppo/rules_python_external/archive/v{version}.zip".format(version = rules_python_external_version),
)

# Install the rule dependencies
load("@rules_python_external//:repositories.bzl", "rules_python_external_dependencies")
rules_python_external_dependencies()

load("@rules_python_external//:defs.bzl", "pip_install")
pip_install(
    name = "py_deps",
    requirements = "//:requirements.txt",
    # (Optional) You can provide a python interpreter (by path):
    python_interpreter = "/usr/bin/python3.8",
    # (Optional) Alternatively you can provide an in-build python interpreter, that is available as a Bazel target.
    # This overrides `python_interpreter`.
    # Note: You need to set up the interpreter target beforehand (not shown here). Please see the `example` folder for further details.
    #python_interpreter_target = "@python_interpreter//:python_bin",
)

Example BUILD file.

load("@py_deps//:requirements.bzl", "requirement")

py_binary(
    name = "main",
    srcs = ["main.py"],
    deps = [
        requirement("boto3"),
    ],
)

Note that above you do not need to add transitively required packages to deps = [ ... ]

Setup requirements.txt

While rules_python_external does not require a transitively-closed requirements.txt file, it is recommended. But if you want to just have top-level packages listed, that also will work.

Transitively-closed requirements specs are very tedious to produce and maintain manually. To automate the process we recommend pip-compile from jazzband/pip-tools.

For example, pip-compile takes a requirements.in like this:

boto3~=1.9.227
botocore~=1.12.247
click~=7.0

pip-compile 'compiles' it so you get a transitively-closed requirements.txt like this, which should be passed to pip_install below:

boto3==1.9.253
botocore==1.12.253
click==7.0
docutils==0.15.2          # via botocore
jmespath==0.9.4           # via boto3, botocore
python-dateutil==2.8.1    # via botocore
s3transfer==0.2.1         # via boto3
six==1.14.0               # via python-dateutil
urllib3==1.25.8           # via botocore

Demo

You can find a demo in the example/ directory.

Development

Testing

bazel test //...

Adopters

Here's a (non-exhaustive) list of companies that use rules_python_external in production. Don't see yours? You can add it in a PR!

• Canva