py-manage

A personal, opinionated guide to managing Python projects. Literature that extends the argumentation and explains it on a more detailed level:

• "Python Project Management Primer".
• "Optimizing Docker Images for Python Production Services".

Table of Contents

1. Tooling
2. Dockerfile Optimizations
3. CUDA Development Workbenches
4. Standard Project Structure
5. Mono-Repository Structure
6. Workflows
   1. Starting a New Project
   2. Installing an Existing Project
   3. Developing Locally
   4. Building Docker Images
   5. Running Docker Containers Locally
   6. CI/CD

Tooling

• Use pyenv to manage Python versions.
• Use pipx to install and run global Python applications in isolated environments (poetry for example).
• Use poetry to manage Python dependencies and packaging.
• [minor] use ruff as a linter/formatter.

Dockerfile Optimizations

This guide recommends the following techniques:

• Multi-stage builds:
• • To parallelize builds to increase speed.
• • To separate build and runtime stages to reduce final image size.
• Effective cache utilization to speed-up build times by:
• • Positioning expensive layers early.
• • Placing frequently changing layers last.
• • Keeping layers small (including only necessary files and dependencies).
• • Minimizing layer count.

Examples could be found in both, standard and monorepo structures.

CUDA Development Workbenches

This guide also covers how to build comprehensive CUDA environment workbenches, that can reliably build deep learning frameworks, such as PyTorch, from source. The same Dockerfile file can be found in standard and monorepo structures.

Standard Project Structure

For details on the standard project structure, refer to the standard structure documentation.

Mono-Repository Structure

For details on the mono-repository structure, refer to the mono-repository structure documentation.

Workflows

Starting a New Project

Follow these steps to start a new project:

pyenv install 3.12.3  # or the version you need
pyenv local 3.12.3
poetry init
... # Configure as needed
poetry install --no-root

Installing an Existing Project

Follow these steps to install an existing project:

pyenv install 3.12.3  # or the version you need
pyenv local 3.12.3
poetry install

Developing Locally

Follow these steps for local development:

poetry run ruff format  # format the files
poetry run ruff check --fix  # apply linting fixes 
poetry run python -m unittest discover
poetry run mypy .

Building Docker Images

Follow these steps to build Docker images for services:

# Inside the service directory
export IMAGE_TAG=python-service-x
docker build -f Dockerfile -t $IMAGE_TAG .

Running Docker Containers Locally

Follow these steps to run Docker containers locally:

# Be sure to have built the Docker image before
export IMAGE_TAG=python-service-x
docker run -p 8080:8080 --name local $IMAGE_TAG

CI/CD

For details on CI/CD workflows, refer to the respective documentation:

• Standard structure CI/CD workflows.
• Mono-repository structure CI/CD workflows.