mlops-initialization

MLOps Initialization

Goal

To initialize a robust, production-ready MLOps project structure using the modern Python toolchain (uv), industry-standard version control (git), and a configured development environment (VS Code). This skill ensures reproducibility, collaboration, and high code quality from day one.

Prerequisites

• Language: Python (latest stable version recommended)
• Manager: uv (replaces pip, venv, poetry, pyenv)
• VCS: Git
• IDE: VS Code (recommended)

Instructions

1. System & Toolchain Verification

Before modifying files, verify that the essential tools are available.

1. Check uv:
   • Ensure uv is installed: uv --version
   • If missing, install it: curl -LsSf https://astral.sh/uv/install.sh | sh
2. Check git:
   • Ensure git is installed: git --version

2. Project Initialization

Initialize the project structure using uv to ensure modern standards (pyproject.toml).

1. Create Directory (if not already inside):
   • mkdir <project_name> && cd <project_name>
2. Initialize Project:
   • Run uv init
   • This creates pyproject.toml, .python-version, and a basic hello.py.
3. Configure pyproject.toml:

   • Update metadata: name, version, description, authors, license.

   • Set requires-python: Ensure it matches the project's target environment (e.g., >=3.10).

   • Example Structure:

     [project]
     name = "my-mlops-project"
     version = "0.1.0"
     description = "A robust MLOps project."
     readme = "README.md"
     requires-python = ">=3.11"
     license = { file = "LICENSE" }
     authors = [{ name = "Your Name", email = "your.email@example.com" }]
     dependencies = [
         "pandas>=2.2.0",
         "loguru>=0.7.0",
         # Add other runtime dependencies here
     ]
     
     [project.urls]
     Repository = "https://github.com/username/my-mlops-project"
     Documentation = "https://username.github.io/my-mlops-project"
     
     [project.optional-dependencies]
     dev = [
         "pytest>=8.0.0",
         "ruff>=0.3.0",
         "mypy>=1.9.0",
     ]
     
     [build-system]
     requires = ["hatchling"]
     build-backend = "hatchling.build"
     

3. Dependency Management

Establish a clean separation between production and development dependencies.

1. Add Runtime Dependencies (Production):
   • Use uv add <package> for libraries needed in production (e.g., fastapi, numpy, torch).
   • These go into [project.dependencies] in pyproject.toml.
2. Add Dev Dependencies (Development):
   • Use uv add --dev <package> (or --group dev) for tools like pytest, ruff, pre-commit.
   • These go into [project.optional-dependencies] and are kept separate from production builds.
3. Sync Environment:
   • Run uv sync to resolve dependencies, create the .venv, and generate the uv.lock file.
   • Critical: The uv.lock file pins exact versions of all dependencies (including transitive ones). It ensures that every developer and CI/CD pipeline uses the exact same environment, preventing "it works on my machine" issues. Commit this file to git.

4. Version Control (Git)

Set up a clean repository and ensure unwanted files are ignored.

1. Initialize Git:
   • git init
   • git branch -M main
2. Create .gitignore:
   • Write a robust .gitignore tailored for Python/MLOps.
   • Must Include:
     • Environment: .venv/, .env
     • Caches: __pycache__/, .pytest_cache/, .ruff_cache/, .mypy_cache/
     • Builds: dist/, build/, *.egg-info/
     • Data/Models: data/, models/, outputs/ (unless using DVC/LFS)
     • IDE: .vscode/ (selectively), .idea/, .DS_Store
     • Note: It is often good practice to commit project-specific .vscode/settings.json but ignore User settings.
3. Verify Status:
   • git status should show only source files, config files, and the lockfile.

5. IDE Configuration (VS Code)

Standardize the developer experience (DX) by committing project-specific settings.

1. Install Recommended Extensions:
   • Python Tier A: ms-python.python, headers.ruff, ms-python.vscode-pylance, ms-toolsai.jupyter.
   • Productivity: eamodio.gitlens, alefragnani.project-manager, usernamehw.errorlens.
2. Create .vscode Directory:
   • mkdir .vscode
3. Create settings.json:

   • Configure settings to enforce code quality and use the uv environment.

   • Key Settings:

     {
       "[python]": {
         "editor.defaultFormatter": "charliermarsh.ruff",
         "editor.formatOnSave": true,
         "editor.codeActionsOnSave": {
           "source.organizeImports": "explicit"
         }
       },
       "python.defaultInterpreterPath": ".venv/bin/python",
       "python.terminal.activateEnvironment": true,
       "python.analysis.typeCheckingMode": "basic",
       "python.testing.pytestEnabled": true,
       "files.trimTrailingWhitespace": true,
       "files.insertFinalNewline": true,
       "editor.rulers": [88],
       "files.exclude": {
         "**/__pycache__": true,
         "**/.pytest_cache": true,
         "**/.ruff_cache": true,
         "**/.venv": true
       }
     }
     

6. Verification & First Commit

Finalize the initialization.

1. Verify Environment:
   • Run uv run python -c "import sys; print(sys.executable)" to confirm it uses the .venv.
2. Initial Commit:
   • git add .
   • git commit -m "chore: initialize project with uv, git, and vscode settings"

7. Best Practices Summary

• One Command Setup: ideally, uv sync should be the only command needed to set up the environment.
• Lockfile: Always commit uv.lock to ensure all environments are identical.
• Editor Config: Checked-in .vscode/settings.json reduces onboarding friction and enforces standards (formatting, linting).
• Dependency Separation: Keep production dependencies light; put testing/linting tools in dev.

Self-Correction Checklist

• Lockfile: Does uv.lock exist?
• Virtual Env: Is .venv/ created and ignored in .gitignore?
• Project Config: Does pyproject.toml validly describe the project?
• Git Cleanliness: Are secrets and large data files excluded?
• Reproducibility: Can another developer git clone and uv sync to get the exact same state?