Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial import from original repo #2

Merged
merged 95 commits into from
Oct 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
95 commits
Select commit Hold shift + click to select a range
1f0266d
Initial commit
jgiannuzzi Jun 24, 2024
cc1adf2
Configure ruff further
jgiannuzzi Jun 24, 2024
d5f6cbd
Import Go code and misc settings
jgiannuzzi Jun 25, 2024
bb4148a
Add README with a few rewrites
jgiannuzzi Jun 25, 2024
6096fe4
Move Go commands under `cmd`
jgiannuzzi Jun 25, 2024
861e7d7
Add ModelRegistry GetLatestVersions endpoint
jgiannuzzi Jun 25, 2024
dbc7351
Initial generation of Go files from proto
nojaf Jun 26, 2024
044eaba
Update gitignore
jgiannuzzi Jun 26, 2024
c6cde6c
Use snake case for JSON
jgiannuzzi Jun 26, 2024
7745759
Make the server work again
jgiannuzzi Jun 26, 2024
424d844
Miscealenous binding improvements
jgiannuzzi Jun 27, 2024
2b10c0c
Add current FFI definitions
jgiannuzzi Jun 27, 2024
13d102d
Fix Go compilation on Windows
jgiannuzzi Jun 28, 2024
85339b3
Use logrus logger
nojaf Jul 1, 2024
7f7838c
Stop the entire process group when stopping the Python command
jgiannuzzi Jul 1, 2024
726af02
Move Go and Python files in separate directories at the top-level
jgiannuzzi Jul 2, 2024
af8ab6b
Fix typo in README
jgiannuzzi Jul 2, 2024
a7dbb82
Mage
nojaf Jul 2, 2024
9d8150d
Keep Go cache across devcontainer rebuilds
jgiannuzzi Jul 2, 2024
e595a08
Monkey eats banana
jgiannuzzi Jul 3, 2024
f9c9e44
Extra logging to know when Go/Python server exits
jgiannuzzi Jul 3, 2024
fcb71f9
Allow passing python_address and python_command to the Go server
jgiannuzzi Jul 3, 2024
eb9e8f4
Use MLFlow's `find_free_port` function
jgiannuzzi Jul 3, 2024
091cd72
Run unmodified server tests
jgiannuzzi Jul 3, 2024
f27c4fd
Improve build process
jgiannuzzi Jul 3, 2024
7e852ed
Improve temporary doc
jgiannuzzi Jul 3, 2024
2e1daf9
Moar mage targets
nojaf Jul 4, 2024
d4b8731
Update repo.go
jgiannuzzi Jul 4, 2024
51e4b3a
Run go mod tidy
jgiannuzzi Jul 4, 2024
77b045a
Use real sqlite instead of wasm one now that we use cgo
jgiannuzzi Jul 4, 2024
8fb08d7
Use floflo's branch to get green server tests
jgiannuzzi Jul 4, 2024
344519e
Parse C header automatically from Python
jgiannuzzi Jul 4, 2024
6274fec
Could not deal with error message inconsistencies
jgiannuzzi Jul 4, 2024
be8e139
Skip file store tests early
jgiannuzzi Jul 4, 2024
374aaeb
Refactor logging, config, and server startup
jgiannuzzi Jul 4, 2024
99e471d
Apply defaults when loading config
jgiannuzzi Jul 4, 2024
ba20e7c
Allow server to start without StaticFolder or PythonAddress
jgiannuzzi Jul 4, 2024
6e47453
Start test server in-process for faster startup
jgiannuzzi Jul 4, 2024
17d14b9
Expose ffi for stores
jgiannuzzi Jul 5, 2024
b532fc7
Fix header parsing to include functions that return pointers
jgiannuzzi Jul 5, 2024
2abae62
Use ModelRegistryStoreURI when it's passed but no TrackingStoreURI is
jgiannuzzi Jul 5, 2024
65c3ad6
Use a context manager for the temporary library build
jgiannuzzi Jul 5, 2024
e651691
Add store bindings
jgiannuzzi Jul 5, 2024
407dfce
Add store tests
jgiannuzzi Jul 5, 2024
d85e575
Use pytest log level in Go server
jgiannuzzi Jul 5, 2024
bc4f543
Add information about running specific tests with more verbosity
jgiannuzzi Jul 5, 2024
8469b1e
Register Go stores automatically, with optional enablement
jgiannuzzi Jul 31, 2024
eb15cff
Remove unnecessary version from docker-compose.ymlt
jgiannuzzi Aug 1, 2024
26412ce
Add dev info about using the Go store directly in Python
jgiannuzzi Aug 5, 2024
eb30a63
Migrate `restoreExperiment` endpoint
dsuhinin Aug 5, 2024
1668128
Clarify README
jgiannuzzi Aug 5, 2024
a26e1b4
Update upstream ref
jgiannuzzi Aug 5, 2024
b5a2dfd
Migrate updateExperiment endpoint
dsuhinin Aug 7, 2024
ebaad04
Passing `context` object from `request`
dsuhinin Aug 8, 2024
9a1cf04
List failing tests
nojaf Aug 8, 2024
ce18039
Add log metric endpoint
nojaf Aug 8, 2024
6b1d085
Revisit order by in search runs
nojaf Aug 12, 2024
95528ca
Enable case_sensitive_like for sqlite
nojaf Aug 13, 2024
9d7325c
Add ruff to pre-commit hooks
jgiannuzzi Aug 14, 2024
335e5f7
Use fiber.Ctx instead of context.Context object
dsuhinin Aug 14, 2024
6764dc9
Fix store call to UpdateExperiment
jgiannuzzi Aug 14, 2024
b6cbebd
Update failing tests
nojaf Aug 14, 2024
c1a01bc
Fix SearchRuns_MaxResults validation
nojaf Aug 19, 2024
c9cfd9f
Allow negative numbers in search run lexing
nojaf Aug 19, 2024
d62cef3
Valid attribute.run_uuid instead of run_id
nojaf Aug 19, 2024
16117b6
Correctly alias Created as start_time
nojaf Aug 19, 2024
7f8a06f
Cleanup of Debian indices in devcontainer image
jgiannuzzi Aug 20, 2024
7fbc402
Introduce basic CI
jgiannuzzi Aug 20, 2024
205db41
Propagate context to downstream objects
dsuhinin Aug 27, 2024
ba5c81d
Use values instead of pointers
dsuhinin Aug 29, 2024
2bca827
Migrate Update Run endpoint
dsuhinin Sep 4, 2024
bf4f074
Migrate `GetRun` endpoint
dsuhinin Sep 5, 2024
1c003a5
Finish run endpoints migration
dsuhinin Sep 9, 2024
a0d5100
Fix test_log_batch_nonexistent_run
nojaf Sep 9, 2024
7a4e2e1
Truncate in validation
nojaf Sep 10, 2024
b932dc4
Force DeletedTime to be NULL
nojaf Sep 11, 2024
4351268
Set DeletedTime for Runs to NULL when restoring experiment.
nojaf Sep 11, 2024
5e3860f
Return None for empty next_page_token
nojaf Sep 12, 2024
3db35cb
Take valid stages into account
nojaf Sep 12, 2024
522834f
Check if run is active in log metrics.
nojaf Sep 12, 2024
79dac8c
Unify work with protos, models, entities.
dsuhinin Sep 12, 2024
a331064
Return contract errors for attribute filter validation.
nojaf Sep 12, 2024
6c278ee
Return correct error message for logBatch uniqueParams
nojaf Sep 13, 2024
ca95e7e
Ignore test via conftest
nojaf Sep 16, 2024
8eaf2c1
Normalise models, protos, entities. simplify logic.
dsuhinin Sep 17, 2024
3cd9545
Test search runs datasets
nojaf Sep 18, 2024
a7addb9
Use correct validation error for max length
nojaf Sep 19, 2024
c538f78
Port extact AppendToURIPath function from Python
nojaf Sep 19, 2024
bdf8178
Allow endtime to be NULL in update run
nojaf Sep 23, 2024
258a61e
Test order by metric tag param
nojaf Sep 24, 2024
e0dbbb2
Trying to fix issue with NaN, Inf, -Inf
dsuhinin Sep 24, 2024
c9c38f6
Fix test_log_metric_concurrent_logging_succeeds test
dsuhinin Sep 25, 2024
17d1ada
Ignore test_log_batch_null_metrics
nojaf Sep 25, 2024
60e1a0e
Add target to list endpoints
nojaf Sep 26, 2024
344307e
Fix Test log inputs with large inputs limit check
nojaf Oct 1, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .devcontainer/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM mcr.microsoft.com/devcontainers/go:1-1.22-bookworm

# [Optional] Uncomment this section to install additional OS packages.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
&& apt-get -y install --no-install-recommends \
postgresql-client \
sqlite3 \
&& rm -rf /var/lib/apt/lists/*

# [Optional] Uncomment the next lines to use go get to install anything else you need
USER vscode
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.33.0 \
&& go install github.com/vektra/mockery/v2@v2.43.2 \
&& go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.59.1 \
&& go install github.com/magefile/mage@v1.15.0 \
&& go clean -cache -modcache
USER root

# [Optional] Uncomment this line to install global node packages.
# RUN su vscode -c "source /usr/local/share/nvm/nvm.sh && npm install -g <your-package-here>" 2>&1
66 changes: 66 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// For format details, see https://aka.ms/devcontainer.json.
{
"name": "MLflow Go",
"dockerComposeFile": "docker-compose.yml",
"service": "app",
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",

// Features to add to the dev container. More info: https://containers.dev/features.
"features": {
"ghcr.io/devcontainers/features/github-cli:1": {},
"ghcr.io/devcontainers/features/python:1": {
"version": "3.8"
},
"ghcr.io/devcontainers/features/docker-in-docker:2": {},
"ghcr.io/devcontainers-contrib/features/k6:1": {},
"ghcr.io/devcontainers-contrib/features/pre-commit:2": {},
"ghcr.io/devcontainers-contrib/features/protoc-asdf:1": {
"version": "26.0"
},
"ghcr.io/devcontainers-contrib/features/ruff:1": {}
},

// Configure tool-specific properties.
"customizations": {
"vscode": {
"settings": {
"terminal.integrated.defaultProfile.linux": "zsh",
"editor.rulers": [
80,
100
],
"editor.formatOnSave": true,
"git.alwaysSignOff": true,
"go.lintTool": "golangci-lint",
"gopls": {
"formatting.local": "github.com/mlflow/mlflow-go",
"formatting.gofumpt": true,
"build.buildFlags": ["-tags=mage"]
},
"[python]": {
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports": "explicit"
},
"editor.defaultFormatter": "charliermarsh.ruff"
}
},
"extensions": [
"charliermarsh.ruff",
"golang.Go",
"humao.rest-client",
"pbkit.vscode-pbkit",
"tamasfe.even-better-toml"
]
}
},

// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [5432],

// Use 'postCreateCommand' to run commands after the container is created.
"postCreateCommand": ".devcontainer/postCreate.sh"

// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
}
38 changes: 38 additions & 0 deletions .devcontainer/docker-compose.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
volumes:
go-cache:
postgres-data:

services:
app:
build:
context: .
dockerfile: Dockerfile
volumes:
- ../..:/workspaces:cached
- go-cache:/var/cache/go
environment:
- GOCACHE=/var/cache/go/build
- GOMODCACHE=/var/cache/go/mod

# Overrides default command so things don't shut down after the process ends.
command: sleep infinity

# Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
network_mode: service:db

# Use "forwardPorts" in **devcontainer.json** to forward an app port locally.
# (Adding the "ports" property to this file will not forward from a Codespace.)

db:
image: postgres:latest
restart: unless-stopped
volumes:
- postgres-data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=postgres
- POSTGRES_HOSTNAME=localhost=value

# Add "forwardPorts": ["5432"] to **devcontainer.json** to forward PostgreSQL locally.
# (Adding the "ports" property to this file will not forward from a Codespace.)
7 changes: 7 additions & 0 deletions .devcontainer/postCreate.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
#!/bin/sh

# Fix permissions for the Go cache
sudo chown -R $(id -u):$(id -g) /var/cache/go

# Install precommit (https://pre-commit.com/)
pre-commit install -t pre-commit -t prepare-commit-msg
16 changes: 16 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for more information:
# https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
# https://containers.dev/guide/dependabot

version: 2
updates:
- package-ecosystem: "devcontainers"
directory: "/"
schedule:
interval: weekly
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: weekly
18 changes: 18 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
name: CI

on:
push:
branches:
- main
pull_request:
schedule:
# Run daily at 01:34 so we get notified if CI is broken before a pull request
# is submitted.
- cron: "34 1 * * *"

jobs:
lint:
uses: ./.github/workflows/lint.yml

test:
uses: ./.github/workflows/test.yml
22 changes: 22 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: Lint

on:
workflow_call:

permissions:
contents: read

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
check-latest: true
cache: false
- name: Run pre-commit hooks
run: pipx run pre-commit run --all-files
59 changes: 59 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Test

on:
workflow_call:

permissions:
contents: read

jobs:
go:
name: Test Go
strategy:
matrix:
runner: [macos-latest, ubuntu-latest, windows-latest]
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
check-latest: true
cache: false
- name: Install mage
run: go install github.com/magefile/mage@v1.15.0
- name: Run unit tests
run: mage test:unit

python:
name: Test Python
strategy:
matrix:
runner: [macos-latest, ubuntu-latest, windows-latest]
python: ["3.8", "3.9", "3.10", "3.11", "3.12"]
runs-on: ${{ matrix.runner }}
steps:
- uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- name: Setup Go
uses: actions/setup-go@v5
with:
go-version: "1.22"
check-latest: true
cache: false
- name: Install mage
run: go install github.com/magefile/mage@v1.15.0
- name: Install our package in editable mode
run: pip install -e .
- name: Initialize MLflow repo
run: mage repo:init
- name: Install dependencies
run: pip install pytest==8.1.1 psycopg2-binary -e .mlflow.repo
- name: Run integration tests
run: mage test:python
# Temporary workaround for failing tests
continue-on-error: ${{ matrix.runner != 'ubuntu-latest' }}
16 changes: 16 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Artifacts
dist/
*.egg-info/
*.so

# Runs
mlruns/

# Cache
__pycache__/

# MLflow repo
.mlflow.repo/

# JetBrains
.idea
55 changes: 55 additions & 0 deletions .golangci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
run:
build-tags:
- mage
timeout: 5m

linters:
enable:
- errcheck
- gosimple
- lll
disable:
- depguard
- gochecknoglobals # Immutable globals are fine.
- exhaustruct # Often the case for protobuf generated code or gorm structs.
- protogetter # We do want to use pointers for memory optimization.
presets:
- bugs
- comment
- complexity
- error
- format
- import
- metalinter
- module
- performance
- sql
- style
- test
- unused

linters-settings:
gci:
custom-order: true

sections:
- standard # Standard section: captures all standard packages.
- default # Default section: contains all imports that could not be matched to another section type.
- alias # Alias section: contains all alias imports. This section is not present unless explicitly enabled.
- prefix(github.com/mlflow/mlflow-go) # Custom section: groups all imports with the specified Prefix.
- blank # Blank section: contains all blank imports. This section is not present unless explicitly enabled.
- dot # Dot section: contains all dot imports. This section is not present unless explicitly enabled.

gofumpt:
module-path: github.com/mlflow/mlflow-go
extra-rules: true

tagliatelle:
case:
rules:
json: snake

issues:
exclude-files:
- ".*\\.g\\.go$"
- ".*\\.pb\\.go$"
1 change: 1 addition & 0 deletions .mlflow.ref
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
https://github.com/mlflow/mlflow.git#master
8 changes: 8 additions & 0 deletions .mockery.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
dir: "{{ .InterfaceDir }}"
filename: "mock_{{ .InterfaceNameSnake }}.go"
with-expecter: true
inpackage: true
packages:
github.com/mlflow/mlflow-go/pkg/tracking/store:
interfaces:
TrackingStore:
34 changes: 34 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: end-of-file-fixer
files: \.(proto|txt|sh|rst)$
- repo: https://github.com/golangci/golangci-lint
rev: "v1.59.1"
hooks:
- id: golangci-lint-full
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.5.7
hooks:
- id: ruff
types_or: [python, pyi, jupyter]
args: [--fix]
- id: ruff-format
types_or: [python, pyi, jupyter]
- repo: local
hooks:
# - id: rstcheck
# name: rstcheck
# entry: rstcheck
# language: system
# files: README.rst
# stages: [commit]
# require_serial: true

- id: must-have-signoff
name: must-have-signoff
entry: 'grep "Signed-off-by:"'
language: system
stages: [prepare-commit-msg]
require_serial: true
Loading
Loading