G3T to git-drs Migration: pain points, developer feedbacks.

[matthewpeterkort]

Steps to reproduce attempting to migrate project from g3t to git-drs

1. create a new git repo in the frontend:
   https://source.ohsu.edu/orgs/CBDS/repositories

2. clone it locally: git clone git@source.ohsu.edu:CBDS/multi-modal.git

3. copy in the old .g3t and MANIFEST directory from old project. In this case cbds-multimodal project on calypr-dev

4. make in a new dir make a new python env, pip install g3t. pull the existing files down from the .g3t project

python -m venv venv  ; source venv/bin/activate
pip install -e ../../../gen3_util
g3t --profile cbds-dev pull --data-only

4.4 Do this g3t operations in a seperate draft to minimize issues between the two different .git directories. This also avoids having to do a
git add remote, because the two utilities conflict with eachother and remove the old remote.

Run this script to make meta files. Adapted from Walsh draft.

#!/usr/bin/env python3

import os
import subprocess
import yaml
import json
import argparse
from pathlib import Path

def run(cmd, dry_run=False):
    print(">>>", " ".join(cmd))
    if not dry_run:
        subprocess.run(cmd, check=True)

def write_meta(file_path, metadata, dry_run=False):
    meta_path = Path(f"{file_path}.meta")
    meta_content = {
        "aliases": [],
        "dvc_metadata": metadata
    }
    print(f"📄 Writing metadata to {meta_path}")
    if not dry_run:
        with open(meta_path, "w") as f:
            json.dump(meta_content, f, indent=2)
        run(["git", "add", str(meta_path)], dry_run=dry_run)

def migrate_dvc_file(dvc_path: Path, dry_run=False):
    print(f"🔄 Processing {dvc_path}")
    with open(dvc_path) as f:
        dvc_yaml = yaml.safe_load(f)

    outs = dvc_yaml.get("outs", [])
    if not outs:
        print(f"⚠️  No 'outs' section in {dvc_path}")
        return

    for out in outs:
        data_path = Path(out["path"])
        if not data_path.exists():
            print(f"⚠️  Missing file: {data_path}")
            continue

        # Write .meta file
        metadata = {k: v for k, v in out.items() if k != "path"}
        if metadata:
            write_meta(data_path, metadata, dry_run=dry_run)

def main():
    parser = argparse.ArgumentParser(description="Migrate DVC-tracked files to Git LFS and Calypr .meta format.")
    parser.add_argument("--path", default=".", help="Directory to search for .dvc files")
    parser.add_argument("--dry-run", action="store_true", help="Preview changes without applying")
    args = parser.parse_args()

    for root, _, files in os.walk(args.path):
        for file in files:
            if file.endswith(".dvc"):
                migrate_dvc_file(Path(root) / file, dry_run=args.dry_run)

if __name__ == "__main__":
    main()

4.5 move all of the pulled files and the meta files to the new repo that you cloned earlier.

4.55 install git-drs using go or maybe some other way so that users don't need to have go installed to get this thing to run.

5. Run git lfs init:

git lfs install --skip-smudge

git drs init --profile cbds-dev --apiendpoint https://calypr-dev.ohsu.edu --cred /Users/peterkor/.gen3/credentials-cbds-dev.json
.drs/lfs/objects already present in .gitignore
Git DRS initialized successfully!

manually make drs directory
mkdir .drs
manually make drs config:

{
  "queryServer": {
    "baseURL": "https://calypr-dev.ohsu.edu"
  },
  "writeServer": {
    "baseURL": "https://calypr-dev.ohsu.edu"
  },
  "gen3Profile": "cbds-dev",
  "gen3Project": "cbds-multimodal",
  "gen3Bucket": "cbds"
}

6. Only track your data files not your meta files.

git lfs track "*.tsv"
git add .gitattributes
git add "*.tsv"
git add "*.tsv.meta"

7. Make sure to commit your .drs/config otherwise clone pull won't work.

git add .drs/config

8. commit your files

git commit -m "test"

9. push your files

git push

Clone use case:

Setup:

if your gen3 config profile name does not match the config specified in the cloned .config this will need to be manually changed. In my opinion, the git drs init command should take care of this for you and update the config file for you.

git@source.ohsu.edu:CBDS/multi-modal.git
git lfs install --skip-smudge
git drs init --profile cbds-dev --apiendpoint https://calypr-dev.ohsu.edu --cred /Users/peterkor/.gen3/credentials-cbds-dev.json
git-lfs ls-files
git-lfs pull -I db66be89-fb7c-41c3-9a0b-9beae75b7850.rna_seq.augmented_star_gene_counts.tsv

Feedbacks

I don't like that os/exec is called for the git commands. I ran into nasty os exec errors from this that was not trivial to debug.
a package like go-git would provide a go native error that could be passed back to the user instead of a black box error to improve error handling for the user.

We need to implement multi file uploading leveraging the existing cdis client bulk uploader. Nobody wants to wait for files to upload one at a time.

Config should be marshaled / unmarshalled directly from the config file that way when the user runs the git-drs init command they can save whatever they specified to the config directory, eg: if their gen3 profile name is different than whatever one was cloned.

The user is expected to add the .drs/config on their very first commit. Using something like go-git this could probably be done silently and incorporated into one of the function calls either drs-init or as part of the hook attached to 'git commit' perhaps, since the user shouldn't be expected to know about the .drs/config file to begin with.

I have added a delete indexd record function as it had helped me debug my issues during the migration, but there should also probably be a delete file function added to this API so that a push can also effectively remove files from an S3 bucket. The delete indexd record function should probably have a pattern based approach just like 'git add'

User docs needed to be added complete with instructions on how to download the executable from the releases page, and other setup pain points. Thinking about this further the git-drs executable should probably be part of the main g3t meta executable so that the user only has to download one executable.

This code block is called alot. It should be its own function.

	myLogger.Logf("using endpoint: %s\n", a.String())
	req, err := http.NewRequest("GET", a.String(), nil)
	if err != nil {
		return nil, err
	}

	err = addGen3AuthHeader(req, cl.profile)
	if err != nil {
		return nil, fmt.Errorf("error adding Gen3 auth header: %v", err)
	}

In some places in the command flow new logger is declared twice, ex: UpdateDrsObjects and precommit hook. Instead of instantiating the logger twice, probably just pass the instance to the function using a method function.

json.Decoder can probably be used in cmd/transfer/main.go to avoid copying the bytes into the unmarshal when they can be read directly from stdin.