breaking-change/simple-oid-bucket-key

[bwalsh]

ADR: Breaking Change — Migrate Bucket Object Naming to SHA-256 Content Addressing for LFS De-duplication

Status: Proposed
Date: 2026-01-21
Deciders: Platform Architecture / Storage / Data Integrity
Context: Git-LFS, object storage, de-duplication, backward compatibility

---

Context

The platform stores large file objects in backing object storage (e.g., S3-compatible buckets).
Today, object keys are derived from project-, path-, or upload-scoped naming conventions, not solely from file content.

As a result:

• Identical files uploaded:

  • Across different projects
  • From different clients
  • At different times
    are stored as distinct physical objects

• Storage-level de-duplication is impossible or incomplete

• Storage identity diverges from Git-LFS’s content identity model

At the same time, Git-LFS already defines file identity as the SHA-256 digest of file contents, computed deterministically by clients.

This mismatch prevents correct, global de-duplication.

---

Problem Statement

The current bucket object naming scheme:

• Is not content-addressed
• Prevents multiple logical references from resolving to a single physical object
• Increases storage cost and operational complexity
• Conflicts with Git-LFS pointer semantics (oid sha256:<digest>)

Without changing the naming convention, true LFS de-duplication cannot be reliably implemented.

---

Decision

We will change the bucket object naming convention to be strictly content-addressed, using the file’s SHA-256 digest as the canonical object key.

This is a breaking change.

Canonical Object Naming Form

<bucket>/<full-sha256>

Where:

• <full-sha256> is the lowercase hex-encoded SHA-256 digest of the file’s bytes

• Object identity is independent of:

  • Project
  • Repository
  • Filename
  • Upload time

No project- or path-derived components are included in the object key.

---

Rationale

1. Align storage identity with Git-LFS identity

Git-LFS already defines content identity as:

oid sha256:<digest>

Using the same identifier at the storage layer:

• Eliminates impedance mismatch
• Simplifies reasoning about correctness
• Makes de-duplication explicit rather than emergent

---

2. Enable true global de-duplication

With content-addressed object keys:

• Identical files are stored once
• Multiple logical references resolve to the same physical object
• Storage cost scales with unique content, not number of uploads

---

3. Improve integrity and auditability

A content-addressed object key ensures:

• Stored bytes must match their identifier
• Corruption is detectable
• Objects are naturally immutable by convention

---

Breaking Change Impact

Existing Clients

All existing clients must be updated.

Specifically:

• Upload paths change
• Download resolution logic changes
• Any client logic assuming project- or path-based bucket keys will break

Clients must treat the SHA-256 digest as the authoritative storage key.

---

Existing Projects and Data

All existing projects must be re-imported and tested.

Reasons:

• Legacy objects are stored under non-canonical names
• Safe de-duplication cannot be inferred retroactively
• Mixed naming schemes would introduce ambiguity and correctness bugs

Re-import ensures:

• Objects are rewritten or re-registered under canonical keys
• Metadata references are consistent
• De-duplication behavior is correct and verifiable

---

Migration Strategy (High Level)

1. Freeze legacy writes

   • Prevent new uploads using the old naming scheme

2. Client upgrade

   • Release updated clients that read/write <bucket>/<full-sha256>

3. Project re-import

   • Recompute SHA-256 for existing files
   • Register or rewrite objects under canonical keys
   • Verify integrity and references

4. Validation

   • Confirm identical files collapse to a single physical object
   • Run project- and workflow-level tests

5. Legacy cleanup (optional)

   • Garbage-collect unreachable legacy objects

---

Consequences

Positive

• Correct, global Git-LFS de-duplication
• Reduced storage footprint
• Stronger integrity guarantees
• Clean alignment with Git-LFS semantics
• Simpler long-term architecture

Negative

• Breaking change for all existing clients
• Mandatory re-import and testing for existing projects
• One-time migration and operational overhead

---

Non-Goals

• Backward compatibility with legacy bucket object paths
• Long-term support for mixed naming conventions
• Fully transparent migration without client or project involvement

---

Decision Summary

To enable correct and scalable Git-LFS de-duplication, the platform will adopt SHA-256–based content-addressed bucket object naming (<bucket>/<full-sha256>), accepting a breaking change that requires client upgrades and project re-imports.

[bwalsh]

Migration Estimate ⚖️

 

Scope Summary

 
This estimate covers the breaking-change migration to canonical SHA-256 storage keys and re-import of all legacy projects/data, including:
 

• Client upgrades for new upload/download behaviors.
• Re-import of existing projects (re-hash, re-register/rewrite, verify).
• Validation of de-duplication correctness and workflow integrity.
• Optional legacy cleanup (garbage collection).
   

Key Assumptions

 

• The new canonical object key format is <bucket>/<full-sha256> and is already finalized.
• Legacy writes can be fully frozen during the migration window.
• Re-import can be automated (batch tool or job), and storage throughput is sufficient.
• We can stage and roll out client changes to all active client types.
   

Rough Order of Magnitude (ROM) Timeline

 

Workstream	Activities	Effort (team weeks)	Notes
Client upgrade	Implement new upload path + download resolution, update SDKs/CLIs, release + rollout	1	Depends on number of clients + compatibility testing
Migration tooling	Build re-import pipeline (hashing, rewrite/register, metadata updates)	1	Reusable for all projects
Data re-import execution	Run re-import for all projects, verify integrity	2	Heavily dependent on data size and IO constraints
Validation & testing	Project/workflow tests, dedupe validation, regression fixes	3	Includes test environment setup
Legacy cleanup (optional)	GC legacy objects, storage verification	3	Optional, can be deferred
Estimated total: 3 team weeks (can be parallelized). Expect ~4 calendar weeks with 2–3 engineers plus QA support.

Risk Drivers / Schedule Sensitivities

 

• Data volume & IO throughput: Large datasets may elongate re-import time.
• Client diversity: Multiple language SDKs, custom clients, or offline clients increase rollout time.
• Integrity validation scope: Depth of workflow testing per project expands schedule.
• Legacy write freeze adherence: Any bypass extends re-import and verification windows.
   

Recommended Phased Plan

 

1. Preparation (Week 1)

• Finalize client API/SDK behavior changes.
• Build and test re-import tooling.
• Draft migration runbooks + rollback steps.
   

2. Client Rollout (Week 1)

• Ship updated clients and enforce new upload/download logic.
• Monitor adoption and detect incompatible callers.
   

3. Re-import Execution (Week 2)

• Batch re-hash and re-register/rewrite objects.
• Update metadata references to canonical keys.
   

4. Validation (Week 3)

• Verify dedupe correctness and storage integrity.
• Run project/workflow-level tests.
   

5. Legacy Cleanup (Optional, Week 3-4)

• Garbage-collect orphaned legacy objects.
   

Definition of Done

 

• All clients use canonical SHA-256 keys for upload and download.
• All projects re-imported and tested successfully.
• Metadata references only canonical keys.
• Dedupe verified for identical objects.
• (Optional) Legacy objects are cleaned up.

[bwalsh]

Alternative Solution: Add Optional use_prefix Parameter

Based on the PR changes to make_signed_url in the BlankIndex class, an proposed alternative approach:

Current PR Changes

The PR removes the GUID prefix from storage paths:

# BEFORE (line 386):
container_url = "{}/{}/{}".format(container, self.guid, file_name)

# AFTER:
container_url = "{}/{}".format(container, file_name)

# BEFORE (line 396):
s3_url = "s3://{}/{}/{}".format(bucket, self.guid, file_name)

# AFTER:
s3_url = "s3://{}/{}".format(bucket, file_name)

---

Proposed Alternative: Add use_prefix Parameter

Here's how the alternative would work:

def make_signed_url(self, file_name, protocol=None, expires_in=None, bucket=None, use_prefix=True):
    """
    Works for upload only; S3 or Azure Blob Storage only
    (only supported case for data upload flow currently).

    Args:
        file_name (str): Name of the file to upload
        protocol (str): Storage protocol ('az' for Azure, defaults to S3)
        expires_in (int): Expiration time in seconds
        bucket (str): Bucket name (optional, uses config default if not provided)
        use_prefix (bool): Whether to prefix the storage path with GUID. 
                          Defaults to True for backwards compatibility.
                          Set to False for Git LFS compatibility (content-addressed storage).

    Return:
        S3IndexedFileLocation or AzureBlobStorageIndexedFileLocation
    """

    # check if azure, and default to S3

    if protocol == "az":
        try:
            container = flask.current_app.config["AZ_BLOB_CONTAINER_URL"]
        except KeyError:
            raise InternalError(
                "fence not configured with data upload container; can't create signed URL"
            )
        
        # Use GUID prefix based on parameter
        if use_prefix:
            container_url = "{}/{}/{}".format(container, self.guid, file_name)
        else:
            container_url = "{}/{}".format(container, file_name)

        url = AzureBlobStorageIndexedFileLocation(container_url).get_signed_url(
            "upload", expires_in
        )
    else:
        if not bucket:
            bucket = flask.current_app.config["DATA_UPLOAD_BUCKET"]

        self.logger.debug("Attempting to upload to bucket '{}'".format(bucket))
        
        # Use GUID prefix based on parameter
        if use_prefix:
            s3_url = "s3://{}/{}/{}".format(bucket, self.guid, file_name)
        else:
            s3_url = "s3://{}/{}".format(bucket, file_name)
            
        url = S3IndexedFileLocation(s3_url).get_signed_url("upload", expires_in)

    self.logger.info(
        "created presigned URL to upload file {} with ID {}".format(
            file_name, self.guid
        )
    )

    return url

Additionally, you'd need to update the call site in IndexedFile._get_signed_url() (around line 624):

blank_record.make_signed_url(
    protocol=protocol,
    file_name=file_name,
    expires_in=expires_in,
    bucket=bucket,
    use_prefix=False  # or pass this through from caller
)

---

Analysis

✅ Advantages

1. Backwards Compatibility: By defaulting use_prefix=True, existing code continues to work without modification
2. No Data Migration Required: Both storage patterns can coexist:
   • Old files remain at {bucket}/{guid}/{filename}
   • New LFS files can be stored at {bucket}/{oid}
3. Gradual Migration: You can migrate incrementally by passing use_prefix=False only for LFS uploads
4. Explicit Intent: The code is self-documenting about when prefix is used vs. not used

❌ Problems & Considerations

1. Complexity: Adds conditional logic that needs to be maintained long-term

2. Indexd URL Consistency: The indexd record URLs field would contain mixed formats:

   • s3://bucket/guid/file.txt (old style)
   • s3://bucket/sha256-oid (new LFS style)

3. Caller Responsibility: Every caller needs to know and specify the correct use_prefix value. This creates opportunity for errors.

4. Not a Complete Solution: The fundamental issue is that Git LFS expects file_name to BE the OID, not an arbitrary filename. So you'd still need to:

   • Pass the SHA-256 OID as the file_name parameter
   • Ensure indexd records store the OID correctly
   • The use_prefix parameter becomes a workaround for path structure, but doesn't address the core semantic change

5. Ambiguous File Retrieval: When downloading, how do you know which pattern to use? You'd need to:

   • Inspect the indexd URL to determine if it has the GUID prefix
   • Add logic to handle both cases in retrieval code
   • Risk confusion when the same system has both patterns

6. Long-term Technical Debt: You're essentially supporting two different storage models indefinitely. This increases:

   • Testing surface area
   • Documentation burden
   • Onboarding complexity for new developers

7. Doesn't Eliminate Migration: While you avoid immediate data migration, you'd still eventually want to migrate old data to the new format to:

   • Simplify the codebase
   • Enable deduplication across all data
   • Ensure consistent behavior

🔄 Data Migration Considerations

Current PR approach (no prefix):

• Requires: Migrating existing indexd records and potentially moving files in S3/Azure
• Benefit: Clean break, single storage model going forward

Alternative approach (use_prefix parameter):

• Avoids immediate migration: Old data stays as-is
• Long-term: Still need migration to realize full benefits of content-addressed storage
• Hybrid state: System operates with two storage models, which has operational complexity

---

Recommendation

The use_prefix parameter approach delays but doesn't eliminate the need for migration. Consider:

• If you need Git LFS compatibility immediately and have significant existing data: The parameter approach provides a migration path
• If the dataset is small or mostly net-new: The clean break in the current PR is simpler long-term
• If you need both models permanently: Document why and ensure the dual-model approach is intentional, not transitional

[matthewpeterkort]

My opinion:

use_prefix sounds like a good middle-ground idea. but this is the type of legacy feature that effectively kicks the can down the road and will fix issues today but cause issues months/years from now.

I suspect that the best solution is to migrate our existing files to the new path format.

This GUID/SHa pattern is a headache to debug anyway since you need to know an arbitrary ID to manually check to see if the file exists in the bucket -- We should just move to the new SHA format immediately as it simplifies things immensely.