Deprecate column "type prefixes"?

[jpfeuffer]

I don't even know how to call them since I never understood their use.
I mean using "organism" instead of "characteristic[organism]"

Especially now with the templates, you can represent "groups" of columns differently if you desperately think this is necessary.

[ypriverol]

Thanks @jpfeuffer, I think this is a good suggestion, but I need to point out the entire picture. I do think it is a big change a it is important to mention, we can do the change now and forever, or wait for the next release. But this is good for discussion; thanks again for triggering this; I have been thinking a lot about this.

Current Design Rationale

The prefixes exist because:

1. MAGE-TAB/ISA-TAB compatibility - SDRF-Proteomics was designed to be "fully compatible with the SDRF
   component of MAGE-TAB v1.1". This enables multi-omics integration with transcriptomics data.
2. Semantic grouping - The prefixes categorize columns:
   • characteristics[...] → biological sample properties
   • comment[...] → technical/data file properties
   • factor value[...] → experimental variables

---

Advantages of Removing Prefixes

• Less verbose: organism vs characteristics[organism]
• Fewer typos: We've already seen errors like ccaratistics
• Easier parsing: Simple key-value pairs without bracket extraction
• Better readability: Column headers are cleaner and more intuitive

---

Challenges

1. Backward Compatibility

This would require a v2.0.0 version bump. Existing tools and SDRFs would need:

• Converters to migrate v1.x files
• Parsers that support both formats during transition

2. The Boundary Problem

Without prefixes, distinguishing column categories becomes ambiguous:

source name | organism | disease | assay name | instrument | data file | treatment
^-- sample props --^ ^-- data props --^ ^-- factor? --^

How does a parser know if treatment is a data property or a factor value?

---

Possible Solutions

Option 1: Keep prefixes (status quo)

• ✅ Backward compatible, ISA-TAB compatible
• ❌ Verbose, error-prone

Option 2: Remove prefixes, use schema as source of truth

The templates already define column categories. A v2.0 parser uses the schema:

columns:
  - name: organism          # was: characteristics[organism]
    category: characteristic
  - name: instrument        # was: comment[instrument]
    category: comment
  - name: treatment
    category: factor_value

- ✅ Clean column names
- ✅ Schema-driven validation
- ❌ Requires template for parsing

Option 3: Section marker for factor values

Use a marker column to delimit where factor values begin:

source name | organism | disease | assay name | instrument | data file | factor values | treatment | dose
            ^-- sample --^        ^-------- data properties --------^   ^-- marker --^  ^-- factors --^

Marker column specification:
┌───────────────┬─────────────────────────────┬─────────────────────────────────────────────────────┐
│  Column name  │         Requirement         │                     Description                     │
├───────────────┼─────────────────────────────┼─────────────────────────────────────────────────────┤
│ factor values │ Required (if factors exist) │ Delimiter marking the start of factor value columns │
└───────────────┴─────────────────────────────┴─────────────────────────────────────────────────────┘
Row values for the marker:

Option A: Count of factors (machine-readable)
source name | organism    | data file      | factor values | treatment | dose
sample_1    | homo sapiens| file1.raw      | 2             | drug_A    | 10mg
sample_2    | homo sapiens| file2.raw      | 2             | drug_B    | 20mg

Option B: Factor names as value (self-documenting)
source name | organism    | data file      | factor values       | treatment | dose
sample_1    | homo sapiens| file1.raw      | treatment;dose      | drug_A    | 10mg
sample_2    | homo sapiens| file2.raw      | treatment;dose      | drug_B    | 20mg

Recommended: Option B - It's self-documenting, machine-parseable, and allows validation that all listed
factors exist as subsequent columns.

- ✅ Clear visual and logical separation
- ✅ No prefixes needed
- ✅ Backward compatible concept (similar to how source name and assay name already act as section markers)
- ❌ Adds one extra column

---

Recommendation

If we pursue this change, Option 2 (schema-driven categories) combined with Option 3 (factor marker) provides
the best balance:

1. v2.0.0 removes prefixes entirely
2. Templates define the category for each column
3. A factor values marker column delimits where factors begin
4. Provide a converter tool in sdrf-pipelines:
   parse_sdrf convert --from v1 --to v2 input.sdrf.tsv
5. Validator accepts both formats during a transition period (e.g., 1 year)

---

Open Questions

1. How important is ISA-TAB/MAGE-TAB compatibility going forward? If multi-omics integration is critical, we
   need a compatibility layer. If proteomics has diverged enough, a cleaner v2.0 format is justified.
2. Timeline: What's a reasonable deprecation period for v1.x format support?
3. Factor marker value: Which option (A or B) is preferred for the marker column values?

[Cajac102]

I would be in favor of dropping them as well, mainly because they are clunky and imho confusingly named (e.g. why are technical details called comments?). If it would break compatibility with other formats, that would be a good argument to keep them anyways, although that might be fixable with a 1-1 mapping (but then there would be a need for that mapping, which might be a pain).

[levitsky]

This seems to come down to question # 1: do we still want the compatibility with ISA-TAB/MAGE-TAB? With v1 my impression was that the format was moving toward more compatibility at the expense of some extra verbosity. If reversing the course is on the table, perhaps we can try to make a list of everything done so far to ensure this compatibility and reevaluate.

This change will indeed simplify the files but also make all existing SDRF files invalid. We would need a plan to deal with that. Realistically we could probably batch-convert them on PRIDE and here.

I think the general consensus was that we don't try to seek simplicity and readability with the TSV representation and find new representations that are more concise (like JSON). However I don't have a strong opinion on that.

[ypriverol]

I think for back-compatibility we can try to use the validator in someway. In PRIDE, we have now around 500 SDRFs, which is like 1% of the entire PRIDE. In this repo, when we do major changes like we are proposing with 1.1.0, I try to do the conversion myself.

I think you make an important point about if we want to keep backward compatibility or not, I will trigger that discussion in PSI, from my point of view it was a good idea 5 years ago, but now because we have encorage some rules we can do that compatibility easily convertible like everything after source name is sample (characteristics), everything after assay name is data file.

You also raise another big point now that we are doing that, which should be done more; I think this change proposed by @jpfeuffer, while it is making the format not valid by previous SDRF transcriptomics parsers, it does allow easy conversion. Other changes, like moving away from ontologies or more complex things, will completely disconnect initiatives. The other things that I have thought in the past are source name -> sample accession and assay name -> assay accession, but they are really minor, and I think source and assay are quite easy to understand.

[ypriverol]

@daichengxin @jonasscheid @julianu @timosachsenberg @TineClaeys @noatgnu @nithujohn @deeptijk @enryH and others @bigbio/sdrf-community Can you give feedback here.

[noatgnu]

From my point of view, the type is like a minimal guardrail with the intention to make it possible to have someone in a linear chronological fashion, create the sdrf file using excel without any additional tool but excel and the spec. Unless we intend to write spec for program to create only, we should keep the spec with light guardrail built-in where the user can start creating the document before they even start the wet lab. Concurrently, all the characteristics would be filled as the experiment begin. The table would expand and got pickup by the MS scientist assigned to run the job where they can fill out the rest. Then the statistical analysis that have been done can be derived by factor value. We should avoid making the file something that is only to be think about at the end of the experiment. We can change how the type look but they should be there.

If we want to keep compatibility and make it easier for other to merge the same experiment across different omics, then we would keep MAGE-TAB. Or maybe have a discussion with other omics on how they would like to adapt MAGE-TAB.

I think at this point, for dev, we might have a future json format but for everyone else, we should have the table with light guardrail that can be reasonably created and completed with only excel and the spec document.

As someone who have convinced their research network to use this as metadata for proteomics, I definitely vote to keep the same format.

[timosachsenberg]

This is a breaking change and I would prefer keeping the format stable for now.

[ypriverol]

BTW I took a picture how this will render for those users that are not familiar with the format.

[timosachsenberg]

not a statistician so could be wrong. Would just "factors" be more accurate?
I think factor values exist as a term in statistics and refer to "levels".

[jonasscheid]

This is a breaking change and I would prefer keeping the format stable for now.

Agreed. As much as I like increasing readability, I am not sure about the consequences for the (still small?) userbase. I think in general the less you do breaking changes in the format, the more the people stick with it because there is no frustration going on.

If you would go for a breaking change I would throw in the discussion cage that we should rethink if it is allowed to have 2x/3x the same colname in the sdrf. It's a bit clunky if you think of e.g. reading in the sdrf with pandas. But that would be another discussion thread I guess.