Think about switching from duplicating columns to list columns

[jpfeuffer]

Working with duplicated columns like "modification parameters" for multiple modifications is not great.
When loading with pandas for example you will receive renamed columns like "modification parameters.1" and then need to do a prefix matching to apply validation rules.

Having a list column would also make the transition to parquet or json or anything more structured easier.

[ypriverol]

This is a common and well-known issue since 1.0.0. We discuss long time back encoding multiple PTMs in one column for each file, but then it will be more unreadable than what is already.

It is a compromise between the human/machine-readable and tsv format. We can do better, as you said, if we go for JSON, @levitsky and I discussed before, a JSON representation with less duplication.

One solution is to provide in the sdrf-pipelines a reader and writer classes and interfaces that people don't need to duplicate that make that mistake, (this happen to me also a lot), specially in pandas; I think we can do an interface that enables that for writer and reading.

[levitsky]

Duplicated columns are indeed hard to deal with when writing code. We can support list columns in the spec and in the parser, however never knowing in advance if a value will be a string or a list is also not very convenient, or at least requires getting used to. Realistically it's almost any column that can be repeated at least in some cases.

To ensure predictable API we would probably have to treat all columns as lists at all times? With exceptions for source name, assay name and a few others.

[noatgnu]

Can we have a wrapper api rather than using pandas directly it would be an SDRF object that aware of possible duplication but having the same api as a normal pandas object. It will be like an SDK for SDRF frame where code involving SDRF can import to work with SDRF table.

[jpfeuffer]

Yeah possible but I was more thinking about external systems and code interfacing with the SDRF structure such that they do not need to depend on an API for every kind of "samplesheet" for every different type of Omics data.

Specifically I was thinking about data warehousing systems that support validation of column types like https://lamin.ai/ which is ubiquitous in the single cell community.

[jpfeuffer]

Similar thing for e.g. nextflow schema validators or any schema validator following open standards. I have not seen any of them (yet) supporting duplicated columns as this usually is notoriously bad for schema inference.

[jpfeuffer]

Doesn't have to be in 1.1 but it is food for thought.