Implement incremental upload of tab delimited data. #38

forus · 2024-05-15T09:34:52Z

See #45 instead.

- Calculate number of lines in the file in the loader - Remove unused imports and fields - Reuse constructors - Reuse common parsing logic in tab delimiter importer - Show full stacktrace which helps in dinding where tests errored out

…pload

It's dangerous as we would further mess up the data in the row

Do it for all data types, not only MAF

- Split big tab. delim test to multiple tests based on data type. - Use ImportProfileData instead of ImportTabDelimData for testing. - We cover more logic with such tests. - This is more stable interface. ImportTabDelimData can be refactored.

…ploader

…ation rows or not

args variable was not declared

haynescd · 2024-05-29T04:15:04Z

src/main/java/org/mskcc/cbio/portal/dao/DaoCnaEvent.java

+        Connection con = null;
+        PreparedStatement pstmt = null;
+        ResultSet rs = null;
+        try {


Is there a reason you don't use try with resources? You might have already answered this

haynescd

just had a couple questions, but looks good

forus added 20 commits May 15, 2024 11:27

Refactor tab delim. data importer

b2c1c21

- Calculate number of lines in the file in the loader - Remove unused imports and fields - Reuse constructors - Reuse common parsing logic in tab delimiter importer - Show full stacktrace which helps in dinding where tests errored out

Implement incremental upload of mRNA data

a7aab3a

Add RPPA test

bd2d8c1

Add normal sample to thest data to test skipping

8b68331

Add rows with more columns then in header to skip

b18aab1

Skip rows that don't have enough sample columns

ea688c3

Test for invalid entrez id

cdae501

Extract common code from inc. tab. delim. tests

cf458a4

Implement incremntal upload of cna data via tab. delim. loader

9ea1ada

Blanken values for genes not mentioned in the file

03f9660

Remove unused code

93cc6ff

Throw unsupported operation exception for GENESET_SCORE incremental u…

842bcd3

…pload

Add generic assay data incremental upload test

22b688a

Fix integration tests

d11a353

Make tab. delimiter data uploader transactional

7dfb1bd

Check for illegal state in tab delim. data update

71cdf70

It's dangerous as we would further mess up the data in the row

Wire incremental tab delim. data upload to cli commands

2d31dac

Expand README with section on how to run incremental upload

4997542

Address TODOs in tab delim. importer

911ae28

Add more data types to incremental data upload folder

c7343f9

forus requested a review from haynescd May 15, 2024 09:35

forus added 7 commits May 15, 2024 11:42

Remove obsolete TODO comment

2ed0bd8

Reuse genetic_profile record if it exists in db already

76b52a9

Do it for all data types, not only MAF

Move counting lines if file inside generic assay patient level data u…

e5ccc3e

…ploader

Give error that generic asssay patient level data is not supported

472f47e

Clean sample_cna_event despite whether it has alteration_driver_annot…

c54e303

…ation rows or not

Fix cbioportalImport script execution

18dbdd3

args variable was not declared

haynescd reviewed May 29, 2024

View reviewed changes

forus closed this Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement incremental upload of tab delimited data. #38

Implement incremental upload of tab delimited data. #38

forus commented May 15, 2024 •

edited

Loading

haynescd May 29, 2024 •

edited

Loading

haynescd left a comment

Implement incremental upload of tab delimited data. #38

Implement incremental upload of tab delimited data. #38

Conversation

forus commented May 15, 2024 • edited Loading

haynescd May 29, 2024 • edited Loading

Choose a reason for hiding this comment

haynescd left a comment

Choose a reason for hiding this comment

forus commented May 15, 2024 •

edited

Loading

haynescd May 29, 2024 •

edited

Loading