Is it possible to merge csv files?

[ElSnato]

Hello, I want to use this package for my masters thesis.
In the description it said, that it is possible to Merge individual files into experiments but I was not able to find out how. I tried merging csv files into one combined file with: (found online)

extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]
#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ])
#export to csv
combined_csv.to_csv( "combined_csv.csv", index=False, encoding='utf-8-sig')

But it didnt work out because I found a different amount of objects in my pyfim experiment after that. Also the csv files were only like stacked on top of each other and I don't think they are usable in that way.

Is there another way, to merge objects from different csv files into one experiment?

[schlegelp]

Hi. I haven't looked at this package in a while but iirc "merging individual files into experiments" refers to Collections. See if the example in the docs helps: https://pyfim.readthedocs.io/en/latest/source/generated/pyfim.Collection.html#pyfim.Collection

Collections are used if you want to compare across different conditions (e.g. genotypes). If you just need to concatenate experiments from the same conditions then pd.concat is the right choice. But you have to concatenate along the second axis (i.e. concatenate the columns) since each trace is a column and rows are time points. By default, pd.concat will add rows at the bottom instead of columns to the left. Try this instead (note the axis=1 parameter).

extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]
#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ], axis=1)
# Check if it has the expect number of rows and columns 
print(combined_csv.shape)

By-the-by: I did reformat your above message into code using three ticks (`) before and after the code block.

[ElSnato]

Hello, thanks for your answer.

Yes, I wanted to combine data from experiments with the same conditions.
The axis=1 parameter was the right choice I think.
But I had to delete the rows like "mom(x)" afterwards. So I converted csv to xsls, then deleted the rows and then converted it back to csv. I guess there are better ways to do so but I'm new to python and using codes and the amount of data I want to merge is small enough, so this is doing it for me.
Saving the excel file as csv didn't work, neither did editing it with notepad and export as csv.

In case someone is having a similar issue here is what i did:

import os
import glob
import pandas as pd
os.chdir("typefolderdirectoryhere")
print(os.getcwd())
extension = 'csv'
all_filenames = [i for i in glob.glob('*.{}'.format(extension))]
#combine all files in the list
combined_csv = pd.concat([pd.read_csv(f) for f in all_filenames ], axis=1)
# Check if it has the expect number of rows and columns 
print(combined_csv.shape)
#export to csv
combined_csv.to_csv( "combined_csv.csv", index=False, encoding='utf-8-sig')

Then I deleted the extra rows (using Text to Columns function)
Then I converted the xsls file back to csv with

import pandas as pd

read_file = pd.read_excel (r'folder/test.xlsx', engine='openpyxl')
read_file.to_csv (r'folder/merge.csv', index = None, header=True)

The file could be opened with pyfim now.

Thanks for your help.