Troubleshooting discussion

[matinnuhamunada]

A user found this issue running BGCFlow

What I did :

# create and activate a new conda environment

conda create -n bgcflow pip -y

conda activate bgcflow

# install `BGCFlow` wrapper

pip install git+[https://github.com/NBChub/bgcflow_wrapper.git](https://github.com/NBChub/bgcflow_wrapper.git)

# make sure to use bgcflow_wrapper version >= 0.2.7

bgcflow --version **My bgcflow version is 0.3.1**

#Set conda channel priorities to flexible

conda config --set channel_priority disabled

conda config --describe channel_priority

#Java (required to run metabase)

conda install openjdk 

#Deploy and run BGCFlow

bgcflow clone <your_bgcflow_directory> # clone `BGCFlow` to your_bgcflow_directory

cd <your_bgcflow_directory> # move to BGCFLOW_PATH

I cloned it as test.

In git hub page it says that

A new folder named will be created, containing this structure:
└── bgcflow
    ├── CITATION.cff
    ├── data
    ├── Dockerfile
    ├── envs.yaml
    ├── LICENSE
    ├── README.md
    └── workflow

But what i have inside the test folder is:

└── bgcflow
    ├── bgcflow_wrapper.log
    ├── CITATION.cff
    ├── config    
    ├── data
    ├── Dockerfile
    ├── emappertmp_dmdn_6nz2b1ki
    ├── emappertmp_dmdn_bj6a02y8
    ├── emappertmp_dmdn_n5d0_l6v
    ├── envs.yaml
    ├── LICENSE
    ├── logs
    ├── README.md
    ├── resources
    └── workflow

bgcflow init

bgcflow run -n

My configuration file looks like this (Screenshot 1)

When I did this dry run the terminal looked like screenshot 2(attached)

When I did example set the terminal looked like screenshot 3(attached)

When I checked the log file it showed me this error (screenshot 4).

There was a fasta folder but it was just empty. I copy pasted four fasta files (from another fasta file that bgcflow has but I couldn't remember) and again did an example set run. There was no error message. Did I do it right ? Was it supposed to be copy pasted by ourselves?

Then I did

bgcflow build report

Then I got three errors (I don't have screenshot for this one)

It was written:

Error in rule mkdocs_py_report

log file for first error (screenshot 5) I have no idea about what's happening here.

log file for second error (screenshot 6)

logfile for third error (screenshot 7)

Screenshot 6 and 7 says that  there is no file or folder as df_antismash_7.0.0_summary.csv and also there is no file or folder as df_arts_as-7.0.0.csv . But when I went to the table folder, there is df_arts_as-6.1.1.csv and df_antismash_6.1.1_summary.csv( screenshot 8).

Now I am confused about what to do?

I know I have asked a lot of questions. I am sorry for that. I am getting confused whether I am doing it in the right way or not. I kindly request you to see this once and give me some suggestions. It would be great if you could give me a basic idea of how this is supposed to be done.

[matinnuhamunada]

Folder structure

Don't worry, there is nothing wrong with the folder structure. There are indeed some new files/folder generated after running the workflow and is explained below:

└── bgcflow
   ├── bgcflow_wrapper.log # a log file generated by panoptes
   ├── CITATION.cff
   ├── config # config folder generated using bgcflow init    
   ├── data # generated data from snakemake run
   ├── Dockerfile
   ├── emappertmp_dmdn_6nz2b1ki # temporary eggnog folders
   ├── emappertmp_dmdn_bj6a02y8 # temporary eggnog folders
   ├── emappertmp_dmdn_n5d0_l6v # temporary eggnog folders
   ├── envs.yaml
   ├── LICENSE
   ├── logs # log folder created from snakemake runs
   ├── README.md
   ├── resources # downloaded resources from snakemake runs
   └── workflow

Notes & suggestion for developer:

• The default temporary emapper should be moved to the interim directory to avoid confusion
• The panoptes sqlite db file is not generated, related to issue Panoptes does not show progress #280

[matinnuhamunada]

GTDBtk Error

There was a fasta folder but it was just empty. I copy pasted four fasta files (from another fasta file that bgcflow has but I couldn't remember) and again did an example set run. There was no error message. Did I do it right ? Was it supposed to be copy pasted by ourselves?

In the latest .examples folder (or this template in particular .examples/Lactobacillus_delbrueckii/project_config.yaml), the GTDBtk is turned off. I believe you turned the setting on, which is totally fine, but there is an issue with the example project:

1. Yes, you did nothing wrong :) and you are not supposed to copy paste the files on your own
2. The reason you got the error is because all of the genome accessions are already in GTDB. This will cause error further along as there will be conflicting accession as the query is already in the database. BGCFlow handle this issue by not preparing the input files, hence the error: no genomes found in directory
3. I will suggest to run GTDBtk only if you have a custom genome (provide your own fasta file) or an NCBI assembly accession which is not already in the GTDB database: https://gtdb.ecogenomic.org/
4. If the genome is already in GTDB, BGCFlow will automatically capture the taxonomic information using GTDB API, and you will get it as a table in the data/processed/<project name>/tables/df_gtdb_meta.csv

Notes & suggestions for developer:

• Make it clear about the issue of already existing NCBI accession in GTDB
• Make the error message clear
• Don't even run the rule if all accession is already in GTDB

[matinnuhamunada]

AntiSMASH version error in building report

Screenshot 6 and 7 says that there is no file or folder as df_antismash_7.0.0_summary.csv and also there is no file or folder as df_arts_as-7.0.0.csv . But when I went to the table folder, there is df_arts_as-6.1.1.csv and df_antismash_6.1.1_summary.csv( screenshot 8).
Screenshot 8

Now I am confused about what to do?

The back compatibility feature of antiSMASH is something that I introduced quite recently, so forgive me if there might still be a lot of issues with it. Nevertheless, I think I have a rough idea of what happens in this case:

1. CMIW. In the config file, you changed the default parameter of antiSMASH from 7 to 6:

rule_parameters:
  install_gtdbtk:
    release: "214.1"
    release_version: "r214"
  gtdbtk:
    ani_screen: FALSE
  antismash:
    version: "6" # valid versions: 6, 7

2. You might have run BGCFlow previously with the default setting, which generate the project metadata, then you changed the rule_parameters. You can find the metadata files here: data/processed/Lactobacillus_delbrueckii/metadata/dependency_versions.json and check what version of antiSMASH is recorded there:

{
  "antismash": "7.0.0",
  "prokka": "1.14.6",
  "mlst": "2.19.0",
  "eggnog-mapper": "2.1.6",
  "roary": "3.13.0",
  "refseq_masher": "0.1.2",
  "seqfu": "1.15.3"
}

3. I'm guessing that the antismash version recorded is still the default (7.0.0). You can try to manually change this into 6.1.1 and see if it resolves the problem. Another way is just to delete the metadata file, and BGCFlow should be able to regenerate it to accommodate the latest config change.

Notes & suggestions for developer:

• BGCFlow should be able to automatically detect if the metadata files are up to date with the latest config or not.

[shailaza1]

I manually changed it to 6.1.1. It looks like its working now.

[shailaza1]

Here are some more issues when I tried to serve the project using :bgcflow serve --project Lactobacillus_delbrueckii

[matinnuhamunada]

If you want, we can have a quick zoom session

[shailaza1]

I am running this on the remote server. I am trying to forward the port to local machine. But its not working right now.
After 36hrs of running bgcflow serve --project Lactobacillus_delbrueckii the terminal looks like:

[shailaza1]

Sure, we can do a zoom session. Plz let me know your which day and time works for you.
Thank you!

[matinnuhamunada]

How about Thursday? I'm quite flexible with the time

[matinnuhamunada]

Hi, I didn't know if you checked the email or not. But can you confirm if we are having this meeting on zoom today?

[shailaza1]

Hi Matin,
I have problem running bigslice (screenshot 1)

The log file shows this error (screenshot2)

[shailaza1]

Do you have any idea how long it will take to fix?

[matinnuhamunada]

I ended up making my own fork of bigslice. The feature is now available in this PR: #291
Will do some tests before integrating it to main.

You can try it out by switching to the development branch:

git fetch
git status
git pull
git checkout dev-0.7.7

[matinnuhamunada]

It's now merged in the main branch. I might have small issue with processing roary output, but will make the release soon.

[matinnuhamunada]

And the report should look like this: https://nbchub.github.io/bgcflow_report_demo/

[shailaza1]

Thank you Matin! Really appreciate your help.

[shailaza1]

Hi Matin,
I have another issue with rule arts (Screenshot 1).
When i checked log file it shows me the error message which I can't solve (screenshot 2)

[matinnuhamunada]

This seems to be numpy vs matplotlib incompatibility?
I can't reproduce this error on a freshly installed Ubuntu VM on Azure.

Can you do two things for me:

1. Which branch are you in right now?

git fetch
git status
git log

2. What is in the conda environment?

conda activate /ddn/home/r31.... # the name of the conda env shown in the snakemake log
conda env export

Maybe you can pass what shown on the terminal to a text file and that will be really helpful to find out what is happening.

[shailaza1]

I will try this and let you know.

[shailaza1]

Actually I didn't wait for your reply and did git fetch followed by git pull.
After doing this, those errors are gone but I can still see other 2 errors ( screenshot_3)
I know error about gtdbtk but have no idea about query_bigslice

[matinnuhamunada]

Can you also pass the log for query_bigslice here?
Are you on the main branch?

For the gtdbtk was it because all the genomes already exist in GTDB?

[shailaza1]

okay.

[shailaza1]

Yes I am on the main branch

[shailaza1]

Can you also pass the log for query_bigslice here? Are you on the main branch?

For the gtdbtk was it because all the genomes already exist in GTDB?

[matinnuhamunada]

I don't think I can solve this issue right now as it seems to be related to the system you are in.

It might be the same issue as mentioned here: python/cpython#87909

Can confirm that this BlockingIOError happens on GPFS (alpine) on Summit supercomputer, tested with Python 3.8 and 3.10a7.

I found that it happens only for file sizes above 65536

Will try to work around by using shell script instead of shutil

[shailaza1]

Okay. I shall check this let you know. Thank you Matin !

[matinnuhamunada]

Actually, it seems simple enough to make a workaround (#295).
Can you try this:

git fetch
git checkout dev-0.7.8

Then run it again?

[shailaza1]

it gives the same error. No any change.

[shailaza1]

I have some questions for you:

1. I did bgcflow build report -c 64. I got 18 error files which are similar (screenshot_4)
   Log file has same error msg (Screenshot_5)

2. Then I tried bgcflow build report -c 32. I can still see that problem.

3. Now when I did bgcflow build report , I can see only two errors (screenshot_6 and 7)

and their logfile has error msg as (screenshot_8) Why is this?Am I doing it in a wrong way?

[matinnuhamunada]

Oh, this is expected I guess, because you haven't finished the bgcflow run for bigslice_query and gtdbtk. You should still be able to serve the report though, albeit the page for those two will be empty

[shailaza1]

Hi Matin,
How can I fix this error?

[matinnuhamunada]

Thanks for pointing this out, it seems to be a problem from Jupyter. There is an open issue here: jupyter/jupyter_client#154