Skip to content
This repository has been archived by the owner on Mar 23, 2023. It is now read-only.

Possible Pyarrow Bug #41

Open
wenluo711 opened this issue Sep 29, 2022 · 4 comments
Open

Possible Pyarrow Bug #41

wenluo711 opened this issue Sep 29, 2022 · 4 comments

Comments

@wenluo711
Copy link

Dear PoreC pipeline developer,

I'm having an issue at the rule create_hicRef and import_basecalls for the test dataset. It seems to suggest pyarrow issue.
Below is the log file from results.test/logs/results.test/juicebox/NlaIII_GRCh38_p13.hicRef.log

$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
Fatal error condition occurred in /home/conda/feedstock_root/build_artifacts/aws-c-io_1636051483861/work/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_backtrace_print+0x59) [0x2aaab8621579]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_fatal_assert+0x48) [0x2aaab86116c8]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x13ad3) [0x2aaab885dad3]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aaab862360d]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../.././././libaws-c-io.so.1.0.0(+0x113ca) [0x2aaab885b3ca]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../../././libaws-c-common.so.1(aws_ref_count_release+0x1d) [0x2aaab862360d]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../../././libaws-crt-cpp.so(_ZN3Aws3Crt2Io15ClientBootstrapD1Ev+0x3a) [0x2aaab85c4f5a]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/lib/python3.8/site-packages/pyarrow/../../.././libaws-cpp-sdk-core.so(+0x5f570) [0x2aaab7fcd570]
/lib64/libc.so.6(+0x39bd9) [0x2aaaaaf24bd9]
/lib64/libc.so.6(+0x39c27) [0x2aaaaaf24c27]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(+0x24b539) [0x55555579f539]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(+0x24b56d) [0x55555579f56d]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(+0x24b5c4) [0x55555579f5c4]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(+0x109f75) [0x55555565df75]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(Py_RunMain+0x313) [0x5555557a2533]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(Py_BytesMain+0x39) [0x5555557a2779]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2aaaaaf0d445]
$PATH/Pore-C-Snakemake/.snakemake/conda/57f2c509/bin/python3.8(+0x1d7863) [0x55555572b863]

And results.test/logs/results.test/basecall/NlaIII_run01.rd.catalog.yaml.log looks exactly like a former solved issue #18

Thanks!
Wen
@eharr
Copy link
Collaborator

eharr commented Oct 3, 2022

Hi @wenluo711 - I just merged @aallahyar's fix, would you mind giving it a try?

@wenluo711
Copy link
Author

Hi @eharr Nice fix!
I'm able to run through the previous kinks with the updated environment.
However, I'm having another error at the last step summarise_contacts. You might have some idea of what is this about again?

Thanks,
Wen

/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.Float64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/utils.py:367: FutureWarning: pandas.UInt64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.
  _numeric_index_types = (pd.Int64Index, pd.Float64Index, pd.UInt64Index)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae88f9b80>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab00991b20>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: UndefinedVariableError("name 'indirect_cis' is not defined")

/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/contextlib.py:120: UserWarning: Creating scratch directories is taking a surprisingly long time. This is often due to running workers on a network file system. Consider specifying a local-directory to point workers to write scratch data to a local disk.
  next(self.gen)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2747: FutureWarning: The `inplace` parameter in pandas.Categorical.set_categories is deprecated and will be removed in a future version. Removing unused categories will always return a new Categorical object.
  res = method(*args, **kwargs)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2747: FutureWarning: The `inplace` parameter in pandas.Categorical.set_categories is deprecated and will be removed in a future version. Removing unused categories will always return a new Categorical object.
  res = method(*args, **kwargs)
/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2747: FutureWarning: The `inplace` parameter in pandas.Categorical.set_categories is deprecated and will be removed in a future version. Removing unused categories will always return a new Categorical object.
  res = method(*args, **kwargs)
Traceback (most recent call last):
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/scope.py", line 208, in resolve
    return self.temps[key]
KeyError: 'indirect_cis'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/bin/pore_c", line 10, in <module>
    sys.exit(cli())
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pore_c/cli.py", line 791, in summarize
    concatemer_df = contacts_df.map_partitions(gather_concatemer_stats, meta=concatemer_meta).compute()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/client.py", line 2725, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/client.py", line 1986, in gather
    return self.sync(
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/client.py", line 832, in sync
    return sync(
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
    result[0] = yield future
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/distributed/client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/optimization.py", line 961, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/utils.py", line 29, in apply
    return func(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/dask/dataframe/core.py", line 5298, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pore_c/analyses/contacts.py", line 471, in gather_concatemer_stats
    contact_df.groupby(["read_name", "contact_is_direct", "contact_is_cis"])
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/frame.py", line 4240, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/eval.py", line 351, in eval
    parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 811, in __init__
    self.terms = self.parse()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 830, in parse
    return self._visitor.visit(self.expr)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 421, in visit_Module
    return self.visit(expr, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 636, in visit_Assign
    return self.visit(node.value, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 536, in visit_BinOp
    op, op_class, left, right = self._maybe_transform_eq_ne(node)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 458, in _maybe_transform_eq_ne
    right = self.visit(node.right, side="right")
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 415, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 549, in visit_Name
    return self.term_type(node.id, self.env, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 98, in __init__
    self._value = self._resolve_name()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 115, in _resolve_name
    res = self.env.resolve(self.local_name, is_local=self.is_local)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/f0b8d608/lib/python3.8/site-packages/pandas/core/computation/scope.py", line 213, in resolve
    raise UndefinedVariableError(key, is_local) from err
pandas.core.computation.ops.UndefinedVariableError: name "name 'indirect_cis' is not defined" is not defined

@eharr
Copy link
Collaborator

eharr commented Oct 6, 2022

Ugh - that looks like some weird interaction between pandas and dask maybe? Can you try pinning the version of pandas to an earlier one by changing your envs/pore_c.yml to this?

channels:
- conda-forge
- bioconda
- defaults
dependencies:
- pore-c==0.4.0
- python==3.8
- aws-sdk-cpp=1.8.186=h9ad65fb_2
- pandas==1.0.5

@wenluo711
Copy link
Author

With the new pore_c environment, the Warnings are all gone but the error still persist

 cat merged_contacts/NlaIII_run02_GRCh38_unphased.concatemers.parquet.log
distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae1aee9d0>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab000c6af0>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: Exception("name 'indirect_cis' is not defined")

distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae1aee9d0>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab0019bac0>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: Exception("name 'indirect_cis' is not defined")

distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae1aee9d0>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab0013cbe0>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: Exception("name 'direct_long_range' is not defined")

distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae1aee9d0>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab0016dca0>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: Exception("name 'indirect_cis' is not defined")

distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aaae1aee9d0>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aab000c6af0>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/data/DCEG_HPV/MikeD
kwargs:    {}
Exception: Exception("name 'direct_cis' is not defined")

Traceback (most recent call last):
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/bin/pore_c", line 10, in <module>
    sys.exit(cli())
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pore_c/cli.py", line 791, in summarize
    concatemer_df = contacts_df.map_partitions(gather_concatemer_stats, meta=concatemer_meta).compute()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/client.py", line 2725, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/client.py", line 1986, in gather
    return self.sync(
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/client.py", line 832, in sync
    return sync(
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
    result[0] = yield future
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/distributed/client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/optimization.py", line 961, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/utils.py", line 29, in apply
    return func(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/dask/dataframe/core.py", line 5298, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pore_c/analyses/contacts.py", line 471, in gather_concatemer_stats
    contact_df.groupby(["read_name", "contact_is_direct", "contact_is_cis"])
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/frame.py", line 3346, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/eval.py", line 332, in eval
    parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 764, in __init__
    self.terms = self.parse()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 781, in parse
    return self._visitor.visit(self.expr)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 375, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 381, in visit_Module
    return self.visit(expr, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 375, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 598, in visit_Assign
    return self.visit(node.value, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 375, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 497, in visit_BinOp
    op, op_class, left, right = self._maybe_transform_eq_ne(node)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 418, in _maybe_transform_eq_ne
    right = self.visit(node.right, side="right")
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 375, in visit
    return visitor(node, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 510, in visit_Name
    return self.term_type(node.id, self.env, **kwargs)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 82, in __init__
    self._value = self._resolve_name()
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 99, in _resolve_name
    res = self.env.resolve(self.local_name, is_local=self.is_local)
  File "/gpfs/gsfs10/users/DCEG_HPV/MikeDean_nanopore/run/Pore-C-Snakemake/.snakemake/conda/82819d64/lib/python3.8/site-packages/pandas/core/computation/scope.py", line 203, in resolve
    raise UndefinedVariableError(key, is_local)
Exception: name 'indirect_cis' is not defined

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eharr @wenluo711