pandas.core.computation.ops.UndefinedVariableError

[sarah872]

Hi!
I am getting an error with the penultimate step of the Snakemake workflow:

pore_c --dask-scheduler-port 0 --dask-num-workers 4 contacts summarize AF_porec/merged_contacts/HhaI_run01_genome.2_unphased.contacts.parquet AF_porec/basecall/HhaI_run01.rd.summary.csv AF_porec/merged_contacts/HhaI_run01_genome.2_unphased.concatemers.parquet AF_porec/merged_contacts/HhaI_run01_genome.2_unphased.concatemer_summary.csv --user-metadata '{"run_id":"run01","enzyme":"HhaI","biospecimen":"lab","refgenome_id":"genome.2","phase_set_id":"unphased"}'

distributed.worker - WARNING -  Compute Failed
Function:  execute_task
args:      ((subgraph_callable, (<function read_parquet_part at 0x2aea104c3430>, <bound method ArrowEngine.read_partition of <class 'dask.dataframe.io.parquet.arrow.ArrowEngine'>>, <fsspec.implementations.local.LocalFileSystem object at 0x2aea28025160>, Empty DataFrame
Columns: [read_name, read_length, read_idx, contact_is_direct, contact_is_cis, contact_read_distance, contact_genome_distance, contact_fragment_adjacent, contact_fragment_distance, haplotype_pair_type, align1_align_idx, align1_chrom, align1_start, align1_end, align1_strand, align1_mapping_quality, align1_align_score, align1_align_base_qscore, align1_phase_set, align1_phase_qual, align1_haplotype, align1_fragment_id, align1_fragment_start, align1_fragment_end, align2_align_idx, align2_chrom, align2_start, align2_end, align2_strand, align2_mapping_quality, align2_align_score, align2_align_base_qscore, align2_phase_set, align2_haplotype, align2_fragment_id, align2_fragment_start, align2_fragment_end]
Index: [], [('/scratch/user/2
kwargs:    {}
Exception: UndefinedVariableError("name 'direct_trans' is not defined")

/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/arrays/categorical.py:2630: FutureWarning: The `inplace` parameter in pandas.Categorical.set_categories is deprecated and will be removed in a future version. Removing unused categories will always return a new Categorical object.
  res = method(*args, **kwargs)
Traceback (most recent call last):
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/scope.py", line 211, in resolve
    return self.temps[key]
KeyError: 'direct_trans'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/bin/pore_c", line 10, in <module>
    sys.exit(cli())
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pore_c/cli.py", line 743, in summarize
    concatemer_df = contacts_df.map_partitions(gather_concatemer_stats, meta=concatemer_meta).compute()
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/base.py", line 452, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/client.py", line 2725, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/client.py", line 1986, in gather
    return self.sync(
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/client.py", line 832, in sync
    return sync(
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/utils.py", line 340, in sync
    raise exc.with_traceback(tb)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/utils.py", line 324, in f
    result[0] = yield future
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/distributed/client.py", line 1851, in _gather
    raise exception.with_traceback(traceback)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/optimization.py", line 961, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/utils.py", line 29, in apply
    return func(*args, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/dask/dataframe/core.py", line 5298, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pore_c/analyses/contacts.py", line 472, in gather_concatemer_stats
    contact_df.groupby(["read_name", "contact_is_direct", "contact_is_cis"])
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/frame.py", line 4186, in eval
    return _eval(expr, inplace=inplace, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/eval.py", line 348, in eval
    parsed_expr = Expr(expr, engine=engine, parser=parser, env=env)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 806, in __init__
    self.terms = self.parse()
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 825, in parse
    return self._visitor.visit(self.expr)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 411, in visit
    return visitor(node, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 417, in visit_Module
    return self.visit(expr, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 411, in visit
    return visitor(node, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 632, in visit_Assign
    return self.visit(node.value, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 411, in visit
    return visitor(node, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 532, in visit_BinOp
    op, op_class, left, right = self._maybe_transform_eq_ne(node)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 452, in _maybe_transform_eq_ne
    left = self.visit(node.left, side="left")
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 411, in visit
    return visitor(node, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/expr.py", line 545, in visit_Name
    return self.term_type(node.id, self.env, **kwargs)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 98, in __init__
    self._value = self._resolve_name()
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/ops.py", line 115, in _resolve_name
    res = self.env.resolve(self.local_name, is_local=self.is_local)
  File "/scratch/user/2021/Pore-C-Snakemake/.snakemake/conda/5d22b593/lib/python3.8/site-packages/pandas/core/computation/scope.py", line 216, in resolve
    raise UndefinedVariableError(key, is_local) from err
pandas.core.computation.ops.UndefinedVariableError: name "name 'direct_trans' is not defined" is not defined

The test run finished successfully, So I'm not so sure what the error is here. I am only using a one-contig reference (a small prokaryotic genome).

The rule merge_contact_files finished successfully, and the input file *.summary.csv and the input directory *.contacts.parquet also exist.

Could you please have a look? Thank you!