Fix a few bugs in fuzzy dedup and docs (NVIDIA#156)

ryantwolf · web-flow · commit e65428138863 · 2024-07-29T10:21:10.000-07:00
* Fix arg type

Signed-off-by: Ryan Wolf &lt;rywolf@nvidia.com&gt;

* Fix arg in docs

Signed-off-by: Ryan Wolf &lt;rywolf@nvidia.com&gt;

---------

Signed-off-by: Ryan Wolf &lt;rywolf@nvidia.com&gt;
diff --git a/docs/user-guide/gpudeduplication.rst b/docs/user-guide/gpudeduplication.rst
@@ -154,7 +154,7 @@ steps (all scripts are included in the :code:`nemo_curator/scripts/` subdirector
 
                  # same as `python connected_components.py`
                  gpu_connected_component \
-                   --jaccard-pairs_path /path/to/dedup_output/jaccard_similarity_results.parquet \
+                   --jaccard-pairs-path /path/to/dedup_output/jaccard_similarity_results.parquet \
                    --output-dir /path/to/dedup_output \
                    --cache-dir /path/to/cc_cache \
                    --jaccard-threshold 0.8
diff --git a/nemo_curator/scripts/fuzzy_deduplication/connected_components.py b/nemo_curator/scripts/fuzzy_deduplication/connected_components.py
@@ -67,7 +67,7 @@ def attach_args(parser=None):
     )
     parser.add_argument(
         "--jaccard-threshold",
-        type=int,
+        type=float,
         default=0.8,
         help="Jaccard threshold below which we don't consider documents"
         " to be duplicate",

Original file line number	Diff line number	Diff line change
`@@ -67,7 +67,7 @@ def attach_args(parser=None):`
`67`	`67`	`)`
`68`	`68`	`parser.add_argument(`
`69`	`69`	`"--jaccard-threshold",`
`70`		`- type=int,`
	`70`	`+ type=float,`
`71`	`71`	`default=0.8,`
`72`	`72`	`help="Jaccard threshold below which we don't consider documents"`
`73`	`73`	`" to be duplicate",`