FB Fails if Indices are Missing/Out of Order

[xehu]

It appears that the featurebuilder (and in particular, the mimicry feature) assumes that successive rows of the input dataframe increment in their row index. If the dataframe has missing indices (for example, because it was filtered before processing such that a few rows were dropped), the FB fails with the following error:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /opt/anaconda3/envs/tpm_virtualenv/lib/python3.11/site-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key)
   3804 try:
-> 3805     return self._engine.get_loc(casted_key)
   3806 except KeyError as err:

File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()

File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()

File pandas/_libs/hashtable_class_helper.pxi:2606, in pandas._libs.hashtable.Int64HashTable.get_item()

File pandas/_libs/hashtable_class_helper.pxi:2630, in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 813

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
Cell In[88], line 19
      1 # first, run the feature builder on the labeled utterances
      2 labeled_reddit_feature_builder = FeatureBuilder(
      3 	input_df=df_labeled_utterances_to_annotate,
      4 	custom_features=[
...
   3815     #  InvalidIndexError. Otherwise we fall through and re-raise
   3816     #  the TypeError.
   3817     self._check_indexing_error(key)

KeyError: 813
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[sundy1994]

@xehu Can you attach the code you ran? I will reproduce the error

[xehu]

@sundy1994 it was:

	labeled_reddit_feature_builder = FeatureBuilder(
		input_df=df_labeled_utterances,
		custom_features=[
			"(BERT) Mimicry",
			"Moving Mimicry",
			"Forward Flow",
			"Discursive Diversity"
		],
		conversation_id_col="CONV_ID",
		speaker_id_col="speaker",
		message_col="text",
		timestamp_col="timestamp",
		vector_directory="./vector_data/",
		output_file_base="labeled_utterances_reddit",
		timestamp_unit='ns'    
	)
	labeled_reddit_feature_builder.featurize()

(see slack DM's for data!)

[sundy1994]

@xehu I can't reproduce the same error. Did you filter the dataframe? Please see my test below.

If I run feature builder like this, I got a timestamp error when generating chat level features:

df_labeled_utterances = pd.read_csv('labeled_utterances_to_annotate.csv')
labeled_reddit_feature_builder = FeatureBuilder(
    input_df=df_labeled_utterances,
    custom_features=[
        "(BERT) Mimicry",
        "Moving Mimicry",
        "Forward Flow",
        "Discursive Diversity"
    ],
    conversation_id_col="CONV_ID",
    speaker_id_col="speaker",
    message_col="text",
    timestamp_col="timestamp",
    vector_directory="./vector_data/",
    output_file_base="labeled_utterances_reddit",
    timestamp_unit='ns'    
)
labeled_reddit_feature_builder.featurize()

# Output
Traceback (most recent call last):
  File "/Users/yuxuanzh/Documents/Code/team-process-map/venv/lib/python3.11/site-packages/team_comm_tools/features/temporal_features.py", line 30, in get_time_diff
    df.loc[i, "time_diff"] = (df.loc[i, on_column] - df.loc[(i-1), on_column]) / pd.Timedelta(seconds=1)
                             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~
TypeError: unsupported operand type(s) for /: 'numpy.float64' and 'Timedelta'

Then, I add a line parsing the data type before calling feature builder:
df_labeled_utterances['timestamp'] = df_labeled_utterances['timestamp'].astype(np.int64)
The output is all good without any error.

# Output
Initializing Featurization...
Confirmed that data has conversation_id_col column: CONV_ID!
Confirmed that data has speaker_id_col column: speaker!
Confirmed that data has message_col column: text!
Confirmed that data has timestamp_col column: timestamp!
Chat Level Features ...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:50<00:00,  2.95s/it]
Generating features for the first 100.0% of messages...
Generating User Level Features ...
Generating Conversation Level Features ...
All Done!

[sundy1994]

The error can't be reproduced because the indexes were reset when saving the csv file. After manually mess up the indexes, it turns out the key error was from computeTF() in features/word_mimicry.py:

tfdict = {}
wf = pd.Series(column_mimc, dtype='str').value_counts()
for i in wf.index:
    tfdict[i] = wf[i]/frequency_dict[i]
return sum(tfdict.values())

Here, frequency_dict will throw a key error if some indexes are missing. We can do reset_index() at the beginning of initialization to resolve this.