Thresholds Not Updated After Thresholds Tuning

[prawiraelang]

I have experimented with tuning the thresholds, but it appears that the threshold is not being updated. I am using AzureOpenAIEncoder with a default threshold value of 0.82. Upon examining the code, I found the following section:

# update best
if acc > best_acc:
    best_acc = acc
    best_thresholds = thresholds

The above code snippet shows that the threshold is only updated if a new threshold with better accuracy is found.

My initial accuracy is around 23%, but it has not changed after attempting to tune the threshold because the threshold is not being updated. However, when I changed the default threshold to 0.0, I achieved nearly 100% accuracy. I am concerned that there might be something wrong with the fit mechanism, especially the search range, because the threshold should at least be updated to 0.0, as it clearly performs better than 0.8. Is anyone else experiencing this behavior?

[andreped]

@prawiraelang If you think this is a bug, you should open an issue about it not a discussion. Otherwise, this bug might go unnoticed.

[prawiraelang]

Thanks @andreped for the suggestion. I opened a Discussions first instead of an Issue to ensure this behavior is not just happening to me. After checking the current Issues, I found that someone has already created a related Issue on #335.

[andreped]

I think I know why this does not change in certain cases. I can see if I can find a fix.

[andreped]

I wonder if it is because the variable you are updating is private (see here), which may be problematic to overwrite in certain situations. Does my intuition make sense, @jamescalam? I could try changing this in a branch and opening a PR if I see that I can fix this.

[jamescalam]

I did miss this but thanks for the guidance @andreped — leaving my response below

[jamescalam]

As far as I'm aware, this issue was resolved some time ago. I have an example here I put together just yesterday where I was optimizing thresholds (towards the end of the notebook).

It could be an input data issue, I would make sure to have plenty of (utterance, route) pairs in the data, covering every route and also including plenty of (utterance, None) pairs for examples of what utterances should not trigger any route. Here's an example dataset.

If anyone is still seeing issues please let us know by raising an issue