Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor Bug: when specifying --config , response is invalid #56

Closed
comorbidity opened this issue Nov 21, 2024 · 1 comment · Fixed by #59
Closed

Minor Bug: when specifying --config , response is invalid #56

comorbidity opened this issue Nov 21, 2024 · 1 comment · Fixed by #59

Comments

@comorbidity
Copy link
Contributor

Expected

  • chart review performs the same way with --config as if the file was in the PWD as "labelstudio-export.json"

Actual:
'list' object has no attribute 'get'

Failure Examples
cd /opt/labelstudio/covid/kappa-bch/

$chart-review --config labelstudio-dec14-626-full.json accuracy andy alon
$chart-review --config /opt/labelstudio/covid/kappa-bch/labelstudio-dec14-626-full.json accuracy andy alon

Working example
ln -s labelstudio-dec14-626-full.json labelstudio-export.json
$chart-review accuracy andy alon

F1     Sens  Spec   PPV    NPV  Kappa  TP  FN  TN   FP  Label                   
0.984  1.0   0.992  0.968  1.0  0.98   61  0   234  2   *                       
0.909  1.0   0.955  0.833  1.0  0.886  5   0   21   1   Congestion or runny nose
1.0    1.0   1.0    1.0    1.0  1.0    7   0   20   0   Cough                   
1.0    1.0   1.0    1.0    1.0  1.0    4   0   23   0   Diarrhea                
1.0    1.0   1.0    1.0    1.0  1.0    5   0   22   0   Dyspnea                 
0.923  1.0   0.952  0.857  1.0  0.899  6   0   20   1   Fatigue                 
1.0    1.0   1.0    1.0    1.0  1.0    11  0   16   0   Fever or chills         
1.0    1.0   1.0    1.0    1.0  1.0    7   0   20   0   Headache                
1.0    1.0   1.0    1.0    1.0  1.0    1   0   26   0   Loss of taste or smell  
1.0    1.0   1.0    1.0    1.0  1.0    3   0   24   0   Muscle or body aches    
1.0    1.0   1.0    1.0    1.0  1.0    8   0   19   0   Nausea or vomiting      
1.0    1.0   1.0    1.0    1.0  1.0    4   0   23   0   Sore throat             
@mikix
Copy link
Contributor

mikix commented Nov 29, 2024

Ah, that error message is not at all helpful, I agree - I've got a pass at a better one in #59. But ultimately, the issue there is that you are passing in the label studio export file as a config, rather than a custom config file.

There's not currently a CLI arg for a custom export file. I believe the thinking was that all such "custom" properties would flow from the config file. So you'd have a custom config pointing at your custom export. But if that's too awkward for everyday use, we can talk about other options.

@mikix mikix closed this as completed in #59 Dec 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants