Worker Quality is NaN

[PeccatorImpius]

For a number of our jobs, I am getting worker quality as NaN for all workers. Please see comments below for steps to reproduce.

[PeccatorImpius]

Step 1:

$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs" -d '{"categories":["blank","confirm_closed","confirm_no","confirm_nonrestaurant","confirm_yes"],"costMatrix":[{"from":"blank","to":"blank","value":5.0},{"from":"blank","to":"confirm_closed","value":5.0},{"from":"blank","to":"confirm_no","value":5.0},{"from":"blank","to":"confirm_nonrestaurant","value":5.0},{"from":"blank","to":"confirm_yes","value":5.0},{"from":"confirm_closed","to":"blank","value":5.0},{"from":"confirm_closed","to":"confirm_closed","value":0.0},{"from":"confirm_closed","to":"confirm_no","value":5.0},{"from":"confirm_closed","to":"confirm_nonrestaurant","value":5.0},{"from":"confirm_closed","to":"confirm_yes","value":5.0},{"from":"confirm_no","to":"blank","value":1.0},{"from":"confirm_no","to":"confirm_closed","value":1.0},{"from":"confirm_no","to":"confirm_no","value":0.0},{"from":"confirm_no","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_no","to":"confirm_yes","value":1.0},{"from":"confirm_nonrestaurant","to":"blank","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_closed","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_no","value":5.0},{"from":"confirm_nonrestaurant","to":"confirm_nonrestaurant","value":0.0},{"from":"confirm_nonrestaurant","to":"confirm_yes","value":5.0},{"from":"confirm_yes","to":"blank","value":1.0},{"from":"confirm_yes","to":"confirm_closed","value":1.0},{"from":"confirm_yes","to":"confirm_no","value":1.0},{"from":"confirm_yes","to":"confirm_nonrestaurant","value":1.0},{"from":"confirm_yes","to":"confirm_yes","value":0.0}],"algorithm":"IDS","iterations":10,"epsilon":0.0001,"scheduler":"NormalScheduler","calculator":"CostBased"}'
{"timestamp":"2013-05-11T16:02:06.659-07:00","result":"New job created with ID: RANDOM__1368313326658_39","status":"OK"}

[PeccatorImpius]

Step 2:

$ curl -X POST -H "Content-Type: application/json" "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/assigns" -d '{"assigns":[{"label":"confirm_yes","worker":"A1R7CJMWXC79UO","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2OVZKPFE4EGD044XHZZIHNZWGWV7HC"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2Z3KH1Q6SVQ80ZQH28XLDOBVMXHL2G"},{"label":"confirm_yes","worker":"A1FQYUBCBNTQX7","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2V6ZFYQS1CST5FXS3RJ4QC1E8S3KN3"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"2MCFJ51Y7QEOI6GL4F3S1MOF76TBEL"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S4JTUHYW2GT8095J6WWU169874PK4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2S3YHVI44OS23CC3P15Z0BAPFQI4YO"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2KXDEY5COW0LZIHYKQIFTC6VX9G4VF"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_no","worker":"A3TXO6RKFIDFUV","object":"2FT49F9SSSHXKS1JZ6K5P5F8TJ4TX2"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"27G2RCJK63ZVXUZMCYLIIQ9JVK87WC"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2JODG67D1X3VJ62VO2B2RE1MH40HAC"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_nonrestaurant","worker":"AH5ZIMHL7TNWC","object":"26ZIT3HVWAVK1XKIV4T1F2Z9J5XEMD"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"20N1Y7QEOZFY9JJ747DONXRD2JYEH4"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"241KM05BMJTU0PEXS7G9ZA7SFB2EJ9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_closed","worker":"A19BL9GZGXFWFT","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2YDWAVKI62NJ9TJ2ED09YH6BU0SRJZ"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2JV21O3W5XH02G7NUGBYMPLSZMLHB7"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2JS1QP6AUC26W7O2PFO330FKAR5811"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2CIFK0COK2JEKDPKWY0LZW6O9PSRKY"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2IJK274J79KQFXJ3ZIXU5FCDEKDHFP"},{"label":"confirm_closed","worker":"A3TXO6RKFIDFUV","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A3PRF4IIM2IQN2","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_closed","worker":"A31X3JCHS0BPFJ","object":"234EGOOGQSCMP9S5E65I2UU0A36EEO"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"23QJIA0RYNJ9LE1FYEWBUJTURDD27P"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2S5VP3TVOK5UYANWALHEWZFY07I52D"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2ETAT2D5NQYNO6LLCX751YMNMBF84E"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"2Z2MN9U8P9Y3RKER9WUS2YIMAY9EVU"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2GXYQS1CSTMOYO984I9C9EYDXCZPM9"},{"label":"blank","worker":"AH5ZIMHL7TNWC","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2BLK4F0OHOVRKV0SW2TLH2HEMBR23K"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A2KET1HL1COET5","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2X6OGQSCM6IATTA9U8LU80OQUZQHHN"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"29UWY2AOO2GM55YJ0UHSKM6IKPI442"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DJVP9746OQ1BE8LJ4X7851QUR3L1Z"},{"label":"confirm_yes","worker":"A31X3JCHS0BPFJ","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"A3PRF4IIM2IQN2","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"25XXRDS4IC1EH45SVTD19A2I5DKWZS"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2DABRI8IUB2YD0QET6KLJ3L0PQZH56"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"AH5ZIMHL7TNWC","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A2L2ZXRHK3SNOP","object":"2TOSK9RJ85OZ1QWCYO1PUOOQFTCU60"},{"label":"confirm_yes","worker":"A3PRF4IIM2IQN2","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"ABT7QTMIYXYO0","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_no","worker":"AH5ZIMHL7TNWC","object":"2IBHV8ER9Y8T6B0HB6D5SFHSC334RM"},{"label":"confirm_yes","worker":"A3TXO6RKFIDFUV","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A19BL9GZGXFWFT","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A1I3CXC17NIRWB","object":"294EZZ2MIKMNSLQKLCU81WWXSI97O0"},{"label":"confirm_yes","worker":"A2VRQML8Q3XYP4","object":"2AOYTWX4H3H282M8LN7IEIJRLKM4ZF"}]}'
{"timestamp":"2013-05-11T16:03:34.374-07:00","status":"OK","redirect":"responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns"}

[PeccatorImpius]

Step 3:

$ curl -X GET "http://localhost/troia-server-1.1/responses/3183/POST/jobs/RANDOM__1368313326658_39/assigns"
{"timestamp":"2013-05-11T16:04:22.189-07:00","result":"Assigns added","executionTime":0.002,"status":"OK"}

[PeccatorImpius]

Step 4:

$ curl -X GET "http://localhost/troia-server-1.1/jobs/RANDOM__1368313326658_39/workers/quality/estimated"
{"timestamp":"2013-05-11T16:05:21.667-07:00","status":"OK","redirect":"responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated"}

[PeccatorImpius]

Step 5:

$ curl -X GET "http://localhost/troia-server-1.1/responses/3204/GET/jobs/RANDOM__1368313326658_39/workers/quality/estimated"
{"timestamp":"2013-05-11T16:06:01.420-07:00","result":[{"workerName":"A2KET1HL1COET5","value":NaN},{"workerName":"A31X3JCHS0BPFJ","value":NaN},{"workerName":"A3PRF4IIM2IQN2","value":NaN},{"workerName":"A1FQYUBCBNTQX7","value":NaN},{"workerName":"ABT7QTMIYXYO0","value":NaN},{"workerName":"A2L2ZXRHK3SNOP","value":NaN},{"workerName":"A2VRQML8Q3XYP4","value":NaN},{"workerName":"A1R7CJMWXC79UO","value":NaN},{"workerName":"A19BL9GZGXFWFT","value":NaN},{"workerName":"AH5ZIMHL7TNWC","value":NaN},{"workerName":"A1I3CXC17NIRWB","value":NaN},{"workerName":"A3TXO6RKFIDFUV","value":NaN}],"executionTime":0.0,"status":"OK"}

[kkonrad]

Thanks for reporting this issue! Dana have turned it into test. We will let you know when we manage to fix this.

[kkonrad]

Quick update:

• it works for BDS
• if you provide priors at job creation it works for IDS (priors = probability distribution across labels, for example: 1/5 for each class)

It looks like there is some problem with dynamic prior calculations done by IDS algorithm. We are still working on this.

[PeccatorImpius]

Thanks for the update. The workaround is very helpful. I will try to use priors while I am waiting for the fix.

[PeccatorImpius]

I have implemented the workaround with priors, but I am seeing some strange numbers:

$ curl -X GET "http://localhost/troia-server-1.1/responses/1148/GET/jobs/RANDOM__1368726379654_33/workers/quality/estimated"
{"timestamp":"2013-05-16T10:53:46.686-07:00","result":[{"workerName":"A1W0N2QACQZEQP","value":-3.1727002269058184},{"workerName":"A224TK7J4KA1LV","value":-3.3285464160473195},{"workerName":"A1GU6OYGOV6PBK","value":0.12620888888888937},{"workerName":"AHHKIENAC9M3M","value":0.6288671324385611},{"workerName":"A239WM193Q1OIY","value":-3.3466950270926326}],"executionTime":0.0,"status":"OK"}

Is it normal for worker quality numbers to be negative?

[kkonrad]

It works that way: quality is computed as 1 - (worker_cost / spammer_cost). workers cost is expected costs that he will generate making an assign. Our baseline is spammer_cost which is using priors and costs to select label which minimizes cost. So in this case spammer is generating lower costs than workers ...
So it looks like quality can be negative. Nevertheless, I will ask someone smarter than me about this ;)
BTW As you can see changing priors will result in different worker qualities. More accurate priors should result in better worker quality estimation

[kkonrad]

It is confirmed that in some cases worker quality can be negative.
It looks like that this is not enough data for IDS to coverage and it is recommended for such small jobs use BDS algorithm. IDS works well with high volume of assigns. We won't tackle this problem any time soon.