HTTP Error 401: Authorization Required -- Update authorization & bot-detection-evasion logic

[dmduck]

Hello,

I am getting HTTP Error 401: Authorization Required every time I try to use the flurry data application. I have verified numerous times that my credentials are correct. Below is a snippet from the splunkd.log file. Has anyone else run into this, and is there a solution?

Thanks!

04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login()
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login
04-13-2015 10:57:39.082 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do')
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout)
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response
04-13-2015 10:57:39.083 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 236, in
04-13-2015 10:57:49.191 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" conn.login()
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/extract.py", line 65, in login
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" self.browser.open('https://dev.flurry.com/secure/login.do')
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" return self._mech_open(url, data, timeout=timeout)
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" File "/Applications/Splunk/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" raise response
04-13-2015 10:57:49.192 -0500 ERROR ExecProcessor - message from "python /Applications/Splunk/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 401: Authorization Required

[davidfstr]

I am seeing a different error myself in splunkd.log:

04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor -     interval: 600000 ms
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor - New scheduled exec process: /Applications/splunk_621_flurry/bin/splunkd instrument-resource-usage
04-15-2015 11:21:39.296 -0700 INFO  ExecProcessor -     interval: 0 ms
04-15-2015 11:25:55.092 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" Traceback (most recent call last):
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 241, in <module>
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     flurry_csv_stream = conn.download_log(year, month, day, offset)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py", line 102, in download_log
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     resp = self.browser.open(url)
04-15-2015 11:25:55.093 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 203, in open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     return self._mech_open(url, data, timeout=timeout)
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"   File "/Applications/splunk_621_flurry/etc/apps/flurry/bin/mechanize/_mechanize.py", line 255, in _mech_open
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py"     raise response
04-15-2015 11:25:55.094 -0700 ERROR ExecProcessor - message from "python /Applications/splunk_621_flurry/etc/apps/flurry/bin/extract.py" mechanize._response.httperror_seek_wrapper: HTTP Error 404: Not Found

Regardless, both my error trace and your error trace indicate that either Flurry has changed their bot detection system or rate limiting system. The actual (private) API being accessed appears to still work, namely: https://dev.flurry.com/eventsLogCsv.do

I no longer have enough cycles to continue updating this app to adjust to Flurry's API changes. So I will mark this item as Community Support.

[hmasanari]

Hi,
How about setting "Next Session ID [Advanced]" as well as your credentials?
I got the same problem for many times without entering it, but the app seems to start again after I entered and saved bigger Session ID. I found the biggest Session ID on the time as below;
index="flurry" | stats max(Session)

And I entered a number next to the search result.
Thanks.