Can't download the report after using 'RetrieveExport' function

[toinou222]

Versions

OS: Linux 4.14.106-97.85.amzn2.x86_64
Python: 3.6.1 :: Anaconda 4.4.0 (64-bit)
krakenex: 2.1.0

What are you trying to achieve?

I'm trying to create and open a report using 'AddExport' and 'RetrieveExport' functions.
Ideally, I would like to be able to retrieve data from the report within my python script to do some custom reporting but downloading it would be acceptable as well.

#!/usr/bin/env python3

###################################
##       Library imports         ##
###################################

import krakenex

# Define public and secret API keys to connect to Kraken client
KEY = 'api_key'
SECRET = 'secret_key'
k = krakenex.API(key=KEY, secret=SECRET)
  
###################################
##           Report              ##
###################################

# Create report on ledgers entries 
create_report = k.query_private('AddExport', 
                    {'description': 'reporting',
                     'report': 'ledgers',
                     'format': 'CSV'
                    })

# Wait for the report to be processed as mentionned in kraken docs

# Retrieve report ID
report = k.query_private('ExportStatus', 
                    {'report': 'ledgers'
                    })

report_id = report['result'][0]['id']

# Trying to retrieve the report
export = k.query_private('RetrieveExport', {'id': report_id})

What do you expect to happen?

Etiher download the report or retrieve the data that it contains.

What happens instead?

Got the following error...
In kraken API docs it says:
"Result: binary zip archive containing the report

Note: If content-type isn't application/json, text, or html, the binary report will be streamed back. If it is one of those types you can expect an error in the response."

Not sure to understand what it means... but as this api calls returns something else (a binary zip file) than usual, maybe the 'query_private' function isn't appropriate?

---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-16-ffda4fa16aaa> in <module>()
----> 1 export = k.query_private('RetrieveExport', {'id': report_id})
      2 #report

/home/ec2-user/anaconda3/lib/python3.6/site-packages/krakenex/api.py in query_private(self, method, data, timeout)
    191         }
    192 
--> 193         return self._query(urlpath, data, headers, timeout = timeout)
    194 
    195     def _nonce(self):

/home/ec2-user/anaconda3/lib/python3.6/site-packages/krakenex/api.py in _query(self, urlpath, data, headers, timeout)
    138             self.response.raise_for_status()
    139 
--> 140         return self.response.json(**self._json_options)
    141 
    142 

/home/ec2-user/anaconda3/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
    895                     # used.
    896                     pass
--> 897         return complexjson.loads(self.text, **kwargs)
    898 
    899     @property

/home/ec2-user/anaconda3/lib/python3.6/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    352             parse_int is None and parse_float is None and
    353             parse_constant is None and object_pairs_hook is None and not kw):
--> 354         return _default_decoder.decode(s)
    355     if cls is None:
    356         cls = JSONDecoder

/home/ec2-user/anaconda3/lib/python3.6/json/decoder.py in decode(self, s, _w)
    337 
    338         """
--> 339         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    340         end = _w(s, end).end()
    341         if end != len(s):

/home/ec2-user/anaconda3/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
    355             obj, end = self.scan_once(s, idx)
    356         except StopIteration as err:
--> 357             raise JSONDecodeError("Expecting value", s, err.value) from None
    358         return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Thanks for your help and for this amazing work!!

[veox]

Note: If content-type isn't application/json, text, or html, the binary report will be streamed back. If it is one of those types you can expect an error in the response."

Not sure to understand what it means... but as this api calls returns something else (a binary zip file) than usual, maybe the 'query_private' function isn't appropriate?

krakenex so far has expected all responses to be JSON - I've never used RetrieveExport myself, so don't know if it's always been there, or if the method has been added recently.

The easiest hacky "fix" would be to check self.response.headers['content-type'] and not do JSON decoding if it's binary, as Kraken's docs try to weasel in, here:

python3-krakenex/krakenex/api.py

Line 140 in d88fc51

return self.response.json(**self._json_options)

I am generally against this approach, as this would mean that the same function returns different-type values depending on the arguments passed to it, while none of the arguments pertain to content-type or de-serialisation.

---

Another hacky workaround would be to add an argument specifically stating not to expect JSON... Which runs into the complication of "what if the remote responds with JSON, e.g. due to error".

---

Finally, to maintain a uniform interface, krakenex itself could "JSONise" the binary response into

{
  'error': [],
  'result': <SOME-HACKY-WAY-TO-PRESENT-BINARY-RESPONSE>
}

[veox]

Meanwhile, perhaps you could try the following workaround, specifically so you can move on:

# Trying to retrieve the report: response is binary if successful
from json import JSONDecodeError
try:
    maybe_error = k.query_private('RetrieveExport', {'id': report_id})
except JSONDecodeError:
    export_bytes = k.response.content
else:
    response_error = maybe_error

From requests.Response.content docs:

Content of the response, in bytes.

[toinou222]

Hi,

Got this piece of code from Kraken support that is suggesting to modify the "_query" function.

def _query(self, urlpath, data, headers=None, timeout=None): 

        if data is None:
            data = {}
        if headers is None:
            headers = {}

        url = self.uri + urlpath

        self.response = self.session.post(url, data = data, headers = headers,
                                          timeout = timeout)

        if self.response.status_code not in (200, 201, 202):
            self.response.raise_for_status()
        
        #edit: below
        #if method is RetrieveExport and result is not json (error), write binary zip to .zip file

        if 'RetrieveExport' not in urlpath:                   
         return self.response.json(**self._json_options)
        else:
            try:
                res = self.response.json(**self._json_options)
            except(Exception) as f:
                name = 'export'+str(self._nonce())+'.zip'
                z = open(name, 'wb')
                z.write(self.response.content)
                res = 'export, '+ name + ' successfully downloaded'
            return res

Didn't manage to test it already as the 'AddExport' function throw me the following error
{'error': ['EDatabase:Internal error']}

I will update you when fixed but it seems that the suggested fix would be worth adding to the master.

[veox]

The solution is wholly unacceptable to me. It writes to file on-disk as a side-effect, returns a plain string instead of a JSON-derived dict, and singles out RetrieveExport.

If that works for you - wonderful, but please don't submit a PR with it. :) I'll think of something else when I get the time.

[Exodus4400]

I would like to bring this issue back up. I have run into the same problem as the original submission.

[veox]

@Exodus4400 Have you tried the workaround suggested in #114 (comment)?

[veox]

I've added an example based on OP's description in examples/retrieve-csv-report.py.

Sorry all for being so stubborn about this, but as far as I see krakenex, it's "mature" and I'd like to touch it as little as possible. Refer to PR #132 for more verbose reasoning.

(Something to keep in mind: some workarounds for this issue are blockers for issue #103.)