ENH: Use IEX Trading data instead of pandas-datareader

[freddiev4]

Got fed up with the breaking/removal of Google & Yahoo Finance APIs, and so since we've been using pandas-datareader to get data from those APIs I removed that dependency for when we want to get data for a specific benchmark (in this case SPY).

I changed our data source to IEX which will give us up to 5 years worth of data. It has really great documentation and seems like its a lot more reliable.

Should fix:

• get_banchmark_returns can't handle google finance http response format #2029
• Issues with running algorithms through latest close #2024
• Benchmark downloading is broken #1965
• Unable to run zipline example with fresh install #2002
• Example script on Windows 7 gives Unicode Error #2036
• cannot read results.pickle #2030
• when running the tutorial code, I get the error "KeyError: 'Close'" #2048

And should be better than PR #1972

[ssanderson]

👍 on moving to the IEX api. Do we need to add an attribution somewhere? From https://iextrading.com/api-terms/:

1. If you sell, lease, furnish, permit or provide access to, or otherwise redistribute Investors Exchange (“IEX”) API data, which includes API data licensed from third-parties, to any other person:
   a. you must provide a link to https://iextrading.com/api-exhibit-a or otherwise obtain consent from such person to the terms of use substantially in the form of the above Exhibit A;
   b. you must cite IEX as the source in accordance with the instructions available on https://iextrading.com/developer;

[ssanderson]

MIght be worth opening an issue/asking a question on https://github.com/iexg/IEX-API.

[freddiev4]

@ssanderson Thanks for catching that. I read that attribution comment at some point and just kinda forgot. Will open an issue there.

[coveralls]

Coverage increased (+0.006%) to 87.378% when pulling 181a0f7 on use-iex-for-benchmark-data into 49a3a4a on master.

[freddiev4]

Opened up an issue: https://github.com/iexg/IEX-API/issues/124

[coveralls]

Coverage increased (+0.006%) to 87.378% when pulling 8b87424 on use-iex-for-benchmark-data into 42b3200 on master.

[coveralls]

Coverage increased (+0.006%) to 87.378% when pulling 8b87424 on use-iex-for-benchmark-data into 42b3200 on master.

[coveralls]

Coverage increased (+0.006%) to 87.378% when pulling c2a735b on use-iex-for-benchmark-data into 42b3200 on master.

[ChrisPappalardo]

This looks like a good change for zipline master, but FYI it still does not fix #2024. I'm getting the same KeyError using this branch and trying to run the do-nothing test algo I described in that issue through latest available close (12/1):

KeyError: 'the label [2017-11-30 00:00:00+00:00] is not in the [index]'

The loader seems to pick the wrong benchmark ending date of 11/29:

root@1400ce2d197f:/projects# zipline run -b quantopian-quandl -f test.py --start=2017-1-1 --end=2017-12-31
[2017-12-02 19:10:15.709130] INFO: Loader: Cache at /root/.zipline/data/SPY_benchmark.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-11-29 00:00:00+00:00.
[2017-12-02 19:10:15.709493] INFO: Loader: Downloading benchmark data for 'SPY' from 1989-12-29 00:00:00+00:00 to 2017-11-29 00:00:00+00:00
[2017-12-02 19:10:16.352607] WARNING: Loader: Still don't have expected benchmark data for 'SPY' from 1989-12-29 00:00:00+00:00 to 2017-11-29 00:00:00+00:00 after redownload!
[2017-12-02 19:10:16.353117] INFO: Loader: Cache at /root/.zipline/data/treasury_curves.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-11-29 00:00:00+00:00.
[2017-12-02 19:10:16.353323] INFO: Loader: Downloading treasury data for 'SPY' from 1990-01-02 00:00:00+00:00 to 2017-11-29 00:00:00+00:00

When data through 12/1 is available:

root@1400ce2d197f:/projects# curl -s https://api.iextrading.com/1.0/stock/SPY/chart/5y | python -c "import sys, json; print(json.load(sys.stdin)[-1])"
{'changePercent': -0.208, 'change': -0.55, 'date': '2017-12-01', 'low': 260.76, 'changeOverTime': 0.8696359137504418, 'label': 'Dec 1', 'vwap': 263.4229, 'unadjustedVolume': 164390902, 'close': 264.46, 'high': 265.31, 'volume': 164390902, 'open': 264.76}

[ChrisPappalardo]

The issue is this line:

zipline/zipline/data/loader.py

Line 156 in 5421ec0

last_date = trading_days[trading_days.get_loc(now, method='ffill') - 2]

It appears to be in place because the treasury data from the Fed H15 report is a day lagged from the Quandl and IEX stock market data.

In my opinion, treasury curve data should be filled through the latest available market data close rather than hold up the trading algorithm a full day behind the market.

Alternatively, it appears users can implement their own load_market_data callable and pass via the load arg in when instantiating the TradingEnvironment object.

[eykiriku]

i am calling it via zipline.data.loader.load_market_data but it throws an error at line 222 of the loader.py where first_date=first_date - trading_day. Error trying to subtract timestamp to string

[shridude]

Currently version 1.1.1 is broken pending this fix getting released. Any chance we can get a conda install for 1.1.0 to keep things going in the meantime?

[jimgwilliam]

Just getting started on zipline. Anxious for this branch to get implemented so the benchmarking is fixed with the IEX historical data.

[ernestoeperez88]

LGTM