REV/MAINT: Switching back to Yahoo (temp) to deal with Google #1972

freddiev4 · 2017-10-03T19:04:54Z

Google limited their API to only 251 days worth of data, most likely as a result of the change that Yahoo had made a few months ago. pandas-datareader made a fix for the Yahoo data reader so I'm making this PR to move back to Yahoo until we can do something better. Ideally we'll have a more robust solution before we do another release, so users will have to install latest Zipline master if this is merged.

Hoping to use this as a temp fix for #1965 and all the other issues that reference it

Ran:
zipline run -f buyapple.py -s 2005-1-1 -e 2017-2-1

Output:

[2017-10-21 17:09:23.349648] INFO: Performance: after split: asset: Equity(2696 [AAPL]), amount: 74.0, cost_basis: 38.14, last_sale_price: 88.99
[2017-10-21 17:09:23.350107] INFO: Performance: returning cash: 0.0
[2017-10-21 17:09:31.018709] INFO: Performance: after split: asset: Equity(2696 [AAPL]), amount: 16870.0, cost_basis: 36.52, last_sale_price: 645.57
[2017-10-21 17:09:31.018856] INFO: Performance: returning cash: 0.0
[2017-10-21 17:09:32.856054] INFO: Performance: Simulated 3042 trading days out of 3042.
[2017-10-21 17:09:32.856223] INFO: Performance: first open: 2005-01-03 14:31:00+00:00
[2017-10-21 17:09:32.856361] INFO: Performance: last close: 2017-02-01 21:00:00+00:00

Also attempting to drop Py3.4 packages here so that we don't have to build our own pandas-datareader and requests-ftp packages on conda (also helps make room for Travis jobs when we begin to build Py3.6 packages) #

freddiev4 · 2017-10-03T23:38:17Z

zipline/data/loader.py

@@ -356,93 +352,6 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
    return None


-def _load_raw_yahoo_data(indexes=None, stocks=None, start=None, end=None):


Should've been removed in #1812

coveralls · 2017-10-04T00:07:26Z

Coverage increased (+0.2%) to 87.405% when pulling b784e24 on back-to-yahoo-we-go into 5421ec0 on master.

freddiev4 · 2017-10-04T01:39:18Z

JoaoAparicio · 2017-10-04T12:36:28Z

Thank you. This is causing issues to a lot of people.

freddiev4 · 2017-10-05T17:07:35Z

@richafrank any thoughts on how to deal with this?

Passing Py27 and Py35 but not Py34. No pandas-datareader conda packages for Py34

Google limited their API to only 251 days worth of data, most likely as a result of the change that Yahoo had made a few months ago. `pandas-datareader` made a fix for the Yahoo data reader so I'm making this PR to move back to Yahoo until we can do something better. Ideally we'll have a more robust solution before we do another release, so users will have to install latest Zipline master if this is merged.

richafrank · 2017-10-05T20:57:43Z

Previously we've added a recipe for the missing package with conda skeleton pypi to our recipes dir, and build it ourselves.

abhilashchowdhary · 2017-10-11T18:05:15Z

Any updates on this PR? It would be great if it's merged in.

freddiev4 · 2017-10-11T19:24:00Z

@abhilashchowdhary Currently working on it 🙂

coveralls · 2017-10-13T17:57:53Z

Coverage increased (+0.2%) to 87.392% when pulling f18b9c9 on back-to-yahoo-we-go into 45257eb on master.

freddiev4 · 2017-10-21T17:01:46Z

@richafrank could you take a look at this? (forgot to ping you here)

richafrank · 2017-10-23T21:46:28Z

ACK. Started looking...

richafrank

Makes general sense to me. I had a couple questions, and I'm currently confirming our use of py3.4.

richafrank · 2017-10-24T12:53:58Z

tests/test_examples.py

@@ -55,7 +55,7 @@ def init_class_fixtures(cls):
            serialization='pickle',
        )

-        market_data = ('SPY_benchmark.csv', 'treasury_curves.csv')
+        market_data = ('^GSPC_benchmark.csv', 'treasury_curves.csv')


Shall we make a global name for the default benchmark symbol, so we don't have to change it in 5 places?

richafrank · 2017-10-24T13:00:49Z

zipline/examples/__init__.py

@@ -69,6 +69,8 @@ def run_example(example_name, environ):
    mod = EXAMPLE_MODULES[example_name]

    register_calendar("YAHOO", get_calendar("NYSE"), force=True)
+    # for when we don't actually have a 'test' bundle


When do we have a 'test' bundle otherwise? Did this work before?

I don't think we ever actually had a test bundle, but I don't think this failed before when we switched to Google 🤔

richafrank · 2017-10-24T16:52:34Z

.travis.yml

@@ -3,7 +3,6 @@ sudo: false
 fast_finish: true
 python:
  - 2.7
-  - 3.4


Just to confirm, this was removed because of the missing conda packages for the newer pandas-datareader dependency?

I'm currently assessing the internal value of testing this python version...

@freddiev4 Can we keep this row in the matrix for running tests, but not for building conda packages?

richafrank · 2017-10-24T16:53:29Z

zipline/data/benchmarks.py

@@ -45,17 +42,11 @@ def get_benchmark_returns(symbol, first_date, last_date):
    """
    data = pd_reader.DataReader(
        symbol,
-        'google',
+        'yahoo',


So the newer version works ok with yahoo again?

Yes on my machine with a fresh zipline install it works.

richafrank · 2017-11-01T15:10:45Z

@freddiev4 Will this PR include fixing the NotImplementedError in rebuild_example_data?

freddiev4 · 2017-11-03T05:14:26Z

@richafrank I don't think it's currently worth adding back Yahoo bundles, which we would need for the NotImplementedError so I went ahead and removed it.

Maybe at some other point in time.

coveralls · 2017-11-03T05:57:30Z

Coverage increased (+0.2%) to 87.427% when pulling b9a76dd on back-to-yahoo-we-go into 45257eb on master.

coveralls · 2017-11-03T05:58:14Z

Coverage increased (+0.2%) to 87.427% when pulling b9a76dd on back-to-yahoo-we-go into 45257eb on master.

freddiev4 · 2017-12-02T19:42:08Z

Closing in favor of #2031

freddiev4 commented Oct 3, 2017

View reviewed changes

DEV: Newer pandas-datareader

906fa73

freddiev4 force-pushed the back-to-yahoo-we-go branch from b784e24 to d5487ab Compare October 13, 2017 16:13

BLD: Don't build Py34 packages anymore

f18b9c9

freddiev4 force-pushed the back-to-yahoo-we-go branch from d5487ab to f18b9c9 Compare October 13, 2017 16:46

freddiev4 requested a review from richafrank October 13, 2017 20:18

freddiev4 mentioned this pull request Oct 14, 2017

Remove deprecated Yahoo code #1805

Closed

richafrank reviewed Oct 24, 2017

View reviewed changes

Freddie Vargus added 5 commits October 27, 2017 10:11

BLD: Don't build conda 3.4 packages

1e7e78d

MAINT: Respond to comments

920be8f

Syntax error b/c of space???

c1bf9c2

STY: Flake8

3056ecc

^GSPC vs GSPC

6010cab

Freddie Vargus added 2 commits November 3, 2017 01:09

Remove conflict & supposed cycle

bfe44ca

Add global name for test examples

b9a76dd

freddiev4 mentioned this pull request Nov 3, 2017

Unable to run zipline example with fresh install #2002

Closed

BLD: Don't conda build if python 3.4

3e56055

freddiev4 mentioned this pull request Nov 30, 2017

ENH: Use IEX Trading data instead of pandas-datareader #2031

Merged

freddiev4 closed this Dec 2, 2017

freddiev4 deleted the back-to-yahoo-we-go branch July 12, 2018 17:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

REV/MAINT: Switching back to Yahoo (temp) to deal with Google #1972

REV/MAINT: Switching back to Yahoo (temp) to deal with Google #1972

freddiev4 commented Oct 3, 2017 •

edited

Loading

freddiev4 Oct 3, 2017

coveralls commented Oct 4, 2017 •

edited

Loading

freddiev4 commented Oct 4, 2017

JoaoAparicio commented Oct 4, 2017

freddiev4 commented Oct 5, 2017 •

edited

Loading

richafrank commented Oct 5, 2017

abhilashchowdhary commented Oct 11, 2017

freddiev4 commented Oct 11, 2017

coveralls commented Oct 13, 2017 •

edited

Loading

freddiev4 commented Oct 21, 2017 •

edited

Loading

richafrank commented Oct 23, 2017

richafrank left a comment

richafrank Oct 24, 2017

richafrank Oct 24, 2017

freddiev4 Oct 27, 2017

richafrank Oct 24, 2017

richafrank Oct 25, 2017

richafrank Oct 24, 2017

freddiev4 Oct 27, 2017

richafrank commented Nov 1, 2017

freddiev4 commented Nov 3, 2017 •

edited

Loading

coveralls commented Nov 3, 2017 •

edited

Loading

coveralls commented Nov 3, 2017 •

edited

Loading

freddiev4 commented Dec 2, 2017

		@@ -356,93 +352,6 @@ def _load_cached_data(filename, first_date, last_date, now, resource_name,
		return None


		def _load_raw_yahoo_data(indexes=None, stocks=None, start=None, end=None):

REV/MAINT: Switching back to Yahoo (temp) to deal with Google #1972

REV/MAINT: Switching back to Yahoo (temp) to deal with Google #1972

Conversation

freddiev4 commented Oct 3, 2017 • edited Loading

Choose a reason for hiding this comment

coveralls commented Oct 4, 2017 • edited Loading

freddiev4 commented Oct 4, 2017

JoaoAparicio commented Oct 4, 2017

freddiev4 commented Oct 5, 2017 • edited Loading

richafrank commented Oct 5, 2017

abhilashchowdhary commented Oct 11, 2017

freddiev4 commented Oct 11, 2017

coveralls commented Oct 13, 2017 • edited Loading

freddiev4 commented Oct 21, 2017 • edited Loading

richafrank commented Oct 23, 2017

richafrank left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richafrank commented Nov 1, 2017

freddiev4 commented Nov 3, 2017 • edited Loading

coveralls commented Nov 3, 2017 • edited Loading

coveralls commented Nov 3, 2017 • edited Loading

freddiev4 commented Dec 2, 2017

freddiev4 commented Oct 3, 2017 •

edited

Loading

coveralls commented Oct 4, 2017 •

edited

Loading

freddiev4 commented Oct 5, 2017 •

edited

Loading

coveralls commented Oct 13, 2017 •

edited

Loading

freddiev4 commented Oct 21, 2017 •

edited

Loading

freddiev4 commented Nov 3, 2017 •

edited

Loading

coveralls commented Nov 3, 2017 •

edited

Loading

coveralls commented Nov 3, 2017 •

edited

Loading