Skip to content

Commit 89db062

Browse files
RomazesRomazes
authored
Migrate CoinApi Integration to Standalone Project (#1)
* feat: remove template files * feat: prepare solution's structure feat: github issue & PR templates feat: prepare GH workflow file * feat: paste coinApi files from ToolBox * feat: paste coinApi.Converter files from ToolBox * feat: paste coinApi.test files from ToolBox feat: TestSetup initializer * feat: divide HistoryProvider in file * remove: does not support exchanges in SymbolMapper * refactor: downgrade pckg Microsoft.NET.Test.Sdk * feat: handle JsonSerializationException feat: simplify code feat: remove one warning msg * feat: DQH tests with different param feat: GetBrokerage CryptoFuture Symbol Test feat: init with wrong api key test feat: helper class for tests * feat: forceTypeNameOnExisting set up to false for GetExportedValueByTypeName * feat: add sync bash script in Converter project * remove: unsupported markets from Converter * feat: ValidateSubscription * fix: productId in ValidateSubscription * refactor: make static of TestHelper class fix: deprecated GDAX to Coinbase Market in Symbol test * feat: CoinAPIDataDownloader fea: test of CoinAPIDataDownloader fix: reset config in wrong api key test * fix: handle exception when parsing response in History * feat: update Readme * Create LICENSE * fix: reset config in test where we change config * refactor: create RestClient at once time * refactor: create RestRequest only once * rename: Converter to DataProcessing * refactor: test OneTimeSetUp to testing class * refactor: increase delay in DQH tests * fix: change delay and init DQH class tests * refactor: GlobalSetup make static * refactor: ProcessFeed in DQH tests * feat: add some explicit and log trace * refactor: validation on null tick remove: thread sleep * feat: add delay in ProcessFeed by cancellationToken refactor: future test * fix: tick symbol in CryptoFuture test * remove: Explicit attribute in tests
1 parent ad7e7f9 commit 89db062

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

56 files changed

+3236
-1373
lines changed

‎.github/ISSUE_TEMPLATE.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
<!--- Provide a general summary of the issue in the Title above -->
2+
3+
## Expected Behavior
4+
<!--- If you're describing a bug, tell us what should happen -->
5+
<!--- If you're suggesting a change/improvement, tell us how it should work -->
6+
7+
## Current Behavior
8+
<!--- If describing a bug, tell us what happens instead of the expected behavior -->
9+
<!--- If suggesting a change/improvement, explain the difference from current behavior -->
10+
11+
## Possible Solution
12+
<!--- Not obligatory, but suggest a fix/reason for the bug, -->
13+
<!--- or ideas how to implement the addition or change -->
14+
15+
## Steps to Reproduce (for bugs)
16+
<!--- Provide a link to a live example, or an unambiguous set of steps to -->
17+
<!--- reproduce this bug. Include code to reproduce, if relevant -->
18+
1.
19+
2.
20+
3.
21+
4.
22+
23+
## Context
24+
<!--- How has this issue affected you? What are you trying to accomplish? -->
25+
<!--- Providing context helps us come up with a solution that is most useful in the real world -->
26+
27+
## Your Environment
28+
<!--- Include as many relevant details about the environment you experienced the bug in -->
29+
* Version used:
30+
* Environment name and version (e.g. PHP 5.4 on nginx 1.9.1):
31+
* Server type and version:
32+
* Operating System and version:
33+
* Link to your project:

‎.github/PULL_REQUEST_TEMPLATE.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<!--- Provide a general summary of your changes in the Title above -->
2+
3+
## Description
4+
<!--- Describe your changes in detail -->
5+
6+
## Motivation and Context
7+
<!--- Why is this change required? What problem does it solve? -->
8+
<!--- If it fixes an open issue, please link to the issue here. -->
9+
10+
## How Has This Been Tested?
11+
<!--- Please describe in detail how you tested your changes. -->
12+
<!--- Include details of your testing environment, and the tests you ran to -->
13+
<!--- see how your change affects other areas of the code, etc. -->
14+
15+
## Screenshots (if appropriate):
16+
17+
## Types of changes
18+
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
19+
- [ ] Bug fix (non-breaking change which fixes an issue)
20+
- [ ] New feature (non-breaking change which adds functionality)
21+
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
22+
23+
## Checklist:
24+
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
25+
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
26+
- [ ] My code follows the code style of this project.
27+
- [ ] My change requires a change to the documentation.
28+
- [ ] I have updated the documentation accordingly.
29+
- [ ] I have read the **CONTRIBUTING** document.
30+
- [ ] I have added tests to cover my changes.
31+
- [ ] All new and existing tests passed.

‎.github/workflows/build.yml‎

Lines changed: 22 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,18 @@ on:
99
jobs:
1010
build:
1111
runs-on: ubuntu-20.04
12-
12+
env:
13+
QC_JOB_USER_ID: ${{ secrets.QC_JOB_USER_ID }}
14+
QC_API_ACCESS_TOKEN: ${{ secrets.QC_API_ACCESS_TOKEN }}
15+
QC_JOB_ORGANIZATION_ID: ${{ secrets.QC_JOB_ORGANIZATION_ID }}
16+
QC_COINAPI_API_KEY: ${{ secrets.QC_COINAPI_API_KEY }}
1317
steps:
14-
- uses: actions/checkout@v2
18+
- name: Checkout
19+
uses: actions/checkout@v2
1520

1621
- name: Free space
1722
run: df -h && rm -rf /opt/hostedtoolcache* && df -h
1823

19-
- name: Pull Foundation Image
20-
uses: addnab/docker-run-action@v3
21-
with:
22-
image: quantconnect/lean:foundation
23-
2424
- name: Checkout Lean Same Branch
2525
id: lean-same-branch
2626
uses: actions/checkout@v2
@@ -40,11 +40,20 @@ jobs:
4040
- name: Move Lean
4141
run: mv Lean ../Lean
4242

43-
- name: BuildDataSource
44-
run: dotnet build ./QuantConnect.DataSource.csproj /p:Configuration=Release /v:quiet /p:WarningLevel=1
43+
- name: Pull Foundation Image
44+
uses: addnab/docker-run-action@v3
45+
with:
46+
image: quantconnect/lean:foundation
47+
options: -v /home/runner/work:/__w --workdir /__w/Lean.DataSource.CoinAPI/Lean.DataSource.CoinAPI -e QC_JOB_USER_ID=${{ secrets.QC_JOB_USER_ID }} -e QC_API_ACCESS_TOKEN=${{ secrets.QC_API_ACCESS_TOKEN }} -e QC_JOB_ORGANIZATION_ID=${{ secrets.QC_JOB_ORGANIZATION_ID }} -e QC_COINAPI_API_KEY=${{ secrets.QC_COINAPI_API_KEY }}
48+
49+
- name: Build QuantConnect.CoinAPI
50+
run: dotnet build ./QuantConnect.CoinAPI/QuantConnect.CoinAPI.csproj /p:Configuration=Release /v:quiet /p:WarningLevel=1
51+
52+
- name: Build DataProcessing
53+
run: dotnet build ./DataProcessing/DataProcessing.csproj /p:Configuration=Release /v:quiet /p:WarningLevel=1
4554

46-
- name: BuildTests
47-
run: dotnet build ./tests/Tests.csproj /p:Configuration=Release /v:quiet /p:WarningLevel=1
55+
- name: Build QuantConnect.CoinAPI.Tests
56+
run: dotnet build ./QuantConnect.CoinAPI.Tests/QuantConnect.CoinAPI.Tests.csproj /p:Configuration=Release /v:quiet /p:WarningLevel=1
4857

49-
- name: Run Tests
50-
run: dotnet test ./tests/bin/Release/net6.0/Tests.dll
58+
- name: Run QuantConnect.CoinAPI.Tests
59+
run: dotnet test ./QuantConnect.CoinAPI.Tests/bin/Release/QuantConnect.CoinAPI.Tests.dll

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
*.hex
3434

3535
# QC Cloud Setup Bash Files
36+
!DataProcessing/*.sh
3637
*.sh
3738
# Include docker launch scripts for Mac/Linux
3839
!run_docker.sh

‎DataProcessing/CLRImports.py‎

Lines changed: 0 additions & 12 deletions
This file was deleted.

‎DataProcessing/CoinApiDataConverter.cs‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
/*
2+
* QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
3+
* Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
4+
*
5+
* Licensed under the Apache License, Version 2.0 (the "License");
6+
* you may not use this file except in compliance with the License.
7+
*
8+
* You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
using QuantConnect.Data;
18+
using QuantConnect.Util;
19+
using System.Diagnostics;
20+
using QuantConnect.Logging;
21+
using QuantConnect.ToolBox;
22+
using QuantConnect.CoinAPI;
23+
24+
namespace QuantConnect.DataProcessing
25+
{
26+
/// <summary>
27+
/// Console application for converting CoinApi raw data into Lean data format for high resolutions (tick, second and minute)
28+
/// </summary>
29+
public class CoinApiDataConverter
30+
{
31+
/// <summary>
32+
/// List of supported exchanges
33+
/// </summary>
34+
private static readonly HashSet<string> SupportedMarkets = new[]
35+
{
36+
Market.Coinbase,
37+
Market.Bitfinex,
38+
Market.Binance,
39+
Market.Kraken,
40+
Market.BinanceUS
41+
}.ToHashSet();
42+
43+
private readonly DirectoryInfo _rawDataFolder;
44+
private readonly DirectoryInfo _destinationFolder;
45+
private readonly SecurityType _securityType;
46+
private readonly DateTime _processingDate;
47+
private readonly string _market;
48+
49+
/// <summary>
50+
/// CoinAPI data converter.
51+
/// </summary>
52+
/// <param name="date">the processing date.</param>
53+
/// <param name="rawDataFolder">path to the raw data folder.</param>
54+
/// <param name="destinationFolder">destination of the newly generated files.</param>
55+
/// <param name="securityType">The security type to process</param>
56+
/// <param name="market">The market to process (optional). Defaults to processing all markets in parallel.</param>
57+
public CoinApiDataConverter(DateTime date, string rawDataFolder, string destinationFolder, string market = null, SecurityType securityType = SecurityType.Crypto)
58+
{
59+
_market = string.IsNullOrWhiteSpace(market)
60+
? null
61+
: market.ToLowerInvariant();
62+
63+
_processingDate = date;
64+
_securityType = securityType;
65+
_rawDataFolder = new DirectoryInfo(Path.Combine(rawDataFolder, "crypto", "coinapi"));
66+
if (!_rawDataFolder.Exists)
67+
{
68+
throw new ArgumentException($"CoinApiDataConverter(): Source folder not found: {_rawDataFolder.FullName}");
69+
}
70+
71+
_destinationFolder = new DirectoryInfo(destinationFolder);
72+
_destinationFolder.Create();
73+
}
74+
75+
/// <summary>
76+
/// Runs this instance.
77+
/// </summary>
78+
/// <returns></returns>
79+
public bool Run()
80+
{
81+
var stopwatch = Stopwatch.StartNew();
82+
83+
var symbolMapper = new CoinApiSymbolMapper();
84+
var success = true;
85+
86+
// There were cases of files with with an extra suffix, following pattern:
87+
// <TickType>-<ID>-<Exchange>_SPOT_<BaseCurrency>_<QuoteCurrency>_<ExtraSuffix>.csv.gz
88+
// Those cases should be ignored for SPOT prices.
89+
var tradesFolder = new DirectoryInfo(
90+
Path.Combine(
91+
_rawDataFolder.FullName,
92+
"trades",
93+
_processingDate.ToStringInvariant(DateFormat.EightCharacter)));
94+
95+
var quotesFolder = new DirectoryInfo(
96+
Path.Combine(
97+
_rawDataFolder.FullName,
98+
"quotes",
99+
_processingDate.ToStringInvariant(DateFormat.EightCharacter)));
100+
101+
var rawMarket = _market != null &&
102+
CoinApiSymbolMapper.MapMarketsToExchangeIds.TryGetValue(_market, out var rawMarketValue)
103+
? rawMarketValue
104+
: null;
105+
106+
var securityTypeFilter = (string name) => name.Contains("_SPOT_");
107+
if (_securityType == SecurityType.CryptoFuture)
108+
{
109+
securityTypeFilter = (string name) => name.Contains("_FTS_") || name.Contains("_PERP_");
110+
}
111+
112+
// Distinct by tick type and first two parts of the raw file name, separated by '-'.
113+
// This prevents us from double processing the same ticker twice, in case we're given
114+
// two raw data files for the same symbol. Related: https://github.com/QuantConnect/Lean/pull/3262
115+
var apiDataReader = new CoinApiDataReader(symbolMapper);
116+
var filesToProcessCandidates = tradesFolder.EnumerateFiles("*.gz")
117+
.Concat(quotesFolder.EnumerateFiles("*.gz"))
118+
.Where(f => securityTypeFilter(f.Name) && (rawMarket == null || f.Name.Contains(rawMarket)))
119+
.Where(f => f.Name.Split('_').Length == 4)
120+
.ToList();
121+
122+
var filesToProcessKeys = new HashSet<string>();
123+
var filesToProcess = new List<FileInfo>();
124+
125+
foreach (var candidate in filesToProcessCandidates)
126+
{
127+
try
128+
{
129+
var entryData = apiDataReader.GetCoinApiEntryData(candidate, _processingDate, _securityType);
130+
CurrencyPairUtil.DecomposeCurrencyPair(entryData.Symbol, out var baseCurrency, out var quoteCurrency);
131+
132+
if (!candidate.FullName.Contains(baseCurrency) && !candidate.FullName.Contains(quoteCurrency))
133+
{
134+
throw new Exception($"Skipping {candidate.FullName} we have the wrong symbol {entryData.Symbol}!");
135+
}
136+
137+
var key = candidate.Directory.Parent.Name + entryData.Symbol.ID;
138+
if (filesToProcessKeys.Add(key))
139+
{
140+
// Separate list from HashSet to preserve ordering of viable candidates
141+
filesToProcess.Add(candidate);
142+
}
143+
}
144+
catch (Exception err)
145+
{
146+
// Most likely the exchange isn't supported. Log exception message to avoid excessive stack trace spamming in console output
147+
Log.Error(err.Message);
148+
}
149+
}
150+
151+
Parallel.ForEach(filesToProcess, (file, loopState) =>
152+
{
153+
Log.Trace($"CoinApiDataConverter(): Starting data conversion from source file: {file.Name}...");
154+
try
155+
{
156+
ProcessEntry(apiDataReader, file);
157+
}
158+
catch (Exception e)
159+
{
160+
Log.Error(e, $"CoinApiDataConverter(): Error processing entry: {file.Name}");
161+
success = false;
162+
loopState.Break();
163+
}
164+
}
165+
);
166+
167+
Log.Trace($"CoinApiDataConverter(): Finished in {stopwatch.Elapsed}");
168+
return success;
169+
}
170+
171+
/// <summary>
172+
/// Processes the entry.
173+
/// </summary>
174+
/// <param name="coinapiDataReader">The coinapi data reader.</param>
175+
/// <param name="file">The file.</param>
176+
private void ProcessEntry(CoinApiDataReader coinapiDataReader, FileInfo file)
177+
{
178+
var entryData = coinapiDataReader.GetCoinApiEntryData(file, _processingDate, _securityType);
179+
180+
if (!SupportedMarkets.Contains(entryData.Symbol.ID.Market))
181+
{
182+
// only convert data for supported exchanges
183+
return;
184+
}
185+
186+
var tickData = coinapiDataReader.ProcessCoinApiEntry(entryData, file);
187+
188+
// in some cases the first data points from '_processingDate' get's included in the previous date file
189+
// so we will ready previous date data and drop most of it just to save these midnight ticks
190+
var yesterdayDate = _processingDate.AddDays(-1);
191+
var yesterdaysFile = new FileInfo(file.FullName.Replace(
192+
_processingDate.ToStringInvariant(DateFormat.EightCharacter),
193+
yesterdayDate.ToStringInvariant(DateFormat.EightCharacter)));
194+
if (yesterdaysFile.Exists)
195+
{
196+
var yesterdaysEntryData = coinapiDataReader.GetCoinApiEntryData(yesterdaysFile, yesterdayDate, _securityType);
197+
tickData = tickData.Concat(coinapiDataReader.ProcessCoinApiEntry(yesterdaysEntryData, yesterdaysFile));
198+
}
199+
else
200+
{
201+
Log.Error($"CoinApiDataConverter(): yesterdays data file not found '{yesterdaysFile.FullName}'");
202+
}
203+
204+
// materialize the enumerable into a list, since we need to enumerate over it twice
205+
var ticks = tickData.Where(tick => tick.Time.Date == _processingDate)
206+
.OrderBy(t => t.Time)
207+
.ToList();
208+
209+
var writer = new LeanDataWriter(Resolution.Tick, entryData.Symbol, _destinationFolder.FullName, entryData.TickType);
210+
writer.Write(ticks);
211+
212+
Log.Trace($"CoinApiDataConverter(): Starting consolidation for {entryData.Symbol.Value} {entryData.TickType}");
213+
var consolidators = new List<TickAggregator>();
214+
215+
if (entryData.TickType == TickType.Trade)
216+
{
217+
consolidators.AddRange(new[]
218+
{
219+
new TradeTickAggregator(Resolution.Second),
220+
new TradeTickAggregator(Resolution.Minute)
221+
});
222+
}
223+
else
224+
{
225+
consolidators.AddRange(new[]
226+
{
227+
new QuoteTickAggregator(Resolution.Second),
228+
new QuoteTickAggregator(Resolution.Minute)
229+
});
230+
}
231+
232+
foreach (var tick in ticks)
233+
{
234+
if (tick.Suspicious)
235+
{
236+
// When CoinAPI loses connectivity to the exchange, they indicate
237+
// it in the data by providing a value of `-1` for bid/ask price.
238+
// We will keep it in tick data, but will remove it from consolidated data.
239+
continue;
240+
}
241+
242+
foreach (var consolidator in consolidators)
243+
{
244+
consolidator.Update(tick);
245+
}
246+
}
247+
248+
foreach (var consolidator in consolidators)
249+
{
250+
writer = new LeanDataWriter(consolidator.Resolution, entryData.Symbol, _destinationFolder.FullName, entryData.TickType);
251+
writer.Write(consolidator.Flush());
252+
}
253+
}
254+
}
255+
}

0 commit comments

Comments
 (0)