Skip to content

Commit

Permalink
Implementation of parallel exchange requests
Browse files Browse the repository at this point in the history
  • Loading branch information
@sjanel
sjanel committed Aug 11, 2023
1 parent 9c77a37 commit 278af63
Show file tree
Hide file tree
Showing 19 changed files with 419 additions and 95 deletions.
21 changes: 11 additions & 10 deletions CONFIG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,17 @@ Configures the logging, tracking activity of relevant commands, and console outp

### Options description

| Name | Value | Description |
| ------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **apiOutputType** | String among {`off`, `table`, `json`} | Configure the default output type of coincenter (can be overridden by command line)queries |
| **fiatConversion.rate** | Duration string (ex: `8h`) | Minimum time between two consecutive requests of the same fiat conversion |
| **log.activityTracking.commandTypes** | Array of strings (ex: `["Buy", "Sell"]`) | Array of command types whose output will be stored to activity history files. |
| **log.activityTracking.dateFileNameFormat** | String (ex: `%Y-%m` for month split) | Defines the date string format suffix used by activity history files. The string should be compatible with [std::strftime](https://en.cppreference.com/w/cpp/chrono/c/strftime). Old data will never be clean-up by `coincenter` (as it may contain important data). User should manage the clean-up / storage. |
| **log.consoleLevel** | String | Defines the log level for standard output. Can be {'off', 'critical', 'error', 'warning', 'info', 'debug', 'trace'} |
| **log.fileLevel** | String | Defines the log level in files. Can be {'off', 'critical', 'error', 'warning', 'info', 'debug', 'trace'} |
| **log.maxFileSize** | String (ex: `5Mi` for 5 Megabytes) | Defines in bytes the maximum logging file size. A string representation of an integral, possibly with one suffix ending such as k, M, G, T (1k multipliers) or Ki, Mi, Gi, Ti (1024 multipliers) are supported. |
| **log.maxNbFiles** | Integer | Number of maximum rotating files for log in files |
| Name | Value | Description |
| ---------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **apiOutputType** | String among {`off`, `table`, `json`} | Configure the default output type of coincenter (can be overridden by command line)queries |
| **fiatConversion.rate** | Duration string (ex: `8h`) | Minimum time between two consecutive requests of the same fiat conversion |
| **log.activityTracking.commandTypes** | Array of strings (ex: `["Buy", "Sell"]`) | Array of command types whose output will be stored to activity history files. |
| **log.activityTracking.dateFileNameFormat** | String (ex: `%Y-%m` for month split) | Defines the date string format suffix used by activity history files. The string should be compatible with [std::strftime](https://en.cppreference.com/w/cpp/chrono/c/strftime). Old data will never be clean-up by `coincenter` (as it may contain important data). User should manage the clean-up / storage. |
| **log.consoleLevel** | String | Defines the log level for standard output. Can be {'off', 'critical', 'error', 'warning', 'info', 'debug', 'trace'} |
| **log.fileLevel** | String | Defines the log level in files. Can be {'off', 'critical', 'error', 'warning', 'info', 'debug', 'trace'} |
| **log.maxFileSize** | String (ex: `5Mi` for 5 Megabytes) | Defines in bytes the maximum logging file size. A string representation of an integral, possibly with one suffix ending such as k, M, G, T (1k multipliers) or Ki, Mi, Gi, Ti (1024 multipliers) are supported. |
| **log.maxNbFiles** | Integer | Number of maximum rotating files for log in files |
| **requests.concurrency.nbMaxParallelRequests** | Integer | Size of the thread pool that makes exchange requests. |


## static/exchangeconfig.json
Expand Down
7 changes: 7 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,13 @@ If only one private exchange is given and start amount is absolute, `coincenter`

#### Options

**Parallel requests**

You may want to query several exchanges for a command at the same time. In this case, it's possible to ask `coincenter` to launch requests in parallel when it's possible.
By default, the number of requests in parallel is `1`. To increase it, change the value of the field `nbMaxParallelRequests` in `generalconfig.json` file (more information [here](CONFIG.md#options-description)).

You will have a nice boost of speed when you query the same thing from multiple exchanges / or accounts. However, the logs may not be ordered anymore.

**Trade timeout**

A trade is **synchronous** by default with the created order of `coincenter` (it follows the lifetime of the order).
Expand Down
6 changes: 5 additions & 1 deletion src/engine/include/exchangesorchestrator.hpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,18 @@
#include "exchangeretriever.hpp"
#include "market.hpp"
#include "queryresulttypes.hpp"
#include "threadpool.hpp"
#include "tradedamounts.hpp"
#include "withdrawoptions.hpp"

namespace cct {

class RequestsConfig;
class ExchangesOrchestrator {
public:
using UniquePublicSelectedExchanges = ExchangeRetriever::UniquePublicSelectedExchanges;

explicit ExchangesOrchestrator(std::span<Exchange> exchangesSpan) : _exchangeRetriever(exchangesSpan) {}
explicit ExchangesOrchestrator(const RequestsConfig &requestsConfig, std::span<Exchange> exchangesSpan);

ExchangeHealthCheckStatus healthCheck(ExchangeNameSpan exchangeNames);

Expand Down Expand Up @@ -77,5 +80,6 @@ class ExchangesOrchestrator {

private:
ExchangeRetriever _exchangeRetriever;
ThreadPool _threadPool;
};
} // namespace cct
2 changes: 1 addition & 1 deletion src/engine/src/coincenter.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Coincenter::Coincenter(const CoincenterInfo &coincenterInfo, const ExchangeSecre
_apiKeyProvider(coincenterInfo.dataDir(), exchangeSecretsInfo, coincenterInfo.getRunMode()),
_metricsExporter(coincenterInfo.metricGatewayPtr()),
_exchangePool(coincenterInfo, _fiatConverter, _cryptowatchAPI, _apiKeyProvider),
_exchangesOrchestrator(_exchangePool.exchanges()),
_exchangesOrchestrator(coincenterInfo.requestsConfig(), _exchangePool.exchanges()),
_queryResultPrinter(coincenterInfo.apiOutputType(), _coincenterInfo.loggingInfo()) {}

int Coincenter::process(const CoincenterCommands &coincenterCommands) {
Expand Down
5 changes: 4 additions & 1 deletion src/engine/src/coincenterinfo_create.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,10 @@ CoincenterInfo CoincenterInfo_Create(std::string_view programName, const Coincen
LoggingInfo loggingInfo(LoggingInfo::WithLoggersCreation::kYes, dataDir,
static_cast<const json &>(generalConfigData["log"]));

GeneralConfig generalConfig(std::move(loggingInfo), fiatConversionQueryRate, apiOutputType);
RequestsConfig requestsConfig(generalConfigData["requests"]["concurrency"]["nbMaxParallelRequests"].get<int>());

GeneralConfig generalConfig(std::move(loggingInfo), std::move(requestsConfig), fiatConversionQueryRate,
apiOutputType);

LoadConfiguration loadConfiguration(dataDir, LoadConfiguration::ExchangeConfigFileType::kProd);

Expand Down
Loading

0 comments on commit 278af63

Please sign in to comment.