Skip to content

Commit

Permalink
Update endpoint config for eth sdk (#550)
Browse files Browse the repository at this point in the history 
* Update endpoint config for eth sdk

* Update endpoint docs

---------

Co-authored-by: James Bayly <james@bayly.xyz>
  • Loading branch information
@stwiname @jamesbayly
stwiname and jamesbayly authored Aug 2, 2024
1 parent 9c29636 commit 0a38ba6
Show file tree
Hide file tree
Showing 9 changed files with 251 additions and 85 deletions.
45 changes: 33 additions & 12 deletions docs/indexer/build/manifest/arbitrum.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ const project: EthereumProject = {
* When developing your project we suggest getting a private API key
# We suggest providing an array of endpoints for increased speed and reliability
*/
endpoint: ["https://arbitrum.api.onfinality.io/public"],
endpoint: ["https://arbitrum.rpc.subquery.network/public"],
dictionary: "https://dict-tyk.subquery.network/query/arbitrum",
},
dataSources: [
Expand Down Expand Up @@ -134,10 +134,8 @@ network:
# This endpoint must be a public non-pruned archive node
# We recommend providing more than one endpoint for improved reliability, performance, and uptime
# Public nodes may be rate limited, which can affect indexing speed
# When developing your project we suggest getting a private API key
# You can get them from OnFinality for free https://app.onfinality.io
# https://documentation.onfinality.io/support/the-enhanced-api-service
endpoint: ["https://arbitrum.api.onfinality.io/public"]
# When developing your project we suggest getting a private API key and using the endpoint config
endpoint: ["https://arbitrum.rpc.subquery.network/public"]
# Recommended to provide the HTTP endpoint of a full chain dictionary to speed up processing
dictionary: "https://api.subquery.network/sq/subquery/arbitrum-one-dictionary"

Expand Down Expand Up @@ -207,16 +205,16 @@ Additionally you will need to update the `endpoint`. This defines the (HTTP or W
- Increased reliability - If an endpoint goes offline, SubQuery will automatically switch to other RPC providers to continue indexing without interruption.
- Reduced load on RPC providers - Indexing is a computationally expensive process on RPC providers, by distributing requests among RPC providers you are lowering the chance that your project will be rate limited.

Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider like [OnFinality](https://onfinality.io/networks/arbitrum).
Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider.

There is only a dictionary for Arbitrum One `https://gx.api.subquery.network/sq/subquery/arbitrum-one-dictionary`. For Arbitrum Nova, please omit the dictionary setting.

| Field | Type | Description |
| ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] | Defines the endpoint of the blockchain to be indexed, this can be a string or an array of endpoints - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |
| Field | Type | Description |
| ---------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] or Record\<String, IEndpointConfig\> | Defines the endpoint of the blockchain to be indexed, this can be a string, an array of endpoints, or a record of endpoints to [endpoint configs](#endpoint-config) - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |

### Runner Spec

Expand Down Expand Up @@ -338,3 +336,26 @@ When declaring a `range` use an string in the format of `"start - end"`. Both st
}
}
```

## Endpoint Config

This allows you to set specific options relevant to each specific RPC endpoint that you are indexing from. This is very useful when endpoints have unique authentication requirements, or they operate with different rate limits.

Here is an example of how to set an API key in the header of RPC requests in your endpoint config.


```ts
{
network: {
endpoint: {
"https://arbitrum.rpc.subquery.network/public": {
headers: {
"x-api-key": "your-api-key",
},
// NOTE: setting this to 0 will not use batch requests
batchSize: 5
}
}
}
}
```
35 changes: 29 additions & 6 deletions docs/indexer/build/manifest/avalanche.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -218,12 +218,13 @@ Additionally you will need to update the `endpoint`. This defines the (HTTP or W

Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider like [OnFinality](https://onfinality.io/networks/avalanche).

| Field | Type | Description |
| ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] | Defines the endpoint of the blockchain to be indexed, this can be a string or an array of endpoints - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |
| Field | Type | Description |
| ---------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] or Record\<String, IEndpointConfig\> | Defines the endpoint of the blockchain to be indexed, this can be a string, an array of endpoints, or a record of endpoints to [endpoint configs](#endpoint-config) - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |

### Runner Spec

| Field | Type | Description |
Expand Down Expand Up @@ -344,3 +345,25 @@ When declaring a `range` use an string in the format of `"start - end"`. Both st
}
}
```

## Endpoint Config

This allows you to set specific options relevant to each specific RPC endpoint that you are indexing from. This is very useful when endpoints have unique authentication requirements, or they operate with different rate limits.

Here is an example of how to set an API key in the header of RPC requests in your endpoint config.

```ts
{
network: {
endpoint: {
"https://arbitrum.rpc.subquery.network/public": {
headers: {
"x-api-key": "your-api-key",
},
// NOTE: setting this to 0 will not use batch requests
batchSize: 5
}
}
}
}
```
43 changes: 32 additions & 11 deletions docs/indexer/build/manifest/bsc.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,10 +134,8 @@ network:
# This endpoint must be a public non-pruned archive node
# We recommend providing more than one endpoint for improved reliability, performance, and uptime
# Public nodes may be rate limited, which can affect indexing speed
# When developing your project we suggest getting a private API key
# You can get them from OnFinality for free https://app.onfinality.io
# https://documentation.onfinality.io/support/the-enhanced-api-service
endpoint: ["https://bsc-dataseed1.binance.org"]
# When developing your project we suggest getting a private API key and using endpoint config
endpoint: ["https://bnb.rpc.subquery.network/public"]
# Recommended to provide the HTTP endpoint of a full chain dictionary to speed up processing
dictionary: "https://gx.api.subquery.network/sq/subquery/bsc-dictionary"

Expand Down Expand Up @@ -207,14 +205,15 @@ Additionally you will need to update the `endpoint`. This defines the (HTTP or W
- Increased reliability - If an endpoint goes offline, SubQuery will automatically switch to other RPC providers to continue indexing without interruption.
- Reduced load on RPC providers - Indexing is a computationally expensive process on RPC providers, by distributing requests among RPC providers you are lowering the chance that your project will be rate limited.

Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider like [OnFinality](https://onfinality.io/networks/bsc).
Public nodes may be rate limited which can affect indexing speed, when developing your project we suggest getting a private API key from a professional RPC provider.

| Field | Type | Description |
| ---------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] or Record\<String, IEndpointConfig\> | Defines the endpoint of the blockchain to be indexed, this can be a string, an array of endpoints, or a record of endpoints to [endpoint configs](#endpoint-config) - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |

| Field | Type | Description |
| ---------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **chainId** | String | A network identifier for the blockchain |
| **endpoint** | String or String[] | Defines the endpoint of the blockchain to be indexed, this can be a string or an array of endpoints - **This must be a full archive node**. |
| **dictionary** | String | It is suggested to provide the HTTP endpoint of a full chain dictionary to speed up processing - read [how a SubQuery Dictionary works](../../academy/tutorials_examples/dictionary.md). |
| **bypassBlocks** | Array | Bypasses stated block numbers, the values can be a `range`(e.g. `"10- 50"`) or `integer`, see [Bypass Blocks](#bypass-blocks) |
### Runner Spec

| Field | Type | Description |
Expand Down Expand Up @@ -335,3 +334,25 @@ When declaring a `range` use an string in the format of `"start - end"`. Both st
}
}
```

## Endpoint Config

This allows you to set specific options relevant to each specific RPC endpoint that you are indexing from. This is very useful when endpoints have unique authentication requirements, or they operate with different rate limits.

Here is an example of how to set an API key in the header of RPC requests in your endpoint config.

```ts
{
network: {
endpoint: {
"https://bnb.rpc.subquery.network/public": {
headers: {
"x-api-key": "your-api-key",
},
// NOTE: setting this to 0 will not use batch requests
batchSize: 5
}
}
}
}
```
Loading

0 comments on commit 0a38ba6

Please sign in to comment.