Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use strong setting types #1490

Merged
merged 43 commits into from
Sep 12, 2024
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d20ea8c
bus: update settings
peterjan Aug 28, 2024
7542b55
bus: combine settings
peterjan Aug 28, 2024
be4a40f
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/settings
peterjan Aug 28, 2024
f43d5d7
stores: add compat code
peterjan Aug 29, 2024
0e43e8d
all: update docs and defaults
peterjan Aug 29, 2024
8c76a21
all: cleanup PR
peterjan Aug 29, 2024
c45aa0b
all: cleanup PR
peterjan Aug 29, 2024
54b3d34
api: get rid of setting constants
peterjan Aug 29, 2024
ed16790
testing: fix TestPinManager
peterjan Aug 29, 2024
f21c2cb
testing: fix race in TestPinManager
peterjan Aug 29, 2024
94a840b
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/settings
peterjan Sep 3, 2024
b2c5eef
stores: unmarshal in store
peterjan Sep 3, 2024
1c4648d
settings: update store
peterjan Sep 3, 2024
36a73db
testing: fix setting key
peterjan Sep 3, 2024
8f268f9
Merge branch 'dev' into pj/settings
peterjan Sep 3, 2024
007f8ff
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/settings
peterjan Sep 5, 2024
0161496
Merge branch 'pj/settings' of github.com:SiaFoundation/renterd into p…
peterjan Sep 5, 2024
2cab31c
all: rename pin setting
peterjan Sep 5, 2024
3f82d0b
stores: move migration
peterjan Sep 6, 2024
c831617
Merge branch 'pj/explorer-config' of github.com:SiaFoundation/renterd…
peterjan Sep 6, 2024
829e6bf
sql: fix migration
peterjan Sep 6, 2024
8aa18a0
db: update migration
peterjan Sep 6, 2024
9df8aab
Merge branch 'pj/explorer-config' of github.com:SiaFoundation/renterd…
peterjan Sep 6, 2024
ca1c077
stores: use bool
peterjan Sep 6, 2024
151fb9c
build(deps): bump the all-dependencies group with 4 updates
dependabot[bot] Sep 9, 2024
fe695fa
stores: remove explorer enabled check
peterjan Sep 9, 2024
1d576bc
testing: fix TestPinManager NDF
peterjan Sep 9, 2024
a07d3ce
testing: add MineTransactions
peterjan Sep 9, 2024
f1748cd
tmp: run TestEphemeralAccountSync in a loop
peterjan Sep 9, 2024
414a239
Revert "tmp: run TestEphemeralAccountSync in a loop"
peterjan Sep 9, 2024
d923729
ui: v0.61.0
ChrisSchinnerl Sep 9, 2024
0d0321f
utils: add SendRequest
peterjan Sep 6, 2024
33c19cb
util: rename method
peterjan Sep 9, 2024
94a3a69
utils: remove named return vars
ChrisSchinnerl Sep 9, 2024
e082fdf
stores: use raw SQL in the migration
peterjan Sep 10, 2024
dafe545
stores: cleanup migration
peterjan Sep 10, 2024
a2e4ee1
db: add missing metrics migration
peterjan Sep 11, 2024
c5554a5
Fix missing metrics migration (#1522)
ChrisSchinnerl Sep 11, 2024
1f4d426
stores: remove default settings
peterjan Sep 11, 2024
458d37d
Merge branch 'dev' of github.com:SiaFoundation/renterd into pj/settings
peterjan Sep 11, 2024
7dc649a
Merge branch 'api-breakers' of github.com:SiaFoundation/renterd into …
peterjan Sep 11, 2024
62dbcd6
stores: fix indentation
peterjan Sep 11, 2024
87f0cc2
bus: remove network
peterjan Sep 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/ISSUE_TEMPLATE/bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@ body:
description: |
The configuration of your bus
```bash
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/contractset
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/gouging
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/redundancy
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/uploadpacking
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/settings/gouging
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/settings/pinned
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/settings/s3
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/settings/uploads
```
placeholder: Paste the output of the above commands here
validations:
Expand Down
59 changes: 0 additions & 59 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -558,65 +558,6 @@ formed.
}
```

### Contract Set

The contract set settings on the bus allow specifying a default contract set.
This contract set will be returned by the `bus` through the upload parameters,
and decides what contracts data is upload or migrated to by default. This
setting does not have a default value, it can be updated using the settings API:

- `GET /api/bus/setting/contractset`
- `PUT /api/bus/setting/contractset`

```json
{
"default": "autopilot"
}
```

In most cases the default set should match the set from your autopilot
configuration in order for migrations to work properly. The contract set can be
overridden by passing it as a query string parameter to the worker's upload and
migrate endpoints.

- `PUT /api/worker/objects/foo?contractset=foo`

### Redundancy

The default redundancy on mainnet is 30-10, on testnet it is 6-2. The redundancy
can be updated using the settings API:

- `GET /api/bus/setting/redundancy`
- `PUT /api/bus/setting/redundancy`

The redundancy can also be passed through query string parameters on the upload
endpoint in the worker API:

- `PUT /api/worker/objects/foo?minshards=2&totalshards=5`

### Gouging

The default gouging settings are listed below. The gouging settings can be
updated using the settings API:

- `GET /api/bus/setting/gouging`
- `PUT /api/bus/setting/gouging`

```json
{
"hostBlockHeightLeeway": 6, // 6 blocks
"maxContractPrice": "15000000000000000000000000", // 15 SC per contract
"maxDownloadPrice": "3000000000000000000000000000", // 3000 SC per 1 TB
"maxRPCPrice": "1000000000000000000000", // 1mS per RPC
"maxStoragePrice": "631593542824", // 3000 SC per TB per month
"maxUploadPrice": "3000000000000000000000000000", // 3000 SC per 1 TB
"migrationSurchargeMultiplier": 10, // overpay up to 10x for sectors migrations on critical slabs
"minAccountExpiry": 86400000000000, // 1 day
"minMaxEphemeralAccountBalance": "1000000000000000000000000", // 1 SC
"minPriceTableValidity": 300000000000 // 5 minutes
}
```

### Blocklist

Unfortunately the Sia blockchain is subject to hosts that announced themselves
Expand Down
42 changes: 11 additions & 31 deletions api/events.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ const (

EventAdd = "add"
EventUpdate = "update"
EventDelete = "delete"
EventArchive = "archive"
EventRenew = "renew"
)
Expand Down Expand Up @@ -51,27 +50,24 @@ type (
Timestamp time.Time `json:"timestamp"`
}

EventHostUpdate struct {
HostKey types.PublicKey `json:"hostKey"`
NetAddr string `json:"netAddr"`
Timestamp time.Time `json:"timestamp"`
}

EventContractSetUpdate struct {
Name string `json:"name"`
ContractIDs []types.FileContractID `json:"contractIDs"`
Timestamp time.Time `json:"timestamp"`
}

EventSettingUpdate struct {
Key string `json:"key"`
Update interface{} `json:"update"`
Timestamp time.Time `json:"timestamp"`
EventHostUpdate struct {
HostKey types.PublicKey `json:"hostKey"`
NetAddr string `json:"netAddr"`
Timestamp time.Time `json:"timestamp"`
}

EventSettingDelete struct {
Key string `json:"key"`
Timestamp time.Time `json:"timestamp"`
EventSettingUpdate struct {
GougingSettings *GougingSettings `json:"gougingSettings,omitempty"`
PinnedSettings *PinnedSettings `json:"pinnedSettings,omitempty"`
S3Settings *S3Settings `json:"s3Settings,omitempty"`
UploadSettings *UploadSettings `json:"uploadSettings,omitempty"`
Timestamp time.Time `json:"timestamp"`
}
)

Expand Down Expand Up @@ -138,15 +134,6 @@ var (
URL: url,
}
}

WebhookSettingDelete = func(url string, headers map[string]string) webhooks.Webhook {
return webhooks.Webhook{
Event: EventDelete,
Headers: headers,
Module: ModuleSetting,
URL: url,
}
}
)

func ParseEventWebhook(event webhooks.Event) (interface{}, error) {
Expand Down Expand Up @@ -201,19 +188,12 @@ func ParseEventWebhook(event webhooks.Event) (interface{}, error) {
return e, nil
}
case ModuleSetting:
switch event.Event {
case EventUpdate:
if event.Event == EventUpdate {
var e EventSettingUpdate
if err := json.Unmarshal(bytes, &e); err != nil {
return nil, err
}
return e, nil
case EventDelete:
var e EventSettingDelete
if err := json.Unmarshal(bytes, &e); err != nil {
return nil, err
}
return e, nil
}
}
return nil, fmt.Errorf("%w: module %s event %s", ErrUnknownEvent, event.Module, event.Event)
Expand Down
126 changes: 66 additions & 60 deletions api/setting.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,19 +5,11 @@ import (
"fmt"
"time"

"go.sia.tech/core/consensus"
rhpv2 "go.sia.tech/core/rhp/v2"
"go.sia.tech/core/types"
)

const (
SettingContractSet = "contractset"
SettingGouging = "gouging"
SettingPricePinning = "pricepinning"
SettingRedundancy = "redundancy"
SettingS3Authentication = "s3authentication"
SettingUploadPacking = "uploadpacking"
)

const (
S3MinAccessKeyLen = 16
S3MaxAccessKeyLen = 128
Expand All @@ -28,15 +20,11 @@ var (
// ErrInvalidRedundancySettings is returned if the redundancy settings are
// not valid
ErrInvalidRedundancySettings = errors.New("invalid redundancy settings")
)

// ErrSettingNotFound is returned if a requested setting is not present in the
// database.
ErrSettingNotFound = errors.New("setting not found")

var (
// DefaultGougingSettings define the default gouging settings the bus is
// configured with on startup. These values can be adjusted using the
// settings API.
//
// configured with on startup.
DefaultGougingSettings = GougingSettings{
MaxRPCPrice: types.Siacoins(1).Div64(1000), // 1mS per RPC
MaxContractPrice: types.Siacoins(15), // 15 SC per contract
Expand All @@ -50,48 +38,48 @@ var (
MigrationSurchargeMultiplier: 10, // 10x
}

// DefaultPricePinSettings define the default price pin settings the bus is
// configured with on startup. These values can be adjusted using the
// settings API.
DefaultPricePinSettings = PricePinSettings{
// DefaultPinnedSettings define the default pin settings the bus is
// configured with on startup.
DefaultPinnedSettings = PinnedSettings{
peterjan marked this conversation as resolved.
Show resolved Hide resolved
Enabled: false,
Currency: "usd",
ForexEndpointURL: "https://api.siascan.com/exchange-rate/siacoin",
Threshold: 0.05,
}

// DefaultUploadPackingSettings define the default upload packing settings
// the bus is configured with on startup.
DefaultUploadPackingSettings = UploadPackingSettings{
Enabled: true,
SlabBufferMaxSizeSoft: 1 << 32, // 4 GiB
}

// DefaultRedundancySettings define the default redundancy settings the bus
// is configured with on startup. These values can be adjusted using the
// settings API.
//
// NOTE: default redundancy settings for testnet are different from mainnet.
DefaultRedundancySettings = RedundancySettings{
MinShards: 10,
TotalShards: 30,
}

// Same as DefaultRedundancySettings but for running on testnet networks due
// to their reduced number of hosts.
// DefaultRedundancySettingsTestnet defines redundancy settings for the
// testnet, these are lower due to the reduced number of hosts on the
// testnet.
DefaultRedundancySettingsTestnet = RedundancySettings{
MinShards: 2,
TotalShards: 6,
}

// DefaultS3Settings defines the 3 settings the bus is configured with on
// startup.
DefaultS3Settings = S3Settings{
Authentication: S3AuthenticationSettings{
V4Keypairs: map[string]string{},
},
}
)

type (
// ContractSetSetting contains the default contract set used by the worker for
// uploads and migrations.
ContractSetSetting struct {
Default string `json:"default"`
// DefaultUploadSettings define the default upload settings the bus is
// configured with on startup.
func DefaultUploadSettings(network *consensus.Network) UploadSettings {
return UploadSettings{
Packing: UploadPackingSettings{
Enabled: true,
SlabBufferMaxSizeSoft: 1 << 32, // 4 GiB
},
Redundancy: RedundancySettings{
MinShards: 10,
TotalShards: 30,
},
}
}

type (
// GougingSettings contain some price settings used in price gouging.
GougingSettings struct {
// MaxRPCPrice is the maximum allowed base price for RPCs
Expand Down Expand Up @@ -132,10 +120,10 @@ type (
MigrationSurchargeMultiplier uint64 `json:"migrationSurchargeMultiplier"`
}

// PricePinSettings holds the configuration for pinning certain settings to
// a specific currency (e.g., USD). It uses a Forex API to fetch the current
// PinnedSettings holds the configuration for pinning certain settings to a
// specific currency (e.g., USD). It uses a Forex API to fetch the current
// exchange rate, allowing users to set prices in USD instead of SC.
PricePinSettings struct {
PinnedSettings struct {
// Enabled can be used to either enable or temporarily disable price
// pinning. If enabled, both the currency and the Forex endpoint URL
// must be valid.
Expand All @@ -160,6 +148,23 @@ type (
GougingSettingsPins GougingSettingsPins `json:"gougingSettingsPins"`
}

// UploadSettings contains various settings related to uploads.
UploadSettings struct {
DefaultContractSet string `json:"defaultContractSet"`
Packing UploadPackingSettings `json:"packing"`
Redundancy RedundancySettings `json:"redundancy"`
}

UploadPackingSettings struct {
Enabled bool `json:"enabled"`
SlabBufferMaxSizeSoft int64 `json:"slabBufferMaxSizeSoft"`
}

RedundancySettings struct {
MinShards int `json:"minShards"`
TotalShards int `json:"totalShards"`
}

// AutopilotPins contains the available autopilot settings that can be
// pinned.
AutopilotPins struct {
Expand All @@ -180,22 +185,15 @@ type (
Value float64 `json:"value"`
}

// RedundancySettings contain settings that dictate an object's redundancy.
RedundancySettings struct {
MinShards int `json:"minShards"`
TotalShards int `json:"totalShards"`
// S3Settings contains various settings related to the S3 API.
S3Settings struct {
Authentication S3AuthenticationSettings `json:"authentication"`
}

// S3AuthenticationSettings contains S3 auth settings.
S3AuthenticationSettings struct {
V4Keypairs map[string]string `json:"v4Keypairs"`
}

// UploadPackingSettings contains upload packing settings.
UploadPackingSettings struct {
Enabled bool `json:"enabled"`
SlabBufferMaxSizeSoft int64 `json:"slabBufferMaxSizeSoft"`
}
)

// IsPinned returns true if the pin is enabled and the value is greater than 0.
Expand All @@ -204,7 +202,7 @@ func (p Pin) IsPinned() bool {
}

// Validate returns an error if the price pin settings are not considered valid.
func (pps PricePinSettings) Validate() error {
func (pps PinnedSettings) Validate() error {
if pps.ForexEndpointURL == "" {
return fmt.Errorf("price pin settings must have a forex endpoint URL")
}
Expand Down Expand Up @@ -239,6 +237,14 @@ func (gs GougingSettings) Validate() error {
return nil
}

// Validate returns an error if the upload settings are not considered valid.
func (us UploadSettings) Validate() error {
if us.Packing.Enabled && us.Packing.SlabBufferMaxSizeSoft <= 0 {
return errors.New("SlabBufferMaxSizeSoft must be greater than zero when upload packing is enabled")
}
return us.Redundancy.Validate()
}

// Redundancy returns the effective storage redundancy of the
// RedundancySettings.
func (rs RedundancySettings) Redundancy() float64 {
Expand Down Expand Up @@ -272,8 +278,8 @@ func (rs RedundancySettings) Validate() error {

// Validate returns an error if the authentication settings are not considered
// valid.
func (s3as S3AuthenticationSettings) Validate() error {
for accessKeyID, secretAccessKey := range s3as.V4Keypairs {
func (s3s S3Settings) Validate() error {
for accessKeyID, secretAccessKey := range s3s.Authentication.V4Keypairs {
if accessKeyID == "" {
return fmt.Errorf("AccessKeyID cannot be empty")
} else if len(accessKeyID) < S3MinAccessKeyLen || len(accessKeyID) > S3MaxAccessKeyLen {
Expand Down
Loading
Loading