Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: configuration validation #1778

Merged
merged 33 commits into from
Oct 5, 2023
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
6ba3dd8
wip
maschad May 24, 2023
68f6e83
wip
maschad May 29, 2023
5874c4e
refactor: added connection manager config validation
maschad May 29, 2023
71490db
refactor: added addresses config validation (#1573)
maschad May 29, 2023
3aeb754
wip
maschad May 31, 2023
36f66ef
Merge branch 'master' into feat/configuration-validation
maschad Jun 30, 2023
722dc7a
wip
maschad Jul 3, 2023
27a9789
feat: added fetch validation
maschad Jul 3, 2023
73f9fc7
feat: added default announce filter (#1573)
maschad Jul 3, 2023
893e068
feat: refactored config to validate before returning service (#1573)
maschad Jul 4, 2023
6c5f7fd
Merge branch 'master' into feat/configuration-validation
maschad Jul 4, 2023
1e9b0e3
chore: linting fixes
maschad Jul 4, 2023
c70f7c1
wip
maschad Jul 4, 2023
dd17fcd
feat: added validation to unpnp and circuit relay transport
maschad Jul 5, 2023
df1ba0e
Merge branch 'master' into feat/configuration-validation
maschad Jul 5, 2023
f86432a
test: refactor tests + updated config (#1573)
maschad Jul 5, 2023
c3ed878
chore: linting (#1573)
maschad Jul 5, 2023
be8bd3a
Merge branch 'master' into feat/configuration-validation
maschad Jul 21, 2023
6bd7bec
feat: updated config validation for identify and circuit relay (#1573)
maschad Jul 21, 2023
d4ec37a
Merge branch 'master' into feat/configuration-validation
maschad Jul 31, 2023
8e23b59
chore: linting
maschad Jul 31, 2023
083414f
Merge branch 'master' into feat/configuration-validation
maschad Aug 4, 2023
0820b6f
Merge branch 'master' into feat/configuration-validation
maschad Aug 7, 2023
83a5f0e
Merge branch 'master' into feat/configuration-validation
maschad Aug 18, 2023
dbbcbbd
Merge branch 'master' into feat/configuration-validation
maschad Aug 18, 2023
0d4c8e4
fix: added transient connection default to identify service (#1573)
maschad Aug 20, 2023
a89818a
refactor: Refactored config to be done inside services (#1573)
maschad Aug 21, 2023
264ec7c
Merge branch 'master' into feat/configuration-validation
maschad Sep 6, 2023
b4fb523
Merge branch 'master' into feat/configuration-validation
maschad Sep 15, 2023
6c7935a
Merge branch 'master' into feat/configuration-validation
maschad Sep 15, 2023
6638573
Update packages/libp2p/src/circuit-relay/server/index.ts
maschad Oct 5, 2023
e0080fb
Merge branch 'master' into feat/configuration-validation
maschad Oct 5, 2023
d45851d
feat: added validation to dcutr
maschad Oct 5, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions packages/libp2p/.aegir.js
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,9 @@ export default {
const peerId = await createEd25519PeerId()
const libp2p = await createLibp2p({
connectionManager: {
inboundConnectionThreshold: Infinity,
inboundConnectionThreshold: 1000,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I take it yup doesn't accept Infinity as a number?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

maxIncomingPendingConnections: 1000,
maxConnections: 1000,
Comment on lines +28 to +29
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default value for these was fine before, why do these now need to be set?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my comment #1778

minConnections: 0
},
addresses: {
Expand All @@ -51,7 +53,7 @@ export default {
fetch: fetchService(),
relay: circuitRelayServer({
reservations: {
maxReservations: Infinity
maxReservations: 100000
}
})
}
Expand Down
3 changes: 2 additions & 1 deletion packages/libp2p/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,8 @@
"uint8arraylist": "^2.4.3",
"uint8arrays": "^4.0.6",
"wherearewe": "^2.0.1",
"xsalsa20": "^1.1.0"
"xsalsa20": "^1.1.0",
"yup": "^1.2.0"
},
"devDependencies": {
"@chainsafe/libp2p-gossipsub": "^10.0.0",
Expand Down
14 changes: 14 additions & 0 deletions packages/libp2p/src/address-manager/utils.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
import { type ObjectSchema, object, array, string, mixed } from 'yup'
import { validateMultiaddr } from '../config/helpers.js'
import type { AddressManagerInit } from '.'
import type { Multiaddr } from '@multiformats/multiaddr'

export function debounce (func: () => void, wait: number): () => void {
let timeout: ReturnType<typeof setTimeout> | undefined

Expand All @@ -11,3 +16,12 @@ export function debounce (func: () => void, wait: number): () => void {
timeout = setTimeout(later, wait)
}
}

export function validateAddressManagerConfig (opts: AddressManagerInit): ObjectSchema<Record<string, unknown>> {
return object({
listen: array().of(string()).test('is multiaddr', validateMultiaddr).default([]),
announce: array().of(string()).test('is multiaddr', validateMultiaddr).default([]),
noAnnounce: array().of(string()).test('is multiaddr', validateMultiaddr).default([]),
announceFilter: mixed().default(() => (addrs: Multiaddr[]): Multiaddr[] => addrs)
})
}
22 changes: 16 additions & 6 deletions packages/libp2p/src/autonat/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ import map from 'it-map'
import parallel from 'it-parallel'
import { pipe } from 'it-pipe'
import isPrivateIp from 'private-ip'
import { number, object, string } from 'yup'
import { codes } from '../errors.js'
import {
MAX_INBOUND_STREAMS,
Expand Down Expand Up @@ -108,14 +109,23 @@ class DefaultAutoNATService implements Startable {
private started: boolean

constructor (components: AutoNATComponents, init: AutoNATServiceInit) {
const validatedConfig = object({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please have config definitions outside of the class so we don't create new validators on every object instantiation:

const configValidator = object({
  protocolPrefix: string().default(PROTOCOL_PREFIX),
  // ...etc
})

class DefaultAutoNATService implements Startable {
  // ...etc

  constructor (components: AutoNATComponents, init: AutoNATServiceInit) {
    const config = configValidator.validateSync(init)
    // ...etc
  }
}

protocolPrefix: string().default(PROTOCOL_PREFIX),
timeout: number().integer().default(TIMEOUT),
startupDelay: number().integer().default(STARTUP_DELAY),
refreshInterval: number().integer().default(REFRESH_INTERVAL),
maxInboundStreams: number().integer().default(MAX_INBOUND_STREAMS),
maxOutboundStreams: number().integer().default(MAX_OUTBOUND_STREAMS)
}).validateSync(init)

this.components = components
this.started = false
this.protocol = `/${init.protocolPrefix ?? PROTOCOL_PREFIX}/${PROTOCOL_NAME}/${PROTOCOL_VERSION}`
this.timeout = init.timeout ?? TIMEOUT
this.maxInboundStreams = init.maxInboundStreams ?? MAX_INBOUND_STREAMS
this.maxOutboundStreams = init.maxOutboundStreams ?? MAX_OUTBOUND_STREAMS
this.startupDelay = init.startupDelay ?? STARTUP_DELAY
this.refreshInterval = init.refreshInterval ?? REFRESH_INTERVAL
this.protocol = `/${validatedConfig.protocolPrefix}/${PROTOCOL_NAME}/${PROTOCOL_VERSION}`
this.timeout = validatedConfig.timeout
this.maxInboundStreams = validatedConfig.maxInboundStreams
this.maxOutboundStreams = validatedConfig.maxOutboundStreams
this.startupDelay = validatedConfig.startupDelay
this.refreshInterval = validatedConfig.refreshInterval
this._verifyExternalAddresses = this._verifyExternalAddresses.bind(this)
}

Expand Down
5 changes: 5 additions & 0 deletions packages/libp2p/src/circuit-relay/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,3 +70,8 @@ export const DEFAULT_HOP_TIMEOUT = 30 * second
* How long to wait before starting to advertise the relay service
*/
export const DEFAULT_ADVERT_BOOT_DELAY = 30 * second

/**
* The default timeout for Incoming STOP requests from the relay
*/
export const DEFAULT_STOP_TIMEOUT = 30 * second
37 changes: 26 additions & 11 deletions packages/libp2p/src/circuit-relay/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,18 @@ import { RecordEnvelope } from '@libp2p/peer-record'
import { type Multiaddr, multiaddr } from '@multiformats/multiaddr'
import { pbStream, type ProtobufStream } from 'it-protobuf-stream'
import pDefer from 'p-defer'
import { object, number, boolean } from 'yup'
import { MAX_CONNECTIONS } from '../../connection-manager/constants.js'
import { DEFAULT_MAX_INBOUND_STREAMS, DEFAULT_MAX_OUTBOUND_STREAMS } from '../../registrar.js'
import {
CIRCUIT_PROTO_CODE,
DEFAULT_DURATION_LIMIT,
DEFAULT_HOP_TIMEOUT,
RELAY_SOURCE_TAG,
RELAY_V2_HOP_CODEC,
RELAY_V2_STOP_CODEC
DEFAULT_MAX_RESERVATION_CLEAR_INTERVAL,
DEFAULT_MAX_RESERVATION_STORE_SIZE,
DEFAULT_MAX_RESERVATION_TTL,
RELAY_SOURCE_TAG
, RELAY_V2_HOP_CODEC, RELAY_V2_STOP_CODEC
maschad marked this conversation as resolved.
Show resolved Hide resolved
maschad marked this conversation as resolved.
Show resolved Hide resolved
} from '../constants.js'
import { HopMessage, type Reservation, Status, StopMessage } from '../pb/index.js'
import { createLimitedRelay } from '../utils.js'
Expand Down Expand Up @@ -95,10 +100,6 @@ export interface RelayServerEvents {
'relay:advert:error': CustomEvent<Error>
}

const defaults = {
maxOutboundStopStreams: MAX_CONNECTIONS
}

class CircuitRelayServer extends EventEmitter<RelayServerEvents> implements Startable, CircuitRelayService {
private readonly registrar: Registrar
private readonly peerStore: PeerStore
Expand All @@ -121,18 +122,32 @@ class CircuitRelayServer extends EventEmitter<RelayServerEvents> implements Star
constructor (components: CircuitRelayServerComponents, init: CircuitRelayServerInit = {}) {
super()

const validatedConfig = object({
hopTimeout: number().min(0).integer().default(DEFAULT_HOP_TIMEOUT),
reservations: object({
maxReservations: number().integer().min(0).default(DEFAULT_MAX_RESERVATION_STORE_SIZE),
reservationClearInterval: number().integer().min(0).default(DEFAULT_MAX_RESERVATION_CLEAR_INTERVAL),
applyDefaultLimit: boolean().default(true),
reservationTtl: number().integer().min(0).default(DEFAULT_MAX_RESERVATION_TTL),
defaultDurationLimit: number().integer().min(0).default(DEFAULT_DURATION_LIMIT).max(init?.reservations?.reservationTtl ?? DEFAULT_MAX_RESERVATION_TTL, `default duration limit must be less than reservation TTL: ${init?.reservations?.reservationTtl}`)
}),
maxInboundHopStreams: number().integer().min(0).default(DEFAULT_MAX_INBOUND_STREAMS),
maxOutboundHopStreams: number().integer().min(0).default(DEFAULT_MAX_OUTBOUND_STREAMS),
maxOutboundStopStreams: number().integer().min(0).default(MAX_CONNECTIONS)
}).validateSync(init)

this.registrar = components.registrar
this.peerStore = components.peerStore
this.addressManager = components.addressManager
this.peerId = components.peerId
this.connectionManager = components.connectionManager
this.connectionGater = components.connectionGater
this.started = false
this.hopTimeout = init?.hopTimeout ?? DEFAULT_HOP_TIMEOUT
this.hopTimeout = validatedConfig.hopTimeout
this.shutdownController = new AbortController()
this.maxInboundHopStreams = init.maxInboundHopStreams
this.maxOutboundHopStreams = init.maxOutboundHopStreams
this.maxOutboundStopStreams = init.maxOutboundStopStreams ?? defaults.maxOutboundStopStreams
this.maxInboundHopStreams = validatedConfig.maxInboundHopStreams
this.maxOutboundHopStreams = validatedConfig.maxOutboundHopStreams
this.maxOutboundStopStreams = validatedConfig.maxOutboundStopStreams

try {
// fails on node < 15.4
Expand Down
22 changes: 16 additions & 6 deletions packages/libp2p/src/circuit-relay/server/reservation-store.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { PeerMap } from '@libp2p/peer-collections'
import { object, mixed, number, boolean } from 'yup'
import { DEFAULT_DATA_LIMIT, DEFAULT_DURATION_LIMIT, DEFAULT_MAX_RESERVATION_CLEAR_INTERVAL, DEFAULT_MAX_RESERVATION_STORE_SIZE, DEFAULT_MAX_RESERVATION_TTL } from '../constants.js'
import { type Limit, Status } from '../pb/index.js'
import type { RelayReservation } from '../index.js'
Expand Down Expand Up @@ -50,12 +51,21 @@ export class ReservationStore implements Startable {
private readonly defaultDataLimit: bigint

constructor (options: ReservationStoreOptions = {}) {
this.maxReservations = options.maxReservations ?? DEFAULT_MAX_RESERVATION_STORE_SIZE
this.reservationClearInterval = options.reservationClearInterval ?? DEFAULT_MAX_RESERVATION_CLEAR_INTERVAL
this.applyDefaultLimit = options.applyDefaultLimit !== false
this.reservationTtl = options.reservationTtl ?? DEFAULT_MAX_RESERVATION_TTL
this.defaultDurationLimit = options.defaultDurationLimit ?? DEFAULT_DURATION_LIMIT
this.defaultDataLimit = options.defaultDataLimit ?? DEFAULT_DATA_LIMIT
const validatedConfig = object({
maxReservations: number().min(0).integer().default(DEFAULT_MAX_RESERVATION_STORE_SIZE),
reservationClearInterval: number().integer().min(0).default(DEFAULT_MAX_RESERVATION_CLEAR_INTERVAL),
applyDefaultLimit: boolean().default(true),
reservationTtl: number().integer().min(0).default(DEFAULT_MAX_RESERVATION_TTL),
defaultDurationLimit: number().integer().min(0).default(DEFAULT_DURATION_LIMIT),
defaultDataLimit: mixed().test('is-bigint', 'Invalid bigint', value => typeof value === 'bigint').default(DEFAULT_DATA_LIMIT)
}).validateSync(options)

this.maxReservations = validatedConfig.maxReservations
this.reservationClearInterval = validatedConfig.reservationClearInterval
this.applyDefaultLimit = validatedConfig.applyDefaultLimit
this.reservationTtl = validatedConfig.reservationTtl
this.defaultDurationLimit = validatedConfig.defaultDurationLimit
this.defaultDataLimit = validatedConfig.defaultDataLimit as bigint
}

isStarted (): boolean {
Expand Down
30 changes: 16 additions & 14 deletions packages/libp2p/src/circuit-relay/transport/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,10 @@ import { streamToMaConnection } from '@libp2p/utils/stream-to-ma-conn'
import * as mafmt from '@multiformats/mafmt'
import { multiaddr } from '@multiformats/multiaddr'
import { pbStream } from 'it-protobuf-stream'
import { number, object } from 'yup'
import { MAX_CONNECTIONS } from '../../connection-manager/constants.js'
import { codes } from '../../errors.js'
import { CIRCUIT_PROTO_CODE, RELAY_V2_HOP_CODEC, RELAY_V2_STOP_CODEC } from '../constants.js'
import { CIRCUIT_PROTO_CODE, DEFAULT_STOP_TIMEOUT, RELAY_V2_HOP_CODEC, RELAY_V2_STOP_CODEC } from '../constants.js'
import { StopMessage, HopMessage, Status } from '../pb/index.js'
import { RelayDiscovery, type RelayDiscoveryComponents } from './discovery.js'
import { createListener } from './listener.js'
Expand Down Expand Up @@ -100,12 +101,6 @@ export interface CircuitRelayTransportInit extends RelayStoreInit {
reservationCompletionTimeout?: number
}

const defaults = {
maxInboundStopStreams: MAX_CONNECTIONS,
maxOutboundStopStreams: MAX_CONNECTIONS,
stopTimeout: 30000
}

class CircuitRelayTransport implements Transport {
private readonly discovery?: RelayDiscovery
private readonly registrar: Registrar
Expand All @@ -116,24 +111,31 @@ class CircuitRelayTransport implements Transport {
private readonly addressManager: AddressManager
private readonly connectionGater: ConnectionGater
private readonly reservationStore: ReservationStore
private readonly maxInboundStopStreams: number
private readonly maxInboundStopStreams?: number
private readonly maxOutboundStopStreams?: number
private readonly stopTimeout: number
private readonly stopTimeout?: number
private started: boolean

constructor (components: CircuitRelayTransportComponents, init: CircuitRelayTransportInit) {
const validatedConfig = object({
discoverRelays: number().min(0).integer().default(0),
maxInboundStopStreams: number().min(0).integer().default(MAX_CONNECTIONS),
maxOutboundStopStreams: number().min(0).integer().default(MAX_CONNECTIONS),
stopTimeout: number().min(0).integer().default(DEFAULT_STOP_TIMEOUT)
}).validateSync(init)

this.registrar = components.registrar
this.peerStore = components.peerStore
this.connectionManager = components.connectionManager
this.peerId = components.peerId
this.upgrader = components.upgrader
this.addressManager = components.addressManager
this.connectionGater = components.connectionGater
this.maxInboundStopStreams = init.maxInboundStopStreams ?? defaults.maxInboundStopStreams
this.maxOutboundStopStreams = init.maxOutboundStopStreams ?? defaults.maxOutboundStopStreams
this.stopTimeout = init.stopTimeout ?? defaults.stopTimeout
this.maxInboundStopStreams = validatedConfig.maxInboundStopStreams
this.maxOutboundStopStreams = validatedConfig.maxOutboundStopStreams
this.stopTimeout = validatedConfig.stopTimeout

if (init.discoverRelays != null && init.discoverRelays > 0) {
if (validatedConfig.discoverRelays > 0) {
this.discovery = new RelayDiscovery(components)
this.discovery.addEventListener('relay:discover', (evt) => {
this.reservationStore.addRelay(evt.detail, 'discovered')
Expand Down Expand Up @@ -321,7 +323,7 @@ class CircuitRelayTransport implements Transport {
* An incoming STOP request means a remote peer wants to dial us via a relay
*/
async onStop ({ connection, stream }: IncomingStreamData): Promise<void> {
const signal = AbortSignal.timeout(this.stopTimeout)
const signal = AbortSignal.timeout(this.stopTimeout ?? DEFAULT_STOP_TIMEOUT)
const pbstr = pbStream(stream).pb(StopMessage)
const request = await pbstr.read({
signal
Expand Down
41 changes: 0 additions & 41 deletions packages/libp2p/src/config.ts

This file was deleted.

44 changes: 44 additions & 0 deletions packages/libp2p/src/config/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import { FaultTolerance } from '@libp2p/interface/transport'
import { publicAddressesFirst } from '@libp2p/utils/address-sort'
import { dnsaddrResolver } from '@multiformats/multiaddr/resolvers'
import mergeOptions from 'merge-options'
import { object } from 'yup'
import { validateAddressManagerConfig } from '../address-manager/utils.js'
import { validateConnectionManagerConfig } from '../connection-manager/utils.js'
import type { AddressManagerInit } from '../address-manager'
import type { ConnectionManagerInit } from '../connection-manager/index.js'
import type { Libp2pInit } from '../index.js'
import type { ServiceMap, RecursivePartial } from '@libp2p/interface'

const DefaultConfig: Partial<Libp2pInit> = {
connectionManager: {
resolvers: {
dnsaddr: dnsaddrResolver
},
addressSorter: publicAddressesFirst
},
transportManager: {
faultTolerance: FaultTolerance.FATAL_ALL
}
}

export function validateConfig <T extends ServiceMap = Record<string, unknown>> (opts: RecursivePartial<Libp2pInit<T>>): Libp2pInit<T> {
const libp2pConfig = object({
addresses: validateAddressManagerConfig(opts?.addresses as AddressManagerInit),
connectionManager: validateConnectionManagerConfig(opts?.connectionManager as ConnectionManagerInit)
})

if ((opts?.services) != null) {
// @ts-expect-error until we resolve https://github.com/libp2p/js-libp2p/pull/1762 and have a better way of discovering type dependencies
// eslint-disable-next-line @typescript-eslint/strict-boolean-expressions
if ((opts.services?.kadDHT || opts.services?.relay || opts.services?.ping) && !opts.services.identify) {
throw new Error('identify service is required when using kadDHT, relay, or ping')
}
}

const parsedOpts = libp2pConfig.validateSync(opts)

const resultingOptions: Libp2pInit<T> = mergeOptions(DefaultConfig, parsedOpts)

return resultingOptions
}
maschad marked this conversation as resolved.
Show resolved Hide resolved
12 changes: 12 additions & 0 deletions packages/libp2p/src/config/helpers.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { multiaddr } from '@multiformats/multiaddr'

export const validateMultiaddr = (value: Array<string | undefined> | undefined): boolean => {
value?.forEach((addr) => {
try {
multiaddr(addr)
} catch (err) {
throw new Error(`invalid multiaddr: ${addr}`)
}
})
return true
}
Loading
Loading