[Tutorial] Dynamic Retrieval Pricing For Storage Provider

[jennijuju]

Retrieval pricing

Lotus allows you to set different strategy to calculate the quoted price of a retrieval deal:

  [Dealmaking.RetrievalPricing]
    Strategy = "external"
    [Dealmaking.RetrievalPricing.Default]
      VerifiedDealsFreeTransfer = true
    [Dealmaking.RetrievalPricing.External]
      Path = "/var/price"

• default: Lotus is shipped with an in-built default pricing policy that offers free retrievals for Verified Unsealed deals. Storage miners can still charge clients for data-transfer if they wish to by turning off free data-transfer for verified deals(VerifiedDealsFreeTransfer) in the Miner config. Miners can also charge clients for Unsealing if they don't have an Unsealed sector file to retrieve the deal payload from.

• external: Storage providers can configure an external pricing script that will be used instead of the default pricing policy to price retrieval deals. The script will be called with the relevant deal parameters and the output quote will be used by Lotus to price deals.

Using the default retrieval pricing policy

• The default pricing policy uses the price configured in the Ask Store (set via the lotus retrieval-deals set-ask CLI command) to price all retrieval deals.
• However, it will not charge for data-transfer if there exists a verified storage deal for the payload being retrieved. This behaviour can be turned off by setting the VerifiedDealsFreeTransfer flag to false in the DealMaking.RetrievalPricing section of the config as shown below.

 [Dealmaking.RetrievalPricing]
    Strategy = "default"
    [Dealmaking.RetrievalPricing.Default]
      VerifiedDealsFreeTransfer = false

• It will also not charge for Unsealing if we already have an Unsealed sector file containing the retrieval payload.

Using the external retrieval pricing policy

• Users can configure an external pricing script(similar to the deal filter mechanism) that takes a json marshalled PricingInput as the input and outputs a json marshalled Ask i.e. the quote. The PricingInput struct is defined as follows:

type PricingInput struct {
	// PayloadCID is the cid of the payload to retrieve.
	PayloadCID cid.Cid
	// PieceCID is the cid of the Piece from which the Payload will be retrieved.
	PieceCID cid.Cid
	// PieceSize is the size of the Piece from which the payload will be retrieved.
	PieceSize abi.UnpaddedPieceSize
	// Client is the peerID of the retrieval client.
	Client peer.ID
	// VerifiedDeal is true if there exists a verified storage deal for the PayloadCID.
	VerifiedDeal bool
	// Unsealed is true if there exists an unsealed sector from which we can retrieve the given payload.
	Unsealed bool
	// CurrentAsk is the ask configured in the ask-store via the `lotus retrieval-deals set-ask` CLI command.
	CurrentAsk Ask
}

The output Ask is defined as:

type Ask struct {
   PricePerByte            abi.TokenAmount
   UnsealPrice             abi.TokenAmount
   PaymentInterval         uint64
   PaymentIntervalIncrease uint64
}

• To use this mode, set the DealMaking.RetrievalPricing strategy to "external" and needs to be given the absolute path of the pricing script as shown below:

[Dealmaking.RetrievalPricing]
    Strategy = "external"
    [Dealmaking.RetrievalPricing.External]
      Path = "<absolute path to the script>"

This example in go sets the ask price of the deals from certain clients to 0.003 FIL/Byte and unseal price to be zero. To use the script:

• Build it by running go build -o retrievalpricing ../fil-dynretprice/main.go, where retrievalpricing is the built script file name.
• Then set the Path of Dealmaking.RetrievalPricing.External to be the absolute path of retrievalpricing.
• Restart your lotus-miner for the script to be effective.

Reference: #6175 (comment) by @aarshkshah1992

[f8-ptrk]

a few questions:

type Ask struct {
   PricePerByte            abi.TokenAmount
   UnsealPrice             abi.TokenAmount
   PaymentInterval         uint64
   PaymentIntervalIncrease uint64
}

• in case of "incomplete" json returned: where are the default values? Are they taken from the lotus side pricing feature or are they hard coded?

https://github.com/raulk/fil-dynretprice/blob/main/main.go as an example for incomplete json returned.

• What are the units for the 4 returned values? FIL/FIL/sec/sec ?

• Is there any way to connect PricingInput to the actual retrieval proposal? I see no way right now to distinguish between proposals if the same client retrieves twice with the same values

[Dealmaking.RetrievalPricing]
    Strategy = "external"

• if set to external: is it exclusive? If not, what is the order of setting the price? external with default lotus impl. as fallback?
• what happens if the external script exits with 1?

[xinaxu]

Looking at the default retrieval pricing, the PaymentInternal and the PaymentInternalIncrease seems to be in bytes and defaults to 1 MiB/1MiB.

[Fatman13]

Do miner get paid only after a successful retrieval? What if client always responds with failed transfer to avoid payments?

[xinaxu]

My guess is that it is after each PaymentInterval.