feat: handle grant spent amount calculation on payment completion #3674

BlairCurrey · 2025-09-30T13:25:12Z

resolves #3374 (linear)

Some relevant context:

we are changing how amounts spent against grants are calculated. currently in main it just sums all payments on outgoing payment creation. instead we want to keep a record with the running total (OutgoingPaymentGrantSpentAmounts)
amounts tracked are the grant total and interval total (if any)
base branch bc/raf-1031/grant-spent-amounts has half of this implemented: incrementing the totals on outgoing payment creation. This PR is to add the other half: adjustments when outgoing payments are completed

Changes

Changes handleSending and handleFailed to get the latest spent amounts and update as needed. This happens within the lifecycle of the worker (outgoingPayment.processNext).
Adds many tests in lifecycle.test.ts which tests the grant total counting by calling outgoingPaymentService.processNext. This required mocking the paymentMethodHandler.pay method since we dont have another real rafiki to run it against and we need the control of manipulating the response (it fails, it completes partially, completes fully)

Deviations from the issue as stated:

issues only mentions "failure" but really the problem is any needed adjustments when payments are completed. This includes when payments fail, but also includes "partial" payments. That is, a payment which completes for an amount that differs from the amounted anticipated/reserved/held etc. during outgoing payment creation.

Manual Test

I mostly tested using unit tests because I needed control over the time (for intervals) and how pay was resolving. Here is how you can do some manual tests via bruno to verify the happy path.

Basic single payment case.

Start Rafiki
Do bruno Example > Open Payments flow
connect to cloud_nine_wallet_backend database using your preferred method. You can connect directly to the psql shell with docker exec -it rafiki-shared-database-1 su - postgres -c "psql cloud_nine_wallet_backend" and then enter \x to make the output more readable.
Query for the records with select * from "outgoingPaymentGrantSpentAmounts"; (filter by grantId/outgoingPaymentId etc. as needed if not a fresh volume).
The flow should produce 1 record with these amounts matching:

cloud_nine_wallet_backend=# select * from "outgoingPaymentGrantSpentAmounts";
-[ RECORD 1 ]----------------+-------------------------------------
id                           | 567489e0-747c-4a4c-9587-efbc34ff9de9
grantId                      | dde5481d-6447-403f-877a-a11fa7f742c4
outgoingPaymentId            | 6fdc1b38-115d-48b7-a7fa-a994c3a8c1fe
receiveAmountScale           | 2
receiveAmountCode            | USD
paymentReceiveAmountValue    | 100
intervalReceiveAmountValue   | 
grantTotalReceiveAmountValue | 100
debitAmountScale             | 2
debitAmountCode              | USD
paymentDebitAmountValue      | 205
intervalDebitAmountValue     | 
grantTotalDebitAmountValue   | 205
paymentState                 | FUNDING
intervalStart                | 
intervalEnd                  | 
createdAt                    | 2025-10-22 17:48:32.643+00

…mount

…ernal server error

Co-authored-by: Max Kurapov <max@interledger.org>

mkurapov

A few thoughts

mkurapov · 2025-10-20T13:03:15Z

packages/backend/src/open_payments/payment/outgoing/service.ts

            state: OutgoingPaymentState.Funding,
-            grantId
+            grantId,
+            createdAt: new Date()


This is done automatically

(we have table.timestamp('createdAt').defaultTo(knex.fn.now()) in the create outgoing payments table migration)

I had to do this so it can be controlled via fake timers in the tests. Otherwise there was a test that failed. It was some business logic checking mocked time vs. real outgoingPayment.createdAt (since it is set by postgres and uncontrollable by jest's fake timers).

Would it still work if the test patched the outgoing payment after it was created?

Good thought on the patch, but I just tried it and no it doesnt work, because this check that comes right after creating the payment in outgoingPaymentService.create:

if ( payment.walletAddressId !== payment.quote.walletAddressId || payment.quote.expiresAt.getTime() <= payment.createdAt.getTime() ) { throw OutgoingPaymentError.InvalidQuote }

Where the quote's expiresAt is subject to the mocked time but the payment is not. Trying to patch after its created results in a InvalidQuote error.

I think it's probably generally useful to be able to control it from the application level, so it can be mocked like this. Functionally setting it to new Date() should be the same as knex.fn.now().

packages/backend/src/payment-method/handler/service.ts

packages/backend/src/open_payments/payment/outgoing/lifecycle.ts

mkurapov · 2025-10-20T15:18:58Z

packages/backend/src/open_payments/payment/outgoing/lifecycle.ts

+        },
+        'No outgoingPaymentGrantSpentAmounts record found for grant interval'
+      )
+      return


Need to think through this a bit, but I think we would still want to update the OutgoingPaymentGrantSpentAmounts instead of returning early

Like, as a fallback? Is that your thinking?

I guess maybe that makes sense. This should never happen though so IDK how we'd get the correct interval/grant totals. Since it's totally unexpected I dont think we can just assume the interval/grant amounts were 0 or something. I suppose we can sum them all up like we do for the legacy path on the payment create side.

packages/backend/src/open_payments/payment/outgoing/lifecycle.ts

njlie · 2025-10-20T23:20:19Z

packages/backend/src/open_payments/payment/outgoing/service.ts

            state: OutgoingPaymentState.Funding,
-            grantId
+            grantId,
+            createdAt: new Date()


Would it still work if the test patched the outgoing payment after it was created?

amount for spent amount recalc

grant spent amount updates

packages/backend/src/open_payments/payment/outgoing/lifecycle.ts

- turned duplicated logic into shared functions - example fn where revert/update is all in one, but felt it was more complicated

mkurapov · 2025-10-30T21:03:53Z

packages/backend/src/open_payments/payment/outgoing/service.ts

+  )
+
+  await OutgoingPaymentGrantSpentAmounts.query(deps.knex).insert({
+    ...latestPaymentSpentAmounts,


To be safe, let's not have this spread operator and just define all of the fields in-line, otherwise we might include fields that we don't want to update

mkurapov · 2025-10-30T21:14:04Z

packages/backend/src/open_payments/payment/outgoing/lifecycle.test.ts

+              .first()
+
+          // Should have new spent amounts with payment factored out
+          expect(latestGrantSpentAmounts).toMatchObject({


minor, but lets check there are two records that were created here

added a check to make sure they are different records.

mkurapov · 2025-10-30T22:05:14Z

packages/backend/src/payment-method/ilp/service.ts

      },
      'ILP payment completed'
    )
+    return { receive: receipt.amountDelivered, debit: finalDebitAmount }


I think we should leave a comment here explaining the behaviour why not the receipt.amountSent

at this point I think its just easier and more clear to just remove the debit amount from the return type. updated w/ that.

…ions

BlairCurrey and others added 18 commits April 28, 2025 14:25

feat: calculate grant spent amounts from new table

9dbcfbe

chore: linting

d5f0f18

fix: revise grant spent amounts calc to track interval

511f66d

fix: throw better error, dont inti interval amounts unless they exist

6a618db

fix: linter, wip commented out code

bda9118

refactor(backend): rename interval statuses

ab8097d

fix: ensure correct amount (interval/total) is used for grant spent a…

9395127

…mount

Merge branch 'bc/raf-1031/grant-spent-amounts' into blair/raf-1034

4ba90ea

test(backend): grant spent amounts

e5ee133

fix(backend): make logic more typesafe, avoid having to defend w/ int…

934e9f6

…ernal server error

test(backend): legacy grant calculations

cf61c42

test(backend): classify payment interval fn

eff27b1

test(backend): rm commented-out classify payment interval case

c42fc44

refactor(backend): improve semantics of interval helper fn

5f9630e

test(backend): add non-interval cases to create op

6a11e97

Apply suggestions from code review

4773e69

Co-authored-by: Max Kurapov <max@interledger.org>

Merge branch 'bc/raf-1031/grant-spent-amounts' into blair/raf-1036

add2084

feat: partially handled grant spent amount scenarios on settle

55333ac

github-actions bot added type: tests Testing related pkg: backend Changes in the backend package. type: source Changes business logic labels Sep 30, 2025

BlairCurrey added 8 commits October 3, 2025 14:08

feat: handle non-interval partial settleamount cases

2086256

test(backend): failure grant spent amount case for sucessive payment

c63e1f9

test(backend): some grant calc interval cases

8dda943

test(backend): counting grant payments across interval boundaries

e44178a

chore(backend): rm commented out test

7e9cd30

test(backend): new grant spent amount on payment completion

500e1c7

test(backend): improve interval grant calc test to show summation

f42090c

test(backend): failure edge case

138ab62

BlairCurrey changed the title ~~Blair/raf 1036~~ feat: handle grant spent amount calculation on payment completion Oct 7, 2025

mkurapov reviewed Oct 20, 2025

View reviewed changes

BlairCurrey changed the base branch from bc/raf-1031/grant-spent-amounts to main October 20, 2025 19:14

BlairCurrey changed the base branch from main to bc/raf-1031/grant-spent-amounts October 20, 2025 19:15

BlairCurrey requested review from cozminu, koekiebox, njlie, oana-lolea and sanducb October 20, 2025 20:41

njlie reviewed Oct 20, 2025

View reviewed changes

BlairCurrey marked this pull request as ready for review October 21, 2025 14:06

BlairCurrey added 3 commits October 21, 2025 13:43

test(backend): payment retries do not add additional spent record

bb0f2c5

fix(backend): use correct debit

2e901c3

amount for spent amount recalc

fix(backend): set correct payment state on

066c9c8

grant spent amount updates

mkurapov reviewed Oct 27, 2025

View reviewed changes

BlairCurrey added 9 commits October 27, 2025 23:31

refactor(backend): dedupe revert/handle grant spent amounts

87216fd

- turned duplicated logic into shared functions - example fn where revert/update is all in one, but felt it was more complicated

chore: rm unused fn

aa3b0c0

test(backend): fix ilp pay return expectations

a95327a

refactor(backend): move revert/update grant spent amounts to op service

22c47a2

chore: rm unused import

53314da

fix: handle grant spent amounts on payment cancellation

6d9433d

chore: cleanup commented out code

5c9606c

chore: rm submodule added in error

80dee6b

chore(backend): add error logs

77f6084

BlairCurrey requested a review from mkurapov October 30, 2025 17:08

mkurapov reviewed Oct 30, 2025

View reviewed changes

BlairCurrey added 3 commits November 1, 2025 13:07

fix(backend): explicit grant spent amount formation, more test assert…

1b55ff2

…ions

refactor(backend): only return receive amt from .pay

61eb9e3

fix(backend): build error

a6f3f2b

BlairCurrey merged commit 1be38b5 into bc/raf-1031/grant-spent-amounts Nov 1, 2025
26 of 38 checks passed

BlairCurrey deleted the blair/raf-1036 branch November 1, 2025 18:14

feat: handle grant spent amount calculation on payment completion #3674

feat: handle grant spent amount calculation on payment completion #3674

Uh oh!

Conversation

BlairCurrey commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Some relevant context:

Changes

Deviations from the issue as stated:

Manual Test

Basic single payment case.

Uh oh!

mkurapov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BlairCurrey Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BlairCurrey Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BlairCurrey Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BlairCurrey commented Sep 30, 2025 •

edited

Loading

BlairCurrey Oct 20, 2025 •

edited

Loading

BlairCurrey Oct 22, 2025 •

edited

Loading

BlairCurrey Oct 22, 2025 •

edited

Loading