*: add invocations to applicationlog #3569

ixje · 2024-09-03T14:48:53Z

Problem

Solution

Implement as extension. Moved the discussion from Dora's backend PR to here

To do

copy arguments to avoid modifications
limit the total number of argument stack items in a single transaction (for safety)
make this a configurable feature
include native contract calls

Do we want to limit the stack item depth (think MaxJSONDepth) or are we content with just limiting the total stack arguments?

AnnaShaleva

A good prototype, but I have several design questions that we should solve before review.

pkg/core/interop/contract/call.go

AnnaShaleva · 2024-09-05T05:12:18Z

@roman-khimov, I think we need some third opinion on these topics.

roman-khimov · 2024-09-05T20:14:36Z

How about System.Runtime.LoadScript calls, btw?

AnnaShaleva · 2024-09-06T05:04:45Z

How about System.Runtime.LoadScript calls

It leads to new execution context creation, thus it's a valid part of invocation tree. But is this information useful in practice? Dynamic invocations are identified by hash160 of the loaded script, as a result user can't get this script because he knows only its hash. But still we may include dynamic invocations into the resulting Invocations tree with some special field like isContractCall: false.

ixje · 2024-10-31T09:42:52Z

Picking this up again. I rebased the branch onto latest master and processed some of the feedback. In particular

use stackitem.Serialize instead of deepcopy and re-use the results when storing the data
make the behaviour configurable through a SaveInvocations config option

Note; It was unclear to me based on #3569 (comment) if I should have made it a tree or keep it flat. I kept it flat for now.

If the feature is enabled the applicationlog output looks as follows

"invocations": [
                    {
                        "contract_hash": "0xd2a4cff31913016155e38e474a2c06d08be276cf",
                        "method": "transfer",
                        "arguments": {
                            "type": "Array",
                            "value": [
                                {
                                    "type": "ByteString",
                                    "value": "krOcd6pg8ptXwXPO2Rfxf9Mhpus="
                                },
                                {
                                    "type": "ByteString",
                                    "value": "AZelPVEEY0csq+FRLl/HJ9cW+Qs="
                                },
                                {
                                    "type": "Integer",
                                    "value": "1000000000000"
                                },
                                {
                                    "type": "Any"
                                }
                            ]
                        },
                        "arguments_count": 4,
                        "is_valid": true
                    }
                ]

and in disabled state it returns

"invocations": []

I'm looking for feedback on the above before taking care of covering System.Runtime.LoadScript calls

ixje · 2024-11-14T14:45:35Z

@AnnaShaleva can this PR also get some review love please

roman-khimov

I don't see any limits for the overall size of saved data. While the feature is optional we still need to protect node from abuse.

pkg/core/state/notification_event.go

pkg/core/interop/contract/call.go

pkg/core/state/notification_event.go

pkg/core/interop/contract/call.go

ixje · 2024-12-01T08:36:47Z

ping

ixje · 2024-12-03T14:41:03Z

I feel there is a lot of resistance against this PR, but I can't really tell why. Other PRs I opened (later than this one, like #3674 and #3677) were swiftly reviewed (multiple times). This one however takes 2-3 weeks per update to get another response despite multiple pings.

Can any of you @AnnaShaleva @roman-khimov elaborate please? If I understand where this resistance is coming from I can perhaps do something about it.

roman-khimov · 2024-12-03T14:47:54Z

@ixje, zero resistance. Sorry, it's just that there are too many things to handle at once. @AnnaShaleva will get back to it soon.

pkg/config/ledger_config.go

pkg/core/interop/context.go

pkg/core/state/notification_event.go

pkg/core/interop/contract/call.go

pkg/core/state/notification_event.go

pkg/core/interop/context.go

pkg/core/state/notification_event.go

AnnaShaleva · 2024-12-06T16:59:46Z

@ixje sorry for the delay. Actually, when we don't need #3569 (comment) and #3569 (comment) to be fixed, there are only a couple of non-critical issues left to be fixed, so the PR is almost ready and looks good.

ixje · 2025-01-02T12:51:27Z

@AnnaShaleva some RPC server tests and TestNEO_CommitteeEvents are failing due to changes requested by you in this comment. If I do write this 1 byte for the length then they pass (with some small updates where necessary).

I've spend some time trying to understand why TestNEO_CommitteeEvents is failing and for some reason the data fetched from the dao does contain the execution events, despite the body of this logic never being called (as per breakpoints not hitting in the debugger)

	if invocLen := len(aer.Invocations); invocLen > 0 {
		w.WriteVarUint(uint64(invocLen))
		for i := range aer.Invocations {
			aer.Invocations[i].EncodeBinaryWithContext(w, sc)
		}
	}

Putting a breakpoint in EncodeBinaryWithContext of ContractInvocation is also not hit. I'm not sure yet what/where I'm overlooking something. Any help/insight there is appreciated.

docs/rpc.md

pkg/core/interop/contract/call.go

pkg/core/state/contract_invocation.go

AnnaShaleva · 2025-01-09T14:33:12Z

pkg/core/state/contract_invocation.go

+	}
+	params, err := stackitem.FromJSONWithTypes(aux.Arguments)
+	if err != nil {
+		return err


You need to handle the case of invalid type ([]byte(fmt.Sprintf("error: %v", err)) set in the marshaller). Try to unmarshal aux.Arguments into string and continue execution flow if unmarshalling is successful (exactly like for state.Execution unmarshalling. Let's add Encode -> Decode -> marshal JSON -> unmarshal JSON unittest for ConstructIvocation structure for the case when argumentsBytes is nil.

I'm starting to wonder if this

if err != nil { item = []byte(fmt.Sprintf(`"error: %v"`, err)) }

in MarhshallJSON() is ever not nil. I'm sure I initially copied it from how NotificationEvent is working. But looking at it now the stackitem.Deserialize(ci.argumentsBytes) call above it uses a deserialization context with allowInvalid: false therefore FromJSONWithTypes should never fail. That logic does not exist for the NotificationEvent MarshallJSON() function.

-edit-
The testcases prove this is unreachable

pkg/core/state/notification_event.go

ixje · 2025-01-22T11:48:26Z

pkg/core/interop/contract/call.go

+			truncated = false
+			argBytes  []byte
+		)
+		if argBytes, err = ic.DAO.GetItemCtx().Serialize(stackitem.NewArray(args), false); err != nil {


I've found that argBytes gets modified by the time EncodeBinaryWithContext of ContractInvocation is called (when persisting to storage). Specifically, the type changes from an Array to a Struct.

If I replace ic.DAO.GetItemCtx() with stackitem.NewSerializationContext() the content of argBytes is preserved. I'm not sure what the implications of this are but wanted to point this out as I find this somewhat worrisome. @AnnaShaleva @roman-khimov

I found this out by a type assertion failure in MarshallJSON() of ContractInvocation, specifically at this line

item, err = stackitem.ToJSONWithTypes(si.(*stackitem.Array))

It's an unexpected behaviour, let me check.

I found this specifically happens when trying to read the following T5 log (it's in block 2872 iirc so you don't have to sync much for it)

curl --location 'http://127.0.0.1:20332' \ --header 'Content-Type: text/plain' \ --data '{ "jsonrpc": "2.0", "method": "getapplicationlog", "params": ["0x2f7b0f78df942212334c0b6c03239a1afc4518aeb4461cb5b77b31aaa9e0c1a5"], "id": 1 } '

Ref. #3569 (comment).

codecov · 2025-01-22T15:53:28Z

Codecov Report

Attention: Patch coverage is 62.39316% with 44 lines in your changes missing coverage. Please review.

Project coverage is 83.02%. Comparing base (e8b8c1a) to head (fa29376).
Report is 216 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/core/state/contract_invocation.go	76.92%	10 Missing and 5 partials ⚠️
pkg/core/state/notification_event.go	20.00%	9 Missing and 3 partials ⚠️
pkg/core/interop/contract/call.go	9.09%	9 Missing and 1 partial ⚠️
pkg/core/blockchain.go	33.33%	3 Missing and 1 partial ⚠️
pkg/core/dao/dao.go	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3569      +/-   ##
==========================================
- Coverage   83.20%   83.02%   -0.18%     
==========================================
  Files         334      336       +2     
  Lines       46488    47005     +517     
==========================================
+ Hits        38681    39028     +347     
- Misses       6234     6378     +144     
- Partials     1573     1599      +26

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

docs/rpc.md

pkg/core/dao/dao_test.go

docs/rpc.md

pkg/core/interop/contract/call.go

pkg/core/state/contract_invocation.go

AnnaShaleva · 2025-01-22T16:18:44Z

pkg/core/state/contract_invocation.go

+// MarshalJSON implements the json.Marshaler interface.
+func (ci ContractInvocation) MarshalJSON() ([]byte, error) {
+	var item []byte
+	if ci.argumentsBytes != nil {


if ci.Arguments == nil && ci.argumentsBytes != nil

AnnaShaleva · 2025-01-22T16:22:26Z

pkg/core/state/contract_invocation.go

+	ci.Hash = aux.Hash
+	ci.ArgumentsCount = aux.ArgumentsCount
+	ci.Truncated = aux.Truncated
+	ci.argumentsBytes = argsBytes


Add ci.Arguments = aux.Arguments.

I explicitly removed this because otherwise MarshalJSON and UnmarshalJSON are not symmetrical. Is that really what we want?

Yes. Because the common usage scenario is constructing response to getapplicationlog which means: decode binary from the DB, marshal it to JSON o nthe server side, then unmarshal from JSON on the client side. Client needs only ci.Arguments, it's not interested in ci.argumentsBytes, so I consider it useless to spend time serializing ci.Arguments in UnmarshalJSON.

pkg/core/state/notification_event.go

AnnaShaleva · 2025-01-22T16:23:18Z

pkg/core/state/notification_event.go

@@ -120,6 +130,9 @@ func (aer *AppExecResult) DecodeBinary(r *io.BinReader) {
 	aer.Stack = arr
 	r.ReadArray(&aer.Events)
 	aer.FaultException = r.ReadString()
+	if aer.VMState&0x80 != 0 {


And reuse this constant here.

👀

neo-go/pkg/encoding/bigint/bigint.go

Line 31 in 673b26f

isNeg := data[size-1]&0x80 != 0

neo-go/pkg/vm/emit/emit.go

Line 57 in 673b26f

if buf[l-1]&0x80 != 0 {

I'm not a fan of this solution, it's harder to maintain. And I treat the case of VM state as kind of special, because originally it's not designed to store additional information in the MSB (and strictly speaking, it can even be any other free bit).

VM and bigint cases are about negative/MSB, it's more obvious from the purpose of this code (especially in the bigint case). This 0x80 however is pure magic and it can be 0x20 or whatever else. Compatibility kludge in its best, we need to remember it. If we're to ever change the DB scheme this will be removed for sure.

pkg/core/state/notification_event.go

AnnaShaleva · 2025-01-23T08:44:11Z

pkg/core/state/contract_invocation.go

+	ci.Hash = aux.Hash
+	ci.ArgumentsCount = aux.ArgumentsCount
+	ci.Truncated = aux.Truncated
+	ci.argumentsBytes = argsBytes


Yes. Because the common usage scenario is constructing response to getapplicationlog which means: decode binary from the DB, marshal it to JSON o nthe server side, then unmarshal from JSON on the client side. Client needs only ci.Arguments, it's not interested in ci.argumentsBytes, so I consider it useless to spend time serializing ci.Arguments in UnmarshalJSON.

AnnaShaleva · 2025-01-23T08:45:11Z

pkg/core/state/notification_event.go

+const (
+	MostSignificantBit          = 0x80
+	ClearMostSignificantBitMask = MostSignificantBit ^ 0xFF


Make them unexported, add documentation.

AnnaShaleva · 2025-01-23T08:56:12Z

pkg/core/state/contract_invocation_test.go

+func TestContractInvocation_JSON_InvalidArguments(t *testing.T) {
+	ci := NewContractInvocation(util.Uint160{}, "fakeMethodCall", nil, 1, false)
+
+	out, err := json.Marshal(&ci)
+	require.NoError(t, err)
+
+	var ci2 ContractInvocation
+	err = json.Unmarshal(out, &ci2)
+	require.NoError(t, err)
+	require.Equal(t, ci, &ci2)
+}
+
+func TestContractInvocation_JSON_ValidArguments(t *testing.T) {


Merge these two tests into one TestContractInvocation_MarshalUnmarshalJSON, create two sub-tests (t.Run), use testserdes.MarshalUnmarshalJSON for non-error case.

AnnaShaleva · 2025-01-23T08:56:42Z

pkg/core/state/contract_invocation_test.go

+func TestContractInvocation_EncodeDecode_InvalidArguments(t *testing.T) {
+	ci := NewContractInvocation(util.Uint160{}, "fakeMethodCall", nil, 1, false)
+
+	w := io.NewBufBinWriter()
+	ci.EncodeBinary(w.BinWriter)
+	require.NoError(t, w.Err)
+
+	ciReader := io.NewBinReaderFromBuf(w.Bytes())
+	ci2 := ContractInvocation{}
+	ci2.DecodeBinary(ciReader)
+	require.Equal(t, ci, &ci2)
+}
+
+func TestContractInvocation_EncodeDecode_ValidArguments(t *testing.T) {


Ditto for these two, and we have testserdes.EncodeDecodeBinary.

AnnaShaleva · 2025-01-23T08:58:09Z

pkg/core/state/contract_invocation.go

+}
+
+// contractInvocationAux is an auxiliary struct for ContractInvocation JSON marshalling.
+type contractInvocationAux struct {


And move this declaration to the top of the file, place it under ContractInvocation declaration. Usually all strucrtures are declared in the start of the file, then methods/functions follow.

AnnaShaleva · 2025-01-23T08:58:30Z

pkg/core/state/contract_invocation.go

+)
+
+// NewContractInvocation returns a new ContractInvocation.
+func NewContractInvocation(hash util.Uint160, method string, argBytes []byte, argCnt uint32, truncated bool) *ContractInvocation {


Move it under types declaration.

AnnaShaleva · 2025-01-23T09:02:21Z

pkg/core/state/contract_invocation.go

+	if b := r.ReadVarBytes(); len(b) > 0 {
+		ci.argumentsBytes = b
+	}


OK, then I have even better suggestion: in EncodeBinary write ci.Truncated firstly, then write ci.Arguments only if !ci.Truncated. In DecodeBinary read ci.Truncated firstly, then use ci.argumentsBytes = r.ReadVarBytes() only if !ci.Truncated.

AnnaShaleva · 2025-01-23T09:11:25Z

pkg/core/interop/contract/call.go

+			truncated bool
+			argBytes  []byte
+		)
+		if argBytes, err = stackitem.NewSerializationContext().Serialize(stackitem.NewArray(args), false); err != nil {


Regarding this: #3569 (comment)

This behaviour is likely caused by the behaviour of (e Element) Array(): args is an underlying content of either VM Array or VM Struct:

neo-go/pkg/vm/stack.go

Lines 96 to 101 in 673b26f

func (e Element) Array() []stackitem.Item {

switch t := e.value.(type) {

case *stackitem.Array:

return t.Value().([]stackitem.Item)

case *stackitem.Struct:

return t.Value().([]stackitem.Item)

When serializing it, you enforce args to be wrapped into array irrespectively of the original type. However, if the old serialization context is used, the information about slice type is stored in the context, hence we've got a struct. The correct behaviour is to preserve the original type of arguments. To solve this problem we need:

argsItm := ic.VM.Estack().Pop() args := argsItm.Array() ... ic.DAO.GetItemCtx().Serialize(argsItm, false); err != nil { ...

It still fails after changing to this

argsItm := ic.VM.Estack().Pop() args := argsItm.Array()

I tried the old one with the

ic.DAO.GetItemCtx().Serialize(stackitem.NewArray(args), false)

and with

ic.DAO.GetItemCtx().Serialize(argsItem.Itm(), false)

In both cases the underlying type changes from Array to Struct

AnnaShaleva · 2025-01-23T09:17:33Z

Everything is almost done, so once finished, please, format commits according to https://github.com/nspcc-dev/.github/blob/master/git.md#commits and add a proper signoff to the commit messages.

ixje · 2025-01-23T10:54:54Z

Everything is almost done, so once finished, please, format commits according to https://github.com/nspcc-dev/.github/blob/master/git.md#commits and add a proper signoff to the commit messages.

I'll wait with the squashing and signoff until everything is really approved. I thought I was close a couple of times before and then the next review requests changes over parts that 'passed' the previous reviews.

AnnaShaleva reviewed Sep 4, 2024

View reviewed changes

ixje added 3 commits October 31, 2024 08:15

Store invocations in application log

d8a0d13

rename params to arguments

04c71f5

make invocation saving configurable, safely handle args

dfd5c9b

ixje force-pushed the applog-invocations branch from f4e91f5 to dfd5c9b Compare October 31, 2024 09:03

re-use serialized arguments for storing

cd7caf5

ixje requested a review from AnnaShaleva October 31, 2024 09:43

ixje marked this pull request as ready for review November 19, 2024 10:26

ixje requested a review from roman-khimov November 19, 2024 10:27

roman-khimov reviewed Nov 20, 2024

View reviewed changes

feedback

e7063a2

ixje requested a review from roman-khimov November 20, 2024 14:17

AnnaShaleva reviewed Dec 5, 2024

View reviewed changes

pkg/core/state/notification_event.go Outdated Show resolved Hide resolved

pkg/core/state/notification_event.go Outdated Show resolved Hide resolved

process feedback

ae3b481

ixje requested a review from AnnaShaleva January 2, 2025 12:51

AnnaShaleva reviewed Jan 9, 2025

View reviewed changes

process feedback

bc4cb3d

ixje commented Jan 22, 2025

View reviewed changes

add better RPC example

5ffb57d

ixje added 2 commits January 22, 2025 13:39

preserve argBytes

90c18b7

use VMState as marker

d1d0921

AnnaShaleva reviewed Jan 22, 2025

View reviewed changes

process feedback

25e93ec

AnnaShaleva reviewed Jan 23, 2025

View reviewed changes

ixje added 5 commits January 23, 2025 10:54

move code around

72caf56

document bit masks consts

4876e70

update encode/decode binary routines, merge binary tests

036e6cb

update marshal/unmarshal routines, merge marshalling tests

7b83ff3

simplify NewContractInvocation

fa29376

ixje requested a review from AnnaShaleva January 23, 2025 10:55

	func (e Element) Array() []stackitem.Item {
	switch t := e.value.(type) {
	case *stackitem.Array:
	return t.Value().([]stackitem.Item)
	case *stackitem.Struct:
	return t.Value().([]stackitem.Item)

*: add invocations to applicationlog #3569

Are you sure you want to change the base?

*: add invocations to applicationlog #3569

Conversation

ixje commented Sep 3, 2024 • edited Loading

Problem

Solution

To do

AnnaShaleva left a comment

Choose a reason for hiding this comment

AnnaShaleva commented Sep 5, 2024

roman-khimov commented Sep 5, 2024

AnnaShaleva commented Sep 6, 2024

ixje commented Oct 31, 2024

ixje commented Nov 14, 2024

roman-khimov left a comment

Choose a reason for hiding this comment

ixje commented Dec 1, 2024

ixje commented Dec 3, 2024

roman-khimov commented Dec 3, 2024

AnnaShaleva commented Dec 6, 2024

ixje commented Jan 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jan 22, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ixje Jan 23, 2025 • edited Loading

Choose a reason for hiding this comment

AnnaShaleva commented Jan 23, 2025

ixje commented Jan 23, 2025

ixje commented Sep 3, 2024 •

edited

Loading

ixje commented Jan 2, 2025 •

edited

Loading

codecov bot commented Jan 22, 2025 •

edited

Loading

ixje Jan 23, 2025 •

edited

Loading