Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rpc-compat: Run speccheck on non-deterministic tests #640

Open
lightclient opened this issue Aug 22, 2022 · 4 comments
Open

rpc-compat: Run speccheck on non-deterministic tests #640

lightclient opened this issue Aug 22, 2022 · 4 comments

Comments

@lightclient
Copy link
Member

Some methods are not deterministic across clients. For example, eth_gasPrice returns as gasPrice in wei that users could use for a transaction to be included quickly. This is subjective based on client so a static test case doesn't make sense. However, we know what the shape the request and response should look like. So it would make sense to make the request anyways and verify the round trip against the spec.

I've written a tool speccheck which more-or-less does this. However it doesn't yet have the ability to verify round-trips that aren't written to file in the format of the test fixtures. So it would need to be librarified to a degree and then imported into rpc-compat and run against certain non-deterministic endpoints (maybe even all of them!).

@fjl
Copy link
Collaborator

fjl commented Jan 30, 2024

I'm adding a speconly flag in ethereum/execution-apis#520 so we can at least mark the tests where this kind of checking should occur.

@lightclient
Copy link
Member Author

This is a really nice idea, I am happy to begin using it.

@fjl
Copy link
Collaborator

fjl commented Jan 30, 2024

I'm working on it right now!

@fjl
Copy link
Collaborator

fjl commented Feb 1, 2024

Parsing of the flag is now implemented in the simulator, but it's unclear to me how to proceed with adding speccheck. I don't want to depend on rpctestgen from the simulator, it defeats the purpose of having the generator and test output separated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants