ci: Refactor scripts #60

febo · 2025-02-06T14:46:31Z

Problem

CI currently has limited checks and the publish.yml workflow is limited to a couple of crates.

Solution

Refactor the CI. This involved using a new set of supporting scripts, which can also be used from the command-line:

audit: audit dependencies of the workspace
build: build a crate with build-sbf (useful to detect issues with code under the target = solana configuration
clippy: run clippy
doc: linting of doc sections
format: run cargo fmt
hack: linting of feature powerset
lint: execute clippy && doc && hack in sequence
semver: semantic versioning validation
test: run cargo test

CI make use of these scripts to run:

audit
For each crate of the repository, it executes:
- format
- clippy
- doc
- hack
- build
- test

The publish.yml workflow was updated to work with any crate of the reposiroty and incorporates semver-checks based on the version level. If the version level selected for publishing violates semantic versioning, the publishing fails. For example, if the code to be published under a patch version contains breaking changes, the semver-check will fail and the publish workflow will abort.

Note

The code in main is currently failing the doc linting - #64 fixes these issues.

febo · 2025-02-08T14:51:14Z

Just realised that the "filter" is not as precise as it should be. For example, when the main pinocchio crate changes, all crates that depend on it should also go through the CI checks – this is not happening with the current filter.

Update: The conditional filter was removed.

joncinque

Nothing show-stopping, mostly small questions -- looks great!

joncinque · 2025-02-10T22:55:06Z

.github/workflows/main.yml

+      - name: build-sbf
+        run: pnpm build ${{ matrix.member }}


nit: How about calling the script build-sbf to make things clearer?

Yep, definitely clearer.

joncinque · 2025-02-10T22:56:46Z

.github/workflows/publish.yml

+      - name: Set Git Author
+        run: |
+          git config --global user.email "github-actions@github.com"
+          git config --global user.name "github-actions"


Sorry for fish brain: this is used so that the github-actions bot doesn't appear as a contributor, correct? Why was this a problem again?

Yes, using these labels, the "github-actions" does not appear as a contributor. It does not solve a particular problem, it is a cosmetic thing. Happy to revert the change.

joncinque · 2025-02-10T22:58:17Z

Cargo.toml

@@ -17,5 +17,14 @@ repository = "https://github.com/anza-xyz/pinocchio"

 [workspace.dependencies]
 five8_const = "0.1.3"
-pinocchio = { path = "sdk/pinocchio", version = ">= 0.6, <= 0.7" }
-pinocchio-pubkey = { path = "sdk/pubkey", version = "0.2.1" }
+pinocchio = { path = "sdk/pinocchio", version = ">= 0.6" }


Why is the version specified like that?

This is probably me not using the workspace dependencies correctly. I was trying to avoid bumping dependencies on other packages unnecessarily. For example, if pinocchio-pubkey can work with multiple versions of pinocchio and pinocchio has been bumped to 0.7, I still want to specify that pinocchio-pubkey works with anything from >= 0.6.

But perhaps this is not the correct way to do that.

joncinque · 2025-02-10T22:58:59Z

package.json

+  "private": true,
+  "scripts": {
+    "audit": "tsx ./scripts/audit.mjs",
+    "build": "tsx ./scripts/build.mjs",


nit: mentioned earlier, but build-sbf would probably be clearer

joncinque · 2025-02-10T23:01:28Z

scripts/build.mts

+
+const [folder, ...args] = cliArguments();
+
+const buildArgs = [...args, '--', '--all-targets', '--all-features'];


I'm surprised to see --all-targets here -- I tried building a couple of programs with cargo build-sbf -- --all-targets and it fails because some dependencies that are incompatible with the chain are probably brought in.

Is that what you're trying to test here? It's pretty cool!

Yeah, it is a sanity check.

joncinque · 2025-02-10T23:12:44Z

.github/workflows/publish.yml


      - name: Check semver
-        uses: obi1kenobi/cargo-semver-checks-action@v2
+        run: |
+          pnpm semver ${{ inputs.crate }} --release-type ${{ inputs.level }}


That's really neat, I didn't know it could do that!

joncinque · 2025-02-10T23:14:17Z

.github/workflows/publish.yml

+      - name: Push Commit and Tag
+        if: github.event.inputs.dry_run != 'true'
+        run: git push origin --follow-tags


It's up to you, but I prefer to use the functionality from cargo release to tag and push, ie: https://github.com/solana-program/token-2022/blob/a46685f457db8e5c68d7f43e1c032500b7cd258b/scripts/rust/publish.mjs#L24

Oh, that is better, I will change it so it does the same thing.

joncinque · 2025-02-10T23:15:32Z

scripts/publish.mts

+// Soft reset the last commit so we can create our own commit and tag.
+await $`git reset --soft HEAD~1`;
+
+// Commit the new version.
+await $`git commit -am "Publish ${crate} v${version}"`;
+
+// Tag the new version.
+await $`git tag -a ${crate}@v${version} -m "${crate} v${version}"`;


As flagged earlier, if you decide to update the flags to cargo release, you don't need to do any of this: https://github.com/solana-program/token-2022/blob/a46685f457db8e5c68d7f43e1c032500b7cd258b/scripts/rust/publish.mjs#L24

Be careful to specify two curly braces around {{version}} since that actually interpolates the version that the repo will be bumped to

febo added 2 commits February 6, 2025 14:45

[wip]: Add new scripts

32034fc

[wip]: Use matric strategy

6be0456

febo force-pushed the febo/refactor-ci branch from c2ad80e to 6be0456 Compare February 6, 2025 14:47

febo added 4 commits February 6, 2025 14:50

[wip]: Add setup action

9455e7d

[wip]: Tweak output

6492a14

[wip]: Debug output

7d2e20c

[wip]: Fix members parsing

db04746

febo force-pushed the febo/refactor-ci branch from 8bb36fb to 20e4cfb Compare February 6, 2025 16:17

febo added 5 commits February 6, 2025 18:06

[wip]: Fix matrix strategy

32e6669

[wip]: All process steps

9542414

[wip]: Add toolchains

d7c6e2a

[wip]: Add CI env variables

ec01d10

[wip]: Remove nothrow

4f7ad78

febo force-pushed the febo/refactor-ci branch from 20e4cfb to 4f7ad78 Compare February 6, 2025 18:07

febo added 15 commits February 6, 2025 21:07

[wip]: Filter changes

4bf599e

[wip]: Add audit step

e64f87f

[wip]: Update names

d376d52

[wip]: Add semver checks

62d045b

[wip]: Add publish workflow

233b5de

[wip]: Debug semver check

a1de716

[wip]: Refactor publish workflow

bc3cc18

[wip]: Tweaks

3bf2d9f

[wip]: Fix set version

0bc005f

[wip]: Install cargo-edit

ab956d7

[wip]: Refactor

84b19e2

[wip]: Fix ci variables

4693995

[wip]: Fix import

343d9d3

[wip]: Install solana

503f35b

[wip]: Fix commands

5982943

febo marked this pull request as ready for review February 7, 2025 23:03

febo requested a review from joncinque February 7, 2025 23:06

Fix formatting

7db5221

febo added 2 commits February 8, 2025 18:58

Remove detect changes step

45cecf0

Relax clippy check

3a7dc55

joncinque reviewed Feb 10, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: Refactor scripts #60

ci: Refactor scripts #60

febo commented Feb 6, 2025 •

edited

Loading

febo commented Feb 8, 2025 •

edited

Loading

joncinque left a comment

joncinque Feb 10, 2025

febo Feb 11, 2025

joncinque Feb 10, 2025

febo Feb 11, 2025

joncinque Feb 10, 2025

febo Feb 11, 2025 •

edited

Loading

joncinque Feb 10, 2025

joncinque Feb 10, 2025

febo Feb 11, 2025

joncinque Feb 10, 2025

joncinque Feb 10, 2025

febo Feb 11, 2025

joncinque Feb 10, 2025


		const [folder, ...args] = cliArguments();

		const buildArgs = [...args, '--', '--all-targets', '--all-features'];

ci: Refactor scripts #60

Are you sure you want to change the base?

ci: Refactor scripts #60

Conversation

febo commented Feb 6, 2025 • edited Loading

Problem

Solution

febo commented Feb 8, 2025 • edited Loading

joncinque left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

febo Feb 11, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

febo commented Feb 6, 2025 •

edited

Loading

febo commented Feb 8, 2025 •

edited

Loading

febo Feb 11, 2025 •

edited

Loading