Update board catalog generator

[szczys]

Zephyr 3.7 implements a new hardware model that changes the path to each board
definition, and the naming of the yaml files in those definitions. This commit
updates the automatic generation of the Golioth device catalog to work with the
new model.

• Search for board.yml files instead of scraping arch directories (which no
  longer exist).
• Use the first image in each board's .rst file as the image in the device
  catalog.
• Make several attempts to automatically recognize the device-specific yaml
  files. This is a tad tricky as different vendors approach this differently,
  especially when it comes to including or excluding the SOC name from the
  device ID and for multi-core processors like the ESP32 family.
• Use the scripts/device-catalog/custom_images directory to override the image
  used for any board. The filename (excluding the extension) must exactly match
  the board ID. This may be used even when no board image is present in the
  Zephyr tree. The scripts/device-catalog/manually_added directory may still
  be used to add boards to the library, as long as the added board file
  structure matches the new hardware model.

Running this script

rm -rf static/img/boards/
rm -rf docs/firmware/hardware/assets/boards.json 
rm -rf docs/firmware/hardware/6-catalog/continuously-verified/
rm -rf docs/firmware/hardware/6-catalog/verified/
rm -rf docs/firmware/hardware/6-catalog/unverified/
npm run build-device-catalog

resolves https://github.com/golioth/devrel-issue-tracker/issues/418

[szczys]

The build will pass when commands listed above are run. That will remove old generated files and regenerate them. I didn't want to do that before the review because it will bloat the PR with 1.2k files changed

[trond-snekvik]

LGTM!

[trond-snekvik]

If you change this to

Suggested change

	files = fs.readdirSync(dirPath)
	files = fs.readdirSync(dirPath, {withFileTypes: true, recursive: true})

You'll get objects that already tell you which ones are files and directories, and you get recursion out of the box.

Can reduce this entire function to

function getAllBoardPaths(dirPath) {
    return fs.readdirSync(dirPath, { withFileTypes: true, recursive: true})
              .filter(e => e.isFile() && e.name === 'board.yml')
              .map(f => f.path) // the directory path relative to dirPath
}

[szczys]

Worked like a charm, thanks!

[trond-snekvik]

Not really a bug, but worth mentioning that this arrayOfFiles = assignment is redundant, as arrays are mutable references to the same data. All pushes that happen to arrayOfFiles within this call are reflected in the original array.
All these three variations of this line have the same outcome:

arrayOfFiles = getAllBoardPaths(dirPath + "/" + file, arrayOfFiles)
getAllBoardPaths(dirPath + "/" + file, arrayOfFiles)
arrayOfFiles.push(...getAllBoardPaths(dirPath + "/" + file))

It's usually safest to go for the third variant in this case, as passing arrays for manipulation is a common source of bugs, because it's easy to forget that mutations have an effect outside of the current scope. For instance, the following variant of this line looks innocent, but actually adds the entries twice:

arrayOfFiles.push(...getAllBoardPaths(dirPath + "/" + file, arrayOfFiles))

[trond-snekvik]

JS convention for locally repeated behavior like this is to make a function within the current scope:

const getFileIfExists = (yamlName) => {
    const yamlPath = path.join(boardsRoot, board.path, yamlName);
    if (fs.existsSync(yamlPath) return yamlPath
}

let yamlPath = getFileIfExists(`${board.boardId}.yaml`);
if (yamlPath) return yamlPath;

if (board.rev) {
    yamlPath = getFileIfExists(`${board.boardId}_${board.rev.replace(/\./g, '_')}.yaml`);
    if (yamlPath) return yamlPath;
}
// ...

[szczys]

This is much cleaner now, thank you!

[szczys]

This PR is waiting on the release of Zephyr v3.7.0 and the adoption of it by the Golioth Firmware SDK. Once this happens, one more commit should be made:

Before merging:

1. Remove all generated device catalog files

   rm -rf static/img/boards/
   rm docs/firmware/hardware/assets/boards.json 
   rm -rf docs/firmware/hardware/6-catalog/continuously-verified/
   rm -rf docs/firmware/hardware/6-catalog/verified/
   rm -rf docs/firmware/hardware/6-catalog/unverified/
   

2. Generate new files

   npm run build-device-catalog
   

3. Add to git in hopes that unchanged files (images) are recognized as a rename

   git add -u
   git add docs/firmware/hardware/6-catalog/
   git add static/img/boards/
   

4. Commit and push

   git commit -s
   git push --force
   

5. Confirm that tests pass

[github-actions]

Visit the preview URL for this PR (updated for commit c306a69):

https://golioth-docs-dev--pr412-szczys-update-board-0x23f2jf.web.app

_{(expires Sat, 05 Oct 2024 18:03:54 GMT)}

_{🔥 via Firebase Hosting GitHub Action 🌎}

_{Sign: f53b02bdc98ce6f5593931ec4c339aa96bac84df}