Guidance on meta-qcom-extras build workflow

[abinjose-qualcomm]

During our build guide review, @ricardosalveti noted that kas users expect to run the build by cloning only one repository and not execute any additional scripts. We wanted to have a discussion on the correct way to build meta-qcom-extras.

Current Workflow:

A kas fragment is hosted in meta-qcom-extras. This file links to the local path where meta-qcom-extras is cloned and sets up the environment variables needed by the layer to compile.

# meta-qcom-extras/ci/extras.yml
header:
  version: 14

repos:
  meta-qcom-extras:
    branch: main
    path: /path/to/extras

env:
  CUST_ID: '<CUST_ID>'
  FWZIP_PATH: <FWZIP_PATH>

During build, the meta-qcom-extras kas fragment is populated and used to figure out meta-qcom-extras repo information.

git clone https://github.com/qualcomm-linux/meta-qcom
git clone https://github.com/qualcomm-linux/meta-qcom-extras

# populate meta-qcom-extras kas fragment with FWZIP_PATH & CUST_ID
export FWZIP_PATH="/path/to/local/binaries"
export CUST_ID="<customer-id>"
meta-qcom-extras/setup_extras_config.sh
cp meta-qcom-extras/ci/extras.yml meta-qcom/ci/extras.yml

kas build meta-qcom/ci/qcs9100-ride-sx.yml:meta-qcom/ci/qcom-distro.yml:meta-qcom/ci/extras.yml

Proposed Workflow:

The suggestion by @rsalveti was to host the extras.yml kas fragment in meta-qcom.

# meta-qcom/ci/extras.yml
header:
  version: 14

repos:
  meta-qcom-extras:
    url: https://github.com/qualcomm-linux/meta-qcom-extras
    branch: main

env:
  CUST_ID: '<CUST_ID>'
  FWZIP_PATH: <FWZIP_PATH>

During builds, the meta-qcom-extras layer can now be automatically synced by kas.

git clone https://github.com/qualcomm-linux/meta-qcom

# we still need some mechanism to expose environment variables required 
# by meta-qcom-extras to bitbake.
export FWZIP_PATH="/path/to/local/binaries"
export CUST_ID="<customer-id>"
meta-qcom/setup_extras_config.sh

kas build meta-qcom/ci/qcs9100-ride-sx.yml:meta-qcom/ci/qcom-distro.yml:meta-qcom/ci/extras.yml

One concern with this approach is that meta-qcom-extras is hosted on GitHub with the development happening in private. However, it will be distributed to customers via Chipcode. So, we would need some mechanism to switch the meta-qcom-extras url for user builds.

Any other insights or suggestions on this problem would be appreciated.

[ricardosalveti]

We are having similar discussions with the duplication of kas files for meta-qcom and meta-qcom-distro (part of qualcomm-linux/meta-qcom-distro#139), so we should also bring meta-qcom-extras to the table (@quaresmajose).

I'm ok to have the kas fragment from a private repository into a public one if that provides a better user experience, as in this case meta-qcom-extras is also an officially supported repository (part of Qualcomm Linux), but be good to see what @ndechesne / @lumag / @quaresmajose / @koenkooi / @vkraleti might think of that.

[lumag]

Please use local.conf instead of environment settings.

[koenkooi]

We are having similar discussions with the duplication of kas files for meta-qcom and meta-qcom-distro (part of qualcomm-linux/meta-qcom-distro#139), so we should also bring meta-qcom-extras to the table (@quaresmajose).

I'm ok to have the kas fragment from a private repository into a public one if that provides a better user experience, as in this case meta-qcom-extras is also an officially supported repository (part of Qualcomm Linux), but be good to see what @ndechesne / @lumag / @quaresmajose / @koenkooi / @vkraleti might think of that.

As long as users/customers can both fetch and build everything listed in those configs themselves, I'm fine with it. We don't have a better solution, yet.

[koenkooi]

Please use local.conf instead of environment settings.

And use those sparingly, KAS configs shouldn't include necessary config changes, since KAS isn't the only tool used to consume OE layers. If there are required changes that you don't want users to manually add, there's layer.conf for that or a specific DISTRO.conf.

[ricardosalveti]

Right, my main point here is about the kas file itself, I agree that the customization of the CUST_ID / FWZIP_PATH variables should be done in a different way.

[quaresmajose]

The problem here is the same as with meta-qcom/meta-qcom-distro. The big issue is we can't use kas config spread by different repos in an easy way. Using the command line it's not possible, and only by duplicating the kas configuration files can we circumvent this limitation.

Since meta-qcom is a dependency of meta-qcom-extras, it should be the last to include the first. Perhaps the meta-qcom-extras layer should, as a reference example, contain the necessary kas configuration to use BSP/meta-qcom. Users started from this reference, then implemented their own kas configurations in their layers with the necessary adaptations. In reality, our users will have many difficulties in using our kas fragments in their products due to the limitations already mentioned.

Having the meta-qcom-extras kas configuration in meta-qcom won't work without some setup_extras_config.sh helper because the layer is proprietary, so having everything behind the wall makes more sense in my opinion.