Add ref-spec matching of spaceros repository (#91).

[xfiderek]

Description

This PR fixes #91, by matching spaceros repo branch to current local branch during build. Assuming that current local branch is feature/foo, the behavior is as follows:

• If SPACEROS_GIT_REF is not supplied by user via CLI (default behavior):
  • If feature/foo exists in spaceros repo, ros2.repos from that branch will be used
  • Otherwise file will be cloned from main branch
• If user overrides SPACEROS_GIT_REF via CLI
  • If that branch exists in spaceros repo, that branch will be used
  • otherwise earthly returns error

It is also now possible to specify different url for spaceros git repository, which can be useful in testing scenarios, when contributors are forking spaceros repo.
If SPACEROS_REPO_URL is not supplied by user, then we use wget to clone ros2.repos file from spaceros repo. Otherwise we use Earthly's GIT CLONE command to get the file (see discussion below).

Check updated readme file for more details.

Summary of changes

• Added logic described above to Earthfile
• Updated README.md

How changes were tested

I have developed python file to test the changes: https://gist.github.com/xfiderek/a697dd33739c035ac6b2b1d6cdee8af1

[Bckempa]

Already way better than what we had and a great addition to the readme as well.

I've got a few questions but none are blockers so I'll approve after I finish checking out my local test build, but I'll hold off on hitting the merge button until you've had a chance to respond.

[Bckempa]

Annoying we need a full clone for this but I understand why - literally the only thing I miss about SVN.

Can we at least make a third case that detects if we're just using the main branch of the default repo and uses the githubusercontent CDN to grab the file? I guess a more complete solution would be checking the url for common host platforms like GitHub and GitLab which expose files directly under a specific URL but that might be out of scope for this issue.

[xfiderek]

These are good points. Let me break it down:

• "Can we at least make a third case that detects if we're just using the main branch of the default repo and uses the githubusercontent CDN to grab the file?" - For default spaceros repo, I would even go one step further and say that we should use CDN whenever SPACEROS_REPO_URL="https://github.com/space-ros/space-ros.git", regardless of branch. I think that it is important, because I can imagine that if we have multiple spaceros distros, we would rarely pull from main (you would rather pull from specific tag). There is just one caveat with this approach.
• "I guess a more complete solution would be checking the url for common host platforms like GitHub and GitLab which expose files directly under a specific URL but that might be out of scope for this issue." -> yes, but here we would ideally need to handle cases when such repos are private. Earthly's GIT CLONE handles that automagically (check earthly docs)

In summary, I would refactor that so that we use CDN whenever we pull from default spaceros repo, and use GIT CLONE otherwise. I will document that in the code so that its clear for future generations :D.

Sounds good to you @Bckempa?

Update: I've just discovered a corner case. In case the branch becomes available only after first run of Earthly, it won't pull new repos as results of "$(git ls-remote ${SPACEROS_REPO_URL} ${SPACEROS_GIT_REF})" are cached across runs. Thats where GIT CLONE command shines because it checks whether there was a new commit added and reruns the pipeline if so. Let me think about the easiest solution that will mitigate that corner case

[Bckempa]

Yep, you got everything I was going for and more. If this too big and needs to be split out to another ticket we can do that too.

[xfiderek]

No worries, I figured it out, I will post the PR today in the evening

[Bckempa]

Not a huge deal, but while I see it executing the feedback line:

+sources | *cached* --> RUN echo "cloning branch '${GIT_BRANCH}' from spaceros repo"

I don't see the actual echo with the substituted output. Can we find a better way to make sure it's clear in logs what branch was used?

Either way, seems to work though earthly's caching is so near magical that I want to play with it more before we merge, but you got my green checkmark.

[Bckempa]

Note that the space_robots CI failure is expected due to #87

[xfiderek]

Not a huge deal, but while I see it executing the feedback line:

+sources | *cached* --> RUN echo "cloning branch '${GIT_BRANCH}' from spaceros repo"

I don't see the actual echo with the substituted output. Can we find a better way to make sure it's clear in logs what branch was used?

Either way, seems to work though earthly's caching is so near magical that I want to play with it more before we merge, but you got my green checkmark.

I think i will just remove these echo statements as they bring unnecessary noise. I could not find any way to log something to stdout using earthly. Is it ok with you?

[xfiderek]

Ok @Bckempa, now it should work as expected. Please take a look again.
To make sure that corner cases are covered, I ended up writing python file to make sure that all works after modifications. The file is available here. I also added you to the repository, and as soon as you accept you will be able to run this file locally.

Summary of behavior
Assuming that we are on feature/foo branch:

• If SPACEROS_GIT_REF is not supplied by user via CLI (default behavior):
  • If feature/foo exists in spaceros repo, ros2.repos file will be cloned from feature/foo
  • Otherwise file will be cloned from main branch
• If user overrides SPACEROS_GIT_REF via CLI
  • If that branch exists in spaceros repo, the supplied branch will be used
  • otherwise earthly returns error

Additionally, if SPACEROS_REPO_URL is not supplied by user, then we use wget to clone ros2.repos file from spaceros repo. Otherwise we use Earthly's GIT CLONE command to get the file (see discussion above).

[xfiderek]

As we want to re-run git ls-remote every time, we can no longer keep these args at the top of the file

[EzraBrooks]

looks like this should be good to go after a rebase

[xfiderek]

Rebase applied