Implement Dockerfile template verification for internal servicing validation

[NikolaMilosavljevic]

This issue was brought up with #5928 (comment)

Our templates support multiple scenarios for dockerfile generation, single and multi-stage are just one example. With internal testing work (#5928) it becomes critical to ensure that no dockerfile template change is ever regressing any of the supported dockerfile generation scenarios, i.e. internal testing.

Verification should include generation of templates and appropriate pipeline run to validate build and test.

Some options:

• Manual verification of internal testing scenario, as a requirement for any template change
• Automated verification of internal testing scenario
• Automated verification of all supported scenarios

[lbussell]

I am experimenting with "snapshot" testing to validate internal templates, using a library like https://github.com/VerifyTests/Verify. The internal Dockerfiles would still be "checked-in", somewhere in the tests/ directory, but the library would automate generating, comparing/validating, and accepting changes to the Dockerfiles.

If we made changes to templates, the workflow would go something like this:

1. Change templates
2. Run pre-build validation tests locally
3. If internal Dockerfiles have changes, tests will fail
4. Verify supports scrubbing files, so in order to reduce the frequency of diffs we can scrub things like versions and shas that will always change.
5. Use a diff tool to accept/reject changes to Dockerfiles
6. Check-in the new accepted metadata

Without running tests locally, then the workflow could be this:

1. Change templates
2. Submit PR
3. PR validation fails tests. In this case, new baselines will be uploaded as pipeline artifacts
4. Download the new baselines as pipeline artifacts, check them in, and push the new baselines to the PR so that tests can pass

@mthalman, curious to hear your opinion on this approach.

[mthalman]

That seems reasonable. We're obviously restricted from being able to build these Dockerfiles as part of verification within the context of GitHub PRs, so this seems like the next best thing.