Enable AUTHORING UX approval tests creation

[DavidKarlas]

Background

Today there is no good story for template authors to test their templates and ensure they work as intended after doing changes and environment around them changes(.NET Framework, Template Engine...)

We have https://github.com/dotnet/templating/tree/main/tools/ProjectTestRunner but its pretty hard to understand and navigate for novice template author, also tooling is not compiled as dotnet global tool to just use...
We run tests in our repo using Process.Start("dotnet new console) and than check output see example here.. Again, not very good way for template author to run/maintain tests.

Outcomes

This enables template development inner loop. We want to support approval tests, which means template author would do:

1. dotnet new console once to create initial content
2. On test run TemplateEngine will provide ability to run tests that compare content from step 1) with what it would generate, if change is intentional author re-runs step 1) and commits to git changes.

Justification

• Customer impact - 1st party customers has easy tools to test their templates (popular request)
• Engineering impact - automated testing contribute to less bugs - automated testing allows to reduce amount of manual testing - teams don't need to invent and support own tooling for testing

Prerequisite

What needs to be solved how to handle random values like PortNumber or GUIDs...

Subtasks

Investigations:

• Get stats on usage of nondeterministic generators (Guid, Now, Port, Random) - @vlada-shubina
• Investigate ways of usage of XUnit Verifier so that multiple verification can be performed and reported (even if multiple are failing)
  • Verify.Net doesn't support verification of multiple files at the moment.
  • Simon is considering to implement it in near future
  • We stick to 1-by-1 file comparison at the moment
• Go through CommonTemplatesTests to access what functionality we'll need from the test framework in order to transform and adopt these tests.
  • stdout and stderr content comparison
  • content regex matching
  • content substring matching, absence of patterns
  • newlines normalization
  • custom content checking (xml parsing)
• Investigate options to programaticaly change dotnet sdk version to run a sdk tool (as a fallback we can programatically create and then discard global.json)
  We will leverage global.json for this - simplified approach:

ren global.json global.json.bak
dotnet new globaljson --sdk-version <version>
ren global.json.bak global.json

Subtasks for MVP (not to be exposed to customers):

• Generalize and repurpose Microsoft.TemplateEngine.TemplateLocalizer as templates authoring toolset. Packaged as nuget - @vlada-shubina
• Define configuration model for a single test case ({template to be tested; dotnet sdk version; parameter values; approvals location}). Create System.CommandLine Parser transforming CLI arguments to this configuration model
• Verification logic module (the API and actual logic doesn't have to be polished for first version) - @JanKrivanek
• Add programatic way of simple scrubbing and/or replacing keyed by files.
• Transform and onboard CommonTemplatesTests to the new framework Template verifier adoption in CommonTemplateTests sdk#28707

V2 (preparation for customers exposed toolset):

• Define Verification module API. CLI and MSBuild interfaces should call the logic through the API
• Extract external process wrapping logic (Command) from Microsoft.DotNet.Cli.Utils or find another utility for wrapping CLI processes - and get rid of copied code within the Microsoft.TemplateEngine.Authoring.TemplateVerifier. (this task might be joined with Consider dropping dependency to Microsoft.DotNet.Cli.Utils. #5296)
• Support batch execution logic for multiple test cases (probably configured by files)
• Support filter/ignore lists (with defualt behavior that should suffice in most common cases) - e.g. to be able to ignore images, bin/* outputs etc.
• Support for not installed templates (arbitrary location on disk)
• Support for switching sdk versions
• Documenting API and CLI in docs/wiki
• Simplify and unify working with paths (include/exclude patterns, paths given in custom scruber and custom verifier etc.) - tooling should be permissive and be able to accept mix-n-matches of dir separator chars, plus it should have settings for enforcing separator char used in path it's passing out (custom scrubber and verifier). Paths passed out should probably be passed as custom type - allowing to fetch relative path (to template root or test root), absolute path, paths with custom spearators
• Telemetry opt out in our integration tests (currently: https://github.com/dotnet/sdk/blob/main/src/Tests/Microsoft.NET.TestFramework/Commands/DotnetNewCommand.cs#L20, possibilities: explicit set of env in integration set fixture; or ability to inject env into instantiatot process via API)

Next iterations (ideally part of the first customer facing version):

• Review (and adjust if needed) signing of tooling imposed by source build - is it required for shipping?
• Rewrite more snapshots-based integration test for templating instantiation in sdk to use the tooling
• Add telemetry
• Implement context detection and extraction for nondeterministic generators handling (so e.g. for Port generator, the logic should be able to detect the resulting value in the generated output and then process the output by replacing all instances of the generator being used).
• Add Template Validator as another tool in the authoring toolset. Implement just a sample of most importatnt validations - [ ] Create MSBuild Task version of the Template Validator
  • Design and use continuable errors during validation - so that as much errors as possible cna be reported during single run (while not reporting nonsense issues caused by inconsistent data detected in previous steps).
• Investigate, Design and implement deterministic mode for Macros (and hence generators): macros 2.0 #5223
• Build in property based testing

[RussKie]

@vlada-shubina @DavidKarlas a number of repos are starting to create .NET 7 templates (e.g, dotnet/winforms#6206), and having a test infra for templates could be very helpful.

[vlada-shubina]

This feature will be useful for testing: #3418 to avoid the need of using custom settings location and installing the template.

[JanKrivanek]

Based on brainstroming session with @vlada-shubina those are the tasks we came up with:

Investigations:

• Get stats on usage of nondeterministic generators (Guid, Now, Port, Random) - @vlada-shubina
• Investigate ways of usage of XUnit Verifier so that multiple verification can be performed and reported (even if multiple are failing)
  • Verify.Net doesn't support verification of multiple files at the moment.
  • Simon is considering to implement it in near future
  • We stick to 1-by-1 file comparison at the moment
• Go through CommonTemplatesTests to access what functionality we'll need from the test framework in order to transform and adopt these tests.
  • stdout and stderr content comparison
  • content regex matching
  • content substring matching, absence of patterns
  • newlines normalization
  • custom content checking (xml parsing)
• Investigate options to programaticaly change dotnet sdk version to run a sdk tool (as a fallback we can programatically create and then discard global.json)
  We will leverage global.json for this - simplified approach:

ren global.json global.json.bak
dotnet new globaljson --sdk-version <version>
ren global.json.bak global.json

Subtasks for MVP (not to be exposed to customers):

• Generalize and repurpose Microsoft.TemplateEngine.TemplateLocalizer as templates authoring toolset. Packaged as nuget - @vlada-shubina
• Define configuration model for a single test case ({template to be tested; dotnet sdk version; parameter values; approvals location}). Create System.CommandLine Parser transforming CLI arguments to this configuration model
• Verification logic module (the API and actual logic doesn't have to be polished for first version) - @JanKrivanek
• Add programatic way of simple scrubbing and/or replacing keyed by files.
• Transform and onboard CommonTemplatesTests to the new framework

V2 (preparation for customers exposed toolset):

• Define Verification module API. CLI and MSBuild interfaces should call the logic through the API
• (In Progress) Implement context detection and extraction for nondeterministic generators handling (so e.g. for Port generator, the logic should be able to detect the resulting value in the generated output and then process the output by replacing all instances of the generator being used).
• Support batch execution logic for multiple test cases (probably configured by files)
• Support filter/ignore lists (with defualt behavior that should suffice in most common cases) - e.g. to be able to ignore images, bin/* outputs etc.

Next iterations (ideally part of the first customer facing version):

• Add telemetry
• Add Template Validator as another tool in the authoring toolset. Implement just a sample of most importatnt validations (more comprehensive list: Authoring tools: templates & template packages validation #2623)
  • Create MSBuild Task version of the Template Validator
  • Design and use continuable errors during validation - so that as much errors as possible cna be reported during single run (while not reporting nonsense issues caused by inconsistent data detected in previous steps).
• Investigate, Design and implement deterministic mode for Macros (and hence generators)

[RussKie]

Great plan!

• Investigate ways of usage of XUnit Verifier so that multiple verification can be performed and reported (even if multiple are failing)

This point sounds strange... Perhaps I don't quite understand the intent here, could you elaborate on this please?

[vlada-shubina]

We plan to use Verify.NET to build the framework. We need to investigate if it can do verification of multiple files out of the box. So we far we are using it only for single file\object validation.

[RussKie]

We use Verify in Windows Forms repos quite a bit, and I don't think it's possible to verify multiple files simultaneously, such verifications need to be serialized. That is, how do you present multiple failures in a diff tool?
This is how we verify multiple files in a single test:

    protected async Task VerifyAsync(
        IVisualStudioDocument mainFile,
        IVisualStudioDocument codeBehindFile,
        [CallerMemberName] string testMethodName = "")
    {
        var (TestMethodName, AdditionalPath) = GetTestContext(testMethodName);

        await VerifyAsync(mainFile.GetTextBuffer().CurrentSnapshot, $"{TestMethodName}_{MainFileSuffix}", AdditionalPath);
        await VerifyAsync(codeBehindFile.GetTextBuffer().CurrentSnapshot, $"{TestMethodName}_{CodeBehindFileSuffix}", AdditionalPath);
    }

    private static Task VerifyAsync(ITextSnapshot textSnapshot, string methodName, string additionalPath)
        => Verifier.Verify(textSnapshot.GetText())
            .UseDirectory($@"TestData\{additionalPath}")
            .UseFileName(methodName);

Pining @SimonCropp for sharing his thoughts on this.

[SimonCropp]

@RussKie yeah thats not ideal. where do i find that code so i can have a go a making it better?

[vlada-shubina]

Just to explain our use case better: we would like to verify the template output, that can be N files with random folder structure.
Example:

• src
  • Project1
    • project 1 content goes here (multiple files)
  • Project2
    • project 2 content goes here (multiple files)
• test
  • test files go here

The template output is pretty static (with some exceptions like guids, dates etc) so approval tests seems to be good match to test them.

Ideally we do the following:

await VerifyFiles(pathToTemplateOutput); 

and this produces the snapshot identical to folder/files in the pathToTemplateOutput folder.
Ideally in case of failure, all the failures will be shown in the one window. Certain settings decorators should be still applicable.
Initial plan was to similar to what @RussKie mentioned above, but we would have many files to check and not to fail on the first incorrect file.

Imho, this use case (verifying all the files in the folder) is pretty generic and implementing it in Verify.NET would be of benefit.

[RussKie]

Thank you for the context, now I think I get it - executing a template will produce several files, and each template may have a different number of those files. So essentially we need to verify a folder content. I don't think it'd be difficult to create a helper method that verifies a content of a folder, though it'd make our lives easier if that helper was provided out of a box. :)

In Windows Forms scenarios we generally compare one or two known files (more like entities that represent files, and those may not even exist on disk).

Ideally in case of failure, all the failures will be shown in the one window.

I can interpret this wish in a few different ways, though not sure if my interpretation aligns with yours.

• Tooling: some diff/merge tools provide folder comparison functionality (e.g., BeyondCompare 4 IIRC) but a lot of tools don't.
• Test result presentation: collecte results from all failed file verifications and concatenate them together. It's possible to catch every "verification failed exception" (there's a specific exception type, I don't remember the name of it), and then provide a custom report to the user. Depending on how much data is presented to a developer this may degrade the developer experience.

[RussKie]

Thank you @SimonCropp for jumping in. In essence, here we're talking about verifying the result of dotnet new command. E.g., dotnet new winforms, dotnet new winforms -n MyApp, dotnet new winforms -f net5.0, etc. will produce folders with own sets of files, and we'll need to verify the content of each file. (@vlada-shubina please correct me if I misinterpret it) We will also need to be able to exclude some folders (e.g., obj).

D:\Development\throwaway\foo>dotnet new winforms -f net5.0
The template "Windows Forms App" was created successfully.

Processing post-creation actions...
Restoring D:\Development\throwaway\foo\foo.csproj:
  Determining projects to restore...
  Restored D:\Development\throwaway\foo\foo.csproj (in 68 ms).
Restore succeeded.


D:\Development\throwaway\foo>dir /b
foo.csproj
foo.csproj.user
Form1.cs
Form1.Designer.cs
obj
Program.cs

Do you think it's a worthy addition to your already awesome library?

The sample I mentioned earlier is in a close source, but I'm sure @vlada-shubina or @JanKrivanek may be able to explain their test procedures in more details.

[vlada-shubina]

Added couple of test drafts in #5163.
Note that this syntax is not set in stone, it may be different.

[vlada-shubina]

now.txt
port.txt
random.txt
guid.txt

Usage stats of non-deterministic macro attached:

• port - 78 templates
• random - 11 templates
• now - 51 templates
• guid - 298 templates

[SimonCropp]

@vlada-shubina i should have a beta "verifydirectory" for you in a couple of days

[SimonCropp]

@vlada-shubina if you update to the current beta of Verify, you can try this https://github.com/VerifyTests/Verify#verifydirectory

[vlada-shubina]

@vlada-shubina if you update to the current beta of Verify, you can try this https://github.com/VerifyTests/Verify#verifydirectory

Awesome work! Thanks. First very basic test passed.

The only obstacle I'm stuck with is here:
https://github.com/VerifyTests/Verify/blob/33f951c84a2e08d5aed9743158abb88795f3067f/src/Verify/Splitters/Target.cs#L83-L86
The filenames can easily contain . in the name. Also subfolders do not work for the same reason as above.

update: if possible I would prefer different naming

- <testname folder>.received
  - content received goes here with unchanged names
- <testname folder>.verified
  - verified content goes here with unchanged names

Having verified and received mixed in the file name is a big confusing, see:

[SimonCropp]

The filenames can easily contain .

i have deployed a new version with that constraint removed. I no longer makes sense anyway given the current file naming logic

if possible I would prefer different naming

hmm. let me think on that and get back to you

[SimonCropp]

@vlada-shubina how does this look https://github.com/VerifyTests/Verify/blob/main/docs/naming.md#usesplitmodeforuniquedirectory
it is in the 18.0.0-beta.17 nuget

[vlada-shubina]

@vlada-shubina how does this look https://github.com/VerifyTests/Verify/blob/main/docs/naming.md#usesplitmodeforuniquedirectory it is in the 18.0.0-beta.17 nuget

Thanks, it works great.
The only improvement that I may think of is to do folder diff in diff tool, if it's available.
For example: DiffMerge has a folder mode, and it's much more convenient to use that one instead of running multiple DiffMerge processes for individual files. CodeCompare supports it as well.

[JanKrivanek]

Closing as work in context of 7.0 was done.
Tracking the next iteration ideas in a new item: #5705