🙏 Feature Request: AST-Based Test Discovery that doesn't run test discovery logic

[PrzemyslawKlys]

It's related probably to: TylerLeonhardt/vscode-pester-test-adapter#16

PS C:\Support\GitHub\GpoZaurr> . 'c:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1' -Discovery -PipeName PesterTestController-17564 c:\Support\GitHub\GpoZaurr\Tests\GPOPermissions.Tests.ps1 -Verbosity None
Import-Module : The value Ignore is not supported for an ActionPreference variable. The provided value sh
ould be used only as a value for a preference parameter, and has been replaced by the default value. For
more information, see the Help topic, "about_Preference_Variables."
At C:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1:337
 char:15
+     $Pester = Import-Module Pester -PassThru
+               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], NotSupportedException
    + FullyQualifiedErrorId : System.NotSupportedException,Microsoft.PowerShell.Commands.ImportModuleCom
   mand

The expression after '&' in a pipeline element produced an object that was not valid. It must result in a
 command name, a script block, or a CommandInfo object.
At C:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1:338
 char:7
+     & $Pester {
+       ~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : BadExpression

And

PS C:\Support\GitHub\GpoZaurr> . 'c:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1' -Discovery -PipeName PesterTestController-17564 c:\Support\GitHub\GpoZaurr\GPOZaurr.Tests.ps1 -Verbosity None
Import-Module : The value Ignore is not supported for an ActionPreference variable. The provided value sh
ould be used only as a value for a preference parameter, and has been replaced by the default value. For
more information, see the Help topic, "about_Preference_Variables."
At C:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1:337
 char:15
+     $Pester = Import-Module Pester -PassThru
+               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], NotSupportedException
    + FullyQualifiedErrorId : System.NotSupportedException,Microsoft.PowerShell.Commands.ImportModuleCom
   mand

The expression after '&' in a pipeline element produced an object that was not valid. It must result in a
 command name, a script block, or a CommandInfo object.
At C:\Users\przemyslaw.klys\.vscode\extensions\pspester.pester-test-0.0.1\Scripts\PesterInterface.ps1:338
 char:7
+     & $Pester {
+       ~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : BadExpression



ModuleName: GPOZaurr Version: 0.0.129
PowerShell Version: 5.1.19041.1151
PowerShell Edition: Desktop
Required modules: 
   [>] PSSharedGoods Version: 0.0.208
   [>] ADEssentials Version: 0.0.130
   [>] PSWriteHTML Version: 0.0.158
   [>] CimCmdlets
   [>] Microsoft.PowerShell.Management
   [>] Microsoft.PowerShell.Utility
   [>] Microsoft.PowerShell.Security

Import-Module : The value Ignore is not supported for an ActionPreference variable. The provided value sh
ould be used only as a value for a preference parameter, and has been replaced by the default value. For
more information, see the Help topic, "about_Preference_Variables."
At C:\Support\GitHub\GpoZaurr\GPOZaurr.Tests.ps1:46 char:1
+ Import-Module $PSScriptRoot\*.psd1 -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], NotSupportedException
    + FullyQualifiedErrorId : System.NotSupportedException,Microsoft.PowerShell.Commands.ImportModuleCom
   mand

Export-ModuleMember : The value Ignore is not supported for an ActionPreference variable. The provided va
lue should be used only as a value for a preference parameter, and has been replaced by the default value
. For more information, see the Help topic, "about_Preference_Variables."
At C:\Users\przemyslaw.klys\Documents\WindowsPowerShell\Modules\Pester\5.1.1\Pester.psm1:15353 char:1
+ & $script:SafeCommands['Export-ModuleMember'] @(
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Export-ModuleMember], NotSupportedException
    + FullyQualifiedErrorId : System.NotSupportedException,Microsoft.PowerShell.Commands.ExportModuleMem
   berCommand

[JustinGrote]

The "ignore" thing is a known 5.1 bug, switch to 7 and it works fine. I haven't ironed out all the 5.1 compatability stuff.

Also it only executes discovery logic in discovery. If you put things like your "Required modules" output above in a BeforeEach or BeforeAll it won't execute.

[JustinGrote]

I will track ignore in a separate issue, however the "Execution on expand" is required in order to discover tests and testcases. A no-execute AST-based method may be possible but I'm not gonna make it, I may consider PRs as a testProvider :)

[PrzemyslawKlys]

Well - the version of Tyler doesn't execute them - so there's that. Surely this is rather unexpected to get something executed while opening as it can have deadly consequences.

[JustinGrote]

@PrzemyslawKlys without it, there's no way to identify test cases without basically duplicating extensive AST parsing. Performing that action has little difference to the Invoke-Pester command with "NoRun" enabled, so if the "execution on discovery" is a concern, this is an issue that should be filed in the Pester repository and this repo will inherit that enhancement.

I get your point in terms of "oh someone puts rm -rf * in the discovery logic" but that's a risk regardless. Tyler's method is here: https://github.com/TylerLeonhardt/vscode-pester-test-adapter/blob/master/src/powershellScripts.ts

I'll reopen this issue and mark it as up-for-grabs.

[PrzemyslawKlys]

Well have you noticed other errors that happened? While one thing is auto-execution - the other is - it crashed, and have not detected a single thing :)

[JustinGrote]

@PrzemyslawKlys not quite sure what you mean but it's an 0.0.1 release, I expect there to be many bugs to squash at the moment :). You are also totally free to use Tyler's extension still, there is a new setting in test adapter that "ports" them to the new testing API

[JustinGrote]

@PrzemyslawKlys because I was curious about this, I checked and Tyler's extension actually does run discovery and execute code, you just don't see it because he runs it as a background runspace whereas I run it directly in the terminal. I plan to add background runspace support, but FYI.

Here's the function Tyler calls (Find-Test):
https://github.com/pester/Pester/blob/b8c64bb1a3436af0e5c234078ca341e40ba558b2/src/Pester.Runtime.ps1#L112-L112

Which in turn calls discover-test which runs the discovery process:
https://github.com/pester/Pester/blob/b8c64bb1a3436af0e5c234078ca341e40ba558b2/src/Pester.Runtime.ps1#L876-L876

TO verify this write to a file on your desktop in your discovery logic. When I ran it, it showed up.

[PrzemyslawKlys]

I mean if you look at gif I presented - you will see that I expand the tests - and when I expand tests it starts to execute and starts throwing all sort of PowerShell errors which are not there when running manually.

I understand it's 0.0.1 - I don't even use pester that much - just wanted to help report errors. I don't use Tyler's extension either because I'm simply not writing any tests lately :)

[JustinGrote]

I mean if you look at gif I presented - you will see that I expand the tests - and when I expand tests it starts to execute and starts throwing all sort of PowerShell errors which are not there when running manually.

I understand it's 0.0.1 - I don't even use pester that much - just wanted to help report errors. I don't use Tyler's extension either because I'm simply not writing any tests lately :)

Appreciated! as far as the error it's because you're running in 5.1 and not 7. Issue for that here, should be fixed next release:
#15

[JustinGrote]

I'm going to close this issue because it makes more sense for Pester to implement this with a function or configuration option, and if it does, then this adapter can support it, I recommend opening this in the Pester repository if it is a desired feature.

[JustinGrote]

@PrzemyslawKlys to follow up on this, the Trusted Workspaces feature already prevents the extension from loading until a workspace is trusted. I may implement support to do the file-based pester discovery but not actual test discovery until a workspace is trusted, that is tracked here:

#78