Run e2e tests locally without spinning smtp servers

[cstns]

Description

add the ability to run e2e tests locally without spinning the docker smtp server processes

Related Issue(s)

Checklist

• I have read the contribution guidelines
• Suitable unit/system level tests have been added and they pass
• Documentation has been updated
  • Upgrade instructions
  • Configuration details
  • Concepts
• Changes flowforge.yml?
  • Issue/PR raised on FlowFuse/helm to update ConfigMap Template
  • Issue/PR raised on FlowFuse/CloudProject to update values for Staging/Production

Labels

• Includes a DB migration? -> add the area:migration label

[cstns]

the only downside to this approach is that e2e tests that require the smtp server will always fail locally when running the cy:web-server-light script

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.65%. Comparing base (ed6cb31) to head (577a935).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4147   +/-   ##
=======================================
  Coverage   78.65%   78.65%           
=======================================
  Files         286      286           
  Lines       13087    13087           
  Branches     2922     2922           
=======================================
  Hits        10293    10293           
  Misses       2794     2794           

Flag	Coverage Δ
backend	78.65% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Steve-Mcl]

the only downside to this approach is that e2e tests that require the smtp server will always fail locally when running the cy:web-server-light script

Could the same env var not be used to .skip the failing tests?

[Steve-Mcl]

Instead of yet another script, can we not simply do the inverse?

i.e.

• local tests always runas they always used to (no SMTP docker image)
• tests on CI self determine the environment and run the SMTP docker image
• If local wants to run smtp tests then local sets an ENV VAR e.g. SMTP_TESTS=true

[cstns]

Can't think of a solution where we'd get away with not duplicating an existing script, but I'm open to suggestions.

We'd need to be able to run the e2e tests locally with and without the smtp server (this applies to all three scenarios running os, running ee and running them together) while CI needs relies on a smtp service (so no smtp server required via docker)

the only downside to this approach is that e2e tests that require the smtp server will always fail locally when running the cy:web-server-light script

Could the same env var not be used to .skip the failing tests?

This would be a good approach but would come with some caveats.

Running the e2e tests locally without the smtp server and associated tests would result in an incomplete run and would render incomplete result. That's why I think there should be another script or method that would run the entire suite with the smtp server enabled locally.