Make Test Runtime Configurable Across Environments (URLs, DB Matrix, Fixtures, Services, Profiles)

[jordicompanym]

Problem statement

The current test suites (utest and ujest) include multiple hardcoded environment assumptions (base URLs, credentials, DB/service availability, dataset shape, fixed timeouts).
This causes fragile runs across different developer setups and CI environments, even when product code is correct.

Current hardcoded points (examples)

• Fixed URLs such as https://127.0.0.1/saltos/code4/...
• Fixed auth credentials (admin/admin)
• Fixed UI timeouts (3000ms)
• Fixed temporary/coverage paths in /tmp/...
• DB assumptions tied to specific engines/hosts/state
• Data assumptions (for example expected initial row counts in UI flows)
• Service assumptions (Greenmail, PostgreSQL, MSSQL availability)

Proposed solution

Introduce a test runtime configuration file with environment overrides, so tests can run consistently in local and CI setups without editing test source code.

Recommended approach:

• Versioned default config: tests/config.xml
• Local override (ignored): tests/config.local.xml
• Optional env var overrides for CI/secrets (highest priority)

Load order:

1. tests/config.xml
2. tests/config.local.xml (if present)
3. Environment variables

Configuration schema (proposed)

• base_url_web, base_url_api
• auth (user, pass)
• timeouts (ui_default_ms, ui_heavy_ms)
• paths (tmp_root, coverage_root, tester_json)
• databases
  • main (ref)
  • per-engine entries: mysql, pgsql, mssql, sqlite with enabled, mode, connection params
  • fixture (policy, source, file)
• services
  • greenmail (enabled, mode, host, port)
• fixtures (for deterministic app-level assumptions, e.g. customers.initial_rows)
• profile (fast|integration|e2e|visual)
• skip (explicit list of suites/tests to skip in a given environment)

Parameter definitions

• base_url_web
  • Base URL used by browser/UI tests (Puppeteer/Jest E2E).
  • Example: https://127.0.0.1/saltos4_desarrollo/code/web
• base_url_api
  • Base URL used by API/backend HTTP tests.
  • Example: https://127.0.0.1/saltos4_desarrollo/code/api
• auth user
  • Default test user login for flows that require authentication.
  • Example: admin
• auth pass
  • Default test user password.
  • Example: admin
• timeouts ui_default_ms
  • Default timeout (ms) for normal UI waits (waitForSelector, waitForFunction).
  • Example: 3000
• timeouts ui_heavy_ms
  • Extended timeout (ms) for slow/heavy screens or first-load scenarios.
  • Example: 10000
• paths tmp_root
  • Root temp directory for intermediate test artifacts.
  • Example: /tmp
• paths coverage_root
  • Output directory root for JS coverage raw files.
  • Example: /tmp/nyc_output
• paths tester_json
  • Path to generated tester metadata/config file used by UI tests.
  • Example: /tmp/tester.json
• databases main ref
  • Defines which DB engine entry is used as the primary DB for normal app tests.
  • Example: mysql
• databases enabled (true|false)
  • Enables/disables tests for that engine in the current environment/profile.
• databases mode (existing|docker)
  • existing: use an already running DB instance.
  • docker: spin up/use DB through test Docker profile.
• databases mysql|pgsql|mssql host/port/name/user/pass
  • Connection parameters for client-server DB engines.
• databases sqlite file
  • SQLite database file path.
  • Example: /home/user/db/saltos4_test.sqlite
• databases fixture policy (reuse|reset|seed)
  • reuse: use DB as-is (fastest, less reproducible).
  • reset: clean/recreate baseline state before tests (no business seed unless explicitly requested).
  • seed: reset + load deterministic seed data.
• databases fixture source (none|builtin|file)
  • none: do not load extra seed data.
  • builtin: load versioned project seed fixtures.
  • file: load seed data from databases fixture file.
• databases fixture file
  • Path to seed file used when source=file.
  • Example: /home/user/workspace/seeds/test_seed.sql
• services greenmail enabled (true|false)
  • Enables/disables Greenmail-dependent test paths.
• services greenmail mode (existing|docker)
  • existing: use an already running Greenmail.
  • docker: use Greenmail container from test profile.
• services greenmail host/port
  • Host and port used by email integration tests.
• fixtures customers initial_rows
  • Expected seeded customer rows for pagination/load-more assertions.
  • Example: 25 or 100 depending on fixture.
• profile (fast|integration|e2e|visual)
  • fast: unit/mocked tests only.
  • integration: backend + DB/services integration tests.
  • e2e: end-to-end flows with real app runtime.
  • visual: screenshot/snapshot-heavy suites.
• skip test (repeatable)
  • Explicit suite/test entries to skip for a given environment/profile.
  • Examples:
    • utest:test_database.pdo_mssql
    • ujest:test_screenshots

Example XML

<tests>
	<base_url_web>https://127.0.0.1/saltos4_desarrollo/code/web</base_url_web>
	<base_url_api>https://127.0.0.1/saltos4_desarrollo/code/api</base_url_api>

	<auth user="admin" pass="admin"/>

	<timeouts ui_default_ms="3000" ui_heavy_ms="10000"/>

	<paths
	  tmp_root="/tmp"
	  coverage_root="/tmp/nyc_output"
	  tester_json="/tmp/tester.json"
	/>

	<databases>
	  <main ref="mysql"/>

	  <mysql
		enabled="true"
		mode="existing"
		host="127.0.0.1"
		port="3306"
		name="saltos4_test"
		user="saltos"
		pass="saltos"
	  />

	  <pgsql
		enabled="false"
		mode="existing"
		host="127.0.0.1"
		port="5432"
		name="saltos4_test"
		user="saltos"
		pass="saltos"
	  />

	  <mssql
		enabled="false"
		mode="existing"
		host="127.0.0.1"
		port="1433"
		name="saltos4_test"
		user="sa"
		pass="***"
	  />

	  <sqlite
		enabled="false"
		mode="existing"
		file="/home/user/db/saltos4_test.sqlite"
	  />

	  <fixture policy="seed" source="builtin" file=""/>
	</databases>

	<services>
	  <greenmail
		enabled="true"
		mode="existing"
		host="127.0.0.1"
		port="3025"
	  />
	</services>

	<fixtures>
	  <customers initial_rows="25"/>
	</fixtures>

	<profile>integration</profile>

	<skip>
	  <test>utest:test_database.pdo_mssql</test>
	  <test>ujest:test_screenshots</test>
	</skip>
</tests>

Implementation plan

1. Add config loader utility for test runtime settings.
2. Replace hardcoded URL/auth/timeout/path values in utest and ujest.
3. Add profile handling (fast, integration, e2e, visual).
4. Add database matrix handling (main ref, per-engine enabled/mode).
5. Add fixture policy handling (reuse, reset, seed) and seed source support.
6. Add service handling flags (starting with Greenmail).
7. Add config.local.xml support and documentation.
8. Update test docs with examples and migration notes.

Acceptance criteria

• Same test command works across at least two different local setups without code edits.
• No hardcoded base URL remains in test source.
• UI timeout and credentials are configurable.
• Database execution can switch between existing and docker.
• Primary DB can be selected via databases main ref.
• Multi-engine checks can be enabled/disabled per environment.
• SQLite can run with explicit file path.
• Data-sensitive tests can run with deterministic fixture policy (seed).
• CI can override config via environment variables.

Non-goals / out of scope

• Full redesign of the current test architecture.
• Rewriting all tests at once.
• Enforcing Docker-only execution for every developer.

[josepsanzcamp]

Thanks for taking the time to write such a detailed proposal. The idea of making the test runtime configurable across environments definitely makes sense and the structure you describe is well thought out.

At the moment our main focus has been on getting the test suites working reliably and ensuring we have good coverage across unit, integration and end-to-end tests. The current setup was built with that goal in mind so the project has a solid testing base.

What you’re proposing would certainly improve flexibility, especially for different environments and CI setups, but it also introduces a fair amount of additional infrastructure around the test system. Given the current resources available for the project, this isn’t something I can prioritize right now.

I’ll keep this proposal in mind for the future, as it’s a useful direction for improving the testing workflow once there is time to dedicate to this kind of enhancement.