test(ffe): add parametric tests for provider initialization blocking

[leoromanovsky]

Motivation

FFL-1843: Python OpenFeature provider's initialize() returns in 0.00s without waiting for Remote Config data. The SDK emits PROVIDER_READY immediately, and flag evaluations silently return defaults. This affects scripts and short-lived processes where there's no time for RC to deliver config before evaluations happen.

The fix is in DataDog/dd-trace-py#16650 which makes initialize() block with a threading.Event until config arrives (matching Java/Go/Node.js).

This PR adds system tests that validate the blocking behavior and a weblog fix needed to support them.

Changes

New test file: tests/parametric/test_ffe/test_initialization_blocking.py (5 tests)

Test	What it validates
test_ffe_init_blocks_until_config_received	Config before start -> provider READY, real values
test_ffe_init_returns_real_values_not_defaults	Config after start but before eval -> real values
test_ffe_evaluation_immediately_after_start_without_config	No config -> provider must NOT claim READY in <5s
test_ffe_init_blocks_and_resolves_when_config_arrives	Config arrives mid-block -> unblocks, real values immediately
test_ffe_init_timeout_returns_error	Custom 3s timeout -> times out, then late recovery via RC

Weblog fix: Removed module-level auto-initialization of the OpenFeature provider in the Python parametric weblog (server.py). The provider should only be initialized when the test calls /ffe/start, not at server startup. With the blocking initialize() fix, module-level init would block server startup for 30s.

Decisions

• Tests are Python-specific (the fix is in dd-trace-py). Other languages already block in initialize() but their weblog architectures differ (e.g. Go initializes at container startup, not via /ffe/start).
• The test_ffe_init_blocks_and_resolves_when_config_arrives test uses a background thread to deliver RC config 2s after ffe_start() is called, proving the blocking wait resolves mid-flight.
• The test_ffe_init_timeout_returns_error test uses DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS=3000 to avoid waiting the full 30s default.

Testing

All 5 tests pass locally against the patched dd-trace-py from #16650:

test_ffe_init_blocks_until_config_received           PASSED  32s
test_ffe_init_returns_real_values_not_defaults        PASSED  61s
test_ffe_evaluation_immediately_after_start_without_config  PASSED  62s
test_ffe_init_blocks_and_resolves_when_config_arrives PASSED  33s
test_ffe_init_timeout_returns_error                   PASSED  38s

Requires: DataDog/dd-trace-py#16650

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test_initialization_blocking.py               @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/build/docker/python/parametric/apm_test_client/server.py          @DataDog/apm-python @DataDog/asm-python @DataDog/system-tests-core

[datadog-datadog-prod-us1-2]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 31 Tests failed

tests.parametric.test_ffe.test_initialization_blocking.Test_FFE_Initialization_Blocks_Until_Config.test_ffe_evaluation_immediately_after_start_without_config[library_env0, parametric-cpp] from system_tests_suite (Datadog) (Fix with Cursor)

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

self = <Response [404]>, kwargs = {}

    def json(self, **kwargs):
        r"""Returns the json-encoded content of a response, if any.
    
        :param \*\*kwargs: Optional arguments that \`\`json.loads\`\` takes.
        :raises requests.exceptions.JSONDecodeError: If the response body does not
            contain valid json.
...

tests.parametric.test_ffe.test_initialization_blocking.Test_FFE_Initialization_Blocks_Until_Config.test_ffe_evaluation_immediately_after_start_without_config[library_env0, parametric-dotnet] from system_tests_suite (Datadog) (Fix with Cursor)

requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

self = <Response [404]>, kwargs = {}

    def json(self, **kwargs):
        r"""Returns the json-encoded content of a response, if any.
    
        :param \*\*kwargs: Optional arguments that \`\`json.loads\`\` takes.
        :raises requests.exceptions.JSONDecodeError: If the response body does not
            contain valid json.
...

tests.parametric.test_ffe.test_initialization_blocking.Test_FFE_Initialization_Blocks_Until_Config.test_ffe_evaluation_immediately_after_start_without_config[library_env0, parametric-golang] from system_tests_suite (Datadog) (Fix with Cursor)

Failed: BUG FFL-1843: ffe_start() returned in 0.00s (should block ~30s). Provider returned default with reason='UNKNOWN' without any config. Full result: {'value': 'default-sentinel'}

self = <tests.parametric.test_ffe.test_initialization_blocking.Test_FFE_Initialization_Blocks_Until_Config object at 0x7fd32c1e9700>
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fd2fbda0b00>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fd2fbf41970>

    @parametrize("library_env", [{**DEFAULT_ENVVARS}])
    def test_ffe_evaluation_immediately_after_start_without_config(
        self,
        test_agent: TestAgentAPI,  # noqa: ARG002
...

View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7f2d4c5 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}