Skip to content

Bug: Incorrect program shutdown order #61

@PiotrKorkus

Description

@PiotrKorkus

Original report: https://github.com/qorix-group/inc_orchestrator_internal/issues/398

Describe the bug:

There are two programs running in parallel. We initiate Shutdown 2 and run a loop checking if Shutdown2 was executed (AtomicBool was set). Once loaded Atomic is true it initiates Shutdown1.
It is expected that Shutdown1 is initiated after Shutdown2 was executed.
The initiation order was correct but execution not. Please check the logs.

Scenario logic: TwoProgramsTwoShutdowns in component_integration_tests/rust_test_scenarios/src/tests/orchestration/orchestration_shutdown.rs

Steps to reproduce the behavior:

Observed only in nightly tests.

  1. scripts/run_component_tests.sh -k shutdown_order

Observed behavior:

Shutdown 1 executed before Shutdown 2.

Expected behavior

Initiated and executed order should be the same - 2 then 1.

Occurrence:

Very rare - observed once in nightly. Not reproduced locally.

Attachments / Logs:

https://github.com/qorix-group/inc_orchestrator_internal/actions/runs/19218611305/job/54932115703

ResultEntry(timestamp=0:00:00.012050, level=DEBUG, message=Initiating shutdown 2..., target=rust_test_scenarios::tests::orchestration::orchestration_shutdown, thread_id=ThreadId(1))
ResultEntry(timestamp=0:00:00.013000, level=DEBUG, message=Initiating shutdown 1..., target=rust_test_scenarios::tests::orchestration::orchestration_shutdown, thread_id=ThreadId(1))
ResultEntry(timestamp=0:00:00.013503, level=INFO, message=ShutdownDesign1::StopAction was executed, target=rust_test_scenarios::tests::orchestration::orchestration_shutdown, thread_id=ThreadId(33))
ResultEntry(timestamp=0:00:00.015167, level=INFO, message=ShutdownDesign2::StopAction was executed, target=rust_test_scenarios::tests::orchestration::orchestration_shutdown, thread_id=ThreadId(3))

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    Status

    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions