Retry sometimes does not work as configured.

[panicboat]

Basically, retries work as set up.
However, the retry did not work only for the attached Workflow.
I don't know the cause of the problem, and I couldn't decide whether I should report it as a bug or not.

If you check the manifest file, you will see that the following retry settings are made in main.

    - name: main
      retryStrategy:
        limit: 2
        retryPolicy: Always

But there appears to be only aya-jobs-import-webdm-tp5wx-881703283 as a child of aya-jobs-import-webdm-tp5wx in node.
What is happening?
If it is a bug, can you please check it for us to isolate it since we don't know what the condition is?

manifest

aya-jobs-import-webdm-tp5wx.txt

workflow-controller

workflow-controller.txt

workflow

@timestamp | container_name | @message
-- | -- | --
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.363411353Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.363Z\" level=warning msg=\"failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/\" error=\"workflowtaskresults.argoproj.io \\\"aya-jobs-import-webdm-tp5wx-881703283\\\" is forbidden: User \\\"system:serviceaccount:pr-90616-workflow-secure:aya-jobs\\\" cannot patch resource \\\"workflowtaskresults\\\" in API group \\\"argoproj.io\\\" in the namespace \\\"pr-90616-workflow-secure\\\"\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.361774526Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.361Z\" level=info msg=\"Alloc=12168 TotalAlloc=52075 Sys=28517 NumGC=68 Goroutines=12\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.346662386Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.346Z\" level=warning msg=\"failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/\" error=\"workflowtaskresults.argoproj.io is forbidden: User \\\"system:serviceaccount:pr-90616-workflow-secure:aya-jobs\\\" cannot create resource \\\"workflowtaskresults\\\" in API group \\\"argoproj.io\\\" in the namespace \\\"pr-90616-workflow-secure\\\"\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.345391047Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.345Z\" level=info msg=\"Save artifact\" artifactName=main-logs duration=325.948708ms error=\"<nil>\" key=aya-jobs-import-webdm-tp5wx/aya-jobs-import-webdm-tp5wx-main-881703283/main.log","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.345429257Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.345Z\" level=info msg=\"not deleting local artifact\" localArtPath=/tmp/argo/outputs/logs/main.log","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.345435728Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.345Z\" level=info msg=\"Successfully saved file: /tmp/argo/outputs/logs/main.log\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.026051658Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.025Z\" level=info msg=\"Creating minio client using IAM role\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.026072447Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.026Z\" level=info msg=\"Saving file to s3\" bucket=sapuri-staging-workflow-artifacts-staging endpoint=s3.amazonaws.com key=aya-jobs-import-webdm-tp5wx/aya-jobs-import-webdm-tp5wx-main-881703283/main.log path=/tmp/argo/outputs/logs/main.log","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:03.019431838Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:03.019Z\" level=info msg=\"S3 Save path: /tmp/argo/outputs/logs/main.log, key: aya-jobs-import-webdm-tp5wx/aya-jobs-import-webdm-tp5wx-main-881703283/main.log\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:02.966390612Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:02.966Z\" level=info msg=\"Main container completed\" error=\"<nil>\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:02.966431973Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:02.966Z\" level=info msg=\"No Script output reference in workflow. Capturing script output ignored\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:02.966438043Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:02.966Z\" level=info msg=\"No output parameters\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:03 | wait | {"time":"2024-07-31T08:47:02.966441293Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:02.966Z\" level=info msg=\"No output artifacts\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:47:02 | main | {"time":"2024-07-31T08:47:02.127817255Z","stream":"stderr","_p":"F","log":"Error: exit status 137","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}
2024-07-31 8:47:02 | main | {"time":"2024-07-31T08:47:02.125932253Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:47:02.125Z\" level=info msg=\"sub-process exited\" argo=true error=\"<nil>\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}
2024-07-31 8:46:58 | wait | {"time":"2024-07-31T08:46:57.898152002Z","stream":"stderr","_p":"F","log":"time=\"2024-07-31T08:46:57.897Z\" level=warning msg=\"failed to patch task result, falling back to legacy/insecure pod patch, see https://argo-workflows.readthedocs.io/en/release-3.5/workflow-rbac/\" error=\"workflowtaskresults.argoproj.io is forbidden: User \\\"system:serviceaccount:pr-90616-workflow-secure:aya-jobs\\\" cannot create resource \\\"workflowtaskresults\\\" in API group \\\"argoproj.io\\\" in the namespace \\\"pr-90616-workflow-secure\\\"\"","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"wait","container_hash":"quay.io/argoproj/argoexec@sha256:cb5a9f6dd344d2b13fdef59f57f4123b94a6936d5da8133a74c58e7225d7c527","container_image":"quay.io/argoproj/argoexec:v3.5.9"}
2024-07-31 8:46:53 | main | {"time":"2024-07-31T08:46:53.203329632Z","stream":"stdout","_p":"F","log":"[webdm_push_notification_v1_20240726_TG0000010010022905_0001.json] [user_id=5cbfd1e8d4b8d65f360002cd] Create UserWebdmPushNotification","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}
2024-07-31 8:46:44 | main | {"time":"2024-07-31T08:46:44.138639185Z","stream":"stdout","_p":"F","log":"[webdm_push_notification_v1_20240726_TG0000010010022905_0001.json] [user_id=5cbfd171d4b8d65f36000280] Create UserWebdmPushNotification","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}
2024-07-31 8:46:37 | main | {"time":"2024-07-31T08:46:36.572751848Z","stream":"stdout","_p":"F","log":"[webdm_push_notification_v1_20240726_TG0000010010022905_0001.json] [user_id=5cbfd12925b8f1429e000af0] Create UserWebdmPushNotification","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}
2024-07-31 8:46:33 | main | {"time":"2024-07-31T08:46:33.210348193Z","stream":"stdout","_p":"F","log":"[webdm_push_notification_v1_20240726_TG0000010010022905_0001.json] [user_id=5cbfc90993083a3dc4000ce7] Create UserWebdmPushNotification","pod":"aya-jobs-import-webdm-tp5wx-main-881703283","namespace":"pr-90616-workflow-secure","host":"ip-10-34-184-24.ap-northeast-1.compute.internal","container_name":"main","container_hash":"575405593970.dkr.ecr.ap-northeast-1.amazonaws.com/monorepo/aya-jobs@sha256:55dbd55c00ab56d2c3bf1e6ea5a8f6ab4e4f56bb2b43f74d31524f32d00c0393","container_image":"sha256:d401231faca442d8ca48a714585967c2a1e5acfe64ca69a3b2481bb8219f4eea"}

[panicboat]

I think it is another cause because OOM confirms that it is retried.

metadata:
  name: lovely-rhino1
  namespace: pr-90616-workflow-secure
  labels:
    example: 'true'
spec:
  arguments:
    parameters:
      - name: message
        value: hello argo
  entrypoint: argosay
  templates:
    - name: argosay
      retryStrategy:
        limit: 2
        retryPolicy: Always
      inputs:
        parameters:
          - name: message
            value: '{{workflow.parameters.message}}'
      container:
        name: main
        image: argoproj/argosay:v2
        command:
          - /argosay
        args:
          - echo
          - '{{inputs.parameters.message}}'
        resources:
          limits:
            memory: 8Mi
          requests:
            memory: 8Mi
  ttlStrategy:
    secondsAfterCompletion: 300
  podGC:
    strategy: OnPodCompletion

lovely-rhino1.txt

[panicboat]

retryStrategy.backoff.maxDuration did not recognize that it included processing time and was small.

argo-workflows/workflow/controller/operator.go

Lines 990 to 1007 in 3bb637c

	if retryStrategy.Backoff != nil {
	maxDurationDeadline := time.Time{}
	// Process max duration limit
	if retryStrategy.Backoff.MaxDuration != "" && len(childNodeIds) > 0 {
	maxDuration, err := parseStringToDuration(retryStrategy.Backoff.MaxDuration)
	if err != nil {
	return nil, false, err
	}
	firstChildNode, err := woc.wf.Status.Nodes.Get(childNodeIds[0])
	if err != nil {
	return nil, false, err
	}
	maxDurationDeadline = firstChildNode.StartedAt.Add(maxDuration)
	if time.Now().After(maxDurationDeadline) {
	woc.log.Infoln("Max duration limit exceeded. Failing...")
	return woc.markNodePhase(node.Name, lastChildNode.Phase, "Max duration limit exceeded"), true, nil
	}
	}