Flaky-test: PulsarFunctionsK8STest.testCreateFunctionInK8sWithSecrets

[lhotari]

Search before reporting

• I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/actions/runs/21718811469/job/62651627693?pr=25207#step:12:5662

Exception stacktrace

Error:  Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 126.1 s <<< FAILURE! -- in TestSuite
  Error:  org.apache.pulsar.tests.integration.functions.k8s.PulsarFunctionsK8STest.testCreateFunctionInK8sWithSecrets -- Time elapsed: 0.068 s <<< FAILURE!
  org.apache.pulsar.client.admin.PulsarAdminException: Function test-function already exists
  	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
  	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:360)
  	at org.apache.pulsar.client.admin.internal.FunctionsImpl.createFunctionWithUrl(FunctionsImpl.java:199)
  	at org.apache.pulsar.tests.integration.functions.k8s.PulsarFunctionsK8STest.testCreateFunctionInK8sWithSecrets(PulsarFunctionsK8STest.java:111)
  	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.InvokeMethodRunnable.runOne(InvokeMethodRunnable.java:47)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:76)
  	at org.testng.internal.invokers.InvokeMethodRunnable.call(InvokeMethodRunnable.java:11)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  	at java.base/java.lang.Thread.run(Thread.java:1583)
  	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException: Function test-function already exists

Are you willing to submit a PR?

• I'm willing to submit a PR!

[pdolif]

If the test fails, it will be retried once. There is no cleanup of created resources between the test runs. Therefore, it is very likely that the second test run fails due to "test-function already exists" after the first run fails. The error shared in this issue is the error from the second test run. The error from the first run is not shown in the CI logs. This might be misleading. Therefore, I suggest disabling the retry for this test: #25230

---

Here are some relevant parts of the logs from the GitHub Actions artifacts:

2026-02-05T17:22:09,687 - INFO - [TestNG-method=testCreateFunctionInK8sWithSecrets-1:PulsarFunctionsK8STest@140] - Testing function 
2026-02-05T17:22:09,991 - INFO - [pulsar-client-io-8-3:ConnectionPool@298] - [[id: 0x46d64a13, L:/127.0.0.1:49994 - R:localhost/127.0.0.1:32771]] Connected to server 
2026-02-05T17:22:10,094 - INFO - [pulsar-client-lookup-21-1:ConsumerStatsRecorderImpl@118] - Starting Pulsar consumer status recorder with config: {"topicNames":["persistent://public/default/test-function-output-..."],...} 
2026-02-05T17:22:10,108 - INFO - [pulsar-client-lookup-21-1:ConsumerStatsRecorderImpl@119] - Pulsar client config: {"serviceUrl":"pulsar://localhost:32771",...} 
2026-02-05T17:22:10,129 - INFO - [pulsar-client-io-8-3:ConsumerImpl@870] - [persistent://public/default/test-function-output-...][sub] Subscribing to topic on cnx [id: 0x46d64a13, L:/127.0.0.1:49994 - R:localhost/127.0.0.1:32771], consumerId 0 
2026-02-05T17:22:10,173 - INFO - [pulsar-client-io-8-3:ConsumerImpl@1044] - [persistent://public/default/test-function-output-...][sub] Subscribed to topic on localhost/127.0.0.1:32771 -- consumer: 0 
2026-02-05T17:22:10,221 - INFO - [pulsar-client-io-8-3:ProducerStatsRecorderImpl@105] - Starting Pulsar producer perf with config: {"topicName":"persistent://public/default/test-function-input-...",...} 
2026-02-05T17:22:10,221 - INFO - [pulsar-client-io-8-3:ProducerStatsRecorderImpl@106] - Pulsar client config: {"serviceUrl":"pulsar://localhost:32771",...} 
2026-02-05T17:22:10,225 - INFO - [pulsar-client-io-8-3:ProducerImpl@1903] - [persistent://public/default/test-function-input-...] [null] Creating producer on cnx [id: 0x46d64a13, L:/127.0.0.1:49994 - R:localhost/127.0.0.1:32771] 
2026-02-05T17:23:10,033 - INFO - [pulsar-client-io-8-4:ProducerImpl@2033] - [persistent://public/default/test-function-input-...] [null] Temporary error in creating producer: request timeout {'durationMs': '30000', 'reqId':'3512025844353319531', 'remote':'localhost/127.0.0.1:32771', 'local':'/127.0.0.1:49994'} 
2026-02-05T17:23:10,035 - WARN - [pulsar-client-io-8-4:ClientCnx@1476] - [id: 0x46d64a13, L:/127.0.0.1:49994 - R:localhost/127.0.0.1:32771] request timeout {'durationMs': '30000', 'reqId':'3512025844353319531', 'remote':'localhost/127.0.0.1:32771', 'local':'/127.0.0.1:49994'} 
2026-02-05T17:23:10,037 - INFO - [pulsar-client-io-8-3:ConsumerImpl@1229] - [persistent://public/default/test-function-output-...] [sub] Closed consumer 
2026-02-05T17:23:10,037 - INFO - [TestNG-method=testCreateFunctionInK8sWithSecrets-1:PulsarClientImpl@864] - Client closing. URL: pulsar://localhost:32771 
2026-02-05T17:23:10,042 - INFO - [pulsar-client-io-8-3:ClientCnx@332] - [id: 0x46d64a13, L:/127.0.0.1:49994 ! R:localhost/127.0.0.1:32771] Disconnected

Pulsar standalone logs:

2026-02-05T17:22:09,535+0000 [bookkeeper-ml-scheduler-OrderedScheduler-3-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.42.0.6:59546] Created subscription on topic persistent://public/default/test-function-input-... / public/default/test-function
2026-02-05T17:22:09,563+0000 [pulsar-web-53-1] INFO  org.eclipse.jetty.server.RequestLog - 10.42.0.1 - - [05/Feb/2026:17:22:09 +0000] "GET /admin/v2/persistent/public/default/test-function-input-.../subscriptions HTTP/1.1" 200 52 "" "Pulsar-Java-v4.2.0-SNAPSHOT" 3
2026-02-05T17:22:10,004+0000 [pulsar-io-20-4] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.42.0.1:28761] connected with clientVersion=Pulsar-Java-v4.2.0-SNAPSHOT, clientProtocolVersion=21, proxyVersion=null
2026-02-05T17:22:10,139+0000 [pulsar-io-20-4] INFO  org.apache.pulsar.broker.service.ServerCnx - [[id: 0x183f927a, L:/10.42.0.5:16650 - R:/10.42.0.1:28761] [SR:10.42.0.1, state:Connected]] Subscribing on topic persistent://public/default/test-function-output-... / sub. consumerId: 0, role: null
2026-02-05T17:22:10,170+0000 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/test-function-output-...] Cursor sub recovered to position 9:-1
2026-02-05T17:22:10,171+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/test-function-output-...] Opened new cursor: ManagedCursorImpl{ledger=public/default/persistent/test-function-output-..., name=sub, ackPos=9:-1, readPos=9:0}
2026-02-05T17:22:10,172+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/test-function-output-...-sub] Rewind from 9:0 to 9:0
2026-02-05T17:22:10,172+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO  org.apache.pulsar.broker.service.persistent.PersistentSubscription - backlog for persistent://public/default/test-function-output-... - 0
2026-02-05T17:22:10,172+0000 [bookkeeper-ml-scheduler-OrderedScheduler-2-0] INFO  org.apache.pulsar.broker.service.ServerCnx - [/10.42.0.1:28761] Created subscription on topic persistent://public/default/test-function-output-... / sub
2026-02-05T17:22:10,232+0000 [pulsar-io-20-4] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-13 (v. Optional.empty)
2026-02-05T17:22:10,233+0000 [pulsar-io-20-4] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-13 (v. Optional.empty)
2026-02-05T17:22:10,234+0000 [pulsar-io-20-4] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [10.42.0.5:36313] for ledger: 13
2026-02-05T17:22:10,238+0000 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-14 (v. Optional.empty)
2026-02-05T17:22:10,238+0000 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-14 (v. Optional.empty)
2026-02-05T17:22:10,239+0000 [BookKeeperClientWorker-OrderedExecutor-2-0] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [10.42.0.5:36313] for ledger: 14
2026-02-05T17:22:10,244+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-15 (v. Optional.empty)
2026-02-05T17:22:10,245+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-15 (v. Optional.empty)
...
2026-02-05T17:22:30,844+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-6767 (v. Optional.empty)
2026-02-05T17:22:30,845+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-6767 (v. Optional.empty)
2026-02-05T17:22:30,845+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [10.42.0.5:36313] for ledger: 6767
2026-02-05T17:22:30,847+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleting path: /ledgers/idgen/ID-6768 (v. Optional.empty)
2026-02-05T17:22:30,847+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.pulsar.metadata.impl.AbstractMetadataStore - Deleted path: /ledgers/idgen/ID-6768 (v. Optional.empty)
2026-02-05T17:22:30,848+0000 [BookKeeperClientWorker-OrderedExecutor-3-0] INFO  org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [10.42.0.5:36313] for ledger: 6768

I was not yet able to reproduce this issue.