Quarkus Container (running as an Azure Function App) fails to start when Instances are Scaling Up and/or Down

[vijay-902]

We are running Quarkus as an Azure Function App, using the "quarkus-azure-functions-http" plugin.
The Quarkus App is mostly Rest Client.
This has been working great in Azure Cloud.

But when we do load testing, which goes on for say 1-3 days, we start seeing that some of the containers fail to start or existing containers are restarting with errors.

So the Quarkus process has encountered and exception on startup and throws the "Quarkus bootstrap failed" exception.

Since it is running in the Azure Functions mode, the quarkus process doesn't exit, and the azure functions layer doesn't detect the quarkus failure, and attempt any restarts of the quarkus process.

So the Azure functions http layer is still running, accepting http requests, and it proxies the requests to quarkus process,
which in-turn returns the exception "Quarkus bootstrap failed".

When we see the Exception Stack Trace, there are different underlying causes,

• No CDI Container available
• NullPointerException (when initialising some startup data)
• NullPointerException (SmallRyeContextPropagationRecorder)
• etc.,

so no single reason/root cause.

The issue occurs when Azure is scaling up and scaling down as well.
For context, we are using Azure "App Service Plan". And the Min Instances is set at 3 and Max Instances is set to 13.

Most times it is limited to 1 or 2 containers, but we have seen cases, where most of the container instances end up in this state.

If we leave the instances without performing any load tests, they stay normal for days/weeks.

so its definitely occurring due to Azure Instances being scaled up/down.
We are working Azure support to debug, if there is anything wrong from there end.

Has anyone encountered similar issues when running Quarkus in Cloud environments ?

If and when the issue occurs in the Quarkus Startup Lifecycle, how do we catch this exception and return a standard error message, rather than returning the entire stack trace.

Also, would there be a way to detect this and exit the Quarkus process.

I have attached our process gradle file.

some sample exception stack traces.

[jwnmulder]

We are having the same issue. Both with dedicated plans and premium plans (all Windows). Running Quarkus on Java 11.

Still debugging as we speak but no improvements so far. Annoyingly, when it happens, Azure is not replacing it with a new instance and you really have to restart the function to get it working again. The Azure http layer is just timing out while trying to forward requests to Quarkus.

We tried changing the amount of function apps in a plan but that doesn't help. Plans with only one function and sufficient memory available exhibit the same problem.

Another error we are seeing

java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.azure.functions.resteasy.runtime.BaseFunction.initQuarkus(BaseFunction.java:51)
	at io.quarkus.azure.functions.resteasy.runtime.Function.run(Function.java:18)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
	at com.microsoft.azure.functions.worker.broker.EnhancedJavaMethodExecutorImpl.execute(EnhancedJavaMethodExecutorImpl.java:55)
	at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:54)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)
	at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
	at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)
	at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
	at io.quarkus.micrometer.runtime.binder.vertx.VertxMeterBinderRecorder.configureBinderAdapter(VertxMeterBinderRecorder.java:32)
	at io.quarkus.deployment.steps.VertxBinderProcessor$setVertxConfig325991315.deploy_0(Unknown Source)
	at io.quarkus.deployment.steps.VertxBinderProcessor$setVertxConfig325991315.deploy(Unknown Source)
	... 20 more

[rob-spoor]

@jwnmulder has been able to narrow this down to a very likely candidate, and I've created an issue for this: #23789.

[vijay-902]

@rob-spoor Thanks for quickly debugging the issue and making the fix.
Can you please provide any estimates on when we can see this in the release ?

[rob-spoor]

Unfortunately not. I'm a Quarkus user just like you. I did see that my PR (#23801) had some activity. It's best to keep watching that.

[rob-spoor]

@vijay-902 FYI, my PR was merged, and it seems like it will be backported to version 2.7.x.

[rob-spoor]

@vijay-902 I saw that Quarkus 2.7.2 has been released 2 days ago, and it includes my PR.