Update to Kombu 5.5.0 stopped processing SQS messages

[mdrachuk]

Hello.

Tbh I'm confused about the debug steps for this, because we didn't have any alerts or error logs. Only the actual usage showed that the tasks from the SQS weren't processing anymore.

Downgrade to 5.4.2 solved the issue for now.

Leaving this here for anybody having similar issues to see and maybe comment on details they can find.

[mgorven]

I'm seeing the same thing. It's not completely broken, but processing is extremely slow.

[Nusnus]

Can you please share a reproduction script or steps to reproduce, I'd like to solve this ASAP.

Thank you.

[mgorven]

Our workers process tasks from four standard SQS queues. Task execution time is fast (~50ms) and we use task_acks_late=True. We run with concurrency of eight and worker_prefetch_multiplier=16. With Kombu 5.4.2 one instance can process ~100 tasks per second, with 5.5.0 it drops to ~3/s. CPU usage remains low.

[Nusnus]

Ok, found it.

Throughput: 0.99 tasks per second with urllib3

python stress_test.py
Processed 50 tasks in 50.68 seconds
Throughput: 0.99 tasks per second

Throughput: 55.25 tasks per second with pycurl

python stress_test.py
Processed 50 tasks in 0.90 seconds
Throughput: 55.25 tasks per second

In PR #2134 we have migrated from pycurl to urllib3, by @spawn-guy.
Following this change, we’ve also removed pycurl from celery in PR celery/celery#9526 by @jmsmkn.

@mgorven

Our workers process tasks from four standard SQS queues. Task execution time is fast (~50ms) and we use task_acks_late=True. We run with concurrency of eight and worker_prefetch_multiplier=16. With Kombu 5.4.2 one instance can process ~100 tasks per second, with 5.5.0 it drops to ~3/s. CPU usage remains low.

By creating a quick stress_test.py to reproduce your case, I could easily see the problem using the latest version.
Reverting Commit 07c8852 from kombu and Commit 9bf0546 from celery was enough to fix the performance, as can be seen in the logs above.

Celery Worker

export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
celery -A myapp worker -l INFO

myapp.py

from celery import Celery
import time

app = Celery(
    "myapp",
    broker="sqs://",
    result_backend="redis://",
    task_acks_late=True,
    worker_prefetch_multiplier=16,
)


@app.task
def add(x, y):
    time.sleep(0.05)  # Simulate 50ms task execution time
    return x + y


if __name__ == "__main__":
    app.start()

stress_test.py

export AWS_DEFAULT_REGION=us-east-1
export AWS_ACCESS_KEY_ID=test
export AWS_SECRET_ACCESS_KEY=test
export AWS_ENDPOINT_URL=http://localhost:4566
python stress_test.py

from myapp import add
from celery import group
import time

num_tasks = 50
tasks = [add.s(1, 2) for _ in range(num_tasks)]
g = group(tasks)
start_time = time.time()
result = g.apply_async()
result.get()
end_time = time.time()
total_time = end_time - start_time
throughput = num_tasks / total_time
print(f"Processed {num_tasks} tasks in {total_time:.2f} seconds")
print(f"Throughput: {throughput:.2f} tasks per second")

We have had multiple pre-releases, both in Celery and Kombu, and this is the first time we have received such a report.

—

@spawn-guy, I wonder if there’s anything we can tweak to fix this instead of reverting back to pycurl.
As this was discovered after the final v5.5.0 release, this issue is a bit urgent.
Would you mind giving it a look please?
If you’re unavailable, please let me know so I can revert the changes and issue a new release, although this is the least wanted outcome.

@auvipy I’d be glad to hear your feedback as well. Thanks.

Note to self: How come we didn’t fall on this in the CI 🤔
I might need to enable SQS in pytest-celery and add a smoke test to the CI to reproduce this case and detect performance failures.
The issue is that SQS support is currently experimental in pytest-celery, and I don’t have the capacity to fix any problems that may arise (IIRC there are issues when using SQS/Redis in the CI, potentially due to a genuine bug).

[auvipy]

the performance regressions are hard to address in short notice to be honest

[auvipy]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

[auvipy]

If we can't figure out by next 1 or 2 weeks, we should revert this #2261

[Nusnus]

@auvipy

If we can't figure out by next 1 or 2 weeks, we should revert this #2261

Unfortunately we don't have the luxury of time my friend. I will take care of everything and release a fixed version on Monday.

I just want to give it a few days for @spawn-guy to check this out, but per my script above we can confirm the issue is only due to the dep changes.

[Nusnus]

the performance regressions are hard to address in short notice to be honest

I released it before the weekend knowing it might have issues. I am prepared to respond quickly.

I take responsibility, please don't stress yourself over the weekend 🙏

[Nusnus]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

Any recommendation?

[auvipy]

the performance regressions are hard to address in short notice to be honest

I released it before the weekend knowing it might have issues. I am prepared to respond quickly.

I take responsibility, please don't stress yourself over the weekend 🙏

there is no need to feel guilty. celery code base is hard.

[auvipy]

I think we have to stick to special implementation of pycurl until we have our homegrown faster or equal alternative of pycurl. some previous attempts also faced the serious performance regressions

Any recommendation?

I have to test alternatives, right now I think we should revert back to pycurl as it was working fine as far as I know.

[spawn-guy]

the main difference between pycurl and urllib3 was is using the available urllib3.PoolManager instead of custom pycurl-pool implementation.

@mdrachuk what is your use-case? does this pr-revert fix your problem?

@Nusnus have you tried tuning the max_clients setting?

i am running my kombu fork (not the rc versions) and things seem to be processed fine, but not at the speeds mentioned

[spawn-guy]

another idea i have is about the SSL-certificates and optional cert package importing

[spawn-guy]

here are my "from-home" on Windows tests of the code here #2258 (comment)

celery -A myapp worker -l INFO -P gevent

kombu = "*"

max_clients=10

Processed 50 tasks in 4.41 seconds
Throughput: 11.35 tasks per second

Processed 50 tasks in 4.29 seconds
Throughput: 11.65 tasks per second

Processed 50 tasks in 3.77 seconds
Throughput: 13.26 tasks per second

Processed 50 tasks in 3.93 seconds
Throughput: 12.72 tasks per second

Processed 50 tasks in 4.25 seconds
Throughput: 11.78 tasks per second

FORKED_BY_MULTIPROCESSING - didn't do anything

kombu = "*"

Processed 50 tasks in 4.51 seconds
Throughput: 11.09 tasks per second

Processed 50 tasks in 4.44 seconds
Throughput: 11.26 tasks per second

Processed 50 tasks in 4.00 seconds
Throughput: 12.50 tasks per second

max_clients: int = 100

Processed 50 tasks in 3.67 seconds
Throughput: 13.64 tasks per second

Processed 50 tasks in 4.24 seconds
Throughput: 11.79 tasks per second

Processed 50 tasks in 4.56 seconds
Throughput: 10.97 tasks per second

Processed 50 tasks in 4.18 seconds
Throughput: 11.95 tasks per second

max_clients: int = 1

Processed 50 tasks in 3.87 seconds
Throughput: 12.93 tasks per second

Processed 50 tasks in 4.00 seconds
Throughput: 12.51 tasks per second

Processed 50 tasks in 4.51 seconds
Throughput: 11.08 tasks per second

kombu = "5.4.2"
celery = { extras = ["sqs", "redis"] }

max_clients: int = 10

Processed 50 tasks in 4.03 seconds
Throughput: 12.41 tasks per second

Processed 50 tasks in 4.50 seconds
Throughput: 11.12 tasks per second

Processed 50 tasks in 4.52 seconds
Throughput: 11.06 tasks per second

max_clients: int = 1

Processed 50 tasks in 4.13 seconds
Throughput: 12.11 tasks per second

Processed 50 tasks in 4.17 seconds
Throughput: 11.99 tasks per second

Processed 50 tasks in 4.21 seconds
Throughput: 11.88 tasks per second

Processed 50 tasks in 4.06 seconds
Throughput: 12.33 tasks per second

i dunno. is it Windows? is it gevent?

billard version is failing Windows with Access denied

\Lib\site-packages\billiard\pool.py", line 406, in _ensure_messages_consumed
    if self.on_ready_counter.value >= completed:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 3, in getvalue
PermissionError: [WinError 5] Access is denied

all speeds seem to be the same. with or without urllib3/pycurl on both kombu versions: latest and 5.4.2

help!

[spawn-guy]

main.py config is a bit different.

app = Celery(
    "myapp",
    broker="sqs://",
    # result_backend="redis://",
    result_backend="redis://localhost:6379/1",  # local docker
    task_acks_late=True,
    worker_prefetch_multiplier=16,
    task_default_queue="default_static",
    task_queues={
        "default_static": {
            "exchange": "default_static",
            "routing_key": "default_static",
        }
    },
    broker_transport_options={
        # # !!!! same as in AWS-SQS: Visibility timeout. match the time of the longest ETA you’re planning to use.
        # # chatGPT: Set the SQS visibility timeout on the queue to be slightly longer than the longest running task.
        # "visibility_timeout": 1200,  # seconds.
        # # Celery: seconds to sleep between unsuccessful polls
        # "polling_interval": 1,  # seconds.
        # # !!!! same as in AWS-SQS: Receive message wait time
        # "wait_time_seconds": 20,  # seconds.
        "region": "eu-west-1",
        "predefined_queues": {
            "default_static": {
                "url": 'https://sqs.eu-west-1.amazonaws.com/00000accNumber/existing-queue',
            }
        }
    }
)

aws access keys are picked up from a default location. no extra env vars are set.
python 3.11

[auvipy]

May be we should highlight these suggestions you shared for people to avoid these performance issues

[spawn-guy]

@auvipy i'd like to hear more use-case specifics from @mdrachuk and @mgorven : environments, os'es, python versions. with-or-without a proxy (as i didn't test that one out)
i'd like to figure the reason of slowdown out.
not that it was unexpected. it was undesired, but possible

i am running not-high-frequency tasks(1+ seconds) on aws elastic beanstalk py3.11 instances with the urllib3 version of kombu and worker_pool=aio_pool.pool.AsyncIOPool, in eu-west-1 region for a month now (since the PR) and didn't notice any errors

another reason of slowdown with max_clients=10 could be - creation of many dynamic queues and pool-draining. but after worker init it shouldn't be a problem
and i think urllib3.PoolManager should re-use the established sqs http connection. this should cover the message-updates/creation

maybe pycurl returns the responses earlier in the http lifecycle. maybe this needs a rewrite on kombu side for urllib3.

it's like, i can agree with slowdown,
but a complete "not working" stage - i doubt it. unless there is a proxy involved ()

maybe, some configuration issue or aws outage or WAF interference on seeing a new User-Agent instead of an old one.

[spawn-guy]

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients.
but i am still waiting for a long promised out-of-the-box asyncio support.
and i'm really looking forward to the outcome of recent @Nusnus experiments ;)

[auvipy]

we will revisit our recent and old experiments for introducing native async support in v6.0. but we have to reach to a consensus for this exact issue for now with recent changes

[auvipy]

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients. but i am still waiting for a long promised out-of-the-box asyncio support.

we can also try httpx and see later

[mgorven]

@spawn-guy We're using Python 3.11.11 on Debian Bookworm/12 aarch64. No proxy for SQS.

[Nusnus]

// On Topic

We will revert back to pycurl and release v5.6.
v5.7+ will be able to have an alternative if it fits better than both of these.

// Off Topic

@spawn-guy

as a side-note:

i am also thinking about aiohttp+aiobotocore in urllib3+botocore place, as i heard that it is even faster than other py-clients. but i am still waiting for a long promised out-of-the-box asyncio support. and i'm really looking forward to the outcome of recent @Nusnus experiments ;)

I am trying to be as creative as I’ve ever been in my life to solve the challenges of migrating to asyncio. It became a mission for me. It requires solving a completely different core challenge first, which makes the difficulty extremely high and multi-dimensional, but this only makes it more attractive tbh 😉

EDIT:
Grok 3, ChatGPT 4.5 and the Google one (lol) all say my plan to solve both+ challenges should work within a reasonable time frame. I’m still working on the execution plan but if they don’t lie, I might be on to something.
I’m also exploring other approaches to address it, so there’s more happening behind the scenes.
I’ll stick with whatever works first 😄

[spawn-guy]

v5.7+ will be able to have an alternative if it fits better than both of these.

i started to have an idea to give an option to pick the client implementation with a config setting.

because i am not rolling back :) i don't want to re-complie curl for Amazon Linux 2023 each time an instance is deployed (as pycurl is available but compiled without open-ssl)

so, shall we instead of reverting things back - add an client-selection option now? and keep both clients and make the pycurl the default with urllib3 as an option and possible other http client implementations later (aiohttp and httpx)?

[Nusnus]

@spawn-guy

v5.7+ will be able to have an alternative if it fits better than both of these.

i started to have an idea to give an option to pick the client implementation with a config setting.

because i am not rolling back :) i don't want to re-complie curl for Amazon Linux 2023 each time an instance is deployed (as pycurl is available but compiled without open-ssl)

so, shall we instead of reverting things back - add an client-selection option now? and keep both clients and make the pycurl the default with urllib3 as an option and possible other http client implementations later (aiohttp and httpx)?

Very interesting. I like your attitude :)
Do you think it can be done in a timly manner? we need to provide a viable solution with pycurl soon. Having that support already for v5.6 with a fast release can be very nice if done correctly.

EDIT:
The time pressure comes from the release cycle of Celery which is now blocked due to the pycurl dep change, so we need it to be resolved pretty fast as the release is planned for the end of this month.

[auvipy]

That's too late for celery v5.5. As it is clear that new change has introduced performance regression with default mechanisms. So It's best to revert now and reconsider for 5.7

[Nusnus]

@spawn-guy

v5.7+ will be able to have an alternative if it fits better than both of these.

i started to have an idea to give an option to pick the client implementation with a config setting.
because i am not rolling back :) i don't want to re-complie curl for Amazon Linux 2023 each time an instance is deployed (as pycurl is available but compiled without open-ssl)
so, shall we instead of reverting things back - add an client-selection option now? and keep both clients and make the pycurl the default with urllib3 as an option and possible other http client implementations later (aiohttp and httpx)?

Very interesting. I like your attitude :) Do you think it can be done in a timly manner? we need to provide a viable solution with pycurl soon. Having that support already for v5.6 with a fast release can be very nice if done correctly.

EDIT: The time pressure comes from the release cycle of Celery which is now blocked due to the pycurl dep change, so we need it to be resolved pretty fast as the release is planned for the end of this month.

I might also need to apply whatever solution you’d bring to pytest-celery, as pycurl was also removed from there in this effort.

[Nusnus]

@auvipy

That's too late for celery v5.5. As it is clear that new change has introduced performance regression with default mechanisms. So It's best to revert now and reconsider for 5.7

That’s my reasoning too. Enabling both by choice with pycurl as default can be an acceptable middle ground. WDYT?

[auvipy]

Yes

[spawn-guy]

sure.
revert now -> introduce choice later.
i'll lock the version on my side (or switch back to my fork) on my environments

and will still try to reproduce the slowdown today/tomorrow. no luck on windows so far.

[millerdev]

Would it be recommended to use 5.4.2 on Python 3.13 to avoid this issue, given that 5.5 is the first version of Kombu to officially support Python 3.13? If not, would it be possible to get a 5.4.3 release with support for Python 3.13 until 5.5+ has stabilized?

[spawn-guy]

As it is clear that new change has introduced

it is not so clear to me anymore.

first: @Nusnus how many times have you tried your test?

second: "our" urllib3 client seems to be only used to FETCH and SEND messages. on DeleteMessage from the queue - the boto3.Session is used and i don't see any urllib3_client logs that i've added

i've deployed things to amazon linux 2023 and now running more iterations of the same test from y home windows(cloud-home delay).

celery -A myapp worker -l DEBUG -Q default_static
total 4 ForkPoolWorkers start.

from datetime import datetime

from myapp import add
from celery import group
import time

num_iters = 10
num_tasks = 500

all_times = []

tasks = [add.s(1, 2) for _ in range(num_tasks)]

print("start,run,time,throughput")
for i in range(num_iters):
    g = group(tasks)
    start_dtime = datetime.now()
    start_time = time.time()
    result = g.apply_async()
    result.get()
    end_time = time.time()
    total_time = end_time - start_time
    throughput = num_tasks / total_time
    all_times.append(total_time)
    print(f"{start_dtime},{i},{total_time},{throughput}")

print(f"iterations: {num_iters:.2f}")
print(f"tasks in iteration: {num_tasks:.2f}")
print(f"total tasks: {num_tasks * num_iters}")
print(f"total time: {sum(all_times):.2f} seconds")
print(f"average time of iteration: {sum(all_times) / len(all_times):.2f} seconds")
print(f"average throughput: {num_tasks * num_iters / sum(all_times):.2f} tasks per second")

testing the speed of urllib3.

start,run,time,throughput
2025-03-21 12:56:43.979081,0,29.73174262046814,16.81704319799225
2025-03-21 12:57:13.710823,1,23.604838371276855,21.18209801463485
2025-03-21 12:57:37.315662,2,23.39216923713684,21.37467435923865
2025-03-21 12:58:00.707831,3,24.549772262573242,20.36678770997248
2025-03-21 12:58:25.257603,4,24.756406545639038,20.196792255703098
2025-03-21 12:58:50.014010,5,33.352821588516235,14.99123540936506
2025-03-21 12:59:23.366831,6,42.59526753425598,11.73839322872875
2025-03-21 13:00:05.962099,7,42.99054718017578,11.630463736701747
2025-03-21 13:00:48.952646,8,43.111043214797974,11.597956410119385
2025-03-21 13:01:32.063689,9,33.36323094367981,14.986558131736276
iterations: 10.00
tasks in iteration: 500
total tasks: 5000
total time: 321.45 seconds
average time of iteration: 32.14 seconds
average throughput: 15.55 tasks per second

the numbers fluctuate too much from 11 to 21 tps

and 500 tasks seem to work faster than 50

start,run,time,throughput
2025-03-21 13:08:27.189244,0,25.04867458343506,1.99611360008108
2025-03-21 13:08:52.237918,1,3.1373138427734375,15.937200581692258
2025-03-21 13:08:55.375233,2,2.7751967906951904,18.01674035068156
2025-03-21 13:08:58.150429,3,12.57353949546814,3.976604997981787
2025-03-21 13:09:10.723969,4,3.0881454944610596,16.19094699057434
2025-03-21 13:09:13.812731,5,2.7244091033935547,18.35260348297891
2025-03-21 13:09:16.537141,6,22.50865936279297,2.2213673055379957
2025-03-21 13:09:39.045800,7,3.0558571815490723,16.36202120370496
2025-03-21 13:09:42.101657,8,12.132184267044067,4.121269418551471
2025-03-21 13:09:54.233842,9,3.0647895336151123,16.3143339702749
iterations: 10
tasks in iteration: 50
total tasks: 500
total time: 90.11 seconds
average time of iteration: 9.01 seconds
average throughput: 5.55 tasks per second

the numbers fluctuate too much from 1.9 to 18.3 tps

[spawn-guy]

As I have pycurl deployment problems again - I am thinking about the fixing strategy.

I will roll back the deletion of pycurl and dependencies.
But then.. I want to keep both options available for (me).

The best way would be to introduce choice via celery configure, like pool choice. But I don't have enough time to do this.
Instead I am going to implement a kombu fallback to urllib3 (as it is required by boto3) when pycurl is not detected (in a way similar to certifi and ssl).

But I need some advice on package dependency : we now use sqs extra, that required pycurl. And I have pycurl problems on instance deployment.
My ideas are:

• add pycurl back to sqs.txt and add sqs-urlib3 or sqs-clean as an extra without pycurl. This should cover existing users to use pycurl and me to use urllib3 instead
• or add http-pycurl extra. But this will require current users to add one extra to their kombu/celery dependency

The ci will use pycurl by default

What do you say?
I'm in favor of pycurl extra

[spawn-guy]

initial code here #2269

when pycurl is found it will be used as a http client. i can't test with pycurl as it brings my old struggle back.
i'll test urllib3-fallback tomorrow to see if it works

i've picked the 3rd route to requirements management:
to use pycurl as http client - it needs to be installed/required by user :)

[spawn-guy]

so i've deployed the "fallback urllib3" version to my Amazon Linux 2023 instances and the code seems to run fine without pycurl

@mgorven will you be so kind to test the branch with pycurl manually installed to see if it works and if there is speed difference?

[spawn-guy]

i have been thinking and commenting with @jmsmkn and..

the thing is.. i can't reliably state that pycurl-urllib3 is the problem here. 'cause my change was not the only one since 5,4,2.

on the other hand i have also enabled ssl connections for sqs which were disabled instead of "fixed" previously.
might it be that this is the actual "not working" and "slow" problem is ssl-with-optional-certifi enablement? too many requests to filesystem?!

it looks like i still need to test pycurl on aws anyway :( and also if ssl works correctly

[mgorven]

@mgorven will you be so kind to test the branch with pycurl manually installed to see if it works and if there is speed difference?

The branch doesn't work at all: https://gist.github.com/mgorven/f1689323acb1a4e981644dfc9afe87ab

This is using rev 77ca118 and pycurl 7.45.6.

[spawn-guy]

@mgorven thanks for the feedback i'll look into it today. even though this is the code form 5.4.2 😝