Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrading to 2.0.0 and using redis backend crashing. Seems to need upgraded redis client to 3.4.0 #239

Open
chrisxstyles opened this issue Sep 28, 2022 · 1 comment

Comments

@chrisxstyles
Copy link
Contributor

After upgrading to the 2.0.0 image, we saw the following crash in our superset-celery pods:

[2022-09-27 16:50:14,091: CRITICAL/MainProcess] Unrecoverable error: TypeError("__init__() got an unexpected keyword argument 'username'")
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 925, in create_channel
    return self._avail_channels.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 988, in get_connection
    connection = self._available_connections.pop()
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 332, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.8/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/connection.py", line 21, in start
    c.connection = c.connect()
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 428, in connect
    conn = self.connection_for_read(heartbeat=self.amqheartbeat)
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 434, in connection_for_read
    return self.ensure_connected(
  File "/usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py", line 460, in ensure_connected
    conn = conn.ensure_connection(
  File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 381, in ensure_connection
    self._ensure_connection(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 433, in _ensure_connection
    return retry_over_time(
  File "/usr/local/lib/python3.8/site-packages/kombu/utils/functional.py", line 312, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 877, in _connection_factory
    self._connection = self._establish_connection()
  File "/usr/local/lib/python3.8/site-packages/kombu/connection.py", line 812, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 949, in establish_connection
    self._avail_channels.append(self.create_channel(self))
  File "/usr/local/lib/python3.8/site-packages/kombu/transport/virtual/base.py", line 927, in create_channel
    channel = self.Channel(connection)
  File "/usr/local/lib/python3.8/site-packages/kombu/transport/redis.py", line 737, in __init__
    self.client.ping()
  File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 1037, in ping
    return self.execute_command('PING')
  File "/usr/local/lib/python3.8/site-packages/redis/client.py", line 772, in execute_command
    connection = pool.get_connection(command_name, **options)
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 990, in get_connection
    connection = self.make_connection()
  File "/usr/local/lib/python3.8/site-packages/redis/connection.py", line 1025, in make_connection
    return self.connection_class(**self.connection_kwargs)
TypeError: __init__() got an unexpected keyword argument 'username'

After some further investigation, we traced it back to the Redis client needing to accept the username parameter. The support for that was added in this commit which became version 3.4.0. The current docker image is using an older version of the Redis client

@amancevice
Copy link
Owner

amancevice commented Sep 28, 2022

Can you open a PR to update the Redis client? Thanks for digging into it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants