Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Possible misconfiguration of systemd unit #506

Open
renatogcarvalho opened this issue Oct 25, 2022 · 3 comments · May be fixed by #603
Open

[BUG] Possible misconfiguration of systemd unit #506

renatogcarvalho opened this issue Oct 25, 2022 · 3 comments · May be fixed by #603
Labels
Priority 2

Comments

@renatogcarvalho
Copy link

On CentOS linux distributions using SELinux the systemd unit doesn't have the required permissions to set the maxclients config via setrlimit(): https://github.com/Snapchat/KeyDB/blob/main/src/server.cpp#L3546

Below are the logs when the service was started:

2342:2342:M 25 Oct 2022 15:20:34.959 # Server can't set maximum open files to 65000 because of OS error: Operation not permitted.
2342:2342:M 25 Oct 2022 15:20:34.959 # Current maximum open files is 10240. maxclients has been reduced to 10208 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.

This happened even though we tuned the kernel with the below:

[root@ip-10-0-1-248 ~]# ulimit -n
500000

After inspecting the systemd unit configured on /lib/systemd/system/keydb.service and reading more about the CAP_SYS_RESOURCE setting on CapabilityBoundingSet this "should have worked". However, after arm-wrestling with this issue for a little while it only worked after I added:

AmbientCapabilities=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE

https://www.freedesktop.org/software/systemd/man/systemd.exec.html#AmbientCapabilities=

Here's my entire systemd unit configuration:

[Unit]
Description=Advanced key-value store
After=network.target
Documentation=https://docs.keydb.dev, man:keydb-server(1)

[Service]
Type=notify
ExecStart=/usr/bin/keydb-server /etc/keydb/keydb.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/var/run/keydb/keydb-server.pid
TimeoutStopSec=0
Restart=always
User=keydb
Group=keydb
RuntimeDirectory=keydb
RuntimeDirectoryMode=2755

UMask=007
PrivateTmp=yes
LimitNOFILE=65535
PrivateDevices=yes
ProtectHome=yes
ReadOnlyDirectories=/
ReadWriteDirectories=-/var/lib/keydb
ReadWriteDirectories=-/var/log/keydb
ReadWriteDirectories=-/var/run/keydb

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
AmbientCapabilities=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# keydb-server can write to its own config file when in cluster mode so we
# permit writing there by default. If you are not using this feature, it is
# recommended that you replace the following lines with "ProtectSystem=full".
ProtectSystem=true
ReadWriteDirectories=-/etc/keydb

[Install]
WantedBy=multi-user.target
Alias=keydb.service

Please note: I also had to adjust the Type to notify as I currently have Monit installed to verify the service health. It looks like this was worked by your team at some point but it was not made the default value: #352

KeyDB log after successful service restart:

1380:signal-handler (1666729774) Received SIGTERM scheduling shutdown...
1380:1388:M 25 Oct 2022 20:29:34.770 # User requested shutdown...
1380:1388:M 25 Oct 2022 20:29:34.770 * Removing the pid file.
1380:1388:M 25 Oct 2022 20:29:34.770 # KeyDB is now ready to exit, bye bye...
1503:1503:C 25 Oct 2022 20:29:34.906 # WARNING supervised by systemd - you MUST set appropriate values for TimeoutStartSec and TimeoutStopSec in your service unit.
1503:1503:C 25 Oct 2022 20:29:34.906 # oO0OoO0OoO0Oo KeyDB is starting oO0OoO0OoO0Oo
1503:1503:C 25 Oct 2022 20:29:34.906 # KeyDB version=6.3.1, bits=64, commit=ee16abf0, modified=0, pid=1503, just started
1503:1503:C 25 Oct 2022 20:29:34.906 # Configuration loaded
1503:1503:M 25 Oct 2022 20:29:34.908 * Increased maximum number of open files to 65032 (it was originally set to 10240).
@msotheeswaran-sc
Copy link
Collaborator

Closing as default value has been updated since.

@renatogcarvalho
Copy link
Author

renatogcarvalho commented Mar 12, 2023 •
edited
Loading

@msotheeswaran

It doesn't look like the values were updated?

https://github.com/Snapchat/KeyDB/blob/v6.3.2/pkg/rpm/keydb_build/keydb_rpm/usr/lib/systemd/system/keydb.service#L7

https://github.com/Snapchat/KeyDB/blob/v6.3.2/pkg/rpm/keydb_build/keydb_rpm/usr/lib/systemd/system/keydb.service#L29

@msotheeswaran-sc
Copy link
Collaborator

Looks like I misunderstood what was updated when the other issue was closed.

@msotheeswaran-sc msotheeswaran-sc linked a pull request Mar 13, 2023 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Priority 2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

update systemd type to notify and add capabilities Snapchat/KeyDB
2 participants
@renatogcarvalho @msotheeswaran-sc