Is --tuning-squid supposed to work with the mgrpxy install podman command?

[ppanon2022]

I was trying to figure out how to increase squid timeouts for one of our proxies that has to deal with longer WAN delays. Poking about the container with podman exec -ti uyuni-proxy-squid /bin/bash, I found /etc/squid/conf.d/squid_example.conf, which says

# Do not change this file in your environment, since
# it's not persisted and changes would be lost on restart.
# Custom squid configuration file can be added using
# --tuning-squid flag on mgrpxy install or mgrpxy upgrade.
# e.g.
# mgrpxy install [podman/kubernetes] --tuning-squid my_local_squid_configuration.conf config.tar.gz
# mgrpxy upgrade [podman/kubernetes] --tuning-squid my_local_squid_configuration.conf

So I tried to to create a squid supplemental configuration that contained

connect_timeout 20 minutes
read_timeout 20 minutes
pconn_timeout 20 minutes

but when I tried to do a mgrpxy upgrade or a remove/reinstall, the pod self destructed and only the <some hex number>-infra container was left standing. I had to do an uninstall+purge and reinstall without the extending conf file to get it to recover.

Is that approach/option just not supported/working yet? Or should it be working, but something about those specific option settings is breaking the installation?

[rjmateus]

This property should be working, and if causing the service to not start properly then it's a bug. Do you have any logs for the squid systems service? The command to show the logs should be journalctl -u uyuni-proxy-squid.service

[ppanon2022]

Since the container doesn't come up, I had to use the mgrpxy logs to get the clue. It says

Dec 12 17:54:24 ca07uyproxy1.semnet.dom sh[2719676]: 679247d2bb840544282feda1c04fbb78f6862066aac89a569b58ed6db03fa150
Dec 12 17:54:24 ca07uyproxy1.semnet.dom systemd[1]: Started Uyuni proxy squid container service.
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| Processing Configuration File: /etc/squid/squid.conf (depth 0)
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| WARNING: (B) '::/0' is a subnetwork of (A) '::/0'
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| WARNING: because of this '::/0' is ignored to keep splay tree searching predictable
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| WARNING: You should probably remove '::/0' from the ACL named 'all'
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| Processing Configuration File: /etc/squid/conf.d/squid_example.conf (depth 1)
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| Processing Configuration File: /etc/squid/conf.d/squid_tuning.conf (depth 1)
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| Not currently OK to rewrite swap log.
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| storeDirWriteCleanLogs: Operation aborted.
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| FATAL: Unable to open configuration file: /etc/squid/conf.d/squid_tuning.conf: (13) Permission denied_
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| Squid Cache (Version 6.10): Terminated abnormally.
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: CPU Usage: 0.007 seconds = 0.002 user + 0.005 sys
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: Maximum Resident Size: 55296 KB
Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: Page faults with physical i/o: 0

The interesting thing is that my config file is not called squid_tuning.conf. I'm presuming it's trying to rename it during a mapping process into the container? I originally had thought it would just copy it into the overlay.

I created a /etc/squid/ directory on the host, moved the config file there, used an absolute path for the config file in the mgrpxy command, and that worked. Thank you for your help on that part.

Looks like the mod_wsgi handler is still timing out and failing though. Any idea on how to increase the timeout for that when dealing with slower WAN speeds? In particular, how to deal with it so it will survive a container restart or update? Is there a similar mechanism to --tuning-squid?

f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.errors: <_io.TextIOWrapper name='<wsgi.errors>' encoding='utf-8'>
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.file_wrapper: <class 'mod_wsgi.FileWrapper'>
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.input: <rhn.SmartIO.SmartIO object at 0x7fb1db4608d0>
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.input_terminated: True
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.multiprocess: True
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.multithread: False
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.run_once: False
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.url_scheme: https
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] \twsgi.version: (1, 0)
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] Exception Handler Information
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] Traceback (most recent call last):
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/share/rhn/proxy/rhnShared.py", line 235, in _serverCommo
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     status, headers, bodyFd = self._proxy2server()
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/share/rhn/proxy/rhnShared.py", line 400, in _proxy2server
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     response = http_connection.getresponse()
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/lib/python3.6/site-packages/rhn/connections.py", line 115, in getresponse
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     response.begin()
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/lib64/python3.6/http/client.py", line 320, in begin
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     version, status, reason = self._read_status()
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]   File "/usr/lib64/python3.6/socket.py", line 586, in readinto
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200]     return self._sock.recv_into(b)
f1e73152c572 [proxy-broker] [Thu Dec 12 23:30:29 2024] [error] [pid 55] src/server/wsgi_logger.c(147): [client 10.155.4.22:61200] socket.timeout: timed out

rhnShared.py seems to pull a time out configuration value from the rhn.conf file, presumably matching the proxy.timeout value mentioned in https://www.uyuni-project.org/uyuni-docs/en/uyuni/administration/troubleshooting/tshoot-rpctimeout.html. However that config file is in the uyuni-proxy-httpd container, and doesn't have any proxy.timeout option set, so how do we modify/add to it in a persistent way? Could /etc/rhn/rhn.conf be externalized into the host file system so that it can be modified? Or can we get a --tuning-proxy option for a file that gets appended to /etc/rhn/rhn.conf?

[rjmateus]

Lets on the first isue from squid first. you have the error in the logs

Dec 12 17:54:25 ca07uyproxy1.semnet.dom uyuni-proxy-squid[2719774]: 2024/12/12 22:54:25| FATAL: Unable to open configuration file: /etc/squid/conf.d/squid_tuning.conf: (13) Permission denied_

The file on the host OS is mapped to /etc/squid/conf.d/squid_tuning.conf inside the container. Check the file permission on your host OS and if you need to re-label it because of SE Linux. You can try manually re-label the file with the command chcon -R -t container_file_t PATHTOFILE/

After this restart the containers.

[ppanon2022]

Ah of course, SELinux contexts. I usually check the audit logs on permissions issues but hadn't thought of that this time. As I said, I created a squid directory in the host under /etc, and then created the file in there. It must have picked up the etc_t context as a result instead of user_home_t or admin_home_t, and that was sufficient. Also the advantage of having the file work under /etc with etc_t, is that the context won't change if the defaults get re-applied by a restorecon job.

I edited that response as I figured it out after getting the proxy logs. Because the containers weren't showing at all, I hadn't thought there would be anything useful in the mgrpxy logs output until your suggestion, then followed the breadcrumbs I found. Sorry if I wasn't clear.

TL;DR, I got lucky on the SELinux permissions on my second go round.
That's why I included the follow-up question on rhn.conf when the squid config settings "worked" but weren't enough.

[ppanon2022]

Lets on the first issue from squid first. you have the error in the logs

Note that, while I've marked the original question as answered I still would appreciate help on how to tune the timeout value in /etc/rhn/rhn.conf since it appears that modifying the squid tuning file is insufficient. If there isn't a --tuning-rhnconf option, can /etc/rhn/rhn.conf in the container be mapped to an rhn.conf file in the container host to make it possible to update?

[ppanon2022]

Hi Ricardo (@rjmateus),

Any suggestions on how to make sticky changes like the timeout setting to the /etc/rhn/rhn.conf file in the proxy container? Is it mapped to an external persistent file with 2024.12 and how would you change it? When I last tried to do it, I couldn't seem to use the podman tools to attach to the container in a way that I would have permissions to modify the file. While you answered my original question, that one change isn't enough to fix the problem which spawned it - something I couldn't know when I asked it. It's silly to have to open a new discussion when anyone who needs to increase timeouts due to WAN lag/throughput bottlenecks will also need to change both files.

[rjmateus]

Hey @ppanon2022 yo don't need to open a new question. Which parameter are you looking to change in that file? We current have no way to change any configuration in there?
Is your goal to the apache timeout?

[ppanon2022]

I'm guessing that's it, yes. As you can see from the wsgi logs in here, even though I made the changes in the squid configuration with --tuning-squid. I am still getting timeouts in the mod_wsgi handler (the python redirection code?). The troubleshooting page suggests changing the server.timeout and proxy.timeout values in rhn.conf, so I would like to try to change those, assuming those are the correct settings, but don't seem to have a way to do it. When I was looking at the python handler code, it did seem to be pulling the timeout value from some sort of configuration object and I was hoping that proxy.timeout was its source. Unlike the page's suggestion however, I want to increase the timeout to give the package files more time to download - they're already failing and I want them to succeed.

Generally, the timeouts happen with the Ubuntu firmware and kernel packages downloads, which are some of the largest and longest to transfer. But they're also pretty critical - I can't just skip them.