Skip to content

Commit d4de59b

Browse files
decarvdecarv
committed
WIP: Add new custom event loop for I/O layer
Introduce ev.h and ev.c, establishing the foundation for the new custom event loop, `pgagroal_ev`. Replace previous dependencies on libev with the custom event loop. For Linux, implement support for io_uring with a fallback to epoll if io_uring is unavailable. For BSD, implement support for kqueue. Changelog ========= 2025-01-13: - Fix ASan report in `pgagroal_ev_io_stop()` 2025-01-16: - Remove `pgagroal_ev_*_stop()` functions from main.c and vault.c as these functions are not intended to be called from a child process inheriting the ev loop. This does not make sense in io_uring and for epoll and kqueue, just closing the fds is enough as a precaution to prevent their use. - Enhance debugging steps in `sigchld_handler()` to help debug child processes termination. - Add error handling to `kill()` to help debug child processes termination. - Allow return error on `pgagroal_ev_io_stop()` instead of exit so the caller function can handle errors appropriately. - Add a call to `setpgid()` when worker is created to ensure child processes do not exit immediately if a SIGINT is issued in the controlling terminal. - Add a call to `pgagroal_ev_fork()` when a worker is created. 2025-01-17: - Fix regression added to the Debug build. `waitpid()` inside `#if DEBUG` would block waiting for the running child processes. Add WNOHANG to the function options so the main process does not block. - Remove '\n' from `pagroal_log_*()` - Run uncrustify.sh - Remove `printf()` debugging - Modify CD/CI pipeline to improve function verify_running and enable the CD/CI for kqueue, which was disabled.
1 parent 92ac986 commit d4de59b

40 files changed

+5029
-597
lines changed

‎.github/ISSUE_TEMPLATE/bug_report.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,9 @@ What is the version of pgagroal ?
2323

2424
What is the version of PostgreSQL ?
2525

26-
**libev**
26+
**liburing**
2727

28-
What is the version of libev ?
28+
What is the version of liburing ?
2929

3030
**OpenSSL**
3131

‎.github/config/pg_hba.conf

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
local all all trust
2+
host all all all trust
3+
local replication all peer
4+
host replication all all trust

‎.github/workflows/ci.yml

Lines changed: 183 additions & 62 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ on:
1111
jobs:
1212
build-linux:
1313

14-
runs-on: ubuntu-latest
14+
runs-on: ubuntu-24.04
1515

1616
steps:
1717
- uses: actions/checkout@v3
@@ -21,8 +21,8 @@ jobs:
2121
sudo wget --quiet --output-document /etc/apt/trusted.gpg.d/apt.postgresql.org.asc https://www.postgresql.org/media/keys/ACCC4CF8.asc
2222
- name: Update system
2323
run: sudo apt update
24-
- name: Install libev
25-
run: sudo apt install -y libev4 libev-dev
24+
- name: Install liburing
25+
run: sudo apt install -y liburing-dev
2626
- name: Install systemd
2727
run: sudo apt install -y libsystemd-dev
2828
- name: Install rst2man
@@ -37,14 +37,43 @@ jobs:
3737
run: sudo apt install graphviz
3838
- name: Install doxygen
3939
run: sudo apt install doxygen
40+
- name: Install crudini
41+
run: sudo apt install -y crudini
4042
- name: Install clang
4143
run: sudo apt install -y clang
4244
- name: Install PostgreSQL
43-
run: sudo apt install -y postgresql
44-
- name: Start postgres
45+
run: sudo apt install -y postgresql postgresql-contrib
46+
- name: Start postgres & setup test table
4547
run: |
4648
version=$(pg_config --version | grep -Eo "[0-9]{1,2}" | head -1)
47-
sudo -u postgres /usr/lib/postgresql/${version}/bin/pg_ctl start -D /etc/postgresql/${version}/main/
49+
sudo cp pg_hba.conf /etc/postgresql/${version}/main
50+
sudo -u postgres /usr/lib/postgresql/${version}/bin/pg_ctl start -D /etc/postgresql/${version}/main/ -o "-p 5432"
51+
netstat -tuln | grep '127.0.0.1:5432' || (echo "Nothing is listening on 127.0.0.1:5432"; exit 1)
52+
netstat -tuln | grep '::1:5432' || (echo "Nothing is listening on ::1:5432"; exit 1)
53+
PGPASSWORD="postgres" pgbench -i -s 1 -h localhost -p 5432 -U postgres -d postgres
54+
working-directory: /home/runner/work/pgagroal/pgagroal/.github/config/
55+
- name: Define functions `verify_running` and `verify_shutdown`
56+
run: |
57+
echo 'verify_running() {
58+
echo "Confirming pgagroal is listening on port 2345"
59+
netstat -tuln | grep "127.0.0.1:2345" || (echo "Nothing is listening on 127.0.0.1:2345"; exit 1)
60+
netstat -tuln | grep "::1:2345" || (echo "Nothing is listening on ::1:2345"; exit 1)
61+
echo "[*] Running pgagroal-cli ping"
62+
./pgagroal-cli ping
63+
echo "[*] Running queries with psql"
64+
PGPASSWORD="postgres" psql -h 127.0.0.1 -p 2345 -U postgres -d postgres -c "SELECT * FROM pgbench_accounts LIMIT 50;" > /dev/null
65+
PGPASSWORD="postgres" psql -h ::1 -p 2345 -U postgres -d postgres -c "SELECT * FROM pgbench_accounts LIMIT 50;" > /dev/null
66+
}
67+
68+
verify_shutdown() {
69+
echo "[*] Running pgagroal-cli shutdown immediate"
70+
./pgagroal-cli shutdown immediate
71+
sleep 5
72+
echo "[*] Confirming there are no dangling pgagroal processes"
73+
pgrep pgagroal > /dev/null && echo "[E] Dangling pgagroal child processes: $(wc -l < <(pgrep pgagroal))" && exit 1
74+
echo "rm -f /tmp/pgagroal.2345.pid"
75+
rm -f /tmp/pgagroal.2345.pid
76+
}' > /tmp/functions.sh
4877
- name: GCC/mkdir
4978
run: mkdir build
5079
working-directory: /home/runner/work/pgagroal/pgagroal/
@@ -54,28 +83,52 @@ jobs:
5483
- name: GCC/make
5584
run: make
5685
working-directory: /home/runner/work/pgagroal/pgagroal/build/
57-
- name: GCC/Run pgagroal & confirm pgagroal is running
86+
- name: GCC/Run pgagroal as daemon with 'io_uring' backend
5887
run: |
5988
sudo mkdir -p /etc/pgagroal
89+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_type file
90+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_path /dev/null
91+
crudini --set ../../doc/etc/pgagroal.conf pgagroal ev_backend io_uring
92+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
6093
sudo cp ../../doc/etc/*.conf /etc/pgagroal
61-
./pgagroal >> /dev/null 2>&1 &
62-
pid=$!
94+
sudo sysctl kernel.io_uring_disabled=0
95+
./pgagroal -d || exit 1
6396
sleep 5
64-
./pgagroal-cli ping
6597
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
66-
- name: GCC/Stop pgagroal & postgres
98+
- name: GCC/Run verify_running for 'io_uring' backend
6799
run: |
68-
./pgagroal-cli shutdown
69-
version=$(pg_config --version | grep -Eo "[0-9]{1,2}" | head -1)
70-
sudo -u postgres /usr/lib/postgresql/${version}/bin/pg_ctl stop -D /etc/postgresql/${version}/main/
100+
source /tmp/functions.sh
101+
verify_running
102+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
103+
- name: GCC/Run verify_shutdown for 'io_uring' backend
104+
run: |
105+
source /tmp/functions.sh
106+
verify_shutdown
107+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
108+
- name: GCC/Run pgagroal as daemon with 'epoll' backend
109+
run: |
110+
sudo mkdir -p /etc/pgagroal
111+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_type file
112+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_path /dev/null
113+
crudini --set ../../doc/etc/pgagroal.conf pgagroal ev_backend epoll
114+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
115+
sudo cp ../../doc/etc/*.conf /etc/pgagroal
116+
./pgagroal -d || exit
117+
sleep 5
118+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
119+
- name: GCC/Run verify_running for 'epoll' backend
120+
run: |
121+
source /tmp/functions.sh
122+
verify_running
123+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
124+
- name: GCC/Run verify_shutdown for 'epoll' backend
125+
run: |
126+
source /tmp/functions.sh
127+
verify_shutdown
71128
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
72129
- name: rm -Rf
73130
run: rm -Rf build/
74131
working-directory: /home/runner/work/pgagroal/pgagroal/
75-
- name: Start postgres
76-
run: |
77-
version=$(pg_config --version | grep -Eo "[0-9]{1,2}" | head -1)
78-
sudo -u postgres /usr/lib/postgresql/${version}/bin/pg_ctl start -D /etc/postgresql/${version}/main/
79132
- name: CLANG/mkdir
80133
run: mkdir build
81134
working-directory: /home/runner/work/pgagroal/pgagroal/
@@ -85,38 +138,60 @@ jobs:
85138
- name: CLANG/make
86139
run: make
87140
working-directory: /home/runner/work/pgagroal/pgagroal/build/
88-
- name: CLANG/Run pgagroal & confirm pgagroal is running
141+
- name: CLANG/Run pgagroal as daemon with 'io_uring' backend
89142
run: |
90143
sudo mkdir -p /etc/pgagroal
144+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_type file
145+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_path /dev/null
146+
crudini --set ../../doc/etc/pgagroal.conf pgagroal ev_backend io_uring
147+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
91148
sudo cp ../../doc/etc/*.conf /etc/pgagroal
92-
./pgagroal >> /dev/null 2>&1 &
93-
pid=$!
149+
sudo sysctl kernel.io_uring_disabled=0
150+
./pgagroal -d || exit 1
94151
sleep 5
95-
./pgagroal-cli ping
96152
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
97-
- name: CLANG/Stop pgagroal & postgres
153+
- name: CLANG/Run verify_running for 'io_uring' backend
98154
run: |
99-
./pgagroal-cli shutdown
100-
version=$(pg_config --version | grep -Eo "[0-9]{1,2}" | head -1)
101-
sudo -u postgres /usr/lib/postgresql/${version}/bin/pg_ctl stop -D /etc/postgresql/${version}/main/
155+
source /tmp/functions.sh
156+
verify_running
157+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
158+
- name: CLANG/Run verify_shutdown for 'io_uring' backend
159+
run: |
160+
source /tmp/functions.sh
161+
verify_shutdown
162+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
163+
- name: CLANG/Run pgagroal as daemon with 'epoll' backend
164+
run: |
165+
sudo mkdir -p /etc/pgagroal
166+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_type file
167+
crudini --set ../../doc/etc/pgagroal.conf pgagroal log_path /dev/null
168+
crudini --set ../../doc/etc/pgagroal.conf pgagroal ev_backend epoll
169+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
170+
sudo cp ../../doc/etc/*.conf /etc/pgagroal
171+
./pgagroal -d || exit 1
172+
sleep 5
173+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
174+
- name: CLANG/Run verify_running for 'epoll' backend
175+
run: |
176+
source /tmp/functions.sh
177+
verify_running
178+
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
179+
- name: CLANG/Run verify_shutdown for 'epoll' backend
180+
run: |
181+
source /tmp/functions.sh
182+
verify_shutdown
102183
working-directory: /home/runner/work/pgagroal/pgagroal/build/src/
103-
104-
105184

106185
build-macos:
107-
186+
108187
runs-on: macos-latest
109-
188+
110189
steps:
111190
- uses: actions/checkout@v3
112-
- name: Install Homebrew
113-
run: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"
114191
- name: Update system
115192
run: brew update
116193
- name: Install openssl
117194
run: brew install openssl
118-
- name: Install libev
119-
run: brew install libev
120195
- name: Install zstd
121196
run: brew install zstd
122197
- name: Install lz4
@@ -135,10 +210,42 @@ jobs:
135210
run: |
136211
latest_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
137212
brew install ${latest_pg} || true # `|| true` prevents install errors from breaking the run
138-
- name: Start postgres
213+
- name: Start PostgreSQL & setup test table
139214
run: |
140215
installed_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
141216
brew services start ${installed_pg}
217+
sleep 5
218+
/opt/homebrew/opt/${installed_pg}/bin/pgbench -i -s 1 -h localhost -p 5432 -U $(whoami) -d postgres
219+
exit 0
220+
- name: Define functions `verify_running` and `verify_shutdown`
221+
run: |
222+
echo 'verify_running() {
223+
echo "[*] Confirming pgagroal is listening on port 2345"
224+
netstat -an | grep "\.2345 .*LISTEN" || (echo "Nothing is listening on port 2345"; exit 1)
225+
echo "[*] Running pgagroal-cli ping"
226+
./pgagroal-cli ping
227+
echo "[*] Running queries with psql"
228+
installed_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
229+
PGPASSWORD="postgres" /opt/homebrew/opt/${installed_pg}/bin/psql -h 127.0.0.1 -p 2345 -U $(whoami) -d postgres -c "SELECT * FROM pgbench_accounts LIMIT 50;" > /dev/null
230+
PGPASSWORD="postgres" /opt/homebrew/opt/${installed_pg}/bin/psql -h ::1 -p 2345 -U $(whoami) -d postgres -c "SELECT * FROM pgbench_accounts LIMIT 50;" > /dev/null
231+
}
232+
233+
verify_shutdown() {
234+
echo "[*] Getting pgid"
235+
pgid=$(ps -o pgid= -p $(cat /tmp/pgagroal.2345.pid) | grep -o "[0-9]*")
236+
echo "[*] Running pgagroal-cli shutdown immediate"
237+
./pgagroal-cli shutdown immediate
238+
sleep 5
239+
echo "[*] Confirming there are no dangling pgagroal processes"
240+
if pgrep -g $pgid > /tmp/dangling; then
241+
echo "[E] Dangling pgagroal child processes:
242+
$(cat /tmp/dangling)"
243+
exit 1
244+
else
245+
echo "Removing PID file"
246+
rm -f /tmp/pgagroal.2345.pid
247+
fi
248+
}' > /tmp/functions.sh
142249
- name: GCC/mkdir
143250
run: mkdir build
144251
working-directory: /Users/runner/work/pgagroal/pgagroal/
@@ -148,49 +255,63 @@ jobs:
148255
- name: GCC/make
149256
run: make
150257
working-directory: /Users/runner/work/pgagroal/pgagroal/build/
151-
- name: GCC/Run pgagroal & confirm pgagroal is running
152-
run: |
153-
sudo mkdir -p /etc/pgagroal
154-
sudo cp ../../doc/etc/*.conf /etc/pgagroal
155-
./pgagroal >> /dev/null 2>&1 &
156-
pid=$!
157-
sleep 5
158-
./pgagroal-cli ping
258+
- name: GCC/Run pgagroal as daemon with 'kqueue' backend
259+
run: |
260+
sudo mkdir -p /etc/pgagroal
261+
sed -i '' 's/^log_type =.*$/log_type = file/' ../../doc/etc/pgagroal.conf
262+
sed -i '' 's|^log_path =.*$|log_path = /dev/null|' ../../doc/etc/pgagroal.conf
263+
sed -i '' 's/^ev_backend =.*$/ev_backend = kqueue/' ../../doc/etc/pgagroal.conf
264+
cat ../../doc/etc/pgagroal.conf
265+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
266+
sudo cp ../../doc/etc/*.conf /etc/pgagroal
267+
./pgagroal -d || exit 1
268+
sleep 5
159269
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
160-
- name: GCC/Stop pgagroal & postgres
270+
- name: GCC/Run verify_running for 'kqueue' backend
161271
run: |
162-
./pgagroal-cli shutdown
163-
installed_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
164-
brew services stop ${installed_pg}
272+
source /tmp/functions.sh
273+
verify_running
165274
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
166-
- name: rm -Rf
167-
run: rm -Rf build/
168-
working-directory: /Users/runner/work/pgagroal/pgagroal/
169-
- name: Start postgres
275+
- name: GCC/Run verify_shutdown for 'kqueue' backend
170276
run: |
171-
installed_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
172-
brew services start ${installed_pg}
277+
source /tmp/functions.sh
278+
verify_shutdown
279+
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
280+
- name: GCC/Clean up build
281+
run: rm -Rf build
282+
working-directory: /Users/runner/work/pgagroal/pgagroal/
173283
- name: CLANG/mkdir
174284
run: mkdir build
175285
working-directory: /Users/runner/work/pgagroal/pgagroal/
176286
- name: CLANG/cmake
177-
run: export CC=/usr/bin/clang && export OPENSSL_ROOT_DIR=`brew --prefix openssl` && cmake -DCMAKE_BUILD_TYPE=Debug ..
287+
run: |
288+
export CC=/usr/bin/clang
289+
export OPENSSL_ROOT_DIR=$(brew --prefix openssl)
290+
cmake -DCMAKE_BUILD_TYPE=Debug ..
178291
working-directory: /Users/runner/work/pgagroal/pgagroal/build/
179292
- name: CLANG/make
180293
run: make
181294
working-directory: /Users/runner/work/pgagroal/pgagroal/build/
182-
- name: CLANG/Run pgagroal & confirm pgagroal is running
295+
- name: CLANG/Run pgagroal as daemon with 'kqueue' backend
183296
run: |
184297
sudo mkdir -p /etc/pgagroal
298+
sed -i '' 's/^log_type =.*$/log_type = file/' ../../doc/etc/pgagroal.conf
299+
sed -i '' 's|^log_path =.*$|log_path = /dev/null|' ../../doc/etc/pgagroal.conf
300+
sed -i '' 's/^ev_backend =.*$/ev_backend = kqueue/' ../../doc/etc/pgagroal.conf
301+
cat ../../doc/etc/pgagroal.conf
302+
echo "host all all all trust" > ../../doc/etc/pgagroal_hba.conf
185303
sudo cp ../../doc/etc/*.conf /etc/pgagroal
186-
./pgagroal >> /dev/null 2>&1 &
187-
pid=$!
304+
./pgagroal -d || exit 1
188305
sleep 5
189-
./pgagroal-cli ping
190306
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
191-
- name: CLANG/Stop pgagroal & postgres
307+
- name: CLANG/Run verify_running for 'kqueue' backend
192308
run: |
193-
./pgagroal-cli shutdown
194-
installed_pg=$(brew search postgresql | grep postgresql@ | tail -n 1)
195-
brew services stop ${installed_pg}
309+
source /tmp/functions.sh
310+
verify_running
196311
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
312+
- name: CLANG/Run verify_shutdown for 'kqueue' backend
313+
run: |
314+
source /tmp/functions.sh
315+
verify_shutdown
316+
working-directory: /Users/runner/work/pgagroal/pgagroal/build/src/
317+

0 commit comments

Comments
 (0)