Server down error (4store)

[syphax-bouazzouni]

UI is down

CODE EROR : 500 Internal Server Error
DATE: 20/10/2021 15:13

API is also down

[syphax-bouazzouni]

The cause, is that the 4s http got stoped , like the following centos messages log (/var/log/messages) shows:

Nov 15 03:19:04 agroportal 4store[5244]: rhash.c:713 resource 6d03e01554f90a5c not found in § 0x53e420-0x53e43f of /srv/ontoportal/data/4store/ontologies_api/0000/res.rhash
Nov 15 03:19:04 agroportal 4store[5249]: rhash.c:713 resource 188ee56622da9bdb not found in § 0x8b6a60-0x8b6a7f of /srv/ontoportal/data/4store/ontologies_api/0003/res.rhash
Nov 15 03:19:04 agroportal 4store[5246]: rhash.c:713 resource 5770d925924840e5 not found in § 0x492100-0x49211f of /srv/ontoportal/data/4store/ontologies_api/0001/res.rhash
Nov 15 03:19:04 agroportal 4store[5246]: rhash.c:713 resource 0bab55fe3222ab39 not found in § 0xc88aa0-0xc88abf of /srv/ontoportal/data/4store/ontologies_api/0001/res.rhash
Nov 15 03:19:04 agroportal 4store[5246]: rhash.c:713 resource 1d43ad91b6dc18d1 not found in § 0xdb7060-0xdb707f of /srv/ontoportal/data/4store/ontologies_api/0001/res.rhash
Nov 15 03:20:01 agroportal systemd: Started Session 8342 of user root.
Nov 15 03:23:19 agroportal 4store[5248]: rhash.c:713 resource 37d12014903f7762 not found in § 0x40fdd0-0x40fdef of /srv/ontoportal/data/4store/ontologies_api/0002/res.rhash
Nov 15 03:30:01 agroportal systemd: Started Session 8343 of user root.
Nov 15 03:30:12 agroportal kernel: 4s-backend[5244]: segfault at 7ffec3d37d98 ip 0000000000403cfc sp 00007ffec3d37da0 error 6 in 4s-backend[400000+35000]
Nov 15 03:30:12 agroportal kernel: 4s-backend[5248]: segfault at 7ffec3d34958 ip 0000000000403cfc sp 00007ffec3d34960 error 6
Nov 15 03:30:12 agroportal kernel: in 4s-backend[400000+35000]
Nov 15 03:30:12 agroportal kernel: 
Nov 15 03:30:12 agroportal kernel: 4s-backend[5246]: segfault at 7ffec3d36f88 ip 0000000000403cfc sp 00007ffec3d36f90 error 6
Nov 15 03:30:12 agroportal kernel: in 4s-backend[400000+35000]
Nov 15 03:30:12 agroportal kernel: 
Nov 15 03:30:12 agroportal kernel: 4s-backend[5249]: segfault at 7ffec3d33278 ip 0000000000403cfc sp 00007ffec3d33280 error 6
Nov 15 03:30:12 agroportal kernel: in 4s-backend[400000+35000]
Nov 15 03:30:12 agroportal kernel: 
Nov 15 03:30:12 agroportal kernel: 
Nov 15 03:30:12 agroportal 4store[8513]: 4s-server.c:319 kb=ontologies_api child 5246 terminated by signal 11

[syphax-bouazzouni]

hot fix : restart 4s-httpd every day at 11:00 pm (0 23 * * * sudo /usr/bin/systemctl restart 4s-httpd.service)

[jonquet]

After investigations (nov 2021) :

• The problem is a some kind of memory leak (segmentation fault) => very hard to identify. After a while 4store does not respond without explanation.
• The API logs do not show a specific query or pattern which provokes the error
• The frequency of the call when crash seems normal

Temporal solution: setting up a cron job to restart 4store every night at 11pm to regularly refresh 4store memory

[syphax-bouazzouni]

Update: 13/12/2021 (15:30)

The bug reappeared even with the hotfix

Some screenshots :

SPARQL endpoint (running)

UI (API request timeout)

API (some routes are still working like 'projects' but maybe it's just cached data)

API (can't retrieve ontologies)

[syphax-bouazzouni]

Close this as too old, and could not figure out the issue behind 4store, our plan is to move out to Virsuoso, follow up here: agroportal/project-management#229