Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: Mixtral 8x7B fails to return a response after a couple of API calls whill running on AWS g6.12xlarge EC2 instance #504

Open
rpchastain opened this issue Jul 24, 2024 · 0 comments
Open

Bug: Mixtral 8x7B fails to return a response after a couple of API calls whill running on AWS g6.12xlarge EC2 instance #504

rpchastain opened this issue Jul 24, 2024 · 0 comments
Labels
bug high severity

Comments

@rpchastain
Copy link

Contact Details

rpchastain@protonmail.com

What happened?

I'm running Mixtral-8x7B-Instruct-v0.1-llamafile in server mode on an AWS g6.12xlarge EC2 instance with 4 NVidia L4 gpus. I'm using full gpu offloading (-ngl 999).

The EC2 instance is running Amazon Linux 3.

I'm using the OpenAI client to access the API using a for loop in python with rather large prompts. After a few iterations (usually about 6, but sometimes as few as 1 and as many as 8) the API stops returning responses and seems to hang.

I've tried tweaking various parameters, but nothing seems to help.

The service is started with this command:

./Mixtral-8x7B-Instruct-v0.1-llamafile --server --nobrowser -ngl 999 --port 9990 --strace

Version

./Mixtral-8x7B-Instruct-v0.1-llamafile --version

llamafile v0.8.9

gcc --version

gcc (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Jun__6_02:18:23_PDT_2024
Cuda compilation tools, release 12.5, V12.5.82
Build cuda_12.5.r12.5/compiler.34385749_0

What operating system are you seeing the problem on?

Linux

Relevant log output

llama server listening at http://127.0.0.1:9990

SYS   9618   9618    227'935'753'155 writev(2, {{u"", 0}, {u"◙llama server listening at http://127.0."..., 50}}, 2) → 50 ENOENT
{"function":"server_cli","hostname":"127.0.0.1","level":"INFO","line":3213,"msg":"HTTP server listening","port":"9990","tid":"10733792","timestamp":1721861859}
SYS   9618   9618    227'935'776'943 write(1, u"{“function“:“server_cli“,“hostname“:“127"..., 160) → 160 ENOENT
SYS   9618   9618    227'935'789'654 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda401ee000 ENOENT
SYS   9618   9618    227'935'799'609 mprotect(0x7fda401ee000, 4'096, 0) → 0 ENOENT
SYS   9618   9618    227'935'837'478 pthread_create([11219], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEZ10server_cliiPPcEUlvE5_EEEEEPvSB_, 0x7fda741b6280) → 0
SYS   9618  11219    227'935'886'626 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda401ae000
SYS   9618  11219    227'935'900'097 mprotect(0x7fda401ae000, 4'096, 0) → 0
SYS   9618   9618    227'935'907'897 sigaction(SIGINT, {.sa_handler=signal_handler(int)}, [NULL]) → 0 ENOENT
{"function":"update_slots","level":"INFO","line":1659,"msg":"all slots are idle and system prompt is empty, clear the KV cache","tid":"10733792","timestamp":1721861859}
SYS   9618   9618    227'935'935'395 write(1, u"{“function“:“update_slots“,“level“:“INFO"..., 169) → 169 ENOENT
SYS   9618  11219    227'935'931'058 pthread_create([11220], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c074c0) → 0
SYS   9618  11219    227'935'949'184 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4016e000
SYS   9618  11219    227'935'978'334 mprotect(0x7fda4016e000, 4'096, 0) → 0
SYS   9618  11219    227'936'008'412 pthread_create([11221], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c08770) → 0
SYS   9618  11219    227'936'026'794 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4012e000
SYS   9618  11219    227'936'046'837 mprotect(0x7fda4012e000, 4'096, 0) → 0
SYS   9618  11219    227'936'076'101 pthread_create([11222], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c09a10) → 0
SYS   9618  11219    227'936'092'982 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda400ee000
SYS   9618  11219    227'936'104'774 mprotect(0x7fda400ee000, 4'096, 0) → 0
SYS   9618  11219    227'936'162'668 pthread_create([11223], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c0aca0) → 0
SYS   9618  11219    227'936'203'964 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda400ae000
SYS   9618  11219    227'936'214'113 mprotect(0x7fda400ae000, 4'096, 0) → 0
SYS   9618  11219    227'936'241'497 pthread_create([11224], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c0bf80) → 0
SYS   9618  11219    227'936'257'406 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4006e000
SYS   9618  11219    227'936'276'459 mprotect(0x7fda4006e000, 4'096, 0) → 0
SYS   9618  11219    227'936'302'553 pthread_create([11225], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c08890) → 0
SYS   9618  11219    227'936'324'804 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4002e000
SYS   9618  11219    227'936'365'861 mprotect(0x7fda4002e000, 4'096, 0) → 0
SYS   9618  11219    227'936'398'712 pthread_create([11226], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c0e490) → 0
SYS   9618  11219    227'936'416'255 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda2cf5d000
SYS   9618  11219    227'936'428'666 mprotect(0x7fda2cf5d000, 4'096, 0) → 0
SYS   9618  11219    227'936'457'419 pthread_create([11227], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c0e5d0) → 0
SYS   9618  11219    227'936'482'099 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda2cf1d000
SYS   9618  11219    227'936'492'946 mprotect(0x7fda2cf1d000, 4'096, 0) → 0
SYS   9618  11219    227'936'519'906 pthread_create([11228], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c10a60) → 0
SYS   9618  11219    227'936'534'481 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda2cedd000
SYS   9618  11219    227'936'549'842 mprotect(0x7fda2cedd000, 4'096, 0) → 0
SYS   9618  11219    227'936'579'336 pthread_create([11229], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c0bef0) → 0
SYS   9618  11219    227'936'596'296 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd270070000
SYS   9618  11219    227'936'607'108 mprotect(0x7fd270070000, 4'096, 0) → 0
SYS   9618  11219    227'936'632'884 pthread_create([11230], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c12f50) → 0
SYS   9618  11219    227'936'651'717 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd270030000
SYS   9618  11219    227'936'660'647 mprotect(0x7fd270030000, 4'096, 0) → 0
SYS   9618  11219    227'936'683'950 pthread_create([11231], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c13090) → 0
SYS   9618  11219    227'936'697'447 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a18e000
SYS   9618  11219    227'936'706'581 mprotect(0x7fd26a18e000, 4'096, 0) → 0
SYS   9618  11219    227'936'734'264 pthread_create([11232], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c15490) → 0
SYS   9618  11219    227'936'745'730 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4001e000
SYS   9618  11219    227'936'762'151 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a14e000
SYS   9618  11219    227'936'770'684 mprotect(0x7fd26a14e000, 4'096, 0) → 0
SYS   9618  11219    227'936'793'121 pthread_create([11233], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c155d0) → 0
SYS   9618  11219    227'936'810'814 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a10e000
SYS   9618  11219    227'936'820'531 mprotect(0x7fd26a10e000, 4'096, 0) → 0
SYS   9618  11219    227'936'841'139 pthread_create([11234], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c168a0) → 0
SYS   9618  11219    227'936'858'019 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a0ce000
SYS   9618  11219    227'936'866'959 mprotect(0x7fd26a0ce000, 4'096, 0) → 0
SYS   9618  11219    227'936'889'545 pthread_create([11235], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c16a20) → 0
SYS   9618  11219    227'936'907'336 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a08e000
SYS   9618  11219    227'936'919'084 mprotect(0x7fd26a08e000, 4'096, 0) → 0
SYS   9618  11219    227'936'941'839 pthread_create([11236], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c16cb0) → 0
SYS   9618  11219    227'936'960'080 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a04e000
SYS   9618  11219    227'936'969'293 mprotect(0x7fd26a04e000, 4'096, 0) → 0
SYS   9618  11219    227'936'992'012 pthread_create([11237], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c16e30) → 0
SYS   9618  11219    227'937'010'094 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd26a00e000
SYS   9618  11219    227'937'017'249 mprotect(0x7fd26a00e000, 4'096, 0) → 0
SYS   9618  11219    227'937'036'965 pthread_create([11238], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40022510) → 0
SYS   9618  11219    227'937'054'252 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd25800c000
SYS   9618  11219    227'937'061'787 mprotect(0x7fd25800c000, 4'096, 0) → 0
SYS   9618  11219    227'937'080'354 pthread_create([11239], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda74c109d0) → 0
SYS   9618  11219    227'937'097'553 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd2340e9000
SYS   9618  11219    227'937'107'640 mprotect(0x7fd2340e9000, 4'096, 0) → 0
SYS   9618  11219    227'937'129'759 pthread_create([11240], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40024990) → 0
SYS   9618  11219    227'937'147'991 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd2340a9000
SYS   9618  11219    227'937'158'344 mprotect(0x7fd2340a9000, 4'096, 0) → 0
SYS   9618  11219    227'937'182'918 pthread_create([11241], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda400213d0) → 0
SYS   9618  11219    227'937'213'587 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd234069000
SYS   9618  11219    227'937'222'562 mprotect(0x7fd234069000, 4'096, 0) → 0
SYS   9618  11219    227'937'245'820 pthread_create([11242], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40020290) → 0
SYS   9618  11219    227'937'264'715 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd234029000
SYS   9618  11219    227'937'274'087 mprotect(0x7fd234029000, 4'096, 0) → 0
SYS   9618  11219    227'937'296'038 pthread_create([11243], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001f150) → 0
SYS   9618  11219    227'937'313'837 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dfc0000
SYS   9618  11219    227'937'322'856 mprotect(0x7fd22dfc0000, 4'096, 0) → 0
SYS   9618  11219    227'937'352'209 pthread_create([11244], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4002a510) → 0
SYS   9618  11219    227'937'371'739 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22df80000
SYS   9618  11219    227'937'380'617 mprotect(0x7fd22df80000, 4'096, 0) → 0
SYS   9618  11219    227'937'405'721 pthread_create([11245], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4002a650) → 0
SYS   9618  11219    227'937'417'178 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22df40000
SYS   9618  11219    227'937'427'654 mprotect(0x7fd22df40000, 4'096, 0) → 0
SYS   9618  11219    227'937'451'867 pthread_create([11246], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4002ca50) → 0
SYS   9618  11219    227'937'468'429 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fda4000e000
SYS   9618  11219    227'937'480'363 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22df00000
SYS   9618  11219    227'937'487'606 mprotect(0x7fd22df00000, 4'096, 0) → 0
SYS   9618  11219    227'937'507'225 pthread_create([11247], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4002cb90) → 0
SYS   9618  11219    227'937'524'521 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dec0000
SYS   9618  11219    227'937'534'979 mprotect(0x7fd22dec0000, 4'096, 0) → 0
SYS   9618  11219    227'937'557'584 pthread_create([11248], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4000f190) → 0
SYS   9618  11219    227'937'576'072 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22de80000
SYS   9618  11219    227'937'585'736 mprotect(0x7fd22de80000, 4'096, 0) → 0
SYS   9618  11219    227'937'608'252 pthread_create([11249], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40010480) → 0
SYS   9618  11219    227'937'625'671 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22de40000
SYS   9618  11219    227'937'636'660 mprotect(0x7fd22de40000, 4'096, 0) → 0
SYS   9618  11219    227'937'659'123 pthread_create([11250], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40011710) → 0
SYS   9618  11219    227'937'676'118 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22de00000
SYS   9618  11219    227'937'683'150 mprotect(0x7fd22de00000, 4'096, 0) → 0
SYS   9618  11219    227'937'701'850 pthread_create([11251], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40012a00) → 0
SYS   9618  11219    227'937'731'009 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22ddc0000
SYS   9618  11219    227'937'741'282 mprotect(0x7fd22ddc0000, 4'096, 0) → 0
SYS   9618  11219    227'937'765'644 pthread_create([11252], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40013ea0) → 0
SYS   9618  11219    227'937'788'620 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dd80000
SYS   9618  11219    227'937'796'764 mprotect(0x7fd22dd80000, 4'096, 0) → 0
SYS   9618  11219    227'937'817'982 pthread_create([11253], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40015130) → 0
SYS   9618  11219    227'937'835'189 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dd40000
SYS   9618  11219    227'937'845'171 mprotect(0x7fd22dd40000, 4'096, 0) → 0
SYS   9618  11219    227'937'867'501 pthread_create([11254], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda400162f0) → 0
SYS   9618  11219    227'937'884'665 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dd00000
SYS   9618  11219    227'937'894'955 mprotect(0x7fd22dd00000, 4'096, 0) → 0
SYS   9618  11219    227'937'917'127 pthread_create([11255], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda400175b0) → 0
SYS   9618  11219    227'937'935'589 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dcc0000
SYS   9618  11219    227'937'945'456 mprotect(0x7fd22dcc0000, 4'096, 0) → 0
SYS   9618  11219    227'937'968'749 pthread_create([11256], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40018870) → 0
SYS   9618  11219    227'937'985'877 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dc80000
SYS   9618  11219    227'937'992'970 mprotect(0x7fd22dc80000, 4'096, 0) → 0
SYS   9618  11219    227'938'011'237 pthread_create([11257], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda40019b30) → 0
SYS   9618  11219    227'938'037'481 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dc40000
SYS   9618  11219    227'938'045'290 mprotect(0x7fd22dc40000, 4'096, 0) → 0
SYS   9618  11219    227'938'064'706 pthread_create([11258], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001adf0) → 0
SYS   9618  11219    227'938'083'715 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dc00000
SYS   9618  11219    227'938'093'158 mprotect(0x7fd22dc00000, 4'096, 0) → 0
SYS   9618  11219    227'938'115'859 pthread_create([11259], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001c0b0) → 0
SYS   9618  11219    227'938'127'095 mmap(0, 65'536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd270020000
SYS   9618  11219    227'938'144'365 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dbc0000
SYS   9618  11219    227'938'151'581 mprotect(0x7fd22dbc0000, 4'096, 0) → 0
SYS   9618  11219    227'938'170'979 pthread_create([11260], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d370) → 0
SYS   9618  11219    227'938'188'107 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22db80000
SYS   9618  11219    227'938'195'333 mprotect(0x7fd22db80000, 4'096, 0) → 0
SYS   9618  11219    227'938'213'989 pthread_create([11261], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d4f0) → 0
SYS   9618  11219    227'938'230'428 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22db40000
SYS   9618  11219    227'938'237'715 mprotect(0x7fd22db40000, 4'096, 0) → 0
SYS   9618  11219    227'938'257'908 pthread_create([11262], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d570) → 0
SYS   9618  11219    227'938'274'762 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22db00000
SYS   9618  11219    227'938'286'820 mprotect(0x7fd22db00000, 4'096, 0) → 0
SYS   9618  11219    227'938'308'744 pthread_create([11263], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d6f0) → 0
SYS   9618  11219    227'938'327'126 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22dac0000
SYS   9618  11219    227'938'336'507 mprotect(0x7fd22dac0000, 4'096, 0) → 0
SYS   9618  11219    227'938'359'333 pthread_create([11264], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d870) → 0
SYS   9618  11219    227'938'378'757 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22da80000
SYS   9618  11219    227'938'390'532 mprotect(0x7fd22da80000, 4'096, 0) → 0
SYS   9618  11219    227'938'413'861 pthread_create([11265], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001d9f0) → 0
SYS   9618  11219    227'938'430'512 mmap(0, 262'144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) → 0x7fd22da40000
SYS   9618  11219    227'938'437'525 mprotect(0x7fd22da40000, 4'096, 0) → 0
SYS   9618  11219    227'938'456'623 pthread_create([11266], 0, _ZNSt3__214__thread_proxyINS_5tupleIJNS_10unique_ptrINS_15__thread_structENS_14default_deleteIS3_EEEEN7httplib10ThreadPool6workerEEEEEEPvSB_, 0x7fda4001db70) → 0
SYS   9618  11219    272'984'168'640 accept4(98, [NULL]) -> 99
SYS   9618  11219    272'984'186'060 setsockopt(99, SOL_SOCKET, SO_RCVTIMEO, u"X☻              ", 16) → 0
SYS   9618  11219    272'984'192'305 setsockopt(99, SOL_SOCKET, SO_SNDTIMEO, u"X☻              ", 16) → 0
SYS   9618  11219    272'984'205'475 futex(7110000008 [1], FUTEX_WAKE_PRIVATE, 1) → 1 woken
SYS   9618  11220    272'984'246'550 futex(7110000008 [1], FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL) → 0

error: Uncaught SIGSEGV (SEGV_ACCERR) at 0x7fd9c01ed6c5 on ip-10-163-209-149.us-west-2.compute.internal pid 9618 tid 11220
  ./_llm/_llamafile/Mixtral-8x7B-Instruct-v0.1-llamafile
  No error information
  Linux Cosmopolitan 3.5.4 MODE=x86_64; #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 ip-10-163-209-149.us-west-2.compute.internal 6.1.94-99.176.amzn2023.x86_64

RAX 0000000000000002 RBX 00007fd9c01ed6c5 RDI 00007fd9c01ed6c5
RCX 0000000000000001 RDX 0000000000000001 RSI 0000000000000000
RBP 00007fda401ed590 RSP 00007fda401ec8c0 RIP 000000000097e110
 R8 0000000000000036  R9 0000000000000000 R10 0000000000000000
R11 0000000000000000 R12 0000000000000000 R13 000000000098c13f
R14 0000000000000000 R15 00007fda401ed6c4
TLS 00007fda74c08500

XMM0  00007fda74aa918000007fda74aa9180 XMM8  00007fda6b7f9e7000007fda6b7f9e80
XMM1  00000000000000000000000000000000 XMM9  00007fda6b7f9e9000007fda6b7f9ea0
XMM2  00000000000000000000000000000000 XMM10 00007fda6b7f9eb000007fda6b7f9ec0
XMM3  00000000000000000000000000000000 XMM11 00007fda6b7f9ed000007fda6b7f9ee0
XMM4  00000000000000000000000000000000 XMM12 00007fda6b7f9ef000007fda6b7f9f00
XMM5  00000000000000000000000000000000 XMM13 00007fda6b7f9f1000007fda6b7f9f20
XMM6  00000000000000000000000000000000 XMM14 00007fda6b7f9f3000007fda6b7f9f40
XMM7  00000000000000000000000000000000 XMM15 00000000000000000000000000000000

cosmoaddr2line /home/ec2-user/cc-llama/_llm/_llamafile/Mixtral-8x7B-Instruct-v0.1-llamafile 97e110 97fd9f 93c675 93056c 45407b 4ef716 4ef91a 45355e 44f66d 91a46c 9255e4 98019f

0x000000000097e110: ?? ??:0
0x000000000097fd9f: ?? ??:0
0x000000000093c675: ?? ??:0
0x000000000093056c: ?? ??:0
0x000000000045407b: ?? ??:0
0x00000000004ef716: ?? ??:0
0x00000000004ef91a: ?? ??:0
0x000000000045355e: ?? ??:0
0x000000000044f66d: ?? ??:0
0x000000000091a46c: ?? ??:0
0x00000000009255e4: ?? ??:0
0x000000000098019f: ?? ??:0

000000400000-000000a161f8 r-xf-- 6232kb
7fda401af000-7fda401ee000 rw-pa- 252kb
000000a17000-000000aae000 rw-f-- 604kb
7fff6664c000-7fff66e4c000 rw-f-- 8192kb
7fda401ef000-7fda402ee000 rw-pa- 1020kb
7fd22da40000-7fd22da41000 ---pa- 4096b
7fd22da41000-7fd22da80000 rw-pa- 252kb
7fd22da80000-7fd22da81000 ---pa- 4096b
7fd22da81000-7fd22dac0000 rw-pa- 252kb
7fd22dac0000-7fd22dac1000 ---pa- 4096b
7fd22dac1000-7fd22db00000 rw-pa- 252kb
7fd22db00000-7fd22db01000 ---pa- 4096b
7fd22db01000-7fd22db40000 rw-pa- 252kb
# 66'542'968'832 bytes in 122 mappings
./_llm/_llamafile/Mixtral-8x7B-Instruct-v0.1-llamafile -m mixtral-8x7b-instruct-v0.1.Q5_K_M.gguf -c 2048 --server --nobrowser -ngl 999 --port 9990
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug high severity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rpchastain