mxnet version error..

[jvrobert]

I was using an older version of compreface and it worked fine, but decided to move to 1.1.0. I did it from scratch, removed postgres docker volume, etc.. I'm getting this when I try to do anything such as uploading a face (using 1.1.0-arcface-r100-gpu with nvidia GPU, same config I used in older version). Any ideas?

compreface-core | [00:01:39] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [00:01:40] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
compreface-core | [32, 16, 8] {'32': {'SCALES': (32, 16), 'BASE_SIZE': 16, 'RATIOS': (1.0,), 'ALLOWED_BORDER': 9999}, '16': {'SCALES': (8, 4), 'BASE_SIZE': 16, 'RATIOS': (1.0,), 'ALLOWED_BORDER': 9999}, '8': {'SCALES': (2, 1), 'BASE_SIZE': 16, 'RATIOS': (1.0,), 'ALLOWED_BORDER': 9999}}
compreface-core | use_landmarks True
compreface-core | [00:01:42] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.3.0. Attempting to upgrade...
compreface-core | [00:01:42] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
compreface-core | [00:01:42] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
compreface-core | [00:01:42] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
compreface-core | {"severity": "DEBUG", "message": "Found: BoundingBoxDTO(x_min=296, y_min=388, x_max=898, y_max=1255, probability=0.9990967512130737, np_landmarks=array([[ 512.8989 , 756.3247 ],\n [ 800.13495, 739.587 ],\n [ 694.2923 , 944.9437 ],\n [ 545.1163 , 1078.0962 ],\n [ 760.3133 , 1067.7125 ]], dtype=float32))", "request": {"method": "POST", "path": "/find_faces", "filename": "20221216_164552.jpg", "api_key": "", "remote_addr": "172.22.0.6"}, "logger": "src.services.facescan.plugins.insightface.insightface", "module": "insightface", "traceback": null, "build_version": "dev"}
compreface-core | [00:01:42] ../src/nnvm/legacy_json_util.cc:203: Warning: loading symbol saved by MXNet version 20000 with lower version of MXNet v10900. May cause undefined behavior. Please update MXNet if you encounter any issue
compreface-core | {"severity": "INFO", "message": "200 OK", "request": {"method": "POST", "path": "/find_faces", "filename": "20221216_164552.jpg", "api_key": "", "remote_addr": "172.22.0.6"}, "logger": "src.services.flask.log_response", "module": "log_response", "traceback": null, "build_version": "dev"}
compreface-core | terminate called after throwing an instance of 'dmlc::Error'
compreface-core | what(): [00:01:43] ../include/mshadow/./stream_gpu-inl.h:133: Check failed: err == CUBLAS_STATUS_SUCCESS (1 vs. 0) : Create cublas handle failed
compreface-core |
compreface-core |
compreface-core | DAMN ! worker 1 (pid: 30) died, killed by signal 6 :( trying respawn ...
compreface-core | Respawned uWSGI worker 1 (new pid: 212)

[pospielov]

What was the last version of CompreFace you successfully used?
We updated the CUDA library in the 1.0.1 release, some old GPUs are not supported anymore.
You can read it in detail here:
https://exadel.com/news/minor-compreface-update/
Could you send your GPU model if it's not old? Not sure we will be able to reproduce it.

[jvrobert]

Thanks.. didn't see that, not great but I blame nvidia not you guys. I jave a t1000 or t800 or something of that gen.

What's a good alternative, 3050 seems more future proof but still fairly pricy. Any good choices likely to be supported for next 3 years or so?

[jvrobert]

Actually, logged in and checked and I have a 4GB T600, CUDA v 11.7. Looks like it supports compute capability 7.5 which should be good enough, no? Here is my smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.86.01 Driver Version: 515.86.01 CUDA Version: 11.7 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA T600 Off | 00000000:01:00.0 Off | N/A |
| 38% 40C P8 N/A / 41W | 3198MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 22206 C uwsgi 3195MiB |
+-----------------------------------------------------------------------------+

[pospielov]

Sorry for a long response
I think the problem can be with memory consumption. As I remember the minimum GPU that is capable of run SubCenter-ArcFace-r100-gpu version requires 6 Gb of RAM, so I don't know how it worked before...
Can you try Mobilenet-gpu version? It should work on 4Gb GPU, if it works then this is the case. Probably new CUDA is not as optimized as we want...

[jvrobert]

yeah, the mobilenet GPU version works, I'm using it now.. SubCenter did work in older releases (if I remember it used 31xx or 32xx MB of RAM out of 4).

Thanks

[Neekazhang]

I also have the same problem, my CUDA version is 12.1, graphics card rtx4060, 8G video memory, can I solve it?thank you

[pospielov]

We released a couple of urgent fixes today.
Could you run docker-compose pull and run CompreFace again?
Not sure if it helps, but it helps with extra RAM consumption.

[jvrobert]

FYI I think my issue was because I had old training data from previous version of compreface. I removed all the training data and readded and it fixed it.

[Neekazhang]

I think it may not be a video memory problem, after turning up face detection, the GPU occupies 50%, and the video memory only accounts for 2G. I tried both MobileNet-GPU and ArcFace-R100-GPU versions. The core error log is as follows:
Respawned uWSGI worker 1 (new pid: 111)
2023-03-11 16:22:41 [08:22:41] ../src/nnvm/legacy_json_util.cc:208: Loading symbol saved by previous version v1.2.0. Attempting to upgrade...
2023-03-11 16:22:41 [08:22:41] ../src/nnvm/legacy_json_util.cc:216: Symbol successfully upgraded!
2023-03-11 16:22:41 [08:22:41] ../src/engine/engine.cc:54: MXNet start using engine: ThreadedEnginePerDevice
2023-03-11 16:22:41 [08:22:41] ../src/base.cc:79: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
2023-03-11 16:22:46 terminate called after throwing an instance of 'dmlc::Error'
2023-03-11 16:22:46 terminate called after throwing an instance of 'dmlc::Error'
2023-03-11 16:22:46 what(): [08:22:46] ../include/mshadow/./stream_gpu-inl.h:159: Check failed: err == CUSOLVER_STATUS_SUCCESS (7 vs. 0) : Create cusolver handle failed
2023-03-11 16:22:46
2023-03-11 16:22:46
2023-03-11 16:22:46 what(): [08:22:46] ../include/mshadow/./stream_gpu-inl.h:159: Check failed: err == CUSOLVER_STATUS_SUCCESS (7 vs. 0) : Create cusolver handle failed
2023-03-11 16:22:46
2023-03-11 16:22:46
2023-03-11 16:22:47 DAMN ! worker 1 (pid: 111) died, killed by signal 6 :( trying respawn ...

It is suspected that it is a CUDA and CUDNN version problem, I have downgraded the version to CUDA 11.2.2, CUDNN 8.1.0.77, but the problem still exists. Please give me a look.Thank you again!

[jvrobert]

Did you try removing all training data and retraining?

[pospielov]

@Neekazhang, it looks like, unfortunately, your GPU is not supported by CompreFace yet.
https://en.wikipedia.org/wiki/CUDA
rtx4060 Compute Capability is 8.9, and we use CUDA 11.2, which can work with a maximum Compute Capability of 8.6.
Updating CUDA on your machine won't help as it doesn't influence what is inside the container.
And we have lots of work to do to update CUDA inside and check that everything still works...

[Neekazhang]

Thank you for your work, looking forward to the early release of the new version, is it expected to be released in the second quarter?

…

---Original--- From: "Pospielov ***@***.***> Date: Sat, Mar 18, 2023 02:01 AM To: ***@***.***>; Cc: ***@***.******@***.***>; Subject: Re: [exadel-inc/CompreFace] mxnet version error.. (Discussion #1000) @Neekazhang, it looks like, unfortunately, your GPU is not supported by CompreFace yet. https://en.wikipedia.org/wiki/CUDA rtx4060 Compute Capability is 8.9, and we use CUDA 11.2, which can work with a maximum Compute Capability of 8.6. Updating CUDA on your machine won't help as it doesn't influence what is inside the container. And we have lots of work to do to update CUDA inside and check that everything still works... — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[pospielov]

We started work on this, and unfortunately, we faced a problem.
We use mxnet library to run neural networks. And they didn't release a version for the new CUDA.
It looks like mxnet is almost dead, and, of course, if they release a new version, we can update quite fast.
But for now, we started looking for a way to migrate to another library.
I hope we will be able to solve it in the second quarter and release it as this is a high-priority problem.