WIP feat:Init commit for rust backend #1180

Aisuko · 2023-10-17T02:37:01Z

Description

This PR relates to #939

Notes for Reviewers

Signed commits

Yes, I signed my commits.

Signed-off-by: GitHub <noreply@github.com>

mudler · 2023-10-17T16:36:36Z

cc @lu-zero

.gitignore

backend/rust/Makefile

backend/rust/src/main.rs

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/bunker/build.rs

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/bunker/src/service.rs

backend/rust/bunker/src/lib.rs

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko · 2023-10-22T00:00:29Z

The process of rust backend:

The basic framework of Rust gRPC backend (Maybe still some issues, will fixed by other commits)
Implement with burn (Working on it now, their website was broken I already open issue on their repo and do more investigation)
candle (I see burn support candle as backend in alpha. So, let's implement the backend with burn first)

Really appreciate your help @lu-zero but still need your help on "burn" backend, thank you.

backend/rust/Cargo.lock

Signed-off-by: Aisuko <urakiny@gmail.com>

.gitignore

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko · 2023-10-26T02:48:53Z

An idea of choosing default burn backend for Rust backend #1219

Aisuko · 2023-10-31T08:15:29Z

Get stuck in some issues like below(only in debug mode), it may related to the Rust implement PyTorch C++ API.

dyld[15803]: Library not loaded: @rpath/libtorch_cpu.dylib
  Referenced from: <B583CD33-2743-323A-B503-5781B34C078F> /Users/tifa/Downloads/workspace/LocalAI/backend/rust/target/debug/deps/server-bc3eca19368e3b4a
  Reason: no LC_RPATH's found

It makes hard to debug the program. I am going to refactor some code and add setting file of IDE. Make sure it can be easy for anyone to debug the program.

lu-zero · 2023-10-31T08:48:30Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Aisuko · 2023-10-31T23:37:39Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Will try it and give a feedback

Update

ndarary backend can be used to debug in IDE. And the torch backend has some issues on Mac M1. Here I am trying to set up LIBTORCH_USE_PYTORCH=1 as env with the conda env which is installed PyTorch. However, it is still hit other issues on M1 environment. So, I'm going to use ndarray to help me debug the conversion part code.

lu-zero · 2023-11-01T08:37:49Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko · 2023-11-01T11:09:56Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Thanks a lot. I have made some change here. I have been migrated the code which is included Llama2 to fork repo, and I am working on the a more simpler model. Here are some reasons:

A simpler model can be more effecient to debug than Llama2, less parameters, and less memory used. (Only load half of Llama2 parameters to tensor can cost at least 13min in my local env now)
We can move faster on this PR. It is good for us to refractor the code, project structure and abstract some common traits.
Easy for code reviewing
Easy for adding some test cases(CI).

Here I hit an issue on reshaping of the Tensor. So, we can try to implement a simple one instead of getting stuck on the Llama2.

backend/rust/models/src/lib.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero · 2023-11-16T06:34:15Z

backend/rust/models/src/whisper/utils.rs

+        // And now the nonlinear scale
+        let min_log_hz = 1000.0; // beginning of log region (Hz)
+        let min_log_mel = (min_log_hz - f_min) / f_sp;
+        let logstep = (6.4f64).ln() / 27.0; // step size for log region


those constants are repeated, being always f64 you can just keep them as consts

thank you, will do.

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/backend/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

netlify · 2023-11-23T01:15:35Z

❌ Deploy Preview for localai failed.

Name	Link
🔨 Latest commit	`c990112`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/655ea7b3d02aec0008ca4cdf

Aisuko · 2023-11-23T01:22:15Z

backend/rust/models/src/llama/llama.rs

+
+        let tensor3=tensor2.transpose();
+
+        let tensor41=tensor3.repeat(2, 2);


@lu-zero Here, I am going to use wgpu backend instead of tch. However, I the repeat function here only support 2 dimensions tensor, (Can only repeat dimension with dim=1) https://github.com/Tracel-AI/burn/blob/b86bc5876149bd73bc59cb5197fd3ee8b92509d4/burn-tensor/src/tensor/ops/tensor.rs#L222C7-L222C7.

I have been tried several solutions, like use swap_dims and flattern these internal function of Tensor, but here hard to say it is correct and also causes other issues. Is there a better example for this?

asking upstream probably it is the best route (sorry for the belated reply, I got very busy and the message got lost in the mailbox)

No worries, thanks for your support. I will continue to work on this one after I applied PhD successfully. Currently, sooo busy. But I still want to get this PR to merged.

Once you are more free please contact me, probably a good deal of the issues will be ironed out by upstream meanwhile :)

Aisuko marked this pull request as draft October 17, 2023 02:38

Aisuko self-assigned this Oct 17, 2023

Init commit for rust backend

ef3fe9a

Signed-off-by: GitHub <noreply@github.com>

Aisuko added the new-backend label Oct 17, 2023

lu-zero reviewed Oct 17, 2023

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/Makefile Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

Aisuko and others added 3 commits October 18, 2023 10:47

Update backend/rust/Makefile

029a71f

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Add tracing

5c67aa6

Signed-off-by: GitHub <noreply@github.com>

Add workspace

1806dd7

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko requested a review from lu-zero October 18, 2023 08:00

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/bunker/build.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Aisuko requested a review from lu-zero October 19, 2023 04:51

Replace the generated file to the generated folder

61bd269

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/service.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/lib.rs Outdated Show resolved Hide resolved

Update backend/rust/bunker/src/lib.rs

b92677b

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Oct 20, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Remove services.rs

a2bb86f

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko requested a review from lu-zero October 20, 2023 01:09

Add test health in Makefile

bc6c1fc

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero reviewed Oct 22, 2023

View reviewed changes

backend/rust/Cargo.lock Outdated Show resolved Hide resolved

Ignore Cargo.lock

7e9215f

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero reviewed Oct 23, 2023

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

Update .gitignore

bd087ca

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Add tracing

4397789

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko added 2 commits November 1, 2023 20:12

Add new model

c0dadcc

Signed-off-by: Aisuko <urakiny@gmail.com>

Implement a new simple model

fb67c91

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Nov 3, 2023

View reviewed changes

backend/rust/models/src/lib.rs Show resolved Hide resolved

Aisuko added 2 commits November 4, 2023 12:16

Implement MNIST model and inference

1d2fd99

Signed-off-by: Aisuko <urakiny@gmail.com>

Add check memory feature

660cc49

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko mentioned this pull request Nov 5, 2023

[EPIC] Model support dashboard (v2) #1126

Open

93 tasks

lu-zero reviewed Nov 16, 2023

View reviewed changes

Aisuko added 2 commits November 18, 2023 18:28

Trying to call mnist model in main

d62c701

Signed-off-by: Aisuko <urakiny@gmail.com>

Add test case for load model and import getusage

b210203

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Nov 19, 2023

View reviewed changes

backend/rust/backend/src/main.rs Show resolved Hide resolved

Add llama for test

c990112

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Nov 23, 2023

View reviewed changes

Aisuko closed this by deleting the head repository Apr 16, 2025


		let tensor3=tensor2.transpose();

		let tensor41=tensor3.repeat(2, 2);

Uh oh!

WIP feat:Init commit for rust backend #1180

WIP feat:Init commit for rust backend #1180

Conversation

Aisuko commented Oct 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mudler commented Oct 17, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Aisuko commented Oct 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Aisuko commented Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Aisuko commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lu-zero commented Oct 31, 2023

Uh oh!

Aisuko commented Oct 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Update

Uh oh!

lu-zero commented Nov 1, 2023

Uh oh!

Aisuko commented Nov 1, 2023

Uh oh!

Uh oh!

lu-zero Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

Aisuko Nov 17, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

netlify bot commented Nov 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Deploy Preview for localai failed.

Uh oh!

Aisuko Nov 23, 2023

Choose a reason for hiding this comment

Uh oh!

lu-zero Dec 21, 2023

Choose a reason for hiding this comment

Uh oh!

Aisuko Jul 2, 2024

Choose a reason for hiding this comment

Uh oh!

lu-zero Jul 2, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Aisuko commented Oct 17, 2023 •

edited

Loading

Aisuko commented Oct 22, 2023 •

edited

Loading

Aisuko commented Oct 26, 2023 •

edited

Loading

Aisuko commented Oct 31, 2023 •

edited

Loading

Aisuko commented Oct 31, 2023 •

edited

Loading

netlify bot commented Nov 23, 2023 •

edited

Loading