SG2002

[tcpipchip]

Hi, congratulations...

I see that has option to RISC-V SG2002...

We use the milk-duo series that has the SG2002...to process cvimodel inferences...

We compiled a SDK for it...

To run picolm...will we need to compile it on our SDK ?

Can you give us some tips ? Can it run too the TPU inferences ?

Thanks!

[tcpipchip]

Compiled on RISCV

[root@milkv-duo]~# ./picolm
PicoLLM — ultra-lightweight LLM inference engine

Usage: ./picolm <model.gguf> [options]

Generation options:
  -p <prompt>    Input prompt (or pipe via stdin)
  -n <int>       Max tokens to generate (default: 256)
  -t <float>     Temperature (default: 0.8, 0=greedy)
  -k <float>     Top-p / nucleus sampling (default: 0.9)
  -s <int>       RNG seed (default: 42)
  -c <int>       Context length override
  -j <int>       Number of threads (default: 4)

Advanced options:
  --json         Grammar-constrained JSON output mode
  --cache <file> KV cache file (saves/loads prompt state)
[root@milkv-duo]~#

Send me some example to test to see if all is good ? Need i donwload a model into SG2002 ?

[sunglee42]

hello I have SG2002 too
but it's LicheeRV Nano (with 256M RAM)

I want to try cross compile from arm platform
just do make riscv and it finish with no error
but I can't execute successfully (will show -sh: ./picolm: not found)
is anything you do too, but I didn't?
haha I forgot to set this LDFLAGS += -static

anyway if you want to run picolm, you need to do make model to download model
https://github.com/RightNow-AI/picolm?tab=readme-ov-file#build-from-source
so yes, model will in SG2002 (or in microSD)
and that will be tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

I try this on my raspberry pi zero
and it's super slow XD
wonder know if it will be faster on RISC-V...

ok...I try on SG2002 now
still pretty slow XDD

[tcpipchip]

yea, @sunglee42
I had to put too the -static too :)

# --- RISC-V (Sipeed LicheeRV, etc.) ---
riscv: CC = riscv64-linux-gnu-gcc
riscv: CFLAGS += -march=rv64gc -mabi=lp64d -static
riscv: $(TARGET)

Where to download the GGUF file to test ?

I will put in my USB PENDRIVE

[tcpipchip]

Did you try cvimodel inference on tpu of sg2002, using the picolm ?

[sunglee42]

@tcpipchip you can see in here

picolm/picolm/Makefile

Line 58 in 862c330

"https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"; \

I manually place the model into the device too
(also put my cross-compile file)
so I use this model to test on pi zero and LicheeRV Nano
the generating takes tens of minutes...might not unable to use
or this is not normal? (but their performance is poor too)

[tcpipchip]

i will test accessing the pendrive

[tcpipchip]

Slow :(

[sunglee42]

I think we might missing something
According to README.md
LicheeRV Nano (SG2002) should have ~1 tok/s (Generation Speed)

I finished my test with echo "Hello" | ./picolm tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf -n 100 -j 1
pi zeo is too old might be fine, but LicheeRV Nano is too low for sure

LicheeRV Nano (1 GHz single-coreRISC-V C906, 256 MB of RAM) + 32G microSD (C10, A1)

Prefill: 3 tokens in 165.76s (0.0 tok/s)
Generation: 38 tokens in 2107.78s (0.0 tok/s)
Total: 2273.53s
Memory: 45.17 MB runtime state (FP16 KV cache)

Raspberry Pi Zero (1 GHz single-core ARM11 CPU,  MB of RAM) + 16G microSD (C10, A1)

Prefill: 3 tokens in 199.15s (0.0 tok/s)
Generation: 38 tokens in 2335.64s (0.0 tok/s)
Total: 2534.79s
Memory: 45.17 MB runtime state (FP16 KV cache)

[tcpipchip]

😬
50% cpu use