Does the type of RAM matter and how much do you need?

[Thresher12]

I currently have about 16GB plain old RAM (not VRAM) and am planning to upgrade to higher capacity sticks to use larger models. My question is twofold.

First is the type of RAM I get going to have a significant effect. Should I for example splashout for DDR5 over DDR4 and get the highest Mhz rating I can find?

Secondly how much RAM should I get? I can relatively easily get to 64GB. Upgrading beyond will necessitate building a new system. What would I miss by sticking to 64GB? I see the short list of memory requirements on the frontpage that I assume are llama models but I wanted to get more detailed opinions.

[FreedomCoder-dev]

Running larger models will require powerful CPU and even using 65B model is a big deal for top processors. Of course you should choose higher speeds just because the model is being read from memory for every computation, but firstly check what your motherboard supports. Having 64Gb RAM shall allow you to run 65B parameters LLM, but still it's a hard task for CPU.

[pugzly]

My two cents. Larger amount of RAM will only allow to load larger models.
While the speed of memory is pretty much directly correlates to speed for inference. I originally kept my 32GB DDR4 kit under-clocked at 2666MHz (I like keeping everything under-clocked by default), but by upping it up to 3200GHz (its designed speed, which is ~20% increase) resulted in pretty much ~20% increase in inference speed on my machine. Which I think is a strong indication that memory speed is the primary variable for inference speed, at least on my system. My CPU (R7 5700G) seems to be underutilized, despite already being one of the cheapest and slowest budget CPUs out there.
Based on that, the next machine I will be building, most surely going to be DDR5 with the fastest 64GB-128GB memory kit I will be able to find. I will rather save up on CPU, SSD, everything else, but not the memory.

[LynxPDA]

I can also confirm that with large models, memory becomes a bottleneck.
I have a Ryzen 9 7950X and with it I didn't get that impressive boost due to memory speed.
On a 30B_q_4 I get about 380ms/token. Memory overclocking is linearly correlated with generation speed.

[Derpford]

I'm having trouble running a 20b model with 32gb of DDR4 RAM--even if I close everything else out, Windows (plus Edge as an interface) uses 4 to 6 gb, and while the model itself typically only takes up about 20gb after that, processing a prompt rapidly consumes the rest of my memory and starts hammering my swap. It does this whether I have --mmap and --smartcontext on or not. (I know it's swap, because it's my C drive--the model I'm using is stored on another drive.)

Does anybody know if there's any other tricks I can do? I'm not exactly in a position to upgrade my ram right now.

[JWNoctis]

Try switching to cmdline to avoid UI and browser overhead, which are significant.

Switching to Linux should also help, since most distributions tend to use much less memory than that. Latest Ubuntu LTS version should work fine. The cmdline interface works better there, too.

[Derpford]

I wasn't aware there was a command line UI.