Crazy virtual mem usage?

[linas]

I'm seeing a harmless but alarming behavior: virtual memory has blown up to 616 GB, but RSS stays at 40GB. Virtual is much larger than what the machine has in physical RAM, but the OS seems happy, cause RSS is low (and no swap usage). Any clue what is causing this? (As I wrote this, I decided it is badly fragmented heap...)

System is using about 20 threads to do LG parsing from a huge dictionary, and it has run 16 hours, now. Most of the 40GB RSS is NOT LG, but other things. However, during parsing, sometimes I hit #1402 (Explosive RAM usage) which is now fixed, because RAM usage is now clamped to a reasonable value. But, with 20 threads, if half of them are "unlucky", RSS does get large -- up to 165GB, but then it shrinks back down after sentence delete.

But Virt...well, it doesn't grow, but overnight, it got large... while I am writing this, it shrank from 616 GB to 595 GB. Any clue what this is about? Is this a badly fragmented heap? I guess it has to be... OK, never mind, that is what it must be. Huh.

[ampli]

Any clue what is causing this? (As I wrote this, I decided it is badly fragmented heap...)

I don't really know, but I suspect that, among other things, pool_alloc() may waste half of the memory (due to typically allocating power-of-2 size blocks). However, this will only inflate the virtual memory by a factor of two and only for memory pool elements.

In case the reason is fragmentation, we can try to see if the following idea helps:

1. Define a per-thread struct for all the memory pools now in Sentence_s.
2. On sentence delete(), use pool_reuse() instead of pool_delete().
3. On pull_new(), use the memory pools from (1).

We may consider adding an API call to the library to initialize it for multi-threading, which will define (1) (and possibly other one-time things).

up to 165GB, but then it shrinks back down after sentence delete.

We can also experience with madvice(); for example, we can mark the tracon table pages as MADV_RANDOM while parsing and extracting links and with MADV_SEQUENTIAL while growing the table. The disjunct/connector memblock may also be marked as MADV_RANDOM while parsing/extracting.

it shrank from 616 GB to 595 GB

Maybe due to malloc_trim() in free_extractor()?

(BTW, I'm also working on improving the library, including saving memory, reducing cache table accesses, converting lists to arrays, avoiding random accesses, etc.)

[linas]

The parsing happens at the same time as extremely rapid malloc/free of small blocks in other threads. This is just C++ doing its thing with std::vector and std::string and smart pointers. Which is why I suspect fragmentation.

FYI, if I use the any parser, there's no explosive ram usage (of course: there is only one connector type!) and LG runs fine in 60 threads at top speed. So that's pretty cool. Small victory, I guess.

[ampli]

Can the vector memory just be deleted at once at certain points, like sentence_delete()? I ask because there may be a way to use a pool allocator for them. This may also speed things up due to bulk allocations/frees (or even reuse).

[linas]

Can the vector memory just be deleted at once at certain points, like sentence_delete()

Ah, the use-case is reversed from what you imagine. LG and the Atomspace interact in two completely different, unrelated ways. You've seen one, where LG is "in charge", and pulling words out of an AtomSpace dict.

The other use-case, where you haven't seen, is a processing pipeline, running in a server, where the server calls LG to get parses, wraps up the parse results, and stuffs them into a "different" AtomSpace. There, the server is "in charge", LG has no choice in what's happening, and most mallocs/frees happen outside of LG (when LG is not running).

I've thought about using pools, but measuring real-world performance is hard; the synthetic benchmarks aren't all that useful, and its a big pain in the neck to recompile everything and run the tests. Usually for negative results. The default malloc/free seem pretty good, and hard to beat, based on what I've read online.

[ampli]

The default malloc/free seem pretty good, and hard to beat, based on what I've read online.

Would not allocate, e.g., 10K items at once (instead of 10k malloc() calls), plus potentially zeroing out the big block at once instead of zeroing out each allocation separately, and not freeing at all until finishing the current request, essentially be much faster?

[linas]

Some of the alloc'ed things are never freed; -- The AtomSpace is an in-RAM database, so when you poke things in there, they stay there "forever". This is mixed in with pipeline processing, which creates temporaries, of varying lifetime. There's no particular way of knowing the lifetime of an object.

For example, during input, a (PhraseNode "this is a sentence that LG will parse") is created. After parsing, I get back a bunch of (WordNode "this") which are kept around "forever". The original PhraseNode is now freed. So the PhraseNode would be a temporary, that had a lifetime of however-long the parse takes (fractions of a second to tens of seconds)

Other mallocs can last hours or days, e.g. counts on things, which last until the next time the count is updated, which could be seconds or hours or days.

[linas]

I'm closing this discussion, since this appears to be "normal" behavior. and I don't see any easy alternatives. So, "noted" and "interesting" bu "won't fix".