Max unique subjects in a single jetstream

[suchen-sci]

Hi, team!

Suppose I create a jetstream in following way:

cfg := jetstream.StreamConfig{
		Subjects:  []string{"events.id.>"},
	}

And there may be different ids to be published to this stream, like events.id.123 or events.id.abc.
So, for a single stream, how many different subjects is allowed in max? Is it possible to put like 50000 different subjects into a single jetstream?

Or any advice to split large number of subjects into multiple jetstreams?

Thanks a lot!

[derekcollison]

Yes you can create a stream that way and place messages into it which have unique subjects.

Currently the recent versions of the the server, >= 2.10.9, use a modified Adaptive Radix Trie to store in memory index data per subject that identifies metadata about the subject. Currently that is number of messages and first and last blocks. Assuming you are talking about a filestore backed stream.

The stree (ART impl) uses path compression and lazy expansion so is efficient at the types of subjects you have mentioned, since only the suffix needs to be stored at the leafs.

We do have plans to implement a version that can be stored to disk as well, but currently this is all kept in memory. 50k subjects should be fine.

[suchen-sci]

Thanks a lot for your help!!!

[bjorndm]

As a test I stored 1 million small JSON files in a single JetStream Key Value store, and it only took 2min30s on a 2 year old MacBook Pro.