For Milvus 2.5.4, recommended max number of partitions and collections?

[xiaobingxia-at]

Hello, team, Milvus 2.5.4 has been released, and it’s a game-changer in many ways.

From the release notes, I noticed that we support 10k collections, and 1 million partitions. So it implies that we use 100 partitions per collections. I'm curious, which option below is recommended if I want to have 1 million partitions:

1. 10k collections, 100 partitions per collection
2. 200 collections, 5k partitions per collection

For each of above options, may I know the rough comparison for following dimension?

1. Query Performance / Ingestion Throughput: Option 1 vs Option 2
2. Load / Release partition time: Option 1 vs Option 2
3. The new query node recovery time (reload all data) after evicting/terminating one query pod: Option 1 vs Option 2

Thanks!

[xiaofan-luan]

I think both could work, there is not a big difference from both implementation.
I would prefer second choice because there will be less work to do on recovering on the channels.

[xiaofan-luan]

can you confirm this is related to GetDataDistribution call? @xiaobingxia-at

@bigsheeper
I remember we did some optimizaiton on the getDistribution call

	segmentVersionInfos = append(segmentVersionInfos, &querypb.SegmentVersionInfo{
		ID:                 s.ID(),
		Collection:         s.Collection(),
		Partition:          s.Partition(),
		Channel:            s.Shard().VirtualName(),
		Version:            s.Version(),
		Level:              s.Level(),
		LastDeltaTimestamp: s.LastDeltaTimestamp(),
		IndexInfo: lo.SliceToMap(s.Indexes(), func(info *segments.IndexedFieldInfo) (int64, *querypb.FieldIndexInfo) {
			return info.IndexInfo.FieldID, info.IndexInfo
		}),
	})

We can optimize it by:

1. change s.Shard().VirtualName() to shardIndex only
2. simplify indexInfo to only some index state

[xiaofan-luan]

I'm using 2.5.4, doing 4k collections, 100 partitions per collection, totally 400k partitions. When this thing happens, there are about 120k partitions. Actually maybe I was wrong, when the networking receiving I/O of coordinator surges, I checked that, only query nodes's networking receiving I/O rate surges at exactly same time window. So it is related with query nodes.

Even 2c16g node can offer 100MB+ network bandwitch and trafiic can be bursted to 1GB+.This might not be a blocker issue for 1M partitions and can definitely be improved in the future

[xiaobingxia-at]

I disabled the hourly clustering compaction, the hourly burst of coordinator networking I/O rate and CPU rate disappeared.
But given 400k partitions, the coordinator networking I/O still has 50+mb/s with query node.
This is not the bottleneck, but will cause costly and large volume cross-az traffic.

[xiaofan-luan]

I disabled the hourly clustering compaction, the hourly burst of coordinator networking I/O rate and CPU rate disappeared. But given 400k partitions, the coordinator networking I/O still has 50+mb/s with query node. This is not the bottleneck, but will cause costly and large volume cross-az traffic.

Do you know which call and which proto actaully cause this issue?

maybe we can enable grpc log for further analysis or use tcpdump or wireshark to analysis

[xiaobingxia-at]

I don't know which call and which proto causing this issue.
But here's how to replicate it:

1. Create a cluster with 500 collections, each collection contain 800 partitions, in total 400k partitions
2. Ingestion 500 rows to each partition one by one, once you fill up the 400k partitions, you will see the high networking i/o.