[Usage]: In the context of pp parallelism, UniformTypeKVCacheSpecs type, the calculation of page size considers all the layers， not only the layerss of one pp rank ?

[yangshanjun]

Your current environment

class UniformTypeKVCacheSpecs(KVCacheSpec):
"""
A KV cache spec for multiple layers with the same type of attention. Here,
same types means always need the same number of token slots. For example,
sliding window attentions with different window sizes are not the same type
and should not be merged into one UniformTypeKVCacheSpecs.
"""

kv_cache_specs: dict[str, KVCacheSpec]

@property
def page_size_bytes(self) -> int:
    return sum(spec.page_size_bytes for spec in self.kv_cache_specs.values())

How would you like to use vllm

I want to run inference of a [specific model](put link here). I don't know how to integrate it with vllm.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.