open-local(kernel 4.18)限流功能没有生效 #233
Replies: 2 comments 3 replies
-
|
相同的困惑 in 0.6.6 |
Beta Was this translation helpful? Give feedback.
-
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"open-local-lvm-io-throttling"},"parameters":{"bps":"1048576","iops":"1024","volumeType":"LVM"},"provisioner":"yodaplugin.csi.alibabacloud.com","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
creationTimestamp: "2023-09-06T06:16:22Z"
name: open-local-lvm-io-throttling
resourceVersion: "16781492"
uid: 72cd9992-4781-464b-8aa9-712b2b52fcde
parameters:
bps: "10485760"
iops: "1024"
volumeType: LVM
provisioner: local.csi.aliyun.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: Service
metadata:
name: test-io-throttling
labels:
app: test-io-throttling
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: test-io-throttling
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: test-io-throttling
spec:
selector:
matchLabels:
app: test-io-throttling
podManagementPolicy: Parallel
serviceName: "test-io-throttling"
replicas: 1
volumeClaimTemplates:
- metadata:
name: html
spec:
accessModes:
- ReadWriteOnce
storageClassName: open-local-lvm-io-throttling
resources:
requests:
storage: 5Gi
template:
metadata:
labels:
app: test-io-throttling
spec:
tolerations:
- key: node-role.kubernetes.io/control-plane
operator: Exists
effect: NoSchedule
containers:
- name: test
image: xridge/fio
imagePullPolicy: Always
volumeMounts:
- mountPath: "/data"
name: html
command:
- sh
- "-c"
- |
while true; do
echo "huizhi testing";
echo "yes ">>/data/yes.txt;
sleep 120s
done;already validated on environment: OS: 3.10.0-1160.76.1.el7.x86_64 |
Beta Was this translation helpful? Give feedback.
-
|
It should be the kernel version, cgroupv1 and cgroupv2 coexistence after kernel 3.10 , even if we use cgroupv1 in kernel 4.18 or higher, k8s: 1.23, but in kernel 3.10 ,There are some known cgroup memory leak issues. suc as : |
Beta Was this translation helpful? Give feedback.
-
|
Similar issue with kubelet cgroup not updating: kubernetes/kubernetes#71233 |
Beta Was this translation helpful? Give feedback.
-
|
On kernel version 4.18, the directIO speed limit for blkio of cgroupv1 does not work.
To simulate the correspondence between pods and containers, I created a parent control group [root@master-1d692-0 blkio]# pwd
/sys/fs/cgroup/blkio
[root@master-1d692-0 blkio]# mkdir rg1
[root@master-1d692-0 rg1]# mkdir rg2
[root@master-1d692-0 rg1]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 40G 0 disk
├─vda1 252:1 0 1G 0 part /boot
└─vda2 252:2 0 39G 0 part
├─centos-root 253:0 0 35G 0 lvm /
└─centos-swap 253:1 0 4G 0 lvm
vdb 252:16 0 300G 0 disk
└─open--local--pool--0-local--eaddd584--2663--4f01--b565--7a167eda1ad3 253:2 0 5G 0 lvm /var/lib/kubelet/pods/ad15b1fd-5c51-4e3f-9b5d-4710ab7779f8/volumes/kubernetes.io~csi/local-eaddd584-2663-4f01-b565-7a167eda1ad3/mount
vdc 252:32 0 80G 0 disk /test
[root@master-1d692-0 rg1]# cd /test/
[root@master-1d692-0 test]# echo $$ > /sys/fs/cgroup/blkio/rg1/rg2/tasks
[root@master-1d692-0 test]# echo "252:32 1048576" > /sys/fs/cgroup/blkio/rg1/blkio.throttle.read_bps_device
[root@master-1d692-0 test]# dd if=/dev/zero of=./test.img bs=10M count=1
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 0.0126231 s, 831 MB/s
[root@master-1d692-0 test]# dd if=./test.img of=/dev/null bs=10M count=1 iflag=direct
1+0 records in
1+0 records out
10485760 bytes (10 MB) copied, 10.0021 s, 1.0 MB/sHowever, on kernel version 4.18, when I performed the same operation and tested it using dd, I found that the limitation was not effective. [root@master-1dc2f-1 test]# dd if=./test.img of=/dev/null bs=10M count=1 iflag=direct
记录了1+0 的读入
记录了1+0 的写出
10485760字节（10 MB，10 MiB）已复制，0.0163331 s，642 MB/s |
Beta Was this translation helpful? Give feedback.
-
环境信息
操作步骤
按照部署文档， 新增限流sc：
修改statefulset的sc为
open-local-lvm-io-throttling, pod的volumeMode为Filesystem, 在pod内部使用dd命令测试IO性能:显然没有限制住.
初步怀疑和
volumeMode有关系, 于是使用example/lvm/sts-io-throttling.yaml进行部署验证, 但显示仍没有生效:查看cgroup/blkio的信息, 相关限制已经被写进去了.
在宿主机内查看对应cgroup的blkio的限制是有值的, 但是在pod内查看却没有任何信息:
pod内查看
宿主机查看
在pod内执行fio测试命令, 有告警
全部测试信息:
创建一个限流卷的csi-plugin日志如下:
期望的带宽应该是1024KiB/s. 但现在远远没有限制住. 有人知道这是什么原因吗?
Beta Was this translation helpful? Give feedback.
All reactions