overcompressible data with blockvarpct

[jayrajput0]

Hello Sven,

Hope you are doing well!

I am running a benchmark on a ceph cluster, my goal is to evaluate compression overhead with object workload.

• I am using --blockvarpct 50 with elbencho to have compressible data, zlib compression algorithm is configured on my pools.
• The data getting over compressed, see the default.rgw.buckets.data pool row, the last two columns.

Explanation for last two columns:

• USED COMPR: The amount of space allocated for compressed data including his includes compressed data, allocation, replication and erasure coding overhead.
• UNDER COMPR: The amount of data passed through compression and beneficial enough to be stored in a compressed form.

# ceph df detail
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    503 TiB  495 TiB  7.5 TiB   7.5 TiB       1.50
TOTAL  503 TiB  495 TiB  7.5 TiB   7.5 TiB       1.50

--- POOLS ---
POOL                       ID   PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
.mgr                        1     1   13 MiB   13 MiB      0 B        6   38 MiB   38 MiB      0 B      0    156 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.index  85   256  271 MiB      0 B  271 MiB      415  814 MiB      0 B  814 MiB      0    156 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.meta           86    32   42 KiB  4.2 KiB   38 KiB       19  306 KiB  192 KiB  114 KiB      0    156 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.log            87    32   28 KiB   28 KiB      0 B      323  2.3 MiB  2.3 MiB      0 B      0    156 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.control        88    32      0 B      0 B      0 B        9      0 B      0 B      0 B      0    156 TiB            N/A          N/A    N/A         0 B          0 B
.rgw.root                  89    32   11 KiB   11 KiB      0 B       19  216 KiB  216 KiB      0 B      0    156 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.data   90  4096  4.5 TiB  4.5 TiB      0 B    1.53M  7.2 TiB  7.2 TiB      0 B   1.52    156 TiB            N/A          N/A    N/A     469 GiB      6.8 TiB

• I am using below syntax for benchmarking.

 elbencho --hosts $ebHosts --numhosts $numhost --s3endpoints $ebEndpoints --s3key $access --s3secret $secret -r -s $size -b $chunksize -t $threads -n $numhost -N $objects --timelimit $testduration --infloop --nolive --lat --latpercent --latpercent9s 0 --direct --csvfile=pgd-${size}-${threads}t.csv --resfile=pgd-${size}-${threads}t.out --port 13001 --label ${threads}t-$comment --s3ignoreerrors --treescan s3://${size}-${threads}t-${comment}-bucket ${additionalargs} ${size}-${threads}t-${comment}-bucket &>> elbencho-${size}-${threads}t.out

• Do you think it's due to the data buffer pattern? any feedback or suggestions to avoid this over compression?

• Just for the sanity check I used fio (obviously for block workload) with buffer_compress_percentage=50 to see how the results looks like, and I see almost the 50% of compression there.

• Point to note that fio enables refill_buffers by default with buffer_compress_percentage to avoid likelihood of over compression.

[jayrajput0]

Hello Sven,
Is it possible that this ticket get your attention on priority,It's blocking me on a very important project.
I have been trying a couple of options available with elbencho, but noned seems to helping.

Any feedback on this would help.

======With --blockvarpct 50 --blockvaralgo strong =====

# ceph df detail
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    503 TiB  502 TiB  1.2 TiB   1.2 TiB       0.24
TOTAL  503 TiB  502 TiB  1.2 TiB   1.2 TiB       0.24

--- POOLS ---
POOL                       ID   PGS   STORED   (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
.mgr                        1     1   14 MiB   14 MiB     0 B        7   41 MiB   41 MiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.index  91   256   27 MiB      0 B  27 MiB       83   81 MiB      0 B  81 MiB      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.meta           92    32   18 KiB  1.8 KiB  16 KiB       11  145 KiB   96 KiB  49 KiB      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.log            93    32   28 KiB   28 KiB     0 B      355  2.3 MiB  2.3 MiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.control        94    32      0 B      0 B     0 B        9      0 B      0 B     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.data   95  4096  1.1 TiB  1.1 TiB     0 B  300.07k  934 GiB  934 GiB     0 B   0.19    317 TiB            N/A          N/A    N/A      55 GiB      879 GiB
.rgw.root                  96    32   11 KiB   11 KiB     0 B       19  216 KiB  216 KiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B

===== With --blockvarpct 50 --randalgo strong =============

Every 2.0s: ceph df detail                                                                                                                                

--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    503 TiB  502 TiB  1.2 TiB   1.2 TiB       0.24
TOTAL  503 TiB  502 TiB  1.2 TiB   1.2 TiB       0.24

--- POOLS ---
POOL                        ID   PGS   STORED   (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
.mgr                         1     1   14 MiB   14 MiB     0 B        7   41 MiB   41 MiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.index   97   256   28 MiB      0 B  28 MiB       83   85 MiB      0 B  85 MiB      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.meta            98    32   19 KiB  1.8 KiB  18 KiB       11  149 KiB   96 KiB  53 KiB      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.log             99    32  8.5 KiB  8.5 KiB     0 B      196  804 KiB  804 KiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.control        100    32      0 B      0 B     0 B        9      0 B      0 B     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.data   101  4096  1.2 TiB  1.2 TiB     0 B  306.83k  955 GiB  955 GiB     0 B   0.20    317 TiB            N/A          N/A    N/A      56 GiB      899 GiB
.rgw.root                  102    32   11 KiB   11 KiB     0 B       19  216 KiB  216 KiB     0 B      0    159 TiB            N/A          N/A    N/A         0 B          0 B

[breuner]

Hi @jayrajput0 ,

regarding "--blockvarpct 50": The way this works is by having the first of 50% of the bytes in each written block (as in -b/--block) filled with completely random data and the remainder of each block filled with the same 64bit number written again and again. This makes the first 50% incompressible and the remaining 50% of each block perfectly compressible for a 2:1 data reduction ratio.

(elbencho always picks completely random numbers as seeds, which is equivalent to the fio parameter refill_buffers being set.)

That's why I can't understand how ceph can claim any different reduction ratio than 2:1 when you set "--blockvarpct 50". (You will probably be able to easily verify this by creating a tar gzip file of some of the data that you have written via elbencho with "--blockvarpct 50" to confirm that it compresses not more than 2:1.)

Does it correctly show no compression at all when you set the default "--blockvarpct 100"?

I assume we can exclude the case that there is other data (not written by elbencho) on the default.rgw.buckets.data pool that would impact the compression number that the ceph df is showing?

Tbh I'm not very familiar with this ceph output, so I consulted an AI tool to better understand the meaning of the output. Thus, what I'm saying now might be total AI non-sense - hopefully you can tell if it makes sense or not. The AI was pointing out that the numbers seem strange in the sense that the "USED COMPR" column shows a relatively small value in comparison to the "UNDER COMPR" column, which usually indicates a high compression ratio. But at the same time the "DATA" and "USED" columns (which supposedly show the actual usage on the drives) is showing a value that is significantly higher than the amount of uncompressed data in the "UNDER COMPR" column.

So despite the fact that the output is indicating a 16:1 compression, the disk usage does not seem to support that.

The AI is suggesting to check if the min alloc size (e.g. ceph config get osd bluestore_min_alloc_size_ssd) is maybe larger than the written objects.

[jayrajput0]

Thank you for looking in to it @breuner

yeah, I have the same expectation with blockvarpct .

The tests without blockvarpct (effectively default blockvarpct :100) then I see no compression, which is expected. See the last two columns showing 0 0.

--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    503 TiB  501 TiB  2.0 TiB   2.0 TiB       0.39
TOTAL  503 TiB  501 TiB  2.0 TiB   2.0 TiB       0.39

--- POOLS ---
POOL                        ID   PGS   STORED   (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
.mgr                         1     1   15 MiB   15 MiB     0 B        7   44 MiB   44 MiB     0 B      0    158 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.index  124   256   27 MiB      0 B  27 MiB       83   81 MiB      0 B  81 MiB      0    158 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.meta           125    32   23 KiB  1.8 KiB  21 KiB       11  159 KiB   96 KiB  63 KiB      0    158 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.log            126    32  8.5 KiB  8.5 KiB     0 B      196  804 KiB  804 KiB     0 B      0    158 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.control        127    32      0 B      0 B     0 B        9      0 B      0 B     0 B      0    158 TiB            N/A          N/A    N/A         0 B          0 B
default.rgw.buckets.data   128  4096  1.1 TiB  1.1 TiB     0 B  289.96k  1.7 TiB  1.7 TiB     0 B   0.35    317 TiB            N/A          N/A    N/A         0 B          0 B
.rgw.root                  129    32   11 KiB   11 KiB     0 B       19  216 KiB  216 KiB     0 B      0    158 TiB            N/A          N/A    N/A         0 B          0 B

• I am reaching out to ceph developers as well in parallel but trying to rule out the data pattern possibilities as well, because fio shows expected results.

[breuner]

Hi @jayrajput0 ,

I did set up a mini ceph cluster using quay.io/ceph/demo to see how the compression numbers show up when I write a certain amount of data.

I enabled compression via these two commands:

$ docker exec -it ceph-demo ceph osd pool set default.rgw.buckets.data compression_mode aggressive
$ docker exec -it ceph-demo ceph osd pool set default.rgw.buckets.data compression_algorithm zstd

Before I wrote any data, the pool was completely empty and looked like this:

$ docker exec -it ceph-demo ceph df detail
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
TOTAL  100 GiB  100 GiB  31 MiB    31 MiB       0.03
 
--- POOLS ---
POOL                       ID  PGS   STORED   (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
[...]
default.rgw.buckets.data    8   64      1 B      1 B     0 B        1    4 KiB    4 KiB     0 B      0     95 GiB            N/A          N/A    N/A         0 B          0 B

Then I wrote exactly 1000MiB of data to it via 10 threads, each writing 100 objects of 1MiB size and using "--blockvarpct 50", for a 2:1 compression ratio.

This is the elbencho command that I used:

$ elbencho -w -s 1m --blockvarpct 50 -n 0 -N 100 -t 10 --s3endpoints http://192.168.10.47:9085 demobucket
OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
WRITE       Elapsed time     :      1.509s      2.090s
            Objects/s        :         546         478
            Throughput MiB/s :         546         478
            Total MiB        :         825        1000
            Objects total    :         825        1000
---

This is what the pool looks like after the write:

$ docker exec -it ceph-demo ceph df detail
--- RAW STORAGE ---
CLASS     SIZE   AVAIL     USED  RAW USED  %RAW USED
TOTAL  100 GiB  99 GiB  807 MiB   807 MiB       0.79
 
--- POOLS ---
POOL                       ID  PGS    STORED    (DATA)  (OMAP)  OBJECTS     USED   (DATA)  (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
[...]
default.rgw.buckets.data    8   64  1000 MiB  1000 MiB     0 B    1.00k  750 MiB  750 MiB     0 B   0.77     94 GiB            N/A          N/A    N/A     250 MiB      500 MiB

This result looks reasonable to me.

When using "--blockvarpct 50" then 50% of the data are not compressible, that's why only 500MiB out of the 1000MiB show up under the "UNDER COMPR" column.

Why the zlib algorithm is not able here to compress the easily compressible 500MiB of data to less than 250MiB is something that I can't tell, but at least otherwise it all looks reasonable. Maybe there's a way to get ceph to try harder to compress the data?

When I try the same with a simple tar+gzip then we see that it perfectly compresses to 50% of the original size:

# Write again 1000MiB of data using "--blockvarpct 50", as before, but this time as files for tar:
$ elbencho -w -s 1m --blockvarpct 50 -n 0 -N 100 -t 10 tmp/
OPERATION   RESULT TYPE         FIRST DONE   LAST DONE
=========== ================    ==========   =========
WRITE       Elapsed time     :        38ms        54ms
            Files/s          :       23212       18239
            Throughput MiB/s :       23212       18239
            Total MiB        :         889        1000
            Files total      :         889        1000
---

# check size of written data on disk without compression:
$ du -sh tmp/
1001M	tmp/

# compress the data using "tar -z" for gzip compression:
$ tar -czf tmp-compressed.tgz tmp/

# check the size of the tar archive to confirm that it is 50% of the original data size:
$ du -sh tmp-compressed.tgz 
501M	tmp-compressed.tgz

[jayrajput0]

Thank you very much for testing it. It's strange, I still see the same result with Zstd as well. By any chance do you have ceph version you used for the reproduction.
ceph versions

• Anyway, I am going to follow up with ceph devs
• Thank you again.

[breuner]

hi @jayrajput0 ,
here's the ceph version from my test:

$ docker exec -it ceph-demo ceph version
ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable)

this is the docker-compose.yaml file that i'm using, just in case you also want to try this for your own testing or for comparison to your other system:

services:
  ceph-demo:
    image: quay.io/ceph/demo:latest
    container_name: ceph-demo
    # Host networking is required for the Monitors to advertise correctly
    network_mode: "host"
    environment:
      # CRITICAL: Change to actual LAN IP (e.g. 192.168.x.y)
      # Do not use 127.0.0.1 or localhost, or the OSDs will fail to connect to the Mon
      - MON_IP=192.168.10.47

      # The subnet of your host network (e.g., 192.168.x.y/24)
      - CEPH_PUBLIC_NETWORK=192.168.10.47/24

      # S3 port
      - RGW_FRONTEND_PORT=9085

      # S3 access configuration
      - CEPH_DEMO_UID=myuser
      - CEPH_DEMO_ACCESS_KEY=demoaccesskey
      - CEPH_DEMO_SECRET_KEY=demosecretkey
      - CEPH_DEMO_BUCKET=demobucket

      - CEPH_DEMO_DAEMON_WHITELIST=mon,mgr,osd,rgw,mds
    volumes:
      # Persistent data dirs outside of container
      - ./ceph/etc:/etc/ceph
      - ./ceph/lib:/var/lib/ceph