Replacing Minio in the Standalone Package

[r0x07k]

Milvus uses Minio in a standalone setup, but there’s an issue because Minio is released under the AGPL license. As I understand it, this would not only require Milvus to use the AGPL for this distribution but also force its users to do the same. So, it might be better to switch to a more permissive alternative, like SeaweedFS.

[r0x07k]

I created a Docker Compose file for Milvus with SeaweedFS, but it didn’t work out of the box. It seems that further adjustments are needed.

[r0x07k]

I was able to run Milvus with this configuration eventually. I had to update the Filer's port to 8889 to avoid conflicts with Jupyter Notebook. Basic operations with a collection were successful.

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.5
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  master:
    container_name: milvus-seaweed-master
    image: chrislusf/seaweedfs:3.82
    ports:
      - 9333:9333
      - 19333:19333
      - 9324:9324
    command: "master -ip=master -ip.bind=0.0.0.0 -metricsPort=9324"
  volume:
    container_name: milvus-seaweed-volume
    image: chrislusf/seaweedfs:3.82
    ports:
      - 8080:8080
      - 18080:18080
      - 9325:9325
    command: 'volume -mserver="master:9333" -ip.bind=0.0.0.0 -port=8080  -metricsPort=9325'
    depends_on:
      - master
  filer:
    container_name: milvus-seaweed-filer
    image: chrislusf/seaweedfs:3.82
    ports:
      - 8889:8888
      - 18888:18888
      - 9326:9326
    command: 'filer -master="master:9333" -ip.bind=0.0.0.0 -metricsPort=9326'
    tty: true
    stdin_open: true
    depends_on:
      - master
      - volume
  s3:
    container_name: milvus-seaweed-s3
    image: chrislusf/seaweedfs:3.82
    ports:
      - 8333:8333
      - 9327:9327
    command: 's3 -filer="filer:8888" -ip.bind=0.0.0.0 -metricsPort=9327'
    depends_on:
      - master
      - volume
      - filer

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.4.1
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: s3:8333
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "s3"

networks:
  default:
    name: milvus

[r0x07k]

Alas, Milvus fails after some time.

2025-01-24 21:36:01     panic: failed to write files/insert_log/455546433213826024/455546433213826025/455546433214026057/0/455546433215029169: IO failed[key=files/insert_log/455546433213826024/455546433213826025/455546433214026057/0/455546433215029169]: Signed request requires setting up SeaweedFS S3 authentication: failed to write files/insert_log/455546433213826024/455546433213826025/455546433214026057/1/455546433215029170: IO failed[key=files/insert_log/455546433213826024/455546433213826025/455546433214026057/1/455546433215029170]: Signed request requires setting up SeaweedFS S3 authentication: failed to write files/insert_log/455546433213826024/455546433213826025/455546433214026057/100/455546433215029171: IO failed[key=files/insert_log/455546433213826024/455546433213826025/455546433214026057/100/455546433215029171]: Signed request requires setting up SeaweedFS S3 authentication: failed to write files/insert_log/455546433213826024/455546433213826025/455546433214026057/101/455546433215029172: IO failed[key=files/insert_log/455546433213826024/455546433213826025/455546433214026057/101/455546433215029172]: Signed request requires setting up SeaweedFS S3 authentication: failed to write files/insert_log/455546433213826024/455546433213826025/455546433214026057/102/455546433215029173: IO failed[key=files/insert_log/455546433213826024/455546433213826025/455546433214026057/102/455546433215029173]: Signed request requires setting up SeaweedFS S3 authentication: failed to write files/stats_log/455546433213826024/455546433213826025/455546433214026057/100/455546433215029174: IO failed[key=files/stats_log/455546433213826024/455546433213826025/455546433214026057/100/455546433215029174]: Signed request requires setting up SeaweedFS S3 authentication

[xiaofan-luan]

@r0x07k
Hi Pavel,

Minio SDK is under hadoop licesne, the server is using AGPL but does that really matters?
I would recommend to use GCS or S3 in your production environment.

Also I think seaweed fs is compatible with MInio/S3 SDK, the reason of the failure

Signed request requires setting up SeaweedFS S3 authentication

[r0x07k]

Hi!

Some of our clients would like to deploy a fully local open-source solution, but AGPL is banned in many companies because it's too broadly formulated and can cause problems even if the software is used as is.

Could you please advise what exactly is missing in SeaweedFS that prevents Milvus from running, so I can create a ticket for them? As I said Milvus starts and even saves some data but fails afterwards.

[xiaofan-luan]

I think it's a authentication issue, maybe you need to tune the config a little bit.

SeaweedFS can be accessed through S3 sdk so I don't think there are any compatibility issue