how to load large number of data into databend table

[qicaiyulin]

when I load a text file as csv data into the databend table, I got the error like this:
Code: 4000, displayText = other error (connection closed before message completed) (while in processor thread 34).
The size of the text file is 2.2g and the line number is 14404374. The storage of my databend cluster is minIO. How can I load data from the minio directly when the data has been put into the minio. Or Is there any other method to load large number of data.

[Xuanwo]

Hi, we need the following information to better help you:

• What's the load method you are using now?
• Can you share the detailed log file with us? (often located at ./databend/logs)

And the following links may be helpful to you: https://databend.rs/doc/reference/sql/dml/dml-copy-into-table

COPY INTO mytable
  FROM 's3://mybucket/data.csv'
  CONNECTION = (
        ENDPOINT_URL = 'http://localhost:9000'
        ACCESS_KEY_ID = '<your-access-key-ID>'
        SECRET_ACCESS_KEY = '<your-secret-access-key>')
  FILE_FORMAT = (type = 'CSV' field_delimiter = ','  record_delimiter = '\n' skip_header = 1) size_limit=10;

[qicaiyulin]

I load data from minio like this:
copy into test.call_center from 's3://databend-2/tmp/call_center.dat' CONNECTION =(endpoint_url = 'http://10.3.73.110:9000' ACCESS_KEY_ID = 'xxxx' SECRET_ACCESS_KEY = 'xxxxxx') file_format=(type='CSV' field_delimiter='|' record_delimiter='\n' shik_header=0);
But get the error like this:
ERROR 1105 (HY000): Code: 3903, displayText = copy from insecure storage is not allowed.
What can I do for this problem?

[Xuanwo]

By default, databend-query doesn't support load from insecure endpoints.

Please enable allow-insecure in config:

[storage]
allow_insecure = true

[qicaiyulin]

Now I can load data from minio ,but very slow. How can I do to improving efficiency? I hive 3 nodes to deploy the databend cluster. The cluster has 3 meta instances(each node has 1 meta instance) and 6 query instances(each node has 2 query instance) .
Cluster information like this:
MySql> select * from system.clusters;
+------------------------+-------------+------+
| name | host | port |
+------------------------+-------------+------+
| 1Sqbc5UROnnwmE9krKE023 | 10.3.69.137 | 9094 |
| 2vfLtElFfksS9EBlrsaL03 | 10.3.73.110 | 9094 |
| RM9xJHQkY7DVvlJ8bgHD52 | 10.3.73.110 | 9095 |
| ez7uwU1BVml3RxozbkujL | 10.3.69.200 | 9094 |
| hGxkCwgsAR2aKgYs0bk4h2 | 10.3.69.200 | 9095 |
| mlUtdqT1hislcoA1MyPFg | 10.3.69.137 | 9095 |
+------------------------+-------------+------+

[Xuanwo]

cc @wubx for advice.

[BohuTANG]

Because you have one big file : s3://databend-2/tmp/call_center.dat', if you have many piece files, I think COPY would be parallel and faster, and each query node copy one directory, so the cluster will be in parallel and faster.

[qicaiyulin]

1,the load method I used is streaming_load.
2. The reason seems to be this. When I load data , the minIO server will stop suddenly.

[Xuanwo]

1,the load method I used is streaming_load.

You can try COPY which will load from minio directly.

When I load data , the minIO server will stop suddenly.

Weird. Can you share the minio's log?

[qicaiyulin]

minio logs:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x4ac832]

goroutine 7108 [running]:
io.(*nopCloser).Read(0xc019994cf0, 0xc003856000, 0x2000, 0x2000, 0x40db9b, 0xc019994ce0, 0x40a6bf)
:1 +0x32
io.discard.ReadFrom(0x7f28b059ce88, 0xc019994cf0, 0x1d141e0, 0x1, 0x7f28b0002b70)
io/io.go:598 +0x95
io.copyBuffer(0x279d760, 0x3494ec0, 0x7f28b059ce88, 0xc019994cf0, 0x0, 0x0, 0x0, 0x0, 0xc019994c01, 0xc0015f3d40)
io/io.go:409 +0x357
io.Copy(...)
io/io.go:382
github.com/minio/minio/cmd/http.DrainBody(0x27b5778, 0xc019994cf0)
github.com/minio/minio/cmd/http/close.go:40 +0xdd
github.com/minio/minio/cmd.(*storageRESTClient).CreateFile(0xc0005b0f00, 0x27cc988, 0xc0001aa018, 0x1f42bbe, 0xe, 0xc001f79720, 0x50, 0x2e6a53, 0x279ac60, 0xc0006ac828, ...)
github.com/minio/minio/cmd/storage-rest-client.go:346 +0x53c
github.com/minio/minio/cmd.newStreamingBitrotWriter.func1(0x2e6873, 0x33334, 0x27d15c0, 0xc002f56820, 0xc0006ac828, 0x27f3658, 0xc0005b0f00, 0x1f42bbe, 0xe, 0xc001f79720, ...)
github.com/minio/minio/cmd/bitrot-streaming.go:85 +0x104
created by github.com/minio/minio/cmd.newStreamingBitrotWriter
github.com/minio/minio/cmd/bitrot-streaming.go:79 +0x2bd

[Xuanwo]

It seems a minio bug...

Can you try upgrading minio instead? We don't know what happened to minio.

[wubx]

Please provide some informathion about minio:

1. minio version
2. startup command
3. databend-query minio endpoint confgiure

[qicaiyulin]

1, minio version
minio.RELEASE.2021-03-26T00-00-41Z
2, startup command

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=admin776655
nohup /opt/soft/minio/minio server http://node{1...3}.bigdata/mnt/ssd{1...6}/minioData >> /root/minio.log 2>&1 &

3 databend-query minio endpoint confgiure

[storage]
# fs | s3 | azblob
type = "s3"
[storage.s3]
bucket = "databend-1"
endpoint_url = "http://10.3.73.110:9000"
access_key_id = "admin"
secret_access_key = "admin776655"
allow_insecure = true

[wubx]

Please upgrade minio to latest version.