Can I use databend without copying data?

[kesavkolla]

I have very large dataset already in s3 (parquet files). I don't want to copy that data again into databend. Is it possible to use databend query engine to run queries against data which is s3?

[BohuTANG]

Not yet, we are working on it, tracking issue: #7211

[kesavkolla]

In that case can fuse engine be accessible via arrow flight? Would be nice to run query engine via databend and the results are sent via arrow flight.

[BohuTANG]

That's easy to do.
Now Databend supports HTTP/MySQL handler, we add an Arrow flight handler https://github.com/datafuselabs/databend/tree/main/src/query/service/src/servers will do it.

[sundy-li]

#8857

[dpy1123]

@kesavkolla u can use apache drill to directly query from ur exsiting s3 files, but drill for s3 does not have query index.

[kesavkolla]

I don't think drill can leverage apache arrow for doing native executions. There is lot of value in arrow and it's native expression engine.

[gianluca-moro]

Any updates on this? I have the same situation: Large files in s3 which I want to directly query.

[BohuTANG]

Yes, Databend can query data from parquet format file, but it will slower than FUSE engine, doc is here:
https://databend.rs/doc/sql-functions/table-functions/stage_table_function#query-examples

[gianluca-moro]

Is it possible to create a view on such a select * from 's3://bucket/' query?

[BohuTANG]

Yes

[gianluca-moro]

I tried the following:

CREATE VIEW region_view AS 
SELECT 
    r_regionkey,
    r_name,
    r_comment
FROM 's3://my_bucket/region/'
(
 access_key_id => '<my_access_key>', 
 secret_access_key => '<my_secret_key>',
 region => 'eu-central-1',
 file_format => 'parquet',
 pattern => '.*parquet$'
);

However, this fails with APIError: ResponseError with 1025: VIEW QUERY: system.read_parquet not exists