real column pruning

[phil-88]

Hello! 

I'm using your example with hidden query_col_name and pred_XX params to build connectors to other storages (clickhouse,orc,mysql,mongodb,...). I succeed in pruing columns while reading from the source, but I still have to pass them filled with nulls through the Vertica SDK. And in my case, it is very time-consuming too. If I query a single column out of external table with 20 columns, it can be 10 times slower than fetching a single column out of external table with only one column.

I saw there is a getParserReturnType method, but it is only a column precision that can be changed from there. I also saw a RLETuple type and guess it is possible to avoid passing nulls by encoding them into RLE, but SDK does not support RLETuple as a type for UDx developers. So for now, I came up with using RLE-like groupping inside UDx with an extra column for count. Another thought is to use arrays to prune columns.

Can you please guide me on whether it is possible to remove null columns from the UDParser output to get better perfomance?

[h-serizawa]

Does a query like 'SELECT column1 FROM <table with 20 columns>' take 10 times longer than a query like 'SELECT column1 FROM <table with 1 column>' even if the data volume returned by both queries is the same? I would like to know how long both queries take to send all data to ODBC-Loader.

It needs to read all rows to know the column has only NULL value. If it can know it beforehand, it removes the column from the query or it uses FILLER option in COPY statement to ignore the column.