v0.9.0

    Release Notes - BDTK - Version 0.9.0

Improvement

• [Doc] One fix for developer doc
• [Code refactor] Remove CiderExecutionKernel from cider module
• [M2] [Integration] VeloxToSubstrait- support variadic function lookup
• Support partial avg check in Velox with old Cider data format
• Support complex data print in CiderBatch

Bug

• DISABLE_FAILED_UT option doesn't take affect
• [VP_Cider_Op_Fil_Proj_Integration] fix values node issue
• [VP_Cider_Op_Fil_Proj_Integration] Fix ciderbatch issue to allow mulit-rows results in filter/project cases
• [VP_Velox_plan_2_Substrait]Change the emit order of projectNode
• [Cider_Substrait_2_EU] Fix EU generator issue in filter case
• [ Cider_runtime_proj_fil_module]Fix CiderRuntimeModule bugs to get correct outBatch in filter case
• [ VP_Cider_data_format_convertor ] Fix bugs caused by changes from Velox, e.g., "cdvi::EMPTY_METADATA"
• Build fails with macro error in eu-mocker
• [VP_Cider_Op_Integration] Remove CiderPlan constructor with VeloxPlan as parameter
• [Bug] return wrong number of columns after processNextBatch
• Don't generate groupby skip mask code for project only case
• velox-plugin can't properly handle the representation of null in BDTK
• Set Column range bug for project case
• Set Column range bug for project case
• Different inner join behavior for JOIN ON clause and where clause
• Bug for query with multiple agg functions.
• [Velox2Substrait] fix convertor problems in project AS
• incorrect col index update
• Groupby result is empty when key is "CASE...WHEN.."
• CiderOperatorBenchmark change to use CiderVeloxPluginCtx transform plan
• Incorrect CiderOperator initialize
• Init folly in CiderOperatorTest
• Fix issues in String support
• Unsupport tinyint and smallint simple qual filter
• Mathematical Op divide zero and overflow
• Mathematical Op decimal constant support
• CiderBatch data lost after a turn around in simple method
• Fix the error about return value of partial avg
• Fix avg with null value
• Fix bug by TPC-H Q21
• fix bug 'Clone velox plannode:Not supported velox plannode type.'
• Not properly handled when fetchResults Return Empty CiderBatch
• Use join substrait plan converter instead of hardcode in cider op
• between...and... scalar function date type support
• enable some TPC-H queries
• verify_function_ir bugfix
• Incorrect non-encoding string conversion
• core dump when string op is used to filter
• fix bug of "not a scalar type! kind: ARRAY"
• fix bug in CiderAggHashTable::fillColsInfo, which group_target_offset_ calculation block not called
• incorrect filter result because of null varchar and len 0 varchar
• Fix bug of PlanPattern write race under multi-threading
• Fix bug of "Remove folly::split of function look up"
• Fix Old CiderBatch Memory Allocation for Group-By Result Fetching
• fix bug of construct col name in substrait to eu
• call loadedVector if DictionaryVector is a lazy vector

Epic

• Separate prototype code into different repos
• Substrait Plan Builder
• The convertor between velox and substrait
• RelAlgExecutionUnit generator based on substrait
• Data convertor
• Join - Velox plugin integration (including operator fusing)
• Modular_SQL test framework
• [Infrastructure] Support Join End 2 End in Cider (from presto_cpp to cider)
• [Infrastructure] Support Agg End 2 End in Cider (from presto_cpp to cider)
• [Infrastructure] Support filter/project End 2 End in Cider (from presto_cpp to cider)
• Function/type system support using Velox alpha release (TPC-H)
• [Functionality] Primary Data Type Support: integer,bigint,decimal,real,double,boolean
• [Functionality] Operation&Function Support: Logical Ops, Compare Ops etc.
• [Functionality] RelAlg Op Support: Filter, Project covering all TPC-H cases
• [Test] Testcase Development for MileStone1 Features
• [Infrastructure] Consolidate jit-engine branch with Cider
• [Customer] Integrate native functions provided by ADB into Codegen framework (e.g., expr eval)
• [Functionality] Primary Data Type Support:String,DataTime
• [Functionality] Operation&Function Support:Compare Op( between…and) etc.
• [Functionality] RelAlg Op Support:Agg with having etc.
• [ Upstream ] Velox-2-Substrait util to Velox repo
• [Infrastructure] Replace BDTK internal format representation with Velox format
• [ Infrastructure ] Implement an expr evaluation module
• [ Documentation ] Implement a API user doc
• Code Refacator
• [M2][Functionality][Expr-AggFunctions] Expr - Aggregate Functions
• [M2][Functionality][JoinOp] Join Ops
• [M2][Functionality][AggOp] Aggregate Ops
• [M2][Functionality][Expr-ScalarFunctions] Expr - Scalar Functions
• [M3] [Code Refactor][Placeholder] Product Improvement

Story

• BDTK code gen module refactor design
• Arrow Data convertor impl of varchar/nested data type
• Port code from presto-java to presto-cpp
• BDTK Codegen deep dive
• Investigate agg in velox and BDTK
• CompileWorkUnit API impl
• Test Framework about Velox-Arrow-BDTK data type converter
• Parse the Presto query json log
• Presto performance projection per query w/ ops break down
• investigate/analyzing mapping info between velox Plan Node and substrait relations
• Support Q6 in Execution Unit generator
• Velox BDTK data transfer micro benchmark
• Integrate cider into velox UT
• Impl the IR converter between substrait and velox in SubstraitIRConverter.cpp
• abstract hash table from BDTK
• Investigate the velox-substrait plan node translation
• Support fallback in execution unit generator
• Data convertor(velox::RowVectorPtr -> BDTK )
• integration - Agg(blocking op) work flow
• CICD system setup
• Data Conversion Utils Implement for Velox and BDTK
• noisepage investigation
• Debug failing unit tests in velox and BDTK
• fix results of query/ data with Null value
• UT could pass but clean up stage have memory related issues.
• create getting started guidance for BDTK+velox
• create dockerfile for BDTK + velox
• rebase velox change to upstream latest
• presto tpch queries plan fragment pattern
• velox driver workflow
• Setup pre-commit check
• enable github actions for BDTK
• Seprate module into deifferent repo
• rebase the substrait code to align with upstream
• pre-commit check migration to github action for velox plugin
• check title and code style
• update substraitRel to meet our requirements
• Refine Cider module API
• check license header
• E2E workflow integration
• Design for substrait plan builder
• Support substrait FilterRel in RelAlgExecutionUnit generator passing "select l_suppkey, l_quantity from lineitem where l_orderkey > 10"
• fix bugs in pr-title-checker and license-header-checker
• Velox Plan Transformer: A Pattern Match - Rewriter Framework
• Code refactor in Substrait to Velox plan util seperate type, function,expr + operator convertors
• update dockerfile
• update velox URL for ci
• BDTK JOIN investigation
• Generate explanation document for the investigation result of Jacques’s code
• Arrow data convertor utils for timestamp support
• Detail design for velox BDTK hash join
• Rebase to substrait latest version and using index based
• Data convertor round trip string support Velox type to BDTK type (non-encoding)
• Group by prototype aggregation support (BDTK ResultSet provider, Velox hybrid operator integration)
• Complete TPC-H Q6 prototype to product (veloxPlugin->BDTK codegen API refactor and pass UTs )
• Test plan + execution plan + design for Modular_SQL (Presto Java + Presto_cpp + Velox_plugin + Velox)
• Generate a CiderOperator prototype with Hash Join with Velox Plan Node contains JoinNode validated by plan node mocking join bridge
• BDTK Join API
• BDTK col_buffer internal layout investigation
• Data convertor code/API refactor with CiderBatch (new API adapter)
• BDTK Join API implements including code refactor to support build hash table with one use case
• Agg support in substrait2EU covering (agg + group by) without count distinct
• update submodule in ci
• Integration test with velox-substrait and substrait-eu: filter + project
• Special semantics support in substrait2EU (between/and, count(*))
• Upstream Velox-To-Substrait code to velox repo (filter+ project)
• Using Yaml File to do the Function Convertor in velox-substrait utils
• CI maintain
• Presto-cpp/velox-plugin/BDTK Integration TPC...