IDIOMS Update & BULKI v0.1 #203

zhangwei217245 · 2024-06-24T03:40:06Z

Related Issues / Pull Requests

Direct
#151 #169 #206

Indirect
#109 #132

Description

Major changes:

IDIOMS -> affix-based query benchmark
IDIOMS -> index persistence mechanism
IDIOMS -> numeric value query support (exact query and range query)
IDIOMS -> numeric value query benchmark
IDIOMS -> simulation test
BULKI V0.1 -> a data serialization and deserialization mechanism
Metadata JSON validator
Metadata JSON Loader for PDC

Minor changes

Updated Dockerfile for faster docker image build
Updated pdc_generic.h for full data type support capability and better ordering of PDC data type enums

In-progress changes

Some code for multi-condition query.

What changes are proposed in this pull request?

Bug fix (non-breaking change which fixes an issue)
[ x ] New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality not to work as expected; for instance, examples in this repository must be updated too)
[ x ] This change requires a documentation update

Checklist:

My code modifies existing public API, or introduces new public API, and I updated or wrote docstrings
[ x ] I have commented my code
My code requires documentation updates, and I have made corresponding changes to the documentation
[ x ] I have added tests that prove my fix is effective or that my feature works
[ x ] New and existing unit tests pass locally with my changes

…dart

Update to avoid fixing compilation issue on src/tools (due to : HDF5 cannot be found)

Dr. Tang fixed a compilation issue in NERSC CI where HDF5 cannot be detected even if the cray-parallel-hdf5 module is loaded on Perlmultter.

src/tests/CMakeLists.txt

zhangwei217245 · 2024-09-05T21:57:45Z

@zhangwei217245, some parallel tests on Permlutter CI are failing, see the logs
  Start 139: idioms_ci_mpi
     24/24 Test #139: idioms_ci_mpi ....................***Failed    7.79 sec
     96% tests passed, 1 tests failed out of 24

There should not be parallel test case for now. I just removed it from CMake.

zhangwei217245 · 2024-09-05T21:58:06Z

Please remove the src/api/test_bulki.bin file unless there is a need for it to be there.

Removed. Thank you for reminding.

* Performace improvement for vpicio (#211) * Paritial fix for the region transfer/wait performance issue * Committing clang-format changes * Improve the async processing for vpicio_mts_all, also fix a few compile issues * Committing clang-format changes * Minor change * Continue to optimize start_all performance for vpicio, add a few time related convinient functions * Committing clang-format changes * Fix hanging issue in CI testing * Committing clang-format changes * Disable debug prints * Revert back for non-all ops * Better pthread management * Better pthread management * Fix timeout issue with CI testing and clang-formatting * Committing clang-format changes * Test * Trigger test * Committing clang-format changes * Trigger CI * Committing clang-format changes * Trigger CI * Switch to static partition for vpicio * Replace vpicio_mts with new implementation --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Multi-thread fix and request merging (#205) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix an issue with region transfer request * Committing clang-format changes * Merge small requests when they are contiguous and 1D, change srun commands in run scripts to detect Perlmutter compute nodes * Merge only for REGION_LOCAL partition * Committing clang-format changes * Fix a bug that causes some tests to fail * Fix a couple of issues with start/wait all * Committing clang-format changes * Add aggregation support for contiguous read operations * Committing clang-format changes * Fix compile issue when multithread is enabled * Committing clang-format changes * minor change with test code * Committing clang-format changes * Remove metadata mutex for multi threading * Committing clang-format changes * Fix mutex * Committing clang-format changes * Fix an issue when closing an obj * Sync develop to stable (v.0.5) (#201) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * enable cache by default (#187) * Removing PDC macro (#189) * Removing gres option for ctest * Removing gres option from scripts * Update check for core * Remove PDC macro * Committing clang-format changes --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * BDCATS fix (#193) * Fix issues with bdcats_batch * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Update mpi_test.sh (#197) * Update .gitlab-ci.yml (#195) * Updates for latest integration with Jacamar and Gitlab tokens in CI * VPICIO bugfix (#196) * Fix VPICIO bug * Add more checks and error out when no server is selected * Committing clang-format changes * Add VPICIO and BDCATS to MPI test --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * Fix vpicio_mts (#199) --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Committing clang-format changes * Fix rebase issue * Add timers * Committing clang-format changes * Add explict transfer start (all) with MPI communicator * Committing clang-format changes * MPI fix * remove debug msg * Committing clang-format changes * Add function comment for doc * Revert script changes * Committing clang-format changes * Revert script changes * Committing clang-format changes * Revert script setting --------- Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * Update CMakeLists.txt to bump version number (#202) * Update CMakeLists.txt to bump version number * Update clang-format-check.yml * IDIOMS Update & BULKI v0.1 (#203) * fix cmake mercury_util not found issue * update for Julia support * fix hdf5.h not found for src/tools * update container config * add libhdf5-dev for Github Actions * update CMake for HDF5 in tools * update logic for finding HDF5 * update * remove use system hdf5 * delete useless find library * update findHDF5 * Feature/dart (#11) Update to avoid fixing compilation issue on src/tools (due to : HDF5 cannot be found) * Use cc on Perlmutter (#161) Dr. Tang fixed a compilation issue in NERSC CI where HDF5 cannot be detected even if the cray-parallel-hdf5 module is loaded on Perlmultter. * update with fixes on tools and llsm example * add gitignore for llsm * update gitignore * Feature/dart (#12) * fix formatting * update clangformat10 * update base dockerfile * Add clang-format10 to docker container. Also fixed clang-format. Add clang-format10 to docker container. Also fixed clang-format. * Fix pdc ls (#154) * pdc import, export, ls compiled successfully * removed requested files * formatting issues * changed install tools * gets checkpoint files * grabbing checkpoint files from within sub-directories, minor comments * Committing clang-format changes * Committing clang-format changes * Fix a few issues with pdc_ls * Committing clang-format changes --------- Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * update documentation * update document update document * sync branch sync branch * no UUID module is required * update document and make UUID an optional package * update docker repo name * updating docker repo name and make UUID optional * Complete support for Docker and Github Codespace (#157) Include support for Docker and Github Codespace so we can run our dev environment with the support of Docker. * SQLite and RocksDB support for KVtags (#165) SQLite and RocksDB support for KVtags * fix round for tag delete * update test * bulki update * BULKI base type worked * BULKI all tests done * new index code * update * update new test * update csv bench * update * update script * adding python scripts for generating large metadata set for LLSM application * update json schema * better json validator * update importer * update code for non-MPI compatibility * update llsm converter * update LLSM data converter * split files * update .gitignore * update * add timing info * update * update tag size * detect object creation failure * update * update object name with date * update for robustness * update * update JMD_DEBUG option * update output for overall output * update inttypes.h * update * update extractor * update inttypes.h * update converter * update importer information * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * fix issue * fixed search issues * update for infix * update * index persistence still needs improvement * update * enable cache by default (#187) * Removing PDC macro (#189) * Removing gres option for ctest * Removing gres option from scripts * Update check for core * Remove PDC macro * Committing clang-format changes --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * range query done' * range query local test passed * multi-condition in progress * clean up code * add comments * new benchmark * update * update range query test * update cmake: * update * update * update * update * someta range query * someta range query * someta range query * fix value serialization * update * update double free * update * update * update * fixed pointer issue * rb_tree delete fixed, now need to check index persistence * update * update * update * update * update * update * update * update * update * update * update * update * update * BDCATS fix (#193) * Fix issues with bdcats_batch * Committing clang-format changes --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * update * clean up code * update test sh * IDIOMS persistence DONE * update * remove old kvtag benchmarks * update * update * update changes * dart info * update * multi data type for the same key, supported now * Monitoring changes from feature/dart to develop (#18) Major changes: * IDIOMS -> affix-based query benchmark * IDIOMS -> Simulation Test * IDIOMS -> Multi data type supported for the same key * IDIOMS -> Range Query and Exact Query for Numeric Values * IDIOMS -> benchmark for numeric values (exact search and range query) * IDIOMS -> Index Persistence * BULKI -> A data serialization and deserialization mechanism. * fix CMakeLists.txt * update * update format * update BULKI interface order * BULKI API sorted * add idioms ci test * Feature/dart (#20) 1. add documentation about BULKI and IDIOMS query conditions 2. add ci test for IDIOMS 3. optimized BULKI to save space on its metadata fields. * Feature/dart (#22) update version * update * update * update * remove unnecessary .bin file * update * update --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> * Fix region transfer with object static partitioning (#214) * Update pdc_region_transfer.c * Committing clang-format changes * Update .gitlab-ci.yml Fix issue with Perlmutter CI libfabric module --------- Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> * EQSIM benchmark code and fixes (#213) * Update getting_started.rst (#184) * Removing gres option for ctest (#182) * Removing gres option for ctest * Removing gres option from scripts * Update check for core --------- Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> * enable cache by default (#187) * Benchmark code for EQSIM data * Committing clang-format changes * Minor adjustments * Committing clang-format changes * Updates * Committing clang-format changes * Change vpicio to use local server partitioning, add some debug prints * Committing clang-format changes * Add metadata query to benchmark code * Committing clang-format changes * Add ZFP compression for read and write * Committing clang-format changes * Add a option to use more ranks to read data so total data of each rank is less than the 4GB chunk limit * Committing clang-format changes * Add a data query code for EQSIM data * Committing clang-format changes * Minor adjustments for the HDF5 read code * Committing clang-format changes * Fix an issue with periodic data flush, minor changes to benchmark code * Committing clang-format changes * fix an issue with 3d read segfault * Committing clang-format changes * Fix compile issue * Update .gitlab-ci.yml * Update sleep time * Replace function * Replace function * Minor updates and doc changes * Committing clang-format changes * Update --------- Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Jean Luca Bez <jlbez@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> --------- Co-authored-by: Houjun Tang <htang4@lbl.gov> Co-authored-by: github-actions <github-actions[bot]@users.noreply.github.com> Co-authored-by: Hyunju Oh <oh.693@osu.edu> Co-authored-by: Hyunju Oh <hjoh16@login15.chn.perlmutter.nersc.gov> Co-authored-by: Zhang Wei <zhangwei217245@lbl.gov> Co-authored-by: nickaruwang <nickwan0318@gmail.com> Co-authored-by: Nick Wang <66816536+nickaruwang@users.noreply.github.com>

zhangwei217245 and others added 30 commits October 13, 2023 23:40

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/…

48a370f

…dart

fix cmake mercury_util not found issue

ec6c6bc

update for Julia support

f99a537

fix hdf5.h not found for src/tools

0751a12

update container config

c446108

update container config

9911445

update container config

8296539

update container config

d1303da

update container config

88a75e6

update container config

923ec93

update container config

9c3fbc4

update container config

2f2c073

update container config

49ef572

update container config

199c1bb

add libhdf5-dev for Github Actions

7fc75f0

Merge branch 'develop' of github.com:zhangwei217245/pdc into feature/…

05511d7

…dart

update CMake for HDF5 in tools

11c8cec

update logic for finding HDF5

71a23d5

update

5e01ca9

remove use system hdf5

d023c61

delete useless find library

06cff1c

update findHDF5

025f977

update findHDF5

fd0e853

update

a44f8a0

update

3bb2529

update

9fe8772

update

71cf8d8

Feature/dart (#11)

ca8e59d

Update to avoid fixing compilation issue on src/tools (due to : HDF5 cannot be found)

Use cc on Perlmutter (hpc-io#161)

1b6d60d

Dr. Tang fixed a compilation issue in NERSC CI where HDF5 cannot be detected even if the cray-parallel-hdf5 module is loaded on Perlmultter.

Merge branch 'hpc-io:develop' into develop

67791ef

zhangwei217245 had a problem deploying to external September 5, 2024 18:30 — with GitHub Actions Failure

houjun requested changes Sep 5, 2024

View reviewed changes

src/tests/CMakeLists.txt Outdated Show resolved Hide resolved

update

4d46ec1

zhangwei217245 had a problem deploying to external September 5, 2024 20:34 — with GitHub Actions Failure

update

df57ec8

zhangwei217245 had a problem deploying to external September 5, 2024 20:38 — with GitHub Actions Failure

update

1f89fbc

zhangwei217245 had a problem deploying to external September 5, 2024 20:45 — with GitHub Actions Failure

This comment was marked as off-topic.

Sign in to view

Merge branch 'develop' into develop

259e0eb

zhangwei217245 temporarily deployed to external September 5, 2024 21:00 — with GitHub Actions Inactive

zhangwei217245 requested review from houjun and jeanbez September 5, 2024 21:12

This comment was marked as resolved.

Sign in to view

remove unnecessary .bin file

171e081

zhangwei217245 had a problem deploying to external September 5, 2024 21:46 — with GitHub Actions Failure

zhangwei217245 requested a review from jeanbez September 5, 2024 21:47

update

4a82328

zhangwei217245 temporarily deployed to external September 5, 2024 21:52 — with GitHub Actions Inactive

update

d0e2c0f

zhangwei217245 temporarily deployed to external September 5, 2024 21:56 — with GitHub Actions Inactive

houjun approved these changes Sep 6, 2024

View reviewed changes

zhangwei217245 changed the title ~~Draft: IDIOMS Update & BULKI v0.1~~ IDIOMS Update & BULKI v0.1 Sep 6, 2024

jeanbez approved these changes Sep 6, 2024

View reviewed changes

sbyna approved these changes Sep 10, 2024

View reviewed changes

jeanbez merged commit 2e5de75 into hpc-io:develop Sep 10, 2024

hpc-io deleted a comment from houjun Sep 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

IDIOMS Update & BULKI v0.1 #203

IDIOMS Update & BULKI v0.1 #203

Uh oh!

zhangwei217245 commented Jun 24, 2024 •

edited

Loading

Uh oh!

Uh oh!

This comment was marked as off-topic.

This comment was marked as resolved.

Uh oh!

zhangwei217245 commented Sep 5, 2024

Uh oh!

zhangwei217245 commented Sep 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

IDIOMS Update & BULKI v0.1 #203

IDIOMS Update & BULKI v0.1 #203

Uh oh!

Conversation

zhangwei217245 commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues / Pull Requests

Description

What changes are proposed in this pull request?

Checklist:

Uh oh!

Uh oh!

This comment was marked as off-topic.

This comment was marked as resolved.

Uh oh!

zhangwei217245 commented Sep 5, 2024

Uh oh!

zhangwei217245 commented Sep 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zhangwei217245 commented Jun 24, 2024 •

edited

Loading