[GraphBolt][Dataset] Contribute IGBH dataset to hetero examples. #7708

BowenYao18 · 2024-08-15T20:36:54Z

Description

I added IGB-dataset to the RGCN folder. Currently the three smaller version (tiny[~2GB], small[~20GB], medium[~100GB]) are supported. The two larger versions (large[~800GB], full[~2.2TB]) are still under test.

Usage:

Run the "download.py --size {choose the size}", the dataset will be downloaded and processed under the default ""dataset/"" folder
Run the ""hetero_rgcn.py"" with the corresponding dataset you just processed

Checklist

Please feel free to remove inapplicable items for your PR.

The PR title starts with [$CATEGORY] (such as [NN], [Model], [Doc], [Feature]])
I've leverage the tools to beautify the python and c++ code.
[] The PR is complete and small, read the Google eng practice (CL equals to PR) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
All changes have test coverage
Code is well-documented
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change
Related issue is referred in this PR
If the PR is for a new model/paper, I've updated the example index here.

Changes

dgl-bot · 2024-08-15T20:37:24Z

To trigger regression tests:

@dgl-bot run [instance-type] [which tests] [compare-with-branch];
For example: @dgl-bot run g4dn.4xlarge all dmlc/master or @dgl-bot run c5.9xlarge kernel,api dmlc/master

dgl-bot · 2024-08-15T20:41:19Z

Commit ID: 021341236243f6f7133a1adf3014b3b0b0499258

Build ID: 1

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-08-15T20:47:36Z

Commit ID: afd0152bb1c5e9ea8b34c1e30e89b416f54821fa

Build ID: 2

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

examples/graphbolt/rgcn/download.py

dgl-bot · 2024-08-15T21:18:44Z

Commit ID: 5fc930b

Build ID: 3

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

examples/graphbolt/rgcn/download.py

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

dgl-bot · 2024-08-15T22:02:30Z

Commit ID: 06f3f29

Build ID: 4

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-08-15T22:02:53Z

Commit ID: 19f18a5

Build ID: 5

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-08-15T22:34:27Z

Commit ID: 97c1735

Build ID: 6

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

dgl-bot · 2024-08-15T22:39:57Z

Commit ID: 93cb70f

Build ID: 7

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

examples/graphbolt/rgcn/download.py

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

dgl-bot · 2024-08-15T22:42:47Z

Commit ID: b170a90

Build ID: 8

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

examples/graphbolt/rgcn/download.py

dgl-bot · 2024-08-15T23:13:44Z

Commit ID: 55079b8

Build ID: 9

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

dgl-bot · 2024-08-16T03:56:23Z

Commit ID: 23f65342bf2552118909addd1ed94e0e16fee8e7

Build ID: 10

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

dgl-bot · 2024-09-09T08:43:34Z

Commit ID: d17ccc264f13fac519ac41a598fc5e22ff4c8f31

Build ID: 37

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-09-09T08:47:06Z

Commit ID: 8211a6e3a591222f356736f84bab4061c85d968d

Build ID: 38

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-09-09T08:49:40Z

Commit ID: 94aadc5bca519209d1de06d940f8ca7b4a324357

Build ID: 39

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-09-09T09:19:45Z

Commit ID: de7dc90

Build ID: 40

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

dgl-bot · 2024-10-07T03:49:37Z

Commit ID: 2537811

Build ID: 41

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

dgl-bot · 2024-10-09T02:30:28Z

Commit ID: 692912e

Build ID: 42

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-10-09T03:03:15Z

Commit ID: 5dfd3fd

Build ID: 43

Status: ✅ CI test succeeded.

Report path: link

Full logs path: link

examples/graphbolt/pyg/hetero/node_classification.py

mfbalin · 2024-10-17T01:13:31Z

examples/graphbolt/rgcn/evaluator.py

We don't need this file. The same functionality can be implemented in a few lines.

mfbalin · 2024-10-17T01:14:01Z

examples/graphbolt/rgcn/hetero_rgcn.py

+            "igb-het-medium",
+            "igb-het-large",
+            "igb-het",
+            "igb-het-MLPerf",


Can we make it all lower case?

python/dgl/graphbolt/impl/ondisk_dataset.py

mfbalin · 2024-10-17T01:14:35Z

python/dgl/graphbolt/impl/ondisk_dataset.py

        The igb-hom-[tiny|small|medium] dataset is a heterogeneous citation network,
        which is designed for developers to train and evaluate GNN models with
-        high fidelity. See more details in `igb-het-[tiny|small|medium]
+        high fidelity. See more details in `igb-het-[tiny|small|medium|large]


also here as well. igb-het-[tiny|small|medium|large|mlperf] and igb-het.

dgl-bot · 2024-10-18T02:29:26Z

Commit ID: 5c689e303debd66465258f96e39df1e4c9a8e014

Build ID: 44

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-10-18T02:36:37Z

Commit ID: 1645605a665e2914c1a66b5fd7378c521665d182

Build ID: 45

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

dgl-bot · 2024-10-18T07:41:35Z

Commit ID: 0030a532f223cc33914caaa916550963d636d121

Build ID: 46

Status: ⚪️ CI test cancelled due to overrun.

Report path: link

Full logs path: link

mfbalin · 2024-10-19T21:40:52Z

LGTM except for the overly complicated evaluator.py file.

mfbalin · 2024-10-28T18:39:08Z

examples/graphbolt/rgcn/evaluator.py

Let's use pure pytorch operations to compute the evaluation score.

Ubuntu and others added 5 commits August 15, 2024 10:14

contribute three IGB dataset (small version)

6441133

Merge branch 'dmlc:master' into add-igbh-to-rgcn

abb9f39

contribute three IGB dataset (small version)

85d92c8

contribute three IGB dataset (small version)

5d2fe56

format the code with ufmt

543f672

added documentation

6bf10cb

Merge branch 'dmlc:master' into add-igbh-to-rgcn

5fc930b

mfbalin reviewed Aug 15, 2024

View reviewed changes

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

mfbalin reviewed Aug 15, 2024

View reviewed changes

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

BowenYao18 and others added 4 commits August 15, 2024 15:01

Update examples/graphbolt/rgcn/download.py

42717af

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

Update examples/graphbolt/rgcn/download.py

06f3f29

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

Update examples/graphbolt/rgcn/download.py

19f18a5

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

Update examples/graphbolt/rgcn/download.py

97c1735

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

Ubuntu added 2 commits August 15, 2024 22:35

added 2983 class task

93cb70f

fix lint

b170a90

mfbalin reviewed Aug 15, 2024

View reviewed changes

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

Update examples/graphbolt/rgcn/download.py

55079b8

Co-authored-by: Muhammed Fatih BALIN <m.f.balin@gmail.com>

mfbalin reviewed Aug 15, 2024

View reviewed changes

examples/graphbolt/rgcn/download.py Outdated Show resolved Hide resolved

remove labels from yaml

ce65746

BowenYao18 added 2 commits September 9, 2024 08:38

add igb-het-large

b043999

fix format

2657ee1

fix lint

d11f815

Merge branch 'dmlc:master' into add-igbh-to-rgcn

de7dc90

Merge branch 'dmlc:master' into add-igbh-to-rgcn

2537811

BowenYao18 added 2 commits October 9, 2024 02:27

fix acc drop bug

692912e

fix acc drop bug

5dfd3fd

mfbalin reviewed Oct 17, 2024

View reviewed changes

examples/graphbolt/pyg/hetero/node_classification.py Show resolved Hide resolved

mfbalin reviewed Oct 17, 2024

View reviewed changes

examples/graphbolt/rgcn/evaluator.py Outdated

Copy link

Collaborator

mfbalin Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this file. The same functionality can be implemented in a few lines.

mfbalin reviewed Oct 17, 2024

View reviewed changes

python/dgl/graphbolt/impl/ondisk_dataset.py Show resolved Hide resolved

mfbalin reviewed Oct 17, 2024

View reviewed changes

BowenYao18 and others added 2 commits October 17, 2024 21:19

Update node_classification.py

65c9b80

add dataset name

4b2a7fb

fix lint

4f3f355

change mlperf to lower

8c21ef4

Ubuntu added 2 commits October 23, 2024 03:38

reduce evaluator

098efbe

reduce evaluator

8e30170

mfbalin requested changes Oct 28, 2024

View reviewed changes

examples/graphbolt/rgcn/evaluator.py

Copy link

Collaborator

mfbalin Oct 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use pure pytorch operations to compute the evaluation score.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GraphBolt][Dataset] Contribute IGBH dataset to hetero examples. #7708

[GraphBolt][Dataset] Contribute IGBH dataset to hetero examples. #7708

BowenYao18 commented Aug 15, 2024 •

edited

Loading

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 16, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Oct 7, 2024

dgl-bot commented Oct 9, 2024

dgl-bot commented Oct 9, 2024

mfbalin Oct 17, 2024

mfbalin Oct 17, 2024

mfbalin Oct 17, 2024 •

edited

Loading

dgl-bot commented Oct 18, 2024

dgl-bot commented Oct 18, 2024

dgl-bot commented Oct 18, 2024

mfbalin commented Oct 19, 2024

mfbalin Oct 28, 2024

[GraphBolt][Dataset] Contribute IGBH dataset to hetero examples. #7708

Are you sure you want to change the base?

[GraphBolt][Dataset] Contribute IGBH dataset to hetero examples. #7708

Conversation

BowenYao18 commented Aug 15, 2024 • edited Loading

Description

Checklist

Changes

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 15, 2024

dgl-bot commented Aug 16, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Sep 9, 2024

dgl-bot commented Oct 7, 2024

dgl-bot commented Oct 9, 2024

dgl-bot commented Oct 9, 2024

mfbalin Oct 17, 2024

Choose a reason for hiding this comment

mfbalin Oct 17, 2024

Choose a reason for hiding this comment

mfbalin Oct 17, 2024 • edited Loading

Choose a reason for hiding this comment

dgl-bot commented Oct 18, 2024

dgl-bot commented Oct 18, 2024

dgl-bot commented Oct 18, 2024

mfbalin commented Oct 19, 2024

mfbalin Oct 28, 2024

Choose a reason for hiding this comment

BowenYao18 commented Aug 15, 2024 •

edited

Loading

mfbalin Oct 17, 2024 •

edited

Loading