Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closes #305 | Create Multilingual open relation dataloader #320

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Closes #305 | Create Multilingual open relation dataloader #320

wants to merge 2 commits into from

Conversation

madenindya
Copy link
Contributor

@madenindya madenindya commented Oct 13, 2022 •
edited
Loading

Closes #305

Checkbox

  • Confirm that this PR is linked to the dataset issue.
  • Create the dataloader script nusantara/nusa_datasets/my_dataset/my_dataset.py (please use only lowercase and underscore for dataset naming).
  • Provide values for the _CITATION, _DATASETNAME, _DESCRIPTION, _HOMEPAGE, _LICENSE, _URLs, _SUPPORTED_TASKS, _SOURCE_VERSION, and _NUSANTARA_VERSION variables.
  • Implement _info(), _split_generators() and _generate_examples() in dataloader script.
  • Make sure that the BUILDER_CONFIGS class attribute is a list with at least one NusantaraConfig for the source schema and one for a nusantara schema.
  • Confirm dataloader script works with datasets.load_dataset function.
  • Confirm that your dataloader script passes the test suite run with python -m tests.test_nusantara --path=nusantara/nusa_datasets/my_dataset/my_dataset.py.
  • If my dataset is local, I have provided an output of the unit-tests in the PR (please copy paste). This is OPTIONAL for public datasets, as we can test these without access to the data files.

@madenindya
Copy link
Contributor Author

Hi, I found many difficulties with this dataset and would like to have some advice:

  1. The data source is from Kaggle, how do I download it here? The original .zip size is huge. In local, I only download Indonesian data and use it for test reference.
  2. When creating entities offset, I found it hard to get as the original data didn't show that. When I tried to make it on my own, it was also hard to infer as there are some exact words in a sentence.
  3. And because I couldn't get the offset, I don't know whether it's better to have a different EntID for the same word in the same sentence. As it might come from the same/different position word, but I don't have that info.

I mark my line of codes as TODO for the things that need further discussion

Additional Questions:

  • Why the entites.text is List & not just a single string? (context: schema's KB)

@madenindya madenindya changed the title Create Multilingual open relation dataloader Closes #305 | Create Multilingual open relation dataloader Oct 17, 2022
@muhsatrio
Copy link
Collaborator

Hi, I found many difficulties with this dataset and would like to have some advice:

  1. The data source is from Kaggle, how do I download it here? The original .zip size is huge. In local, I only download Indonesian data and use it for test reference.
  2. When creating entities offset, I found it hard to get as the original data didn't show that. When I tried to make it on my own, it was also hard to infer as there are some exact words in a sentence.
  3. And because I couldn't get the offset, I don't know whether it's better to have a different EntID for the same word in the same sentence. As it might come from the same/different position word, but I don't have that info.

I mark my line of codes as TODO for the things that need further discussion

Additional Questions:

  • Why the entites.text is List & not just a single string? (context: schema's KB)

Same like bottleneck that I faced.

cc: @SamuelCahyawijaya @holylovenia @bryanwilie

@muhsatrio
Copy link
Collaborator

I think you can continue discussing it in slack for the faster response kak @madenindya, thank you!

@SamuelCahyawijaya
Copy link
Member

@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away

@muhsatrio
Copy link
Collaborator

@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away

I think still there is no any changes kak

@SamuelCahyawijaya
Copy link
Member

IMO, for this dataset we can just implement the source schema for now, as it will be complicated to extend it to the KB schema. What do you think? @madenindya @muhsatrio

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SamuelCahyawijaya SamuelCahyawijaya Awaiting requested review from SamuelCahyawijaya SamuelCahyawijaya is a code owner

@afaji afaji Awaiting requested review from afaji afaji is a code owner

@holylovenia holylovenia Awaiting requested review from holylovenia holylovenia is a code owner

@gentaiscool gentaiscool Awaiting requested review from gentaiscool gentaiscool is a code owner

@bryanwilie bryanwilie Awaiting requested review from bryanwilie bryanwilie is a code owner

@fajri91 fajri91 Awaiting requested review from fajri91 fajri91 is a code owner

@christianwbsn christianwbsn Awaiting requested review from christianwbsn christianwbsn is a code owner

@muhsatrio muhsatrio Awaiting requested review from muhsatrio muhsatrio is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Create dataset loader for Multilingual Open Relations
3 participants
@madenindya @muhsatrio @SamuelCahyawijaya