Skip to content

Commit

Permalink
Docs for bulk annotation (#2506)
Browse files Browse the repository at this point in the history
# Description

Please include a summary of the changes and the related issue. Please
also include relevant motivation and context. List any dependencies that
are required for this change.

Closes #<issue_number>

**Type of change**

(Please delete options that are not relevant. Remember to title the PR
according to the type of change)

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Refactor (change restructuring the codebase without changing
functionality)
- [ ] Improvement (change adding some improvement to an existing
functionality)
- [x] Documentation update

**How Has This Been Tested**

(Please describe the tests that you ran to verify your changes. And
ideally, reference `tests`)

- [ ] Test A
- [ ] Test B

**Checklist**

- [x] I have merged the original branch into my forked branch
- [ ] I added relevant documentation
- [ ] follows the style guidelines of this project
- [ ] I did a self-review of my code
- [x] I made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
  • Loading branch information
nataliaElv authored Mar 8, 2023
1 parent 29c9ee3 commit a97b7d0
Show file tree
Hide file tree
Showing 13 changed files with 67 additions and 53 deletions.
Binary file modified docs/_source/_static/reference/webapp/features-annotate.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file removed docs/_source/_static/reference/webapp/homepage.png
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
77 changes: 47 additions & 30 deletions docs/_source/reference/webapp/features.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
The Argilla UI has a dedicated mode to quickly label your data in a very intuitive way, or revise previous gold labels and correct them.
Argilla's powerful search and filter functionalities, together with potential model predictions, can guide the annotation process and support the annotator.

You can access the _Annotate mode_ via the sidebar of the [Dataset page](dataset.md).
The _Annotate mode_ is the default mode of the [Dataset page](dataset.md).

### Create labels

Expand All @@ -18,51 +18,69 @@ On the right side of the bulk validation bar, you will find a _"+ Create new lab

### Annotate

To annotate the records, the Argilla UI provides a simple and intuitive interface that tries to follow the same interaction pattern as in the [Explore mode](explore_records.md).
As in the _Explore mode_, the record cards in the _Annotate mode_ are also customized depending on the [task](../../guides/task_examples.ipynb) of the dataset.
The Argilla UI provides a simple and intuitive interface for annotating records. Each task offers a variety of small features and interactions to optimize annotations.

#### Text Classification

##### Single label
![Single label text classification record](../../_static/reference/webapp/features-single_textclass_record.png)
You can annotate the records with one click on the label. The record will be validated automatically.

When switching in the _Annotate mode_ for a text classification dataset, the labels in the record cards become clickable and you can annotate the records by simply clicking on them.
For multi-label classification tasks, you can also annotate a record with no labels by either validating an empty selection or deselecting all labels.
##### Multi-label
![Multi-label text classification record](../../_static/reference/webapp/features-multi_textclass_record.png)
To annotate a record, click on one or multiple labels. Once a record is annotated, its status will show as *Pending*. Click *Reset* if you want to discard your changes and recover the previous state of the record or *Validate* to save your annotation into the dataset. The status will then change to *Validated* in the upper left corner of the record card.

Once a record is annotated, it will be marked as _Validated_ in the upper right corner of the record card.
You can also remove all annotations using the *Clear* button or discard a record from the dataset by clicking *Discard*.

```{note}
When the dataset contains a large list of labels (10 +) we encourage viewing the record one by one (from the footer) and using the arrow keys to move quickly to the next record.
When the dataset contains a large list of labels (10 +), we encourage viewing the record cards one by one (select 1 record per page from the footer) and using the arrow keys to move quickly between records.
```

#### Token Classification

![Token classification record](../../_static/reference/webapp/features-tokclass_record.png)

For token classification datasets, you can highlight words (tokens) in the text and annotate them with a label.
Under the hood, the highlighting takes advantage of the `tokens` information in the [Token Classification data model](../python/python_client.rst#Argilla.client.models.TokenClassificationRecord).
You can also remove annotations by hovering over the highlights and pressing the _X_ button.
To remove specific labels, hover over the highlights and press the _X_ button or double-click directly on the colored area. You can also click the _Clear_ button to remove all annotations in the record.

After modifying a record, either by adding or removing annotations, its status will change to _Pending_. When you have finished your annotation of the whole record, click *Validate* to save the changes. The status will change to *Validated* in the upper left corner of the record card. Alternatively, you can use the *Reset* button to discard your changes and recover the previous state of the record.

After modifying a record, either by adding or removing annotations, its status will change to _Pending_ and a _Save_ button will appear.
Once a record is saved, its status will change to _Validated_.
If you want to discard the record from the dataset, click *Discard*.

#### Text2Text
![Text2Text record](../../_static/reference/webapp/features-text2text_record.png)
For Text2Text datasets, there is a text box available for drafting or editing annotations. If a record has one or more predictions, the prediction with the highest score will be pre-filled. You can empty the text field by clicking the *Clear* button.

Remember to validate to save your changes after editing or drafting your annotation. Alternatively, you can discard your changes and revert to the previous state of the record using the *Reset* button.

After validating your annotation, predictions can still be accessed in the prediction panel on the right side of the record card.

For Text2Text datasets, you have a text box available, in which you can draft or edit an annotation.
After editing or drafting your annotation, don't forget to save your changes.
If you want to discard the record from the dataset, click *Discard*.

### Bulk annotate


For all [tasks](../../guides/task_examples.ipynb), you can **bulk validate** the predictions of the records.
You can either select the records one by one with the selection box on the upper left of each card, or you can use the global selection box below the search bar, which will select all records shown on the page.
Then you can either _Validate_ or _Discard_ the selected records.
For all [tasks](../../guides/task_examples.ipynb), you can use **bulk actions**. You can either select the records one by one with the selection box on the upper left of each card, or you can use the global selection box below the search bar, which will select all records shown on the page. Then you can _Validate_ and _Discard_ the selected records using the icons next to the global selection box. For the multi-label text classification and token classification tasks you can also _Clear_ and _Reset_ at bulk level.

![Bulk actions bar](../../_static/reference/webapp/features-bulk-actions.png)

For the text classification task, you can additionally **bulk annotate** the selected records by simply clicking on the pencil icon next to the global selection box and selecting the label(s) from the list.

In multi-label text classification, you can remove labels from selected records by clicking the cross next to the label. You can also assign a partially used label to the whole selection by clicking on the tag button. Once you have made your selection, click *Select* to apply the annotations to all selected records. To finalize the validation, click on the "validate" icon.

For the text classification task, you can additionally **bulk annotate** the selected records with a specific label, by simply selecting the label from the _"Annotate as ..."_ list.
![Multi-label bulk annotation](../../_static/reference/webapp/features-multiclass-bulk-labels.png)

In single-label text classification, there is no need to validate after selecting your label.

### Validate predictions

In Argilla you can pre-annotate your data by including model predictions in your records.
Assuming that the model works reasonably well on your dataset, you can filter for records with high prediction scores, and simply _validate_ their predictions to quickly annotate records.

```{note}
If you make any changes to predictions, validated annotations or annotate from scratch, you will need to validate the records to save the changes.
```

#### Text Classification

For this task, model predictions are shown as percentages in the label tags.
Expand All @@ -71,17 +89,20 @@ You can validate the predictions shown in a slightly darker tone by pressing the
- for a **single label** classification task, this will be the prediction with the highest percentage
- for a **multi label** classification task, this will be the predictions with a percentage above 50%

If you select a different label in the single label task, the status will change to _Validated_ automatically. In the multi-label task, if you make any changes or clear the predictions/annotations, the status will change to _Pending_. You can then click _Validate_ to save the changes or _Reset_ to revert them and restore the previous status.

#### Token Classification

For this task, predictions are shown as underlines.
You can also validate the predictions (or the absence of them) by pressing the _Validate_ button.

Once the record is saved or validated, its status will change to _Validated_.
If you make any changes or clear the predictions/annotations, the status will change to _Pending_. You can then click _Validate_ to save the changes or _Reset_ to revert them and restore the previous status.

#### Text2Text

You can validate or edit a prediction, by first clicking on the _view predictions_ button, and then the _Edit_ or _Validate_ button.
After editing or drafting your annotation, don't forget to save your changes.
The prediction and its score will appear in a text box below the main text. You can validate the prediction by clicking on the *Validate* button or edit the text making the changes directly in the text box.

If you make any changes, the status will change to *Pending* and you will need to press shift+Enter or click the *Validate* button to save the changes. You can also click the *Clear* button to start writing from scratch or *Reset* to undo your changes and restore the previous status.

### Search and filter

Expand Down Expand Up @@ -200,29 +221,25 @@ Not all filters are available for all [tasks](../../guides/task_examples.ipynb).
### Predictions and annotations

Predictions and annotations are an integral part of Argilla's [data models](../python/python_client.rst#module-Argilla.client.models).
The way they are presented in the Argilla UI depends on the [task](../../guides/task_examples.ipynb) of the dataset.
The way they are presented in the Argilla UI depends on the [task](../../guides/task_examples.ipynb) of the dataset and the mode activated in the right column.

#### Text classification

In this task, the predictions are given as tags below the input text. They contain the label as well as a percentage score.

In this task the predictions are given as tags below the input text.
They contain the label as well as a percentage score.
Annotations are shown as tags on the right together with a symbol indicating if the predictions match the annotations or not.
In the annotation view, the predicted labels come in a slightly darker tone, while annotations have a much darker shade and white lettering. In the exploration view, the predicted labels appear below the input text and annotations are shown as tags on the right side of the card.

#### Token classification


In this task, predictions and annotations are displayed as highlights in the input text. To easily identify them at a glance, **annotations** are highlighted with the color of their corresponding label, while **predictions** are underlined with a solid line (see picture).
In this task, predictions and annotations are displayed as highlights in the input text. To easily identify them at a glance, **annotations** are <span style="background-color:rgba(175,200,240,0.5)">highlighted</span> with the color of their corresponding label, while **predictions** are <span style="color: rgba(175,200,240);text-decoration:underline">underlined</span> with a solid line.

For datasets with available score, the solid line for **predictions** disappears when the **score filter** (in **Predictions filter** section) is applied.

#### Text2Text

In this task, the prediction with the highest score appears in a text field below the input text. You can switch between predictions by clicking on their associated score, which appears on the upper right corner.

In this task predictions and the annotation are given in a text field below the input text.
You can switch between prediction and annotation via the "_View annotation_"/"_View predictions_" buttons.
For the predictions you can find an associated score in the lower left corner.
If you have multiple predictions you can toggle between them using the arrows on the button of the record card.
Once you start making changes to the text or after you have validated the annotation, the predictions will still be visible from the *Prediction* banner at the right hand side of the text field.

### Metrics

Expand Down
10 changes: 7 additions & 3 deletions docs/_source/reference/webapp/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@
Argilla UI
==========

The Argilla UI reference:
Argilla's UI is designed to offer flexibility in the way you explore, annotate, curate, and analyze your data, depending on what step of your NLP pipeline you're working on. The platform is designed with a focus on multi-profile users, making it easy and fast to annotate, but also providing powerful filters that allow you to enter deeper into the data analysis.

* :ref:`Pages`: Overview of UI pages
* :ref:`Features`: Overview of UI features
Whether you're working on basic exploration or in-depth analysis, Argilla's UI allows you to move seamlessly between different tasks and offers a powerful way to work with your data and analyze it at any level of detail.

In the following sections, you will find a reference guide to:

* :ref:`Pages`: Overview of Argilla’s pages and their elements
* :ref:`Features`: Overview of what the platform can do and how to use it

.. toctree::
:maxdepth: 2
Expand Down
33 changes: 13 additions & 20 deletions docs/_source/reference/webapp/pages.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@

## Home page

![Home page](../../_static/reference/webapp/homepage.png)
![Home page](../../_static/reference/webapp/pages-homepage.png)

The _Home page_ is mainly a **filterable, searchable and sortable list** of **datasets**.
It is the **entry point** to the Argilla web app and is composed of the following three components.
Expand Down Expand Up @@ -33,7 +33,7 @@ You can find a user icon and a refresh button on the top right:


## Dataset

![Dataset page](../../_static/reference/webapp/pages-dataset_page.png)

The _Dataset_ page is the main page of the Argilla web app.
From here you can access most of Argilla's features, like **exploring and annotating** the records of your dataset.
Expand Down Expand Up @@ -61,26 +61,26 @@ On the right side of the filters, the info button can be used to open a small ov

The record cards are at the heart of the _Dataset_ page and contain your data.
There are three different flavors of record cards depending on the [task](../../guides/task_examples.ipynb) of your dataset.
All of them share the same basic structure showing the input text and a vertical ellipsis (or "kebab menu") on the top right that lets you access the record's metadata.
All of them share the same basic structure showing the input text and a horizontal ellipsis (or "kebab menu") on the top right that lets you access the record's metadata.
Predictions and annotations are shown depending on the current [mode](#modes) and [task](../../guides/task_examples.ipynb) of the dataset.

Check out our [exploration](explore_records.md) and [annotation](annotate_records.md) guides to see how the record cards work in the different [modes](#modes).
```{hint}
Use the menu on the bottom left corner of the page to select how many record cards you want to see per page: 1, 20, 50 or 100. This allows you to annotate record by record or in bigger bulks.
```

#### Text classification
Check out our [exploration](../webapp/features.md#explore-records) and [annotation](../webapp/features.md#annotate-records) guides to see how the record cards work in the different [modes](#modes).

![Text classification view](../../_static/reference/webapp/explore-text-classification.png)
#### Text classification

In this task the predictions are given as tags below the input text.
They contain the label as well as a percentage score.
When in [Explore mode](#modes) annotations are shown as tags on the right together with a symbol indicating if the predictions match the annotations or not.
When in [Annotate mode](#modes) predictions and annotations share the same labels (annotation labels are darker).

A text classification dataset can support either single-label or multi-label classification - in other words, records are either annotated with one single label or various.
A text classification dataset can support either single-label or multi-label classification - in other words, records are either annotated with one or multiple labels.

#### Token classification

![Token classification view](../../_static/reference/webapp/explore-token-classification.png)

In this task annotations are given as colored highlights in the input text, while predictions are indicated by underlines.
At the top of the record list you will find a legend that connects the colors to the respective labels.
When in [Annotate mode](#modes) you can remove annotations or add new ones by simply selecting the desired text.
Expand All @@ -91,12 +91,8 @@ When using the [score filter](./features.md), the predictions that do **not** fa

#### Text2Text

![Text2Text view](../../_static/reference/webapp/explore-text2text.png)
In this task, predictions and annotations are displayed at the same level to allow for easy comparison of versions. The user interface supports the display of multiple predictions for one record, ordered by score. In [Annotate mode](#modes), you can edit a prediction directly in the text box to produce your annotation.

In this task predictions and the annotation are given in a text field below the input text.
You can switch between prediction and annotation via the "_View annotation_"/"_View predictions_" buttons.
For the predictions you can find an associated score in the lower left corner.
If you have multiple predictions you can toggle between them using the arrows on the button of the record card.

### Sidebar

Expand All @@ -107,13 +103,10 @@ The right sidebar is divided into three sections.

This section of the sidebar lets you switch between the different Argilla modes that are covered extensively in their respective guides:

- **Explore**: this mode is for [exploring your dataset](explore_records.md) and gain valuable insights
- **Annotate**: this mode lets you conveniently [annotate your data](annotate_records.md)
- **Define rules**: this mode helps you to [define rules](define_rules.md) to automatically label your data
- **Hand labelling**: this mode lets you conveniently [annotate your data](./features.md#annotate-records)
- **Weak labelling**: this mode helps you to [define rules](./features.md#weak-labelling) to automatically label your data (Text Classification only)
- **Explore**: this mode is for [exploring your dataset](./features.md#explore-records) and gain valuable insights

```{note}
Not all modes are available for all [tasks](../../guides/task_examples.ipynb).
```

#### Metrics

Expand Down

0 comments on commit a97b7d0

Please sign in to comment.