Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update ASReviewLAB tutorial #28

Merged
merged 5 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
145 changes: 53 additions & 92 deletions ASReviewLAB.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,12 @@ title: "Introductory exercise to ASReview LAB"
author: "The ASReview Academy Team"
---

## Introduction to the software ASReview LAB
**This exercise was made using ASReview LAB version 1.6. If you have a different version you can observe differences between your setup and the exercise.**

## Introduction to the software ASReview LAB v1.6

The **goal** of this exercise is to get familiar with AI-aided screening
by making use of ASReview LAB v1.x.
by making use of ASReview LAB v1.6.

You will learn how to install and set up the software, upload data,
screen records, and export and interpret the results. The exercise will
Expand All @@ -29,92 +31,50 @@ Before you start, you might want to read a bit more on:
- The paper that was published in [Nature Machine
Intelligence](https://www.nature.com/articles/s42256-020-00287-7){target="_blank"}

## The software

### Step 1: Installing Python and ASReview LAB

First, you need to install [Python](https://www.python.org/downloads/){target="_blank"}.

Once you have Python installed, you can go through the easy 3-step
[guide to installing (or upgrading)
ASReview](https://asreview.nl/download/){target="_blank"} on the ASReview-website.

More detailed installation information, troubleshooting, and
installation on a server or via a Docker are available on
[ReadTheDocs](https://asreview.readthedocs.io/en/latest/installation.html#install-asreview){target="_blank"}.

*Have you installed the latest version of ASReview? You can proceed to
step 2!*

### Step 2: Starting ASReview LAB

To open ASReview LAB in your browser, you need to start it in the
command prompt (i.e. CMD.exe or Terminal). You can open your command
prompt by typing ‘cmd’ (for Windows) or ‘terminal’ (for Apple) in your
computer’s search bar (select ‘Run as administrator’ if you have this
option).

![](images/ASReviewLAB/CMD1.png){width=95%}

The command prompt will open, in which you can type the following
command and press enter:
## Prerequisites

``` bash
asreview lab
```
### Installing ASReview LAB

![](images/ASReviewLAB/CMD2.png){width=95%}
Before you start the tutorial ASReview LAB needs to be installed; see for instructions the [ASReview website](https://asreview.nl/download/){target="_blank"}.

It takes a few seconds for ELAS - your Electronic Learning Assistant -
to start the software. It will appear in your (default) web browser.
*Have you installed ASReview LAB? You can proceed to the exercise!*

But why do you need to start it up by running code in your command
prompt? This ensures that ASReview LAB is run locally. More
specifically, <u>your data is and stays your own</u>. Small price to pay
for complete privacy, right?! Read more about the key principles in the
[Zen of Elas](https://asreview.nl/blog/the-zen-of-elas/){target="_blank"}!
## Exercise

Note that you have to keep your command-line interpreter running while
using ASReview LAB, even though the interface is in your browser!

You can also run the software via a server, but you need to take care of
[hosting the server
yourself](https://asreview.readthedocs.io/en/latest/installation.html#server-installation){target="_blank"}
(or ask your IT-department).
### Step 1: Starting ASReview LAB

Open ASReview LAB in your browser. Note that if you do this via Command Prompt (Windows) or Terminal (MacOS) you have to keep your command-line interpreter running while using ASReview LAB, even though the interface is in your browser!

*Have you opened ASReview LAB in your browser? If so, you can proceed to
step 3!*
step 2!*

### Step 3: Creating a project
### Step 2: Creating a project

Now that you have installed and opened ASReview LAB, you can create a
new project. Below you will find a step-by-step guide. Note that the
screenshots shown below are made in [dark
mode](https://asreview.readthedocs.io/en/latest/screening.html#display){target="_blank"}.
Now that you have opened ASReview LAB, you can create a
new project. Below you will find a step-by-step guide.

1. *New project;*

Hover your mouse over the ‘create’ button with the plus sign in the
Hover your mouse over the ‘`create`’ button with the plus sign in the
bottom right corner.

![](images/ASReviewLAB/Interface2.png){width=95%}
![](images/ASReviewLAB/step_2a_V1_6.png){width=85% fig-align="center"}

2. *Project name;*

Select Validation Mode, fill out a project name and press ‘NEXT’. Note
Select Validation Mode, fill out a project name and press ‘`NEXT`’. Note
that you can fill out your name and a description as well.

![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/setup_project_modes.png){width=95%}
![](images/ASReviewLAB/step_2b_V1_6.png){width=85% fig-align="center"}

For this exercise we are screening in the so-called ‘Validation Mode’
For this exercise we are screening in the so-called ‘`Validation Mode`
of ASReview. By screening in the [Validation
Mode](https://asreview.readthedocs.io/en/latest/project_create.html#project-modes){target="_blank"},
we are going to make use of a [benchmark
dataset](https://asreview.readthedocs.io/en/latest/data_labeled.html#fully-labeled-data){target="_blank"}.
This means that all records in the dataset have already been labeled as
relevant or irrelevant. This is indicated to the user through a banner
above each article. Note that in ‘Oracle Mode’ - when screening your own
above each article. Note that in ‘`Oracle Mode`’ - when screening your own
dataset - the relevant papers will not be marked; you, the oracle, have
to make the decisions.

Expand All @@ -123,33 +83,33 @@ More detailed information about setting up a project can be found on


*Have you started creating a new project? If so, you can proceed to step
4!*
3!*

## Project setup

### Step 4: The dataset
### Step 3: The dataset

Now that you have created your ASReview project (woohoo!), you need to set it
up. Without data, we have nothing to screen. So, you need to tell ELAS
which dataset you want to screen for relevant articles.

Click on the ‘ADD’ button next to ‘Add dataset’. Now a menu appears
Click on the ‘`ADD`’ button next to ‘`Add dataset`’. Now a menu appears
where you can choose how to load the dataset. You can add your dataset
by selecting a file or providing an URL. For this exercise, we will use
a benchmark dataset.

Go to the ‘Benchmark datasets’ button, open the first dataset (i.e. the
Go to the ‘`Benchmark datasets`’ button, open the first dataset (i.e. the
Van de Schoot (2017) dataset about PTSD trajectories) and click on
SELECT’. After you select the dataset, click on ‘SAVE’.
`ADD`’. After you select the dataset, click on ‘`SAVE`’.

![](images/ASReviewLAB/Interface5.png){width=95%}
![](images/ASReviewLAB/step_3_V1_6.png){width=85% fig-align="center"}



*Have you successfully selected/uploaded the dataset? If so, you can
proceed to step 5!*
proceed to step 4!*

### Step 5: Prior knowledge
### Step 4: Prior knowledge

Before you can start screening the records, you need to tell ELAS what
kind of records you <u>are</u> and what kind of records you <u>are
Expand All @@ -165,16 +125,15 @@ irrelevant record). However, because you are using the Validation Mode
of ASReview, the relevant records are known; the original authors have
already read ALL records.

To select the prior knowledge you first need to click on the ‘ADD’
button next to ‘Add prior knowledge’; see also the documentation
To select the prior knowledge you first need to click on the ‘`ADD`
button next to ‘`Add prior knowledge`’; see also the documentation
about the selection of [prior knowledge](https://asreview.readthedocs.io/en/latest/project_create.html#select-prior-knowledge){target="_blank"}.
Now you will see a menu about selecting prior knowledge.

The following five papers are known to be relevant:

- Latent trajectories of trauma symptoms and resilience: the 3-year
longitudinal prospective USPER study of Danish veterans deployed in
Afghanistan (DOI: [10.4088/JCP.13m08914](https://doi.org/10.4088/jcp.13m08914){target="_blank"})
- Latent Trajectories of Trauma Symptoms and Resilience
(DOI: [10.4088/JCP.13m08914](https://doi.org/10.4088/jcp.13m08914){target="_blank"})
- A Latent Growth Mixture Modeling Approach to PTSD Symptoms in Rape
Victims (DOI: [10.1177/1534765610395627](https://doi.org/10.1177/1534765610395627){target="_blank"})
- Peace and War: Trajectories of Posttraumatic Stress Disorder
Expand All @@ -185,22 +144,22 @@ The following five papers are known to be relevant:
- Trajectories of trauma symptoms and resilience in deployed US
military service members: Prospective cohort study (DOI: [10.1192/bjp.bp.111.096552](https://doi.org/10.1192/bjp.bp.111.096552){target="_blank"})

To add the relevant records, you click on ‘Search’, copy and paste the titles
To add the relevant records, you click on ‘`Search`’, copy and paste the titles
of these relevant records one by one in the search bar and add them as
relevant.

![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/setup_prior_knowledge_random_validate.png){width=95%}
![](images/ASReviewLAB/step_4_V1_6.png){width=85% fig-align="center"}


After adding all five relevant records, you can add some irrelevant ones
by clicking the ‘Random’ button (use the arrow in the upper left corner
to be able to select this button) and by changing ‘relevant’ to
‘irrelevant’. Select five irrelevant records and click on ‘CLOSE’.
by clicking the ‘`Random`’ button (use the arrow in the upper left corner
to be able to select this button) and by changing ‘`relevant`’ to
`irrelevant`’. Select five irrelevant records and click on ‘`CLOSE`’.

*Have you selected five relevant and five irrelevant records? If so, you
can proceed to step 6!*
can proceed to step 5!*

### Step 6: Active learning model
### Step 5: Active learning model

The last step to complete the setup is to select the active learning
model you want to use. The default settings (i.e. Naïve Bayes, Max and
Expand All @@ -210,8 +169,8 @@ read which options are
or add your own model via a
[template](https://github.com/asreview/template-extension-new-model){target="_blank"}.

You can click on ‘NEXT’. A menu with the defaults will appear. Since we
are using the defaults, you can click on ‘NEXT’ again. In the last step
You can click on ‘`NEXT`’. A menu with the defaults will appear. Since we
are using the defaults, you can click on ‘`NEXT`’ again. In the last step
of the setup, ASReview LAB runs the feature extractor, trains a model,
and ranks the records in your dataset. Depending on the model and the
size of your dataset, this can take a couple of minutes (meanwhile, you
Expand All @@ -220,13 +179,15 @@ Learning?](https://asreview.nl/blog/active-learning-explained/){target="_blank"}

After the project is successfully initialized, you can start reviewing.

![](images/ASReviewLAB/Interface9.png){width=95%}
![](images/ASReviewLAB/step_5a_V1_6.png){width=85% fig-align="center"}

*Have you finished the setup? If so, you can proceed to step 6!*

*Have you finished the setup? If so, you can proceed to step 7!*
![](images/ASReviewLAB/step_5b_V1_6.png){width=85% fig-align="center"}

## Screening phase

### Step 7: Screening the records
### Step 6: Screening the records

Everything is set up and ready to screen, well done!

Expand Down Expand Up @@ -254,7 +215,7 @@ statistics and interpret the corresponding charts, check out the
[documentation](https://asreview.readthedocs.io/en/latest/progress.html#analytics){target="_blank"}
on the Analytics page.

The Van de Schoot (2017) dataset contained 43 relevant records in this
The Van de Schoot (2017) dataset contained 38 relevant records in this
particular example. Did you get to label all of them as relevant before
you reached your Stopping Rule? If you did, great!

Expand All @@ -278,14 +239,14 @@ Rule. Read more about Stopping Rules and how to decide on a good
strategy for your data on the [discussion
platform](https://github.com/asreview/asreview/discussions){target="_blank"}.

### Step 8: Extracting and inspecting the data
### Step 7: Extracting and inspecting the data

Now that you found all or most relevant records, you can export your
data using [these
instructions](https://asreview.readthedocs.io/en/latest/progress.html#export-results){target="_blank"}.
If you choose to inspect your data in Excel, download the data in
‘Excel’ format. If you prefer to inspect your data in R, download the
‘CSV (UTF-8)’ format and open it in R.
`Excel`’ format. If you prefer to inspect your data in R, download the
`CSV (UTF-8)`’ format and open it in R.

You can find all the data that was originally imported to ASReview in
the exported data file, in a new order and with two new columns added at
Expand All @@ -302,7 +263,7 @@ the original ordering, do the included articles come from?

For the last exercise, it is important to change the order back to the order
provided by ASReview. Lastly, check out the first few records with no number
in the `included` column. Are those articles labeled as ‘relevant’ in the
in the `included` column. Are those articles labeled as ‘`relevant`’ in the
original dataset? (Whether or not a record was pre-labeled as relevant is
shown in the column `label_included` in the original dataset.)

Expand Down Expand Up @@ -342,8 +303,8 @@ Some suggestions:
code](https://github.com/asreview/asreview/tree/master/asreview){target="_blank"} on
Github… +1 for open-science!)

![](images/ASReviewLAB/game_V1_6.png){width=85% fig-align="center"}

![](https://raw.githubusercontent.com/asreview/asreview/main/docs/images/game.png){width=95%}

[^1]: Wang Z, Nayfeh T, Tetzlaff J, O’Blenis P, Murad MH (2020) Error
rates of human reviewers during abstract screening in systematic
Expand Down
2 changes: 1 addition & 1 deletion _quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ website:
- section: "Introduction Courses"
contents:
- href: ASReviewLAB.qmd
text: ASReview LAB
text: ASReview LAB v1.6
- href: datatools.qmd
text: Datatools
- href: simulation.qmd
Expand Down
Loading