MAG-WoS papers for open science/reproducility topics

[everyxs]

We intend to collect a set of paper covering the general topics of open science/reproducibility. We want to first replicate our earlier studies based on our own crawled MAG data at 02-22-2108. We also want to conduct a similar analysis of the latest MAG data. In the end, we would like to map the papers in MAG to those in WoS, especially for journal names and disciplinary mapping.

[XiaoranYan]

The pipeline is designed to test many working components of CADRE-RAC from U-sql data ingestion to spark data comparison and integration, nested dockerized gephi visualizations, under a containerized Jupyter Lab environment. It will also use both WoS and MAG datasets.

The repo in development can be found at
https://github.com/everyxs/openScience

[XiaoranYan]

Finished:

• Azure Blob connection established for U-sql result ingestion
• Data cleaning and transformation pipeline optimized using R
• Run Gephi scripts as a JAR package from Jupyter Lab
• Submit U-sql scripts from Jupyter Lab
• Established remote jupyter kernel for Karst Spark cluster using Toree, only possible outside of container, with ssh tunnels and browsers
• Using binder and repo2docker to build reproducible container images
• Prepare a demo based this project, with wiki instructions and binder image deployments

---

To do list:

• Deploy BinderHub on AWS Kubernetes cluster
• Test u-sql on AWS cluster while adding new overlapping output tables
• Explore ways to dockerize customizable Toree image for local cluster integration on CADRE
• Explore ways to pre-mount hdfs, S3, WASB and other Hadoop file system interface in customize docker image
• Explore possible ways to dockerize Gephi for interactive visualizations