This repository contains the code for reproducing the results of our Computational Linguistics article which was presented at EMNLP 2021.
It contains:
- our cross-document event coreference resolution (CDCR) system implementation
- the corresponding corpus preprocessing pipeline for the ECB+, Gun Violence Corpus (GVC) and Football Coreference Corpus (FCC) corpora
- baseline implementations
- CoNLL files and scores from our experiments
- the annotation guidelines used
For obtaining the Football Coreference Corpus (FCC), please visit https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2305
Please cite our work as follows:
@article{bugert2021crossdocument,
author = {Bugert, Michael and Reimers, Nils and Gurevych, Iryna},
title = {{Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora}},
journal = {Computational Linguistics},
volume = {47},
number = {3},
pages = {575-614},
year = {2021},
month = {11},
issn = {0891-2017},
doi = {10.1162/coli_a_00407},
url = {https://doi.org/10.1162/coli_a_00407},
eprint = {https://direct.mit.edu/coli/article-pdf/47/3/575/1971857/coli_a_00407.pdf},
}
Abstract: Cross-document event coreference resolution (CDCR) is an NLP task in which mentions of events need to be identified and clustered throughout a collection of documents. CDCR aims to benefit downstream multidocument applications, but despite recent progress on corpora and system development, downstream improvements from applying CDCR have not been shown yet. We make the observation that every CDCR system to date was developed, trained, and tested only on a single respective corpus. This raises strong concerns on their generalizabilityβa must-have for downstream applications where the magnitude of domains or event mentions is likely to exceed those found in a curated corpus. To investigate this assumption, we define a uniform evaluation setup involving three CDCR corpora: ECB+, the Gun Violence Corpus, and the Football Coreference Corpus (which we reannotate on token level to make our analysis possible). We compare a corpus-independent, feature-based system against a recent neural system developed for ECB+. Although being inferior in absolute numbers, the feature-based system shows more consistent performance across all corpora whereas the neural system is hit-or-miss. Via model introspection, we find that the importance of event actions, event time, and so forth, for resolving coreference in practice varies greatly between the corpora. Additional analysis shows that several systems overfit on the structure of the ECB+ corpus. We conclude with recommendations on how to achieve generally applicable CDCR systems in the futureβthe most important being that evaluation on multiple CDCR corpora is strongly necessary. To facilitate future research, we release our dataset, annotation guidelines, and system implementation to the public.
Contact person: Michael Bugert
This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.
- Contents
- Setup
- Reproducing our results
- Instructions for other experiments
- Troubleshooting
- References
You need:
- a machine with Docker installed
- at least 140GB of free disk space at the location where the Docker container will end up (
/var/lib/docker
by default) - For running hyperparameter optimization:
- A powerful CPU compute server (the more cores and RAM, the better).
- Fast temporary storage which can accomodate several hundred GB of cached features. Its location doesn't have to be
/tmp
(see python tempfile docs). For your options with temporary storage and Docker, see for example this post on stackoverflow.
Not required, but recommended:
- A self-hosted instance of the DBpedia Spotlight entity linker, see https://github.com/dbpedia-spotlight/dbpedia-spotlight/wiki/Installation. We used the English model from 2020-03-11. If you don't want to host it yourself you can use the public endpoint, but be aware that it is rate limited.
- A self-hosted instance of DBpedia. We used the latest release as of April 2020 and imported it into a Virtuoso instance. The alternative to self-hosting is using the public endpoint which is rate limited and (at the time of writing) runs an older release from October 2016.
Please note that using the public endpoints will produce preprocessed corpora different from the ones we used in our experiments, hence scores obtained with the public endpoints will not match those reported in the paper. We provide our preprocessed corpora for reference.
-
Open a terminal and run
docker run --name cdcr-container -it mbugert/cdcr-beyond-corpus-tailored
This opens a shell in a docker container in which all experiments can be performed. To detach it (without stopping the container), press Ctrl+p, Ctrl+q in sequence. Use
docker attach cdcr-container
to re-attach. -
In the container, run
make -f resources/scripts/prepare.makefile
What this does:
- It downloads Pytorch-Biggraph embeddings for Wikidata.
- It downloads and prepares SpanBERT embeddings pretrained for AllenNLP for the use with our model.
- It prepares two virtualenvironments, one with allennlp 0.9.0 and one with allennlp 1.0.0. This is unfortunately necessary because the pretrained SRL system and the pretrained SpanBERT system used during preprocessing require different allennlp versions.
- It downloads and prepares the ECB+ and GVC corpora.
-
To prepare the FCC-T corpus, visit https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2305 and follow the instructions there. Assuming the output of the corpus generation script is
/home/user/datasets
, copy the output into the docker container by runningdocker cp /home/user/datasets/. cdcr-container:/cdcr/resources/data/football
.
We explain how to reproduce the results of our feature-based CDCR model on ECB+, FCC-T and GVC reported in table 5 in the paper. All steps should be performed in the docker container.
- Edit
/cdcr/resources/yaml/device_docker.yaml
(nano
andvim
are available).- Set
max_cores
to the number of logical CPU cores on your machine. - If you have self-hosted instances of DBpedia Spotlight and DBpedia available, you can enter these in the
dbpedia
anddbpedia_spotlight
sections.
- Set
- Before being used in model experiments, each corpus requires preprocessing. To do this for ECB+, run:
This will perform preprocessing twice for each corpus split: once using allennlp 0.9.0 for predicting semantic roles (the results are cached to disk), then a second time with allennlp 1.0.0 for preparing embedding features and for exporting the resulting split into a pickle file. Upon completion,
cd /cdcr resources/scripts/run_data_preprocessing.sh ecbp
/cdcr/working_dir
will contain a folder structure similar to the one shown below (hexadecimal config hashes and timestamps will be different on your end)./cdcr/working_dir/ βββ global βββ preprocess_ECBP_dev βΒ Β βββ 478ea5b0 βΒ Β βββ b6072997 βΒ Β βββ 2020-11-03_16-52-51 |Β Β βββ ... βΒ Β βββ 9_DatasetExporterStage βΒ Β Β Β βββ ecbp_dev_preprocessed_b6072997_2020-11-03_16-52-51.pickle βββ preprocess_ECBP_test βΒ Β βββ bf1a5dab βΒ Β βΒ Β βββ 2020-11-03_17-06-35 |Β Β | βββ ... βΒ Β βΒ Β βββ 9_DatasetExporterStage βΒ Β βΒ Β Β Β βββ ecbp_test_preprocessed_bf1a5dab_2020-11-03_17-06-35.pickle βΒ Β βββ ef8f28d8 βββ preprocess_ECBP_train | βββ 115b6261 | βΒ Β βββ 2020-11-03_16-14-03 | βΒ Β βββ ... | βΒ Β βββ 9_DatasetExporterStage | βΒ Β Β Β βββ ecbp_train_preprocessed_115b6261_2020-11-03_16-14-03.pickle | βββ b4739c2c βββ preprocess_ECBP_traindev Β Β βββ 20b39565 Β Β | βββ 2020-11-05_16-02-19 Β Β | βββ ... Β Β | βββ 10_DatasetExporterStage Β Β | Β Β βββ ecbp_traindev_preprocessed_20b39565_2020-11-05_16-02-19.pickle Β Β βββ 2b97be94
- To train on ECB+:
- First, edit
/cdcr/resources/yaml/train/ecbp/ecbp_clustering_xgboost.yaml
.- This file is pre-filled for training five models (with different random seeds) using XGBoost, with optimal features and hyperparameters for ECB+.
- Fill in the
train_data_path
variable with the path to theecbp_traindev_preprocessed_*.pickle
file created in step 2.
- Start the training process by running
Once complete, the serialized models will be located at
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py train resources/yaml/device_docker.yaml resources/yaml/train/ecbp/ecbp_clustering_xgboost.yaml
/cdcr/working_dir/ecbp_training_clustering_xgboost/[config hash]/[timestamp]/serialized_models
. Take note of this path for the next step.
- First, edit
- To predict and evaluate the trained model(s) on the ECB+ test split:
- Edit
/cdcr/resources/yaml/evaluate/ecbp/ecbp_clustering.yaml
and fill ineval_data_path
with the path to theecbp_test_preprocessed_*.pickle
file created in step 2. - Run (with
/path/to/serialized_models
set to the path from step 3):This triggers prediction and evaluation with the coreference evaluation tool of Moosavi et al. 2019. The following files are generated:python run_feature_model.py evaluate /path/to/serialized_models resources/yaml/device_docker.yaml resources/yaml/evaluate/ecbp/ecbp_clustering.yaml
The content of/cdcr/working_dir/ecbp_evaluate_clustering/ βββ 4907b0d0 βββ 2020-11-05_14-50-01 βββ 0 # evaluation of the model trained with random seed 0 βΒ Β βββ cross_doc # evaluation scenario where all documents are merged into a single meta-document in the gold and system CoNLL files. This is what we report in the paper. βΒ Β βΒ Β βββ eval_cross_doc.txt # output from the scoring script βΒ Β βΒ Β βββ gold.conll # gold annotations - the naming scheme for mentions is `metadoc_{document id}_{mention id}` βΒ Β βΒ Β βββ system.conll # predictions βΒ Β βββ within_doc # evaluation scenario where documents are scored in isolation, i.e. only within-document links are scored and cross-document links are left out entirely. βΒ Β βββ eval_within_doc.txt # output from the scoring script βΒ Β βββ gold.conll # gold annotations - the naming scheme for mentions is `{document id}_{mention id}` βΒ Β βββ system.conll # predictions βββ 1 # evaluation of the model trained with random seed 1, and so on βΒ Β βββ ... βββ ... βββ metrics_unaggregated.csv # all scores from all random seeds collected in one file βββ metrics_aggregated.csv # all scores aggregated over the random seeds: the mean/std/min/max achieved for P, R, F1 of MUC, B3, CEAFe, CoNLL, LEA for the cross_doc (`meta_doc=True`) and within_doc (`meta_doc=False`) scenarios βββ metrics_aggregated_pretty.txt # the above in human-readable form: the format of the first column is `(meta-doc, metric)`
metrics_aggregated_pretty.txt
is logged to stdout upon completion of the script.
- Edit
The steps for reproducing our FCC-T and GVC results differ only slightly from those reported above for ECB+.
- Run
export CORPUS=fcct
orexport CORPUS=gvc
. - Run preprocessing via
cd /cdcr; resources/scripts/run_data_preprocessing.sh ${CORPUS}
- To train:
- Edit
/cdcr/resources/yaml/train/${CORPUS}/${CORPUS}_clustering_xgboost.yaml
and fill intrain_data_path
with the path to${CORPUS}_traindev_preprocessed_*.pickle
created in step 2. - Run
Once complete, the serialized models will be located at
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py train resources/yaml/device_docker.yaml resources/yaml/train/${CORPUS}/${CORPUS}_clustering_xgboost.yaml
/cdcr/working_dir/${CORPUS}_training_clustering_xgboost/[config hash]/[timestamp]/serialized_models
.
- Edit
- To evaluate:
- Edit
/cdcr/resources/yaml/evaluate/${CORPUS}/${CORPUS}_clustering.yaml
and fill ineval_data_path
with the path to the${CORPUS}_test_preprocessed_*.pickle
file created in step 2. - Run (with
/path/to/serialized_models
set to the path produced by step 3):python run_feature_model.py evaluate /path/to/serialized_models resources/yaml/device_docker.yaml resources/yaml/evaluate/${CORPUS}/${CORPUS}_clustering.yaml
- The results appear inside
/cdcr/working_dir/${CORPUS}_evaluate_clustering/
.
- Edit
To perform joint training over multiple corpora (see section 8 in the paper), one needs to create pickle files containing multiple corpora.
This can be achieved by chaining multiple python.handwritten_baseline.pipeline.data.loader
sections in the pipeline
section of a preprocessing YAML config.
Example YAML files for merging the FCC-T and GVC corpora are provided in /cdcr/resources/yaml/data_preprocessing/cross_dataset
: there are six files, two for each split (train
, dev
and train+dev
) to account for allennlp version differences.
Run cd /cdcr; resources/scripts/run_fcct_gvc_cross-dataset_preprocessing.sh
to create the FCC-T+GVC pickle files in /cdcr/working_dir/preprocess_FCC-T_GVC_[split]/
. These files can be used interchangeably in place of other pickle files for training/predicting/optimizing the model.
It can be useful to reduce the size of a corpus for speeding up debugging sessions. There is a python.handwritten_baseline.pipeline.data.processing.reducer
preprocessing pipeline stage which can be used for this purpose. See for example /cdcr/resources/yaml/data_preprocessing/ecbp/ecbp_dev_pt1_allennlp0.9.0.yaml
which contains a commented out configuration for this pipeline stage.
The masking of action/participant/time/location mentions or the document publication date (see experiments in section 7.2 in the paper) is achieved with an extra pipeline stage during corpus preprocessing. See for example /cdcr/resources/yaml/data_preprocessing/gvc/gvc_test_pt2_allennlp1.0.0.yaml
which contains the commented out python.handwritten_baseline.pipeline.data.processing.masking
pipeline stage for this purpose.
Corpus statistics such as the number of mentions, the distribution of cluster sizes, the number of coreference links per type, and more (see table 1 in the paper) can be exported automatically.
- Depending on the corpus to export statistics for, run
export CORPUS=ecbp
,export CORPUS=fcct
,export CORPUS=fcc
, orexport CORPUS=gvc
. - Then, run:
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_pipeline.py run resources/yaml/device_docker.yaml resources/yaml/data_preprocessing/stats/${CORPUS}.yaml
- See
/cdcr/working_dir/${CORPUS}_stats/
for the results.
- Depending on the corpus to perform feature selection for, run
export CORPUS=ecbp
,export CORPUS=fcct
, orexport CORPUS=gvc
. - Edit
resources/yaml/feature_selection/${CORPUS}.yaml
and fill ineval_data_path
with the path to the pickle file of the dev split of the respective corpus. - Then, run:
This will perform recursive feature elimination via 6-fold CV over all mention pairs. This process is repeated 7 times with different random seeds, after which the results are aggregated, printed and plotted.
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py feature-selection resources/yaml/device_docker.yaml resources/yaml/feature_selection/${CORPUS}.yaml
- The results land in
/cdcr/working_dir/${CORPUS}_feature_selection
. In order to use the selected features in subsequent experiments, the contents ofselected_features.txt
need to be integrated in a YAML config for hyperparameter optimization or training. The exact destination in those YAML files is undermodel
βfeatures
βselected_features
(see/cdcr/resources/yaml/train/ecbp/ecbp_clustering_xgboost.yaml
for an example).
Our hyperparameter optimization approach is split into two stages: identifying and optimizing the best mention pair classifier, and (building on top of that) identifying the best hyperparameters for agglomerative clustering.
The hyperparameter sets are sampled automatically with optuna. See the sample_classifier_config_with_optuna
method in /cdcr/python/handwritten_baseline/pipeline/model/scripts/train_predict_optimize.py
for the sampling ranges of each hyperparameter.
- Depending on the corpus to optimize hyperparameters for, run
export CORPUS=ecbp
,export CORPUS=fcct
, orexport CORPUS=gvc
. - First, to optimize the mention pair classifier:
- Pick an ML algorithm: run
export ML=
using the valuelr
for logistic regression,mlp
for a multi-layer perception,svm
for a probabilistic SVC classifier orxgboost
for an XGBoost tree boosting classifier. - Edit
/cdcr/resources/yaml/hyperopt/${CORPUS}/${CORPUS}_classifier_${ML}.yaml
.- This YAML file is pre-filled with our best results from the feature selection stage for this corpus.
- Fill in
train_data_path
andeval_data_path
with the paths to the pickle files of your preprocessed train and dev splits respectively. - Update the settings in the
hyperopt
section to your liking. The optimization uses 6-fold cross-validation and runs parallelized. We therefore recommend to choosecv_num_repeats
so that6 * cv_num_repeats % max_cores == 0
to minimize the number of idling CPU cores during the optimization.
- Run
A plot with the optimization progress will be generated every 10 trials in
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py hyperopt resources/yaml/device_docker.yaml resources/yaml/hyperopt/${CORPUS}/${CORPUS}_classifier_${ML}.yaml
/cdcr/working_dir/${CORPUS}_hyperopt_classifier_${ML}
. - See
/cdcr/working_dir/${CORPUS}_hyperopt_classifier_${ML}
for the results. To use the optimal hyperparameters in subsequent experiments, the contents ofbest_model_config.yaml
need to be integrated in a YAML config for hyperparameter optimization or training inside themodel
section (see/cdcr/resources/yaml/train/ecbp/ecbp_clustering_xgboost.yaml
for an example).
- Pick an ML algorithm: run
- To optimize the agglomerative clustering step:
- Edit
/cdcr/resources/yaml/hyperopt/${CORPUS}/${CORPUS}_clustering_${ML}.yaml
. Due to XGBoost performing best in our experiments, we provide a pre-filled version of this file only forML=xgboost
. Fill intrain_data_path
andeval_data_path
. - Run
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py hyperopt resources/yaml/device_docker.yaml resources/yaml/hyperopt/${CORPUS}/${CORPUS}_clustering_${ML}.yaml
- See
/cdcr/working_dir/${CORPUS}_hyperopt_clustering_${ML}
for the results, particularly thebest_model_config.yaml
file which contains the best identified hyperparameters as in the previous step.
- Edit
The system can be evaluated with predefined document clusters at test time.
This can be achieved by specifying a path to a pickle file for the hard_document_clusters_file
option in any of the YAML configurations inside /cdcr/resources/yaml/evaluate/
.
The pickle file is expected to contain a list of lists of document identifiers representing the clustering. One such file is produced by the data preprocessing pipeline (named *_gold_transitive_closure_doc_clustering.pkl
) which contains the gold clustering of the split as defined by the transitive closure over mentions.
The mention pair classifier component of the system can be evaluated separately (see sections 6.3.2 and 6.3.3 in the paper):
- Perform steps 1 and 2i of the hyperparameter optimization instructions.
- Edit
/cdcr/resources/yaml/train/${CORPUS}/${CORPUS}_classifier_${ML}.yaml
:- Due to XGBoost performing best in our experiments, we provide a pre-filled version of this file only for
ML=xgboost
. When using xgboost as a classifier, it is possible to enable theanalyze_feature_importance
option which will produce thefeature_importances.csv
andfeature_importances_aggregated.txt
files reporting the gain from each feature. - Fill in
train_data_path
with the paths to the pickle files of your preprocessed train split.
- Due to XGBoost performing best in our experiments, we provide a pre-filled version of this file only for
- Run
The trained models appear in
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py train resources/yaml/device_docker.yaml resources/yaml/train/${CORPUS}/${CORPUS}_classifier_${ML}.yaml
/cdcr/working_dir/${CORPUS}_training_classifier_${ML}
in aserialized_models
folder. - Edit
/cdcr/resources/yaml/evaluate/${CORPUS}/${CORPUS}_classifier.yaml
. Fill ineval_data_path
with the path to the pickle file of your preprocessed test split. - Run (with
/path/to/serialized_models
set to the path with models from step 3):The following files are generated:python run_feature_model.py evaluate /path/to/serialized_models resources/yaml/device_docker.yaml resources/yaml/evaluate/${CORPUS}/${CORPUS}_classifier.yaml
The content of/cdcr/working_dir/${CORPUS}_evaluate_classifier/ βββ bec23be8 βββ 2020-11-12_12-37-42 βββ 0 βΒ Β βββ outcomes.pkl # serialized predictions of model with random seed 0 βββ ... βββ detailed_metrics βΒ Β βββ p_r_f1_average_over_runs_ignoring_link_types.txt # Mean P, R, F1 over all runs, ignoring CDCR link types βΒ Β βββ p_r_f1_average_over_runs_for_each_link_type.txt # Mean P, R, F1 over all runs, but for each CDCR link type separately βΒ Β βββ mean_absolute_confusion_matrix_quadrants_over_runs.txt # Mean absolute number of TP, FP, ... predictions for each CDCR link type over all runs βΒ Β βββ *.pkl # serialized pandas DataFrames of each of the pretty-printed tables βββ ... βββ metrics_aggregated_pretty.txt # same content as p_r_f1_average_over_runs_for_each_link_type.txt βββ prediction_examples βββ prediction_examples.csv # contains as many predicted mention pairs per confusion matrix quadrant and each link type as specified by the 'num_samples_per_quadrant' setting, in machine-readable form βββ cross-subtopic pairs # contains pretty-printed cross-subtopic link predictions βΒ Β βββ TXT_FN_prediction_examples.txt # pretty-printed false negatives mention pairs, with document context and action mention spans >>>emphasized<<< βΒ Β βββ TXT_FP_prediction_examples.txt # false positives, and so on βΒ Β βββ TXT_TN_prediction_examples.txt βΒ Β βββ TXT_TP_prediction_examples.txt βΒ Β βββ TEX_*.tex # all the above in LaTeX form βββ cross-topic pairs # pretty-printed cross-topic link predictions, and so on βΒ Β βββ ... βββ within-document pairs βΒ Β βββ ... βββ within-subtopic pairs βββ ...
metrics_aggregated_pretty.txt
is logged to stdout upon completion of the script.
To reproduce the lemma
, lemma-delta
and lemma-time
baseline experiments (reported in table 5 in the paper):
- Depending on which corpus to run baselines for, run
export CORPUS=ecbp
,export CORPUS=fcct
, orexport CORPUS=gvc
. - Edit
/cdcr/resources/yaml/lemma_baselines/${CORPUS}.yaml
and fill intrain_data_path
andeval_data_path
with the paths to the pickle files of your preprocessed train and test splits respectively. - Run
This triggers optimization of the
cd /cdcr source venv_allennlp_1.0.0/bin/activate python run_feature_model.py lemma-baselines resources/yaml/device_docker.yaml resources/yaml/lemma_baselines/${CORPUS}.yaml
lemma-delta
andlemma-time
baselines, followed by prediction and evaluation with coreference resolution metrics. - The following files are generated:
/cdcr/working_dir/${CORPUS}_lemma_baselines βββ b9d92fee βββ 2020-11-12_15-02-52 βββ results βΒ Β βββ lemma βΒ Β βΒ Β βββ cross_doc # evaluation scenario where all documents are merged into a single meta-document in the gold and system CoNLL files. This is what we report in the paper. βΒ Β βΒ Β βΒ Β βββ eval_lemma_cross_doc.txt # output from the scoring script βΒ Β βΒ Β βΒ Β βββ gold.conll # gold annotations - the naming scheme for mentions is `metadoc_{document id}_{mention id}` βΒ Β βΒ Β βΒ Β βββ system.conll # predictions βΒ Β βΒ Β βββ within_doc # evaluation scenario where documents are scored in isolation, i.e. only within-document links are scored and cross-document links are left out entirely. βΒ Β βΒ Β βββ ... βΒ Β βββ lemma-delta βΒ Β βΒ Β βββ ... βΒ Β βββ lemma-time βΒ Β βΒ Β βββ ... βΒ Β βββ lemma-wd # not reported in the paper; only performs within-document coreference resolution (see Upadhyay 2016 et al. https://www.aclweb.org/anthology/C16-1183/) βΒ Β βββ ... βββ tfidf # optimization progress reports of the lemma-tfidf optimization βββ time # reports of the lemma-time optimization
- This implementation caches the results of several operations to disk to speed up subsequent executions. For example, the corpus preprocessing pipeline caches queries to CoreNLP, DBpedia, the SRL system and more. The training/prediction code caches mention pair features which considerably speeds up hyperparameter optimization. In case of problems (or when modifying these components), it may be necessary to clear the affected caches. Do this by manually removing the corresponding cache folders inside
/cdcr/working_dir/global
. - If you encounter OOM issues during hyperparameter optimization, try reducing
max_cores
indevice_docker.yaml
. - Please open a github issue in case of other problems.
- ECB+ corpus: http://www.newsreader-project.eu/results/data/the-ecb-corpus/
- Gun Violence Corpus (GVC): https://github.com/cltl/GunViolenceCorpus
- CoVal: A coreference evaluation tool for the CoNLL and ARRAU datasets: https://github.com/ns-moosavi/coval