feedback by Vincent Carey

[ci skip]

This build is based on
5eaecd1.

This commit was created by the following CI build and job:
https://github.com/biocypher/biochatter-paper/commit/5eaecd1c15465add63457d91832d56774ca2564a/checks
https://github.com/biocypher/biochatter-paper/actions/runs/8022516515

README.md

@@ -1,7 +1,7 @@
 # Output directory containing the formatted manuscript
 
 The [`gh-pages`](https://github.com/biocypher/biochatter-paper/tree/gh-pages) branch hosts the contents of this directory at <https://biocypher.github.io/biochatter-paper/>.
-The permalink for this webpage version is <https://biocypher.github.io/biochatter-paper/v/0cdd37790a8dda42978a9ad0e39a66b21daa9f83/>.
+The permalink for this webpage version is <https://biocypher.github.io/biochatter-paper/v/5eaecd1c15465add63457d91832d56774ca2564a/>.
 To redirect to the permalink for the latest manuscript version at anytime, use the link <https://biocypher.github.io/biochatter-paper/v/freeze/>.
 
 ## Files
@@ -35,4 +35,4 @@ Verifying timestamps with the `ots verify` command requires running a local bitc
 ## Source
 
 The manuscripts in this directory were built from
-[`0cdd37790a8dda42978a9ad0e39a66b21daa9f83`](https://github.com/biocypher/biochatter-paper/commit/0cdd37790a8dda42978a9ad0e39a66b21daa9f83).
+[`5eaecd1c15465add63457d91832d56774ca2564a`](https://github.com/biocypher/biochatter-paper/commit/5eaecd1c15465add63457d91832d56774ca2564a).

index.html

@@ -13,7 +13,7 @@
   <meta name="author" content="Nils Krehl" />
   <meta name="author" content="Qin Ma" />
   <meta name="author" content="Julio Saez-Rodriguez" />
-  <meta name="dcterms.date" content="2024-02-17" />
+  <meta name="dcterms.date" content="2024-02-23" />
   <meta name="keywords" content="biomedicine, large language models, framework, retrieval-augmented generation, knowledge graph" />
   <title>A Platform for the Biomedical Application of Large Language Models</title>
   <style>
@@ -121,11 +121,11 @@
   <meta name="citation_title" content="A Platform for the Biomedical Application of Large Language Models" />
   <meta property="og:title" content="A Platform for the Biomedical Application of Large Language Models" />
   <meta property="twitter:title" content="A Platform for the Biomedical Application of Large Language Models" />
-  <meta name="dc.date" content="2024-02-17" />
-  <meta name="citation_publication_date" content="2024-02-17" />
-  <meta property="article:published_time" content="2024-02-17" />
-  <meta name="dc.modified" content="2024-02-17T07:09:01+00:00" />
-  <meta property="article:modified_time" content="2024-02-17T07:09:01+00:00" />
+  <meta name="dc.date" content="2024-02-23" />
+  <meta name="citation_publication_date" content="2024-02-23" />
+  <meta property="article:published_time" content="2024-02-23" />
+  <meta name="dc.modified" content="2024-02-23T17:03:33+00:00" />
+  <meta property="article:modified_time" content="2024-02-23T17:03:33+00:00" />
   <meta name="dc.language" content="en-UK" />
   <meta name="citation_language" content="en-UK" />
   <meta name="dc.relation.ispartof" content="Manubot" />
@@ -169,9 +169,9 @@
   <meta name="citation_fulltext_html_url" content="https://biocypher.github.io/biochatter-paper/" />
   <meta name="citation_pdf_url" content="https://biocypher.github.io/biochatter-paper/manuscript.pdf" />
   <link rel="alternate" type="application/pdf" href="https://biocypher.github.io/biochatter-paper/manuscript.pdf" />
-  <link rel="alternate" type="text/html" href="https://biocypher.github.io/biochatter-paper/v/0cdd37790a8dda42978a9ad0e39a66b21daa9f83/" />
-  <meta name="manubot_html_url_versioned" content="https://biocypher.github.io/biochatter-paper/v/0cdd37790a8dda42978a9ad0e39a66b21daa9f83/" />
-  <meta name="manubot_pdf_url_versioned" content="https://biocypher.github.io/biochatter-paper/v/0cdd37790a8dda42978a9ad0e39a66b21daa9f83/manuscript.pdf" />
+  <link rel="alternate" type="text/html" href="https://biocypher.github.io/biochatter-paper/v/5eaecd1c15465add63457d91832d56774ca2564a/" />
+  <meta name="manubot_html_url_versioned" content="https://biocypher.github.io/biochatter-paper/v/5eaecd1c15465add63457d91832d56774ca2564a/" />
+  <meta name="manubot_pdf_url_versioned" content="https://biocypher.github.io/biochatter-paper/v/5eaecd1c15465add63457d91832d56774ca2564a/manuscript.pdf" />
   <meta property="og:type" content="article" />
   <meta property="twitter:card" content="summary_large_image" />
   <link rel="icon" type="image/png" sizes="192x192" href="https://manubot.org/favicon-192x192.png" />
@@ -188,10 +188,10 @@ <h1 class="title">A Platform for the Biomedical Application of Large Language Mo
 </header>
 <p><small><em>
 This manuscript
-(<a href="https://biocypher.github.io/biochatter-paper/v/0cdd37790a8dda42978a9ad0e39a66b21daa9f83/">permalink</a>)
+(<a href="https://biocypher.github.io/biochatter-paper/v/5eaecd1c15465add63457d91832d56774ca2564a/">permalink</a>)
 was automatically generated
-from <a href="https://github.com/biocypher/biochatter-paper/tree/0cdd37790a8dda42978a9ad0e39a66b21daa9f83">biocypher/biochatter-paper@0cdd377</a>
-on February 17, 2024.
+from <a href="https://github.com/biocypher/biochatter-paper/tree/5eaecd1c15465add63457d91832d56774ca2564a">biocypher/biochatter-paper@5eaecd1</a>
+on February 23, 2024.
 </em></small></p>
 <h2 id="authors">Authors</h2>
 <ul>
@@ -402,7 +402,7 @@ <h3 id="benchmarking">Benchmarking</h3>
 For transparent and reproducible evaluation of LLMs, we implement a benchmarking framework that allows the comparison of models, prompt sets, and all other components of the pipeline.
 The generic Pytest framework <span class="citation" data-cites="14upAJPXR">[<a href="#ref-14upAJPXR" role="doc-biblioref">31</a>]</span> allows for the automated evaluation of a matrix of all possible combinations of components.
 The results are stored and displayed on our website for simple comparison, and the benchmark is updated upon the release of new models and extensions to the datasets and BioChatter capabilities (<a href="https://biochatter.org/benchmark/">https://biochatter.org/benchmark/</a>).</p>
-<p>Since the biomedical domain has its own tasks and requirements, we created a bespoke benchmark that allows us to be more precise in the evaluation of components <span class="citation" data-cites="uYvzQA7w">[<a href="#ref-uYvzQA7w" role="doc-biblioref">25</a>]</span>.
+<p>Since the biomedical domain has its own tasks and requirements <span class="citation" data-cites="uYvzQA7w">[<a href="#ref-uYvzQA7w" role="doc-biblioref">25</a>]</span>, we created a bespoke benchmark that allows us to be more precise in the evaluation of components.
 This is complementary to the existing, general-purpose benchmarks and leaderboards for LLMs <span class="citation" data-cites="KONKs6Pw LE2GwIqT foK1oImy">[<a href="#ref-LE2GwIqT" role="doc-biblioref">24</a>,<a href="#ref-KONKs6Pw" role="doc-biblioref">32</a>,<a href="#ref-foK1oImy" role="doc-biblioref">33</a>]</span>.
 Furthermore, to prevent leakage of the benchmark data into the training data of the models, a known issue in the general-purpose benchmarks <span class="citation" data-cites="yT66jV6G">[<a href="#ref-yT66jV6G" role="doc-biblioref">34</a>]</span>, we implemented an encrypted pipeline that contains the benchmark datasets and is only accessible to the workflow that executes the benchmark (see Methods).</p>
 <p>Analysis of these benchmarks confirmed the prevailing opinion of OpenAI’s leading role in LLM performance (Figure <a href="#fig:benchmark">3</a> A).
@@ -652,7 +652,7 @@ <h2 class="page_break_before" id="references">References</h2>
 <div class="csl-left-margin">18. </div><div class="csl-right-inline"><strong>Mixtral of Experts</strong> <div class="csl-block">Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, … William El Sayed</div> <em>arXiv</em> (2024) <a href="https://doi.org/gtc2g3">https://doi.org/gtc2g3</a> <div class="csl-block">DOI: <a href="https://doi.org/10.48550/arxiv.2401.04088">10.48550/arxiv.2401.04088</a></div></div>
 </div>
 <div id="ref-mGEvmJGA" class="csl-entry" role="doc-biblioentry">
-<div class="csl-left-margin">19. </div><div class="csl-right-inline"><strong>xorbitsai/inference</strong> <div class="csl-block">Xorbits</div> (2024-02-17) <a href="https://github.com/xorbitsai/inference">https://github.com/xorbitsai/inference</a></div>
+<div class="csl-left-margin">19. </div><div class="csl-right-inline"><strong>xorbitsai/inference</strong> <div class="csl-block">Xorbits</div> (2024-02-23) <a href="https://github.com/xorbitsai/inference">https://github.com/xorbitsai/inference</a></div>
 </div>
 <div id="ref-PDhRVYjU" class="csl-entry" role="doc-biblioentry">
 <div class="csl-left-margin">20. </div><div class="csl-right-inline"><a href="https://www.reuters.com/technology/european-data-protection-board-discussing-ai-policy-thursday-meeting-2023-04-13/">https://www.reuters.com/technology/european-data-protection-board-discussing-ai-policy-thursday-meeting-2023-04-13/</a></div>
@@ -688,7 +688,7 @@ <h2 class="page_break_before" id="references">References</h2>
 <div class="csl-left-margin">30. </div><div class="csl-right-inline"><strong>A Survey on Large Language Model based Autonomous Agents</strong> <div class="csl-block">Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, … Ji-Rong Wen</div> <em>arXiv</em> (2023) <a href="https://doi.org/gsv93m">https://doi.org/gsv93m</a> <div class="csl-block">DOI: <a href="https://doi.org/10.48550/arxiv.2308.11432">10.48550/arxiv.2308.11432</a></div></div>
 </div>
 <div id="ref-14upAJPXR" class="csl-entry" role="doc-biblioentry">
-<div class="csl-left-margin">31. </div><div class="csl-right-inline"><strong>pytest-dev/pytest</strong> <div class="csl-block">pytest-dev</div> (2024-02-17) <a href="https://github.com/pytest-dev/pytest">https://github.com/pytest-dev/pytest</a></div>
+<div class="csl-left-margin">31. </div><div class="csl-right-inline"><strong>pytest-dev/pytest</strong> <div class="csl-block">pytest-dev</div> (2024-02-23) <a href="https://github.com/pytest-dev/pytest">https://github.com/pytest-dev/pytest</a></div>
 </div>
 <div id="ref-KONKs6Pw" class="csl-entry" role="doc-biblioentry">
 <div class="csl-left-margin">32. </div><div class="csl-right-inline"><strong>Large language models encode clinical knowledge</strong> <div class="csl-block">Karan Singhal, Shekoofeh Azizi, Tao Tu, SSara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, … Vivek Natarajan</div> <em>Nature</em> (2023-07-12) <a href="https://doi.org/gsgp8c">https://doi.org/gsgp8c</a> <div class="csl-block">DOI: <a href="https://doi.org/10.1038/s41586-023-06291-2">10.1038/s41586-023-06291-2</a> · PMID: <a href="https://www.ncbi.nlm.nih.gov/pubmed/37438534">37438534</a> · PMCID: <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10396962">PMC10396962</a></div></div>
@@ -736,7 +736,7 @@ <h2 class="page_break_before" id="references">References</h2>
 <div class="csl-left-margin">46. </div><div class="csl-right-inline"><strong>Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks</strong> <div class="csl-block">Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, … Douwe Kiela</div> <em>Advances in Neural Information Processing Systems</em> (2020) <a href="https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf">https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf</a></div>
 </div>
 <div id="ref-12p6amlLS" class="csl-entry" role="doc-biblioentry">
-<div class="csl-left-margin">47. </div><div class="csl-right-inline"><strong>Hugging Face – The AI community building the future.</strong> (2024-02-13) <a href="https://huggingface.co/">https://huggingface.co/</a></div>
+<div class="csl-left-margin">47. </div><div class="csl-right-inline"><strong>Hugging Face – The AI community building the future.</strong> (2024-02-22) <a href="https://huggingface.co/">https://huggingface.co/</a></div>
 </div>
 <div id="ref-19HQJNP46" class="csl-entry" role="doc-biblioentry">
 <div class="csl-left-margin">48. </div><div class="csl-right-inline"><strong>ABC transporters affects tumor immune microenvironment to regulate cancer immunotherapy and multidrug resistance</strong> <div class="csl-block">Jingyi Fan, Kenneth Kin Wah To, Zhe-Sheng Chen, Liwu Fu</div> <em>Drug Resistance Updates</em> (2023-01) <a href="https://doi.org/gtg7tg">https://doi.org/gtg7tg</a> <div class="csl-block">DOI: <a href="https://doi.org/10.1016/j.drup.2022.100905">10.1016/j.drup.2022.100905</a> · PMID: <a href="https://www.ncbi.nlm.nih.gov/pubmed/36463807">36463807</a></div></div>