-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
364 lines (316 loc) · 18.1 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Adverse Events</title>
<link rel="icon" href="https://raw.githubusercontent.com/gramener/assets/main/straive-favicon.svg">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/css/bootstrap.min.css" crossorigin="anonymous">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.min.css">
<style>
.narrative {
max-width: 40rem;
}
.custom-scrollbar {
scrollbar-width: thin;
scrollbar-color: rgba(0, 0, 0, 0.3) transparent;
}
.custom-scrollbar::-webkit-scrollbar {
width: 8px;
}
.custom-scrollbar::-webkit-scrollbar-track {
background: transparent;
}
.custom-scrollbar::-webkit-scrollbar-thumb {
background-color: rgba(0, 0, 0, 0.3);
border-radius: 4px;
border: 2px solid transparent;
}
.custom-scrollbar::-webkit-scrollbar-thumb:hover {
background-color: rgba(0, 0, 0, 0.5);
}
</style>
</head>
<body>
<nav class="navbar navbar-expand-lg bg-body-tertiary" data-bs-theme="dark">
<div class="container-fluid">
<a class="navbar-brand" href=".">Adverse Events</a>
<button class="navbar-toggler" type="button" data-bs-toggle="collapse" data-bs-target="#navbarSupportedContent" aria-controls="navbarSupportedContent" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>
<div class="collapse navbar-collapse" id="navbarSupportedContent">
<div class="nav-item dropdown ms-auto" role="group" aria-label="Toggle dark mode" title="Toggle Dark Mode">
<button class="dark-theme-toggle btn btn-outline-light dropdown-toggle" type="button" data-bs-toggle="dropdown" aria-expanded="false" aria-label="Toggle theme (auto)">
<i class="bi bi-circle-half"></i> <span class="d-lg-none ms-2">Toggle theme</span>
</button>
<ul class="dropdown-menu dropdown-menu-end">
<li><button class="dropdown-item" data-bs-theme-value="light"><i class="me-2 bi bi-sun-fill"></i> Light</button></li>
<li><button class="dropdown-item" data-bs-theme-value="dark"><i class="me-2 bi bi-moon-stars-fill"></i> Dark</button></li>
<li><button class="dropdown-item" data-bs-theme-value="auto"><i class="me-2 bi bi-circle-half"></i> Auto</button></li>
</ul>
</div>
</div>
</div>
</nav>
<div class="container-fluid">
<h1 class="display-1 my-4 text-center">Adverse Events</h1>
<h2 class="display-6 text-center">How can we explainably identify adverse events?</h2>
<div class="mx-auto my-3 narrative">
<p>This application uses a <strong>multi-agent LLM workflow</strong> to identify whether user feedback from a clinical trial is an adverse event or not.</p>
<p>The workflow has tasks, each running a different model to evaluate the text for potential adverse events. Finally, it summarizes the results in a Pharmacovigilance assessment summary.</p>
<p>Here's a breakdown of the workflow:</p>
<ol>
<li><strong>Context Analysis</strong>: The basic model analyzes the clinical description to determine if there is any mention of an adverse event.</li>
<li><strong>BioClin Analysis</strong>: An intermediate model performs a more detailed evaluation, considering clinical reasoning to assess the presence of adverse reactions.</li>
<li><strong>LLM as a Judge Analysis</strong>: A specialized model acts as a judge, evaluating the feedback from the previous analyses based on predefined criteria.</li>
<li><strong>Judge Feedback to Context Analysis</strong>: The judge provides feedback on the context analysis, which is then used to refine the initial evaluation.</li>
<li><strong>Judge Feedback to BioClin Analysis</strong>: Similar to the previous step, the judge reviews the bio-clinical analysis and offers feedback for improvement.</li>
<li><strong>Context Analysis - Revised</strong>: The basic model re-evaluates the clinical description, incorporating the judge's feedback to enhance its analysis.</li>
<li><strong>BioClin Analysis - Revised</strong>: The intermediate model revisits its analysis, integrating the judge's feedback for a more accurate assessment.</li>
<li><strong>Judge Summary</strong>: Finally, the judge summarizes the findings from all analyses, generating a comprehensive report that includes a pharmacovigilance assessment summary in JSON format.</li>
</ol>
</div>
<pre class="mermaid container-fluid mb-5 text-center">
%%{
init: {
'theme': 'base',
'themeVariables': {
'primaryColor': '#BB2528',
'primaryTextColor': '#fff',
'primaryBorderColor': '#7C0000',
'lineColor': '#F8B229',
'secondaryColor': '#006100',
'tertiaryColor': '#fff'
}
}
}%%
flowchart LR
A[Context Analysis] --> C[LLM as a Judge Analysis]
B[BioClin Analysis] --> C
C --> D[Judge Feedback to Context Analysis]
C --> E[Judge Feedback to BioClin Analysis]
D --> F[Context Analysis - Revised]
E --> G[BioClin Analysis - Revised]
F --> H[Judge Summary]
G --> H
C --> H
</pre>
<form id="adverse-event-form" class="container my-4">
<div class="mb-3">
<label for="samples" class="form-label h5">Pick a sample clinical trial description</label>
<select class="form-select" id="samples"></select>
</div>
<div class="mb-4">
<label for="clinical-description" class="form-label h5">... or enter your clinical trial description</label>
<textarea class="form-control" name="clinical-description" id="clinical-description" rows="5" placeholder="Enter the clinical trial description here..." required></textarea>
</div>
<details class="mb-3 d-none" id="advanced-settings">
<summary class="h5 mb-3">Advanced Settings</summary>
<section class="eval">
<div class="mb-3">
<label for="basic-prompt" class="form-label">Basic Prompt</label>
<textarea class="form-control" name="basic-prompt" id="basic-prompt" rows="2">Is there an adverse event in this text? Be very concise and talk in points. Err on the side of caution.
End with one of these:
**Conclusion: Adverse event confirmed**
**Conclusion: Not an adverse event**</textarea>
</div>
<div class="mb-3">
<label for="basic-model" class="form-label">Basic Model</label>
<select class="form-select models" name="basic-model" id="basic-model"></select>
</div>
</section>
<section class="eval">
<div class="mb-3">
<label for="intermediate-prompt" class="form-label">Intermediate Prompt</label>
<textarea class="form-control" name="intermediate-prompt" id="intermediate-prompt" rows="5">You are a clinical expert.
Think step by step. But be very concise. Respond with ✅ or ❌ against each category you evaluate. Each category should be a new bullet.
End with one of these:
**Conclusion: Adverse event confirmed**
**Conclusion: Not an adverse event**
But remember that you should conclude that a health condition is an adverse event only if you are able to clearly establish relationship between drug intake and symptoms / health condition. Err on the side of caution.</textarea>
</div>
<div class="mb-3">
<label for="intermediate-model" class="form-label">Intermediate Model</label>
<select class="form-select models" name="intermediate-model" id="intermediate-model"></select>
</div>
</section>
<section class="eval">
<div class="mb-3">
<label for="judge-prompt" class="form-label">LLM Judge Prompt</label>
<textarea class="form-control" name="judge-prompt" id="judge-prompt" rows="5">A subject has provided assessment of presence of an adverse event. Evaluate it on the following parameters:
- Temporal Ambiguity and Causality: Determines if there's a clear timeline between drug intake and symptoms. Without a specific temporal link, an adverse event will not be assumed.
- Language Variability and Colloquialisms: Adjusts for non-medical or varied terms describing symptoms (e.g., "feeling woozy" could imply different conditions). The assistant will only confirm an AE when it can interpret symptoms accurately.
- Context Dependencies: Identifies whether symptoms could relate to the condition being treated, unrelated issues, or drug side effects. A conclusion is only drawn if context supports a drug-related cause.
- Multiple Drug Interactions: When multiple medications are mentioned, the assistant evaluates if symptoms could result from interactions rather than single drugs, erring on the side of caution.
- Incomplete Information: If key details are missing (medication names, timing, etc.), it will avoid conclusions and recommend more information.
- Severity Assessment Challenges: Avoids conclusions where severity is unclear from casual descriptions, as phrases like "massive headache" are hard to quantify.
- Reliability and Verification Issues: Uses only reliable information and will avoid conclusions based on second-hand reports or exaggerations.
- Cultural and Regional Variations: Considers cultural differences in symptom descriptions; avoids conclusions when meaning may vary.
- Sarcasm and Figurative Language: Ignores non-literal expressions and does not conclude adverse events from figurative statements.
- Conditional Symptoms: Examines if symptoms occur only under specific conditions (e.g., "only dizzy on an empty stomach") and avoids conclusions unless these patterns are clear.
- Evolving Symptoms: Takes into account symptom progression over time to differentiate adverse events from disease progression or other causes.
- Privacy and Identity Verification: Without clear user verification, it refrains from drawing definite conclusions.
- Natural Disease Progression vs. AE: Differentiates between disease symptoms, side effects, and disease progression to avoid misattributions.
- Reporting Bias: Considers reporting tendencies, recognizing that severe or negative experiences are more likely to be shared online.
- Technical Challenges: Handles issues with spelling, abbreviations, or context-dependent meanings to avoid inaccuracies.
Then decide whether the feedback is an adverse reaction or not. Remember that you should call a medical event an adverse event only if a clear relationship is established between drug intake and medical condition.
Render each step of the reasoning like this:
<details>
<summary>Evaluating: Temporal Ambiguity and Causality</summary>
<p>...</p>
</details>
<p>Result: <strong>No temporal link</strong> between drug intake and symptoms. No adverse event</p>
<details>
<summary>Evaluating: Language Variability and Colloquialisms</summary>
<p>...</p>
</details>
<p>Result: <strong>Interpretation of symptoms is accurate</strong> (feeling woozy could be dizziness). Adverse event confirmed</p>
End with one of these:
**Conclusion: Adverse event confirmed**
**Conclusion: Not an adverse event**
But remember that you should conclude that a health condition is an adverse event only if you are able to clearly establish relationship between drug intake and symptoms / health condition. Avoid false positives as much as you can.</textarea>
</div>
<div class="mb-3">
<label for="judge-feedback-prompt" class="form-label">LLM Judge Feedback Prompt</label>
<textarea class="form-control" name="judge-feedback-prompt" id="judge-feedback-prompt" rows="2">You evaluated presence of an adverse event based on your criteria. Original Text and your analysis - both are below.
The user is a model that has done the same with simpler criteria. Give short concise feedback to the model. Don't implicitly or explicitly ask it to change or reconsider the position of their judgement. Also, DO NOT reveal your evaluation criteria.</textarea>
</div>
<div class="mb-3">
<label for="revision-prompt" class="form-label">Model Revision Prompt</label>
<textarea class="form-control" name="revision-prompt" id="revision-prompt" rows="2">Here is some feedback. Consider this feedback and give one final assessment. Don't feel compelled changing the assessment based on the feedback. Stick to your original analysis if you are confident. In your response, be very short and concise. Talk in points.</textarea>
</div>
<div class="mb-3">
<label for="summary-prompt" class="form-label">Summary Prompt</label>
<textarea class="form-control" name="summary-prompt" id="summary-prompt" rows="2">A subject has provided assessment of presence of adverse event.
The evaluations from 3 models are below.
Generate a Pharmacovigilance Assessment Summary as JSON in the provided schema.</textarea>
</div>
<div class="mb-3">
<label for="summary-schema" class="form-label">Summary Schema</label>
<textarea class="form-control font-monospace" name="summary-schema" id="summary-schema" rows="5">{
"type": "object",
"properties": {
"status": {
"type": "string",
"enum": ["VALID CASE", "INVALID CASE", "PENDING REVIEW", "CLOSED"]
},
"causality": {
"type": "string",
"enum": ["Certain", "Probable", "Possible", "Unlikely", "Conditional", "Unassessable", "Unrelated"]
},
"seriousness": {
"type": "string",
"enum": ["Serious", "Non-Serious", "Life-threatening", "Hospitalization required", "Disability", "Congenital anomaly", "Death"]
},
"expectedness": {
"type": "string",
"enum": ["Listed", "Unlisted", "Unknown", "Unexpected"]
},
"primary_event": {
"type": "string",
"description": "The main adverse event category. Examples include 'Gastrointestinal Disorder', 'Cardiac Event', 'Skin Reaction', 'Neurological Disorder', etc."
},
"meddra_pt": {
"type": "string",
"description": "The MedDRA (Medical Dictionary for Regulatory Activities) preferred term code for the primary event. Typically a numeric identifier, e.g., '10000060' for 'Gastrointestinal Disorder'."
},
"case_completeness": {
"type": "string",
"enum": ["Perfectly documented (5/5)", "Well-documented (4/5)", "Moderate documentation (3/5)", "Low documentation quality (2/5)", "Poorly documented (1/5)"]
},
"report_type": {
"type": "string",
"enum": ["Expedited (15-day)", "Periodic", "Annual", "Initial report", "Follow-up report", "None required"]
},
"pbrer_inclusion": {
"type": "string",
"enum": ["Required", "Not Required", "Optional", "Pending review"]
},
"signal_status": {
"type": "string",
"enum": ["Under Monitoring", "Not Under Monitoring", "Signal Closed", "New Signal Detected", "Further Investigation Required"]
},
"expedited_report": {
"type": "string",
"enum": ["Submit within 15 days", "Submit within 7 days", "Submit within 30 days", "No expedited report required", "Submission required upon request"]
},
"include_in_pbrer_psur": {
"type": "string",
"enum": ["Include", "Do not include", "Review for inclusion", "Include if recurring"]
},
"update_safety_database": {
"type": "string",
"enum": ["Yes", "No", "Pending approval", "Already updated"]
},
"update_signal_detection_database": {
"type": "string",
"enum": ["Yes", "No", "Pending review", "Already updated"]
},
"ror_score": {
"type": "number",
"description": "Rate of Reporting Odds Ratio (ROR) score for the signal, usually numeric. Example values include '2.5', '1.0', etc."
},
"ror_status": {
"type": "string",
"enum": ["Above threshold", "Below threshold", "At threshold"]
},
"cases_in_database": {
"type": "integer",
"description": "Number of similar cases in the database. Examples: 15, 50, 100."
},
"labeling_status": {
"type": "string",
"enum": ["No update needed", "Update recommended", "Update required", "Pending further data"]
}
},
"required": [
"status",
"causality",
"seriousness",
"expectedness",
"primary_event",
"meddra_pt",
"case_completeness",
"report_type",
"pbrer_inclusion",
"signal_status",
"expedited_report",
"include_in_pbrer_psur",
"update_safety_database",
"update_signal_detection_database",
"ror_score",
"ror_status",
"cases_in_database",
"labeling_status"
],
"additionalProperties": false
}
</textarea>
</div>
<div class="mb-3">
<label for="judge-model" class="form-label">LLM Judge Model</label>
<select class="form-select models" name="judge-model" id="judge-model"></select>
</div>
<!-- Add a Bootstrap switch to slow down the rendering -->
<div class="mb-3 form-check form-switch">
<input class="form-check-input" type="checkbox" role="switch" id="slow-rendering">
<label class="form-check-label" for="slow-rendering">Slow down the rendering</label>
</div>
</section>
</details>
<div id="analyze" class="d-none">
<button type="submit" class="btn btn-primary">Analyze</button>
<button type="reset" class="btn btn-secondary">Reset</button>
</div>
</form>
<section id="results" class="container-fluid"></section>
<footer class="my-5 vh-100 d-flex align-items-center justify-content-center">
<h1 class="display-4">Designed by <a href="https://gramener.com/" class="text-reset link-offset-3 link-underline link-underline-opacity-25">Gramener</a></h1>
</footer>
</div>
<script src="https://cdn.jsdelivr.net/npm/bootstrap@5.3.3/dist/js/bootstrap.bundle.min.js" type="module"></script>
<script src="https://cdn.jsdelivr.net/npm/@gramex/ui@0.3/dist/dark-theme.js" type="module"></script>
<script src='https://cdn.jsdelivr.net/gh/FThompson/FormPersistence.js@2.0.6/form-persistence.min.js'></script>
<script src="script.js" type="module"></script>
</body>
</html>