Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2024 04 gg tx control #886

Merged
merged 5 commits into from
Apr 23, 2024
Merged

2024 04 gg tx control #886

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
eb135ee
Fix bug checking authorised javascript - not checking anonymous scrip…
Apr 22, 2024
b7a734c
render versions in profile links when necessary
Apr 22, 2024
dcfef33
better handle duplicate references in the IG resource + add translati…
Apr 22, 2024
0337c33
release
Apr 22, 2024
d1b82da
set up release
Apr 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions RELEASE_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
* General: fix spelling of heirarchy
* Dependencies: Bump lombok (#1603, #882), jackson, graalvm (#881)
* Loader: fix NPE loading resources
* Loader: better handle duplicate references in the IG resource
* Terminology Support: Send supplements to tx server
* Terminology Support: fix bug processing code bindings when value sets are complex (multiple filters)
* FHIRPath: Remove the alias/aliasAs custom functions (use standard defineVariable now)
* Validator: Don't enforce ids on elements when processing CDA
* Validator: rework OID handling for better OID -> CodeSystem resolution
* Renderer: render versions in profile links when necessary
* QA: Fix bug checking authorised javascript - not checking anonymous script element properly

## WHO Multi-language support

* More work on language support
* add translations to json data files
* create _data/languages.json
* generate language specific resource instances
5 changes: 3 additions & 2 deletions Mulit-lingual IGs.md → multi-lingual-IGs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,9 @@ Mulit-lingual IGs

#1 Introduction

All IGs are authored in a single primary language that is the master language.
This does not have to be in english (even though most existing IGs are)
All IGs are authored in a single primary language that is the master language, and then
additional languages are provided through the use of translation infrastructure.
The primary language does not have to be in english (even though most existing IGs use english)
Note that the external resources to the IG are also have a single primary
language, but may provide translations.

Expand Down
30 changes: 27 additions & 3 deletions org.hl7.fhir.publisher.core/src/main/java/org/hl7/fhir/igtools/publisher/HTMLInspector.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,10 @@ public String getNextId() {
private Set<String> foundFragments = new HashSet<>();
private List<FetchedFile> sources;
private IPublisherModule module;
private boolean isCIBuild;
private Map<String, ValidationMessage> jsmsgs = new HashMap<>();

public HTMLInspector(String rootFolder, List<SpecMapManager> specs, List<SpecMapManager> linkSpecs, ILoggingService log, String canonical, String packageId, Map<String, List<String>> trackedFragments, List<FetchedFile> sources, IPublisherModule module) {
public HTMLInspector(String rootFolder, List<SpecMapManager> specs, List<SpecMapManager> linkSpecs, ILoggingService log, String canonical, String packageId, Map<String, List<String>> trackedFragments, List<FetchedFile> sources, IPublisherModule module, boolean isCIBuild) {
this.rootFolder = rootFolder.replace("/", File.separator);
this.specs = specs;
this.linkSpecs = linkSpecs;
Expand All @@ -222,6 +224,7 @@ public HTMLInspector(String rootFolder, List<SpecMapManager> specs, List<SpecMap
this.trackedFragments = trackedFragments;
this.sources = sources;
this.module = module;
this.isCIBuild = isCIBuild;
requirePublishBox = Utilities.startsWithInList(packageId, "hl7.");
}

Expand Down Expand Up @@ -665,8 +668,29 @@ public String genID(LoadedFile lf) {
private void checkScriptElement(String filename, Location loc, String path, XhtmlNode x, List<ValidationMessage> messages) {
String src = x.getAttribute("src");
if (!Utilities.noString(src) && Utilities.isAbsoluteUrl(src) && !Utilities.existsInList(src,
"http://hl7.org/fhir/history-cm.js", "http://hl7.org/fhir/assets-hist/js/jquery.js") && !src.contains("googletagmanager.com"))
messages.add(new ValidationMessage(Source.Publisher, IssueType.NOTFOUND, filename+(loc == null ? "" : " at "+loc.toString()), "The <script> src '"+src+"' is llegal", IssueSeverity.FATAL));
"http://hl7.org/fhir/history-cm.js", "http://hl7.org/fhir/assets-hist/js/jquery.js") && !src.contains("googletagmanager.com")) {
messages.add(new ValidationMessage(Source.Publisher, IssueType.INVALID, filename+(loc == null ? "" : " at "+loc.toString()), "The <script> src '"+src+"' is illegal", IssueSeverity.FATAL));
} else if (src == null && x.allText() != null) {
String js = x.allText();
if (jsmsgs.containsKey(js)) {
ValidationMessage vm = jsmsgs.get(js);
vm.incCount();
} else {
ValidationMessage vm;
if (isCIBuild) {
vm = new ValidationMessage(Source.Publisher, IssueType.INVALID, filename+(path == null ? "" : "#"+path+(loc == null ? "" : " at "+loc.toString())), "The <script> tag containing the javascript '"+subset(x.allText())+"'... is illegal - put the script in a .js file in a trusted template", IssueSeverity.FATAL);
} else {
vm = new ValidationMessage(Source.Publisher, IssueType.INVALID, filename+(path == null ? "" : "#"+path+(loc == null ? "" : " at "+loc.toString())), "The <script> tag containing the javascript '"+subset(x.allText())+"'... is illegal and not allowed on the HL7 cibuild - put the script in a .js file in a trusted template", IssueSeverity.ERROR);
}
messages.add(vm);
jsmsgs.put(js, vm);
}
}
}

private String subset(String src) {
src = Utilities.stripEoln(src.strip()).replace("\"", "'");
return src.length() < 20 ? src : src.substring(0, 20);
}

private boolean checkLinkElement(String filename, Location loc, String path, String href, List<ValidationMessage> messages, String uuid) {
Expand Down
16 changes: 14 additions & 2 deletions ...fhir.publisher.core/src/main/java/org/hl7/fhir/igtools/publisher/IGKnowledgeProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -673,12 +673,24 @@ public String getLinkForProfile(StructureDefinition profile, String url) {
if (xver.matchingUrl(url)) {
return xver.getReference(url);
}
if (sd != null && sd.hasWebPath())
return sd.getWebPath()+"|"+sd.getName();
if (sd != null && sd.hasWebPath()) {
if (url.contains("|") || hasMultipleVersions(context.fetchResourcesByUrl(StructureDefinition.class, url))) {
return sd.getWebPath()+"|"+sd.getName()+"("+sd.getVersion()+")";
} else {
return sd.getWebPath()+"|"+sd.getName();
}
}
brokenLinkMessage("?pkp-1?", url, false);
return "unknown.html|?pkp-2?";
}

private boolean hasMultipleVersions(List<? extends CanonicalResource> list) {
Set<String> vl = new HashSet<>();
for (CanonicalResource cr : list) {
vl.add(cr.getVersion());
}
return vl.size() > 1;
}

@Override
public String resolveType(String type) {
Expand Down
116 changes: 100 additions & 16 deletions org.hl7.fhir.publisher.core/src/main/java/org/hl7/fhir/igtools/publisher/Publisher.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@
import org.hl7.fhir.r5.model.Parameters;
import org.hl7.fhir.r5.model.PlanDefinition;
import org.hl7.fhir.r5.model.PlanDefinition.PlanDefinitionActionComponent;
import org.hl7.fhir.r5.model.PrimitiveType;
import org.hl7.fhir.r5.model.Property;
import org.hl7.fhir.r5.model.Provenance;
import org.hl7.fhir.r5.model.Provenance.ProvenanceAgentComponent;
Expand Down Expand Up @@ -335,6 +336,9 @@
import org.hl7.fhir.utilities.i18n.JsonLangFileProducer;
import org.hl7.fhir.utilities.i18n.LanguageFileProducer;
import org.hl7.fhir.utilities.i18n.LanguageFileProducer.TranslationUnit;
import org.hl7.fhir.utilities.i18n.LanguageTag;
import org.hl7.fhir.utilities.i18n.subtag.LanguageSubtagRegistry;
import org.hl7.fhir.utilities.i18n.subtag.LanguageSubtagRegistryLoader;
import org.hl7.fhir.utilities.i18n.PoGetTextProducer;
import org.hl7.fhir.utilities.i18n.XLIFFProducer;
import org.hl7.fhir.utilities.json.model.JsonArray;
Expand Down Expand Up @@ -3159,10 +3163,10 @@ else if (vsCache == null) {
txLog = null;
} else {
log("Connect to Terminology Server at "+txServer);
context.connectToTSServer(new TerminologyClientFactory(version), txServer, "fhir/publisher", txLog);
context.connectToTSServer(new TerminologyClientFactory(version), txServer, "fhir/publisher", txLog, true);
}
} else {
context.connectToTSServer(new TerminologyClientFactory(version), webTxServer.getAddress(), "fhir/publisher", txLog);
context.connectToTSServer(new TerminologyClientFactory(version), webTxServer.getAddress(), "fhir/publisher", txLog, true);
}

loadPubPack();
Expand Down Expand Up @@ -3201,7 +3205,7 @@ else if (vsCache == null) {
}
}

inspector = new HTMLInspector(outputDir, specMaps, linkSpecMaps, this, igpkp.getCanonical(), sourceIg.getPackageId(), trackedFragments, fileList, module);
inspector = new HTMLInspector(outputDir, specMaps, linkSpecMaps, this, igpkp.getCanonical(), sourceIg.getPackageId(), trackedFragments, fileList, module, mode == IGBuildMode.AUTOBUILD || mode == IGBuildMode.WEBSERVER);
inspector.getManual().add("full-ig.zip");
if (historyPage != null) {
inspector.getManual().add(historyPage);
Expand Down Expand Up @@ -4310,9 +4314,17 @@ private boolean load() throws Exception {
needToBuild = loadBundles(needToBuild, igf);
needToBuild = loadTranslationSupplements(needToBuild, igf);
int i = 0;
Set<String> resLinks = new HashSet<>();
for (ImplementationGuideDefinitionResourceComponent res : publishedIg.getDefinition().getResource()) {
if (!res.hasReference())
if (!res.hasReference()) {
throw new Exception("Missing source reference on a resource in the IG with the name '"+res.getName()+"' (index = "+i+")");
} else if (!res.getReference().hasReference()) {
throw new Exception("Missing source reference.reference on a resource in the IG with the name '"+res.getName()+"' (index = "+i+")");
} else if (resLinks.contains(res.getReference().getReference())) {
throw new Exception("Duplicate source reference '"+res.getReference().getReference()+"' on a resource in the IG with the name '"+res.getName()+"' (index = "+i+")");
} else {
resLinks.add(res.getReference().getReference());
}
i++;
FetchedFile f = null;
if (!bndIds.contains(res.getReference().getReference()) && !res.hasUserData("loaded.resource")) {
Expand Down Expand Up @@ -8531,6 +8543,7 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
item.add("url", sd.getUrl());
item.add("name", sd.getName());
item.add("title", sd.present());
addTranslationsToJson(item, "title-translations", sd.getTitleElement(), false);
item.add("path", sd.getWebPath());
if (sd.hasKind()) {
item.add("kind", sd.getKind().toCode());
Expand All @@ -8540,6 +8553,7 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
StructureDefinition base = sd.hasBaseDefinition() ? context.fetchResource(StructureDefinition.class, sd.getBaseDefinition()) : null;
if (base != null) {
item.add("basename", base.getName());
addTranslationsToJson(item, "basename-translations", base.getNameElement(), false);
item.add("basepath", Utilities.escapeXml(base.getWebPath()));
} else if ("http://hl7.org/fhir/StructureDefinition/Base".equals(sd.getBaseDefinition())) {
item.add("basename", "Base");
Expand All @@ -8556,8 +8570,11 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
item.add("derivation", sd.getDerivation().toCode());
}
item.add("publisher", sd.getPublisher());
addTranslationsToJson(item, "publisher-translations", sd.getPublisherElement(), false);
item.add("copyright", sd.getCopyright());
addTranslationsToJson(item, "copyright-translations", sd.getCopyrightElement(), false);
item.add("description", ProfileUtilities.processRelativeUrls(sd.getDescription(), "", igpkp.specPath(), context.getResourceNames(), specMaps.get(0).listTargets(), pageTargets(), false));
addTranslationsToJson(item, "description-translations", sd.getDescriptionElement(), true);

if (sd.hasContext()) {
JsonArray contexts = new JsonArray();
Expand Down Expand Up @@ -8643,12 +8660,16 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
item.add("index", i);
item.add("url", q.getUrl());
item.add("name", q.getName());
addTranslationsToJson(item, "name-translations", q.getNameElement(), false);
item.add("path", q.getWebPath());
item.add("status", q.getStatus().toCode());
item.add("date", q.getDate().toString());
item.add("publisher", q.getPublisher());
addTranslationsToJson(item, "publisher-translations", q.getPublisherElement(), false);
item.add("copyright", q.getCopyright());
addTranslationsToJson(item, "copyright-translations", q.getCopyrightElement(), false);
item.add("description", ProfileUtilities.processRelativeUrls(q.getDescription(), "", igpkp.specPath(), context.getResourceNames(), specMaps.get(0).listTargets(), pageTargets(), false));
addTranslationsToJson(item, "description-translations", q.getDescriptionElement(), true);

i++;
}
Expand Down Expand Up @@ -8696,7 +8717,7 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
contained.add(jo);
jo.add("type", crd.getType());
jo.add("id", crd.getId());
jo.add("title", crd.getType());
jo.add("title", crd.getTitle());
jo.add("description", ProfileUtilities.processRelativeUrls(crd.getDescription(), "", igpkp.specPath(), context.getResourceNames(), specMaps.get(0).listTargets(), pageTargets(), false));

JsonObject citem = new JsonObject();
Expand All @@ -8722,7 +8743,25 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {

json = org.hl7.fhir.utilities.json.parser.JsonParser.compose(data, true);
TextFile.stringToFile(json, Utilities.path(tempDir, "_data", "resources.json"));

data = new JsonObject();
if (sourceIg.hasLanguage()) {
data.add("ig", sourceIg.getLanguage());
}
data.add("hasTranslations", hasTranslations);
data.add("defLang", defaultTranslationLang);
JsonObject langs = new JsonObject();
data.add("langs", langs);
for (String s : translationLangs) {
JsonObject lu = new JsonObject();
langs.add(s, lu);
lu.add("en", getLangDesc(s));
// todo: translation to other languages
}

json = org.hl7.fhir.utilities.json.parser.JsonParser.compose(data, true);
TextFile.stringToFile(json, Utilities.path(tempDir, "_data", "languages.json"));

if (publishedIg.getDefinition().hasPage()) {
JsonObject pages = new JsonObject();
addPageData(pages, publishedIg.getDefinition().getPage(), "0", "");
Expand Down Expand Up @@ -8750,12 +8789,41 @@ private void generateSummaryOutputs(DBBuilder db) throws Exception {
generateViewDefinitions(db);
}

private String getLangDesc(String s) throws IOException {
if (registry == null) {
registry = new LanguageSubtagRegistry();
LanguageSubtagRegistryLoader loader = new LanguageSubtagRegistryLoader(registry);
loader.loadFromDefaultResource();
}
LanguageTag tag = new LanguageTag(registry, s);
return tag.present();
}

private void addTranslationsToJson(JsonObject item, String name, PrimitiveType<?> element, boolean processDesc) {
JsonArray jt = null;
for (Entry<String, String> t : ToolingExtensions.getLanguageTranslations(element).entrySet()) {
if (jt == null) {
jt = new JsonArray();
item.add(name, jt);
}
JsonObject ji = new JsonObject();
jt.add(ji);
ji.add("lang", t.getKey());
if (processDesc) {
ji.add("content",
ProfileUtilities.processRelativeUrls(t.getValue(), "", igpkp.specPath(), context.getResourceNames(), specMaps.get(0).listTargets(), pageTargets(), false));
} else {
ji.add("content", t.getValue());
}
}
}

private void generateViewDefinitions(DBBuilder db) {
for (String vdn : viewDefinitions) {
logMessage("Generate View "+vdn);
Runner runner = new Runner();
try {
runner.setContext(context);;
runner.setContext(context);
PublisherProvider pprov = new PublisherProvider(context, npmList, fileList, igpkp.getCanonical());
runner.setProvider(pprov);
runner.setStorage(new StorageSqlite3(db.getConnection()));
Expand Down Expand Up @@ -10422,6 +10490,8 @@ private String processRefTag(DBBuilder db, String src, FetchedFile f) {
private int sqlIndex = 0;

private boolean isSushi;

private LanguageSubtagRegistry registry;
private String processSQlCommand(DBBuilder db, String src, FetchedFile f) throws FHIRException, IOException {
String output = db == null ? "<span style=\"color: maroon\">No SQL this build</span>" : db.processSQL(src);
int i = sqlIndex++;
Expand Down Expand Up @@ -10855,46 +10925,60 @@ private Map<String, String> makeVars(FetchedResource r) {
private byte[] saveNativeResourceOutputs(FetchedFile f, FetchedResource r) throws FHIRException, IOException {
ByteArrayOutputStream bsj = new ByteArrayOutputStream();
org.hl7.fhir.r5.elementmodel.JsonParser jp = new org.hl7.fhir.r5.elementmodel.JsonParser(context);
jp.compose(r.getElement(), bsj, OutputStyle.NORMAL, igpkp.getCanonical());
npm.addFile(isExample(f,r ) ? Category.EXAMPLE : Category.RESOURCE, r.getElement().fhirType()+"-"+r.getId()+".json", bsj.toByteArray());
Element element = r.getElement();
jp.compose(element, bsj, OutputStyle.NORMAL, igpkp.getCanonical());
npm.addFile(isExample(f,r ) ? Category.EXAMPLE : Category.RESOURCE, element.fhirType()+"-"+r.getId()+".json", bsj.toByteArray());
String path = Utilities.path(tempDir, "_includes", r.fhirType()+"-"+r.getId()+".json");
TextFile.bytesToFile(bsj.toByteArray(), path);
String pathEsc = Utilities.path(tempDir, "_includes", r.fhirType()+"-"+r.getId()+".escaped.json");
XmlEscaper.convert(path, pathEsc);

saveNativeResourceOutputFormats(f, r, element, "");
for (String lang : translationLangs) {
Element e = (Element) element.copy();
if (LanguageUtils.switchLanguage(e, lang)) {
saveNativeResourceOutputFormats(f, r, e, "-"+lang);
}
}

return bsj.toByteArray();
}

private void saveNativeResourceOutputFormats(FetchedFile f, FetchedResource r, Element element, String suffix) throws IOException, FileNotFoundException {
String path;
if (igpkp.wantGen(r, "xml") || forHL7orFHIR()) {
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+".xml");
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+suffix+".xml");
f.getOutputNames().add(path);
FileOutputStream stream = new FileOutputStream(path);
org.hl7.fhir.r5.elementmodel.XmlParser xp = new org.hl7.fhir.r5.elementmodel.XmlParser(context);
if (suppressId(f, r)) {
xp.setIdPolicy(IdRenderingPolicy.NotRoot);
}
xp.compose(r.getElement(), stream, OutputStyle.PRETTY, igpkp.getCanonical());
xp.compose(element, stream, OutputStyle.PRETTY, igpkp.getCanonical());
stream.close();
}
if (igpkp.wantGen(r, "json") || forHL7orFHIR()) {
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+".json");
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+suffix+".json");
f.getOutputNames().add(path);
FileOutputStream stream = new FileOutputStream(path);
org.hl7.fhir.r5.elementmodel.JsonParser jp = new org.hl7.fhir.r5.elementmodel.JsonParser(context);
if (suppressId(f, r)) {
jp.setIdPolicy(IdRenderingPolicy.NotRoot);
}
jp.compose(r.getElement(), stream, OutputStyle.PRETTY, igpkp.getCanonical());
jp.compose(element, stream, OutputStyle.PRETTY, igpkp.getCanonical());
stream.close();
}
if (igpkp.wantGen(r, "ttl")) {
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+".ttl");
path = Utilities.path(tempDir, r.fhirType()+"-"+r.getId()+suffix+".ttl");
f.getOutputNames().add(path);
FileOutputStream stream = new FileOutputStream(path);
org.hl7.fhir.r5.elementmodel.TurtleParser tp = new org.hl7.fhir.r5.elementmodel.TurtleParser(context);
if (suppressId(f, r)) {
tp.setIdPolicy(IdRenderingPolicy.NotRoot);
}
tp.compose(r.getElement(), stream, OutputStyle.PRETTY, igpkp.getCanonical());
tp.compose(element, stream, OutputStyle.PRETTY, igpkp.getCanonical());
stream.close();
}
return bsj.toByteArray();
}
}

private boolean isExample(FetchedFile f, FetchedResource r) {
Expand Down
Loading
Loading