-
Notifications
You must be signed in to change notification settings - Fork 85
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
O3-3512: SDK should handle reading dependencies from content.properties and ensuring they are part of the distro #280
Conversation
d82557e
to
abb9623
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mherman22! Great work here. A few things that need to be fixed, but a very good first pass. Also note the O
in O3
is the letter "O" rather than the number 0
.
maven-plugin/pom.xml
Outdated
<dependency> | ||
<groupId>com.vdurmont</groupId> | ||
<artifactId>semver4j</artifactId> | ||
<version>3.1.0</version> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this be implemented using the java-semver library we already have instead of adding a new dependency?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
True, I've just seen that, lemi use the existing library!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I asked this of the other PR, but could we use the org.semver:semver4j
version. It's more maintained than this one.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done!
* @param distroProperties | ||
* @throws MojoExecutionException | ||
*/ | ||
private void parseContentProperties(File userDir, DistroProperties distroProperties) throws MojoExecutionException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These checks should be implemented for both the build-distro
and setup
commands, so it might make sense to extract the "meat" of them into one of the shared components (or add a new one).
for (String dependency : contentProperties.stringPropertyNames()) { | ||
String versionRange = contentProperties.getProperty(dependency); | ||
String distroVersion = distroProperties.get(dependency); | ||
|
||
if (distroVersion == null) { | ||
String latestVersion = findLatestMatchingVersion(dependency, versionRange); | ||
if (latestVersion == null) { | ||
throw new MojoExecutionException("No matching version found for dependency " + dependency); | ||
} | ||
|
||
distroProperties.add(dependency, latestVersion); | ||
} else { | ||
checkVersionInRange(dependency, versionRange, distroVersion); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not all properties in the content.properties
specify some sort of library. Specifically, I think it's only the keys that start: omod.
, owa.
, and spa.frontendModule.
(plus, I suppose, content.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh! and war
!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ibacher For example, what will be the value for content.
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
content.hiv=1.0.0
or something. If we have a content package that requires another content package.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Assume the default groupId is org.openmrs.content
, but can be anything using the content.hiv.groupId=xxx
pattern).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ooh perfect!
} | ||
|
||
private String findLatestMatchingVersion(String dependency, String versionRange) throws MojoExecutionException { | ||
return distroHelper.findLatestMatchingVersion(dependency, versionRange); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Here we need slightly different handling for findLatestMatchingVersion
for keys that are spa.frontendModule.
and others, since frontendModule versions need to be looked up from an NPM registry rather than a Maven registry.
*/ | ||
private void checkVersionInRange(String contentDependencyKey, String contentDependencyVersionRange, String distroPropertyVersion) throws MojoExecutionException { | ||
Semver semverVersion = new Semver(distroPropertyVersion, Semver.SemverType.NPM); | ||
String[] ranges = contentDependencyVersionRange.split(","); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need to support comma-separated ranges. The default SemVer ranges have enough disjuctivity.
if (range.startsWith("^") || range.startsWith("~")) { | ||
inRange = semverVersion.satisfies(range); | ||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the test here should always be something like semverVersion.satisfies(range)
. Ranges can also be specified as things like 1.x
or 1.10.1 || ^2 || ~4.0 || > 5
.
} | ||
|
||
if (!inRange) { | ||
throw new MojoExecutionException( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Error message should probably also include a way of identifying the content package in question since a distro may have several.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, makes perfect sense
@@ -517,4 +518,11 @@ public Properties getArtifactProperties(Artifact artifact, Server server, String | |||
return properties; | |||
} | |||
|
|||
public String findLatestMatchingVersion(String dependency, String versionRange) throws MojoExecutionException { | |||
if (dependency.startsWith("omod") || dependency.startsWith("owa") || dependency.startsWith("spa.frontend")) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should probably support war
too, no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree
d79d0f4
to
691f901
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Quick remark. I haven't fully reviewed. Also, based on today's conversation, can we switch from the java-semver library to semver4j. Sorry for the whiplash!
|
||
public class NpmVersionHelper { | ||
|
||
public String getLatestReleasedVersionFromNpmRegistry(PackageJson packageJson) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Clever! Could I suggest just using npm dist-tag ls <package>
though? Easier output to parse, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That said, the overall goal here is that the user supplies some semver range and we resolve that to a version which may or may not always be the latest version. Here's what I'd suggest: use the existing NPM helper stuff to run npm pack --dry-run --json <package>@<version>
, e.g., npm pack --dry-run --json '@openmrs/esm-login-app@>=5.0.0-pre.1'
will get you a JSON document that has the actual resolved version, etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mherman22 Any thoughts on this?
maven-plugin/pom.xml
Outdated
<dependency> | ||
<groupId>com.vdurmont</groupId> | ||
<artifactId>semver4j</artifactId> | ||
<version>3.1.0</version> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, I asked this of the other PR, but could we use the org.semver:semver4j
version. It's more maintained than this one.
@@ -78,7 +76,7 @@ public class BuildDistro extends AbstractTask { | |||
|
|||
private static final Logger log = LoggerFactory.getLogger(BuildDistro.class); | |||
|
|||
/** | |||
/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The indentation here looks funny to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you double-check that this is a tab and not spaces?
* currently used in the distribution. | ||
* @throws MojoExecutionException if there is an error reading the content.properties file or if a version conflict is detected. | ||
*/ | ||
public static void parseContentProperties(File userDir, DistroProperties distroProperties) throws MojoExecutionException { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure I quite get the use-case here? The distro.properties should have entries like:
content.hiv=1.0.0-SNAPSHOT
This should correspond to a Zip file stored in a Maven repository and the content.properties
file is in that Zip file. I know there are some bits missing here, but for arguments sake, it should be similar to the code to load org.openmrs:distro-emr-frontend:3.0.0:zip
and read the files.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i am struggling to get what you mean here @ibacher.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah... past Ian doesn't make a lot of sense. I think what I was getting at is that this assumes that the content.properties
file is in the directory with all the other files, but it's likely inside a zip file inside that folder or something.
I also couldn't quite see how things would work if there is more than one content package (which is a use-case we're supporting)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks , however I thought this particular ticket was to basically pick a content.properties that has already been extracted by https://openmrs.atlassian.net/browse/O3-3640 and then do the rest of the stuff?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cc: @ibacher
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise if I'm wrong, feel free to review the latest commit 😄
sdk-commons/pom.xml
Outdated
<dependency> | ||
<groupId>com.vdurmont</groupId> | ||
<artifactId>semver4j</artifactId> | ||
<version>3.1.0</version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only specify the version in the main pom.xml. Also, I don't think the scope
is, strictly speaking necessary.
|
||
public class NpmVersionHelper { | ||
|
||
public String getLatestReleasedVersionFromNpmRegistry(PackageJson packageJson) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mherman22 Any thoughts on this?
if (dependency.startsWith("omod") || dependency.startsWith("war")) { | ||
properties.setProperty(dependency, version); | ||
log.info("Added dependency: {} with version: {}", dependency, version); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should work with spa.frontendModules
and owa
too, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
true!
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream())); | ||
StringBuilder outputBuilder = new StringBuilder(); | ||
String line; | ||
while ((line = reader.readLine()) != null) { | ||
outputBuilder.append(line); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More about the try()
part, but reading with a single array buffer is slightly more efficient.
BufferedReader reader = new BufferedReader(new InputStreamReader(process.getInputStream())); | |
StringBuilder outputBuilder = new StringBuilder(); | |
String line; | |
while ((line = reader.readLine()) != null) { | |
outputBuilder.append(line); | |
} | |
StringBuilder outputBuilder = new StringBuilder(); | |
char[] buffer = new char[4096]; | |
try (InputStream is = process.getInputStream()) { | |
while ((int read = is.read(buf) >= 0) { | |
outputBuilder.append(buf, 0, read); | |
} | |
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks, though im gonna use a Reader
than an InputStream
. It is really a byte vs char war
|
||
int exitCode = process.waitFor(); | ||
if (exitCode != 0) { | ||
throw new RuntimeException("npm pack --dry-run --json command failed with exit code " + exitCode); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably good to also dump the contents of the output builder here.
We may need to think about how to get the content packages Zip. I'm not entirely sure that we store the Zip file anywhere other than in the Maven repo. Maybe we use the Maven utilities to move the content packages to a temporary directory? |
sdk-commons/pom.xml
Outdated
<dependency> | ||
<groupId>org.json</groupId> | ||
<artifactId>json</artifactId> | ||
<version>20230618</version> | ||
</dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Forgot this: we actually have Jackson as a dependency already. Can we convert the code to use that instead of adding a new JSON-handling library please?
/** | ||
* Adds a dependency with the specified version to the properties. | ||
* | ||
* @param dependency The name of the dependency. | ||
* @param version The version of the dependency. | ||
*/ | ||
public void add(String dependency, String version) { | ||
if (StringUtils.isBlank(dependency) || StringUtils.isBlank(version)) { | ||
log.error("Dependency name or version cannot be blank"); | ||
return; | ||
} | ||
|
||
if (dependency.startsWith("omod.") || dependency.startsWith("owa.") || dependency.startsWith("war") | ||
|| dependency.startsWith("spa.frontendModule")) { | ||
properties.setProperty(dependency, version); | ||
log.info("Added dependency: {} with version: {}", dependency, version); | ||
} | ||
} | ||
|
||
public String get(String contentDependencyKey) { | ||
return properties.getProperty(contentDependencyKey); | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is mostly indented with spaces, but this change adds tabs for this section only. It's preferable to be consistent about this.
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More tabs v spaces changes.
@@ -48,14 +50,18 @@ public class DistroHelper { | |||
|
|||
final VersionsHelper versionHelper; | |||
|
|||
NpmVersionHelper npmVersionHelper; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this ever get set?
@@ -48,14 +50,18 @@ public class DistroHelper { | |||
|
|||
final VersionsHelper versionHelper; | |||
|
|||
NpmVersionHelper npmVersionHelper; | |||
|
|||
PackageJson packageJson; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we ever use this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes we do! within the same class at:
public String findLatestMatchingVersion(String dependency, String versionRange) {
if (dependency.startsWith("omod") || dependency.startsWith("owa") || dependency.startsWith("content.") || dependency.startsWith("war.")) {
return versionHelper.getLatestReleasedVersion(new Artifact(dependency, "latest"));
} else if (dependency.startsWith("spa.frontendModule")) {
packageJson.setName(dependency.substring("spa.frontendModules.".length()));
return npmVersionHelper.getResolvedVersionFromNpmRegistry(packageJson, versionRange);
}
throw new IllegalArgumentException("Unsupported dependency type: " + dependency);
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah! Where does it get initialized?
is this the implementation that is expected? i.e we have a
Should be able to filter out the properties starting with Now go into the temp directory and process/unzip the zip files in there by also making sure each zip contains at least one And then we process the For dependencies defined in both For dependencies not defined in My latest commit tries to include all the above, still looking into fetching from maven for now. |
Sounds about right. Now that I think about it, I think we can do something very similar to the
Both the ModuleInstaller and the DistroHelper have similar code paths that download Maven artifacts and copy them to appropriate locations. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice work! A few more pointers here, but this is really shaping up!
String[] contentPackages = getContentPackages(distroProperties); | ||
|
||
for (String contentPackage : contentPackages) { | ||
String version = distroProperties.get(contentPackage); | ||
File zipFile = fetchZipFromMavenRepo(contentPackage, version, tempDirectory); | ||
if (zipFile != null) { | ||
processZipFile(zipFile, distroProperties); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't it be more efficient to just do this:
String[] contentPackages = getContentPackages(distroProperties); | |
for (String contentPackage : contentPackages) { | |
String version = distroProperties.get(contentPackage); | |
File zipFile = fetchZipFromMavenRepo(contentPackage, version, tempDirectory); | |
if (zipFile != null) { | |
processZipFile(zipFile, distroProperties); | |
} | |
for (String key : distroProperties.getAllKeys()) { | |
if (!key.startsWith(CONTENT_PREFIX)) { | |
continue; | |
} |
Instead of basically iterating things twice? Also, it might make sense to add getContentArtifacts()
to BaseSdkProperties
that matches, e.g., getModuleArtifacts()
?
|
||
private static File fetchZipFromMavenRepo(String contentPackage, String version, File tempDirectory) throws MojoExecutionException { | ||
// Base URL for the OpenMRS JFrog repository | ||
String repositoryUrl = "https://openmrs.jfrog.io/artifactory/public/"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe refactor DistroHelper
and ModuleInstaller
into a MavenHelper
to do this? We don't want to assume everything is available in the JFrog repo (there shouldn't be any references to it in the SDK; use mavenrepo.openmrs.org
in preference to it)
private static File fetchZipFromMavenRepo(String contentPackage, String version, File tempDirectory) throws MojoExecutionException { | ||
// Base URL for the OpenMRS JFrog repository | ||
String repositoryUrl = "https://openmrs.jfrog.io/artifactory/public/"; | ||
String groupId = "org.openmrs.content"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a way to override groupIds that needs to be supported here too. IIRC the syntax is something like:
omod.event=1.0.0
omod.event.groupId=org.openmrs
So we'd need to support something like:
content.hiv=1.0.0
content.hiv.groupId=com.ozone-his
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code I'm thinking of is here.
00bd1ea
to
dc74eb4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I moved stuff to DistroHelper because of the so many changes i would have to make to get rid of the NullPointerException
i was encountering.
@@ -13,3 +13,4 @@ spa.frontendModules.@openmrs/esm-login-app=3.3.1 | |||
spa.frontendModules.@openmrs/esm-patient-chart-app=3.1.0 | |||
spa.apiUrl=notopenmrs | |||
spa.configUrls=foo | |||
content.hiv=1.0.0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @mherman22!
@@ -78,7 +76,7 @@ public class BuildDistro extends AbstractTask { | |||
|
|||
private static final Logger log = LoggerFactory.getLogger(BuildDistro.class); | |||
|
|||
/** | |||
/** |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you double-check that this is a tab and not spaces?
* @param distroProperties The distro properties file. | ||
* @return The groupId to use for this dependency. | ||
*/ | ||
private static String checkIfOverwritten(String dependencyKey, DistroProperties distroProperties) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason we can't use the default checkIfOverwritten()
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated!
…es and ensuring they are part of the distro
@@ -167,7 +168,7 @@ protected String getArtifactType(String key){ | |||
} | |||
|
|||
protected String checkIfOverwritten(String key, String param) { | |||
String newKey = key + "." + param; | |||
String newKey = key + "." + param; //content.hiv |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably don't need this long-term.
@@ -189,6 +190,8 @@ protected String checkIfOverwritten(String key, String param) { | |||
case TYPE_DISTRO: | |||
case TYPE_CONFIG: | |||
return properties.getProperty(PROPERTY_DISTRO_GROUP_ID, Artifact.GROUP_DISTRO); | |||
case TYPE_CONTENT: | |||
return properties.getProperty(param + ".groupId",Artifact.GROUP_CONTENT); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we get this to a closer parallel to TYPE_OMOD
? Here we should only need to handle the default value.
import org.xml.sax.SAXException; | ||
|
||
public class PropertiesUtils { | ||
public class PropertiesUtils { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we remove the changes from this file, since they are now all not substantive changes?
It looks like the CI build is failing with a NPE here: https://github.com/openmrs/openmrs-sdk/pull/280/files#diff-c64d06786c01f8050b86ec442ebad27e09a108f0f546875e26dc55a3ce57a334R643 |
Looking into it |
Description of what I changed
Issue I worked on
see https://openmrs.atlassian.net/browse/O3-3512
Checklist: I completed these to help reviewers :)
My IDE is configured to follow the code style of this project.
No? Unsure? -> configure your IDE, format the code and add the changes with
git add . && git commit --amend
I have added tests to cover my changes. (If you refactored
existing code that was well tested you do not have to add tests)
No? -> write tests and add them to this commit
git add . && git commit --amend
I ran
mvn clean install
right before creating this pull request andadded all formatting changes to my commit.
No? -> execute the above command
All new and existing tests passed.
No? -> figure out why and add the fix to your commit. It is your responsibility to make sure your code works.
My pull request is based on the latest changes of the master branch.
No? Unsure? -> execute command
git pull --rebase upstream master