ERDDAP: Improve experience managing datasets

[ChrisJohnNOAA]

Project Description

Provide an improved experience for adding new datasets, as well as tools and notifications to help maintain the datasets. The potential improvements include updating the XML parser to a modern implementation with features like XInclude (ERDDAP/erddap#112), improving the generatedatasets helper (https://github.com/ERDDAP/erddap/blob/main/WEB-INF/classes/gov/noaa/pfel/erddap/GenerateDatasetsXml.java), and more. Monitoring the dataset definitions and notifying admins of problems that arise could also be implemented.

Expected Outcomes

Improvements to the dataset management process for ERDDAP admins.

Skills required

Java, XML

Mentor(s)

Chris John (@ChrisJohnNOAA), Tylar (@7yl4r )

Expected Project Size

175

What is the difficulty of the project?

Intermediate

[NEGRAH-S]

Hey @ChrisJohnNOAA ,I feel like this project suits my insterest more then the previous one not only that but I feel in this we need to do following

• Update XML Parser
• Improve generatedatasets Helper
• Notification System and lot more
  The project aims to improve the efficiency and effectiveness of dataset management within the ERDDAP platform. I would about it in more details.kindly, provide me with email or link to any communication for same.

[ChrisJohnNOAA]

Hey @ChrisJohnNOAA ,I feel like this project suits my insterest more then the previous one not only that but I feel in this we need to do following

• Update XML Parser
• Improve generatedatasets Helper
• Notification System and lot more
  The project aims to improve the efficiency and effectiveness of dataset management within the ERDDAP platform. I would about it in more details.kindly, provide me with email or link to any communication for same.

The best way to get started would be to set up your dev environment. There are directions in the documentation on this page: https://erddap.github.io/setup.html

After that looking at the issues, in particular those labelled good first issue could allow you to make a smaller change to familiarize yourself with ERDDAP:
https://github.com/ERDDAP/erddap/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22

[7yl4r]

Hi, I administrate a small ERDDAP server for IMaRS (and the SE-US MBON). I would happily serve as a test user (and mentor?) for this project.

Our ERDDAP config is public here and I wrote this GitHub Action to help my team (just me and Dan Otis, really) manage the datasets. We run the server in a docker container living on hardware managed here at the University of South Florida, but we have also used the same config system as part of the MBON Dashboard stack.

[ChrisJohnNOAA]

Hi, I administrate a small ERDDAP server for IMaRS (and the SE-US MBON). I would happily serve as a test user (and mentor?) for this project.

Our ERDDAP config is public here and I wrote this GitHub Action to help my team (just me and Dan Otis, really) manage the datasets. We run the server in a docker container living on hardware managed here at the University of South Florida, but we have also used the same config system as part of the MBON Dashboard stack.

I think your input would be incredibly valuable. I added you as a mentor, though you may still need to accept the official invite to the 2024 program.

[7yl4r]

Hi, I administrate a small ERDDAP server for IMaRS (and the SE-US MBON). I would happily serve as a test user (and mentor?) for this project.
Our ERDDAP config is public here and I wrote this GitHub Action to help my team (just me and Dan Otis, really) manage the datasets. We run the server in a docker container living on hardware managed here at the University of South Florida, but we have also used the same config system as part of the MBON Dashboard stack.

I think your input would be incredibly valuable. I added you as a mentor, though you may still need to accept the official invite to the 2024 program.

done! Thanks!

[NEGRAH-S]

@ChrisJohnNOAA and @7yl4r am working on proposal for this problem statement. It would be great if you both could mention or email here.

[7yl4r]

@NEGRAH-S : Sorry. I don't understand what you are asking for. Proposals should be submitted through the GSoC website. We'll review them once the submissions close.

[NEGRAH-S]

@7yl4r before final submission we have chance of getting it reviewed by mentors by personally reaching out to them.

[mwengren]

@NEGRAH-S @7yl4r @ChrisJohnNOAA At the mentor discretion, it is possible for mentors to review and give feedback on student proposals before the official GSoC application period, starting March 18.

It's up to you as mentors whether or not to do this and if so how you'd like to handle it. Sharing your email address to allow students to send their applications with you privately is recommended, however, to minimize risk of plagiarism if the student accidentally posts a publicly-accessible URL.

[NEGRAH-S]

Thanks @mwengren , exactly what and why i wanted email.

[ChrisJohnNOAA]

My e-mail address is chris.john@noaa.gov

[ayushsingh01042003]

Hello @ChrisJohnNOAA and @7yl4r I am a prefinal year student and have been lurking around erddap for a few days now, I am familiar with Java and XML have used both in projects earlier.

For the past few days i was able to

• Set up the erddap server locally on my linux system
• Have gone through a significant amount of its documentation and practical videos
• Have gone throught the programmers guide from here https://erddap.github.io/setup.html#programmersGuide
• Have been looking into some of good first issues for better understanding.

I think the problem statement for this project and requiremments are very straightforward and fall right under my belt and as I have good amount of time this summer Id love to work on this.

Would appreciate any further steps according to you be it related to - submitting a PR, connecting to the mentors or working on the proposal.

Thanks for your time!

[ChrisJohnNOAA]

@ayushsingh01042003

A great next step would be selecting one of the good first issues (https://github.com/ERDDAP/erddap/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and making a pull request for it. Feel free to select any on the list that interest you. A couple that might be worth a look are:
ERDDAP/erddap#134
ERDDAP/erddap#119

Feel free to ask any questions about your selected issue. This is intended to be a small change to get you more familiar with ERDDAP.

[ayushsingh01042003]

Hello, while working on the issue ERDDAP/erddap#119 I had to work with the datasets.xml file and serving data and I have a few questions about it and how erddap serves data. Is it alright to ask them here?

I tried to go through the documentation for working with datasets xml but I am unable to access the website from yesterday along with the NOAA(.gov) homepage website for erddap.

[ChrisJohnNOAA]

Hello, while working on the issue ERDDAP/erddap#119 I had to work with the datasets.xml file and serving data and I have a few questions about it and how erddap serves data. Is it alright to ask them here?

I tried to go through the documentation for working with datasets xml but I am unable to access the website from yesterday along with the NOAA(.gov) homepage website for erddap.

@ayushsingh01042003

You can ask questions here or on the issue ERDDAP/erddap#119

Much of the documentation is available on the erddap github io pages: https://erddap.github.io/setupDatasetsXml.html

As for the trouble accessing the noaa.gov coastwatch page, that is under maintenance (requiring the power to be off at the facility), and is expected to be down through the end of their work day (5pm pacific time) today. It should be back up after that.

[NEGRAH-S]

@ChrisJohnNOAA ,I have mailed you a proposal for same problem. Kindly, review it whenever you get time.

[ayushsingh01042003]

@ChrisJohnNOAA I too would like to submit a draft of my proposal for review by tomorrow. Would appreciate if you take a look.

[ayushsingh01042003]

Hello, Initially I set up my Erddap environment using the erddap.war folder and realised I cannot make changes to the code here, I have replaced it with the Erddap repository on github inside Tomcat/webapps/ and then ran mvn compile which resulted in a successful build.
mvn package though resulted in the following errors

[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   HashDigestTests.basicTest:21 » Runtime 
ERROR in Test.ensureEqual(Strings) line #1, col #1 '3[end]'!='9[end]':
Specifically, at line #1, col #1:
s1: 327fbb2aa6c6297d4fdd5fdf4b14e289[end]
s2: 96e9b4d974b734874a66f433b276a41a[end]
    ^

[ERROR]   NcHelperTests.testUnlimited:578 » FileNotFound /temp/unlimited.nc (No such file or directory)
[ERROR]   TableTests.testReadOpendapSequenceSpeed:2715 Simple readOpendapSequence took too long (time=2351ms).
[INFO] 
[ERROR] Tests run: 57, Failures: 0, Errors: 3, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  54.853 s
[INFO] Finished at: 2024-03-23T23:33:12+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.2.1:test (default-test) on project ERDDAP: 

I gave it a shot and ran
javac -encoding UTF-8 -cp "classes;../../../lib/servlet-api.jar;lib/*" classes/gov/noaa/pfel/coastwatch/TestAll.java anyway in erddap/WEB-INF which resulted in error messages like

import org.json.JSONObject;
               ^
classes/gov/noaa/pfel/coastwatch/TestAll.java:45: error: package ucar.nc2.constants does not exist
import ucar.nc2.constants.FeatureType;
                         ^
classes/gov/noaa/pfel/coastwatch/TestAll.java:47: error: package ucar.nc2.dt.grid does not exist
import ucar.nc2.dt.grid.GridDataset;

[ChrisJohnNOAA]

@ayushsingh01042003 The tests runnable through maven are a new thing. Among other things, the tests used to have a ton of hard coded paths. I've been trying to fix them as I migrate them, but I missed some. Could you file an issue with those test failures from your maven run to make sure they get fixed?

As for the javac ... TestAll errors, it looks like the dependencies aren't there. Can you check that the WEB-INF/lib directory has a lot of jar files in it? Running "mvn compile" should find and load any missing dependencies. If it doesn't let me know,

[ayushsingh01042003]

@ChrisJohnNOAA Sure I will go ahead and put up an issue for the failing tests in a bit.

• Yes there are a lot of jar files in WEB-INF/lib
• I run mvn compile in the webapps/erddap directory and get a Build Success message.
• And then I run this command inside webapps/erddap/WEB-INF javac -encoding UTF-8 -cp "classes;../../../lib/servlet-api.jar;lib/*" classes/gov/noaa/pfel/coastwatch/TestAll.java.
  This results in hundreds of error messages saying package does not exist.
• Everything I followed was from this link https://erddap.github.io/setup.html#programmersGuide
• I am not much of a maven user so maybe there is a very silly step that I may be missing.

Important: This may not be relevant but earlier I changed the pom.xml file because when I initially used mvn compile I was getting a netcdf dependency error. So I changed the < version>5.5.3< /version> to < version>5.4.1< /version>.
How did I get to know? Because there was a commit message by Bob saying:
Bob Simons, 3 years ago (January 26th, 2021 5:16 AM) switch to netcdf-java 5.4.1,

[ChrisJohnNOAA]

@ayushsingh01042003 What were the netcdf errors you were getting when using 5.5.3? That should be working. It's possible the change to the netcdf version is causing issues for you.

[ayushsingh01042003]

@ChrisJohnNOAA
This is what my terminal looks like on running mvn compile in webapps/erddap when version is 5.5.3

[INFO] Building erddap 2.23-SNAPSHOT
[INFO] --------------------------------[ war ]---------------------------------
Downloading from unidata-all: https://artifacts.unidata.ucar.edu/repository/unidata-all/edu/ucar/netcdfAll/5.5.3/netcdfAll-5.5.3.pom
Downloading from central: https://repo.maven.apache.org/maven2/edu/ucar/netcdfAll/5.5.3/netcdfAll-5.5.3.pom
[WARNING] The POM for edu.ucar:netcdfAll:jar:5.5.3 is missing, no dependency information available
Downloading from unidata-all: https://artifacts.unidata.ucar.edu/repository/unidata-all/edu/ucar/netcdfAll/5.5.3/netcdfAll-5.5.3.jar
Downloading from central: https://repo.maven.apache.org/maven2/edu/ucar/netcdfAll/5.5.3/netcdfAll-5.5.3.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.556 s
[INFO] Finished at: 2024-03-25T21:24:58+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project ERDDAP: Could not resolve dependencies for project gov.noaa.pfel.erddap:ERDDAP:war:2.23-SNAPSHOT: Could not find artifact edu.ucar:netcdfAll:jar:5.5.3 in unidata-all (https://artifacts.unidata.ucar.edu/repository/unidata-all/) -> [Help 1]

(P.S. I raised the issues for the test cases you can take a look #140)

[ChrisJohnNOAA]

@ayushsingh01042003 What version of maven are you running? Can you share your whole pom file? Those messages look different than what I see when I run mvn compile.

[ayushsingh01042003]

Okay maybe I have an old version...

mvn --version
Apache Maven 3.6.3
Maven home: /usr/share/maven
Java version: 17.0.10, vendor: Eclipse Adoptium, runtime: /usr/lib/jvm/jdk-17.0.10+7

The pom file is the same as https://github.com/ERDDAP/erddap/blob/main/pom.xml
pom.txt

[ChrisJohnNOAA]

The most recent version of maven is 3.9.6.

The other possibility is issues on linux machines with how the netcdf dependency is downloaded. You could try manually getting that one dependency.

It might be helpful to see what the commands used in for the development docker script:
https://github.com/ERDDAP/erddap/blob/main/development/Dockerfile

[ayushsingh01042003]

@ChrisJohnNOAA updated maven to 3.9.6 but sadly no luck its the same problem with netcdf and when I hover over the error in pom.xml in my editor just the message Missing artifact edu.ucar:netcdfAll:jar:5.5.3

And I dont really understand manually getting it. Should I do the build with 4.5.1 and then download and replace it with 5.5.3 in WEB-INF/lib maybe something like that?

[ChrisJohnNOAA]

@ayushsingh01042003 maven should be downloading the dependencies and placing them in the WEB-INF/lib directory. I just made a change to make sure the lib folder is populated during the validate step (it was previously in the install step). Please sync the project and try it again.

After syncing you'll need to run mvn validate or mvn compile.

If it's successful there should be over 100 files in the WEB-INF\lib directory.

[ayushsingh01042003]

@ChrisJohnNOAA Yes I did a clean clone and install of Erddap, ran mvn validate and mvn compile both ran successfully without any problems and yes there are over 100 jar files in WEB-INF/lib but again we have the same error saying package does not exist the same as the file I attached Earlier on running javac -encoding UTF-8 -cp "classes;../../../lib/servlet-api.jar;lib/*" classes/gov/noaa/pfel/coastwatch/TestAll.java

This is what I was trying to say that maybe its not looking at the right place for packages and some wrong path or command is the problem. Sorry if it was not very clear.

[ChrisJohnNOAA]

Just to check is there a "json-20230227.jar" in your /Tomcat/webapps/erddap/WEB-INF/lib/ directory?

It's also possible there's something different needed for linux machines. I don't have one available right now to check,

[ayushsingh01042003]

@ChrisJohnNOAA Yep, there is definitely a "json-20230227.jar" in Tomcat/webapps/erddap/WEB-INF/lib/ directory.

If thats the case maybe I can ask the question in the https://groups.google.com/g/erddap ? Maybe someone who has encountered the problem and uses linux can help out.

[ChrisJohnNOAA]

Yeah, you can ask there.

[mwengren]

@ayushsingh01042003 I believe the command you're trying to run:

javac -encoding UTF-8 -cp "classes;../../../lib/servlet-api.jar;lib/*" classes/gov/noaa/pfel/coastwatch/TestAll.java

has an issue with an incorrect classpath somewhere. I think some steps in the Programmer's Guide you referenced need an update. Basically, it's not resolving the correct root directory to find ERDDAP's compiled dependencies.

The Programmer's Guide is written assuming that you are running ERDDAP via Tomcat, and includes paths that reference locations for dependencies like servlet-api.jar in Tomcat.

@ChrisJohnNOAA knows more than I do, but I don't think running TestAll.java when ERDDAP is compiled with Maven is the right approach. It can work, but it is not pretty, at least the last time I tried.

I have another approach I worked on awhile ago to run ERDDAP in development mode via Maven and Jetty. I never submitted a PR for it. If I can update/rebase my code so it's working again, I'll at least push a new branch so you can work off of that if you want even if I don't submit a PR right away. I'll post a link here if I can get it working. That was the easiest way I found to get ERDDAP running in development move at localhost from the various options.

Also, it sounds like you resolved the Maven dependency issues already, but you can also run the maven-antrun-plugin goals individually, for example the download-netcdf goal as defined in pom.xml here as follows if you want:

mvn antrun:run@download-netcdf

[ayushsingh01042003]

@mwengren Thanks please do post the link if you were able to make it work, I shall try to figure the approach I am working on till then if possible.

[ayushsingh01042003]

@ChrisJohnNOAA @mwengren I noticed something else:
The command I was running was
javac -encoding UTF-8 -cp "classes;../../../lib/servlet-api.jar;lib/*" classes/gov/noaa/pfel/coastwatch/TestAll.java
and the command given in the docs was:
javac -encoding UTF-8 -cp classes;../../../lib/servlet-api.jar;lib/* classes/gov/noaa/pfel/coastwatch/TestAll.java

As you see the quotes(" ") were added in the command I was using, I remember doing that because on running the original command this was the error i got,

ayush@pop-os:/usr/local/Tomcat/webapps/erddap/WEB-INF$ javac -encoding UTF-8 -cp classes;../../../lib/servlet-api.jar;lib/* classes/gov/noaa/pfel/coastwatch/TestAll.java
error: no source files
bash: ../../../lib/servlet-api.jar: cannot execute binary file: Exec format error
bash: lib/animal-sniffer-annotations-1.22.jar: Permission denied

[ayushsingh01042003]

Update: Turns out it was the command after all, change both the ; to : and what do you know after everything it was something this simple........

[mwengren]

@ayushsingh01042003 I finished updating my Maven/Jetty based ERDDAP development environment code. This should help you to more easily get a local ERDDAP running at http://localhost:8080/erddap.

The code base is in my ERDDAP fork in this branch: https://github.com/mwengren/erddap/tree/jetty-dev-2024

Please have a look at the instructions here: https://github.com/mwengren/erddap/blob/jetty-dev-2024/development/jetty/erddap_jetty_setup.md

You will need to add my ERDDAP fork as a remote in your local checkout directory (git remote add mwengren https://github.com/mwengren/erddap.git), checkout the jetty-dev-2024 branch, and then follow the steps above.

Please let me know if it works!

I hope to submit this via PR soon, but will need to coordinate with @ChrisJohnNOAA first, and I'm also too busy at the moment to fix any unforeseen issues. That said, I was able to get it running this morning and it's far easier than following the present directions in the Programmer's Guide you're working from.

[ayushsingh01042003]

@mwengren quite literally missed by a minute! I am sorry you had to go though all the work but I really appreciate it! I can try to see if works if you need me to once I get comfortable with Erddap code changes for a while :)

[mwengren]

@ayushsingh01042003 I think it will save you some time when you start making code changes to ERDDAP rather than trying to run it via Tomcat, so I'd encourage you to try it out anyway.

[ayushsingh01042003]

@mwengren Sure, I will give it a shot!

[mwengren]

@ayushsingh01042003 I realized I missed an important step in my ERDDAP Jetty deployment instructions. You need to run mvn compile before mvn war:exploded, it's not executed automatically.

If you're able, please give it another shot and let me know what you think:

https://github.com/mwengren/erddap/blob/jetty-dev-2024/development/jetty/erddap_jetty_setup.md

This should help with the dev/test cycle for working with ERDDAP a lot.

[ayushsingh01042003]

@mwengren I was caught up with something and have not yet tried setting up Erddap with jetty yet. I'l finish it in a bit, try the setup and let you know!

[ayushsingh01042003]

@mwengren I tried running Erddap through jetty on my system, I took me a matter of minutes to get it up and running and I think I understand why you emphasized on how easy it would be because IT REALLY IS! Even erddap seems to be running more smooth using jetty on my machine and all it takes is pressing return to redeploy. Its great!

However on trying to run GenerateDatasestsXml.sh I was running into some trouble

ayush@pop-os:~/Progs/erddap-jetty$ git status
On branch jetty-dev-2024
Your branch is up to date with 'origin/jetty-dev-2024'.

nothing to commit, working tree clean
ayush@pop-os:~/Progs/erddap-jetty$ cd WEB-INF
ayush@pop-os:~/Progs/erddap-jetty/WEB-INF$ bash ./GenerateDatasetsXml.sh
Error: Could not find or load main class gov.noaa.pfel.erddap.GenerateDatasetsXml
Caused by: java.lang.ClassNotFoundException: gov.noaa.pfel.erddap.GenerateDatasetsXml

Weirdly enough its unable to find the class for it? Or am I missing something here, could you have a look?

[mwengren]

@ayushsingh01042003 Great! Glad to hear it's running well for you. I think it will help others pick up and run ERDDAP experimentally much more easily than the current approach. I'll be working on a PR soon.

I have not tested the GenerateDatasetsXml.sh script at all in this configuration, but I'm quite sure it's due to differences in the classpath parameter used here: https://github.com/ERDDAP/erddap/blob/main/WEB-INF/GenerateThreddsXml.sh#L46 not matching that of a Tomcat deployment.

Most likely those .sh scripts would all need to be updated in some way.

Alternatively, it could be the directory where you are running them from - it looks like you're running from the 'source' WEB-INF folder for the Java files, which doesn't include the compiled Java classes. Try running the script from the ./target/ERDDAP-2.23-SNAPSHOT/WEB-INF directory instead.

[mwengren]

When I run it from the directory I suggested, I get a different error, which is definitely classpath-related:

sh GenerateDatasetsXml.sh 
Exception in thread "main" java.lang.NoClassDefFoundError: jakarta/servlet/ServletOutputStream
        at gov.noaa.pfel.erddap.GenerateDatasetsXml.<clinit>(GenerateDatasetsXml.java:45)
Caused by: java.lang.ClassNotFoundException: jakarta.servlet.ServletOutputStream
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
        ... 1 more

This fix for that would be to update the script to reference Jetty's servlet-api.jar, which is a bit of hack to be honest, but it could be done. You'd need to track down the Jetty dependency path in your local Maven repository ($HOME/.m2/repository). Here's a path to it specific to my environment: .m2/repository/jakarta/servlet/jakarta.servlet-api/6.0.0/jakarta.servlet-api-6.0.0.jar.

Hopefully a better fix could be developed at some point.

[ChrisJohnNOAA]

So I believe GenerateDatasetsXml.sh is looking for the servlet-api.jar from Tomcat based on installing ERDDAP in a Tomcat server. Making the script function outside of a Tomcat install would be great.

[ayushsingh01042003]

@mwengren I changed my ./target/ERDDAP-2.23-SNAPSHOT/WEB-INF/GenerateDatasetsXml.sh from

java -cp classes:../../../lib/servlet-api.jar:lib/* -Xms1000M -Xmx1000M gov.noaa.pfel.erddap.GenerateDatasetsXml "$@"
to
java -cp classes:/home/ayush/.m2/repository/jakarta/servlet/jakarta.servlet-api/6.0.0:lib/* -Xms1000M -Xmx1000M gov.noaa.pfel.erddap.GenerateDatasetsXml "$@"

and seem to get the same error as you got even after modifying the script, is the classpath incorrect? is it working fine for you?

[ayushsingh01042003]

Also a question on working with Erddap in tomcat, I've set it up a few times now and everytime after running TestAll when I run git status the following shows up:

On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   images/erddapStart2.css
	deleted:    public/.includeInGit

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	WEB-INF/DoubleCenterGrids.log
	WEB-INF/[specify <unitTestDataDir> in setup.xml]/
	WEB-INF/classes/gov/noaa/pfel/coastwatch/griddata/BlueWhiteRed_Linear_-10_10_8_true.cpt
	WEB-INF/classes/gov/noaa/pfel/coastwatch/griddata/Rainbow_Linear_0_10_5_false.cpt
	WEB-INF/temp/

Untracked files I think I can just add to .gitIgnore but what about the other two changes that show up

	modified:   images/erddapStart2.css
	deleted:    public/.includeInGit

Do I just stash them? ignore them? whats the right approach for this?

[ChrisJohnNOAA]

@mwengren I changed my ./target/ERDDAP-2.23-SNAPSHOT/WEB-INF/GenerateDatasetsXml.sh from

java -cp classes:../../../lib/servlet-api.jar:lib/* -Xms1000M -Xmx1000M gov.noaa.pfel.erddap.GenerateDatasetsXml "$@"
to
java -cp classes:/home/ayush/.m2/repository/jakarta/servlet/jakarta.servlet-api/6.0.0:lib/* -Xms1000M -Xmx1000M gov.noaa.pfel.erddap.GenerateDatasetsXml "$@"

and seem to get the same error as you got even after modifying the script, is the classpath incorrect? is it working fine for you?

You might need to modify the lib/ part of the path, depending on where you are running the command from. I believe that should be pointing to the web-inf/lib/ directory. The classes path should similarly be pointing to web-inf/classes/

As for those file differences after running in tomcat, you can ignore them.

[ayushsingh01042003]

Happy to be accepted in the program and thanks to all the mentors who helped me by answering my questions!

Really looking forward to this summer and putting in my best work!

[ChrisJohnNOAA]

Congratulations! I'm looking forward to working with you this summer.

[ayushsingh01042003]

Hey Chris, I was wondering when we will start conversing about our project and also the means of communication.

[ChrisJohnNOAA]

Hey Ayush, I sent an email about setting up a meeting.

In the meantime, I know you already set up a an ERDDAP environment. There have been a number of changes to testing since then so it might be good for you to sync. I've found there's some tests that fail on other machines, and am working on improving that. Help tracking down those tests is appreciated.

If you have time and interest during the bonding period, fixing issues is always appreciated. A few issues of note:
ERDDAP/erddap#112 is directly related to your project.
ERDDAP/erddap#132 is improving the output of the generate datasets script, though I have not yet tracked down the right link to use
ERDDAP/erddap#139 is an improvement for admins, though not related to the datasets xml.

[ayushsingh01042003]

Thanks for the reply Chris.

Understood I've been following the updates and synced up.
I'll take a look around for any failing test cases and inform you.

Sure. I'll take the time and work on the issues

If you have time and interest during the bonding period, fixing issues is always appreciated. A few issues of note: ERDDAP/erddap#112 is directly related to your project. ERDDAP/erddap#132 is improving the output of the generate datasets script, though I have not yet tracked down the right link to use ERDDAP/erddap#139 is an improvement for admins, though not related to the datasets xml.

Also for any questions in case I am stuck. Is this dicussions good for now?

[ChrisJohnNOAA]

If it's about a particular issue, the issue is the best place for discussion. For other general things email or here is good.

[ayushsingh01042003]

The attached file has all the failing test cases through maven I get on my Linux system.
Most of them are for the class SgtMap and EDStatic which seem of to
be missing?
Unless its just my environment setup.

failingTests.txt

[ChrisJohnNOAA]

Thanks. The NoClassDefFound and ExceptionInInitializer messages are consistent with the errors that happen when EDStatic or webinfParentDirectory (which SgtMap does) hasn't been initialized, but is used by a test class. Technically it needs to happen once per thread running the tests, but in JUnit each file needs to have an @BeforeAll that calls the initialization. When I was mass migrating tests, I missed adding the initializer to a few files that need it (because if a test that has the initialization runs first, the tests are fine). The other thing to note is that once the initialization has failed other tests that relies on EDStatic or webinfParentDirectory may fail as well (even if their file has the initialization).

I have a branch where I'm working on enabling more tests and add a few more initializers: https://github.com/ChrisJohnNOAA/erddap/tree/enable_more_tests