-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ERDDAP: Improve experience managing datasets #45
Comments
Hey @ChrisJohnNOAA ,I feel like this project suits my insterest more then the previous one not only that but I feel in this we need to do following
|
The best way to get started would be to set up your dev environment. There are directions in the documentation on this page: https://erddap.github.io/setup.html After that looking at the issues, in particular those labelled good first issue could allow you to make a smaller change to familiarize yourself with ERDDAP: |
Hi, I administrate a small ERDDAP server for IMaRS (and the SE-US MBON). I would happily serve as a test user (and mentor?) for this project. Our ERDDAP config is public here and I wrote this GitHub Action to help my team (just me and Dan Otis, really) manage the datasets. We run the server in a docker container living on hardware managed here at the University of South Florida, but we have also used the same config system as part of the MBON Dashboard stack. |
I think your input would be incredibly valuable. I added you as a mentor, though you may still need to accept the official invite to the 2024 program. |
done! Thanks! |
@ChrisJohnNOAA and @7yl4r am working on proposal for this problem statement. It would be great if you both could mention or email here. |
@NEGRAH-S : Sorry. I don't understand what you are asking for. Proposals should be submitted through the GSoC website. We'll review them once the submissions close. |
@7yl4r before final submission we have chance of getting it reviewed by mentors by personally reaching out to them. |
@NEGRAH-S @7yl4r @ChrisJohnNOAA At the mentor discretion, it is possible for mentors to review and give feedback on student proposals before the official GSoC application period, starting March 18. It's up to you as mentors whether or not to do this and if so how you'd like to handle it. Sharing your email address to allow students to send their applications with you privately is recommended, however, to minimize risk of plagiarism if the student accidentally posts a publicly-accessible URL. |
Thanks @mwengren , exactly what and why i wanted email. |
My e-mail address is chris.john@noaa.gov |
Hello @ChrisJohnNOAA and @7yl4r I am a prefinal year student and have been lurking around erddap for a few days now, I am familiar with Java and XML have used both in projects earlier. For the past few days i was able to
I think the problem statement for this project and requiremments are very straightforward and fall right under my belt and as I have good amount of time this summer Id love to work on this. Would appreciate any further steps according to you be it related to - submitting a PR, connecting to the mentors or working on the proposal. Thanks for your time! |
A great next step would be selecting one of the good first issues (https://github.com/ERDDAP/erddap/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) and making a pull request for it. Feel free to select any on the list that interest you. A couple that might be worth a look are: Feel free to ask any questions about your selected issue. This is intended to be a small change to get you more familiar with ERDDAP. |
Hello, while working on the issue ERDDAP/erddap#119 I had to work with the datasets.xml file and serving data and I have a few questions about it and how erddap serves data. Is it alright to ask them here? I tried to go through the documentation for working with datasets xml but I am unable to access the website from yesterday along with the NOAA(.gov) homepage website for erddap. |
You can ask questions here or on the issue ERDDAP/erddap#119 Much of the documentation is available on the erddap github io pages: https://erddap.github.io/setupDatasetsXml.html As for the trouble accessing the noaa.gov coastwatch page, that is under maintenance (requiring the power to be off at the facility), and is expected to be down through the end of their work day (5pm pacific time) today. It should be back up after that. |
@ChrisJohnNOAA ,I have mailed you a proposal for same problem. Kindly, review it whenever you get time. |
@ChrisJohnNOAA I too would like to submit a draft of my proposal for review by tomorrow. Would appreciate if you take a look. |
Hello, Initially I set up my Erddap environment using the
I gave it a shot and ran
|
@ayushsingh01042003 The tests runnable through maven are a new thing. Among other things, the tests used to have a ton of hard coded paths. I've been trying to fix them as I migrate them, but I missed some. Could you file an issue with those test failures from your maven run to make sure they get fixed? As for the javac ... TestAll errors, it looks like the dependencies aren't there. Can you check that the WEB-INF/lib directory has a lot of jar files in it? Running "mvn compile" should find and load any missing dependencies. If it doesn't let me know, |
@ChrisJohnNOAA Sure I will go ahead and put up an issue for the failing tests in a bit.
Important: This may not be relevant but earlier I changed the |
@ayushsingh01042003 What were the netcdf errors you were getting when using 5.5.3? That should be working. It's possible the change to the netcdf version is causing issues for you. |
@ChrisJohnNOAA
(P.S. I raised the issues for the test cases you can take a look #140) |
@ayushsingh01042003 What version of maven are you running? Can you share your whole pom file? Those messages look different than what I see when I run mvn compile. |
Okay maybe I have an old version...
The pom file is the same as https://github.com/ERDDAP/erddap/blob/main/pom.xml |
The most recent version of maven is 3.9.6. The other possibility is issues on linux machines with how the netcdf dependency is downloaded. You could try manually getting that one dependency. It might be helpful to see what the commands used in for the development docker script: |
@ChrisJohnNOAA updated maven to 3.9.6 but sadly no luck its the same problem with netcdf and when I hover over the error in And I dont really understand manually getting it. Should I do the build with 4.5.1 and then download and replace it with 5.5.3 in WEB-INF/lib maybe something like that? |
@ayushsingh01042003 maven should be downloading the dependencies and placing them in the WEB-INF/lib directory. I just made a change to make sure the lib folder is populated during the validate step (it was previously in the install step). Please sync the project and try it again. After syncing you'll need to run If it's successful there should be over 100 files in the WEB-INF\lib directory. |
@ChrisJohnNOAA Yes I did a clean clone and install of Erddap, ran This is what I was trying to say that maybe its not looking at the right place for packages and some wrong path or command is the problem. Sorry if it was not very clear. |
Just to check is there a "json-20230227.jar" in your /Tomcat/webapps/erddap/WEB-INF/lib/ directory? It's also possible there's something different needed for linux machines. I don't have one available right now to check, |
@ChrisJohnNOAA Yep, there is definitely a "json-20230227.jar" in Tomcat/webapps/erddap/WEB-INF/lib/ directory. If thats the case maybe I can ask the question in the https://groups.google.com/g/erddap ? Maybe someone who has encountered the problem and uses linux can help out. |
Yeah, you can ask there. |
@ayushsingh01042003 I believe the command you're trying to run:
has an issue with an incorrect classpath somewhere. I think some steps in the Programmer's Guide you referenced need an update. Basically, it's not resolving the correct root directory to find ERDDAP's compiled dependencies. The Programmer's Guide is written assuming that you are running ERDDAP via Tomcat, and includes paths that reference locations for dependencies like @ChrisJohnNOAA knows more than I do, but I don't think running TestAll.java when ERDDAP is compiled with Maven is the right approach. It can work, but it is not pretty, at least the last time I tried. I have another approach I worked on awhile ago to run ERDDAP in development mode via Maven and Jetty. I never submitted a PR for it. If I can update/rebase my code so it's working again, I'll at least push a new branch so you can work off of that if you want even if I don't submit a PR right away. I'll post a link here if I can get it working. That was the easiest way I found to get ERDDAP running in development move at localhost from the various options. Also, it sounds like you resolved the Maven dependency issues already, but you can also run the maven-antrun-plugin goals individually, for example the
|
@mwengren Thanks please do post the link if you were able to make it work, I shall try to figure the approach I am working on till then if possible. |
@ChrisJohnNOAA @mwengren I noticed something else: As you see the quotes(" ") were added in the command I was using, I remember doing that because on running the original command this was the error i got,
|
Update: Turns out it was the command after all, change both the ; to : and what do you know after everything it was something this simple........ |
@ayushsingh01042003 I finished updating my Maven/Jetty based ERDDAP development environment code. This should help you to more easily get a local ERDDAP running at http://localhost:8080/erddap. The code base is in my ERDDAP fork in this branch: https://github.com/mwengren/erddap/tree/jetty-dev-2024 Please have a look at the instructions here: https://github.com/mwengren/erddap/blob/jetty-dev-2024/development/jetty/erddap_jetty_setup.md You will need to add my ERDDAP fork as a remote in your local checkout directory ( Please let me know if it works! I hope to submit this via PR soon, but will need to coordinate with @ChrisJohnNOAA first, and I'm also too busy at the moment to fix any unforeseen issues. That said, I was able to get it running this morning and it's far easier than following the present directions in the Programmer's Guide you're working from. |
@mwengren quite literally missed by a minute! I am sorry you had to go though all the work but I really appreciate it! I can try to see if works if you need me to once I get comfortable with Erddap code changes for a while :) |
@ayushsingh01042003 I think it will save you some time when you start making code changes to ERDDAP rather than trying to run it via Tomcat, so I'd encourage you to try it out anyway. |
@mwengren Sure, I will give it a shot! |
@ayushsingh01042003 I realized I missed an important step in my ERDDAP Jetty deployment instructions. You need to run If you're able, please give it another shot and let me know what you think: https://github.com/mwengren/erddap/blob/jetty-dev-2024/development/jetty/erddap_jetty_setup.md This should help with the dev/test cycle for working with ERDDAP a lot. |
@mwengren I was caught up with something and have not yet tried setting up Erddap with jetty yet. I'l finish it in a bit, try the setup and let you know! |
@mwengren I tried running Erddap through jetty on my system, I took me a matter of minutes to get it up and running and I think I understand why you emphasized on how easy it would be because IT REALLY IS! Even erddap seems to be running more smooth using jetty on my machine and all it takes is pressing However on trying to run
Weirdly enough its unable to find the class for it? Or am I missing something here, could you have a look? |
@ayushsingh01042003 Great! Glad to hear it's running well for you. I think it will help others pick up and run ERDDAP experimentally much more easily than the current approach. I'll be working on a PR soon. I have not tested the Most likely those Alternatively, it could be the directory where you are running them from - it looks like you're running from the 'source' |
When I run it from the directory I suggested, I get a different error, which is definitely classpath-related:
This fix for that would be to update the script to reference Jetty's servlet-api.jar, which is a bit of hack to be honest, but it could be done. You'd need to track down the Jetty dependency path in your local Maven repository ( Hopefully a better fix could be developed at some point. |
So I believe GenerateDatasetsXml.sh is looking for the servlet-api.jar from Tomcat based on installing ERDDAP in a Tomcat server. Making the script function outside of a Tomcat install would be great. |
@mwengren I changed my
and seem to get the same error as you got even after modifying the script, is the classpath incorrect? is it working fine for you? |
Also a question on working with Erddap in tomcat, I've set it up a few times now and everytime after running
Untracked files I think I can just add to
Do I just stash them? ignore them? whats the right approach for this? |
You might need to modify the lib/ part of the path, depending on where you are running the command from. I believe that should be pointing to the web-inf/lib/ directory. The classes path should similarly be pointing to web-inf/classes/ As for those file differences after running in tomcat, you can ignore them. |
Happy to be accepted in the program and thanks to all the mentors who helped me by answering my questions! Really looking forward to this summer and putting in my best work! |
Congratulations! I'm looking forward to working with you this summer. |
Hey Chris, I was wondering when we will start conversing about our project and also the means of communication. |
Hey Ayush, I sent an email about setting up a meeting. In the meantime, I know you already set up a an ERDDAP environment. There have been a number of changes to testing since then so it might be good for you to sync. I've found there's some tests that fail on other machines, and am working on improving that. Help tracking down those tests is appreciated. If you have time and interest during the bonding period, fixing issues is always appreciated. A few issues of note: |
Thanks for the reply Chris. Understood I've been following the updates and synced up. Sure. I'll take the time and work on the issues
Also for any questions in case I am stuck. Is this dicussions good for now? |
If it's about a particular issue, the issue is the best place for discussion. For other general things email or here is good. |
The attached file has all the failing test cases through maven I get on my Linux system. |
Thanks. The NoClassDefFound and ExceptionInInitializer messages are consistent with the errors that happen when EDStatic or webinfParentDirectory (which SgtMap does) hasn't been initialized, but is used by a test class. Technically it needs to happen once per thread running the tests, but in JUnit each file needs to have an @BeforeAll that calls the initialization. When I was mass migrating tests, I missed adding the initializer to a few files that need it (because if a test that has the initialization runs first, the tests are fine). The other thing to note is that once the initialization has failed other tests that relies on EDStatic or webinfParentDirectory may fail as well (even if their file has the initialization). I have a branch where I'm working on enabling more tests and add a few more initializers: https://github.com/ChrisJohnNOAA/erddap/tree/enable_more_tests |
Project Description
Provide an improved experience for adding new datasets, as well as tools and notifications to help maintain the datasets. The potential improvements include updating the XML parser to a modern implementation with features like XInclude (ERDDAP/erddap#112), improving the generatedatasets helper (https://github.com/ERDDAP/erddap/blob/main/WEB-INF/classes/gov/noaa/pfel/erddap/GenerateDatasetsXml.java), and more. Monitoring the dataset definitions and notifying admins of problems that arise could also be implemented.
Expected Outcomes
Improvements to the dataset management process for ERDDAP admins.
Skills required
Java, XML
Mentor(s)
Chris John (@ChrisJohnNOAA), Tylar (@7yl4r )
Expected Project Size
175
What is the difficulty of the project?
Intermediate
The text was updated successfully, but these errors were encountered: