fix: optimise GitLab commit history scraper #1374

cmeessen · 2025-01-24T12:56:23Z

Fixes #777

Changes proposed in this pull request:

optimise gitlab scrapers so that they do not scrape the entire commit history every time they run
scraper uses commit_history_scraped_at to start checking for new commits
the commit history could become incorrect if
- there are commits between finishing scraping and writing the timestamp to the database
- if the entire commit history was rewritten
added delete buttons to reset scraped metadata (commit history, programming languages, repository statistics)

How to test:

docker compose down --volumes && docker compose build --parallel && docker compose up

login and create a new software that uses a gitlab repository url, e.g.

https://codebase.helmholtz.cloud/research-software-directory/RSD-as-a-service

publish the software entry

start the commit scrapers

docker compose exec scrapers /opt/java/openjdk/bin/java -cp /usr/myjava/scrapers.jar nl.esciencecenter.rsd.scraper.git.MainCommits

the first time the scraping will take a while until it is finished
rerun the commit scrapers, this time the scraping should stop

PR Checklist:

Increase version numbers in docker-compose.yml
Link to a GitHub issue
Update documentation
Tests

sonarqubecloud · 2025-02-04T11:23:57Z

Quality Gate passed for 'scrapers'

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
2.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-02-04T11:25:29Z

Quality Gate passed for 'rsd-frontend'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cmeessen · 2025-03-12T09:24:11Z

I rebased the branch to the current status of the main branch

ewan-escience

Thanks for the pull request! I have some comments:

The delete button for the commit history doesn't clear the commit_history_scraped_at field, leading to an incomplete commit history on the next harvest.

Furthermore, I have some comments on the code.

I didn't check the frontend code.

frontend/components/software/edit/services/PackageManagerServices.tsx

ewan-escience · 2025-03-12T13:05:53Z

scrapers/src/main/java/nl/esciencecenter/rsd/scraper/git/BasicRepositoryDataWithHistory.java

+
+import java.util.UUID;
+
+public record BasicRepositoryDataWithHistory(UUID software, String url, String commitHistoryScrapedAt, CommitsPerWeek commitsPerWeek) {


I would prefer a specialised type for commitHistoryScrapedAt, probably ZonedDateTime. I see this in other places as well, like GitlabScraper. Or do you need strings to make it work with existing code? If so, this can also be changed later.

ewan-escience · 2025-03-12T13:09:45Z

scrapers/src/main/java/nl/esciencecenter/rsd/scraper/git/CommitsPerWeek.java

+		return new TreeMap<>(data);
+	}
+
+	public void setData(SortedMap<Instant, Long> data) {


I'm against this method, as the data provided in the parameter might not be consistent with what this class does.

If you really need this (do you?), you should iterate over all entries in the data map and add them with the respective addCommits method.

ewan-escience · 2025-03-12T13:11:40Z

scrapers/src/main/java/nl/esciencecenter/rsd/scraper/git/CommitsPerWeek.java

 	static final Gson gson = new GsonBuilder()
 			.enableComplexMapKeySerialization()
 			.registerTypeAdapter(Instant.class, (JsonSerializer<Instant>) (src, typeOfSrc, context) -> new JsonPrimitive(src.getEpochSecond()))
 			.create();

+	public SortedMap<Instant, Long> getData() {


Do you really need this other than for testing? If not, it should be package private (i.e. no visibility modifier), but I'm open for arguments for making this public. :)

dmijatovic

@paulastock Nice work for the first PR!

I have some suggestions concerning the frontend code. I would appreciate if you can join one of our standups to discuss these suggestions. I have my suggestion in code and could push additional commit with my suggestion if you like.

frontend/components/software/edit/services/config.ts

frontend/components/software/edit/services/ServiceInfoListItem.tsx

frontend/components/software/edit/services/apiSoftwareServices.tsx

frontend/components/software/edit/services/SoftwareRepoServices.tsx

dmijatovic

Please, double check my frontend suggestions.

fix: deselecting top level category in dialog

…on ordering of endpoint

fix: sonar improvement suggestions chore: remove communties from default settings.json

If commit_history was scraped before, start scraping from the last timestamp stored in commit_history to reduce API calls.

sonarqubecloud · 2025-03-17T12:01:29Z

Quality Gate passed for 'rsd-database'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-03-17T12:01:38Z

Quality Gate passed for 'scrapers'

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
2.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2025-03-17T12:03:17Z

Quality Gate passed for 'rsd-frontend'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

cmeessen · 2025-03-17T12:09:48Z

Closing this one due to issues with the history. Moving to #1432.

cmeessen force-pushed the 777-optimise-gitlab-scraper branch 2 times, most recently from 1649d23 to 8c84e4c Compare January 24, 2025 15:01

paulastock force-pushed the 777-optimise-gitlab-scraper branch 5 times, most recently from dc11bb5 to eb732f4 Compare March 12, 2025 08:31

cmeessen marked this pull request as ready for review March 12, 2025 08:48

cmeessen requested review from jmaassen, dmijatovic and ewan-escience March 12, 2025 08:49

cmeessen force-pushed the 777-optimise-gitlab-scraper branch from ff1b5aa to 7ad0ec3 Compare March 12, 2025 09:23

ewan-escience requested changes Mar 12, 2025

View reviewed changes

paulastock force-pushed the 777-optimise-gitlab-scraper branch 2 times, most recently from 77f6105 to 8faee73 Compare March 13, 2025 10:51

dmijatovic requested changes Mar 13, 2025

View reviewed changes

dmijatovic self-requested a review March 14, 2025 11:06

dmijatovic approved these changes Mar 14, 2025

View reviewed changes

cmeessen force-pushed the 777-optimise-gitlab-scraper branch 2 times, most recently from 7963756 to 1df7834 Compare March 17, 2025 11:59

dmijatovic and others added 7 commits March 17, 2025 13:00

WIP: filter lint for categories

566263a

feat: add status to expanded category paths

30c53e4

feat: filter out custom categories with rejected status

bcdc875

feat: category filter links

3b6915f

fix: deselecting top level category in dialog

feat: add status to expanded category paths

dfe6811

test: improve checkRPC_categoryPathsBySoftwareExpanded to not depend …

043115c

…on ordering of endpoint

fix: ensure complete category tree is included in the filters

a0d4d3c

fix: sonar improvement suggestions chore: remove communties from default settings.json

dmijatovic and others added 6 commits March 17, 2025 13:00

feat: add filter links to category highlight items

21e0d5a

fix: include complete category tree (parents) in the filters

41b9c0d

fix: optimise GitLab commit history scraper

7a59483

If commit_history was scraped before, start scraping from the last timestamp stored in commit_history to reduce API calls.

feat: add delete button for background services

c11c166

doc: update background service documentation

d850414

refactor: suggestions for frontend changes by Dusan

876f633

cmeessen force-pushed the 777-optimise-gitlab-scraper branch from 1df7834 to 876f633 Compare March 17, 2025 12:00

cmeessen mentioned this pull request Mar 17, 2025

777 optimise gitlab scraper new #1432

Merged

4 tasks

cmeessen closed this Mar 17, 2025


		import java.util.UUID;

		public record BasicRepositoryDataWithHistory(UUID software, String url, String commitHistoryScrapedAt, CommitsPerWeek commitsPerWeek) {

fix: optimise GitLab commit history scraper #1374

fix: optimise GitLab commit history scraper #1374

Uh oh!

Conversation

cmeessen commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 4, 2025

Quality Gate passed for 'scrapers'

Uh oh!

sonarqubecloud bot commented Feb 4, 2025

Quality Gate passed for 'rsd-frontend'

Uh oh!

cmeessen commented Mar 12, 2025

Uh oh!

ewan-escience left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ewan-escience Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

ewan-escience Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

ewan-escience Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

dmijatovic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dmijatovic left a comment

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Mar 17, 2025

Quality Gate passed for 'rsd-database'

Uh oh!

sonarqubecloud bot commented Mar 17, 2025

Quality Gate passed for 'scrapers'

Uh oh!

sonarqubecloud bot commented Mar 17, 2025

Quality Gate passed for 'rsd-frontend'

Uh oh!

cmeessen commented Mar 17, 2025

Uh oh!

Uh oh!

cmeessen commented Jan 24, 2025 •

edited

Loading

dmijatovic left a comment •

edited

Loading