feat: Add Open Graph tag support

[JasonLovesDoggo]

Adds support for OG tags.

supersedes/closes #178
closes #168
closes #168
closes #131
closes #118

NOTES:

• This PR also adds some potentially out of scope stuff (like an air config. If @Xe want's me to remove them, I will :)
• This is still a bit of a WIP, though it is fully functional. It needs a refactor/maybe some more through tests

Checklist:

• Added a description of the changes to the [Unreleased] section of docs/docs/CHANGELOG.md
• Added test cases to the relevant parts of the codebase
• Ran integration tests npm run test:integration (unsupported on Windows, please use WSL)

[JasonLovesDoggo]

Just a note here. The integration tests will not support OG Tags given given the way it's setup. The OGTags rely on the --target + the request's URL's Path. Yet given the ordering of the playwrite tests, the temp server's URL is only avail AFTER libanubis is configured

...
s, err := libanubis.New(libanubis.Options{
		Next:           h,
		Policy:         policy,
		ServeRobotsTXT: true, // Target would be set here
	})
	if err != nil {
		t.Fatalf("can't construct libanubis.Server: %v", err)
	}

	ts := httptest.NewServer(s) // ts URL is here 
	t.Log(ts.URL)
...

Can I leave it that way yet keep/add dedicated OG tests @Xe?

[Xe]

👍 the happy path is there for simple things, manually constructing things is always acceptable when the happy path doesn't work.

[JasonLovesDoggo]

👍 the happy path is there for simple things, manually constructing things is always acceptable when the happy path doesn't work.

Awesome thanks :D

This PR should be ready tomorrow evening

[Xe]

Take all the time you need, this is gonna be awesome to see in action :D

[JasonLovesDoggo]

May have gone overboard with tests but the pain of the playwrite made me do this 😨

[JasonLovesDoggo]

Will fix the rest @ ~4pm EDT unless Xe pushes them first lol

[JasonLovesDoggo]

There should definitely be some limit to the amount of data the fetcher loads from the target.

Testing with a 4GB video file, it seems to load all of it into memory and taking quite a while to even get to the challenge part.

The main branch does not have this issue.

EDIT: by "loading into memory" I mean that Anubis's memory usage is increased by the size of the file for a long time after the request is processed, it may also have something to do with Go's GC, I don't really know.

Hmm, when the target is a raw mp4 file? I've patched this by introducing a check that limits the content length to 16 MiB to hopefully prevent this.

[jaredallard]

hype hype hype

[JasonLovesDoggo]

don't merge yet.. forgot something We're back in business

[JasonLovesDoggo]

There should definitely be some limit to the amount of data the fetcher loads from the target.

Testing with a 4GB video file, it seems to load all of it into memory and taking quite a while to even get to the challenge part.

The main branch does not have this issue.

EDIT: by "loading into memory" I mean that Anubis's memory usage is increased by the size of the file for a long time after the request is processed, it may also have something to do with Go's GC, I don't really know.

Just tested with a 400mb file, ram usage of Anubis was between 3-6mb

Though the chrome tab tanked chal difficulty 6 and reached 4gb...

[Xe]

You want: https://pkg.go.dev/net/http#MaxBytesReader

[SlyEcho]

It looks like the memory issue is resolved, one part of it is, of course, about ignoring video files now 👍

[Xe]

Just opened this for testing. I think it's pretty much good to go! Thanks for sticking to it, this review cycle kinda sucked and we'll try to make sure something like this doesn't happen again.

Either way, very happy that this is gonna be in. This will be one of the headline features of the next release :)