prepare rama-ua for UA profiles, that can be used for UA Emulation (c…

…lient-side) (#399)

- [x] Closes #20
- [x] Closes #52: respect client http request by default even when emulating 
- [x] Closes #66: proxy web client: make basic headers overwrite opt-in
- [x] Closes #67: support opt-in client hints (http headers)

To do in a follow up PR:

-  #62: create small script to generate profile rust code based on DB

---

* move ua parser and type to ua module of rama-ua

as to make room for also adding the profile types

* start defining UA profile types and prepare for encoding

* wip comments

* implement ua db + improve types

next steps:

- add unit tests
- refactor rama-fp to make use of this
- add storage trait in rama-fp
- store data in postgres for rama-fp (@fly.io)
- expose data in website
- test & iterate
- ship

embedding profiles automatically via a script
into rama-ua we'll do in a follow-up PR

* fix QA lints (UA)

* making ua profiles a single vec for runtime usage

* prepare rama-ua for new profile structure

next up... db.. again again

* impl db more compatible with other UA layers

* fix lint error

* start working towards UA emulation: add UserAgentProvider logic

this provider will be used by the layer logic to select a UA based on the given ctx
which also allows someone else to implent their own logic

* start to write the UA emulate layer

* remove highway from dep tree: no longer used in latest version of rama-ua

* fix tls import from rama-net in rama-http-backend

* fix more warnings and support preserving http and/or tls if instructed

* support random UA selection and cleaner select fallbacks in general

* fix tests and improve UA (db) profile selection randomness

* support client config based on context in tls connectors

* merge http headers with base headers (UA Emulation)

* minor fixes in merge_http_headers (ua emulation)

* add UA db tests ; get + rnd work as expected

* prepare fp js script for rama custom header marker

* support auto detecting user agent as part of emulation layer

can be useful so one doesn't require two layers
if there is no use of the user agent outside of emulating

* add authorization header to an opt-in only base header

* add more tests for sub routines of rama-ua emulate service

so far all seems to be without any bugs found so far,
still good to have these tests to prevent mistakes in future

* fix test with random profiles

* add final unit test for rama-ua emulate service

fixed bug regarding preserve UA opt-in flag, thx unit test

* be aware of more opt-in headers + sec-fetch headers only for secure reqs

* support decompression request if no compresion was requested

and make sure that we respect accept encoding value,
as to make that kind of difficulty easier to handle

* differentiate between h1 and h2 headers

turns out to be different as well

* refactor q-value and cleanup http deps

unrelated from the PR but seems like I kinda got into a refactor dfrift,
whoops

* improve decompress req detect logic in rama-ua emulate service

decompression will no longer happen in case
the client req had a compatible encoding with the response content-encoding

* add initial client hints support

* ensure rama-fp requests _all_ Client-Hints that we are aware of

* fix client hint header name iterator errors

header names => lowercase

* add fly config for rama-fp-pg

* integrate storage logic (for now dummy) into rama-fp

and adapt python+js code to it

* fix minor issues with rama-fp storage & script

next steps: integrate PG db

* integrate actual postgres storage into rama-fp

* fix fp storage minor issues in code/queries

Cargo.toml

@@ -57,6 +57,7 @@ http = "1"
 http-body = "1"
 http-body-util = "0.1"
 http-range-header = "0.4.0"
+deadpool-postgres = "0.14.1"
 httpdate = "1.0"
 boring = "4.9.1"
 tokio-boring = "4.9.1"
@@ -142,6 +143,7 @@ slab = "0.4.2"
 indexmap = "2"
 rand = "0.9.0"
 walkdir = "2.3.2"
+tokio-postgres = "0.7.13"
 env_logger = "0.11.6"
 httparse = "1.8"
 smallvec = { version = "1.12", features = ["const_generics", "const_new"] }

examples/http_user_agent_classifier.rs

@@ -47,8 +47,8 @@ async fn handle(ctx: Context<()>, _req: Request) -> Result<Response, Infallible>
         "kind": ua.info().map(|info| info.kind.to_string()),
         "version": ua.info().and_then(|info| info.version),
         "platform": ua.platform().map(|p| p.to_string()),
-        "http_agent": ua.http_agent().to_string(),
-        "tls_agent": ua.tls_agent().to_string(),
+        "http_agent": ua.http_agent().as_ref().map(ToString::to_string),
+        "tls_agent": ua.tls_agent().as_ref().map(ToString::to_string),
     }))
     .into_response())
 }