Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare rama-ua for UA profiles, that can be used for UA Emulation (client-side) #399

Merged
merged 52 commits into from
Mar 2, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
6969d6b
move ua parser and type to ua module of rama-ua
GlenDC Jan 20, 2025
bff7d85
start defining UA profile types and prepare for encoding
GlenDC Jan 21, 2025
1bf044a
wip comments
GlenDC Jan 26, 2025
5eda20f
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 5, 2025
5accf8c
implement ua db + improve types
GlenDC Feb 6, 2025
effd81c
fix QA lints (UA)
GlenDC Feb 6, 2025
1307de1
making ua profiles a single vec for runtime usage
GlenDC Feb 7, 2025
75b4ae9
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 8, 2025
836d1f6
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 9, 2025
6bd8a29
prepare rama-ua for new profile structure
GlenDC Feb 10, 2025
49e7323
impl db more compatible with other UA layers
GlenDC Feb 10, 2025
5c8a124
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 13, 2025
ce35460
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 14, 2025
e7d1380
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 14, 2025
4ea52f0
fix lint error
GlenDC Feb 15, 2025
8a4a8ba
start working towards UA emulation: add UserAgentProvider logic
GlenDC Feb 15, 2025
6f0478b
start to write the UA emulate layer
GlenDC Feb 17, 2025
2b6d306
remove highway from dep tree: no longer used in latest version of ram…
GlenDC Feb 17, 2025
5aa1eff
fix tls import from rama-net in rama-http-backend
GlenDC Feb 17, 2025
9ba8f4e
fix more warnings and support preserving http and/or tls if instructed
GlenDC Feb 17, 2025
b45d6aa
support random UA selection and cleaner select fallbacks in general
GlenDC Feb 18, 2025
6c894f6
fix tests and improve UA (db) profile selection randomness
GlenDC Feb 18, 2025
a5f0c66
support client config based on context in tls connectors
GlenDC Feb 18, 2025
ee7a49c
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 19, 2025
aab34c8
merge http headers with base headers (UA Emulation)
GlenDC Feb 19, 2025
34b3691
minor fixes in merge_http_headers (ua emulation)
GlenDC Feb 19, 2025
5f92b0e
add UA db tests ; get + rnd work as expected
GlenDC Feb 20, 2025
ace252a
prepare fp js script for rama custom header marker
GlenDC Feb 20, 2025
2e713f1
support auto detecting user agent as part of emulation layer
GlenDC Feb 20, 2025
abd91b4
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 20, 2025
c35f2bf
add authorization header to an opt-in only base header
GlenDC Feb 21, 2025
c734b33
add more tests for sub routines of rama-ua emulate service
GlenDC Feb 21, 2025
5625f88
fix test with random profiles
GlenDC Feb 21, 2025
57f98e3
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 21, 2025
59693dc
add final unit test for rama-ua emulate service
GlenDC Feb 22, 2025
939e57a
be aware of more opt-in headers + sec-fetch headers only for secure reqs
GlenDC Feb 22, 2025
b15e014
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 22, 2025
96f74f4
support decompression request if no compresion was requested
GlenDC Feb 22, 2025
50db61e
differentiate between h1 and h2 headers
GlenDC Feb 22, 2025
2cc5908
refactor q-value and cleanup http deps
GlenDC Feb 23, 2025
366913d
improve decompress req detect logic in rama-ua emulate service
GlenDC Feb 23, 2025
ce67607
add initial client hints support
GlenDC Feb 24, 2025
7f0fccb
ensure rama-fp requests _all_ Client-Hints that we are aware of
GlenDC Feb 24, 2025
50facd8
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 25, 2025
4dc6c12
fix client hint header name iterator errors
GlenDC Feb 25, 2025
39fb9c3
add fly config for rama-fp-pg
GlenDC Feb 25, 2025
655df9f
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 26, 2025
a02e474
Merge branch 'main' into feat/issue-20-ua-profiles
GlenDC Feb 27, 2025
252618b
integrate storage logic (for now dummy) into rama-fp
GlenDC Feb 28, 2025
b89860b
fix minor issues with rama-fp storage & script
GlenDC Feb 28, 2025
ecb7d74
integrate actual postgres storage into rama-fp
GlenDC Mar 2, 2025
ef26145
fix fp storage minor issues in code/queries
GlenDC Mar 2, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
211 changes: 207 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ http = "1"
http-body = "1"
http-body-util = "0.1"
http-range-header = "0.4.0"
deadpool-postgres = "0.14.1"
httpdate = "1.0"
boring = "4.9.1"
tokio-boring = "4.9.1"
Expand Down Expand Up @@ -142,6 +143,7 @@ slab = "0.4.2"
indexmap = "2"
rand = "0.9.0"
walkdir = "2.3.2"
tokio-postgres = "0.7.13"
env_logger = "0.11.6"
httparse = "1.8"
smallvec = { version = "1.12", features = ["const_generics", "const_new"] }
Expand Down
4 changes: 2 additions & 2 deletions examples/http_user_agent_classifier.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,8 @@ async fn handle(ctx: Context<()>, _req: Request) -> Result<Response, Infallible>
"kind": ua.info().map(|info| info.kind.to_string()),
"version": ua.info().and_then(|info| info.version),
"platform": ua.platform().map(|p| p.to_string()),
"http_agent": ua.http_agent().to_string(),
"tls_agent": ua.tls_agent().to_string(),
"http_agent": ua.http_agent().as_ref().map(ToString::to_string),
"tls_agent": ua.tls_agent().as_ref().map(ToString::to_string),
}))
.into_response())
}
Loading
Loading