Implement OpenAPI endpoints from Turbo spec (#35)

* Add OpenAPI endpoints from Turbo spec

* end-to-end tests for HEAD requests

README.md

@@ -1,7 +1,7 @@
 <p align="center"><br><img src="./icon.png" width="128" height="128" alt="Turbo engine" /></p>
 <h2 align="center">Turbo Cache Server</h2>
 <p align="center">
-  <a href="https://turbo.build/repo">Turborepo</a> remote cache server as a Github Action with S3-compatible storage support.
+  <a href="https://turbo.build/repo">Turborepo</a> remote cache server, <a href="https://turbo.build/repo/docs/core-concepts/remote-caching#self-hosting">API-compliant</a> as a Github Action with S3-compatible storage support.
 </p>
 
 ### How can I use this in my monorepo?
@@ -20,21 +20,22 @@ env:
   TURBO_TOKEN: "turbo-token"
 ```
 
-> [!NOTE]
-> These environment variables are required by Turborepo so it can call
+> [!NOTE] These environment variables are required by Turborepo so it can call
 > the Turbo Cache Server with the right HTTP body, headers and query strings.
-> These environment variables are necessary so the Turborepo binary can
-> identify the Remote Cache feature is enabled and can use them across all steps.
-> You can [read more about this here](https://turbo.build/repo/docs/ci#setup) on the Turborepo official docs.
+> These environment variables are necessary so the Turborepo binary can identify
+> the Remote Cache feature is enabled and can use them across all steps. You can
+> [read more about this here](https://turbo.build/repo/docs/ci#setup) on the
+> Turborepo official docs.
 
-Make sure that you have an S3-compatible storage available. We currently tested with:
+Make sure that you have an S3-compatible storage available. We currently tested
+with:
 
 - [Amazon S3](https://aws.amazon.com/s3/)
 - [Cloudflare R2](https://www.cloudflare.com/en-gb/developer-platform/r2/)
 - [Minio Object Storage](https://min.io/)
 
-1. Still on your `yml` file, after checking out your code, use our custom
-   action to start the Turbo Cache Server in the background:
+1. Still on your `yml` file, after checking out your code, use our custom action
+   to start the Turbo Cache Server in the background:
 
 ```yml
 - name: Checkout repository
@@ -68,17 +69,21 @@ Make sure that you have an S3-compatible storage available. We currently tested
   run: turbo run test build typecheck
 ```
 
-And that is all you need to use our remote cache server for Turborepo.
-As a reference, take a look at [this example workflow file](https://github.com/brunojppb/turbo-decay/blob/main/.github/workflows/ci.yml) for inspiration.
+And that is all you need to use our remote cache server for Turborepo. As a
+reference, take a look at
+[this example workflow file](https://github.com/brunojppb/turbo-decay/blob/main/.github/workflows/ci.yml)
+for inspiration.
 
 ## How does that work?
 
-Turbo Cache Server is a tiny web server written in [Rust](https://www.rust-lang.org/) that
-uses any S3-compatible bucket as its storage layer for the artifacts generated by Turborepo.
+Turbo Cache Server is a tiny web server written in
+[Rust](https://www.rust-lang.org/) that uses any S3-compatible bucket as its
+storage layer for the artifacts generated by Turborepo.
 
 ### What happens when there is a cache hit?
 
-Here is a diagram showing how the Turbo Cache Server works within our actions during a cache hit:
+Here is a diagram showing how the Turbo Cache Server works within our actions
+during a cache hit:
 
 ```mermaid
 sequenceDiagram
@@ -106,8 +111,8 @@ sequenceDiagram
 
 ### What happens when there is a cache miss?
 
-When a cache isn't yet available, the Turbo Cache Server will handle new uploads and store the
-artifacts in S3 as you can see in the following diagram:
+When a cache isn't yet available, the Turbo Cache Server will handle new uploads
+and store the artifacts in S3 as you can see in the following diagram:
 
 ```mermaid
 sequenceDiagram
@@ -140,8 +145,8 @@ sequenceDiagram
 
 ## Development
 
-Turbo Cache Server requires [Rust](https://www.rust-lang.org/) 1.75 or above. To setup your
-environment, use the rustup script as recommended by the
+Turbo Cache Server requires [Rust](https://www.rust-lang.org/) 1.75 or above. To
+setup your environment, use the rustup script as recommended by the
 [Rust docs](https://www.rust-lang.org/learn/get-started):
 
 ```shell
@@ -156,7 +161,10 @@ cargo run
 
 ### Setting up your environment
 
-During local development, you might want to try the Turbo Dev Server locally against a JS monorepo. As it depends on a S3-compatible service for storing Turborepo artifacts, we recommend using [Minio](https://min.io/) with Docker with the following command:
+During local development, you might want to try the Turbo Dev Server locally
+against a JS monorepo. As it depends on a S3-compatible service for storing
+Turborepo artifacts, we recommend using [Minio](https://min.io/) with Docker
+with the following command:
 
 ```shell
 docker run \
@@ -185,3 +193,19 @@ To execute the test suite, run:
 ```shell
 cargo test
 ```
+
+While running our end-to-end tests, you might run into the following error:
+
+```log
+thread 'actix-server worker 9' panicked at /src/index.crates.io-6f17d22bba15001f/actix-server-2.4.0/src/worker.rs:404:34:
+called `Result::unwrap()` on an `Err` value: Os { code: 24, kind: Uncategorized, message: "Too many open files" }
+thread 'artifacts::list_team_artifacts_test' panicked at tests/e2e/artifacts.rs:81:29:
+Failed to request /v8/artifacts
+```
+
+This is likely due the the maximum number of open file descriptors defined for
+your user. Just run the following command to fix it:
+
+```shell
+ulimit -n 1024
+```

src/routes/artifacts.rs

@@ -14,9 +14,41 @@ struct Artifact {
     filename: String,
 }
 
+#[derive(Serialize)]
+struct PostTeamArtifactsResponse {
+    hashes: Vec<String>,
+}
+
+const EMPTY_HASHES: PostTeamArtifactsResponse = PostTeamArtifactsResponse { hashes: vec![] };
+
+/// As of now, we do not need to list all artifacts for a given
+/// team. This seems to be an Admin endpoint for Vercel to map/reduce
+/// on the artifacts for a given team and report metrics.
+#[tracing::instrument(name = "List team artifacts", skip(req))]
+pub async fn post_list_team_artifacts(req: HttpRequest) -> impl Responder {
+    let team = extract_team_from_req(&req);
+
+    tracing::info!(team = team, "Listing team artifacts");
+
+    HttpResponse::Ok().json(&EMPTY_HASHES)
+}
+
+#[tracing::instrument(name = "Check artifact presence", skip(req, storage))]
+pub async fn head_check_file(req: HttpRequest, storage: Data<Storage>) -> impl Responder {
+    let artifact_info = match ArtifactRequest::from(&req) {
+        Some(info) => info,
+        None => return HttpResponse::NotFound().finish(),
+    };
+
+    match storage.file_exists(&artifact_info.file_path()).await {
+        true => HttpResponse::Ok().finish(),
+        false => HttpResponse::NotFound().finish(),
+    }
+}
+
 #[tracing::instrument(name = "Store turbo artifact", skip(storage, body))]
 pub async fn put_file(req: HttpRequest, storage: Data<Storage>, body: Bytes) -> impl Responder {
-    let artifact_info = match ArtifactRequest::from(req) {
+    let artifact_info = match ArtifactRequest::from(&req) {
         Some(info) => info,
         None => return HttpResponse::BadRequest().finish(),
     };
@@ -37,7 +69,7 @@ pub async fn put_file(req: HttpRequest, storage: Data<Storage>, body: Bytes) ->
 
 #[tracing::instrument(name = "Read turbo artifact", skip(storage))]
 pub async fn get_file(req: HttpRequest, storage: Data<Storage>) -> impl Responder {
-    let artifact_info = match ArtifactRequest::from(req) {
+    let artifact_info = match ArtifactRequest::from(&req) {
         Some(info) => info,
         None => return HttpResponse::NotFound().finish(),
     };
@@ -50,6 +82,16 @@ pub async fn get_file(req: HttpRequest, storage: Data<Storage>) -> impl Responde
     HttpResponse::Ok().streaming(stream)
 }
 
+fn extract_team_from_req(req: &HttpRequest) -> String {
+    let query_string = Query::<HashMap<String, String>>::from_query(req.query_string()).unwrap();
+    let default_team_name = "no_team".to_owned();
+    query_string
+        .get("slug")
+        .or_else(|| query_string.get("teamId"))
+        .unwrap_or(&default_team_name)
+        .to_string()
+}
+
 struct ArtifactRequest {
     hash: String,
     team: String,
@@ -61,19 +103,13 @@ impl ArtifactRequest {
         format!("/{}/{}", self.team, self.hash)
     }
 
-    fn from(req: HttpRequest) -> Option<Self> {
+    fn from(req: &HttpRequest) -> Option<Self> {
         let hash = match req.match_info().get("hash") {
             Some(h) => h.to_owned(),
             None => return None,
         };
 
-        let query_string =
-            Query::<HashMap<String, String>>::from_query(req.query_string()).unwrap();
-        let default_team_name = "no_team".to_owned();
-        let team = query_string
-            .get("slug")
-            .unwrap_or(&default_team_name)
-            .to_string();
+        let team = extract_team_from_req(req);
 
         Some(ArtifactRequest { hash, team })
     }

src/startup.rs

@@ -3,7 +3,9 @@ use std::net::TcpListener;
 
 use crate::{
     app_settings::AppSettings,
-    routes::{get_file, health_check, post_events, put_file},
+    routes::{
+        get_file, head_check_file, health_check, post_events, post_list_team_artifacts, put_file,
+    },
     storage::Storage,
 };
 
@@ -14,9 +16,12 @@ pub fn run(listener: TcpListener, app_settings: AppSettings) -> Result<Server, s
         App::new()
             .wrap(Logger::default())
             .route("/management/health", web::get().to(health_check))
+            .route("/v8/artifacts/status", web::get().to(health_check))
+            .route("/v8/artifacts", web::post().to(post_list_team_artifacts))
             .route("/v8/artifacts/events", web::post().to(post_events))
             .route("/v8/artifacts/{hash}", web::put().to(put_file))
             .route("/v8/artifacts/{hash}", web::get().to(get_file))
+            .route("/v8/artifacts/{hash}", web::head().to(head_check_file))
             .app_data(storage.clone())
             .app_data(actix_web::web::PayloadConfig::new(
                 app_settings.max_payload_size_in_bytes,

src/storage.rs

@@ -39,17 +39,24 @@ impl Storage {
         Self { bucket }
     }
 
+    /// Streams the file from the S3 bucket
     pub async fn get_file(&self, path: &str) -> Option<ResponseDataStream> {
         match self.bucket.get_object_stream(path).await {
             Ok(file_stream) => Some(file_stream),
             Err(_) => None,
         }
     }
 
+    /// Stores the given data in the S3 bucket under the given path
     pub async fn put_file(&self, path: &str, data: &[u8]) -> Result<(), String> {
         match self.bucket.put_object(path, data).await {
             Ok(_response) => Ok(()),
             Err(e) => Err(format!("Could not upload file: {}", e)),
         }
     }
+
+    /// Checks whether the given file path exists on the S3 bucket
+    pub async fn file_exists(&self, path: &str) -> bool {
+        self.bucket.head_object(path).await.is_ok()
+    }
 }

tests/e2e/artifacts.rs

@@ -6,7 +6,7 @@ use wiremock::{
 use crate::helpers::{spawn_app, TurboArtifactFileMock};
 
 #[tokio::test]
-async fn upload_artifact_to_s3() {
+async fn upload_artifact_to_s3_test() {
     let app = spawn_app().await;
 
     let client = reqwest::Client::new();
@@ -40,7 +40,7 @@ async fn upload_artifact_to_s3() {
 }
 
 #[tokio::test]
-async fn download_artifact_from_s3() {
+async fn download_artifact_from_s3_test() {
     let app = spawn_app().await;
 
     let client = reqwest::Client::new();
@@ -67,3 +67,77 @@ async fn download_artifact_from_s3() {
     assert!(response.status() == 200);
     assert!(response.text().await.unwrap().as_bytes() == file_mock.file_bytes);
 }
+
+#[tokio::test]
+async fn list_team_artifacts_test() {
+    let app = spawn_app().await;
+
+    let client = reqwest::Client::new();
+
+    let response = client
+        .post(format!("{}/v8/artifacts", &app.address))
+        .send()
+        .await
+        .unwrap_or_else(|_| panic!("Failed to request /v8/artifacts"));
+
+    assert_eq!(response.status(), 200);
+}
+
+#[tokio::test]
+async fn artifact_exists_test() {
+    let app = spawn_app().await;
+
+    let client = reqwest::Client::new();
+    let file_mock = TurboArtifactFileMock::new();
+
+    mock_s3_head_req(&app, &file_mock, 200).await;
+
+    let response = client
+        .head(format!(
+            "{}/v8/artifacts/{}?slug={}",
+            &app.address, file_mock.file_hash, file_mock.team
+        ))
+        .send()
+        .await
+        .expect("Failed to HEAD and check artifact from Sake");
+
+    assert_eq!(response.status(), 200);
+}
+
+#[tokio::test]
+async fn artifact_does_not_exist_test() {
+    let app = spawn_app().await;
+
+    let client = reqwest::Client::new();
+    let file_mock = TurboArtifactFileMock::new();
+
+    mock_s3_head_req(&app, &file_mock, 404).await;
+
+    let response = client
+        .head(format!(
+            "{}/v8/artifacts/{}?slug={}",
+            &app.address, file_mock.file_hash, file_mock.team
+        ))
+        .send()
+        .await
+        .expect("Failed to HEAD and check artifact from Sake");
+
+    assert_eq!(response.status(), 404);
+}
+
+/// A head request must be performed to the S3 bucket
+/// to check whether the artifact exists
+async fn mock_s3_head_req(
+    app: &crate::helpers::TestApp,
+    file_mock: &crate::helpers::TurboArtifactFileMock,
+    response_code: u16,
+) {
+    Mock::given(path(format!(
+        "/{}/{}/{}",
+        app.bucket_name, file_mock.team, file_mock.file_hash
+    )))
+    .and(method("HEAD"))
+    .respond_with(ResponseTemplate::new(response_code))
+    .mount(&app.storage_server)
+    .await;
+}

tests/e2e/health_check.rs

@@ -1,17 +1,33 @@
+use reqwest::Response;
+
 use crate::helpers::spawn_app;
 
 #[tokio::test]
 async fn health_check_test() {
     let app = spawn_app().await;
 
-    let client = reqwest::Client::new();
+    let response = check_endpoint("/management/health", &app).await;
 
-    let response = client
-        .get(format!("{}/management/health", &app.address))
-        .send()
-        .await
-        .expect("Failed to request /management/health");
+    assert!(response.status().is_success());
+    assert_eq!(Some(0), response.content_length());
+}
+
+#[tokio::test]
+async fn turborepo_status_check_test() {
+    let app = spawn_app().await;
+
+    let response = check_endpoint("/v8/artifacts/status", &app).await;
 
     assert!(response.status().is_success());
     assert_eq!(Some(0), response.content_length());
 }
+
+async fn check_endpoint(endpoint: &str, app: &crate::helpers::TestApp) -> Response {
+    let client = reqwest::Client::new();
+
+    client
+        .get(format!("{}{}", &app.address, endpoint))
+        .send()
+        .await
+        .unwrap_or_else(|_| panic!("Failed to request {}", endpoint))
+}