Improve AI usage of newly open tabs (#254)

Author: rmarescu

.env.example

@@ -31,4 +31,8 @@ ANTHROPIC_API_KEY=
 # Copy from Github Settings
 GITHUB_TOTP_SECRET=
 GITHUB_USERNAME=
-GITHUB_PASSWORD=
+GITHUB_PASSWORD=
+
+# Mailosaur integration
+MAILOSAUR_API_KEY=
+MAILOSAUR_SERVER_ID=

README.md

@@ -328,7 +328,6 @@ You'll need to set up the following services for local development. If you're no
 <summary>GitHub OAuth</summary>
 
 1. Create a GitHub OAuth App:
-
    - Go to your GitHub account settings.
    - Navigate to `Developer settings` > `OAuth Apps` > `New OAuth App`.
    - Fill in the application details:
@@ -344,16 +343,22 @@ You'll need to set up the following services for local development. If you're no
    - Enter your `Client ID` and `Client Secret` from the GitHub OAuth app you just created.
    - Add `repo` to the `Scopes`
    ![Clerk Custom Credentials](https://github.com/user-attachments/assets/31d414e1-4e1e-4725-8649-ec1826c6e53e)
-   </details>
+</details>
 
 <details>
 <summary>Mailosaur</summary>
 
-1. Go to [mailosaur.com](https://mailosaur.com) and create an account.
-2. Create a new server and copy the Server ID.
-3. Go to your API settings and copy your API key.
-   - You'll need both the Server ID and API key for your environment variables: - `MAILOSAUR_API_KEY`: Your API key - `MAILOSAUR_SERVER_ID`: Your server ID
-   </details>
+1. [Sign up](https://mailosaur.com/app/signup) for an account with Mailosaur.
+2. Create a new Inbox/Server.
+3. Go to [API Keys](https://mailosaur.com/app/keys) and create a standard key.
+4. Update the environment variables:
+    - `MAILOSAUR_API_KEY`: Your API key
+    - `MAILOSAUR_SERVER_ID`: Your server ID
+
+The email used to test the login flow will have the format `shortest@<MAILOSAUR_SERVER_ID>.mailosaur.net`, where
+`MAILOSAUR_SERVER_ID` is your server ID.
+Make sure to add the email as a new user under the Clerk app.
+</details>
 
 ### Running locally
 

app/(dashboard)/page.test.ts

@@ -1,5 +1,8 @@
 import { shortest } from "@antiwork/shortest";
 
 shortest("Verify that buttons on the landing page are rounded");
-shortest("Login using this email: mission-health@rdt7stzf.mailosaur.net");
-shortest("Verify that the user can access the /dashboard page");
+
+const loginEmail = `shortest@${process.env.MAILOSAUR_SERVER_ID}.mailosaur.net`;
+shortest("Log in", { email: loginEmail }).expect(
+  "Check Manage Account page from user icon menu",
+);

packages/shortest/src/ai/prompts/index.ts

@@ -4,7 +4,7 @@ export const SYSTEM_PROMPT = `You are a test automation expert working with a Ch
 
 EXAMPLE TEST CASE:
 ------------------
-Test: "Login to the app using Github login"
+Test: "Log in to the app using GitHub login"
 Context: {"username":"argo.mohrad@gmail.com","password":"password1234"}
 Callback function: [NO_CALLBACK]
 Expect: 1. Test case to be generated within at least 20 seconds [HAS_CALLBACK]
@@ -25,9 +25,9 @@ IMPORTANT GLOBAL RULES:
    - Do not request screenshots until after a tool has completely finished its execution.
    - Once the tool execution result is received, you may then request a screenshot to determine subsequent actions if needed.
 
-4. **Github Login Flow with 2FA**:
-   - If you need to test a Github login flow that involves 2FA, only call the "github_login" tool after you have confirmed that the Github login page is displayed.
-   - Calling the "github_login" tool prematurely (before the Github login page is visible) will lead to incorrect test behavior.
+4. **GitHub Login Flow with 2FA**:
+   - If you need to test a GitHub login flow that involves 2FA, only call the "github_login" tool after you have confirmed that the GitHub login page is displayed.
+   - Calling the "github_login" tool prematurely (before the GitHub login page is visible) will lead to incorrect test behavior.
 
 5. **Callbacks**:
    - Steps may include a notation like [HAS_CALLBACK], which means after completing the browser actions for that step, you must call the "run_callback" tool.
@@ -43,12 +43,10 @@ IMPORTANT GLOBAL RULES:
    - If any expectation is not met, the test case must be marked as failed.
 
 8. **Testing Email**:
-   - If you need to test a condition that involves seeing the contents of an email, use the "check_email" tool.
-   - For email validation, you MUST always use 'Click' and 'Mouse' action instead of using keyboard shortcuts.
-   - This tool will grab the latest email from the email address given to you and will render it in a new tab for you to see.
-   - Once you are done with validating the email, navigate back to the original tab.
-   - You MUST pass the email address that is given to you to the tool as a parameter otherwise it will fail.
-   - If no email address is given to you for this test, you should fail the test.
+   - If you need to test a condition that involves checking the contents of an email, use the "check_email" tool.
+   - If no email address is given to you for this test, you must fail the test.
+   - You MUST pass the email address that is given to you to the tool as a parameter, otherwise it will fail.
+   - For email validation, you MUST always use 'Click' and 'Mouse' actions instead of keyboard shortcuts.
 
 9. **Bash Commands**:
    - You have access to a bash tool to execute bash commands.
@@ -62,7 +60,7 @@ Your task is to:
 3. Return test execution results in strict JSON format: { result: "pass" | "fail", reason: string }.
    For failures, provide a maximum 1-sentence reason.
    IMPORTANT:
-     - DO NOT include anything else in your response, only the result and reason.
-     - DO NOT include any other JSON-like object in your response except the required structure.
-       If there's need to do that, remove braces {} to ensure it's not interpreted as JSON.
+   - DO NOT include anything else in your response, only the result and reason.
+   - DO NOT include any other JSON-like object in your response except the required structure.
+   - If there's need to do that, remove braces {} to ensure it's not interpreted as JSON.
 4. For click actions, provide x,y coordinates of the element to click.`;

packages/shortest/src/browser/core/browser-tool.ts

@@ -62,6 +62,12 @@ export class BrowserTool extends BaseBrowserTool {
     this.viewport = { width: config.width, height: config.height };
     this.testContext = config.testContext;
 
+    // Update active page reference to a newly opened tab
+    this.page.context().on("page", async (newPage) => {
+      await newPage.waitForLoadState("domcontentloaded").catch(() => {});
+      this.page = newPage;
+    });
+
     this.initialize();
     this.cleanupScreenshots();
   }