feat: add 'aiWaitFor' (#61)

* feat: add

* feat: add

* feat: add  for playwright

* feat: add docs for 'aiWaitFor'

* feat: update docs for report

* feat: add 'wait-for' param in cli

apps/site/docs/en/docs/getting-started/quick-start.mdx

@@ -113,8 +113,7 @@ npx playwright test ./e2e/ebay-search.spec.ts
 
 ### Step 5. view test report after running
 
-After the above command executes successfully, it will output in the console: `Midscene - report file updated: xxx/midscene_run/report/xxx.html`. Open this file in a browser to view the report.
-
+After the above command executes successfully, the console will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. You can open this file in a browser to view the report.
 
 ## Integrate with Puppeteer
 
@@ -203,18 +202,12 @@ npx ts-node demo.ts
 # ]
 ```
 
-
 ### Step 4: View the Run Report
 
-After the above command executes successfully, the console will output: `Midscene - report file updated: xxx/midscene_run/report/xxx.html`. You can open this file in a browser to view the report.
+After the above command executes successfully, the console will output: `Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`. You can open this file in a browser to view the report.
 
 Alternatively, you can import the `./midscene_run/report/latest.web-dump.json` file into the [Visualization Tool](/visualization/) to view it.
 
-
-### View test report after running
-
-After running, Midscene will generate a log dump, which is placed in `./midscene_run/report/latest.web-dump.json` by default. Then put this file into [Visualization Tool](/visualization/), and you will have a clearer understanding of the process.
-
 ## View demo report
 
 Click the 'Load Demo' button in the [Visualization Tool](/visualization/), you will be able to see the results of the previous code as well as some other samples.

apps/site/docs/en/docs/usage/API.md

@@ -100,6 +100,12 @@ expect(onesieItem.price).toBe(7.99);
 ```
 :::
 
+### `.aiWaitFor(assertion: string, {timeoutMs?: number, checkIntervalMs?: number })` - wait until the assertion is met
+
+`.aiWaitFor` will help you check if your assertion has been met or an timeout error occurred. Considering the AI service cost, the check interval will not exceed `checkIntervalMs` milliseconds. The default config sets `timeoutMs` to 15 seconds and `checkIntervalMs` to 3 seconds: i.e. check at most 5 times if all assertions fail and the AI service always responds immediately.
+
+When considering the time required for the AI service, `.aiWaitFor` may not be very efficient. Using a simple `sleep` method might be a useful alternative to `waitFor`.
+
 ## Use LangSmith (Optional)
 
 LangSmith is a platform designed to debug the LLMs. To integrate LangSmith, please follow these steps:

apps/site/docs/en/docs/usage/cli.md

@@ -57,12 +57,13 @@ Options:
   --help                      Display this help message
   --version                   Display the version
 
-Actions (order matters, can be used multiple times):
+Actions (the order matters, can be used multiple times):
   --action <action>           Perform an action, optional
   --assert <assert>           Perform an assert, optional
   --query-output <path>       Save the result of the query to a file, this must be put before --query, optional
   --query <query>             Perform a query, optional
-  --sleep <ms>                Sleep for a number of milliseconds, optional`
+  --wait-for <assertion>      Wait for a condition to be met. The timeout is set to 15 seconds. optional
+  --sleep <ms>                Sleep for a number of milliseconds, optional
 ```
 
 

apps/site/docs/zh/docs/getting-started/quick-start.mdx

@@ -116,8 +116,7 @@ npx playwright test ./e2e/ebay-search.spec.ts
 
 ### Step 5. 查看测试报告
 
-当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: xxx/midscene_run/report/xxx.html` 通过浏览器打开该文件即可看到报告。
-
+当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`，通过浏览器打开该文件即可看到报告。
 
 ## 集成到 Puppeteer
 
@@ -213,17 +212,9 @@ npx ts-node demo.ts
 
 ### 第四步：查看运行报告
 
-当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: xxx/midscene_run/report/xxx.html` 通过浏览器打开该文件即可看到报告。
-
-也可以将 `./midscene_run/report/latest.web-dump.json` 文件导入 [可视化工具](/visualization/) 查看。
-
-
-## 访问示例报告
-
-在 [可视化工具](/visualization/) 中，点击 `Load Demo` 按钮，你将能够看到上方代码的运行结果以及其他的一些示例。
-
-![](/view-demo-visualization.gif)
+当上面的命令执行成功后，会在控制台输出：`Midscene - report file updated: ./current_cwd/midscene_run/report/some_id.html`， 通过浏览器打开该文件即可看到报告。
 
+你也可以将 `./midscene_run/report/latest.web-dump.json` 文件导入 [可视化工具](/visualization/) 查看。
 
 ## 查看示例报告
 

apps/site/docs/zh/docs/usage/API.md

@@ -98,11 +98,16 @@ expect(onesieItem.price).toBe(7.99);
 ```
 :::
 
+### `.aiWaitFor(assertion: string, {timeoutMs?: number, checkIntervalMs?: number })` - 等待断言执行成功
+
+`.aiWaitFor` 帮助你检查你的断言是否满足，或是是否发生了超时错误。考虑到 AI 服务的成本，检查间隔不会超过 `checkIntervalMs` 毫秒。默认配置将 `timeoutMs` 设为 15 秒，`checkIntervalMs` 设为 3 秒：也就是说，如果所有断言都失败，并且 AI 服务总是立即响应，则最多检查 5 次。
+
+考虑到 AI 服务的时间消耗，`.aiWaitFor` 并不是一个特别高效的方法。使用一个普通的 `sleep` 可能是替代 `waitFor` 的另一种方式。
+
 ## 使用 LangSmith （可选）
 
 LangSmith 是一个用于调试大语言模型的平台。想要集成 LangSmith，请按以下步骤操作：
 
-
 ```bash
 # 设置环境变量
 

apps/site/docs/zh/docs/usage/cli.md

@@ -61,11 +61,12 @@ Actions (参数顺序很重要，可以支持多次使用):
   --assert <assert>           Perform an assert, optional
   --query-output <path>       Save the result of the query to a file, this must be put before --query, optional
   --query <query>             Perform a query, optional
+  --wait-for <assertion>      Wait for a condition to be met. The timeout is set to 15 seconds. optional
   --sleep <ms>                Sleep for a number of milliseconds, optional`
 ```
 
 ## 注意事项
 
 1. Options 参数（任务信息）应始终放在 Actions 参数之前。
 2. Actions 参数的顺序很重要。例如，`--action "某操作" --query "某数据"` 表示先执行操作，然后再查询。
-3. 如果有更复杂的需求，比如循环操作，使用 SDK 版本（而不是这个命令行工具）会更容易实现。
+3. 如果有更复杂的需求，比如循环操作，使用 SDK 版本（而不是这个命令行工具）会更合适。

packages/cli/src/help.ts

@@ -17,23 +17,24 @@ if (process.argv.indexOf('--help') !== -1) {
     --assert <assert>           Perform an assert, optional
     --query-output <path>       Save the result of the query to a file, this must be put before --query, optional
     --query <query>             Perform a query, optional
+    --wait-for <assertion>      Wait for a condition to be met. The timeout is set to 15 seconds. optional
     --sleep <ms>                Sleep for a number of milliseconds, optional
 
   Examples:
     # headed mode (i.e. visible browser) to visit bing.com and search for 'weather today'
-    midscene --headed --url https://wwww.bing.com --action "type 'weather today' in search box, hit enter" --sleep 3000
+    midscene --headed --url "https://wwww.bing.com" --action "type 'weather today' in search box, hit enter" --wait-for "there is weather info in the page"
 
     # visit github status page and save the status to ./status.json
-    midscene --url https://www.githubstatus.com/ \\
+    midscene --url "https://www.githubstatus.com/" \\
       --query-output status.json \\
       --query '{name: string, status: string}[], service status of github page'
 
   Examples with Chinese Prompts
     # headed 模式（即可见浏览器）访问 baidu.com 并搜索“天气”
-    midscene --headed --url https://www.baidu.com --action "在搜索框输入 '天气', 敲回车" --sleep 3000
+    midscene --headed --url "https://www.baidu.com" --action "在搜索框输入 '天气', 敲回车" --wait-for 界面上出现了天气信息
 
     # 访问 Github 状态页面并将状态保存到 ./status.json
-    midscene --url https://www.githubstatus.com/ \\
+    midscene --url "https://www.githubstatus.com/" \\
       --query-output status.json \\
       --query '{serviceName: string, status: string}[], github 页面的服务状态，返回服务名称'
   `);

packages/cli/src/index.ts

@@ -52,6 +52,7 @@ const actionArgs = {
   queryOutput: 'query-output',
   query: 'query',
   sleep: 'sleep',
+  waitFor: 'wait-for',
 };
 
 const defaultUA =
@@ -113,21 +114,23 @@ Promise.resolve(
 
     const page = await browser.newPage();
     await page.setUserAgent(ua);
-
     await page.setViewport(viewportConfig);
 
-    updateSpin(stepString('launch', url));
-    await page.goto(url);
-    updateSpin(stepString('waitForNetworkIdle', url));
-    await page.waitForNetworkIdle();
-    printStep('launched', url);
-
-    const agent = new PuppeteerAgent(page);
-
     let errorWhenRunning: Error | undefined;
     let argName: string;
     let argValue: ArgumentValueType;
+    let agent: PuppeteerAgent | undefined;
     try {
+      updateSpin(stepString('launch', url));
+      await page.goto(url);
+      updateSpin(stepString('waitForNetworkIdle', url));
+      await page.waitForNetworkIdle();
+      printStep('launched', url);
+
+      agent = new PuppeteerAgent(page, {
+        autoPrintReportMsg: false,
+      });
+
       let index = 0;
       let outputPath: string | undefined;
       let actionStarted = false;
@@ -197,11 +200,23 @@ Promise.resolve(
             printStep(argName, String(argValue));
             break;
           }
+          case actionArgs.waitFor: {
+            const param = arg.value;
+            assert(param, 'missing assertion for waitFor');
+            assert(typeof param === 'string', 'assertion must be a string');
+            await agent.aiWaitFor(param);
+            printStep(argName, String(argValue));
+            break;
+          }
         }
         index += 1;
       }
+      printStep('Done', `report: ${agent.reportFile}`);
     } catch (e: any) {
       printStep(`${argName!} - Failed`, String(argValue!));
+      if (agent?.reportFile) {
+        printStep('Report', agent.reportFile);
+      }
       printStep('Error', e.message);
       errorWhenRunning = e;
     }
@@ -210,5 +225,3 @@ Promise.resolve(
     process.exit(errorWhenRunning ? 1 : 0);
   })(),
 );
-
-// TODO: print report after running

packages/midscene/src/action/executor.ts

@@ -124,8 +124,7 @@ export class Executor {
         }
 
         Object.assign(task, returnValue);
-
-        task.status = 'success';
+        task.status = 'finished';
         task.timing.end = Date.now();
         task.timing.cost = task.timing.end - task.timing.start;
         taskIndex++;

packages/midscene/src/types.ts

@@ -166,6 +166,15 @@ export type ElementById = (id: string) => BaseElement | null;
 
 export type InsightAssertionResponse = AIAssertionResponse;
 
+/**
+ * agent
+ */
+
+export interface AgentWaitForOpt {
+  checkIntervalMs?: number;
+  timeoutMs?: number;
+}
+
 /**
  * planning
  *
@@ -182,6 +191,7 @@ export interface PlanningAction<ParamType = any> {
     | 'Scroll'
     | 'Error'
     | 'Assert'
+    | 'AssertWithoutThrow'
     | 'Sleep';
   param: ParamType;
 }
@@ -213,6 +223,13 @@ export interface PlanningActionParamSleep {
   timeMs: number;
 }
 
+export interface PlanningActionParamError {
+  thought: string;
+}
+
+export type PlanningActionParamWaitFor = AgentWaitForOpt & {
+  assertion: string;
+};
 /**
  * misc
  */
@@ -293,7 +310,7 @@ export type ExecutionTask<
       ? TaskLog
       : unknown
   > & {
-    status: 'pending' | 'running' | 'success' | 'failed' | 'cancelled';
+    status: 'pending' | 'running' | 'finished' | 'failed' | 'cancelled';
     error?: string;
     errorStack?: string;
     timing?: {

packages/midscene/tests/ai/executor/index.test.ts

@@ -87,7 +87,7 @@ describe('executor', () => {
       expect(element).toBeTruthy();
 
       expect(tasks.length).toBe(inputTasks.length);
-      expect(tasks[0].status).toBe('success');
+      expect(tasks[0].status).toBe('finished');
       expect(tasks[0].output).toMatchSnapshot();
       expect(tasks[0].log?.dump).toBeTruthy();
       expect(tasks[0].timing?.end).toBeTruthy();
@@ -151,7 +151,7 @@ describe('executor', () => {
 
     expect(initExecutor.status).toBe('completed');
     expect(initExecutor.tasks.length).toBe(3);
-    expect(initExecutor.tasks[2].status).toBe('success');
+    expect(initExecutor.tasks[2].status).toBe('finished');
 
     // append while completed
     initExecutor.append(actionTask);

packages/visualizer/src/component/misc.tsx

@@ -5,6 +5,7 @@ import {
   CloseCircleFilled,
   LogoutOutlined,
   MinusOutlined,
+  WarningFilled,
 } from '@ant-design/icons';
 
 export function timeCostStrElement(timeCost?: number) {
@@ -32,13 +33,20 @@ export function timeCostStrElement(timeCost?: number) {
 
 export const iconForStatus = (status: string): JSX.Element => {
   switch (status) {
-    case 'success':
+    case 'finished':
     case 'passed':
       return (
         <span style={{ color: '#2B8243' }}>
           <CheckCircleFilled />
         </span>
       );
+
+    case 'finishedWithWarning':
+      return (
+        <span style={{ color: '#f7bb05' }}>
+          <WarningFilled />
+        </span>
+      );
     case 'failed':
     case 'timedOut':
     case 'interrupted':

packages/visualizer/src/component/sidebar.tsx

@@ -23,6 +23,11 @@ const SideItem = (props: {
     statusText = timeCostStrElement(task.timing.cost);
   }
 
+  const statusIcon =
+    task.status === 'finished' && task.error
+      ? iconForStatus('finishedWithWarning')
+      : iconForStatus(task.status);
+
   return (
     <div
       className={`side-item ${selectedClass}`}
@@ -40,7 +45,7 @@ const SideItem = (props: {
     >
       {' '}
       <div className={'side-item-name'}>
-        <span className="status-icon">{iconForStatus(task.status)}</span>
+        <span className="status-icon">{statusIcon}</span>
         <div className="title">{typeStr(task)}</div>
         <div className="status-text">{statusText}</div>
       </div>

packages/visualizer/src/utils.ts

@@ -16,7 +16,7 @@ export function insightDumpToExecutionDump(
     const task: ExecutionTaskInsightLocate = {
       type: 'Insight',
       subType: insightDump.type === 'locate' ? 'Locate' : 'Query',
-      status: insightDump.error ? 'failed' : 'success',
+      status: insightDump.error ? 'failed' : 'finished',
       param: {
         ...(insightDump.userQuery.element
           ? { query: insightDump.userQuery }