feat(ai): introduce vercel ai sdk support #316

gladyshcodes · 2025-02-04T22:55:23Z

Issue #291

What

Why

Simplify the introduction of new providers and models in the future, specifically, Amazon Bedrock (#310) and OpenAI when available

vercel · 2025-02-04T22:55:27Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
shortest	⬜️ Ignored (Inspect)	Visit Preview		Feb 19, 2025 7:38pm

CLAassistant · 2025-02-04T22:55:28Z

All committers have signed the CLA.

gladyshcodes · 2025-02-04T22:57:37Z

In progress:

Nits
Testing
Feedback iterations

rmarescu

Added some initial comments.

packages/shortest/src/ai/client-provider.ts

packages/shortest/src/ai/client.ts

packages/shortest/src/ai/client_v2.ts

gladyshcodes · 2025-02-06T22:18:04Z

@rmarescu Feedback addressed

rmarescu

Submitting a partial review. High-level thoughts:

No need to map Vercel's models with a local list, it complicates the implementation without too much gain
Use AI instead of LLM (don't need 2 names that represent almost the same thing within the codebase)
Config schema seems complicated. Can be simplified?

packages/shortest/tests/unit/test-config.ts

packages/shortest/package.json

packages/shortest/src/types/config.ts

rmarescu · 2025-02-07T07:12:42Z

packages/shortest/src/types/config.ts

+}
+
+export interface LLMPPublicConfig {
+  provider: LLMSupportedProvidersType;


I think even this one can be optional. Ideally, there should be zero config to run Shortest.

No ai prop provided should default to the provider we think is the best, e.g. anthropic, which tries to read the apiKey from ENV (as Vercel AI SDK supports it by default).

I personally don't see a strong case for simplifying the configuration. Keeping all settings in a single file actually reduces the cognitive load, as it makes it clear where each value comes from and how the final configuration is assembled

Ideally, there should be zero config to run Shortest

Achieving that is unlikely—especially as users increasingly demand more flexibility and control over Shortest (see, for example, issue #313)

What is possible, though, is keeping all non-sensitive data in config and sensitive data (e.g keys) in the env, not to duplicate them in the config

packages/shortest/src/types/config.ts

Co-authored-by: Razvan Marescu <razvan@marescu.net>

vercel · 2025-02-07T18:55:22Z

You must have Developer access to commit code to Antiwork on Vercel. If you contact an administrator and receive Developer access, commit again to see your changes.

Learn more: https://vercel.com/docs/accounts/team-members-and-roles/access-roles#team-level-roles

Co-authored-by: Razvan Marescu <razvan@marescu.net>

rmarescu · 2025-02-15T16:25:21Z

Fix error when running examples/youtube.test.rb

Details

trace  | 11:23:13 | examples/youtube.test.ts | Visit a YouTube channel and verify latest content | 🤖 | 1 | Generating text | currentPrompt=
  Test: "Visit a YouTube channel and verify latest content"
  Context: {"channelName":"TED","sortBy":"newest"}
  Callback function:  [NO_CALLBACK]

  Expect:
  1. "Visit a YouTube channel and verify latest content" expected to be successful

  Current Page State:
  URL: http://localhost:3000/
  Title: Shortest
 messageCount=1 tools={
  "0": "computer",
  "1": "bash",
  "2": "github_login",
  "3": "check_email",
  "4": "sleep",
  "5": "run_callback",
  "6": "navigate"
}

error  | 11:23:16 | examples/youtube.test.ts | Visit a YouTube channel and verify latest content | 🤖 | 1 | Error making request | message=
  Invalid arguments for tool navigate: Type validation failed: Value: {"url":"https://www.youtube.com/@TED"}.
  Error message: [
    {
      "code": "invalid_literal",
      "expected": "navigate",
      "path": [
        "action"
      ],
      "message": "Invalid literal value, expected "navigate""
    }
  ]
 name=AI_InvalidToolArgumentsError stack=
  Error message: [
    {
      "code": "invalid_literal",

error  | 11:23:16 | examples/youtube.test.ts | Visit a YouTube channel and verify latest content | 🤖 | 1 | Invalid arguments for a tool were provided
error  | 11:23:16 | examples/youtube.test.ts | Visit a YouTube channel and verify latest content | 🤖 | Action failed | message=
  Invalid arguments for tool navigate: Type validation failed: Value: {"url":"https://www.youtube.com/@TED"}.
  Error message: [
    {
      "code": "invalid_literal",
      "expected": "navigate",
      "path": [
        "action"
      ],
      "message": "Invalid literal value, expected "navigate""
    }
  ]
 name=AI_InvalidToolArgumentsError stack=
  Error message: [
    {
      "code": "invalid_literal",

trace  | 11:23:16 | examples/youtube.test.ts | Visit a YouTube channel and verify latest content | 🤖 | Retry attempt | retries=1 maxRetries=3

rmarescu · 2025-02-19T16:50:32Z

packages/shortest/src/log/log.ts

+    } else {
+      if (event.level === "warn") {
+        console.warn(
+          pc.bgYellowBright(pc.black(" WARN ")),
+          pc.yellow(event.message),
+        );
+      }
+    }


Forces console output for warnings (used for deprecations at this moment)

rmarescu · 2025-02-19T18:25:11Z

packages/shortest/src/ai/client.ts

+    };
+    this.conversationHistory.push(initialMessageOptions);
+    this.log.trace("💬", "New conversation message", initialMessageOptions);
+    this.log.trace("💬", "Conversation history initialized", {


Each test triggers runAction > runConversation. This tracing should make is easier to follow the AI conversation and the tools executed.

rmarescu · 2025-02-19T18:55:25Z

packages/shortest/src/ai/client.ts

+  private get tools(): Record<string, CoreTool> {
+    if (this._tools) return this._tools;
+
+    this._tools = {


All tools should have the same interface internally, and be managed via a ToolsRegistry or such (can be addressed in a future PR).

rmarescu · 2025-02-19T19:06:35Z

packages/shortest/src/browser/core/browser-tool.ts

@@ -607,6 +611,8 @@ export class BrowserTool extends BaseBrowserTool {
        };
      }
      throw new ToolError(`Action failed: ${error}`);
+    } finally {


Refactored all resetGroup calls to be called within a finally block. This approach ensures the log groups are properly defined with minimal logic.

try { this.log.setGroup("New group"); } finally { this.log.resetGroup(); }

rmarescu · 2025-02-19T19:08:36Z

packages/shortest/src/cli/bin.ts

@@ -122,10 +122,8 @@ function getParamValue(args: string[], paramName: string): string | undefined {
 async function main() {
  const args = process.argv.slice(2);
  const logLevel = getParamValue(args, "--log-level");
-  const logFormat = getParamValue(args, "--log-format");


Missed to removed in a previous PR where --log-format was removed as a CLI arg.

rmarescu · 2025-02-19T19:13:25Z

packages/shortest/src/core/runner/test-reporter.ts


  private filesCount: number = 0;
  private testsCount: number = 0;
  private passedTestsCount: number = 0;
  private failedTestsCount: number = 0;
-  private totalInputTokens: number = 0;
-  private totalOutputTokens: number = 0;
+  private totalPromptTokens: number = 0;


Renamed to match the terminology from AI providers.

rmarescu · 2025-02-19T19:14:47Z

packages/shortest/src/log/event.ts

+    );
+  }
+
+  private parseMetadata():


Mostly moved from log/output.ts, with some adjustments.

rmarescu · 2025-02-19T19:15:49Z

packages/shortest/src/log/output.ts

    if (event.level === "error") {
      message = pc.red(message);
    }

    let outputParts = [];
    outputParts.push(colorFn(`${level}`.padEnd(LogOutput.MAX_LEVEL_LENGTH)));
-    outputParts.push(timestamp);
+    outputParts.push(


.Simplified fro full timestamp, to only HH:MM:SS

rmarescu · 2025-02-19T19:18:01Z

packages/shortest/src/types/ai.ts

+// import { LanguageModelUsage } from "ai";
+import { z } from "zod";
+
+// TODO: Validate against LanguageModelUsage


Can be done in a follow-up PR.

rmarescu · 2025-02-19T20:10:54Z

packages/shortest/src/ai/prompts/index.ts

@@ -20,6 +22,7 @@ IMPORTANT GLOBAL RULES:
   - After invoking a tool, wait until the tool finishes its execution and you receive a success/failure result.
   - You will also receive metadata about the tool's execution to help you interpret its outcome.
   - Only after the tool finishes and you know the result should you request any screenshots or proceed to the next action.
+   - Always include the "action" field matching the tool name in your tool calls (e.g. for "navigate" tool, include 'action: "navigate"').


This is critial to ensure that Vercel AI SDK doesn't return AI_InvalidToolArgumentsError.

feat(ai): vercel ai sdk suppport

8f6d0dd

rmarescu assigned gladyshcodes Feb 5, 2025

rmarescu added this to the v0.4.4 milestone Feb 5, 2025

rmarescu reviewed Feb 5, 2025

View reviewed changes

rmarescu mentioned this pull request Feb 5, 2025

fix(cli): Component UI elements are different, running test in normal mode #305

Open

gladyshcodes and others added 9 commits February 5, 2025 23:28

chore: address feeback

1ae4fa0

Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk

0d9f840

chore: update pnpm lock

eeb6fa5

chore: update config

481ac23

c

3f2d949

Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk

6f5d04f

c

79bc27c

c

7dc366c

[autofix.ci] apply automated fixes

54d538e

gladyshcodes marked this pull request as ready for review February 6, 2025 22:17

gladyshcodes requested a review from rmarescu February 6, 2025 22:23

rmarescu reviewed Feb 7, 2025

View reviewed changes

Update packages/shortest/src/types/config.ts

e67a172

Co-authored-by: Razvan Marescu <razvan@marescu.net>

gladyshcodes and others added 7 commits February 7, 2025 19:59

Update packages/shortest/tests/unit/test-config.ts

a0c3bf2

Co-authored-by: Razvan Marescu <razvan@marescu.net>

Update packages/shortest/tests/unit/test-config.ts

58cdcac

Co-authored-by: Razvan Marescu <razvan@marescu.net>

Update packages/shortest/package.json

2c583a6

Co-authored-by: Razvan Marescu <razvan@marescu.net>

Update packages/shortest/package.json

bce5142

Co-authored-by: Razvan Marescu <razvan@marescu.net>

Update packages/shortest/package.json

d28058a

Co-authored-by: Razvan Marescu <razvan@marescu.net>

Update packages/shortest/package.json

13f104a

Co-authored-by: Razvan Marescu <razvan@marescu.net>

address feedback

740df7a

rmarescu and others added 9 commits February 14, 2025 10:55

f

f4036d5

f

1294959

[autofix.ci] apply automated fixes

1f4a886

Remove @anthropic-ai/sdk

d9ff707

Update package.json

aecbfe8

Add test for console.warn test

c5f4aa2

f

aee107f

Comment test-loop.ts

416aff1

[autofix.ci] apply automated fixes

3166b5a

This was referenced Feb 18, 2025

refactor(vercel): migrate codebase to vercel #341

Closed

Add support for Amazon Bedrock as AI provider #310

Open

feat(ai): introduce amazon bedrock provider #327

Closed

rmarescu added 14 commits February 18, 2025 13:01

Logging changes

ae1bd71

f

146bf7d

log.test.ts

403d6fc

event.test.ts

39d3762

f

7bde3f1

Add tools memoization

8ea86f9

Improve conversation history

517e500

More logging

68d49bf

Merge remote-tracking branch 'origin/main' into gladish/vercel-ai-sdk

6c6183f

f

3ed4dc9

Fix reset group logging

fec6709

f

99760c9

Docs & clean-up

c031adf

Revert changes to package.json

ac05490

rmarescu reviewed Feb 19, 2025

View reviewed changes

rmarescu merged commit 346ac29 into main Feb 19, 2025
6 checks passed

rmarescu deleted the gladish/vercel-ai-sdk branch February 19, 2025 20:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ai): introduce vercel ai sdk support #316

feat(ai): introduce vercel ai sdk support #316

gladyshcodes commented Feb 4, 2025 •

edited

Loading

vercel bot commented Feb 4, 2025 •

edited

Loading

CLAassistant commented Feb 4, 2025 •

edited

Loading

gladyshcodes commented Feb 4, 2025

rmarescu left a comment

gladyshcodes commented Feb 6, 2025

rmarescu left a comment

rmarescu Feb 7, 2025

gladyshcodes Feb 7, 2025 •

edited

Loading

vercel bot commented Feb 7, 2025

rmarescu commented Feb 15, 2025 •

edited

Loading

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

rmarescu Feb 19, 2025

feat(ai): introduce vercel ai sdk support #316

feat(ai): introduce vercel ai sdk support #316

Conversation

gladyshcodes commented Feb 4, 2025 • edited Loading

What

Why

vercel bot commented Feb 4, 2025 • edited Loading

CLAassistant commented Feb 4, 2025 • edited Loading

gladyshcodes commented Feb 4, 2025

rmarescu left a comment

Choose a reason for hiding this comment

gladyshcodes commented Feb 6, 2025

rmarescu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gladyshcodes Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

vercel bot commented Feb 7, 2025

rmarescu commented Feb 15, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gladyshcodes commented Feb 4, 2025 •

edited

Loading

vercel bot commented Feb 4, 2025 •

edited

Loading

CLAassistant commented Feb 4, 2025 •

edited

Loading

gladyshcodes Feb 7, 2025 •

edited

Loading

rmarescu commented Feb 15, 2025 •

edited

Loading