Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ai): introduce vercel ai sdk support #316

Merged
merged 83 commits into from
Feb 19, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
8f6d0dd
feat(ai): vercel ai sdk suppport
gladyshcodes Feb 4, 2025
1ae4fa0
chore: address feeback
gladyshcodes Feb 5, 2025
0d9f840
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 5, 2025
eeb6fa5
chore: update pnpm lock
gladyshcodes Feb 5, 2025
481ac23
chore: update config
gladyshcodes Feb 5, 2025
3f2d949
c
gladyshcodes Feb 6, 2025
6f5d04f
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 6, 2025
79bc27c
c
gladyshcodes Feb 6, 2025
7dc366c
c
gladyshcodes Feb 6, 2025
54d538e
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 6, 2025
e67a172
Update packages/shortest/src/types/config.ts
gladyshcodes Feb 7, 2025
a0c3bf2
Update packages/shortest/tests/unit/test-config.ts
gladyshcodes Feb 7, 2025
58cdcac
Update packages/shortest/tests/unit/test-config.ts
gladyshcodes Feb 7, 2025
2c583a6
Update packages/shortest/package.json
gladyshcodes Feb 7, 2025
bce5142
Update packages/shortest/package.json
gladyshcodes Feb 7, 2025
d28058a
Update packages/shortest/package.json
gladyshcodes Feb 7, 2025
13f104a
Update packages/shortest/package.json
gladyshcodes Feb 7, 2025
740df7a
address feedback
gladyshcodes Feb 7, 2025
6d34ee2
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 7, 2025
3fa2f29
chore: run pnpm install
gladyshcodes Feb 7, 2025
0abff69
chore: update readme
gladyshcodes Feb 8, 2025
46be5f2
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 11, 2025
293b97e
chore: fix import
gladyshcodes Feb 11, 2025
a90a899
chore: resolve autofix errors
gladyshcodes Feb 11, 2025
15516e3
temp
gladyshcodes Feb 11, 2025
b7f1917
c
gladyshcodes Feb 11, 2025
0208e59
c
gladyshcodes Feb 11, 2025
2f4b1fd
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 11, 2025
d7a8629
c
gladyshcodes Feb 11, 2025
77a21a0
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 11, 2025
63508ed
Merge main into gladish/vercel-ai-sdk
gladyshcodes Feb 11, 2025
0c15d00
chore: update import
gladyshcodes Feb 11, 2025
cc16d83
Merge remote-tracking branch 'origin' into gladish/vercel-ai-sdk
gladyshcodes Feb 11, 2025
e56faf1
Merge branch 'gladish/vercel-ai-sdk' of github.com:anti-work/shortest…
gladyshcodes Feb 11, 2025
0153125
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 11, 2025
25539b1
Add AnthropicClient
rmarescu Feb 11, 2025
afae142
Add AI log group back
rmarescu Feb 11, 2025
f16c0f1
Improve logging
rmarescu Feb 12, 2025
acafc7c
Rename client file
rmarescu Feb 12, 2025
e8d470c
Improve logging metadata
rmarescu Feb 12, 2025
9d62d70
f
rmarescu Feb 12, 2025
a11a123
f
rmarescu Feb 12, 2025
2bf524d
Remove --log-format CLI param
rmarescu Feb 12, 2025
b1e5c54
Improve config
rmarescu Feb 13, 2025
760c8e3
f
rmarescu Feb 13, 2025
a44376d
f
rmarescu Feb 13, 2025
40f5013
Add support for SHORTEST_ env
rmarescu Feb 13, 2025
35284d0
f
rmarescu Feb 13, 2025
b513673
f
rmarescu Feb 13, 2025
c4d5f22
Merge remote-tracking branch 'origin/main' into gladish/vercel-ai-sdk
rmarescu Feb 13, 2025
24b7b9d
rename ai-client.ts
rmarescu Feb 13, 2025
f31f495
f
rmarescu Feb 13, 2025
ef570fa
Fix token usage
rmarescu Feb 13, 2025
717fae8
f
rmarescu Feb 13, 2025
267a0f4
AI JSON schema
rmarescu Feb 13, 2025
01f2c86
Deprecation warning
rmarescu Feb 13, 2025
e31acea
f
rmarescu Feb 13, 2025
cc1d0db
Move unused interfaces
rmarescu Feb 13, 2025
409b3f5
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 13, 2025
9d2c52a
Add tests
rmarescu Feb 14, 2025
f4036d5
f
rmarescu Feb 14, 2025
1294959
f
rmarescu Feb 14, 2025
1f4a886
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 14, 2025
d9ff707
Remove @anthropic-ai/sdk
rmarescu Feb 14, 2025
aecbfe8
Update package.json
rmarescu Feb 14, 2025
c5f4aa2
Add test for console.warn test
rmarescu Feb 14, 2025
aee107f
f
rmarescu Feb 14, 2025
416aff1
Comment test-loop.ts
rmarescu Feb 14, 2025
3166b5a
[autofix.ci] apply automated fixes
autofix-ci[bot] Feb 14, 2025
ae1bd71
Logging changes
rmarescu Feb 18, 2025
146bf7d
f
rmarescu Feb 18, 2025
403d6fc
log.test.ts
rmarescu Feb 18, 2025
39d3762
event.test.ts
rmarescu Feb 18, 2025
7bde3f1
f
rmarescu Feb 18, 2025
8ea86f9
Add tools memoization
rmarescu Feb 18, 2025
517e500
Improve conversation history
rmarescu Feb 19, 2025
68d49bf
More logging
rmarescu Feb 19, 2025
6c6183f
Merge remote-tracking branch 'origin/main' into gladish/vercel-ai-sdk
rmarescu Feb 19, 2025
3ed4dc9
f
rmarescu Feb 19, 2025
fec6709
Fix reset group logging
rmarescu Feb 19, 2025
99760c9
f
rmarescu Feb 19, 2025
c031adf
Docs & clean-up
rmarescu Feb 19, 2025
ac05490
Revert changes to package.json
rmarescu Feb 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ npx @antiwork/shortest init
```

This will:

- Automatically install the `@antiwork/shortest` package as a dev dependency if it is not already installed
- Create a default `shortest.config.ts` file with boilerplate configuration
- Generate a `.env.local` file (unless present) with placeholders for required environment variables, such as `ANTHROPIC_API_KEY`
Expand All @@ -49,9 +50,12 @@ export default {
headless: false,
baseUrl: "http://localhost:3000",
testPattern: "**/*.test.ts",
anthropicKey: process.env.ANTHROPIC_API_KEY,
ai: {
provider: "anthropic",
},
} satisfies ShortestConfig;
```
Anthropic API key will default to `SHORTEST_ANTHROPIC_API_KEY` / `ANTHROPIC_API_KEY` environment variables. Can be overwritten via `ai.config.apiKey`.

2. Create test files using the pattern specified in the config: `app/login.test.ts`

Expand Down Expand Up @@ -245,12 +249,14 @@ This guide will help you set up the Shortest web app for local development.
### Getting started

1. Clone the repository:

```bash
git clone https://github.com/anti-work/shortest.git
cd shortest
```

2. Install dependencies:

```bash
npm install -g pnpm
pnpm install
Expand Down Expand Up @@ -335,6 +341,7 @@ You'll need to set up the following services for local development. If you're no
<summary>GitHub OAuth</summary>

1. Create a GitHub OAuth App:

- Go to your GitHub account settings.
- Navigate to `Developer settings` > `OAuth Apps` > `New OAuth App`.
- Fill in the application details:
Expand All @@ -349,7 +356,7 @@ You'll need to set up the following services for local development. If you're no
- Select `Use custom credentials`
- Enter your `Client ID` and `Client Secret` from the GitHub OAuth app you just created.
- Add `repo` to the `Scopes`
![Clerk Custom Credentials](https://github.com/user-attachments/assets/31d414e1-4e1e-4725-8649-ec1826c6e53e)
![Clerk Custom Credentials](https://github.com/user-attachments/assets/31d414e1-4e1e-4725-8649-ec1826c6e53e)

</details>

Expand Down
48 changes: 38 additions & 10 deletions lib/config/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,13 @@ const CONFIG_FILE_PATH = "shortest.yml";

let config: Readonly<Config> = Object.freeze({});

/**
* Loads configuration file.
*
* @throws {Error} When configuration file cannot be parsed
*
* @private
*/
async function loadConfig(): Promise<void> {
try {
const fileContents = await fs.readFile(CONFIG_FILE_PATH, "utf8");
Expand All @@ -37,14 +44,32 @@ async function loadConfig(): Promise<void> {
}
}

async function getConfig(): Promise<Readonly<Config>> {
/**
* Returns the current configuration, loading it if necessary.
*
* @returns {Promise<Readonly<Config>>} Frozen configuration object
* @throws {Error} When configuration cannot be loaded
*
* @private
*/
export async function getConfig(): Promise<Readonly<Config>> {
if (Object.keys(config).length === 0) {
await loadConfig();
}
return config;
}

async function getRepoConfig({
/**
* Gets repository-specific configuration.
*
* @param {Object} params - Repository parameters
* @param {string} params.owner - Repository owner
* @param {string} params.repo - Repository name
* @returns {Promise<RepositoryConfig>} Repository configuration
*
* @private
*/
export async function getRepoConfig({
owner,
repo,
}: {
Expand All @@ -58,7 +83,17 @@ async function getRepoConfig({
);
}

async function getTestPatternsConfig({
/**
* Gets test patterns configuration for a repository.
*
* @param {Object} params - Repository parameters
* @param {string} params.owner - Repository owner
* @param {string} params.repo - Repository name
* @returns {Promise<RepositoryConfig["test_patterns"]>} Test patterns configuration
*
* @private
*/
export async function getTestPatternsConfig({
owner,
repo,
}: {
Expand All @@ -68,10 +103,3 @@ async function getTestPatternsConfig({
const repoConfig = await getRepoConfig({ owner, repo });
return repoConfig.test_patterns;
}

export { getConfig, getRepoConfig, getTestPatternsConfig };

// Add this export for testing
export function _resetConfigForTesting(): void {
config = Object.freeze({});
}
19 changes: 14 additions & 5 deletions packages/shortest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
AI-powered natural language end-to-end testing framework.

## Features

- Natural language test writing
- AI-powered test execution using Claude computer use API
- Built on Playwright
Expand All @@ -19,9 +20,11 @@ npx @antiwork/shortest init
```

This will:

- Automatically install the `@antiwork/shortest` package as a dev dependency if it is not already installed
- Create a default `shortest.config.ts` file with boilerplate configuration
- Generate a `.env.local` file (unless present) with placeholders for required environment variables, such as `ANTHROPIC_API_KEY`
- Generate a `.env.local` file (unless present) with placeholders for required environment variables, such as
`SHORTEST_ANTHROPIC_API_KEY` or `ANTHROPIC_API_KEY`
- Add `.env.local` and `.shortest/` to `.gitignore`

### Quick start
Expand All @@ -35,9 +38,13 @@ export default {
headless: false,
baseUrl: "http://localhost:3000",
testPattern: "**/*.test.ts",
anthropicKey: process.env.ANTHROPIC_API_KEY,
ai: {
provider: "anthropic",
},
} satisfies ShortestConfig;
```
Anthropic API key will default to `SHORTEST_ANTHROPIC_API_KEY` / `ANTHROPIC_API_KEY` environment variables. Can be overwritten via `ai.config.apiKey`.


2. Create test files using the pattern specified in the config: `app/login.test.ts`

Expand Down Expand Up @@ -217,11 +224,13 @@ GITHUB_TOTP_SECRET=your_secret # Only for GitHub auth tests
You can run Shortest in your CI/CD pipeline by running tests in headless mode. Make sure to add your Anthropic API key to your CI/CD pipeline secrets.

## Resources
* Visit [GitHub](https://github.com/anti-work/shortest) for detailed docs
* [Contributing guide](./CONTRIBUTING.md)
* [Changelog](https://github.com/anti-work/shortest/releases)

- Visit [GitHub](https://github.com/anti-work/shortest) for detailed docs
- [Contributing guide](./CONTRIBUTING.md)
- [Changelog](https://github.com/anti-work/shortest/releases)

### Prerequisites

- React >=19.0.0 (if using with Next.js 14+ or Server Actions)
- Next.js >=14.0.0 (if using Server Components/Actions)

Expand Down
12 changes: 7 additions & 5 deletions packages/shortest/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,10 +27,10 @@
"prepare": "pnpm build",
"prepublishOnly": "pnpm build",
"postinstall": "node -e \"if (process.platform !== 'win32') { try { require('child_process').execSync('chmod +x dist/cli/bin.js') } catch (_) {} }\"",
"build:types": "tsup src/index.ts --dts-only --format esm --outDir dist",
"build:js": "esbuild src/index.ts --bundle --platform=node --format=esm --outfile=dist/index.js --external:esbuild --external:punycode --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv",
"build:cjs": "esbuild src/index.ts --bundle --platform=node --format=cjs --outfile=dist/index.cjs --external:esbuild --external:punycode --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv",
"build:cli": "esbuild src/cli/bin.ts --bundle --platform=node --format=esm --outdir=dist/cli --metafile=dist/meta-cli.json --external:fsevents --external:chokidar --external:glob --external:esbuild --external:events --external:path --external:fs --external:util --external:stream --external:os --external:assert --external:url --external:playwright --external:@anthropic-ai/sdk --external:expect --external:dotenv --external:otplib --external:picocolors --external:punycode --external:https --external:http --external:net --external:tls --external:crypto --external:mailosaur",
"build:types": "tsup src/index.ts --dts-only --format esm --outDir dist",
"build:js": "esbuild src/index.ts --bundle --platform=node --format=esm --outfile=dist/index.js --external:esbuild --external:punycode --external:playwright --external:expect --external:dotenv --external:ai --external:@ai-sdk/*",
"build:cjs": "esbuild src/index.ts --bundle --platform=node --format=cjs --outfile=dist/index.cjs --external:esbuild --external:punycode --external:playwright --external:expect --external:dotenv --external:ai --external:@ai-sdk/*",
"build:cli": "esbuild src/cli/bin.ts --bundle --platform=node --format=esm --outdir=dist/cli --metafile=dist/meta-cli.json --external:fsevents --external:chokidar --external:glob --external:esbuild --external:events --external:path --external:fs --external:util --external:stream --external:os --external:assert --external:url --external:playwright --external:expect --external:dotenv --external:otplib --external:picocolors --external:punycode --external:https --external:http --external:net --external:tls --external:crypto --external:mailosaur --external:ai --external:@ai-sdk/*",
"dev": "pnpm build --watch",
"test:unit": "npx vitest run",
"test:unit:watch": "npx vitest --watch",
Expand All @@ -57,7 +57,9 @@
"node": ">=18"
},
"peerDependencies": {
"@anthropic-ai/sdk": "^0.32.0",
"@ai-sdk/anthropic": "^1.1.8",
"@ai-sdk/provider": "^1.0.7",
"ai": "^3.4.0",
"dotenv": "^16.4.5",
"esbuild": "^0.20.1",
"expect": "^29.7.0",
Expand Down
Loading
Loading