Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add validated check #23

Merged
merged 18 commits into from
Jan 4, 2024
86 changes: 84 additions & 2 deletions tests/functions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,36 @@ import { z } from "zod"
async function extractUser() {
const UserSchema = z.object({
age: z.number(),
name: z.string().refine(name => name.includes(" "), {
message: "Name must contain a space"
name: z.string()
})

type User = z.infer<typeof UserSchema>

const oai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY ?? undefined,
organization: process.env.OPENAI_ORG_ID ?? undefined
})

const client = Instructor({
client: oai,
mode: "FUNCTIONS"
})

const user: User = await client.chat.completions.create({
messages: [{ role: "user", content: "Jason Liu is 30 years old" }],
model: "gpt-3.5-turbo",
response_model: UserSchema,
})

return user
}

async function extractUserValidated() {
const UserSchema = z.object({
age: z.number(),
// check if name is uppercase
name: z.string().refine(name => name === name.toUpperCase(), {
message: "Name must be uppercase, please try again"
})
})

Expand All @@ -33,6 +61,41 @@ async function extractUser() {
return user
}


async function extractUserMany() {
const UserSchema = z.object({
age: z.number(),
name: z.string()
})

const UsersSchema = z.object({
users: z.array(UserSchema)
}).describe("Correctly formatted list of users")

type Users = z.infer<typeof UsersSchema>

const oai = new OpenAI({
apiKey: process.env.OPENAI_API_KEY ?? undefined,
organization: process.env.OPENAI_ORG_ID ?? undefined
})

const client = Instructor({
client: oai,
mode: "FUNCTIONS"
})

const user: Users = await client.chat.completions.create({
messages: [{ role: "user", content: "Jason is 30 years old, Sarah is 12" }],
model: "gpt-3.5-turbo",
response_model: UsersSchema,
max_retries: 3
})

return user
}



describe("FunctionCall", () => {
test("Should return extracted name and age based on schema", async () => {
const user = await extractUser()
Expand All @@ -41,3 +104,22 @@ describe("FunctionCall", () => {
expect(user.age).toEqual(30)
})
})

describe("FunctionCallValidated", () => {
test("Name should be uppercase based on validation check", async () => {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pulled this down and played around a bit - looks like we are really struggling to get things in uppercase

The only pass I was able to get was to switch over to using MD_JSON - my guess is likely because that mode inserts the schema in a system prompt and includes some instruction

function calls (via tools or otherwise) dont really have any system level instruction at all - maybe we should include something

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the goal was that the error messages "this should be in uppercase" should prompt the llm to do that, lets try with gpt-4

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we should change the prompt to not just try again but to fix the erros

const user = await extractUserValidated()

expect(user.name).toEqual("JASON LIU")
expect(user.age).toEqual(30)
})
})

describe("FunctionCallMany", () => {
test("Should return extracted name and age based on schema", async () => {
const users = await extractUserMany()
expect(users.users[0].name).toEqual("Jason")
expect(users.users[0].age).toEqual(30)
expect(users.users[1].name).toEqual("Sarah")
expect(users.users[1].age).toEqual(12)
})
})
Loading