Should we switch to the Claude API?

[MaxGhenis]

I and several economists have found Claude to outperform GPT-4 for economic reasoning. Let's test out our AI prompts against both services to see if we should switch.

[anth-volk]

Is there any significant price and/or speed differential?

[MaxGhenis]

Not sure about speed. Claude is more expensive but our bill is pretty low today so I don't think that's a big factor:

The API pricing is on the slightly higher side where Opus is priced at $15 per million tokens (input) and $75 per million tokens (output), in comparison to GPT-4 Turbo (128k context) $10.00 per million tokens (input) and $30.00 per million tokens (output).

[anth-volk]

I'd be supportive of shifting over, then. Curious what others think.

[MaxGhenis]

I tried the TRAFWA CTC in all three modes, between GPT4 and Claude: https://docs.google.com/document/d/1PLp3xrWXsBdggvfUJaFn9OZqXWzLsU07FtmWo0ad8ws/edit?usp=sharing

Claude could use some slight prompt modifications, e.g. to avoid citing the PolicyEngine version and dataset so many times, but otherwise I think I slightly prefer it. What do others think? @nikhilwoodruff

[anth-volk]

Based on this one example, Claude definitely seems preferable.

[daemon-reconfig]

Claude is a bit better than ChatGPT albeit it is a bit expensive but it makes up for it with it's better answers.
if the pricing is manageable then I think shifting to claude wouldn't be a bad option

[MaxGhenis]

I've added Gemini and Meta AI (Llama 3) to the comparison. I think Claude beats them.

[anth-volk]

For sure

[MaxGhenis]

Just added GPT 4o to the comparison. I think it's as good as Claude 3 Opus, except that it didn't respect ELI5; it suggested moving the style to the beginning of the prompt to address this. Given the speed and cost, would suggest we switch back. But maybe we can keep both options and make it a toggle in case they keep besting each other.

[anth-volk]

Do we know how much the price differential is? And how much of the price is flat vs. usage-based? I could imagine keeping both if there's a low initial fee and pricing scales with usage, but wonder if it could become pricey if 40-50% of the cost is some sort of monthly subscription fee.

[MaxGhenis]

There's no flat/subscription fee.

Claude Opus is 3x the price of GPT-4o for input, 5x for output

https://openai.com/api/pricing/
https://www.anthropic.com/api

[anth-volk]

That is quite a difference. I'd be in support of switching back, as long as we can prompt engineer the voice patterns correctly. In fact, Claude has an option of passing specific directions to the AI model as a "system prompt," and Claude's docs specifically list tone as one good usage for this functionality. I wouldn't be surprised if GPT also has a similar option within its API, which could clear up the tone issue and make it the clear choice over Claude.

[MaxGhenis]

Could someone try with Claude 3.5 Sonnet? My account is having issues.

[MaxGhenis]

Done. It looks slightly better than Claude 3 Opus. Given the lower cost we should switch.

[anth-volk]

Will do. I'll start working on this today.

[anth-volk]

Never mind in light of PolicyEngine/policyengine-api#1587