Add semantic similarity score to `TestEvalLLMs` #37

markuswustenberg · 2025-01-21T10:01:33Z

No description provided.

github-actions · 2025-01-21T10:04:51Z

Evals

Name	Input	Expected	Output	Type	Score	Duration
TestEvalLLMs/gpt-4o-mini	Hi!	Hello! How can I assist you today?	Hello! How can I assist you today?	LexicalSimilarity	1.00 ➡️	462.234611ms
TestEvalLLMs/gpt-4o-mini	Hi!	Hello! How can I assist you today?	Hello! How can I assist you today?	SemanticSimilarity	1.00 🆕	967.003217ms
TestEvalLLMs/gemini-1.5-flash	Hi!	Hi there! How can I help you today?	Hi there! How can I help you today?	LexicalSimilarity	1.00 ➡️	420.455661ms
TestEvalLLMs/gemini-1.5-flash	Hi!	Hi there! How can I help you today?	Hi there! How can I help you today?	SemanticSimilarity	1.00 🆕	778.171465ms
TestEvalLLMs/claude-3.5-haiku	Hi!	Hello! How are you doing today? Is there anything …	Hello! How are you doing today? Is there anything …	LexicalSimilarity	1.00 ➡️	577.305544ms
TestEvalLLMs/claude-3.5-haiku	Hi!	Hello! How are you doing today? Is there anything …	Hello! How are you doing today? Is there anything …	SemanticSimilarity	1.00 🆕	936.860041ms
TestEvalPrompt/answers_with_a_pong	ping	pong	plong	LexicalSimilarity	0.80 ➡️	1.402µs

Total					0.97	4.142031941s

github-actions · 2025-01-21T10:08:32Z

Name	Input	Expected	Output	Type	Score	Duration
TestEvalLLMs/gpt-4o-mini	Hi!	Hello! How can I assist you today?	Hello! How can I assist you today?	LexicalSimilarity	1.00 ➡️	535.851744ms
TestEvalLLMs/gpt-4o-mini	Hi!	Hello! How can I assist you today?	Hello! How can I assist you today?	SemanticSimilarity	1.00 🆕	1.16922224s
TestEvalLLMs/gemini-1.5-flash	Hi!	Hi there! How can I help you today?	Hi there! How can I help you today?	LexicalSimilarity	1.00 ➡️	361.698281ms
TestEvalLLMs/gemini-1.5-flash	Hi!	Hi there! How can I help you today?	Hi there! How can I help you today?	SemanticSimilarity	1.00 🆕	1.66232499s
TestEvalLLMs/claude-3.5-haiku	Hi!	Hello! How are you doing today? Is there anything …	Hello! How are you doing today? Is there anything …	LexicalSimilarity	1.00 ➡️	567.407103ms
TestEvalLLMs/claude-3.5-haiku	Hi!	Hello! How are you doing today? Is there anything …	Hello! How are you doing today? Is there anything …	SemanticSimilarity	1.00 🆕	895.050092ms
TestEvalPrompt/answers_with_a_pong	ping	pong	plong	LexicalSimilarity	0.80 ➡️	1.613µs

Total					0.97	5.191556063s

Add semantic similarity score to TestEvalLLMs

2711d94

increase lint timeout

5842528

markuswustenberg merged commit 112246e into main Jan 21, 2025
3 checks passed

markuswustenberg deleted the semantic-eval branch January 21, 2025 10:09