Code for a letter-counting benchmark for LLMs; Measuring how reliably LLMs can answer prompts like "How many times does the letter 'r' appear in the word 'barrier'?".
More information in the blog post: https://brianpeiris.github.io/llm-basic-reasoning-benchmark/