Retrieval-with-Wordle

Contributors: Martinus Kleiweg, Shawn Kim, and Matthew Hernandez

Note Access to our paper here

Last updated May 1st, 2024.

Note The scope of this project involves the following points: indexing and retrieval, measuring performance, error analysis, and a proposed improved implementation.

This project was created for the purpose of applying techniques in Information Retrieval (IR) to develop a strategy to efficiently play Wordle.

Table of Contents

• Objective
• Instructions
• ChatGPT Assist
• Default System
• Reinforcement Learning

Objective

In this repository we describe our end-to-end-implementation of the popular Wordle game, using various Information Retrieval (IR) techniques, together with Reinforcement Learning and large language models. The goal of the game is to guess the word-of-the-day under six attempts with the help of feedback in the form of colored tiles. The system operates on algorithms that index five-letter words and perform a boolean search over them. We present analysis over two popular starting words, and make use of an inverted-index to reduce the search after each guess.

Instructions

The code is intended to be run in the terminal. There are three main files to run: benchmark_inv_index_v2.py, test_model_oneword.pyand prompt.py. You will need a list of previous solutions to Wordle to use the prompt file. This can be found here.

In order to reproduce the results from the paper please follow these steps:

1. Clone the Repository

git clone git@github.com:weezymatt/Retrieval-with-Wordle.git

2. Change Directory

cd src/

3. Choose which implementation you would like to test:
   • ChatGPT Assist prompt.py
   • Default System benchmark_inv_index_v2.py
   • Reinforcement Learning System test_model_oneword.py

ChatGPT Assist

1. Running the prompt.py file requires an OpenAI account to be able to programatically run prompts with an API key as an environment variable. Unfortunately, you must fund your account ($5.00 minimum) even though you can run some free API calls. Sorry! Skip if you are not interested.

• Create an environment variable.

  nano .env
  OPENAI_API_KEY=<paste-your-openai-key-here>

• Run the prompt and use the previous solutions to Wordle.

  python3 prompt.py

• Read the prompt and use the recommended word in brackets.

  To choose the most helpful starting word for the next game of Wordle, I will consider the previous solutions - "vapid," "gleam," and "prune.

  Looking at these words, I see that they are quite diverse in terms of their starting letters and vowel/consonant distributions. To increase our chances of hitting > the target word in the fewest guesses possible, I will go with a word that has a good mix of vowels and consonants, as well as a variety of starting letters.

  Considering this information, a good starting word could be "charm" [charm]. This word has a nice balance of vowels and consonants, and the starting letter > is different from the previous solutions. The variety in letters can help cover a wider range of possible words in the Wordle game.

Default System

The default code in benchmark_inv_index_v2.py is written such that it will run against all words for the selected character and print the number of guess in the terminal. We recommend simply running the code against a random letter to see the code in action.

1. Choose the starting letter for the word-of-the-day or a word that you want to test in the Python file.

def main():
  guess = input("Provide your guess: ")
  b = BenchmarkInvIndex(guess.lower())
  # b.benchmark_alphabet() # Returns text file of the benchmark results
  b.benchmark_words_starting_with('<LETTER>')

2. Run the benchmark by providing a guess to initialize the intersection.

Tip: You may use adieu, slate, ChatGPT's recommendation, or your choice!

python3 benchmark_inv_index_v2.py
Provide your guess: <INSERT-YOUR-WORD>

3. If you are interested in the benchmark of your alphabet you may run uncomment the code in the same file.

def main():
  guess = input("Provide your guess: ")
  b = BenchmarkInvIndex(guess.lower())
  b.benchmark_alphabet() # Returns benchmark results in a text file
  # b.benchmark_words_starting_with('c')

Reinforcement Learning System

We recommend running the test_model_oneword.py for the RL system. However, you may run either test_model_randomwords or test_model_allwords to reproduce similar behavior from the default system.

1. Choose the starting word and specific word in the Python file.

   if __name__ == "__main__":
     env = WordleEnv()
     model_path = 'wordle_model.h5'
     model = load_model(model_path)
     start_word = 'adieu' # preset initial word 
     specific_word = 'craft' # preset start word for evaluation

2. Run the system.

   python3 test_model_oneword.py

3. A text file is created in the directory and contains an evaluation of the system.

The End

You've reached the end of our project. Thanks for reading!