Generating the conversation automatically.

[arthurwolf]

In the video, you have a human (you) checking if the solution is correct or not, and ask the model to think through its mistakes.

I think we can (in a lot of cases, maybe not all) do without that (costly) human requirement, using prompt engineering.

I tried some things, this works pretty well (and can surely be improved with experience) :

I want you to solve a problem, using a specific method.

I will first give you a description of the method, then an example of using the method, then I will give you the actual problem to solve, and finally you will solve the problem using the given method.

The method is as follows, it is a loop:

1. Create a list of requirements that the solution must meet, based on the question.
2. Determine a strategy that you believe would guarantee you find a solution to the problem, explain that strategy.
3. Reason step by step to find a possible solution to the problem using your strategy.
4. Check if the solution satisfies the requirements we created in step 1, you must do this in the most complete and exhaustive fashion possible, do not hesitate to spend a long time on this step. For each requirement, you must **in extreme detail** explain why you believe your answer satisfies this requirement. A common mistake you make here, is that you do not "check" if your reasonning is correct, instead you "give a reason why you think it is correct". You must use critical thinking, and make sure, in detail, for each requirement, that your solution does indeed satisfy that requirement. 
5. Presume you just got the reasonning wrong, and your solution in this loop is incorrect, go over each step of the reasonning in step 2 to find where a mistake was made, presuming one was made for each.
6. Write down a JSON array structure in which each entry is the proposed solution given each time you reached step 2 in past iterations of the loop, followed by whether you currently believe that answer was correct or not. The solutions should be listed in the order they were found. Here is the JSON format:
[ { "solution": "the first solution", "believed_correct_currently": false }, { "solution": "the second solution", "believed_correct_currently": true }, etc...
7. If you have discovered a reason why the current solution is correct, but the previous solution was not correct, explain in detail what mistake was made in the previous solution.
8. If you reach 3 correct solutions in a row, stop the loop and output the solution to the user.
9. Start again from step 2 (this is a loop, and that loop will be executed as many times as required to find a solution that matches the step 1 requirements).

Note you must at no point trust your own reasonning abilities, you must break down every problem in as many simpler components as possible, and you must verify every single thought and step of your reasonning.

Here is the question I want to use this method on: 

Write me a sentence without any words that appear in The Bible, do not use made-up words.

(note the question comes from https://arxiv.org/html/2405.19616 )
(note this would be much more powerful if along with the instructions came multiple examples of using the instructions on real world questions, I just have not had time to do that for this demo).

The important thing here is really step 7.

Now, in my testing, this finds the solution to a lot of difficult questions, with reasonning, and from that output, you can "extract" chain of thought reasonning, by giving it an example of what chain of thought reasonning looks like (for example using the o1 demo examples), and asking it to "transform" the loop into chain of thought.

I don't think this will work for all problems, but in my testing it works for a lot of those I tried out.

The main issue I'm running into right now, is that with claude at least, I often hit the answer length limit before getting to the end of the loop. But in most cases even if that happens, the data generated is enough to convert to CoT (and I believe there are now models with longer answer lengths)

Here's an example of using the prompt, note how similar the actual reasonning is to what o1 does, without requiring a human to tell it whether or not it made a mistake:

I understand you want me to solve the problem of writing a sentence without any words that appear in The Bible, using only real words, by following the specific method you've outlined. I'll go through the steps as described:

1. Requirements:
   - The sentence must be grammatically correct and meaningful.
   - The sentence must not contain any words that appear in The Bible.
   - All words used must be real, existing words (not made-up).

2. Strategy:
   My strategy will be to focus on modern concepts, scientific terms, and relatively recent inventions or discoveries that would not have been mentioned in The Bible. I'll try to construct a sentence using specialized vocabulary from fields like computer science, modern physics, or contemporary culture.

3. Possible solution:
   "Quantum computers utilize superconducting circuits for processing qubits."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar rules. It has a subject ("Quantum computers"), a verb ("utilize"), and an object ("superconducting circuits"). The prepositional phrase "for processing qubits" provides additional information. The sentence conveys a meaningful concept about the functioning of quantum computers.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Quantum": This term comes from Latin "quantus" meaning "how much" and was first used in physics in the early 20th century. It's highly unlikely to appear in The Bible.
      - "Computers": This word didn't exist until the mid-20th century, long after The Bible was written.
      - "Utilize": While this word has Latin roots, it wasn't coined until the early 19th century, well after biblical times.
      - "Superconducting": This scientific term was coined in the early 20th century to describe a phenomenon unknown in biblical times.
      - "Circuits": While "circuit" has ancient roots, its use in the electrical sense dates to the 19th century.
      - "For": This is a common word that likely appears in The Bible. I made a mistake including this word.
      - "Processing": This word in its current meaning related to computing didn't exist in biblical times.
      - "Qubits": This is a modern portmanteau of "quantum" and "bits", coined in the late 20th century.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. None are made-up or invented for this purpose.

5. Presuming a mistake:
   I made a mistake by including the word "for" in the sentence. This common word likely appears in The Bible, violating the second requirement.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     }
   ]

7. Explanation of mistake:
   The mistake in the previous solution was the inclusion of the word "for". As a common preposition, it's almost certainly used in The Bible. I overlooked this common word while focusing on the more specialized vocabulary.

8. Not applicable (haven't reached 3 correct solutions).

9. Starting again from step 2.

2. Revised strategy:
   I will maintain the focus on modern concepts and scientific terms, but I'll be more vigilant about excluding common words. I'll try to construct a sentence using only specialized vocabulary and technical terms that postdate The Bible by centuries.

3. New possible solution:
   "Microprocessors execute algorithmic instructions via transistorized logic gates."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar. It has a subject ("Microprocessors"), a verb ("execute"), and an object ("algorithmic instructions"). The prepositional phrase "via transistorized logic gates" provides additional information. The sentence conveys a meaningful concept about how microprocessors function.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Microprocessors": This term was coined in the 1960s, millennia after The Bible was written.
      - "Execute": While this word has ancient roots, its use in the context of computing instructions is modern.
      - "Algorithmic": This adjective is derived from "algorithm," a term that entered English in the 13th century, well after biblical times.
      - "Instructions": This word has Latin roots and might appear in some Bible translations. I should reconsider its use.
      - "Via": This Latin word meaning "by way of" entered English in the 18th century, long after The Bible was written.
      - "Transistorized": This word is derived from "transistor," a term coined in the 1940s.
      - "Logic": While this word has ancient Greek roots, its use in the context of computing is modern.
      - "Gates": In the context of logic circuits, this usage dates to the 20th century, though the word itself is ancient and likely appears in The Bible.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. None are made-up or invented for this purpose.

5. Presuming mistakes:
   I made two mistakes:
   1. Using "instructions," which might appear in some Bible translations.
   2. Using "gates," which, although used in a modern context here, is an ancient word likely to appear in The Bible.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     },
     {
       "solution": "Microprocessors execute algorithmic instructions via transistorized logic gates.",
       "believed_correct_currently": false
     }
   ]

7. Explanation of mistakes:
   In the previous solution, I overlooked the possibility that "instructions" and "gates" might appear in The Bible. Even though they're used in a modern context here, their roots are ancient, and they likely appear in biblical text. I need to be more careful about using words with ancient origins, even if their modern usage is different.

8. Not applicable (haven't reached 3 correct solutions).

9. Starting again from step 2.

2. Revised strategy:
   I will focus exclusively on scientific and technological terms that were coined in the 20th century or later. This should ensure that none of the words could possibly appear in The Bible.

3. New possible solution:
   "Nanotechnology manipulates fullerenes utilizing scanning tunneling microscopy."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar. It has a subject ("Nanotechnology"), a verb ("manipulates"), and an object ("fullerenes"). The phrase "utilizing scanning tunneling microscopy" provides additional information. The sentence conveys a meaningful concept about a technique used in nanotechnology.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Nanotechnology": This term was coined in 1974, millennia after The Bible was written.
      - "Manipulates": While this word has Latin roots, its use in scientific context is modern. However, it's possible that some form of this word might appear in certain Bible translations. I should reconsider its use.
      - "Fullerenes": This term was coined in the 1980s, named after Buckminster Fuller.
      - "Utilizing": This word, while having Latin roots, wasn't coined until the early 19th century, well after biblical times.
      - "Scanning": While "scan" has ancient roots, its use in the context of microscopy is entirely modern.
      - "Tunneling": In the context of quantum mechanics, this usage dates to the 20th century.
      - "Microscopy": This word was coined in the 19th century, long after The Bible was written.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. None are made-up or invented for this purpose.

5. Presuming mistakes:
   I made a mistake by using "manipulates" and "utilizing". While these words are used in a modern scientific context, they have ancient roots and might appear in some Bible translations.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     },
     {
       "solution": "Microprocessors execute algorithmic instructions via transistorized logic gates.",
       "believed_correct_currently": false
     },
     {
       "solution": "Nanotechnology manipulates fullerenes utilizing scanning tunneling microscopy.",
       "believed_correct_currently": false
     }
   ]

7. Explanation of mistakes:
   In the previous solution, I again included words with ancient roots ("manipulates" and "utilizing") that might appear in some Bible translations. Even though they're used in a modern context, I need to be more stringent in my word selection to ensure no possibility of overlap with biblical text.

8. Not applicable (haven't reached 3 correct solutions).

9. Starting again from step 2.

2. Revised strategy:
   I will use only scientific terms and technological jargon that were coined in the mid-20th century or later, avoiding any words with ancient roots. I'll focus on creating a sentence about a very modern concept using only recently invented terminology.

3. New possible solution:
   "Cryptocurrency blockchain technologists debugged multithreaded APIs."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar. It has a subject ("Cryptocurrency blockchain technologists"), a verb ("debugged"), and an object ("multithreaded APIs"). The sentence conveys a meaningful concept about a task performed by professionals in the cryptocurrency field.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Cryptocurrency": This term was coined in the late 20th century, long after The Bible was written.
      - "Blockchain": This term was first used in 2008, millennia after biblical times.
      - "Technologists": While "technology" has Greek roots, the term "technologist" in its current usage dates to the 20th century.
      - "Debugged": This term originated in the 1940s with the rise of computer programming.
      - "Multithreaded": This computing term was coined in the late 20th century.
      - "APIs": This acronym (Application Programming Interfaces) and the concept it represents date to the 1960s.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. "APIs" is a widely recognized acronym in the field of computer science. None of the words are made-up or invented for this purpose.

5. Presuming mistakes:
   After careful consideration, I don't see any obvious mistakes in this solution. All words are modern inventions or technical terms that postdate The Bible by centuries or millennia. However, I should remain cautious and open to the possibility that I might have overlooked something.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     },
     {
       "solution": "Microprocessors execute algorithmic instructions via transistorized logic gates.",
       "believed_correct_currently": false
     },
     {
       "solution": "Nanotechnology manipulates fullerenes utilizing scanning tunneling microscopy.",
       "believed_correct_currently": false
     },
     {
       "solution": "Cryptocurrency blockchain technologists debugged multithreaded APIs.",
       "believed_correct_currently": true
     }
   ]

7. Not applicable (no mistake found in the current solution).

8. Not applicable (haven't reached 3 correct solutions in a row).

9. Starting again from step 2.

2. Strategy:
   I will maintain the focus on using only modern technological and scientific terms, ensuring each word was coined well after The Bible was written. I'll create a sentence about a different aspect of contemporary technology to further validate the approach.

3. Possible solution:
   "Cybersecurity experts encrypted datacenter mainframes autonomously."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar. It has a subject ("Cybersecurity experts"), a verb ("encrypted"), and an object ("datacenter mainframes"). The adverb "autonomously" modifies the verb. The sentence conveys a meaningful concept about an action performed by cybersecurity professionals.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Cybersecurity": This term was coined in the late 20th century, long after The Bible was written.
      - "Experts": While "expert" has Latin roots, its use as a noun dates to the 19th century, after biblical times.
      - "Encrypted": This term in its current usage related to secure communication dates to the 20th century.
      - "Datacenter": This compound word was coined in the late 20th century with the rise of large-scale computing facilities.
      - "Mainframes": This term for large computers was coined in the 1960s.
      - "Autonomously": While "autonomous" has Greek roots, its use in the context of computer systems is entirely modern.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. None are made-up or invented for this purpose.

5. Presuming mistakes:
   After careful consideration, I don't see any obvious mistakes in this solution. All words are modern inventions or technical terms that postdate The Bible by centuries or millennia. However, I should remain cautious and open to the possibility that I might have overlooked something.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     },
     {
       "solution": "Microprocessors execute algorithmic instructions via transistorized logic gates.",
       "believed_correct_currently": false
     },
     {
       "solution": "Nanotechnology manipulates fullerenes utilizing scanning tunneling microscopy.",
       "believed_correct_currently": false
     },
     {
       "solution": "Cryptocurrency blockchain technologists debugged multithreaded APIs.",
       "believed_correct_currently": true
     },
     {
       "solution": "Cybersecurity experts encrypted datacenter mainframes autonomously.",
       "believed_correct_currently": true
     }
   ]

7. Not applicable (no mistake found in the current solution).

8. Not applicable (haven't reached 3 correct solutions in a row).

9. Starting again from step 2.

2. Strategy:
   I will continue using only modern technological and scientific terms, ensuring each word was coined well after The Bible was written. I'll create a sentence about yet another aspect of contemporary technology to further validate the approach.

3. Possible solution:
   "Biotechnologists genetically engineered CRISPR nanorobots offline."

4. Checking requirements:
   a) Is the sentence grammatically correct and meaningful?
      The sentence follows standard English grammar. It has a subject ("Biotechnologists"), a verb phrase ("genetically engineered"), an object ("CRISPR nanorobots"), and an adverb ("offline"). The sentence conveys a meaningful concept about an action performed by biotechnology professionals.

   b) Does the sentence contain any words that appear in The Bible?
      Let's examine each word:
      - "Biotechnologists": This term combines "bio" and "technologists", both of which postdate The Bible. The field of biotechnology emerged in the 20th century.
      - "Genetically": This adverb is derived from "genetics", a term coined in 1905.
      - "Engineered": While "engineer" has ancient roots, its use as a verb in the context of manipulating genetics is entirely modern.
      - "CRISPR": This acronym (Clustered Regularly Interspaced Short Palindromic Repeats) was coined in 2002.
      - "Nanorobots": This term combines "nano" (coined in 1947) and "robots" (coined in 1920).
      - "Offline": This computing term was coined in the mid-20th century.

   c) Are all words real, existing words?
      All words used in the sentence are real, existing words found in modern English dictionaries. "CRISPR" is a widely recognized acronym in the field of biotechnology. None of the words are made-up or invented for this purpose.

5. Presuming mistakes:
   After careful consideration, I don't see any obvious mistakes in this solution. All words are modern inventions or technical terms that postdate The Bible by centuries or millennia. However, I should remain cautious and open to the possibility that I might have overlooked something.

6. JSON array of solutions:
   [
     {
       "solution": "Quantum computers utilize superconducting circuits for processing qubits.",
       "believed_correct_currently": false
     },
     {
       "solution": "Microprocessors execute algorithmic instructions via transistorized logic

And poof, hit the limit.

Now from there, we need to convert this into a CoT conversation.

The prompt for this is super long, because it contains an example of what a "CoT" conversation should look like, so I'm not pasting it here, but you can find it at

https://gist.github.com/arthurwolf/7011e5f5ba06867f558c2da0dbf4f6df#file-conversion-md

It gives us an answer in CoT format like this:

Let's approach this task of writing a sentence without any words that appear in The Bible, using only real words. This is quite a challenge, as The Bible contains a vast vocabulary covering many aspects of life. We'll need to think strategically about this.
First, let's consider our strategy. The Bible was written thousands of years ago, so our best bet is to focus on modern concepts, scientific terms, and recent inventions or discoveries. These would be things that simply didn't exist when The Bible was written. Fields like computer science, modern physics, or contemporary culture are good places to start.
Let's try our first attempt: "Quantum computers utilize superconducting circuits for processing qubits."
Now, let's break this down:

"Quantum": This term comes from Latin "quantus" meaning "how much" and was first used in physics in the early 20th century. Definitely not in The Bible.
"Computers": This word didn't exist until the mid-20th century. Safe.
"Utilize": Coined in the early 19th century. Should be fine.
"Superconducting": A scientific term from the early 20th century. Safe.
"Circuits": In the electrical sense, this dates to the 19th century. Good.
"For": Oh no! This is a common word that's almost certainly in The Bible. We've made a mistake here.
"Processing": In its computing context, this is modern. Safe.
"Qubits": A modern portmanteau of "quantum" and "bits", coined in the late 20th century. Definitely safe.

So close! But we've stumbled on the simple word "for". This shows we need to be careful not just with technical terms, but also with common words. Let's try again.
For our second attempt, let's go with: "Microprocessors execute algorithmic instructions via transistorized logic gates."
Breaking it down:

"Microprocessors": Coined in the 1960s. Safe.
"Execute": While this word has ancient roots, its use in computing is modern. Probably safe.
"Algorithmic": Derived from "algorithm," which entered English in the 13th century. Safe.
"Instructions": Uh oh. This has Latin roots and might appear in some Bible translations. We should avoid this.
"Via": Entered English in the 18th century. Safe.
"Transistorized": Derived from "transistor," coined in the 1940s. Very safe.
"Logic": In the computing context, this is modern usage. Probably safe.
"Gates": Although we're using it in a modern context, this word is ancient and likely appears in The Bible. Another mistake.

We've made progress, but we're still not there. "Instructions" and "gates" are problematic. This shows we need to be even more careful about words with ancient roots, even if their modern usage is different.
Let's try a third time, focusing exclusively on scientific and technological terms from the 20th century or later: "Nanotechnology manipulates fullerenes utilizing scanning tunneling microscopy."
Analyzing this:

"Nanotechnology": Coined in 1974. Perfect.
"Manipulates": While used in a modern context, this has Latin roots. It might appear in some Bible translations. We should replace this.
"Fullerenes": Coined in the 1980s. Excellent.
"Utilizing": Like "manipulates," this has Latin roots and could be in some translations. Another word to replace.
"Scanning": In the context of microscopy, this is modern usage. Probably safe.
"Tunneling": In quantum mechanics context, this is 20th century. Safe.
"Microscopy": Coined in the 19th century. Safe.

We're getting closer, but "manipulates" and "utilizing" are still problematic. We need to be even more stringent in our word selection.
For our fourth attempt, let's use only terms coined in the mid-20th century or later, focusing on very modern concepts: "Cryptocurrency blockchain technologists debugged multithreaded APIs."
Let's examine this:

"Cryptocurrency": Late 20th century. Perfect.
"Blockchain": First used in 2008. Excellent.
"Technologists": While "technology" has Greek roots, this specific term is 20th century. Should be safe.
"Debugged": Originated in the 1940s with computer programming. Safe.
"Multithreaded": Late 20th century computing term. Perfect.
"APIs": Acronym from the 1960s. Safe.

This sentence looks promising! All terms are modern inventions or technical jargon that postdate The Bible by centuries or millennia.
To be thorough, let's try one more: "Cybersecurity experts encrypted datacenter mainframes autonomously."
Analyzing this:

"Cybersecurity": Late 20th century. Perfect.
"Experts": Noun usage dates to the 19th century. Should be safe.
"Encrypted": Current usage related to secure communication is 20th century. Safe.
"Datacenter": Late 20th century compound word. Perfect.
"Mainframes": Coined in the 1960s. Safe.
"Autonomously": In the context of computer systems, this is entirely modern. Safe.

This sentence also looks good! All words are modern inventions or technical terms that postdate The Bible by centuries or millennia.
For a final attempt, let's try: "Biotechnologists genetically engineered CRISPR nanorobots offline."
Breaking it down:

"Biotechnologists": Combines "bio" and "technologists", both post-biblical. Safe.
"Genetically": Derived from "genetics", coined in 1905. Perfect.
"Engineered": In genetic context, entirely modern. Safe.
"CRISPR": Acronym coined in 2002. Excellent.
"Nanorobots": Combines "nano" (1947) and "robots" (1920). Perfect.
"Offline": Mid-20th century computing term. Safe.

This sentence also meets all our criteria!
In conclusion, we've successfully created several sentences that don't use any words appearing in The Bible, while using only real words. Our best solutions are:

"Cryptocurrency blockchain technologists debugged multithreaded APIs."
"Cybersecurity experts encrypted datacenter mainframes autonomously."
"Biotechnologists genetically engineered CRISPR nanorobots offline."

These sentences fulfill the requirements by using only modern, technical terminology that didn't exist when The Bible was written, thus ensuring no overlap with biblical vocabulary.

As you can see, this process, using two LLM calls, converted a question/puzzle, into a "chain of thought" answer that can be used in a dataset, without requiring a human to participate in the process.

(note, this tries to do everything with two llm calls, but I believe there is a way to do this much better / with a higher success rate, by extracting the logic (looping, going step by step) outside of the prompts, and doing it with code, which would result in much more llm calls, but make it much less likely the llm gets "confused" by context. I can write down an example of doing that if somebody is interrested)

[daveshap]

I demonstrated that Claude can do a full check of the data and can serve as the user agent. Do not get ahead of the conversation, I have done multi-step data synthesis plenty of times.