How to return generated question with create_retrieval_chain?

[nstebbins]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.retrievers import RetrieverOutputLike
from langchain.chains.combine_documents import create_stuff_documents_chain

from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import Runnable, RunnableParallel, RunnablePassthrough
from langchain_core.vectorstores import VectorStoreRetriever

...

def build_history_aware_retriever(
    retriever: VectorStoreRetriever,
    llm: Runnable,
    contextualize_question_prompt: ChatPromptTemplate,
) -> RetrieverOutputLike:
    return create_history_aware_retriever(
        llm=llm,
        retriever=retriever,
        prompt=contextualize_question_prompt,
    )


def build_rag_chain_with_history(
    retriever: VectorStoreRetriever,
    llm: Runnable,
    contextualize_question_prompt_retriever: ChatPromptTemplate,
    contextualize_question_prompt_chain: ChatPromptTemplate,
) -> Runnable:
    history_aware_retriever = build_history_aware_retriever(
        retriever=retriever,
        llm=llm,
        contextualize_question_prompt=contextualize_question_prompt_retriever,
    )
    question_answer_chain = create_stuff_documents_chain(
        llm, contextualize_question_prompt_chain
    )
    return create_retrieval_chain(
        retriever=history_aware_retriever, combine_docs_chain=question_answer_chain
    )

Description

Hey guys,

I followed this tutorial here basically to a tee for creating a chain that performs RAG using a history-aware retriever.

I would however like to display the generated question that was created out of the chat history + original question used to retrieve the K nearest documents.

With the old Langchain paradigm you used to be able to accomplish this by basically setting return_generated_question as part of the BaseConversationalRetrievalChain class.

But now that's not an option as ConversationalRetrievalChain is deprecated in favor of the create_retrieval_chain approach as outlined in the docs here.

System Info

System Information
------------------
> OS:  Darwin
> OS Version:  Darwin Kernel Version 23.5.0: Wed May  1 20:17:33 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6031
> Python Version:  3.10.14 (main, May 18 2024, 12:53:40) [Clang 15.0.0 (clang-1500.3.9.4)]

Package Information
-------------------
> langchain_core: 0.2.1
> langchain: 0.2.1
> langchain_community: 0.2.1
> langsmith: 0.1.63
> langchain_anthropic: 0.1.13
> langchain_cli: 0.0.23
> langchain_openai: 0.1.7
> langchain_postgres: 0.0.9
> langchain_text_splitters: 0.2.0
> langserve: 0.2.1

Packages not installed (Not Necessarily a Problem)
--------------------------------------------------
The following packages were not found:

> langgraph

[dosubot[bot]]

Hey @nstebbins! I'm here to help you with your coding questions and support you as you work through any issues. Let me know how I can assist you!

To return the generated question with create_retrieval_chain, you can set the return_generated_question parameter to True when creating the retrieval chain. Here is an example of how you can modify your code to achieve this:

from langchain.chains import create_history_aware_retriever, create_retrieval_chain
from langchain_core.documents import Document
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.retrievers import RetrieverOutputLike
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import Runnable, RunnableParallel, RunnablePassthrough
from langchain_core.vectorstores import VectorStoreRetriever

def build_history_aware_retriever(
    retriever: VectorStoreRetriever,
    llm: Runnable,
    contextualize_question_prompt: ChatPromptTemplate,
) -> RetrieverOutputLike:
    return create_history_aware_retriever(
        llm=llm,
        retriever=retriever,
        prompt=contextualize_question_prompt,
    )

def build_rag_chain_with_history(
    retriever: VectorStoreRetriever,
    llm: Runnable,
    contextualize_question_prompt_retriever: ChatPromptTemplate,
    contextualize_question_prompt_chain: ChatPromptTemplate,
) -> Runnable:
    history_aware_retriever = build_history_aware_retriever(
        retriever=retriever,
        llm=llm,
        contextualize_question_prompt=contextualize_question_prompt_retriever,
    )
    question_answer_chain = create_stuff_documents_chain(
        llm, contextualize_question_prompt_chain
    )
    return create_retrieval_chain(
        retriever=history_aware_retriever, 
        combine_docs_chain=question_answer_chain,
        return_generated_question=True  # Add this line to return the generated question
    )

# Example usage
inputs = {
    "input": "Your user input here",
    "chat_history": []  # Collect chat history here (a sequence of messages)
}

chain = build_rag_chain_with_history(
    retriever=your_retriever_instance,
    llm=your_llm_instance,
    contextualize_question_prompt_retriever=your_contextualize_question_prompt_retriever,
    contextualize_question_prompt_chain=your_contextualize_question_prompt_chain
)

result = chain.invoke(inputs)

# Extract the generated question
generated_question = result.get("generated_question")
print(generated_question)

In this example, the key addition is setting return_generated_question=True when creating the retrieval chain. This will include the generated question in the result, which you can then extract and display ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[nstebbins]

Hi @dosu that didn't help.

I got the following exception when I added the line you mentioned.

TypeError: create_retrieval_chain() got an unexpected keyword argument 'return_generated_question'

[dosubot[bot]]

To include the return_generated_question functionality in the create_retrieval_chain function, you need to modify the function to handle this parameter. Here is how you can update the _do_retrieval and _call methods to include this functionality:

1. Modify _do_retrieval to return the generated questions.
2. Modify _call to collect and return the generated questions.

Here is the updated code:

def _do_retrieval(
        self,
        low_confidence_spans: List[str],
        _run_manager: CallbackManagerForChainRun,
        user_input: str,
        response: str,
        initial_response: str,
    ) -> Tuple[str, bool, List[str]]:
        question_gen_inputs = [
            {
                "user_input": user_input,
                "current_response": initial_response,
                "uncertain_span": span,
            }
            for span in low_confidence_spans
        ]
        callbacks = _run_manager.get_child()
        question_gen_outputs = self.question_generator_chain.apply(
            question_gen_inputs, callbacks=callbacks
        )
        questions = [
            output[self.question_generator_chain.output_keys[0]]
            for output in question_gen_outputs
        ]
        _run_manager.on_text(
            f"Generated Questions: {questions}", color="yellow", end="\n"
        )
        marginal, finished = self._do_generation(questions, user_input, response, _run_manager)
        return marginal, finished, questions

def _call(
        self,
        inputs: Dict[str, Any],
        run_manager: Optional[CallbackManagerForChainRun] = None,
    ) -> Dict[str, Any]:
        _run_manager = run_manager or CallbackManagerForChainRun.get_noop_manager()

        user_input = inputs[self.input_keys[0]]

        response = ""
        all_generated_questions = []
        for i in range(self.max_iter):
            _run_manager.on_text(
                f"Current Response: {response}", color="blue", end="\n"
            )
            _input = {"user_input": user_input, "context": "", "response": response}
            tokens, log_probs = self.response_chain.generate_tokens_and_log_probs(
                _input, run_manager=_run_manager
            )
            low_confidence_spans = _low_confidence_spans(
                tokens,
                log_probs,
                self.min_prob,
                self.min_token_gap,
                self.num_pad_tokens,
            )
            initial_response = response.strip() + " " + "".join(tokens)
            if not low_confidence_spans:
                response = initial_response
                final_response, finished = self.output_parser.parse(response)
                if finished:
                    return {self.output_keys[0]: final_response, "generated_questions": all_generated_questions}
                continue

            marginal, finished, questions = self._do_retrieval(
                low_confidence_spans,
                _run_manager,
                user_input,
                response,
                initial_response,
            )
            all_generated_questions.extend(questions)
            response = response.strip() + " " + marginal
            if finished:
                break
        return {self.output_keys[0]: response, "generated_questions": all_generated_questions}

This modification ensures that the generated questions are collected and returned as part of the output dictionary under the key "generated_questions" ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[nstebbins]

Hi @dosu I honestly don't follow this solution at all nor do I understand how to incorporate this into the code I pasted initially.

[dosubot[bot]]

To include the generated question in the output of the create_retrieval_chain function, you can modify the chain to propagate the generated question through to the output. Here is how you can do it:

1. Create the history-aware retriever:

   from langchain.chains import create_history_aware_retriever
   from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
   
   contextualize_q_system_prompt = (
       "Given a chat history and the latest user question "
       "which might reference context in the chat history, "
       "formulate a standalone question which can be understood "
       "without the chat history. Do NOT answer the question, "
       "just reformulate it if needed and otherwise return it as is."
   )
   
   contextualize_q_prompt = ChatPromptTemplate.from_messages(
       [
           ("system", contextualize_q_system_prompt),
           MessagesPlaceholder("chat_history"),
           ("human", "{input}"),
       ]
   )
   
   history_aware_retriever = create_history_aware_retriever(
       llm, retriever, contextualize_q_prompt
   )

2. Build the full QA chain:

   from langchain.chains import create_retrieval_chain
   from langchain.chains.combine_documents import create_stuff_documents_chain
   
   qa_system_prompt = (
       "You are an assistant for question-answering tasks. "
       "Use the following pieces of retrieved context to answer "
       "the question. If you don't know the answer, say that you "
       "don't know. Use three sentences maximum and keep the "
       "answer concise."
       "\n\n"
       "{context}"
   )
   qa_prompt = ChatPromptTemplate.from_messages(
       [
           ("system", qa_system_prompt),
           MessagesPlaceholder("chat_history"),
           ("human", "{input}"),
       ]
   )
   question_answer_chain = create_stuff_documents_chain(llm, qa_prompt)
   
   rag_chain = create_retrieval_chain(history_aware_retriever, question_answer_chain)

3. Manage chat history:

   from langchain_community.chat_message_histories import ChatMessageHistory
   from langchain_core.chat_history import BaseChatMessageHistory
   from langchain_core.runnables.history import RunnableWithMessageHistory
   
   store = {}
   
   def get_session_history(session_id: str) -> BaseChatMessageHistory:
       if session_id not in store, store[session_id] = ChatMessageHistory()
       return store[session_id]
   
   conversational_rag_chain = RunnableWithMessageHistory(
       rag_chain,
       get_session_history,
       input_messages_key="input",
       history_messages_key="chat_history",
       output_messages_key="answer",
   )

4. Invoke the chain and display the questions:

   result = conversational_rag_chain.invoke(
       {"input": "What is Task Decomposition?"},
       config={
           "configurable": {"session_id": "abc123"}
       }
   )
   
   # Display the generated question and the original question
   print("Generated Question:", result.get("generated_question", "N/A"))
   print("Original Question:", "What is Task Decomposition?")

By following these steps, you can display both the generated question from the chat history and the original question used to retrieve the nearest documents in a RAG chain using a history-aware retriever in LangChain ^{[1][2][3][4][5]}.

It seems like this issue requires a deeper look, @eyurtsev could you please lend your expertise?

To continue talking to Dosu, mention @dosu.

[ahmedbr]

Hi @nstebbins, did you solve this problem?

[pingusix]

I'm seeing this as well after a recent upgrade of libraries, etc.
Did anyone ever find a solution to this?