Efficient retrieval of unevaluated span documents in Phoenix

[firetix]

I'm encountering an issue while attempting to retrieve unevaluated Documents from Phoenix. The documented methods evals['EVALUATOR'].score is None and evals['EVALUATOR'].label is None are not yielding the expected results. Additionally, DOCUMENT_SCORE = "document.score" doesn't seem to be effective for retrieving scores.
My objective is to efficiently retrieve the following metrics:

NDCG (Normalized Discounted Cumulative Gain)
Hit rate
Overall score

Here's the current implementation:

import phoenix as px

project_name = "rag"
evaluation_name = "relevance_evaluator"
phoenix_endpoint = "http://localhost:7442"

query = (
    SpanQuery()
    .select("trace_id", **INPUT)
    .explode("retrieval.documents")
)

spans_df = px.Client(endpoint=phoenix_endpoint).query_spans(query, project_name=project_name)

I've attempted to use the SpanQuery to retrieve the necessary data, but I'm unable to filter for unevaluated documents effectively. Has anyone encountered a similar issue or can suggest an optimal approach to achieve this?
Any insights on best practices for querying unevaluated documents and retrieving specific metrics in Phoenix would be greatly appreciated. Thank you in advance for your assistance.

[dosubot[bot]]

To retrieve documents that have not been evaluated yet and compute the ndcg, hit rate, and score, you can follow these steps:

1. Retrieve Documents Not Evaluated Yet:

   spans_with_docs_df = spans_df[spans_df["attributes.retrieval.documents"].notnull()]

2. Compute NDCG:

   import numpy as np
   from sklearn.metrics import ndcg_score
   
   def _compute_ndcg(df: pd.DataFrame, k: int):
       n = max(2, len(df))
       eval_scores = np.zeros(n)
       doc_scores = np.zeros(n)
       eval_scores[: len(df)] = df.eval_score
       doc_scores[: len(df)] = df.document_score
       try:
           return ndcg_score([eval_scores], [doc_scores], k=k)
       except ValueError:
           return np.nan
   
   ndcg_at_2 = pd.DataFrame(
       {"score": combined.groupby("context.span_id").apply(_compute_ndcg, k=2)}
   )

3. Compute Precision at 3:

   precision_at_3 = pd.DataFrame(
       {
           "score": combined.groupby("context.span_id").apply(
               lambda x: x.eval_score[:3].sum(skipna=False) / 3
           )
       }
   )

4. Compute Hit Rate:

   hit = pd.DataFrame(
       {
           "hit": combined.groupby("context.span_id").apply(
               lambda x: x.eval_score[:2].sum(skipna=False) > 0
           )
       }
   )

5. Combine Results:

   retrievals_df = px.Client().get_spans_dataframe("span_kind == 'RETRIEVER'")
   rag_evaluation_dataframe = pd.concat(
       [
           retrievals_df["attributes.input.value"],
           ndcg_at_2.add_prefix("ndcg@2_"),
           precision_at_3.add_prefix("precision@3_"),
           hit,
       ],
       axis=1,
   )

6. Aggregate Results:

   results = rag_evaluation_dataframe.mean(numeric_only=True)

This code will help you retrieve documents that have not been evaluated yet and compute the ndcg, hit rate, and score for your RAG system. For more detailed steps and explanations, you can refer to the Arize Phoenix: Quickstart - Retrieval tutorial ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[firetix]

@dosu I would like to retrieve the ndcg that already been stored using log_evaluation, How can I retrieve them?

[dosubot[bot]]

To retrieve the NDCG that has already been stored using log_evaluation in Phoenix, you can use the get_evals_dataframe method from the TraceDataset class. Here is the relevant code snippet:

from phoenix.trace import TraceDataset

trace_dataframe = px.active_session().get_spans_dataframe()
trace_ds = TraceDataset(
    trace_dataframe,
    evaluations=evaluations,
)

ndcg_evaluations = trace_ds.get_evals_dataframe()
ndcg_evaluations

This will give you a DataFrame containing the evaluation results, including the NDCG scores ^[1][2].

To continue talking to Dosu, mention @dosu.