Add multi api inference engine

docs/docs/examples.rst

@@ -1,6 +1,6 @@
 .. _examples:
 ==============
-Examples 
+Examples
 ==============
 
 Here you will find complete coding samples showing how to perform different tasks using Unitxt.
@@ -97,16 +97,16 @@ Related documentation: :ref:`Templates tutorial <adding_template>`, :ref:`Format
 Long Context
 +++++++++++++++++++++++++++++
 
-This example explores the effect of long context in classification.  
+This example explores the effect of long context in classification.
 It converts a standard multi class classification dataset (sst2 sentiment classification),
 where single sentence texts are classified one by one, to a dataset
-where multiple sentences are classified using a single LLM call.  
+where multiple sentences are classified using a single LLM call.
 It compares the f1_micro in both approaches on two models.
 It uses serializers to verbalize and enumerated list of multiple sentences and labels.
 
 `Example code <https://github.com/IBM/unitxt/blob/main/examples/evaluate_batched_multiclass_classification.py>`_
 
-Related documentation:  :ref:`Sst2 dataset card in catalog <catalog.cards.sst2>` :ref:`Types and Serializers Guide <types_and_serializers>`. 
+Related documentation:  :ref:`Sst2 dataset card in catalog <catalog.cards.sst2>` :ref:`Types and Serializers Guide <types_and_serializers>`.
 
 Construct a benchmark of multiple datasets and obtain the final score
 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@@ -265,4 +265,3 @@ This example show how to define new data types as well as the way these data typ
 
 Related documentation: :ref:`Types and Serializers Guide <types_and_serializers>`.
 
-

examples/evaluate_a_judge_model_capabilities_on_arena_hard.py

@@ -1,32 +1,28 @@
 from unitxt import evaluate, load_dataset
-from unitxt.inference import MockInferenceEngine
+from unitxt.inference import CrossProviderInferenceEngine
 from unitxt.text_utils import print_dict
 
-model_id = "meta-llama/llama-3-70b-instruct"
-model_format = "formats.llama3_instruct"
-
 """
-We are evaluating only on a small subset (by using "select(range(4)), in order for the example to finish quickly.
+We are evaluating only on a small subset (by using `max_test_instances=4`), in order for the example to finish quickly.
 The dataset full size if around 40k examples. You should use around 1k-4k in your evaluations.
 """
 dataset = load_dataset(
     card="cards.arena_hard.response_assessment.pairwise_comparative_rating.both_games_gpt_4_judge",
     template="templates.response_assessment.pairwise_comparative_rating.arena_hard_with_shuffling",
-    format=model_format,
-)["test"].select(range(4))
+    format="formats.chat_api",
+    max_test_instances=4,
+    split="test",
+)
 
-inference_model = MockInferenceEngine(model_name=model_id)
+inference_model = CrossProviderInferenceEngine(
+    model="llama-3-2-1b-instruct", provider="watsonx"
+)
 """
-We are using a mock inference engine (and model) in order for the example to finish quickly.
-In real scenarios you can use model from Huggingface, OpenAi, and IBM, using the following:
-from unitxt.inference import (HFPipelineBasedInferenceEngine, IbmGenAiInferenceEngine, OpenAiInferenceEngine)
-and switch them with the MockInferenceEngine class in the example.
-For the arguments these inference engines can receive, please refer to the classes documentation.
+We are using a CrossProviderInferenceEngine inference engine that supply api access to provider such as:
+watsonx, bam, openai, azure, aws and more.
 
-Example of using an IBM model:
-from unitxt.inference import (IbmGenAiInferenceEngine, IbmGenAiInferenceEngineParamsMixin)
-params = IbmGenAiInferenceEngineParamsMixin(max_new_tokens=1024, random_seed=42)
-inference_model = IbmGenAiInferenceEngine(model_name=model_id, parameters=params)
+For the arguments these inference engines can receive, please refer to the classes documentation or read
+about the the open ai api arguments the CrossProviderInferenceEngine follows.
 """
 
 predictions = inference_model.infer(dataset)

examples/evaluate_a_model_using_arena_hard.py

@@ -1,35 +1,31 @@
 from unitxt import evaluate, load_dataset
-from unitxt.inference import MockInferenceEngine
+from unitxt.inference import CrossProviderInferenceEngine
 from unitxt.text_utils import print_dict
 
-model_id = "meta-llama/llama-3-70b-instruct"
-model_format = "formats.llama3_instruct"
-
 """
-We are evaluating only on a small subset (by using "select(range(4)), in order for the example to finish quickly.
+We are evaluating only on a small subset (by using `max_test_instances=4`), in order for the example to finish quickly.
 The dataset full size if around 40k examples. You should use around 1k-4k in your evaluations.
 """
 dataset = load_dataset(
     card="cards.arena_hard.generation.english_gpt_4_0314_reference",
     template="templates.empty",
-    format=model_format,
+    format="formats.chat_api",
     metrics=[
-        "metrics.llm_as_judge.pairwise_comparative_rating.llama_3_8b_instruct_ibm_genai_template_arena_hard_with_shuffling"
+        "metrics.llm_as_judge.pairwise_comparative_rating.llama_3_8b_instruct.template_arena_hard"
     ],
-)["test"].select(range(4))
+    max_test_instances=4,
+    split="test",
+)
 
-inference_model = MockInferenceEngine(model_name=model_id)
+inference_model = CrossProviderInferenceEngine(
+    model="llama-3-2-1b-instruct", provider="watsonx"
+)
 """
-We are using a mock inference engine (and model) in order for the example to finish quickly.
-In real scenarios you can use model from Huggingface, OpenAi, and IBM, using the following:
-from unitxt.inference import (HFPipelineBasedInferenceEngine, IbmGenAiInferenceEngine, OpenAiInferenceEngine)
-and switch them with the MockInferenceEngine class in the example.
-For the arguments these inference engines can receive, please refer to the classes documentation.
+We are using a CrossProviderInferenceEngine inference engine that supply api access to provider such as:
+watsonx, bam, openai, azure, aws and more.
 
-Example of using an IBM model:
-from unitxt.inference import (IbmGenAiInferenceEngine, IbmGenAiInferenceEngineParamsMixin)
-params = IbmGenAiInferenceEngineParamsMixin(max_new_tokens=1024, random_seed=42)
-inference_model = IbmGenAiInferenceEngine(model_name=model_id, parameters=params)
+For the arguments these inference engines can receive, please refer to the classes documentation or read
+about the the open ai api arguments the CrossProviderInferenceEngine follows.
 """
 
 predictions = inference_model.infer(dataset)

examples/evaluate_batched_multiclass_classification.py

@@ -28,7 +28,7 @@ class ParseEnumeratedList(FieldOperator):
     def process_value(self, text: Any) -> Any:
         result = []
         for x in text.split("\n"):
-            line_result = re.findall(r"(\d+)\.\s*(\w+)", x)
+            line_result = re.findall(r"(\d+)\.\s*(.*)", x)
             if len(line_result) == 1:
                 result.append(line_result[0])
         return result
@@ -63,96 +63,115 @@ def serialize(self, value: EnumeratedList, instance: Dict[str, Any]) -> str:
 
 template = InputOutputTemplate(
     input_format="Classify each of the texts to its corresponding {type_of_class} from one of these options:\n{classes}\nReturn for each index the correspond class in a separate line.\nTexts:\n{texts}",
-    #   target_prefix="Answer:\n",
+    target_prefix="Answer:\n",
     output_format="{labels}",
-    postprocessors=[PostProcess(ParseEnumeratedList())],
+    postprocessors=["processors.lower_case", PostProcess(ParseEnumeratedList())],
     serializer=MultiTypeSerializer(serializers=[EnumeratedListSerializer()]),
 )
 df = pd.DataFrame(
-    columns=["model", "batch_size", "num_instances", "f1_micro", "ci_low", "ci_high"]
+    columns=[
+        "provider",
+        "model",
+        "batch_size",
+        "num_instances",
+        "f1_micro",
+        "ci_low",
+        "ci_high",
+        "hellucinations",
+    ]
 )
 
-for model_name in [
-    "ibm/granite-3-8b-instruct",
-    "meta-llama/llama-3-8b-instruct",
+for provider in [
+    "watsonx",
+    "bam",
 ]:
-    if model_name.startswith("ibm"):
-        format = SystemFormat(
-            demo_format=(
-                "{instruction}\\N{source}\\N<|end_of_text|>\n"
-                "<|start_of_role|>assistant<|end_of_role|>{target}\\N<|end_of_text|>\n"
-                "<|start_of_role|>user<|end_of_role|>"
-            ),
-            model_input_format=(
-                "<|start_of_role|>system<|end_of_role|>{system_prompt}<|end_of_text|>\n"
-                "<|start_of_role|>user<|end_of_role|>{demos}{instruction}\\N{source}\\N<|end_of_text|>\n"
-                "<|start_of_role|>assistant<|end_of_role|>"
-            ),
-        )
-        batch_sizes = [50, 30, 10, 1]
-
-    if model_name.startswith("meta-llama"):
-        format = "formats.llama3_instruct"
-        batch_sizes = [100, 50, 10, 1]
-
-    for batch_size in batch_sizes:
-        card, _ = fetch_artifact("cards.sst2")
-        card.preprocess_steps.extend(
-            [
-                CollateInstances(batch_size=batch_size),
-                Rename(field_to_field={"text": "texts", "label": "labels"}),
-                Copy(field="text_type/0", to_field="text_type"),
-                Copy(field="classes/0", to_field="classes"),
-                Copy(field="type_of_class/0", to_field="type_of_class"),
+    for model_name in [
+        "granite-3-8b-instruct",
+        "llama-3-8b-instruct",
+    ]:
+        batch_sizes = [30, 20, 10, 5, 1]
+
+        for batch_size in batch_sizes:
+            card, _ = fetch_artifact("cards.banking77")
+            card.preprocess_steps.extend(
+                [
+                    CollateInstances(batch_size=batch_size),
+                    Rename(field_to_field={"text": "texts", "label": "labels"}),
+                    Copy(field="text_type/0", to_field="text_type"),
+                    Copy(field="classes/0", to_field="classes"),
+                    Copy(field="type_of_class/0", to_field="type_of_class"),
+                ]
+            )
+            card.task = task
+            card.templates = [template]
+            format = "formats.chat_api"
+            if provider == "bam" and model_name.startswith("llama"):
+                format = "formats.llama3_instruct"
+            if provider == "bam" and model_name.startswith("granite"):
+                format = SystemFormat(
+                    demo_format=(
+                        "{instruction}\\N{source}\\N<|end_of_text|>\n"
+                        "<|start_of_role|>assistant<|end_of_role|>{target}\\N<|end_of_text|>\n"
+                        "<|start_of_role|>user<|end_of_role|>"
+                    ),
+                    model_input_format=(
+                        "<|start_of_role|>system<|end_of_role|>{system_prompt}<|end_of_text|>\n"
+                        "<|start_of_role|>user<|end_of_role|>{demos}{instruction}\\N{source}\\N<|end_of_text|>\n"
+                        "<|start_of_role|>assistant<|end_of_role|>"
+                    ),
+                )
+
+            dataset = load_dataset(
+                card=card,
+                template_card_index=0,
+                format=format,
+                num_demos=1,
+                demos_pool_size=5,
+                loader_limit=1000,
+                max_test_instances=200 / batch_size,
+            )
+
+            test_dataset = dataset["test"]
+            from unitxt.inference import CrossProviderInferenceEngine
+
+            inference_model = CrossProviderInferenceEngine(
+                model=model_name, max_tokens=1024, provider=provider
+            )
+            """
+            We are using a CrossProviderInferenceEngine inference engine that supply api access to provider such as:
+            watsonx, bam, openai, azure, aws and more.
+
+            For the arguments these inference engines can receive, please refer to the classes documentation or read
+            about the the open ai api arguments the CrossProviderInferenceEngine follows.
+            """
+            predictions = inference_model.infer(test_dataset)
+
+            evaluated_dataset = evaluate(predictions=predictions, data=test_dataset)
+            # import pandas as pd
+            # result_df = pd.json_normalize(evaluated_dataset)
+            # result_df.to_csv(f"output.csv")
+            # Print results
+            print_dict(
+                evaluated_dataset[0],
+                keys_to_print=[
+                    "source",
+                    "prediction",
+                    "processed_prediction",
+                    "processed_references",
+                ],
+            )
+
+            global_scores = evaluated_dataset[0]["score"]["global"]
+            df.loc[len(df)] = [
+                provider,
+                model_name,
+                batch_size,
+                global_scores["num_of_instances"],
+                global_scores["score"],
+                global_scores["score_ci_low"],
+                global_scores["score_ci_high"],
+                1.0 - global_scores["in_classes_support"],
             ]
-        )
-        card.task = task
-        card.templates = [template]
-
-        dataset = load_dataset(
-            card=card,
-            template_card_index=0,
-            format=format,
-            num_demos=1,
-            demos_pool_size=5,
-            loader_limit=10000,
-            max_test_instances=1000 / batch_size,
-        )
-
-        test_dataset = dataset["test"]
-
-        # inference_model = IbmGenAiInferenceEngine(
-        #    model_name=model_name, max_new_tokens=1024
-        # )
-
-        from unitxt.inference import WMLInferenceEngine
-
-        inference_model = WMLInferenceEngine(model_name=model_name, max_new_tokens=1024)
-
-        predictions = inference_model.infer(test_dataset)
-
-        evaluated_dataset = evaluate(predictions=predictions, data=test_dataset)
-
-        # Print results
-        print_dict(
-            evaluated_dataset[0],
-            keys_to_print=[
-                "source",
-                "prediction",
-                "processed_prediction",
-                "processed_references",
-            ],
-        )
-
-        global_scores = evaluated_dataset[0]["score"]["global"]
-        df.loc[len(df)] = [
-            model_name,
-            batch_size,
-            global_scores["num_of_instances"],
-            global_scores["score"],
-            global_scores["score_ci_low"],
-            global_scores["score_ci_high"],
-        ]
-
-        df = df.round(decimals=2)
-        logger.info(df.to_markdown())
+
+            df = df.round(decimals=2)
+            logger.info(df.to_markdown())

examples/evaluate_benchmark.py

@@ -1,7 +1,7 @@
 from unitxt.api import evaluate
 from unitxt.benchmark import Benchmark
 from unitxt.inference import (
-    HFPipelineBasedInferenceEngine,
+    CrossProviderInferenceEngine,
 )
 from unitxt.standard import StandardRecipe
 from unitxt.text_utils import print_dict
@@ -47,11 +47,17 @@
 test_dataset = list(benchmark()["test"])
 
 
-# Infere using flan t5 base using HF API
-model_name = "google/flan-t5-base"
-inference_model = HFPipelineBasedInferenceEngine(
-    model_name=model_name, max_new_tokens=32
+# Infere using llama-3-2-1b base using Watsonx API
+inference_model = CrossProviderInferenceEngine(
+    model="llama-3-2-1b-instruct", provider="watsonx"
 )
+"""
+We are using a CrossProviderInferenceEngine inference engine that supply api access to provider such as:
+watsonx, bam, openai, azure, aws and more.
+
+For the arguments these inference engines can receive, please refer to the classes documentation or read
+about the the open ai api arguments the CrossProviderInferenceEngine follows.
+"""
 
 predictions = inference_model.infer(test_dataset)
 evaluated_dataset = evaluate(predictions=predictions, data=test_dataset)