Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add multi api inference engine #1343

Merged
merged 33 commits into from
Nov 19, 2024
Merged

Add multi api inference engine #1343

merged 33 commits into from
Nov 19, 2024

Conversation

elronbandel
Copy link
Member

@elronbandel elronbandel commented Nov 12, 2024 •
edited
Loading

With multi api inference engine you can feed into your application:

     MyClass(
           model=CrossProviderInferenceEngine("llama-3.1-405b", api="aws")
      )

Then change it by editing the model.api argument or by general settings:
You can set unitxt.settings.default_provider="watsonx"
or export UNITXT_DEFAULT_PROVIDER="watsonx"

@elronbandel
Add multi api inference engine
de868ab 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix
c40d87d 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Set to greedy decoding
d53eb69 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Merge branch 'main' into multi-api-engine
e4a0799
yoavkatz
yoavkatz reviewed Nov 17, 2024
View reviewed changes
src/unitxt/catalog/engines/model/llama_3_8b_instruct.json Outdated
@@ -0,0 +1,12 @@
{
"__type__": "multi_api_inference_engine",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if adding to the catalog is needed. The API mapping can be done in a single place, and not repeated per model.

@elronbandel
Merge branch 'main' into multi-api-engine
e41a0fa
@elronbandel
Some fixes
b36b7ab 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix consistency and preparation
0593788 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Update
28bafa2 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Merge branch 'main' into multi-api-engine
ccc72ae
@elronbandel
Merge branch 'main' into multi-api-engine
06991ca
@elronbandel
Fix test
3c861fb 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Merge branch 'multi-api-engine' of https://github.com/IBM/unitxt into…
086aae8 
… multi-api-engine
@elronbandel
Make all args None
f9cd539 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Try
4165c78 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix grammar
f202c3a 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix
bd8e176 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel elronbandel requested a review from yoavkatz November 18, 2024 13:45
elronbandel and others added 6 commits November 18, 2024 15:54
@elronbandel
Change api to provider
b686f95 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Merge branch 'main' into multi-api-engine
b4dfe3b
@yoavkatz
Added support for param renaming.
4c91d5e 
Added BAM and improved error messages.

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
Fix merge issues
eaead52 
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@yoavkatz
Updated to CrossProviderModel
4c5ba45 
Signed-off-by: Yoav Katz <katz@il.ibm.com>
@elronbandel
Update name back to InferenceEngine terminology
00dbd30 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Align all examples with chat api and cross provider engines
a0373f8 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Add vllm inference engine
4fa6f8e 
Signed-off-by: elronbandel <elronbandel@gmail.com>
elronbandel and others added 7 commits November 19, 2024 11:00
@elronbandel
Fix blue bench to use cross provider engine
8115091 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Merge branch 'main' into multi-api-engine
986d268
@yoavkatz
Added watsonx-sdk to MultiProviderInferenceEngine
728fcc3 
Add example to evaluate same datasets  and models with multiple providers and formats

Signed-off-by: Yoav Katz <katz@il.ibm.com>
@elronbandel
Make hf tests deterministic
9414f54 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix llmaj with chat api
69388b5 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Add inference documentation
e921b01 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel
Fix examples
9946ab6 
Signed-off-by: elronbandel <elronbandel@gmail.com>
yoavkatz
yoavkatz reviewed Nov 19, 2024
View reviewed changes
elronbandel and others added 2 commits November 19, 2024 17:18
@elronbandel
Fix examples
ececf85 
Signed-off-by: elronbandel <elronbandel@gmail.com>
@elronbandel @yoavkatz
Update docs/docs/inference.rst
71365e7 
Co-authored-by: Yoav Katz <68273864+yoavkatz@users.noreply.github.com>
@elronbandel elronbandel merged commit ccc5338 into main Nov 19, 2024
18 checks passed
@elronbandel elronbandel deleted the multi-api-engine branch November 19, 2024 15:37
ShirApp pushed a commit that referenced this pull request Nov 28, 2024
@elronbandel @yoavkatz @ShirApp
Add multi api inference engine (#1343)
1243918 
* Add multi api inference engine

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Set to greedy decoding

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Some fixes

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix consistency and preparation

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix test

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Make all args None

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Try

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix grammar

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Change api to provider

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Added support for param renaming.

Added BAM and improved error messages.

Signed-off-by: Yoav Katz <katz@il.ibm.com>

* Fix merge issues

Signed-off-by: Yoav Katz <katz@il.ibm.com>

* Updated to CrossProviderModel

Signed-off-by: Yoav Katz <katz@il.ibm.com>

* Update name back to InferenceEngine terminology

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Align all examples with chat api and cross provider engines

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add vllm inference engine

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix blue bench to use cross provider engine

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Added watsonx-sdk to MultiProviderInferenceEngine

Add example to evaluate same datasets  and models with multiple providers and formats

Signed-off-by: Yoav Katz <katz@il.ibm.com>

* Make hf tests deterministic

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix llmaj with chat api

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Add inference documentation

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix examples

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Fix examples

Signed-off-by: elronbandel <elronbandel@gmail.com>

* Update docs/docs/inference.rst

Co-authored-by: Yoav Katz <68273864+yoavkatz@users.noreply.github.com>

---------

Signed-off-by: elronbandel <elronbandel@gmail.com>
Signed-off-by: Yoav Katz <katz@il.ibm.com>
Co-authored-by: Yoav Katz <katz@il.ibm.com>
Co-authored-by: Yoav Katz <68273864+yoavkatz@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yoavkatz yoavkatz yoavkatz approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@elronbandel @yoavkatz