Skip to content

Commit

Permalink
README updates, bump up version number
Browse files Browse the repository at this point in the history
  • Loading branch information
@aarora79
aarora79 committed Aug 6, 2024
1 parent 04cc80e commit 0535390
Show file tree
Hide file tree
Showing 7 changed files with 68 additions and 45 deletions.
53 changes: 29 additions & 24 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,29 +42,38 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl

### Full list of benchmarked models

| Model | SageMaker g4dn/g5/p3 | SageMaker Inf2 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:------------------|:-----------------|:----------------|:--------------|:--------------|:------------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | ✅ | |
| **Mistral-7b-instruct** |✅ | |✅ |✅ | ✅ | |
| **Mistral-7b-AWQ** || | |✅ | | |
| **Mixtral-8x7b-instruct** | | | | | ✅ | |
| **Llama3-8b instruct** |✅ |✅|✅ | ✅|✅ | |
| **Llama3-70b instruct** |✅ |✅|✅ | |✅ | |
| **Llama2-13b chat** |✅ |✅ |✅ | | ✅ | |
| **Llama2-70b chat** |✅ |✅ |✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | ✅ | |
| **Amazon Titan text express** | | | | | ✅ | |
| **Cohere Command text** | | | | | ✅ | |
| **Cohere Command light text** | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | ✅ | |
| **Gemma-2b** |✅ | | | | | |
| **Phi-3-mini-4k-instruct** |✅ | | | | | |
| **distilbert-base-uncased** | ✅ | | | || |
| Model | EC2 g5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:--------------|:---------------------|:---------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | | | | | | | ✅ | |
| **Llama3.1-70b instruct** | | | | | | | ✅ | |
| **Llama3-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | ✅ | |
| **Gemma-2b** | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | ✅ | | | | | |



## New in this release

## v1.0.51

1. `FMBench` has a [website](https://aws-samples.github.io/foundation-model-benchmarking-tool/index.html) now. Rework the README file to make it lightweight.
1. `Llama3.1` config files for Bedrock.

## v1.0.50
1. `Llama3-8b` on Amazon EC2 `inf2.48xlarge` config file.
1. Update to new version of DJL LMI (0.28.0).
Expand All @@ -74,10 +83,6 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl
1. Per-token latency metrics such as time to first token (TTFT) and mean time per-output token (TPOT).
1. Misc. bug fixes.

### v1.0.48
1. Faster result file download at the end of a test run.
1. `Phi-3-mini-4k-instruct` configuration file.
1. Tokenizer and misc. bug fixes.

[Release history](./release_history.md)

Expand Down
43 changes: 23 additions & 20 deletions docs/benchmarking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,26 @@

## Full list of benchmarked models

| Model | SageMaker g4dn/g5/p3 | SageMaker Inf2 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:------------------|:-----------------|:----------------|:--------------|:--------------|:------------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | ✅ | |
| **Mistral-7b-instruct** |✅ | |✅ |✅ | ✅ | |
| **Mistral-7b-AWQ** || | |✅ | | |
| **Mixtral-8x7b-instruct** | | | | | ✅ | |
| **Llama3-8b instruct** |✅ |✅|✅ | ✅|✅ | |
| **Llama3-70b instruct** |✅ |✅|✅ | |✅ | |
| **Llama2-13b chat** |✅ |✅ |✅ | | ✅ | |
| **Llama2-70b chat** |✅ |✅ |✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | ✅ | |
| **Amazon Titan text express** | | | | | ✅ | |
| **Cohere Command text** | | | | | ✅ | |
| **Cohere Command light text** | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | ✅ | |
| **Gemma-2b** |✅ | | | | | |
| **Phi-3-mini-4k-instruct** |✅ | | | | | |
| **distilbert-base-uncased** | ✅ | | | || |
| Model | EC2 g5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:--------------|:---------------------|:---------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | | | | | | | ✅ | |
| **Llama3.1-70b instruct** | | | | | | | ✅ | |
| **Llama3-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | ✅ | |
| **Gemma-2b** | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | ✅ | | | | | |

3 changes: 3 additions & 0 deletions docs/manifest.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ Here is a listing of the various configuration files available out-of-the-box wi

**bedrock**
[├── bedrock/config-bedrock-claude.yml](configs/bedrock/config-bedrock-claude.yml)
[├── bedrock/config-bedrock-llama3-1-70b-streaming.yml](configs/bedrock/config-bedrock-llama3-1-70b-streaming.yml)
[├── bedrock/config-bedrock-llama3-1-8b-streaming.yml](configs/bedrock/config-bedrock-llama3-1-8b-streaming.yml)
[├── bedrock/config-bedrock-llama3-1.yml](configs/bedrock/config-bedrock-llama3-1.yml)
[├── bedrock/config-bedrock-llama3-streaming.yml](configs/bedrock/config-bedrock-llama3-streaming.yml)
[├── bedrock/config-bedrock-llama3.yml](configs/bedrock/config-bedrock-llama3.yml)
[├── bedrock/config-bedrock-titan-text-express.yml](configs/bedrock/config-bedrock-titan-text-express.yml)
Expand Down
4 changes: 4 additions & 0 deletions docs/releases.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Releases

## v1.0.51
1. `FMBench` has a [website](https://aws-samples.github.io/foundation-model-benchmarking-tool/index.html) now. Rework the README file to make it lightweight.
1. `Llama3.1` config files for Bedrock.

## v1.0.50
1. `Llama3-8b` on Amazon EC2 `inf2.48xlarge` config file.
1. Update to new version of DJL LMI (0.28.0).
Expand Down
3 changes: 3 additions & 0 deletions manifest.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@ configs/bedrock/config-bedrock-llama3-streaming.yml
configs/bedrock/config-bedrock-llama3.yml
configs/bedrock/config-bedrock-titan-text-express.yml
configs/bedrock/config-bedrock.yml
configs/bedrock/config-bedrock-llama3-1-70b-streaming.yml
configs/bedrock/config-bedrock-llama3-1-8b-streaming.yml
configs/bedrock/config-bedrock-llama3-1.yml
configs/bert/config-distilbert-base-uncased.yml
configs/byoe/config-model-byo-sagemaker-endpoint.yml
configs/gemma/config-gemma-2b-g5.yml
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "fmbench"
version = "1.0.50"
version = "1.0.51"
description ="Benchmark performance of **any Foundation Model (FM)** deployed on **any AWS Generative AI service**, be it **Amazon SageMaker**, **Amazon Bedrock**, **Amazon EKS**, or **Amazon EC2**. The FMs could be deployed on these platforms either directly through `FMbench`, or, if they are already deployed then also they could be benchmarked through the **Bring your own endpoint** mode supported by `FMBench`."
authors = ["Amit Arora <aroraai@amazon.com>", "Madhur prashant <Madhurpt@amazon.com>"]
readme = "README.md"
Expand Down
5 changes: 5 additions & 0 deletions release_history.md
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
### v1.0.48
1. Faster result file download at the end of a test run.
1. `Phi-3-mini-4k-instruct` configuration file.
1. Tokenizer and misc. bug fixes.

### v1.0.47
1. Run `FMBench` as a Docker container.
1. Bug fixes for GovCloud support.
Expand Down

0 comments on commit 0535390

Please sign in to comment.