Skip to content

Commit

Permalink
version number update, docs update
Browse files Browse the repository at this point in the history
  • Loading branch information
@aarora79
aarora79 committed Sep 10, 2024
1 parent 1db3cdd commit d5210f9
Show file tree
Hide file tree
Showing 10 changed files with 146 additions and 57 deletions.
59 changes: 28 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,31 +56,37 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl

### Full list of benchmarked models

| Model | EC2 g5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2/Trn1 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:--------------|:---------------------|:---------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | ✅ | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | | ✅ | ✅ | ✅ | | | ✅ | |
| **Llama3.1-70b instruct** | | ✅ | | ✅ | | | ✅ | |
| **Llama3-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | ✅ | |
| **Gemma-2b** | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | ✅ | | | | | |
| Model | EC2 g5 | EC2 p4 | EC2 p5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2/Trn1 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:-------|:-------|:--------------|:---------------------|:--------------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | | | ✅ | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | ✅ | |
| **Llama3.1-70b instruct** | | | | ✅ | | ✅ | | | ✅ | |
| **Llama3-8b instruct** | ✅ | | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | | | ✅ | |
| **Gemma-2b** | | | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | | | ✅ | | | | | |

## New in this release

## 2.0.6

1. Run multiple model copies with the DJL serving container and an Nginx load balancer on Amazon EC2.
1. Config files for `Llama3.1-8b` on `g5`, `p4de` and `p5` Amazon EC2 instance types.
1. Better analytics for creating internal leaderboards.

## 2.0.5

1. Support for Intel CPU based instances such as `c5.18xlarge` and `m5.16xlarge`.
Expand All @@ -89,15 +95,6 @@ Llama3 is now available on SageMaker (read [blog post](https://aws.amazon.com/bl

1. Support for AMD CPU based instances such as `m7a`.

## 2.0.3

1. Support for a EFA directory for benchmarking on EC2.






[Release history](./release_history.md)

## Getting started
Expand Down
44 changes: 22 additions & 22 deletions docs/benchmarking.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,25 @@

## Full list of benchmarked models

| Model | EC2 g5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2/Trn1 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:--------------|:---------------------|:---------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | ✅ | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | | ✅ | | ✅ | | | ✅ | |
| **Llama3.1-70b instruct** | | ✅ | | ✅ | | | ✅ | |
| **Llama3-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | ✅ | |
| **Gemma-2b** | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | ✅ | | | | | |
| Model | EC2 g5 | EC2 p4 | EC2 p5 | EC2 Inf2/Trn1 | SageMaker g4dn/g5/p3 | SageMaker Inf2/Trn1 | SageMaker P4 | SageMaker P5 | Bedrock On-demand throughput | Bedrock provisioned throughput |
|:--------------------------------|:-------|:-------|:-------|:--------------|:---------------------|:--------------------|:-------------|:-------------|:-----------------------------|:--------------------------------|
| **Anthropic Claude-3 Sonnet** | | | | | | | | | ✅ | ✅ |
| **Anthropic Claude-3 Haiku** | | | | | | | | | ✅ | |
| **Mistral-7b-instruct** | | | | ✅ | ✅ | | ✅ | ✅ | ✅ | |
| **Mistral-7b-AWQ** | | | | | | | | ✅ | | |
| **Mixtral-8x7b-instruct** | | | | | | | | | ✅ | |
| **Llama3.1-8b instruct** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | | ✅ | |
| **Llama3.1-70b instruct** | | | | ✅ | | ✅ | | | ✅ | |
| **Llama3-8b instruct** | ✅ | | | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | |
| **Llama3-70b instruct** | ✅ | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-13b chat** | | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Llama2-70b chat** | | | | | ✅ | ✅ | ✅ | | ✅ | |
| **Amazon Titan text lite** | | | | | | | | | ✅ | |
| **Amazon Titan text express** | | | | | | | | | ✅ | |
| **Cohere Command text** | | | | | | | | | ✅ | |
| **Cohere Command light text** | | | | | | | | | ✅ | |
| **AI21 J2 Mid** | | | | | | | | | ✅ | |
| **AI21 J2 Ultra** | | | | | | | | | ✅ | |
| **Gemma-2b** | | | | | ✅ | | | | | |
| **Phi-3-mini-4k-instruct** | | | | | ✅ | | | | | |
| **distilbert-base-uncased** | | | | | ✅ | | | | | |
Loading

0 comments on commit d5210f9

Please sign in to comment.