|
| 1 | + |
| 2 | +# 🤖 Quest: I want to build a local Gen AI prototype |
| 3 | + |
| 4 | +> To reset your progress and select a different quest, click this button: |
| 5 | +> |
| 6 | +> [](../../issues/new?title=Reset+Progress:+Back+to+Welcome&labels=reset&body=🔄+I+want+to+reset+my+progress+and+go+back+to+the+Welcome+page.%0A%0A**Please+click+on+Create+below,+then+wait+about+15+seconds.+Your+progress+will+be+reset,+this+issue+will+automatically+close,+and+you+will+be+taken+back+to+the+Welcome+page+to+select+a+new+quest.**) |
| 7 | +
|
| 8 | +## 📋 Pre-requisites |
| 9 | + |
| 10 | +1. A GitHub account |
| 11 | +2. [Visual Studio Code](https://code.visualstudio.com/) installed |
| 12 | +3. [Node.js](https://nodejs.org/en) installed |
| 13 | + |
| 14 | +## 📝 Overview |
| 15 | +You will build a **local Gen AI prototype** using JavaScript and TypeScript. This prototype will allow you to experiment with different AI models, parameters, and prompts. |
| 16 | + |
| 17 | +## 🧠 GitHub Models |
| 18 | + |
| 19 | +GitHub Models is a FREE service that provides access to a variety of AI models from different providers and a playground to experiment with them. You can use these models to build your own AI applications (prototypes), or just to learn more about how AI works. |
| 20 | + |
| 21 | +With GitHub Models, you can use GitHub Personal Access Tokens (PAT) to authenticate and access the models locally, or use a single API key to access all the models. |
| 22 | + |
| 23 | +1. Right click [GitHub Models](https://github.com/marketplace/models) and open in a new tab |
| 24 | + |
| 25 | + ‼️ Open the link IN A NEW TAB so you can keep this page open for reference.‼️ |
| 26 | + |
| 27 | +2. Click on **explore the full model catalog** to see the available models. |
| 28 | + |
| 29 | +  |
| 30 | + |
| 31 | + You will see a broad range of models listed in the catalog. |
| 32 | + |
| 33 | + 🤔 _But which model should you use for what?_ |
| 34 | + |
| 35 | +3. Scroll down to the **Filter section** to see the available filters. You can filter the models by: |
| 36 | + - **Publisher:** Cohere, DeepSeek, Meta, Mistral AI, Microsoft _(research)_, Azure OpenAI Service, and more. |
| 37 | + - **Category:** Conversation _(models optimized for dialogue use cases)_, Agents, Multimodal _(models capable of processing input in multiple formats - audio, visual etc.)_, Reasoning, and more. |
| 38 | + |
| 39 | +4. Select a model from the list to open the model card. The model card provides detailed information about the model, and may include: |
| 40 | + |
| 41 | + ### A. README |
| 42 | + - **Model Abstract:** A brief description of the model and its capabilities. |
| 43 | + - **Model Architecture:** The data used to train the model and their modalities for input and output _(text-image pairs)_, the model size _(parameters)_, model context length _(how much text the model can process at once)_, training date _(knowledge cut-off date/data freshness for the model)_, supported languages, and more. |
| 44 | + |
| 45 | +  |
| 46 | + |
| 47 | + ### B. Transparency Note |
| 48 | + - **Model Use cases:** Primary and out-of-scope use cases for the model, responsible AI considerations, content filtering configurations and more. |
| 49 | + |
| 50 | +  |
| 51 | + |
| 52 | + ### C. License |
| 53 | + - **Model License:** The license under which the model is released, including any restrictions on use or distribution. |
| 54 | + |
| 55 | +  |
| 56 | + |
| 57 | + ### D. Evaluation Report |
| 58 | + - **Model Benchmarks:** A summary of the model's performance on various benchmarks, including accuracy, speed, and other relevant metrics. |
| 59 | + - **Model Benchmarks:** A list of benchmarks used to evaluate the model's performance, including details on how the model performed on each benchmark. Metrics may include: |
| 60 | + - MMLU Pass@1 (Measuring Massive Multitask Language Understanding) - _Knowledge and reasoning across science, math, and humanities_. |
| 61 | + - DROP - _Measures reading comprehension and numerical reasoning capabilities_ |
| 62 | + - among others |
| 63 | + |
| 64 | +  |
| 65 | + |
| 66 | +5. After selecting a model and reviewing the model card, you can use the **Playground** to experiment with the model. The playground provides a user-friendly interface for testing the model's capabilities and understanding how it works. |
| 67 | + |
| 68 | +  |
| 69 | + |
| 70 | + You can directly send questions (prompts) to the model and see how it responds. Throughout the session, you can monitor the token usage and the model's response time at the top of the chat UI. |
| 71 | + |
| 72 | +  |
| 73 | + |
| 74 | +6. To check your token usage against your GitHub Models free quota (input/ output tokens, latency), click on the **Input: Output: Time** note at the top right of the chat UI to open Model usage insights. |
| 75 | + |
| 76 | +  |
| 77 | + |
| 78 | +7. Before going further, on the right side of the playground, switch from **Details** to **Parameters** to see the available parameters that you can adjust to change the model's behavior. |
| 79 | + |
| 80 | +  |
| 81 | + |
| 82 | + The parameters include: |
| 83 | + - **Max Tokens:** The maximum number of tokens the model can generate in response to a prompt. Adjusting this parameter can help control the length of the model's output. |
| 84 | + - **Temperature:** Controls the randomness of the model's output. A higher temperature (e.g., 0.8) makes the output more random, while a lower temperature (e.g., 0.2) makes it more focused and deterministic. |
| 85 | + - **Top P:** Controls the diversity of the model's output. A higher value (e.g., 0.9) allows for more diverse outputs, while a lower value (e.g., 0.1) makes the output more focused on the most likely tokens. |
| 86 | + - **Presence Penalty:** Controls the model's tendency to repeat itself. A higher value (e.g., 1.0) discourages repetition, while a lower value (e.g., 0.0) allows for more repetition. |
| 87 | + - **Frequency Penalty:** Similar to the penalty penalty, this parameter controls the model's tendency to repeat the same words or phrases. |
| 88 | + - **Stop:** A list of tokens that, when generated, will stop the model's output. |
| 89 | + |
| 90 | + You can continue interacting with the model in the playground, as you adjust the parameters to ensure you get the desired output. |
| 91 | + |
| 92 | + 🤔 _How do models compare across different prompts and parameters?_ |
| 93 | + |
| 94 | +7. GitHub Models provides a **Compare** feature that allows you to compare the performance of different models on the same prompt. This is useful for understanding how different models respond to the same input and can help you choose the best model for your specific use case. |
| 95 | + |
| 96 | + Click on the **Compare** button at the top right of the playground. |
| 97 | + |
| 98 | +  |
| 99 | + |
| 100 | + Select the models you want to compare from the list of available models from the drop-down. |
| 101 | + |
| 102 | + This will open a chat UI for the selected models side by side, and your prompt will be sent to both models. |
| 103 | + |
| 104 | +  |
| 105 | + |
| 106 | + In the example provided, you can compare the performance of a reasoning model and a conversation model on the same prompt to understand their strengths and limitations. |
| 107 | + |
| 108 | +## 👨💻 Playground to VS Code |
| 109 | + |
| 110 | +Now that you have a better understanding of the models from the GitHub Models playground, let's look at how to use them in JavaScript code. |
| 111 | + |
| 112 | +### Clone the repository |
| 113 | + |
| 114 | +1. Clone the repository to your local machine using the following command: |
| 115 | + |
| 116 | + ```bash |
| 117 | + git clone <your-repo-url> |
| 118 | + ``` |
| 119 | + Replace `<your-repo-url>` with the URL of your GitHub repository. |
| 120 | + |
| 121 | +2. Open the cloned repository in Visual Studio Code. |
| 122 | + |
| 123 | + ```bash |
| 124 | + cd <your-repo-name> |
| 125 | + code . |
| 126 | + ``` |
| 127 | + Replace `<your-repo-name>` with the name of your cloned repository. |
| 128 | + |
| 129 | +### Get sample code |
| 130 | + |
| 131 | +1. On the far right, click on **Use this model** and select **Language: JavaScript** and **SDK: Azure AI Inference SDK**. |
| 132 | + |
| 133 | +  |
| 134 | + |
| 135 | + Follow the instructions provided to: |
| 136 | + - Get a free developer key, (Personal Access Token (classic)) and store it in an environment variable either using bash, PowerShell or command line. |
| 137 | + - Install dependencies |
| 138 | + - Run basic code sample. **Ensure your local file is named `sample.js`** |
| 139 | + |
| 140 | +  |
| 141 | + |
| 142 | +## 📌 Exercise: Convert a hand-drawn sketch to a web page |
| 143 | + |
| 144 | +1. Download the contoso website hand-drawn sketch from [this link](https://github.com/juliamuiruri4/JS-Journey-to-AI-Foundry/blob/assets/js-ai-journey-assets/contoso_layout_sketch.jpg), and save it `contoso_layout_sketch.jpg` in the same directory as your `sample.js` file. |
| 145 | + |
| 146 | + > **Note** |
| 147 | + > |
| 148 | + > If you aren't using a muiltimodal model, swap out the `modelName` in the code sample with a multimodal model of your choice. You can find a list of multimodal models in the GitHub Models catalog. |
| 149 | +
|
| 150 | + 2. Update the code to pass the image to the model as input. |
| 151 | +
|
| 152 | + > **Note** |
| 153 | + > You can use [GitHub Copilot](https://github.com/features/copilot) to help you with this task. |
| 154 | + > |
| 155 | + >  |
| 156 | +
|
| 157 | + 3. Run the code and check the output in the console. |
| 158 | +
|
| 159 | +  |
| 160 | +
|
| 161 | +## 🧰 Use AI Toolkit in VS Code |
| 162 | +
|
| 163 | +The AI Toolkit in Visual Studio Code is a powerful extension that provides a set of tools and features to help you build AI applications more efficiently. |
| 164 | +
|
| 165 | +1. Click on the **Extensions** icon in the left sidebar of Visual Studio Code, search for **AI Toolkit** and **install**. |
| 166 | +
|
| 167 | +2. Similar to GitHub Models, with the AI Toolkit now installed, you can browse through the catalog of available models, and use the **Playground** to experiment with the models, all on VS Code. |
| 168 | +
|
| 169 | +  |
| 170 | +
|
| 171 | + Let's execute the exercise above using the AI Toolkit in VS Code. |
| 172 | + |
| 173 | +3. Select a multimodal model from the catalog and open the **Playground**. |
| 174 | + |
| 175 | + In the playground, upload the `contoso_layout_sketch.jpg` image and enter a prompt to write the HTML code for the website. |
| 176 | + |
| 177 | +4. On the generated code, click on the **New file** icon to copy the generated code into a new file. Save it as `index.html` in the same directory as your `sample.js` file. |
| 178 | + |
| 179 | +  |
| 180 | + |
| 181 | + Do the same for the CSS code and save it as `style.css` in the same directory. |
| 182 | + |
| 183 | + You can preview the generated code and iterate on the code to improve it (optionally using GitHub Copilot). |
| 184 | + |
| 185 | +  |
| 186 | + |
| 187 | +## ✅ Activity: Push `sample.js` code to your repository |
| 188 | + |
| 189 | +### Quest Checklist |
| 190 | + |
| 191 | +To complete this quest and **AUTOMATICALLY UPDATE** your progress, you MUST push your code to the repository as described below. |
| 192 | + |
| 193 | +**Checklist** |
| 194 | +- [ ] Have a `sample.js` file at the root of your project |
| 195 | +- [ ] The file MUST include a reference to your GITHUB_TOKEN environment variable |
| 196 | + |
| 197 | +1. In the terminal, run the following commands to add, commit, and push your changes to the repository: |
| 198 | + |
| 199 | + ```bash |
| 200 | + git add . |
| 201 | + git commit -m "Working with GitHub Models and AI Toolkit" |
| 202 | + git push |
| 203 | + ``` |
| 204 | + |
| 205 | +2. After pushing your changes, **WAIT ABOUT 15 SECONDS FOR GITHUB ACTIONS TO UPDATE YOUR README**. |
| 206 | + |
| 207 | + |
| 208 | +> To skip this quest and select a different one, click this button: |
| 209 | +> |
| 210 | +> [](../../issues/new?title=Skip+Quest:+Back+to+Welcome&labels=reset&body=🔄+I+want+to+skip+this+quest+and+go+back+to+the+Welcome+page.%0A%0A**Please+click+on+Create+below,+then+wait+about+15+seconds.+Your+progress+will+be+reset,+this+issue+will+automatically+close,+and+you+will+be+taken+back+to+the+Welcome+page+to+select+a+new+quest.**) |
| 211 | + |
| 212 | +## 📚 Further Reading |
| 213 | + |
| 214 | +Here are some additional resources to help you learn more about experimenting with AI models and building prototypes: |
| 215 | + |
| 216 | +- [Lesson 1: Introduction to Generative AI and LLMs for JavaScript Developers](https://github.com/microsoft/generative-ai-with-javascript/tree/main/lessons/01-intro-to-genai#lesson-1-introduction-to-generative-ai-and-llms-for-javascript-developers) |
| 217 | +- [Lesson 2: Writing your first AI app](https://github.com/microsoft/generative-ai-with-javascript/blob/main/lessons/02-first-ai-app/README.md#lesson-2-writing-your-first-ai-app) |
| 218 | +- [Lesson 3: Prompt Engineering](https://github.com/microsoft/generative-ai-with-javascript/blob/main/lessons/03-prompt-engineering/README.md) |
0 commit comments