This repository is a collection of educational materials gathered while studying generative artificial intelligence (AI). In a field that is always changing like generative AI, staying updated with key information is crucial for building skills, choosing competencies, sharing examples, and showing experimental results.
This repository is helpful for anyone who wants to learn, dive deeper, and effectively use generative AI technologies in a structured way. It covers a wide range of topics, from high-level concepts that are easy for beginners and quickly applicable, like prompting, to technical fundamentals like neural networking.
Whether you are a beginner or an experienced professional, this repository aims to provide valuable resources to support your study in the world of generative AI and to share your study materials with others
AI key professional roles
🏆 LMSYS Chatbot Arena Leaderboard is a crowdsourced open platform for LLM evals. They collected over 1,000,000 human pairwise comparisons to rank LLMs with the Bradley-Terry model and display the model ratings in Elo-scale.
- JUNE, 2023 The Rise of the AI Engineer “In numbers, there's probably going to be significantly more AI Engineers than there are ML engineers / LLM engineers. One can be quite successful in this role without ever training anything.” - Andrej Karpathy
- MAY, 2023 Meta’s Bizarre AI Infrastructure Choice Costs Them $100s of Millions In March 2023, the open-source community gained access to Meta's LLaMA model, a powerful foundation model. Despite lacking advanced tuning and RLHF, its potential was quickly recognized, leading to rapid innovation. Within a month, significant advancements were made, including instruction tuning, quality improvements, and multimodality. Crucially, the scaling problem was solved, allowing individuals to experiment with the model using just a powerful laptop, dramatically lowering the barrier to entry for training and experimentation.
| Date | Event | Key Points |
|---|---|---|
| Feb 24, 2023 | LLaMA Launched | Meta releases LLaMA, open-sourcing code but not weights. Not instruction or conversation tuned. |
| Mar 3, 2023 | LLaMA Leaked | LLaMA leaked to public. Increased experimentation due to open access. |
| Mar 12, 2023 | LLaMA on Raspberry Pi | Artem Andreenko runs LLaMA on Raspberry Pi, initiating minification efforts. |
| Mar 13, 2023 | Alpaca Released | Stanford releases Alpaca, adding instruction tuning to LLaMA. Eric Wang's alpaca-lora repo enables low-cost fine-tuning. |
| Mar 18, 2023 | LLaMA on MacBook CPU | Georgi Gerganov runs LLaMA on MacBook CPU using 4-bit quantization. |
| Mar 19, 2023 | Vicuna Released | Cross-university collaboration releases Vicuna, claiming "parity" with Bard. Training cost: $300. |
| Mar 25, 2023 | GPT4All Released | Nomic creates GPT4All, an ecosystem for various models including Vicuna. Training cost: $100. |
| Mar 28, 2023 | Open Source GPT-3 | Cerebras trains GPT-3 architecture using optimal compute schedule and μ-parameterization. |
| Mar 28, 2023 | LLaMA-Adapter | Introduces instruction tuning and multimodality in one hour of training using PEFT. |
| Apr 3, 2023 | Koala Released | Berkeley launches Koala, demonstrating human preference close to ChatGPT. Training cost: $100. |
| Apr 15, 2023 | Open Assistant | Launches model and dataset for Alignment via RLHF, achieving results close to ChatGPT. |
To get started, check out the Generative AI Glossary for key terms and concepts
To enhance your training, review this proposal on a training path available on Notion
To improve your coding skills, explore this proposal on Colab code
- Prompt engineering guide
- Learn prompting
- OpenAI prompt engineering
- Best practices for prompt engineering with the OpenAI API
- Anthropic prompt engineering
- Awesome-Prompt-Engineering
- Prompt Engineering 201: Advanced methods and toolkits
- The Illustrated Transformer
- Glossary of deep learning word embedding
- wevi: word embedding visual inspector
- Word Embedding Explained and Visualized - word2vec and wevi
- Google Colab Notebook: hello_t2t.ipynb
- AskFSDL create an AI app - askFSDL is a demonstration of a retrieval-augmented question-answering application
- OpenAI tokenizer
- Google colab Cloud-based platform for running Jupyter notebooks with access to powerful computing resources like GPUs. You can use Google Colab to develop your first MVP app.
- Tiktoken is a fast BPE tokeniser for use with OpenAI's models.
- DAIR.AI Democratizing Artificial Intelligence Research, Education, and Technologies
- Microsoft Generative AI for Beginners
- Openai cookbook
This repository encourages contributions! Whether you have additional resources to share, suggestions for improvement, or feedback on existing content, you can participate by opening a pull request or creating an issue.
For questions, suggestions, or collaborations, please feel free to contact us directly through GitHub or Discord
This repository includes links to materials distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.
- Articles written by Jay Alammar and published on https://jalammar.github.io are available under the CC BY-NC-SA 4.0 license.
- Denis Marini