The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.
-
Updated
Dec 12, 2024 - Python
#
flash-attention
Here are 23 public repositories matching this topic...
Official release of InternLM2.5 base and chat models. 1M context support
chatbot chinese gpt pretrained-models llm long-context rlhf large-language-model flash-attention fine-tuning-llm
-
Updated
Nov 21, 2024 - Python
📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉
sora llm llms vllm llm-inference awesome-llm flash-attention flash-attention-2 tensorrt-llm paged-attention deepseek open-sora flash-attention-3
-
Updated
Dec 22, 2024
FlashInfer: Kernel Library for LLM Serving
-
Updated
Dec 25, 2024 - Cuda
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
pytorch multi-modal gemma pipeline-parallelism transformers-models tensor-parallelism llava llm-training internlm flash-attention zero3 llm-framework sequence-parallelism internlm2 ring-attention deepspeed-ulysses llama3 910b
-
Updated
Dec 20, 2024 - Python
The official CLIP training codebase of Inf-CL: "Breaking the Memory Barrier: Near Infinite Batch Size Scaling for Contrastive Loss". A super memory-efficiency CLIP training scheme.
-
Updated
Oct 30, 2024 - Python
Train llm (bloom, llama, baichuan2-7b, chatglm3-6b) with deepspeed pipeline mode. Faster than zero/zero++/fsdp.
nlp bloom pipeline pytorch deepspeed llm full-finetune model-parallization flash-attention llama2 baichuan2-7b chatglm3-6b mixtral-8x7b
-
Updated
Feb 5, 2024 - Python
Triton implementation of FlashAttention2 that adds Custom Masks.
deep-learning triton attention cuda-kernels attention-mechanism triton-lang flash-attention flash-attention-2
-
Updated
Aug 14, 2024 - Python
Performance of the C++ interface of flash attention and flash attention v2 in large language model (LLM) inference scenarios.
gpu cuda inference nvidia cutlass mha multi-head-attention llm tensor-core large-language-model flash-attention flash-attention-2
-
Updated
Sep 7, 2024 - C++
Decoding Attention is specially optimized for multi head attention (MHA) using CUDA core for the decoding stage of LLM inference.
gpu cuda inference nvidia mha multi-head-attention llm large-language-model flash-attention cuda-core decoding-attention flashinfer
-
Updated
Nov 5, 2024 - C++
Python package for rematerialization-aware gradient checkpointing
-
Updated
Oct 31, 2023 - Python
Utilities for efficient fine-tuning, inference and evaluation of code generation models
-
Updated
Oct 3, 2023 - Python
Fast and memory efficient PyTorch implementation of the Perceiver with FlashAttention.
-
Updated
Nov 4, 2024 - Python
Flash Attention Implementation with Multiple Backend Support and Sharding This module provides a flexible implementation of Flash Attention with support for different backends (GPU, TPU, CPU) and platforms (Triton, Pallas, JAX).
-
Updated
Dec 3, 2024 - Python
An simple pytorch implementation of Flash MultiHead Attention
artificial-intelligence transformer attention artificial-neural-networks attention-mechanisms attentionisallyouneed gpt4 flash-attention
-
Updated
Feb 5, 2024 - Jupyter Notebook
🚀 Automated deployment stack for AMD MI300 GPUs with optimized ML/DL frameworks and HPC-ready configurations
machine-learning deep-learning hpc axolotl slurm conda gpu-computing rocm deepspeed pytorch-rocm flash-attention amd-mi300
-
Updated
Nov 30, 2024 - Shell
Long term project about a custom AI architecture. Consist of cutting-edge technique in machine learning such as Flash-Attention, Group-Query-Attention, ZeRO-Infinity, BitNet, etc.
python machine-learning deep-learning jupyter-notebook pytorch transformer bitnet pytorch-lightning vision-transformer large-language-models low-rank-adaptation flash-attention
-
Updated
Oct 15, 2024 - Python
Poplar implementation of FlashAttention for IPU
-
Updated
Mar 12, 2024 - C++
Training GPT-2 on FineWeb-Edu in JAX/Flax
-
Updated
Aug 20, 2024 - Python
Improve this page
Add a description, image, and links to the flash-attention topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the flash-attention topic, visit your repo's landing page and select "manage topics."