Named entity recognition in Malayalam using BiLSTM and TENER (Transformer Encoder)

This is project for sequence to sequence NLP task. We developed a custom model to understand the process of task using PyTorch. We also fine tuned pre-trained transformer models to improve the performance of translation task.

Transformer implementation in pytorch trained on NVIDIA A100 in fp16

A byte-level Byte Pair Encoding (BPE) algorithm for tokenization in Large Language Models (LLMs), similar to those used in GPT, Llama, and Mistral.

Byte pair encoding tokenizer as used in some large language models.

This repository houses my assignments completed during the Deep Learning course as part of my Master's in Data Analytics program. Explore diverse projects showcasing hands-on applications of advanced neural networks and machine learning techniques.

Morphologically biased byte-pair encoding

This project aims to implement word-based, character-based and subword-based tokenization techniques.

An implementation of the GPT(generative pretrained transformer) model, from scratch, which produces Shakespearean text by training on the dialogues written by Shakespeare along with the GPT Encoder.

An Introduction to Natural Language Processing (NLP)

Fast BPE algorithm to generate byte pair encodings from text corpus, it's written in rust and approximately 20x faster than it's python implementation

Byte-pair encoding implementation in Python.

an efficient ranked retrieval system for English corpora, optimised with VBE and BPE.

Code for the publication of WWW'22

Modern Eager TensorFlow implementation of Attention Is All You Need

Order-agnostic lossless compressor using BPE and Huffman Coding.

A Visualizer to check how BPE Tokenizer in an LLM Works

Auto summarization from BPE tokenization