Chunker v0.1.0

Full Changelog: https://github.com/revelrylabs/text_chunker_ex/commits/v0.1.0

Chunker v0.1.0 - 02/27/2024

Introducing Chunker, a context-aware text chunking library for Elixir.

Key Features:

• Semantic Chunking: Prioritizes splitting text into meaningful blocks based on format for better embedding and RAG performance.
• Configurable: Options for chunk size, overlap, and text format.
• Metadata Tracking: Automatic tracking of chunk byte ranges.
• Extensible Design: Foundation for adding more chunking strategies.

Getting Started

1. Add to mix.exs:

   def deps do
     [
       {:chunker, "~> 0.1.0"}
     ]
   end

2. Basic Usage:

   alias Chunker.TextChunker
   text = "Your text to be split..."
   chunks = TextChunker.split(text) 

Learn More

• GitHub Repository: https://github.com/revelrylabs/text_chunker_ex
• Documentation: Hex docs documentation

Thank You to Contributors

• Stuart Page
• Jason Pollentier
• Revelry Labs

Feedback Welcome!