ContextCalc 📏

tree like CLI tool and library for token counting.

contextcalc.mp4

Table of Contents

• ContextCalc 📏 - tree like CLI tool but with token counts.
  • Table of Contents
  • Installation
  • Quick Start
  • Input Sources
    • Directory Analysis (default)
    • Single File Analysis
    • Stdin Pipe
    • Clipboard Content
  • Output Formats
    • Tree View (default)
    • Flat View
    • JSON Output
    • CSV Output
    • AST Output (Code Symbol Extraction)
  • Options Reference
  • Usage Examples
    • Basic Usage
    • File Filtering
    • Output Customization
    • Performance & Analysis
    • Input Source Comparison
    • Advanced Workflows
  • Library Usage
    • Installation as Library
    • Basic API
    • Common Use Cases
    • Function Reference
    • TypeScript Support
  • License

Installation

npm install -g contextcalc
# or
npx contextcalc [path] [options]

Quick Start

# Tree view
npx contextcalc .

# Flat file list sorted by token count  
npx contextcalc . --output flat

# Flat file list sorted by token count (only .txt, .md and etc.)
npx contextcalc . --mode docs

# Tree with visual bars
npx contextcalc . --bars

Input Sources

ContextCalc supports multiple input sources for maximum flexibility:

Directory Analysis (default)

Analyze entire directories with hierarchical structure:

npx contextcalc ./src --depth 2
npx contextcalc . --output flat --min-tokens 1000 # Get the biggest files

Single File Analysis

Analyze individual files directly:

npx contextcalc README.md
npx contextcalc src/cli.ts --metrics tokens,lines,size

Stdin Pipe

Pipe content from other commands or files:

cat README.md | npx contextcalc
echo "Your text here" | npx contextcalc --metrics tokens,lines
git diff | npx contextcalc --metrics tokens --no-colors

Clipboard Content

Analyze content directly from your clipboard:

# Copy some text, then analyze it:
npx contextcalc --from-clipboard
npx contextcalc --from-clipboard --metrics tokens
npx contextcalc --from-clipboard --metrics size --no-colors

Output Formats

Tree View (default)

Shows hierarchical structure with absolute percentages:

npx contextcalc . --depth 2

 ~/git/contextcalc ╱ main  npx -y contextcalc@latest --depth 2 .                                                        ✔ ╱ 06:59:13 
. (23.4k tokens, 87.6KB) (100.0%)
└── src (12.8k tokens, 50.0KB) (54.5%)
    ├── src/formatters (4.0k tokens, 14.9KB) (17.0%)
    ├── src/core (2.9k tokens, 12.4KB) (12.5%)
    ├── src/cli.ts (2.8k tokens, 339 lines, 12.5KB) (12.1%)
    ├── src/utils (2.5k tokens, 8.5KB) (10.8%)
    ├── src/types (484 tokens, 1.8KB) (2.1%)
└── test (4.7k tokens, 17.4KB) (20.0%)
    ├── test/enhancedTreeFormatter.test.ts (2.0k tokens, 240 lines, 7.2KB) (8.7%)
    ├── test/cli.test.ts (1.8k tokens, 229 lines, 7.1KB) (7.8%)
    ├── test/fileDetector.test.ts (371 tokens, 33 lines, 1.5KB) (1.6%)
    ├── test/pathUtils.test.ts (223 tokens, 19 lines, 793B) (1.0%)
    ├── test/hasher.test.ts (218 tokens, 26 lines, 832B) (0.9%)
└── README.md (2.2k tokens, 283 lines, 7.8KB) (9.4%)
└── CHANGELOG.md (1.1k tokens, 62 lines, 2.9KB) (4.8%)
└── CLAUDE.md (747 tokens, 111 lines, 2.5KB) (3.2%)
└── .github (592 tokens, 2.2KB) (2.5%)
    ├── .github/workflows (592 tokens, 2.2KB) (2.5%)
└── package.json (454 tokens, 55 lines, 1.3KB) (1.9%)
└── eslint.config.mjs (281 tokens, 44 lines, 1.1KB) (1.2%)
└── LICENSE (221 tokens, 21 lines, 1.0KB) (0.9%)
└── tsconfig.json (193 tokens, 29 lines, 713B) (0.8%)
└── .releaserc.json (103 tokens, 14 lines, 413B) (0.4%)
└── .claude (46 tokens, 129B) (0.2%)
    ├── .claude/settings.local.json (46 tokens, 10 lines, 129B) (0.2%)
└── index.ts (8 tokens, 1 lines, 30B) (0.0%)

Summary: 23.4k tokens, 31 files, 87.6KB
Completed in 0.21s

Flat View

Lists all files sorted by token count (perfect for finding large files):

npx contextcalc . --output flat --min-tokens 500

src/formatters/enhancedTreeFormatter.ts 1.8k tokens, 226 lines, 6.5KB (13.1%)
src/core/scanner.ts 1.5k tokens, 226 lines, 6.7KB (11.5%)
src/cli.ts 1.2k tokens, 131 lines, 5.3KB (9.3%)
src/utils/fileDetector.ts 922 tokens, 123 lines, 3.0KB (6.7%)

JSON Output

Structured data for scripts and automation:

npx contextcalc . --output json

CSV Output

Export data in comma-separated values format for spreadsheets and data analysis:

npx contextcalc . --output csv

Path,Tokens,Lines,Size(bytes),Type,Percentage
src/cli.ts,2834,339,12504,ts,12.1
src/formatters/enhancedTreeFormatter.ts,1847,226,6498,ts,7.9
src/core/scanner.ts,1544,226,6706,ts,6.6
README.md,2208,283,7806,md,9.4
"Total",23421,1247,87604,"",100.0

AST Output (Code Symbol Extraction)

Extract and display code structure as symbols (functions, classes, interfaces, etc.) using tree-sitter AST parsing:

npx contextcalc . --output ast

src/core/scanner.ts (1750 tokens, 258 lines, 7.6KB)
├─ ← from "node:path" { join, relative } line 1
├─ C DirectoryScanner lines 11-258
│  ├─ v cache line 12
│  ├─ v tokenizer line 13
│  ├─ ƒ constructor(projectPath: string, mode: AnalysisMode, ...): void lines 19-32
│  ├─ ƒ async initialize(useGitignore: boolean, ...): Promise<void> lines 34-45
│  ├─ ƒ async scan(): Promise<ScanResult> lines 47-69
│  └─ ƒ static calculatePercentages(nodes: Node[]): Node[] lines 237-257
└─ c CACHE_VERSION line 7

Found 326 symbols across 16 files

Supported Languages:

Language	Extensions	Functions	Classes	Interfaces	Types	Enums	Imports/Exports
TypeScript	.ts, .tsx	✅	✅	✅	✅	✅	✅
JavaScript	.js, .jsx, .mjs, .cjs	✅	✅	❌	❌	❌	✅
Python	.py, .pyi	✅	✅	❌	❌	❌	✅

Notes:

• JavaScript supports all symbol types except interfaces, types, and enums (not part of JS spec)
• Python supports functions, classes, and imports; interfaces/types/enums are not native Python constructs
• All languages support nested symbols (e.g., class methods, nested functions)

Symbol Icons:

• ƒ Function/Method - with parameters, return types, async/generator flags
• C Class - with extends, implements, abstract modifiers
• I Interface - with extends and members
• T Type - type aliases and definitions
• E Enum - with member values
• v Variable - with type annotations
• c Constant - with type annotations
• ← Import - showing source and imported items
• → Export - showing exported items

Features:

• Performance: AST results are cached - subsequent runs are fast
• Selective: Only enabled with --output ast flag
• Smart: Respects --max-size limit to prevent OOM on large files
• Compatible: Works with all existing filters (--mode, --depth, --min-tokens)

Examples:

# Analyze TypeScript/JavaScript project structure
npx contextcalc src --output ast --mode code

# Find all classes and functions (depth 2 to see class members)
npx contextcalc . --output ast --depth 2

# Analyze Python codebase
npx contextcalc . --output ast --mode code

Options Reference

Option	Description	Default
-o, --output <format>	Output format: tree, flat, json, csv, ast	tree
--mode <mode>	Files to analyze: all, code, docs	all
--max-size <size>	Maximum file size to analyze (e.g., 10M, 500k)	10M
--sort <by>	Sort by: tokens, size, name	tokens
--depth <n>	Tree depth levels (0=root only, 1=root+children, etc.)	unlimited
--min-tokens <n>	Hide files with fewer than n tokens	0
--metrics <list>	Metrics to display: tokens,lines,size,percentage	tokens,lines,size,percentage
--percentages	Show absolute percentages (enabled by default)	true
--no-percentages	Disable percentage display
--relative-percentages	Show percentages relative to parent directory	false
--bars	Show visual weight bars	false
--no-colors	Disable colored output
--no-gitignore	Ignore .gitignore files
--no-default-ignores	Disable default ignore patterns
--from-clipboard	Read content from system clipboard

Usage Examples

Basic Usage

# Analyze current directory
npx contextcalc .

# Analyze specific directory
npx contextcalc ./src

# Analyze single file
npx contextcalc README.md

# Get help
npx contextcalc --help

File Filtering

# Only analyze code files (.js, .ts, .py, etc.)
npx contextcalc . --mode code

# Only analyze documentation files (.md, .txt, .rst, etc.)
npx contextcalc . --mode docs

# Limit by file size (skip large files)
npx contextcalc . --max-size 1M

# Hide small files (less than 100 tokens)
npx contextcalc . --min-tokens 100

# Ignore .gitignore patterns
npx contextcalc . --no-gitignore

# Disable all default ignore patterns
npx contextcalc . --no-default-ignores

Output Customization

# Flat list sorted by token count
npx contextcalc . --output flat

# Tree view with visual bars
npx contextcalc . --bars

# Limit tree depth
npx contextcalc . --depth 2

# Sort by file size instead of tokens
npx contextcalc . --sort size

# Sort alphabetically
npx contextcalc . --sort name

# Show only specific metrics
npx contextcalc . --metrics tokens,lines

# Clean output for scripts
npx contextcalc . --no-colors --no-percentages

# Relative percentages (% of parent directory)
npx contextcalc . --relative-percentages

Performance & Analysis

# Find the largest files in your project
npx contextcalc . --output flat --min-tokens 1000

# Export analysis data for further processing
npx contextcalc . --output json > analysis.json
npx contextcalc . --output csv > analysis.csv

# Quick overview of just top-level directories
npx contextcalc . --depth 1

# Focus on heavyweight files only
npx contextcalc . --output flat --min-tokens 2000 --sort tokens

Input Source Comparison

# All these should give the same token count for the same content:

# File analysis
npx contextcalc README.md --metrics tokens

# Stdin pipe
cat README.md | npx contextcalc --metrics tokens

# Copy README.md content to clipboard, then:
npx contextcalc --from-clipboard --metrics tokens

Advanced Workflows

# Git workflow: analyze staged changes
git diff --cached | npx contextcalc --metrics tokens --no-colors

# Find files that exceed token limits for LLM context
npx contextcalc . --output flat --min-tokens 4000

# Analyze documentation structure
npx contextcalc ./docs --mode docs --bars --depth 3

# Compare token usage between branches
git show main:README.md | npx contextcalc --metrics tokens
git show feature-branch:README.md | npx contextcalc --metrics tokens

# Process multiple files through stdin
find . -name "*.md" -exec cat {} \; | npx contextcalc --metrics tokens,lines

# Generate reports for different file types
npx contextcalc . --mode code --output json > code-analysis.json
npx contextcalc . --mode docs --output csv > docs-analysis.csv

# Clean analysis for CI/CD pipelines
npx contextcalc . --output flat --no-colors --min-tokens 500 | head -10

Library Usage

ContextCalc can be imported and used as a library in your Node.js applications for programmatic token counting.

⚠️ Bundling Notice: If you're bundling an application that uses contextcalc, mark tiktoken as external in your bundler configuration to avoid WebAssembly loading issues. See DOCS.md for detailed guidance.

Installation as Library

npm install contextcalc
# or
yarn add contextcalc
# or
bun add contextcalc

📖 Complete API Reference: See DOCS.md for detailed documentation, advanced examples, and integration guides.

Basic API

import { countTokens, countTokensWithOptions } from 'contextcalc';

// Simple token counting
const tokens = countTokens("Hello, world!");
console.log(tokens); // 4

// Works with any data type
const jsonTokens = countTokens({ message: "Hello", data: [1, 2, 3] });
console.log(jsonTokens); // 24

const bufferTokens = countTokens(Buffer.from("Hello"));
console.log(bufferTokens); // 2

// Advanced options
const result = countTokensWithOptions("Line 1\nLine 2", {
  includeLines: true,
  format: 'formatted'
});
console.log(result); // { tokens: 6, lines: 2, formatted: "6" }

Common Use Cases

import { 
  countTokens,
  countTokensBatch,
  countTokensFromFile,
  estimateTokens
} from 'contextcalc';

// Batch processing multiple inputs
const inputs = ["Hello", "World", { key: "value" }];
const results = countTokensBatch(inputs);
console.log(results); // [2, 1, 8]

// File processing
const fileTokens = await countTokensFromFile('./README.md');
console.log(fileTokens); // 1247

// Quick estimation for large datasets
const estimate = estimateTokens("Large text content...");
console.log(estimate); // Fast approximate count

Function Reference

Function	Purpose	Example
countTokens(input)	Basic token counting	countTokens("Hello") → 2
countTokensBatch(inputs)	Multiple inputs efficiently	countTokensBatch(["a", "b"]) → [1, 1]
countTokensFromFile(path)	File processing	await countTokensFromFile("file.txt")
estimateTokens(input)	Quick estimation	estimateTokens("text") → 1
countWithLines(input)	Include line count	countWithLines("Hi\nBye") → {tokens: 3, lines: 2}

TypeScript Support

import { 
  countTokens,
  type TokenInput,
  type TokenCountResult 
} from 'contextcalc';

const input: TokenInput = "Hello, TypeScript!";
const tokens: number = countTokens(input);

License

MIT