Skip to content

paceaux/methodius

Repository files navigation

Methodius (an NGram utility)

A utility for analyzing frequency of text chunks on the web.

Supply a bit o' text to the Methodius class, and let it determine your bigrams, trigrams, ngrams, letter-frequencies, word frequencies, bigram relationships, and create ngram trees.

Hippocratic License HL3-LAW-MEDIA-MIL-SOC-SV

npm

Example

const { Methodius } = require('methodius');
// or import { Methodius } from 'methodius';

const udhr1 = `
All human beings are born free and equal in dignity and rights. They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood.
`;
const nGrams = new Methodius(udhr1);

const topLetters = nGrams.getTopLetters(10);
const topWords = nGrams.getTopWords(10);

API

Methodius

Global Class

new Methodius(text)

Parameters

name type Description
text string raw text to be analyzed

Static Members

Punctuations

characters to ignore when analyzing text period, comma, semicolon, colon, bang, question mark, interrobang, Spanish bang+, parens, bracket, brace, single quote, some spaces

\\.,;:!?‽¡¿⸘()\\[\\]{}<>’'…\"\n\t\r

wordSeparators

characters to ignore AND CONSUME when trying to find words em-dash, period, comma, semicolon, colon, bang, question mark, interrobang, Spanish bang+, parens, bracket, brace, single quote, space

—\\.,;:!?‽¡¿⸘()\\[\\]{}<>…"\\s

Static Methods

hasPunctuation(string)

determines if string contains punctuation

Parameters

name type Description
string string

Returns boolean

hasSymbols(string)

determines if string contains symbols

Parameters

name type Description
string string

Returns boolean

hasSpace(string)

determines if a string has a space

Parameters

name type Description
string string

Returns boolean

sanitizeText(string)

lowercases text and removes diacritics and other characters that would throw off n-gram analysis

Parameters

name type Description
string string

Returns string

getWords(text)

extracts an array of words from a string

Parameters

name type Description
text string

Returns Array<string>

getNGrams(text, gramSize)

gets ngrams from text

Parameters

name type Description
text string
gramSize Number Default = 2

Returns Array<string>

getMeanWordSize(wordArray)

Gets average size of a word

Parameters

name type Description
wordArray string[]

Returns number

getMedianWordSize(wordArray)

Gets the median (middle) size of a word

Parameters

name type Description
wordArray string[]

Returns number

getWordNGrams(text)

Gets 2-word pairs from text.

Note: This doesn't use sentence punctuation as a boundary. Should it?

Parameters

name type Description
text string
gramSize number default=2

Returns Array<string>

getFrequencyMap(frequencyMap)

converts an array of strings into a map of those strings and number of occurences

Parameters

name type Description
ngramArray Array<string>

Returns Map<string, number>

getPercentMap(frequencyMap)

converts a frequency map into a map of percentages

Parameters

name type Description
frequencyMap Map<string, number>

Returns Map<string, number>

getTopGrams(frequencyMap)

filters a frequency map into only a small subset of the most frequent ones

Parameters

name type Description
frequencyMap Map<string, number>
limit number default=20

Returns Map<string, number>

getIntersection(iterable1, iterable2)

Returns an array of items that occur in both iterables

Parameters

name type Description
iterable1 `Map Array`
iterable2 `Map Array`

Returns Array<any> An array of items that occur in both iterables. It will compare the keys, if sent a map

getUnion(iterable1, iterable2)

Returns an array that is the union of two iterables

Parameters

name type Description
iterable1 `Map Array`
iterable2 `Map Array`

Returns Array<any> A union of the items that occur in both iterables.

getDisjunctiveUnion(iterable1, iterable2)

Returns an array of arrays of the unique items in either iterable. Also known as the symmetric difference

Parameters

name type Description
iterable1 `Map Array`
iterable2 `Map Array`

Returns Array<Array<any> An array of arrays of the unique items. The first item is the first parameter, 2nd item second param

getDifference

Returns an array of items that are unique only to the first parameter.

Parameters

name type Description
iterable1 `Map Array`
iterable2 `Map Array`

Returns Array<Array<any> An array of items unique only to the first parameter

getComparison(iterable1, iterable2)

Returns a map containing various comparisons between two iterables

Parameters

name type Description
iterable1 `Map Array`
iterable2 `Map Array`

Returns Map<string, <array>> A map containing various comparisons between two iterables. Those comparisons will be arrays of intersection, disjunctiveUnion, difference, and union.

getWordPlacementForNGram(ngram, wordsArray)

determines the placement of a single ngram in an array of words

Parameters

name type Description
ngram string
wordsArray Array<string>

Returns Map<string, number> a map with the keys 'start', 'middle', and 'end' whose values correspond to how often the provided ngram occurs in this position

getWordPlacementForNGrams(ngrams, wordsArray)

determines the placement of ngrams in an array of words

Parameters

name type Description
ngram Array<string>
wordsArray Array<string>

Returns Map<string, Map<string, number>> a map with the key of the ngram, and the value that is a map containing start, middle, end

getNgramCollections(ngrams, wordsArray)

gets ngrams from an array of words

Parameters

name type Description
wordArray Array<string> an array of words
ngramSize number default = 2. The size of the ngrams to return

Returns Array<Array<string>> An array containing arrays of ngrams, each array corresponds to a word.

getNgramSiblings(searchText, ngramCollections, siblingSize)

using a collection returned from getNgramCollections, searches for a string and returns what comes before and after it

Parameters

name type Description
searchText string the string to search for
ngramCollections `Array Array<Array>`
siblingSize number default = 1. How many siblings to find in front or behind

Returns Map<'before'|'after',Map<string, number>> a Map with the keys 'before' and 'after' which contain maps of what comes before and after

Example

        const words = ['revolution', 'nation'];
        const ngramCollections = Methodius.getNgramCollections(words, 2);
        const onSiblings = Methodius.getNgramSiblings('io', ngramCollections);
        /* 
        new Map([
          ['before', new Map(
            ['ti', 2]
          )],
          ['after', new Map(
            ['on', 2]
          )]
        ])
        */

getRelatedNgrams(words, ngrams, ngramSize)

Gets the ngrams that will occur before or after other ngrams. Useful for finding patterns of ngrams.

Parameters

name type Description
words Array<string> an array of words to evaluate
ngrams Map<string, number> a frequency map of ngrams
ngramSize number default = 2. the size of the ngram

Returns

Map<string, number> A frequency map of how often ngrams occured before or after other ngrams

Example

This requires several steps. You'll need an array of words and a frequency map of ngrams.

    const ngrams = getNGrams('the revolution of the nation was on television. It was about pollution and the terrible situation ', 2);
    const frequencyMap = getFrequencyMap(ngrams);
    const topNgrams = getTopGrams(frequencyMap, 5);
    const words = ['the', 'revolution', 'of', 'the', 'nation', 'was', 'on', 'television', 'it', 'was', 'about', 'pollution', 'and', 'the', 'terrible', 'situation' ];
    const relatedNgrams = getRelatedNgrams(words, topNgrams, 2, 5);

getNgramTreeCollection(words)

Gets a nested map of maps that breaks down unique words into their smallest ngrams

Parameters

name type Description
words Array<string> an array of words to evaluate

Returns

Map<string, Array<string>| Map<string, <Array|string>> A nested map of maps that breaks down unique words into their smallest ngrams.

Instance Members

sanitizedText

lowercased text with diacritics removed

string

letters

an array of letters in the text

Array<string>

words

an array of words in the text

Array<string>

bigrams

an array of letter bigrams in the text

Array<string>

trigrams

an array of letter trigrams in the text

Array<string>

uniqueLetters

an array of unique letters in the text

Array<string>

uniqueBigrams

an array of unique bigrams in the text

Array<string>

uniqueTrigrams

an array of unique trigrams in the text

Map<string, Map<string, number>>

letterPositions

a map of placements of letters within words

Map<string, Map<string, number>>

bigramPositions

a map of placements of bigrams within words

Map<string, Map<string, number>>

uniqueTrigrams

a map of placements of trigrams within words

Array<string>

uniqueWords

an array of unique words in the text

Array<string>

letterFrequencies

a map of letter frequencies in the sanitized text

Map<string, number>

bigramFrequencies

a map of bigram frequencies in the sanitized text

Map<string, number>

trigramFrequencies

a map of trigram frequencies in the sanitized text

Map<string, number>

wordFrequencies

a map of word frequencies in the sanitized text

Map<string, number>

letterPercentages

a map of letter percentages in the sanitized text

Map<string, number>

bigramPercentages

a map of bigram percentages in the sanitized text

Map<string, number>

trigramPercentages

a map of trigram percentages in the sanitized text

Map<string, number>

wordPercentages

a map of word percentages in the sanitized text

Map<string, number>

meanWordSize

The average size of a word

number

medianWordSize

The middle size of a word

number

ngramTreeCollection

A nested map of maps that breaks down unique words into their smallest ngrams.

Instance Methods

getLetterNGrams(size)

gets an array of customizeable ngrams in the text

Parameters

name type Description
size number default = 2 size of the n-gram to return

Returns Array<string>

getTopLetters(limit)

a map of the most used letters in the text

Parameters

name type Description
limit number default = 20 number of top letters to return

Returns Map<string, number>

getTopBigrams(limit)

a map of the most used bigrams in the text

Parameters

name type Description
limit number default = 20 number of top bigrams to return

Returns Map<string, number>

getTopTrigrams(limit)

a map of the most used trigrams in the text

Parameters

name type Description
limit number default = 20 number of top trigrams to return

Returns Map<string, number>

getTopWords(limit)

a map of the most used words in the text

Parameters

name type Description
limit number default = 20 number of top words to return

Returns Map<string, number>

compareTo(methodius)

Compare this methodius instance to another

Parameters

name type Description
methodius Methodius another Methodius instance

Returns Map<string, Map> A map of property names and their comparisons (intersection, disjunctiveUnions, etc) for a set of properties

getRelatedTopNgrams(ngramSize, limit)

Gets the ngrams that will occur before or after other ngrams based on what the most frequent ngrams are. Useful for finding patterns of ngrams.

Parameters

name type Description
ngramSize number default = 2. the size of the ngram
limit number default = 20. the number of top ngrams to use

Returns

Map<string, number> A frequency map of how often the most common ngrams occured before or after other common ngrams