Note:
Tokenization – process of converting a text into tokens Tokens – words or entities present in the text Text object – a sentence or a phrase or a word or an article
import nltk
nltk.download()Text Preprocessing Since, text is the most unstructured form of all the available data, various types of noise are present in it and the data is not readily analyzable without any pre-processing. The entire process of cleaning and standardization of text, making it noise-free and ready for analysis is known as text preprocessing.
It is predominantly comprised of three steps:
Noise Removal Lexicon Normalization Object Standardization
Noise Removal
# Sample code to remove noisy words from a text
noise_list = ["is", "a", "this", "..."]
def _remove_noise(input_text):
words = input_text.split()
noise_free_words = [word for word in words if word not in noise_list]
noise_free_text = " ".join(noise_free_words)
return noise_free_text
_remove_noise("this is a sample text")'sample text'
# Sample code to remove a regex pattern
import re
def _remove_regex(input_text, regex_pattern):
urls = re.finditer(regex_pattern, input_text)
for i in urls:
input_text = re.sub(i.group().strip(), '', input_text)
return input_text
regex_pattern = "#[\w]*"
_remove_regex("remove this #hashtag from analytics vidhya", regex_pattern)'remove this from analytics vidhya'
Lexicon Normalization
#importing required libraries
from nltk.stem.wordnet import WordNetLemmatizer
lem = WordNetLemmatizer()
from nltk.stem.porter import PorterStemmer
stem = PorterStemmer()
word = "multiplying"
print('\n\nStemming\n\n')
print(stem.stem(word))
#>> "multipli"Stemming
multipli
stem.stem(word)'multipli'
Object Standardization Text data often contains words or phrases which are not present in any standard lexical dictionaries. These pieces are not recognized by search engines and models.
Some of the examples are – acronyms, hashtags with attached words, and colloquial slangs. With the help of regular expressions and manually prepared data dictionaries, this type of noise can be fixed, the code below uses a dictionary lookup method to replace social media slangs from a text.
lookup_dict = {'rt':'Retweet', 'dm':'direct message', "awsm" : "awesome", "luv" :"love", "..."}
def _lookup_words(input_text):
words = input_text.split()
new_words = []
for word in words:
if word.lower() in lookup_dict:
word = lookup_dict[word.lower()]
new_words.append(word) new_text = " ".join(new_words)
return new_text
_lookup_words("RT this is a retweeted tweet by Shivam Bansal")
#>> "Retweet this is a retweeted tweet by Shivam Bansal" File "<ipython-input-25-2048df645e48>", line 1
lookup_dict = {'rt':'Retweet', 'dm':'direct message', "awsm" : "awesome", "luv" :"love", "..."}
^
SyntaxError: invalid syntax
import nltk
nltk.download('averaged_perceptron_tagger')
from nltk import word_tokenize, pos_tag
text = "I am learning Natural Language Processing on Analytics Vidhya"
tokens = word_tokenize(text)
print(pos_tag(tokens))
#>>> [('I', 'PRP'), ('am', 'VBP'), ('learning', 'VBG'), ('Natural', 'NNP'),('Language', 'NNP'),
#('Processing', 'NNP'), ('on', 'IN'), ('Analytics', 'NNP'),('Vidhya', 'NNP')][nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data] C:\Users\Welcome\AppData\Roaming\nltk_data...
[nltk_data] Unzipping taggers\averaged_perceptron_tagger.zip.
[('I', 'PRP'), ('am', 'VBP'), ('learning', 'VBG'), ('Natural', 'NNP'), ('Language', 'NNP'), ('Processing', 'NNP'), ('on', 'IN'), ('Analytics', 'NNP'), ('Vidhya', 'NNP')]
Entity Extraction (Entities as features) Entities are defined as the most important chunks of a sentence – noun phrases, verb phrases or both. Entity Detection algorithms are generally ensemble models of rule based parsing, dictionary lookups, pos tagging and dependency parsing. The applicability of entity detection can be seen in the automated chat bots, content analyzers and consumer insights.
Topic Modelling & Named Entity Recognition are the two key entity detection methods in NLP.
A. Named Entity Recognition (NER) The process of detecting the named entities such as person names, location names, company names etc from the text is called as NER. For example :
Sentence – Sergey Brin, the manager of Google Inc. is walking in the streets of New York.
Named Entities – ( “person” : “Sergey Brin” ), (“org” : “Google Inc.”), (“location” : “New York”)
A typical NER model consists of three blocks:
Noun phrase identification: This step deals with extracting all the noun phrases from a text using dependency parsing and part of speech tagging.
Phrase classification: This is the classification step in which all the extracted noun phrases are classified into respective categories (locations, names etc). Google Maps API provides a good path to disambiguate locations, Then, the open databases from dbpedia, wikipedia can be used to identify person names or company names. Apart from this, one can curate the lookup tables and dictionaries by combining information from different sources.
Entity disambiguation: Sometimes it is possible that entities are misclassified, hence creating a validation layer on top of the results is useful. Use of knowledge graphs can be exploited for this purposes. The popular knowledge graphs are – Google Knowledge Graph, IBM Watson and Wikipedia.
B. Topic Modeling Topic modeling is a process of automatically identifying the topics present in a text corpus, it derives the hidden patterns among the words in the corpus in an unsupervised manner. Topics are defined as “a repeating pattern of co-occurring terms in a corpus”. A good topic model results in – “health”, “doctor”, “patient”, “hospital” for a topic – Healthcare, and “farm”, “crops”, “wheat” for a topic – “Farming”.
Latent Dirichlet Allocation (LDA) is the most popular topic modelling technique, Following is the code to implement topic modeling using LDA in python. For a detailed explanation about its working and implementation,
doc1 = "Sugar is bad to consume. My sister likes to have sugar, but not my father."
doc2 = "My father spends a lot of time driving my sister around to dance practice."
doc3 = "Doctors suggest that driving may cause increased stress and blood pressure."
doc_complete = [doc1, doc2, doc3]
doc_clean = [doc.split() for doc in doc_complete]
from gensim import corpora
# Creating the term dictionary of our corpus, where every unique term is assigned an index.
dictionary = corpora.Dictionary(doc_clean)
# Converting list of documents (corpus) into Document Term Matrix using dictionary prepared above.
doc_term_matrix = [dictionary.doc2bow(doc) for doc in doc_clean]
# Creating the object for LDA model using gensim library
Lda = gensim.models.ldamodel.LdaModel
# Running and Training LDA model on the document term matrix
ldamodel = Lda(doc_term_matrix, num_topics=3, id2word = dictionary, passes=50)
# Results
print(ldamodel.print_topics())[(0, '0.029*"driving" + 0.029*"My" + 0.029*"sister" + 0.029*"my" + 0.029*"to" + 0.029*"and" + 0.029*"blood" + 0.029*"stress" + 0.029*"Doctors" + 0.029*"suggest"'), (1, '0.063*"to" + 0.036*"have" + 0.036*"not" + 0.036*"bad" + 0.036*"consume." + 0.036*"likes" + 0.036*"sugar," + 0.036*"Sugar" + 0.036*"but" + 0.036*"father."'), (2, '0.053*"driving" + 0.053*"My" + 0.053*"sister" + 0.053*"my" + 0.053*"to" + 0.053*"around" + 0.053*"lot" + 0.053*"father" + 0.053*"time" + 0.053*"spends"')]
N-Grams as Features A combination of N words together are called N-Grams. N grams (N > 1) are generally more informative as compared to words (Unigrams) as features. Also, bigrams (N = 2) are considered as the most important features of all the others. The following code generates bigram of a text.
def generate_ngrams(text, n):
words = text.split()
output = []
for i in range(len(words)-n+1):
output.append(words[i:i+n])
return output>>> generate_ngrams('this is a sample text', 2)
# [['this', 'is'], ['is', 'a'], ['a', 'sample'], , ['sample', 'text'][['this', 'is'], ['is', 'a'], ['a', 'sample'], ['sample', 'text']]
erm Frequency – Inverse Document Frequency (TF – IDF) TF-IDF is a weighted model commonly used for information retrieval problems. It aims to convert the text documents into vector models on the basis of occurrence of words in the documents without taking considering the exact ordering. For Example – let say there is a dataset of N text documents, In any document “D”, TF and IDF will be defined as –
Term Frequency (TF) – TF for a term “t” is defined as the count of a term “t” in a document “D”
Inverse Document Frequency (IDF) – IDF for a term is defined as logarithm of ratio of total documents available in the corpus and number of documents containing the term T.
TF . IDF – TF IDF formula gives the relative importance of a term in a corpus (list of documents), given by the following formula below. Following is the code using python’s scikit learn package to convert a text into tf idf vectors:
from sklearn.feature_extraction.text import TfidfVectorizer
obj = TfidfVectorizer()
corpus = ['This is sample document.', 'another random document.', 'third sample document text']
X = obj.fit_transform(corpus)
print (X) (0, 7) 0.5844829010200651
(0, 2) 0.5844829010200651
(0, 4) 0.444514311537431
(0, 1) 0.34520501686496574
(1, 1) 0.3853716274664007
(1, 0) 0.652490884512534
(1, 3) 0.652490884512534
(2, 4) 0.444514311537431
(2, 1) 0.34520501686496574
(2, 6) 0.5844829010200651
(2, 5) 0.5844829010200651
Word Embedding (text vectors) Word embedding is the modern way of representing words as vectors. The aim of word embedding is to redefine the high dimensional word features into low dimensional feature vectors by preserving the contextual similarity in the corpus. They are widely used in deep learning models such as Convolutional Neural Networks and Recurrent Neural Networks.
Word2Vec and GloVe are the two popular models to create word embedding of a text. These models takes a text corpus as input and produces the word vectors as output.
Word2Vec model is composed of preprocessing module, a shallow neural network model called Continuous Bag of Words and another shallow neural network model called skip-gram. These models are widely used for all other nlp problems. It first constructs a vocabulary from the training corpus and then learns word embedding representations. Following code using gensim package prepares the word embedding as the vectors.
from gensim.models import Word2Vec
sentences = [['data', 'science'], ['vidhya', 'science', 'data', 'analytics'],['machine', 'learning'], ['deep', 'learning']]
# train the model on your corpus
model = Word2Vec(sentences, min_count = 1)
print(model.similarity('data', 'science'))
#>>> 0.11222489293
print(model['learning'])0.07695229
[-4.23155958e-03 -2.59936438e-03 -1.34786498e-03 -3.81647330e-03
-1.91119977e-03 3.42173665e-03 1.73668913e-03 -1.03614631e-03
1.16325655e-05 4.28344170e-03 3.88916908e-03 4.80439281e-03
-2.30876904e-04 -1.65993383e-03 -2.46129371e-03 -2.83390144e-03
-1.11720164e-03 4.81742201e-03 3.92471626e-03 -3.41004389e-03
-1.36997888e-03 2.59301881e-03 -2.52595288e-03 -1.54679618e-03
2.15128460e-03 7.72733940e-04 -2.99606565e-03 -1.67619903e-03
-4.53938870e-03 4.58160508e-03 4.17015189e-03 -3.52614396e-03
4.13179584e-03 6.14689430e-04 1.96552310e-06 -1.39345985e-03
-6.54549163e-04 3.07444111e-03 -1.12224596e-04 1.87312963e-03
1.50469260e-03 -1.85877303e-04 -4.98030894e-03 -2.26566405e-03
3.86062288e-03 -2.90717860e-03 -3.68980109e-03 9.99477925e-04
-1.59697537e-03 -4.79555083e-03 6.14248158e-04 -1.73279142e-03
-8.28273303e-04 -2.65643862e-03 1.46472230e-04 1.03476911e-03
-4.88899462e-03 -4.04638425e-03 2.82594771e-03 2.03759037e-03
-2.66080443e-03 1.93613314e-03 8.03272182e-04 -3.13281291e-03
-3.28092161e-03 -4.99446690e-03 4.95615834e-03 4.77669528e-03
4.62079421e-03 1.24532613e-03 2.44785770e-04 -1.44386082e-03
-2.32771461e-04 1.34228671e-03 1.02868944e-03 3.64723353e-04
7.70242186e-05 -1.00581604e-03 1.16082176e-03 9.39346210e-05
-5.37181877e-05 -2.21645366e-03 -2.04437482e-03 1.77637639e-03
2.69129081e-03 4.44668205e-03 3.83504736e-03 -3.08015058e-03
-2.04605702e-03 2.68455339e-03 1.33605895e-03 1.93028292e-03
-3.68680828e-03 -4.82220086e-04 -3.68467299e-04 2.02698703e-03
-3.57293431e-03 1.83322304e-03 2.19798461e-03 3.93993640e-03]
C:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py:7: DeprecationWarning: Call to deprecated `similarity` (Method will be removed in 4.0.0, use self.wv.similarity() instead).
import sys
C:\ProgramData\Anaconda3\lib\site-packages\ipykernel_launcher.py:10: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).
# Remove the CWD from sys.path while we load stuff.
Text Classification
import nltk
nltk.download('punkt')
from textblob.classifiers import NaiveBayesClassifier as NBC
from textblob import TextBlob
training_corpus = [
('I am exhausted of this work.', 'Class_B'),
("I can't cooperate with this", 'Class_B'),
('He is my badest enemy!', 'Class_B'),
('My management is poor.', 'Class_B'),
('I love this burger.', 'Class_A'),
('This is an brilliant place!', 'Class_A'),
('I feel very good about these dates.', 'Class_A'),
('This is my best work.', 'Class_A'),
("What an awesome view", 'Class_A'),
('I do not like this dish', 'Class_B')]
test_corpus = [
("I am not feeling well today.", 'Class_B'),
("I feel brilliant!", 'Class_A'),
('Gary is a friend of mine.', 'Class_A'),
("I can't believe I'm doing this.", 'Class_B'),
('The date was good.', 'Class_A'), ('I do not enjoy my job', 'Class_B')]
model = NBC(training_corpus)
print(model.classify("Their codes are amazing."))[nltk_data] Downloading package punkt to
[nltk_data] C:\Users\Welcome\AppData\Roaming\nltk_data...
[nltk_data] Unzipping tokenizers\punkt.zip.
Class_A
#class A
print(model.classify("I don't like their computer."))Class_B
#class B
print(model.accuracy(test_corpus))0.8333333333333334
#Scikit.Learn also provides a pipeline framework for text classification:
from sklearn.feature_extraction.text import TfidfVectorizer
#import TfidfVectorizer from sklearn.metrics
from sklearn.metrics import classification_report
#import classification_report
from sklearn import svm
# preparing data for SVM model (using the same training_corpus, test_corpus from naive bayes example)
train_data = []
train_labels = []
for row in training_corpus:
train_data.append(row[0])
train_labels.append(row[1])
test_data = []
test_labels = []
for row in test_corpus:
test_data.append(row[0])
test_labels.append(row[1])
# Create feature vectors
vectorizer = TfidfVectorizer(min_df=4, max_df=0.9)
# Train the feature vectors
train_vectors = vectorizer.fit_transform(train_data)
# Apply model on test data
test_vectors = vectorizer.transform(test_data)
# Perform classification with SVM, kernel=linear
model = svm.SVC(kernel='linear')
model.fit(train_vectors, train_labels)
prediction = model.predict(test_vectors)
#>>> ['Class_A' 'Class_A' 'Class_B' 'Class_B' 'Class_A' 'Class_A']
print (classification_report(test_labels, prediction)) precision recall f1-score support
Class_A 0.50 0.67 0.57 3
Class_B 0.50 0.33 0.40 3
accuracy 0.50 6
macro avg 0.50 0.50 0.49 6
weighted avg 0.50 0.50 0.49 6
Text Matching / Similarity One of the important areas of NLP is the matching of text objects to find similarities. Important applications of text matching includes automatic spelling correction, data de-duplication and genome analysis etc.
A number of text matching techniques are available depending upon the requirement. This section describes the important techniques in detail.
A. Levenshtein Distance – The Levenshtein distance between two strings is defined as the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character. Following is the implementation for efficient memory computations.
def levenshtein(s1,s2):
if len(s1) > len(s2):
s1,s2 = s2,s1
distances = range(len(s1) + 1)
for index2,char2 in enumerate(s2):
newDistances = [index2+1]
for index1,char1 in enumerate(s1):
if char1 == char2:
newDistances.append(distances[index1])
else:
newDistances.append(1 + min((distances[index1], distances[index1+1], newDistances[-1])))
distances = newDistances
return distances[-1]
print(levenshtein("analyze","analyse"))1
Phonetic Matching – A Phonetic matching algorithm takes a keyword as input (person’s name, location name etc) and produces a character string that identifies a set of words that are (roughly) phonetically similar. It is very useful for searching large text corpuses, correcting spelling errors and matching relevant names. Soundex and Metaphone are two main phonetic algorithms used for this purpose. Python’s module Fuzzy is used to compute soundex strings for different words, for example –
import fuzzy
soundex = fuzzy.Soundex(4)
print(soundex('ankit'))
#>>> “A523”
print(soundex('aunkit'))---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-47-c8cc861c962c> in <module>
----> 1 import fuzzy
2 soundex = fuzzy.Soundex(4)
3 print(soundex('ankit'))
4 #>>> “A523”
5 print(soundex('aunkit'))
ModuleNotFoundError: No module named 'fuzzy'
C. Flexible String Matching – A complete text matching system includes different algorithms pipelined together to compute variety of text variations. Regular expressions are really helpful for this purposes as well. Another common techniques include – exact string matching, lemmatized matching, and compact matching (takes care of spaces, punctuation’s, slangs etc).
D. Cosine Similarity – W hen the text is represented as vector notation, a general cosine similarity can also be applied in order to measure vectorized similarity. Following code converts a text to vectors (using term frequency) and applies cosine similarity to provide closeness among two text.
import math
from collections import Counter
def get_cosine(vec1, vec2):
common = set(vec1.keys()) & set(vec2.keys())
numerator = sum([vec1[x] * vec2[x] for x in common])
sum1 = sum([vec1[x]**2 for x in vec1.keys()])
sum2 = sum([vec2[x]**2 for x in vec2.keys()])
denominator = math.sqrt(sum1) * math.sqrt(sum2)
if not denominator:
return 0.0
else:
return float(numerator) / denominator
def text_to_vector(text):
words = text.split()
return Counter(words)
text1 = 'This is an article on analytics vidhya'
text2 = 'article on analytics vidhya is about natural language processing'
vector1 = text_to_vector(text1)
vector2 = text_to_vector(text2)
cosine = get_cosine(vector1, vector2)4.3 Coreference Resolution Coreference Resolution is a process of finding relational links among the words (or phrases) within the sentences. Consider an example sentence: ” Donald went to John’s office to see the new table. He looked at it for an hour.“
Humans can quickly figure out that “he” denotes Donald (and not John), and that “it” denotes the table (and not John’s office). Coreference Resolution is the component of NLP that does this job automatically. It is used in document summarization, question answering, and information extraction. Stanford CoreNLP provides a python wrapper for commercial purposes
4.4 Other NLP problems / tasks Text Summarization – Given a text article or paragraph, summarize it automatically to produce most important and relevant sentences in order. Machine Translation – Automatically translate text from one human language to another by taking care of grammar, semantics and information about the real world, etc. Natural Language Generation and Understanding – Convert information from computer databases or semantic intents into readable human language is called language generation. Converting chunks of text into more logical structures that are easier for computer programs to manipulate is called language understanding. Optical Character Recognition – Given an image representing printed text, determine the corresponding text. Document to Information – This involves parsing of textual data present in documents (websites, files, pdfs and images) to analyzable and clean format.
- Important Libraries for NLP (python) Scikit-learn: Machine learning in Python Natural Language Toolkit (NLTK): The complete toolkit for all NLP techniques. Pattern – A web mining module for the with tools for NLP and machine learning. TextBlob – Easy to use nl p tools API, built on top of NLTK and Pattern. spaCy – Industrial strength N LP with Python and Cython. Gensim – Topic Modelling for Humans Stanford Core NLP – NLP services and packages by Stanford NLP Group.