Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MY FIRST PR #589

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

MY FIRST PR #589

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
384 changes: 384 additions & 0 deletions Day_5_codes/Day 5 codes/AI chatbot/Chatbot.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,384 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Building a Simple Chatbot from Scratch in Python (using NLTK)\n",
"\n",
"![Alt text](https://cdn-images-1.medium.com/max/800/1*pPcVfZ7i-gLMabUol3zezA.gif)\n",
"\n",
"History of chatbots dates back to 1966 when a computer program called ELIZA was invented by Weizenbaum. It imitated the language of a psychotherapist from only 200 lines of code. You can still converse with it here: [Eliza](http://psych.fullerton.edu/mbirnbaum/psych101/Eliza.htm?utm_source=ubisend.com&utm_medium=blog-link&utm_campaign=ubisend). \n",
"\n",
"On similar lines let's create a very basic chatbot utlising the Python's NLTK library.It's a very simple bot with hardly any cognitive skills,but still a good way to get into NLP and get to know about chatbots.\n",
"\n",
"For detailed analysis, please see the accompanying blog titled:**[Building a Simple Chatbot in Python (using NLTK](https://medium.com/analytics-vidhya/building-a-simple-chatbot-in-python-using-nltk-7c8c8215ac6e)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## NLP\n",
"NLP is a way for computers to analyze, understand, and derive meaning from human language in a smart and useful way. By utilizing NLP, developers can organize and structure knowledge to perform tasks such as automatic summarization, translation, named entity recognition, relationship extraction, sentiment analysis, speech recognition, and topic segmentation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Import necessary libraries"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import random\n",
"import string # to process standard python strings\n",
"import warnings\n",
"import numpy as np\n",
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"from sklearn.metrics.pairwise import cosine_similarity\n",
"import warnings\n",
"warnings.filterwarnings('ignore')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Downloading and installing NLTK\n",
"NLTK(Natural Language Toolkit) is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning, wrappers for industrial-strength NLP libraries.\n",
"\n",
"[Natural Language Processing with Python](http://www.nltk.org/book/) provides a practical introduction to programming for language processing.\n",
"\n",
"For platform-specific instructions, read [here](https://www.nltk.org/install.html)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Requirement already satisfied: nltk in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (3.4.1)\n",
"Requirement already satisfied: six in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from nltk) (1.12.0)\n",
"Note: you may need to restart the kernel to use updated packages.\n"
]
}
],
"source": [
"pip install nltk"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Installing NLTK Packages\n",
"\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"showing info https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml\n"
]
},
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import nltk\n",
"from nltk.stem import WordNetLemmatizer\n",
"nltk.download('popular', quiet=True) # for downloading packages\n",
"#nltk.download('punkt') # first-time use only\n",
"#nltk.download('wordnet') # first-time use only"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reading in the corpus\n",
"\n",
"For our example,we will be using the Wikipedia page for chatbots as our corpus. Copy the contents from the page and place it in a text file named ‘chatbot.txt’. However, you can use any corpus of your choice."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"f=open('chatbot.txt','r',errors = 'ignore')\n",
"raw=f.read()\n",
"raw = raw.lower()# converts to lowercase"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"The main issue with text data is that it is all in text format (strings). However, the Machine learning algorithms need some sort of numerical feature vector in order to perform the task. So before we start with any NLP project we need to pre-process it to make it ideal for working. Basic text pre-processing includes:\n",
"\n",
"* Converting the entire text into **uppercase** or **lowercase**, so that the algorithm does not treat the same words in different cases as different\n",
"\n",
"* **Tokenization**: Tokenization is just the term used to describe the process of converting the normal text strings into a list of tokens i.e words that we actually want. Sentence tokenizer can be used to find the list of sentences and Word tokenizer can be used to find the list of words in strings.\n",
"\n",
"_The NLTK data package includes a pre-trained Punkt tokenizer for English._\n",
"\n",
"* Removing **Noise** i.e everything that isn’t in a standard number or letter.\n",
"* Removing the **Stop words**. Sometimes, some extremely common words which would appear to be of little value in helping select documents matching a user need are excluded from the vocabulary entirely. These words are called stop words\n",
"* **Stemming**: Stemming is the process of reducing inflected (or sometimes derived) words to their stem, base or root form — generally a written word form. Example if we were to stem the following words: “Stems”, “Stemming”, “Stemmed”, “and Stemtization”, the result would be a single word “stem”.\n",
"* **Lemmatization**: A slight variant of stemming is lemmatization. The major difference between these is, that, stemming can often create non-existent words, whereas lemmas are actual words. So, your root stem, meaning the word you end up with, is not something you can just look up in a dictionary, but you can look up a lemma. Examples of Lemmatization are that “run” is a base form for words like “running” or “ran” or that the word “better” and “good” are in the same lemma so they are considered the same.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tokenisation"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"sent_tokens = nltk.sent_tokenize(raw)# converts to list of sentences \n",
"word_tokens = nltk.word_tokenize(raw)# converts to list of words"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Preprocessing\n",
"\n",
"We shall now define a function called LemTokens which will take as input the tokens and return normalized tokens."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"lemmer = nltk.stem.WordNetLemmatizer()\n",
"#WordNet is a semantically-oriented dictionary of English included in NLTK.\n",
"def LemTokens(tokens):\n",
" return [lemmer.lemmatize(token) for token in tokens]\n",
"remove_punct_dict = dict((ord(punct), None) for punct in string.punctuation)\n",
"\n",
"def LemNormalize(text):\n",
" return LemTokens(nltk.word_tokenize(text.lower().translate(remove_punct_dict)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Keyword matching\n",
"\n",
"Next, we shall define a function for a greeting by the bot i.e if a user’s input is a greeting, the bot shall return a greeting response.ELIZA uses a simple keyword matching for greetings. We will utilize the same concept here."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [],
"source": [
"GREETING_INPUTS = (\"hello\", \"hi\", \"greetings\", \"sup\", \"what's up\",\"hey\",)\n",
"GREETING_RESPONSES = [\"hi\", \"hey\", \"*nods*\", \"hi there\", \"hello\", \"I am glad! You are talking to me\"]\n",
"def greeting(sentence):\n",
" \n",
" for word in sentence.split():\n",
" if word.lower() in GREETING_INPUTS:\n",
" return random.choice(GREETING_RESPONSES)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generating Response\n",
"\n",
"### Bag of Words\n",
"After the initial preprocessing phase, we need to transform text into a meaningful vector (or array) of numbers. The bag-of-words is a representation of text that describes the occurrence of words within a document. It involves two things:\n",
"\n",
"* A vocabulary of known words.\n",
"\n",
"* A measure of the presence of known words.\n",
"\n",
"Why is it is called a “bag” of words? That is because any information about the order or structure of words in the document is discarded and the model is only **concerned with whether the known words occur in the document, not where they occur in the document.**\n",
"\n",
"The intuition behind the Bag of Words is that documents are similar if they have similar content. Also, we can learn something about the meaning of the document from its content alone.\n",
"\n",
"For example, if our dictionary contains the words {Learning, is, the, not, great}, and we want to vectorize the text “Learning is great”, we would have the following vector: (1, 1, 0, 0, 1).\n",
"\n",
"\n",
"### TF-IDF Approach\n",
"A problem with the Bag of Words approach is that highly frequent words start to dominate in the document (e.g. larger score), but may not contain as much “informational content”. Also, it will give more weight to longer documents than shorter documents.\n",
"\n",
"One approach is to rescale the frequency of words by how often they appear in all documents so that the scores for frequent words like “the” that are also frequent across all documents are penalized. This approach to scoring is called Term Frequency-Inverse Document Frequency, or TF-IDF for short, where:\n",
"\n",
"**Term Frequency: is a scoring of the frequency of the word in the current document.**\n",
"\n",
"```\n",
"TF = (Number of times term t appears in a document)/(Number of terms in the document)\n",
"```\n",
"\n",
"**Inverse Document Frequency: is a scoring of how rare the word is across documents.**\n",
"\n",
"```\n",
"IDF = 1+log(N/n), where, N is the number of documents and n is the number of documents a term t has appeared in.\n",
"```\n",
"### Cosine Similarity\n",
"\n",
"Tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus\n",
"\n",
"```\n",
"Cosine Similarity (d1, d2) = Dot product(d1, d2) / ||d1|| * ||d2||\n",
"```\n",
"where d1,d2 are two non zero vectors.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To generate a response from our bot for input questions, the concept of document similarity will be used. We define a function response which searches the user’s utterance for one or more known keywords and returns one of several possible responses. If it doesn’t find the input matching any of the keywords, it returns a response:” I am sorry! I don’t understand you”"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
"def response(user_response):\n",
" robo_response=''\n",
" sent_tokens.append(user_response)\n",
" TfidfVec = TfidfVectorizer(tokenizer=LemNormalize, stop_words='english')\n",
" tfidf = TfidfVec.fit_transform(sent_tokens)\n",
" vals = cosine_similarity(tfidf[-1], tfidf)\n",
" idx=vals.argsort()[0][-2]\n",
" flat = vals.flatten()\n",
" flat.sort()\n",
" req_tfidf = flat[-2]\n",
" if(req_tfidf==0):\n",
" robo_response=robo_response+\"I am sorry! I don't understand you\"\n",
" return robo_response\n",
" else:\n",
" robo_response = robo_response+sent_tokens[idx]\n",
" return robo_response\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Finally, we will feed the lines that we want our bot to say while starting and ending a conversation depending upon user’s input."
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ROBO: My name is Robo. I will answer your queries about Chatbots. If you want to exit, type Bye!\n",
"hi\n",
"ROBO: hey\n",
"hello\n",
"ROBO: hey\n",
"bye\n",
"ROBO: Bye! take care..\n"
]
}
],
"source": [
"flag=True\n",
"print(\"ROBO: My name is Robo. I will answer your queries about Chatbots. If you want to exit, type Bye!\")\n",
"while(flag==True):\n",
" user_response = input()\n",
" user_response=user_response.lower()\n",
" if(user_response!='bye'):\n",
" if(user_response=='thanks' or user_response=='thank you' ):\n",
" flag=False\n",
" print(\"ROBO: You are welcome..\")\n",
" else:\n",
" if(greeting(user_response)!=None):\n",
" print(\"ROBO: \"+greeting(user_response))\n",
" else:\n",
" print(\"ROBO: \",end=\"\")\n",
" print(response(user_response))\n",
" sent_tokens.remove(user_response)\n",
" else:\n",
" flag=False\n",
" print(\"ROBO: Bye! take care..\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
1 change: 1 addition & 0 deletions Day_5_codes/Day 5 codes/AI chatbot/_config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
theme: jekyll-theme-midnight
Loading