Before we start doing something, we should start with why?. And in this section, it'll explain to you why? FAQ-Bot made.
When we make a product, we want our customers or users can use it as we wish. However, along with the development of this product, the more information is spread widely and make our customers lose their way.
To solve that, FAQ was created. FAQ stands for Frequently Asked Question, FAQ is a list of questions that most often asked from customer or user about a certain topic along with the answer.
To increase the quality of FAQ, as a producer we should be able to answer the following question :
- Is the answer appropriate to customer?
- What other questions do customers often ask?
As we know, Facebook messenger-platform has released an API that we can use to create a smart virtual assistant. This smart virtual assistant automatically reply to the messages from the costumers. This assistant can also be integrated with wit.ai in order to make it easier to understand what customers saying.
By using the features, we are going to make a bot in messenger that able to understand the question from customers. Then, we will process the question to provide an appropriate answer.
- Introduction
- Table of Contents
- Analyze app that we're going to make
- It's Time to Develop the App!
- Closing
Table of contents generated with markdown-toc
The key of making high-quality applications is knowing the details. If we understand what we're doing, then it would be effective and efficient in the development process since we already know the tech specifications we need and what steps should be taken if errors/bugs occurs.
We need to define the scope so that in this app development we can focus on primary things. The following is the diagram use case :
-
Actor that ables to interact with the system is: 1) customer
-
Costumer actor ables to 1) Send a message and 2) receive an answer.
To know any specifications of the technology that we'll use, We need to try making the simple version or we can say hello-world app. The method we will use is spike[2].
To make an application in messenger platform that could receive, read then answer the message, we need to understand how the platform architecture works. To understand that, we can directly go to short tutorial from messenger-platform through this link https://developers.facebook.com/docs/messenger-platform/getting-started.
To make the application understand the meaning of customer questions, we need to integrate AI. We train this AI by giving samples from customer questions then we specify from each sample, what does customer mean. To understand how, we can directly go to short tutorial from wit.ai through this link https://wit.ai/docs/quickstart.
After doing spikes above, the following is diagram of the specifications technology that we'll use. The diagram is made as simple as possible, so it may miss some information. However, this should be enough to explain the high-level.
Each event conducted by the customers will be forwarded to the webhook that we have created. On this webhook your application will receive, process, and response to the message sent by the customer through Facebook page.
The webhook requirements are :
-
Supports HTTPS protocol
-
Valid SSL certificate
-
Open the port that ables to request
POST
andGET
.
We need to save the webhook URL to settings in Facebook application.
When we save it, Facebook will send GET
request which contains the verification process to make sure webhook requirements are met.
Facebook will sends POST
request to the webhook URL that we already save on settings.
This request contain information such as ID from sender, text messages, time sent, etc.
Our application will process a message corresponding to the logic we made.
After that, the application will answer the message from sender by sending POST
request to Facebook Graph API.
- Ruby ~> 2.5 installed on your device.
You may use rbenv or rvm, to install ruby version needed.
To check if your device has ruby installed, you can run the following command in terminal/command prompt:
$ ruby -v
# example: ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-darwin18]
if your terminal sent back ruby <xxx>
then it's ready to use.
- Bundler ~> v2
To install, open terminal/command prompt then run the following:
$ gem install bundler -v 2.0.2
To check if your device has bundler installed, you can run the following command in terminal/command prompt:
$ bundler -v
# example: Bundler version 2.1.4
if you see Bundler version <xxx>
on your terminal then it's ready to use.
- Rails ~> v6
To install, open terminal/command prompt then run the following:
$ gem install rails -v 6
To check if your device has rails installed, you can run the following command in terminal/command prompt:
$ rails -v
# example: Rails 6.0.3.4
if you see Rails <xxx>
on your terminal then it's ready to use.
- Docker
You can install docker through this link: https://docs.docker.com/get-docker.
- Facebook Developer account
Facebook developer account is required to make a new application, that is core of Facebook integration. You can open the link https://developers.facebook.com to create a new account.
- Wit.ai account
This account is required to make NLP model[1], used to understand he user's question. If you don't have one, click on the link to create new account https://wit.ai.
- Ngrok account
This ngrok account is required to make https URL from your webserver to internet needed by Facebook so that able to sends data to your webhook. You can create a new account by click the link https://ngrok.com. Don't forget to follows the setup tutorial on https://dashboard.ngrok.com/get-started/setup.
Open terminal/cmd, get into your any directory then run the following command below:
$ rails new my-faq-bot --force --api --skip-action-mailbox --skip-action-mailer --skip-active-storage --skip-system-test --skip-action-text --skip-javascript --skip-spring --skip-action-cable --no-skip-active-record --database=postgresql
This command will create a new folder named my-faq-bot
contains the project template for REST API application.
Get into your project directory, then create a new file named docker-compose.yml
with the following code:
# <root_project_directory>/docker-compose.yml
version: '3'
services:
postgres:
image: postgres:11-alpine
ports:
- "5432:5432"
environment:
- POSTGRES_DB=faq-bot_development
- POSTGRES_USER=postgres
- POSTGRES_HOST_AUTH_METHOD=trust
Add key username
and host
in config/databse.yml
file as the following:
# <root_project_directory>/config/database.yml
development:
<<: *default
database: faq-bot_development
username: postgres
host: localhost
Open terminal/cmd, then run the following command:
$ docker-compose up -d
That command will run one PostgreSQL container that we can use as our database.
Open terminal/cmd, get into your project directory then run the following command:
$ bin/rails server -port 3000
The command above will run your project as web server on the port:3000.
Next open http://localhost:3000
on your browser and make sure you see the page: "Yay! You’re on Rails!"
Go to your project directory, then add webhook
resource in config/routes.rb
.
# <root_project_directory>/config/routes.rb
Rails.application.routes.draw do
resource :webhook, only: [:show, :create]
end
To see if the URL works, open terminal/cmd, then run the following command:
$ bin/rails routes
# Prefix Verb URI Pattern Controller#Action
# webhook GET /webhook(.:format) webhooks#show
# POST /webhook(.:format) webhooks#create
From the example result above, then:
-
If there is request
GET /webhook
it should be routed toWebhooksController
in a function#show
. -
If there is request
POST /webhook
it should be routed toWebhooksController
in a function#create
.
Get into your project directory, then create a new file webhooks_controller.rb
in app/controllers/
.
# <root_project_directory>/app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
def show
mode = params['hub.mode']
token = params['hub.verify_token']
challenge = params['hub.challenge']
if mode == 'subscribe' && token == 'foo'
print 'WEBHOOK_VERIFIED'
render json: challenge, status: :ok
return
end
render json: 'FAILED', status: :forbidden
end
end
The contents of the #show
function in the WebhooksController
class are:
-
Read the
mode
,token
, andchallenge
of the query parameters. -
Check if the mode and token are suitable.
-
If suitable, display the value of
challenge
with http status200
ok. -
If it doesn't match, display
"FAILED"
with http status403
forbidden.
Get into your project directory, then add a function #create
in WebhooksController
.
# <root_project_directory>/app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
# show ..
def create
webhook_data = params['webhook'].as_json
if webhook_data['object'] != 'page'
render json: 'FAILED', status: :not_found
return
end
entries = webhook_data['entry']
entries.each do |entry|
messaging = entry['messaging'].first
print messaging['message']
end
render json: 'EVENT_RECEIVED', status: :ok
end
end
The contents of the #create
function in the WebhooksController
class are:
-
Read the
webhook_data
from the request body which is entered in json form. -
Check whether the object is suitable or not.
-
If suitable, retrieve the value of the message sent then display
"EVENT_RECEIVED"
with http status200
ok. -
If it doesn't match, display
"FAILED"
with http status404
not found.
To check if our webhook is successful, open terminal/cmd, then Running web server, after that run the following command to verify token:
$ curl -X GET "localhost:3000/webhook?hub.verify_token=foo&hub.challenge=CHALLENGE_ACCEPTED&hub.mode=subscribe"
To know if our webhook successful, is you can check:
-
if the contents of the query parameter
hub.verify_token
is equal to our specified token above, then we expect webhook should sent back contents from query parameterhub.challenge
and the status200 ok
. -
There is message
WEBHOOK_VERIFIED
shows in console.
For events on the webhook, run the following command:
$ curl -H "Content-Type: application/json" -X POST "localhost:3000/webhook" -d '{"object": "page", "entry": [{"messaging": [{"message": "TEST_MESSAGE"}]}]}'
We can check if our webhook successful is by seeing:
-
if the contents from request body
object
ispage
then the expectations, webhook should send backEVENT_RECEIVED
and the status200 ok
. -
There is message
TEST_MESSAGE
shows in console.
As explained in Conclusion, several requirements must be met so that we can put our webhook in the app configuration. For now, since we are still in development stage, we can use a software named ngrok. The steps are:
1. Change app configuration
Get into your project directory, then allow ngrok hostname in config/environments/development.rb
by adding config.hosts << /[a-z0-9]+\.ngrok\.io/
as follos:
# <root_project_directory>/config/environments/development.rb
Rails.application.configure do
# configs ..
config.hosts << /[a-z0-9]+\.ngrok\.io/
end
2. Run web server
Do as follows this step.
3. Run ngrok
Open a new tab in terminal/cmd, then run:
$ ngrok http 3000
Example output:
Get https URL from line Forwarding
for example https://fe01d05e31bc.ngrok.io
4. Save webhook URL in Facebook app configuration
If you follow this section, then you should already have Facebook developer account and ever made Facebook app. The next step is :
-
Open the settings page from Facebook app that you've made.
-
Get into
messenger > settings > webhooks
Add host and verify token according to the form in the image. For example, the url webhook is https://fe01d05e31bc.ngrok.io/webhook
and the token verification is foo
.
- Connect the bot with the Facebook page
Add messages
and messaging_postback
as subcription settings.
- Check if the configuration is correct like this section by sending message to Facebook page that you have chosen.
-
Open settings page from facebook app you've made.
-
Get into
messenger > settings > webhooks
- Get token from section
access tokens
⚠️ Be careful with this token. Don't share with anyone, unless you believe to them.
Faraday is one of gem that serves as HTTP client.
Get into your project directory, then add Faraday Gemfile
:
# <root_project_directory>/Gemfile
gem 'faraday'
Open terminal/cmd, then run the following command:
$ bundle install
Get into your project directory, then modify #create
function in WebhooksController
.
# <root_project_directory>/app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
# show ..
def create
webhook_data = params['webhook'].as_json
if webhook_data['object'] != 'page'
render json: 'FAILED', status: 404
return
end
entries = webhook_data['entry']
entries.each do |entry|
messaging = entry['messaging'].first
message = messaging['message']
sender_id = messaging['sender']['id']
text_for_answer = message['text']
send_message(sender_id, text_for_answer)
end
render json: 'EVENT_RECEIVED', status: 200
end
private
def send_message(recipient_id, text)
access_token = '<put your page access token>'
request_url = URI("https://graph.facebook.com/v2.6/me/messages?access_token=#{access_token}")
request_body = {
recipient: {
id: recipient_id
},
message: {
text: text
}
}.to_json
resp = Faraday.post(request_url, request_body, "Content-Type" => "application/json")
end
end
The contents of the modified #create
function in the WebhooksController
class are:
-
Retrieve the contents of the message sent and the sender id.
-
Make a
POST
request to the Facebook graph API to send a message on behalf of that page we choose. The contents of the request are the recipient id and the text which contains a message which we will send as a reply to the customer.
After that, you can test by sending message to your facebook page. Don't forget to make sure you have done this section and web server is already running.
To understand the meaning of customer question, we can train our app so that it can extract the meaning from the sentences based on how high the level of confidence.
If you have done this section, then it should be similiar. First thing first, we create an app in wit.ai. Next, get into understanding page.
After that, entry utterances, select the entity and the intent. The more utterance sample are used for this training, then the quality of level of confidence is way better.
- Get access token from wit.ai account
⚠️ Be careful with this token. Don't share it with anyone, unless you believe in them.
-
Open Facebook app settings page that you've made.
-
Get into
messenger > settings > Built-In NLP
-
Select
Other Language Support
, then selectEnglish
, afterward selectCustom
. Next, enter the access token we got from wit.ai
Get into your project directory, then modify #create
function in WebhooksController
.
# <root_project_directory>/app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
# show ..
def create
webhook_data = params['webhook'].as_json
if webhook_data['object'] != 'page'
render json: 'FAILED', status: 404
return
end
entries = webhook_data['entry']
entries.each do |entry|
messaging = entry['messaging'].first
message = messaging['message']
sender_id = messaging['sender']['id']
send_message(sender_id, text_for_answer(message))
end
render json: 'EVENT_RECEIVED', status: 200
end
private
# send_message ...
def text_for_answer(message)
selected_intents = message['nlp']['intents'].select { |item| item['confidence'].to_f > 0.5 }
sorted_intents = selected_intents.sort_by { |item| -item['confidence'].to_f }
intent = sorted_intents.first['name'] unless sorted_intents.empty?
return 'You can put anyname on Facebook' if intent == 'allowed_name'
message['text']
end
end
The contents of the modified #create
function in the WebhooksController
class are:
-
There is a new private function called
#text_for_answer
. The contents of this method is to take the results of the NLP analysis provided by wit.ai. -
Sort the analysis based on the highest level of truth and do not forget filter with a threshold such as
0.5
-
Get the contents of the obtained intent, if the intent is ʻallowed_name
, then return the text
"You can put anyname on Facebook"`. -
If the intent is not what is intended, then return the text according to the message sent by the sender.
-
The text returned above is used in the
POST
request as the body of the message which we will send as a reply to the customer.
After that, you can test by sending message to your facebook page. Don't forget to make sure you have done this section and web server is already running.
Open terminal/cmd, get into any directory you want, then run the following commands:
$ bin/rails generate migration CreateAnswers
# invoke active_record
# create db/migrate/20201025103252_create_answers.rb
These commands will create a new file that contains a script template to change the scheme on database.
Next, get into the new file that just generated before, for example db/migrate/20201025103252_create_answers.rb
.
Then add the contents as follows:
# <root_project_directory>/db/migrate/20201025103252_create_answers.rb
class CreateAnswers < ActiveRecord::Migration[6.0]
def change
create_table :answers do |t|
t.string :question_type, null: false
t.string :text, null: false
t.timestamps
end
add_index :answers, :question_type, unique: true
end
end
The script is used for adding a new table named answers
that has column question_type
with type : string
, and text
with type string
.
Back into terminal/cmd, then run:
$ docker-compose up -d
$ bin/rails db:migrate
That command will runs the container contains PostgreSQL that we have prepared in this section. Next, we execute the migration script to apply a new database scheme that we have designed.
Get into your project directory, then create a new file answer.rb
in app/models/
.
# <root_project_directory>/app/models/answer.rb
class Answer < ApplicationRecord
validates :question_type, uniqueness: true
end
Adding record on answers table
Open terminal/cmd, get into any directory you want, then run the following command:
$ bin/rails console
irb(main):001:0> Answer.create(question_type: 'allowed_name', text: 'You can put anyname on Facebook')
Get into your project directory, then modify private#text_for_answer
function in WebhooksController
.
# <root_project_directory>/app/controllers/webhooks_controller.rb
class WebhooksController < ApplicationController
# show ..
# create ..
private
# send_message ...
def text_for_answer(message)
selected_intents = message['nlp']['intents'].select { |item| item['confidence'].to_f > 0.5 }
sorted_intents = selected_intents.sort_by { |item| -item['confidence'].to_f }
intent = sorted_intents.first['name'] unless sorted_intents.empty?
answer = Answer.find_by_question_type(intent)
return answer.text if answer.present?
message['text']
end
end
The contents of the modified #text_for_answer
function in the WebhooksController
class are:
-
If previously we hardcode the text to be used in return to the customer, we now perform a query to the database via
Answer
model. -
Then we take the contents of the
text
field which we get from the query results. -
Use the text as the text of the message that will be the reply to the customer.
After that you can test by sending message to your facebook page. Don't forget to make sure you have done in this section and web server is already running.
Congratulations! You have successfully created FAQ-bot using messenger and wit.ai. In this tutorial, we have learned many things such as preparation, how to make messenger app, how did NLP works, etc. Hopefully what we have learned can be really useful and you can develop into even cooler things 😁
You can access this project source code on : https://github.com/qornanali/faq-bot/blob/master/docs/source_code_example.
Of course, this app isn't perfect yet. In the future, you may add:
-
Since ngrok is very limited and it is temporarily for development process, therefore we need to deploy the app on server as you wish so that the application always serving customer questions. You can use Heroku, Azure, AWS, etc.
-
For now, to add record on the answers table, we are still using the rails console. If we provide API CRUD (create, read, update, delete) which can be consumed internally, the process will be even easier.
Anything else?
Of course! You can contribute to this repository by following issues in this repos. Any kind of contribution will be accepted! 😁