Build Chat Bot to Control Your Home

ghiscure

Simple chat bot to control your home
Explore the docs »

View Demo · Report Bug · Request Feature

Languange

Read this in other languages: English, Indonesian.

Table of Contents

• Introduction
• Smart Home Unit and Component
• System Configuration
• Chatbot as Control System
  • Facebook Messenger
  • Wit.ai

• How it works

  • Facebook Messenger
  • Wit.ai
  • NodeMCU

• How To Use

  • Installation
    • NodeJs
    • Git
    • Wit.ai
    • Server
    • NodeMCU
  • Deploy
    • Heroku
    • Ngrok

• Reference

• License

Introduction

These days, most people use gadget to support their activities. The development of computer and embedded devices make poeple able to control other devices with their smartphone over internet. This is called Internet of Things. The example of Internet of Things application is smart home.

In this repository we will design a smart home system that integrates with Chat bot to control electronic devices. Message from user recognized by chat bot, and that command will be forwarded to electronic devices. User can control their electronic devices anytime and anywhere.

Smart Home Unit and Component

In IoT, there are 3 important things => node, gateaway and server. Node is the edge devices that have function to be an input or output. The example of nodes are Lamp, Air Conditioner, Pump, and Selenium Door. Gateaway is a connector between server and node. Gateaway can be a microcontroller or mini computer. These are components needed in this tutorial:

1. LED
2. Breadboard
3. Resistor 1K ohm
4. NodeMCU (ESP8266)
5. Jumper

System Configuration

Chatbot as Control System

Facebook Messenger

Facebook Messenger is used for connecting server and user. To get started you can follow this tutorial

Wit.ai

Build Your APP

On this section we will show you how to build an app to control lamp.

1. For intermezzo you can follow this tutorial
2. After you create an account you can check this gif
3. There are 4 important things in wit.ai
   • Utterance
   • Intent
   • Entity
   • Trait
4. To improve the detection you need to train your app. Just add Utterance and labelling it. After that wit.ai will train your data set

How it works

A. Facebook Messenger

1. Received text message from user and send it to wit.ai

   if (received_message.text) {
      
      // Send text to wit.ai and get response
      var result = await getMessage(received_message.text)
      var caps= `turning ${result[0][0]} the ${result[1][0]}`
   
      // Send command to user
      response = {
        "text": caps
      }
      callSendAPI(sender_psid, response);
   
   
      //send result to MQTT Broker and NodeMCU
      var topic = `esp8266/ghiscure/${result[1][0]}` 
      if(result[0][0]=='on'){
      listen.publish(topic, "1");
      console.log(`${result[0][0]}, ${topic}`);
      }else{
      listen.publish(topic, "0");
      console.log(`${result[0][0]}, ${topic}`);
      }
      
    } 

2. Received voice notes from user and send it to wit.ai

   else if (received_message.attachments[0].type=="audio") {
      console.log('audio')
      
      // Get the URL of the message attachment
      let attachment_url = received_message.attachments[0].payload.url;
      
    // Convert voice notes to mp3
     var result = await fetch(attachment_url)
     proc = new ffmpeg({source:result.body})
     proc.setFfmpegPath('ffmpeg')
     result = proc.saveToFile('output.mp3',  function(stdout, stderr){
          return "success"
     })
   
     // Send mp3 to wit.ai
    var mimetype_ = "audio/mpeg3"
     var readStream = fs.createReadStream("output.mp3")
   
     // Get Result from wit.ai
     result = await getMessagefromAudio(readStream, mimetype_)
     console.log(result)
     var caps= `turning ${result[0][0]} the ${result[1][0]}`
     response = {
      "text": caps
    }
   
    // send result to MQTT Broker and NodeMCU
     callSendAPI(sender_psid, response);
     if(result[0][0]=='on'){
       listen.publish(topic, "1");
       console.log(`${result[0][0]}, ${topic}`);
       }else{
       listen.publish(topic, "0");
       console.log(`${result[0][0]}, ${topic}`);
       }     
    }

B. Wit.ai

1. Get Message from text parameter

    // This function is used to get response from wit.ai. This function requires a string parameter
      getMessage: async function(query){
    var url =`https://api.wit.ai/message?v=20201020&q=${encodeURI(query)}` 

    var response = await fetch(url, {
        headers: {
            'Authorization': 'Your Api Token'
        }
    })
    var json_data = await response.json()
    try {

      // Get Command Value
      var cmd_value =json_data.traits.wit$on_off[0].value
      var cmd_confidence = json_data.traits.wit$on_off[0].confidence

      // Get Object Value
      var object_value = json_data.entities['object:object'][0].value
      var object_confidence = json_data.entities['object:object'][0].confidence

      return [[cmd_value,cmd_confidence],[object_value,object_confidence]]   

    } catch (error) {
      console.log(error)
      
    }
    
  }

2. Get Message from voice notes

    // This function is used to get response from wit.ai. This function requires an audio file as parameter
      getMessagefromAudio: async function(bin_data, mimetype_){
    var options = {
        method: 'POST',
        headers: {
            'Authorization': process.env.witai_token,
            'Content-Type': mimetype_
        },
        encoding: null,
        body: bin_data
      }
    var url =`https://api.wit.ai/speech?v=20200513` 
    try {
        // Get response from wit.ai
      var response = await fetch(url, options)
      var json_data = await response.json()
      
      // Get Command Value
      var cmd_value =json_data.traits.wit$on_off[0].value
      var cmd_confidence = json_data.traits.wit$on_off[0].confidence

      // Get Object Value
      var object_value = json_data.entities['object:object'][0].value
      var object_confidence = json_data.entities['object:object'][0].confidence

      console.log(cmd_value, cmd_confidence)
      console.log(object_value, object_confidence)  
      return [[cmd_value,cmd_confidence],[object_value,object_confidence]]     
    } catch (error) {
      console.log(error)
      
    }
  }

C. NodeMCU

1. Received data from MQTT Broker and Turn on/off the lamp/AC

// If a message is received on the topic esp8266/ghiscure/AC, check if the message is either 1 or 0. Turns the ESP GPIO according to the message

  if(topic=="esp8266/ghiscure/AC"){
      Serial.print("Changing GPIO 4 to ");
      if(messageTemp == "1"){
        digitalWrite(ledGPIO4, HIGH); //Turn on AC
        Serial.print("On");
      }
      else if(messageTemp == "0"){
        digitalWrite(ledGPIO4, LOW); //Turn off AC
        Serial.print("Off");
      }
  }
    // If a message is received on the topic esp8266/ghiscure/lamp, check if the message is either 1 or 0. Turns the ESP GPIO according to the message

  if(topic=="esp8266/ghiscure/lamp"){
      Serial.print("Changing GPIO 5 to ");
      if(messageTemp == "1"){
        digitalWrite(ledGPIO5, HIGH); // turn on lamp
        Serial.print("On");
      }
      else if(messageTemp == "0"){
        digitalWrite(ledGPIO5, LOW); //turn off lamp
        Serial.print("Off");
      }
  }

2. Subscribe topic

// You need to add subscribe function to get message from spesific topic
client.subscribe("esp8266/ghiscure/AC");
client.subscribe("esp8266/ghiscure/lamp");

How to Use

Installation

A. Git

1. Start by updating the package index

sudo apt update

2. Run the following command to install Git:

sudo apt install git

3. Verify the installation by typing the following command which will print the Git version:

git --version

4. Windows Installations

https://git-scm.com/download/win

B. NodeJs

1. Debian Based

# Using Ubuntu
curl -sL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs

# Using Debian, as root
curl -sL https://deb.nodesource.com/setup_lts.x | bash -
apt-get install -y nodejs

2. Windows

https://nodejs.org/en/download/

C. Wit.ai

1. To get wit.ai token you can follow this tutorial.

https://wit.ai/docs/quickstart

D. Server

Prerequisite

1. Before you use this app, you must register your app in facebook platform. You can follow this tutorial to get PAGE_ACCESS_TOKEN.

https://developers.facebook.com/docs/messenger-platform/getting-started-app-setup

There are 4 items that you must pay attention to

2. For VERIFY_TOKEN, it is up to you
3. You must have Wit.ai Token
4. After you have PAGE_ACCESS_TOKEN,VERIFY_TOKEN and Wit.ai Token, just follow this step

git clone https://github.com/ghiscure/DevC
cd DevC
mv .env.example .env
#Edit .env with your app credentials
npm install
npm start

5. Server configuration is done

E. NodeMCU

Because of limited devices, in this demonstration we will use LED. This led can represent AC, Pump or another electronic devices.

1. Install Arduino

https://www.arduino.cc/en/main/software

2. Install NodeMCU board in Arduino

https://randomnerdtutorials.com/how-to-install-esp8266-board-arduino-ide/

3. Wiring Schematic
4. Edit config of wireless connection

// Change the credentials below, so your ESP8266 connects to your router
const char* ssid = ""; // your ssid name
const char* password = ""; // password ssid

5. Upload file to NodeMCU

Deploy

1. Heroku
   1. You can use this tutorial to deploy app

   https://devcenter.heroku.com/articles/deploying-nodejs
   

   2. You must edit the environment variable. You can use this tutorial to edit the env variable.

   https://devcenter.heroku.com/articles/config-vars
   

   3. There are 3 environment variable that you must set in config vars.
      • PAGE_ACCESS_TOKEN
      • VERIFY_TOKEN
      • witai_token
   4. Change your Facebook callback URL to heroku
2. Ngrok
   You can use ngrok for forwarding http protocol. Follow this tutorial to forward your localhost to public. Change your Facebook's URL callback to ngrok url.

   ngrok http 3000

Reference

1. ESP32 MQTT – Publish and Subscribe with Arduino IDE
2. Getting Started with Messenger Platfrom
3. Build Your First Wit App
4. Getting Started on Heroku with Node.js
5. Configuration and Config Vars
6. Expose a local web server to the internet

Demonstration

License

Usage is provided under the MIT License. See LICENSE for the full details.