Necessary tools and machines

Apps and online services

About this project

Why tackle the Smart Home problem?

Nowadays, people have smartphones with them all the time. So it makes sense to use these to control home appliances. Presented here is a home automation system using a simple Android app which you can use to control electrical appliances with voice commands. Using the app you can control all the appliances like TV, fans, light etc. It’s like a Siri which can be controlled using voice (command given by user). You can give the command to switch on or off the devices (like light, fan etc) and also manipulate them like fan speed, light intensity. Commands are sent via a Bluetooth module to Arduino. So there is no need for you to get up to switch on or switch off the device while watching a movie or doing some important work.

What’s new?

We are also using an AR-augmented reality app (self made) to control the appliances and manipulate them. If we want to switch on the electrical appliance we just need to focus the camera and it will start it and using a virtual button we can switch it off .

If we want to know the electricity unit consumed by our appliances along with the cost of units we just need to send a text message to the device (Arduino) through our phone. The Arduino would be connected to the network through a GSM module which would reply with another text message containing the electricity unit and the total cost consumed by the appliances. This will help the user to get update on the electricity bill and they can decide whether to use the appliances on a regular routine or reduce it usage to save electricity.

Why did we decide to make it?

Working

The Digital Energy Meter

In the energy meter, a LDR (Light Dependent Resistor) is used to take the blink speed according to the load. The blink per second value is fed into the Arduino for further calculation units and the value of energy consumed. These are sent to the user via the GSM module.The user can then comprehend which appliance is consuming more power and take action accordingly.

The AI Virtual Assistant

The AI Virtual Assistant works using an android app which is installed on the user’s android phone. The user activates the agent using the wake up command, ‘Listen up Jarvis’.

The android app is connected to the Arduino 101 via Bluetooth which is inturn connected to the relay and hence the electrical appliances.

The flow of signals is as follows:

1. Voice Input to Android App

2. Processing on Android App

3. Sending encrypted data to Arduino via Bluetooth.

4. Processing on Arduino 101

5. Digital signals to relay

Pins connections:

Pin 8 -- > PulseIn [Digital Input]

Pin 6 -- > IN1 -- > Fan Output

Pin 4 -- > IN2 -- > Light Output

Pin 5 -- > IN3 -- > Coffee Maker

Connections of Energy meter:

Connect the meter to the supply.

Cover the energy meter with a black tape, preventing the LDR inside the energy meter from outside light.

Connect the neutral wire directly to the loads and the phase wire to the relay.

Take the wires from the LDR outside the meter to the counter circuit.

Now take the output from the counter circuit to the assigned pin 8 pulsepin as in the code.

Connect the TX, RX pin of the Arduino 101 to RX, TX pin of the GSM module.

· For programming the Arduino 101, we have used the default Arduino IDE.

· We have developed our Android app using Android Studio (JAVA and XML).

· Database connectivity is done using Google's Firebase.

· For augmented reality we have used Unity along with the Vuforia SDK.

Implementation:

All the components are giving readings to the Arduino, where we are converting these readings into character code in order to send the bytes efficiently and quickly to the computer. Inside a never ending loop, every reading corresponds to a character.

A code should run on the computer in order to receive the data that is being sent by arduino via serial ports.

For developing the android app we are using android studio and to convert voice to text we are using the standard google api.

For speech recognition we are using acoustic-phonetic speech parameters for speaker-independent speech recognition in the android app.

We have the AI (artificial intelligence) virtual assistant which uses deep learning techniques like CNN (Convolutional Neural Network) using python to implement the natural language processing.

For the augmented reality app we are using Unity IDE and VuforiaSDK (It is an Augmented Reality Software Developments Kit (SDK) for mobile devices that enables the creation of Augmented Reality applications).

Videos

Conversations with personal AI Assistant

Conversations with personal AI Assistant

Controlling appliances using personal AI Assistant

Controlling appliances using personal AI Assistant

Controlling appliances using Augmented Reality

Digital energy reading and cost of appliances connected to J.A.R.V.I.S.

Code

Arduino code for digital energy reading and cost calculationArduino

This code implements the energy units consumed by an appliance connected to the Arduino. Instructions :1. Make the connections and insert a mobile number in the Arduino code2. Upload the code to an Arduino 1013. Run the code4. Send an SMS from any mobile to the mobile number specified in the code in the format #anytext*5. A reply containing the units consumed and the cost would be received

Custom parts and enclosures

Augmented Reality Android App for speech impaired people

1. Download and install the app2. Make sure that the phone is connected to the internet3. Start the app and point it to an appliance which has been configured with Vuforia (the default Vuforia marker is used here)4. The appliance should start5. To turn off the appliance, click on the virtual button that appears on the screen whenever phone is pointed towards the device

jarvisaugmented_I2zWAu2jGG.apk

Schematics

Overall Schematics and circuit daigram

J.A.R.V.I.S. Android App

Kindly note: Active Internet Connection on phone is required at all timesInstructions : 1. Install the app on an android phone2. After setting up the hardware and running it successfully, start the app3. For using Virtual Assistant (Voice), say the keyword, 'Listen Up JARVIS'. This should activate the agent 3.1 Speak your command clearly 3.2 The agent should provide a response4. To start the augmented reality app, click on the augment button; a blank screen should appear 4.1 Open the Jarvis Augmented App on a different phone (NOTE : The J.A.R.V.I.S. android app should be on a different phone) 4.2 Proceed to instructions given in JarvisAugmented for implementing augmented reality

NOTE : 1. If app is stuck at any point, try restarting the app 2. If a message similar to, 'Could not open Microphone' appears on the screen, restart the app3. This app is implemented using HC05 Bluetooth module to increase the scope of the project. Hence, HC05 Bluetooth Module is recommended