Apps and online services

About this project

Photo shows placement of receiver module and plate

The Project

This project is a combination of the many smart fridges and pantry's of the past. The idea is to combine all of them while also introducing Amazon's DRS system through Alexa and the Echo products as a bonus feature.

The process will be simple:

as you put the item you purchased via Amazon Prime Pantry in your fridge, it will be scanned by an RFID tag scanner. This will detect an assigned UID, which will then correspond to a given ASIN. The ASIN will give details VIA web connection, such as weight, size, product name, etc.

Once scanned, the item will then be put onto a shelf with a built in scale. It will keep in mind the item that is going on, then detect the weight change, and track a percent and amount of that given item left.

These details will then go to an Amazon-hosted webpage where your Alexa can respond to simple questions, such as asking how much of an item is left, or asking Alexa to order something for you. She can also track expiration dates and give warnings.

The system will not be solely based around Alexa, though. It can have a small LCD screen that will display the information, and will allow programming to reorder if a certain item drops below a pre-determined amount.

Additionally, when you order a Pantry Box, it will include an RFID tag that you can scan to your pantry. This will assume that all of the items are full and ready to stock, and in turn saves time.

What will make this system better than others?

While there have been smart pantry systems in the past, many of them required too much user input, and many do not have the interconnections between the sensors that will be in this product. RFID smart pantrys only track what has been put in the fridge, not what is left; Scale based pantry's only track weight and require the user interface with it and record what was put in.

As stated above, not only will this project fix those, but it will also interweave it into a service, this case is Amazon.

Building the Project:

Part 1, Materials:

This is very vague because it can be done in many methods, this is just the method I chose to do.

Arduino of choice (Preferably AVR over STM)

MFRC522 RFID scanner and Tag

ESP8266, source of internet connection.

Load cell ( I recommend one over 20 kg load)

HX711 (For load cell amplification)

5 Volt Power Supply

Ethernet cable (If card)

Breadboard for testing

Wire

Acrylic or plastic for scale

Part 2, Libraries:

Again, much of this is up to personal choice. I am using these because I found them first and they worked for me best.

To install the ones you do not have (HX711, ENC, MFRC) just unzip then install them into the Library Folder of where your IDE is installed. For the Ethernet you do NOT have to delete the Ethernet folder, instead, all you have to do is rename all "Ethernet.h" Libraries to "UIPEthernet.h"

Update: Because of the library being used, this will only work with utilization of an ESP8266 module

Part 3, Other Resources:

Lots of time. Lots and Lots of time. Seriously though, I have put in 150+ hours over the past 4 weeks and it is no where near done, and half of those hours were from just simple wiring issues and code bugs that were not even relevant to the whole project. This is why I am bumping up there experience level.

Along with time, patience. For obvious reasons.

Google Developer/API

Part 4, Wiring:

I'll post a diagram later, but for now:

RFID on UNO #1 ( I will refer to it as #1 for the rest of the writeup)

RFID Arduino
VCC 3.3 Volts--While it will work on 5V, IT IS HIGHLY ADVISED AGAINST
RST Any digital lower than 10; I have it assigned to pin 9
GND Ground on Arduino, who would have guessed
MISO PIN 12 on UNO, 50 on MEGA
MOSI PIN 11 on UNO, 51 on MEGA
SCK PIN 13 on UNO, 52 on MEGA
SS PIN10, however this can be changed.
IRQ Unassigned.
Ground Also to Ground on Arduino #2

Serial Connections (WIRE) among Arduinos #1 and #2

UNO #1 MEGA
Analog_A4 20
Analog_A5 21
Ground Ground

Ethernet on UNO #2 (again, will refer to it as #2 for the whole thing)

Updated: The Library being used does not work with Ethernet, instead it uses the ESP module. Similar to the HX711 below, wiring the ESP will be completely dependent on if you have a 3.3v power supply, or if you need a voltage divider.

ETHERNET Arduino
VCC 3.3 Volts--While it will work on 5V, IT IS HIGHLY ADVISED AGAINST
RST Unassigned how I have it right now, I might add a reset function later
GND Ground on Arduino, again
MISO PIN 12 on UNO, 50 on MEGA
MOSI PIN 11 on UNO, 51 on MEGA
SCK PIN 13 on UNO, 52 on MEGA
CS PIN10, Not sure yet if it can be changed. I haven't snooped that far
CLOCKOUT Unassigned.
INT Unassigned.
WOL Unassigned.
Ground Also to Ground on Arduino #1

HX711: I would check the datasheet you get it with, whether it is Amazon, Ebay, or anywhere else. Two of mine were marked incorrectly and did not work as planned. Trial and error will get you there!

How my finalized module turned out. I set the MEGA on bottom, then stacked the UNO upsidedown on top. Also notice the placement of the ESP in the bottom right corner, and the RFID up top

Part 5, Setting up all of the forwarding accounts.

First off, set up a Google Drive account. I will start assuming you already have one, and Create a new forms. The key here is that for any data variables you want, make a question and have it written as a short/long answer.

Then, when it is done, go ahead and click "send" and copy the link. The long list of numbers/letters in that URL is the FormKey. Save it for later.

Next up is to right click the text entry box and "inspect element." This will bring up the source for the webpage, as well as the target entry ID's. Find then, and save for later.

Where to find ID for entry Source.

You now have two of the parts for making an "active/autofill link for Forms." All to do next is create it!

This form is what has worked for me:

https://docs.google.com/forms/d/INSERT_FORM_KEY_HERE/

The long chain of numbers at the end is irrelevant for what knowledge I know, I just know it needs it in order to "submit" the form.

Congratulations, now you can submit through Google forms with just a URL!

Part 6, setting up an account with Pushingbox:

I will reiterate why we can't just use that Google forms link with our Arduino, but for a more in depth reason, search "Arduino SSL" or look at my update from 2/21 below. That "HTTPS" you see every so often means that website has an encryption with it, for a provided security. Arduino's cannot handle that Encryption themselves, so we call upon a non-SSL service to translate, essentially.

Obviously the first thing to do is set up the account. It is straightforwards and simple, no private details, no payments. You can also set it up through Google.

The first page to visit is the "My Services" Tab, then "New Service." Scroll to the bottom of the list and choose "CustomURL." Name it, paste the link UP TILL "formRESPONSE=..."in the middle, and choose GET. POST will work, there is no difference here. I just choose GET.

Now that we have the Host URL submitted, we need to create a "Scenario." Go to My Scenarios, click "new." Fill out the details, as seen below:

Scenario

It will spit out a "DEVID" that you will use in a link to trigger the Google Forms URL you submitted.

Subsection: Variables:

Notice how on the picture above, it has the entry as $Test$, not just Test. This is because I have chose to write Test as a variable. On top of adding this tag, you must add &VARIABLE_NAME=VARIABLE into your code, but there is more on that later.

What this does it it plays telephone between the sensor, Arduino, Pushing Box, and Google.

Your final link through PushingBox should be something like this:

http://api.pushingbox.com/pushingbox?devid=STRINGOFNUMBERS

But, congrats! Now you have a Non-SSL Link that you can use to submit. If you want to test it, you can just throw it into your WebClient Example and see if it works!

Part 7, Arduino #2 Code:

These next two code files will be without a "Wire.h" library, just so you can test your link.

Here is an example code for the "EtherCard" Library, which is no longer updated.

Update: I changed the first one to the Wifi module/ESP. I ended up using it for the final project as the DRS library I am using uses ESP, not Ethernet.

Note that these are just for using Pushingbox, scroll down further for the complete code:

Then go to the Login With Amazon page (LWA) to create a security profile. This is essentially the gateway for allowed clients or devices to login to your Amazon account. This would be helpful, considering that in order to buy from Amazon, you must log in.

The next step is to sign up for the DRS push services, using that LWA you just created. Unfortunately, one of these must be created for each item, but they are very quick to make.

While making the DRS push service, you will need AWS SNS service. This is a "Simple Notificiation Service." All it does is send alerts, in our case, alerts about DRS. To make SNS, go to the developer console and search "SNS," Then start, then under topics, "Create new Topic." You will now have the Topic ARN for the above DRS.

Fill out all the details it asks for (Device name, picture, etc) and eventually you will find yourself on this page:

This is what we have wanted since we started this goose chase! This is the page for defining slots for your device. Again, because this is a private venture, we have to go about creating slots manually. This is slight hassle, but realistically it should not take longer than a minute or so per item if you have the ASIN.

Details page of the Project. We can see the slots for each of the items added.

Returning to the details page gives us the overview and the slot ID. This is needed to insert into the next bit of code below! So save it!

Last but not least, we need to return the the Login With Amazon and incorporate it into our Arduino. Once again, Mr. Carbonette has made it quite easy to do this. He has provided an example script that calls all of the security profiles and keys to provide a oAUTH security token. This will then go in the "AmazonTokens.h" and will provide a clearance for you to access Amazon!

Just for a heads up, we can use this Token system here, but for Alexa, google no longer will accept the tokens. We must use a full key system! Yikes!

This code is pretty straightforwards. IF there is an RFID card detected, the Arduino then transmits the first four digits of the ID of this card to the second, as well as the weight after a delay of 5 seconds. This will allow the user to set the item in in the pantry or fridge and have the scale get an accurate reading. This number can be manipulated, or a button can simply be added if desired.

This Arduino is a bit more complicated than the previous. In the Setup, the ESP and DRS system are initialized. Upon a "recieveEvent" event in the loop, the ID and weight are decoded and saved into a new array, still UID[]. The Arduino then sends these two to the Google forms, where they are saved for access later via phone, computer, or Alexa. The Arduino then does a quick check to see if the amount of the item is less than 10 percent. If so, it goes ahead and attempts to reorder. If there is no internet connection or the slot cannot be called, it gives an error.

Each UID will have to be programmed into the code as a new item, along with it's slotId and "specific weight." This is the weight that the scale measures it at. This issue of running a given weight is composed of two issues. The first is package weight and how it isn't factored into the food weight, and the second is that when transferring data over Wire, only integers can be sent and received. This is not an issue if you have all the SPI devices on one Arduino, though you may start running into other problems like possible interference and memory.

Again, Many other add ins can be places. We have an error LED connected to pin digital 4, but a success LED can be added as well.

Part 10, Test It Out!

That's all! The opportunities are endless, well as long as you have the RFID cards to fit them!

Also try expanding your range. The DXF files provided only makea 10"x10" surface, but why not a shelf? Or a shelf of a fridge? Or a full fridge or pantry!

So, What's next? As it seems, this is Version 1. It runs off of two Arduinos: an Uno and a Mega, and ESP8266, and and the extra peripherals. Here is a list of what I would like to change on it for the second version:

Larger surface area, possibly an entire fridge tray

Running on a single Arduino Mega

Adding a third SPI device, an SD card, to store texts of the card ID, specific weight, and item names. It would make it easier to update

Better long term connection to WiFi. This has a tendency to cut out after a given time, it might be code and it might be the module itself, or my WiFi.

Better Construction

Half bridges, spread among the scale, rather than a single full bridge load cell.

Have a 'level-out' system on the scale that auto-detects the final weight of the item placed.

Alexa! Scroll down

On a similar topic, errors. While running and building, I noticed a few errors. First of all is the ESP problems. It is very finicky, and quite frankly I can't give a proper way to wire it. It will work for 20 minutes, stop for 5, then work again for 2 hours. This could be the one that I have, too.

I also noticed a lot of random errors pop up while compiling. They would be for un-explainable reasons and I would often have to copy the code into a new file, to which it would then work. No clue on this one.

Right now I do not have an organization system for the Google Sheets. The Forms Dumps all of the "reads" into an excel, but I would like for it to be able to have a detection of each individual item and repeated items, then sort it out.

Alright, here is the bonus event I tried doing while waiting for my parts to come in. While the main part of this project was to create a DRS-enabled project, I thought it would be cool if I could get Alexa to read a Google Sheets file, one which had my inventory in it. This would be useful in a kitchen or household. The owner could ask "if he had any butter," "how much butter," or ask to order more butter through a separate DRS system. Here are my attempts at learning how to program Alexa.

Starting Off

Programming an Echo is not too far off from what we have sign up for previous to this. We will still be using the Amazon Web Services and LWA, but we will also be using the Alexa Skill Set. This is essentially broken into 3 parts:

Intent schema, which define the voice interface and system,

Utterances, which are what words the program will respond to, and

The Lamba function program, which is the program that runs when asked to

All this program does is respond with a simple "Hello World!" when the user says "Alexa, Hello."

Now this is neat, but now we need something much more complicated, we need a system to READ an excel file, and this means more authentication, but through Google. Go ahead and sign up for Google Developers then click "Enable API," and choose Google Drive or Google Sheets.

Click "Credentials" on the left side, then "Create Credentials," and choose the Service Account Key one. You will want to choose "p12" not JSON, as that is what we are using. You will be provided with a download of your security key- SAVE IT. this is the only one you will be provided for his ID and Key, so you don't want to lose it. Using the ZIP I have provided, throw the key in there and edit the "index.js" to match the files and details that you have.

If you wish to do the full command prompt base by yourself, you need to install node.js. This will allow you to install the Alexa Skills Kit, Google Spreadsheets, and many more Node Modules into the Modules file. I have it all set up for Sheets, so I suggest you just edit the file ;)

Warning: As of 2/25/27, I have yet to have the program call up the Excel Sheet data. I have, however, successfully logged into google and the sheets. I just now need to learn how to read them!

Now, Lambda:

Lambda is a computing function under the AWS services. It is essentially a cloud service for JS files in the Amazon Cloudbase, for certain triggers.

Instructions:

Click "create new"

For your trigger, you should select "Alexa Skills Set." These are the triggers I told you about above.

Throw the ZIP in there! It should work fine. If we didn't need the "Node Modules" file we could just put the code in the inline editor.

Your configuration page should look like this:

I also added in a 30 second time out under advanced settings.

The only problem I am having with this currently is that every other access command or so, It will not connect. No clue why it is doing this, which is a reason why I added the 30 second time out. It did fix it a little bit, but not as much as I had hoped, considering it was a 2 second time out before.

First Update: February 5th, 2017:

The rough outline of the project is there, however there are a lot of thins preventing me from moving forwards. I managed to get the RFID to work, which has been the easiest part so far. I used this library https://github.com/miguelbalboa/rfid, so many thanks to those who put this together.

The only issue is that this library is arduino, AVR systems. The Intel Galileo that I am using is a stm32f1 system, and while there are people out there running RFID on the Galileo, most are the 125Mhz systems. I will be moving to Arduino unless I can figure the RFID system and library on the Galileo, although so far I have had no luck in creating my own.

However, on the Galileo, I have been (kind of) able to connect it to the internet. What I have been able to do is use the example sketch to determine the IP address, which works. What I have not been able to do is host a web server. Instead, the program resets too fast and does not ping with my computer.

Considering it is the example sketch, I have high doubts that the program is the issue, and because there are only the 3 cables coming in and out of the Galileo, it's not there either. Here are my guesses:

The overall problem is the fact that I am on a college campus

It could be that my computer is connected to a WiFi router, but needs to be connected to the same Ethernet as the Galileo. I can test this by using my spare router I have as a switch.

It could be the settings I have on my computer, although I have high doubts because I can't even ping it.

The Galileo is not having a problem connecting to the internet, it is just the connection between the computer and the Arduino.

Chrome with IP Address, Command Prompt pinging IP address, and the Arduino IDE showing that there is in fact a connection

IP address and Code, CMD Prompt Pinging

In other news, I managed to get the Load Cell to read properly, then one of my friends knocked it off my desk which tore a wire inside. Bummer! I have another one on order and it will get here soon.

Once I can get the Galileo to host the webserver, I am going to send the data from the Arduino Uno which is running the RFID card to it. This will let me begin working on Alexa while I try and figure out the ESP8266 for the Arduino, or the RFID for the Galileo.

Will hopefully update soon!

-Tanner

Mini Update: February 6th, 2017

I finally got it the Galileo to connect and transmit to a "server"! The problem was in fact that hosting on Ethernet and loading over wifi would not work. I think this is because Arizona State University has them set up as different networks, which for whatever reason, does not allow them to connect. Anyways, I ran to goodwill and grabbed a second Ethernet cable to connect to my own router(Not exactly allowed to do this, but sometimes for progress you gotta break rules!). This immediately solved the problem. Next up on my plate is still going to be trying to figure out the library for RFID or the ESP8266. Woohoo!

Don't worry, I'll eventually post code, too! Right now everything has been down with examples, so there is not much missing!

BIG Update: February 7th, 2017

Made big progress the past hour. I decided to do a fresh install of Arduino IDE, along with trying out a Library (for Galileo RFID) that I have not been able to get to work, but I finally managed to get it to work! It is an older version of the on I posted above. I am not quite sure what quite happened, but sometimes clean installs do that! I am going to try putting the newer library in, as I have a code for it already, but that is the one giving me the conflict between the AVR and STM systems. I guess well see!

Now I have successfully:

Connected RFID 522 chip to a Galileo, received a UID Serial

Connected, calibrated, and received data from a Load Cell through an MX711 chip.

Connected and hosted a web server via IP on the Galileo(Ethernet).

Next will be to create a PHP website that is dependent on data put out through the Galileo, over internet. I am also going to buy a WiFi/bluetooth chip for the Galileo.

Next update I will also post my code and schematics for the electronics.

Cheers!

-Tanner

Update: 2/12/2017

This isn't too much of a update involving me as much as links posting to stuff around the web.

I found this link while looking for a handshake between AWS and Arduino. unfortunetly, it is "not possible." It's mainly the fact that the software and libraries can't handle it, and the Arduino can't physically handle the authorization process. The second link is somewhat of a disagreement, from AWS itself. It provides a good list of startup kits for IoT, and lists two for Arduino. This require the YUN board, which I don't have, and the SEEDuino and Grove. It also includes information for the Edison board, which I also don't have. Also from what I have read is that I do not think this will be able to handle my requests, yet.

I would also like to give a huge thanks to Brian Carbonette and his library for the DRS button system. It's a good stepping stone for what I need to do; Once the RFID tag is read and the weight is calculated, if it is under a certain amount, it will automatically reorder for you. An LCD and button can be incorporated. Alexa comes in as a side job; she can't exactly order for you, but you can ask inventory.

Another Major update (almost kind of): 2/13/17

So while I've been taking a break from connecting my Arduino to internet and google docs, I decided to spend the last week trying to connect Alexa and my Echo Dot to Google docs. Here's what all I did:

My first step was to load hello world' and get familiar with JS. With Arduino, I am familiar with the code, I can understand almost everything, with the exception of creating libraries. It also helped get familiar with intents and schema, as well as loading node.js and the mode_modules.

Afterwards, I spent some time fiddling around with a calendar reader. This got me acquainted and helped me learn about the different node modules. I also may use this for personal use.

Next, I tried modifying this code to download a text file rather than an ical file. Obviously this didn't work, but it was worth a shot.

After that, I found a simple looking Google Bulletin/docs reader, however I still could not get it to work.

Finally, I gave up on Alexa and branched out to node.js programs instead. This was very very worthwhile! I found two modules, edit-google-spreadsheets and some other module, I can't remember. I combined an example program with 'hello world' to proof it and voila, it worked!

The program was limited, though. I have been trying to read the document and it is giving me permissions errors. Google API does not recognize them as attempted log-ins, though, so I am not sure what's going on. my thoughts are that I don't have the access keys set up properly. I'll throw the screen-caps down below, and I am going to post on the AWS forums as well.

Update 2/10/17

I felt that this update was necessary so I put it on the top. I have successfully been able to load and tamper with examples on Alexa. What I am working on now is to incorporate HttpRequest into a program, so that Alexa can download a public text file, break it into lines, then read from it. I am also moving strictly to Arduino, rather than Galileo. It seems as though each board can pick two out of the three: RFID, internet, or the Load Cells.

I'm going to use Google Docs as a "handshake" for both Arduino and Alexa. Interfacing Arduino and DRS has been done, but that seems to be it. I have been looking and looking and as of right now, it seems there are no strict Alexa Skills for just reading a document, or at least I haven't found any.

As I have stated before, my programming experience is very, very, very, limited to single library, arduino based C. This adventure in JS is really a doozy. Getting it to compile is straightforwards, but once index goes into AWS, it seems to break and gives an error such as an inability to locate index js or more recently, an inability to initialize a module at line 103, column 16. Well, this is the XMLHttpRequest, so is it just because my nodes_Module needs to include this file? Does Amazon AWS not like this code? It compiles on my JS software just fine. I understand that this will most likely not be answered, but I am keeping it here for my thoughts. Thanks!

Update: 2/14/17

Read Below for other updates!!

Today I started creating the prototype. I used a laser to cut two 1/4" acrylic pieces. These are about 10"x10", for testing purpose, but can be up to whatever size and design you want. The Load cell fits right into the middle of the two pieces, and vinyl was cut to adhere to the bottom and top of the top peice. Pictures for more details:

Piece of acrylic, 1'x2', in laser cutter. Beginning of cuts

Both final cuts

Black vinyl was put on the bottom of the top acrylic piece, then the "KitchenDRS" logo was cut to put on top.

I also managed to get the AWS program to pass security for my google drive, now I just need to figure out how to have Alexa read the data. Exciting stuff!

In the meantime I am going to get my Arduino working for posting the data.

-Tanner

Most Recent Update: 2/21/2017

Okay, so here we are, a week before the contest ends and the project is still fragmented. Anyways, lets get into the update:

As of a while ago I had the ethernet and RFID working seperatly; great! Putting them together on the other hand, is much more difficult. Both are SPI devices, which means that they both want to communicate with the SPI ports. In case you don't know, there are 4:

SCK: Generates the clock from master. It essentially keeps everything in step and in time, syncronized.

MOSI: "Master out Slave in" -As simple as it sounds, it sends data out of master and into the slave

MISO:" Master in Slave Out: Same as above, just as it sounds.

SS/CS: This pin chooses the client or slave that you want to function. To choose one, the pin must be wrote as digital where the other clients must be written as high so they are not read.

And of course Ground, VCC

There are plenty more articles you can read online about this so I am just barely going to touch on it. I could get two SPI devices working JUST FINE for about an hour, then all of a sudden they stopped working. Not entirely sure why though. I tried rewiring completely, swapping to a different uno, swapping to a MEGA, still nothing. Definetely wasn't code either, because I had changed nothing. There was some interference involving the SCK and MOSI ports of the RFID that would mess up the connection. I tried for a couple hours to get it to work but it just would not work, so instead, I wired two UNO's together via Slave/Master Wire library and communicated through that. This worked to my advantage because the memory of a single Uno could not handle all of the library's, and I am currently using the MEGA for a different project I am working on. While it is not very efficient and not the most professional method, two arduinos will work if you cannot get the SPI system working. And yes, to end this, I did make sure to use an auxiliary 3.3 volt power supply.

Next is the internet side of things, connecting it to Google Forms. I chose Google Forms because I can submit simple data by calling a URL on the Ethernet Client. Or so I thought. Turns out that in 2013, Google switched over to SSL, so here is another quick history lesson:

SSL(Secure Sockets Layer), or HTTPS, uses a more secured method of data transfer among website and browser. Because it is so secure, it is too much for an Arduino to handle. Many IoT applications do not use SSL because of exactly that.

So how do we get around this? Well, the truth is we don't. We find a service to handle it for us! The source I am using is PushingBox.com ; It works by plugging in your google forms URL and variables, creating a new, non-SSL link for you to link your device through. The connection between variables and the link is quite tricky, and as of this post I do not have it quite figured out, but I do have it posting to my google forms.

What's next? Creating a scan box for the two Arduino's, RFID, and ethernet, then trimming up and fixing my current code problems, then posting it! Alexa is still halfway, however it may be difficult to get finished before the contest ends. I will still fiddle with it afterwards to try and get the code working, though.