Tuesday, November 17, 2009

Nimbits is a free, social data logging service. It provides web services to feed and store time series data (such as a changing temperature) into "Data Points" online. As your data is fed up into Nimbits you'll be able to use its many online data processing services, perform calculations, and receive alerts. You can also pull your data down into any connected system. Today, as values are recorded into Nimbits users see their changes in Spreadsheets, Visio Diagrams, on Facebook, IM, or using our free windows desktop interface. Software developers also use our free and open source SDK to write their own interfaces to the Nimbits Engine. There is even an app on the Android Phone Marketplace to view and enter data.

Here is a sample screen showing the main view of nimbits. It shows a category i made on my account called "Aquarium" and a Data point called TempF. I'm feeding a temperature reading from a temp probe connected to USB port on my computer.

Here, a google visualization API time series chart just like a google financial page! You can zoom and analyse your data just as easily. You can also set GPS Coordinates for data points and see your changes on Google earth.

The Blue line shows the temperature in degrees F being stored in a data point called TempF. I'm also performing a calculation on TempF each time it's written to.

After creating a new point called TempC we then click on the TempF point in the navigation tree and set a calculation. The Calculation is executed each time the Trigger (TempF) is written to. The formula is executed using the X Point (TempF again - usually the trigger is also the x variable) and the result is fed into the target point (TempC)

I also want an alarm on TempF. This will send me an email from the Nimbits system if a values is ever written that is greater than or equal to 90 degrees.

Lastly, lets set our properties on TempF.

Data compression means only new values written outside this value will actually be recored. I'm writing to nimbits every second so with compression of .5 - repeated writes of 80.0 will be ignored. As will 79.6 or 80.2 etc. 80.6 will be recorded.

Keep your data clean by setting experiation dates - if you don't care what a Temp was last month, then set data to be permenatly deleted if it's older than 30 days like i do here.

If you want to share your data point with other people or applications without giving out your SOA Key, make it public. I do here, so anyone can read this value via this URL:

(If you do click on this link, take a look at the resulting page source, you should just see one value returned, no XML)

Just the point name and owner email is enough if the point is public. Anyone can get this value to read into an excel spreadsheet, google spreadsheet or whatever. Click the link above - provided my Temp Probe is running you should see the current value.

You can also see here the options to have new values posted to your Facebook Wall or to send you an Instant Message.

Friday, October 30, 2009

The best example I like to give that helps explain why we developed Nimibits is:

Say you want to record a reading from a probe thermometer that read a temperature every second. How would you do it? An excel spreadsheet? SQL Server? Flat file? Well, that's 86,400 new records in your database or spreadsheet every day - 31,556,926 a year. Want more than one? Want to record a humidity level or anything else. You start to get a real performance and volume problem.

A data historian solves this basic need. Advanced compression algorithms store this data more efficiently than a table based system. Some other basic compression can take place as well, such as only recording changes in the data - so if your temperature stays at 80 degrees for an hour, that's really only one record - even though your probe is feeding you that same temp over and over.

It then provides a means to ask it: What was the last reading? What was the average over the last 8 hours? This time last year? Record high ever?

On top of that, let's say you'd like to get an email if your temperature goes over 80 degrees or how about automatically plugging your temperature reading into a calculation for relative humidity and storing the result of that calculation in another data point? Then, let's see the average value over the last year.

One of my favorite examples is the home brewer who connects his carboy full of beer to a data historian to log the flow of C02 coming out, the PH, Specific Gravity and temperature over time. Not only to automatically calculate the % alcohol but to permanently store this information in the database to compare to future brews.

Another is the system administrator who reads all of his servers memory/CPU utilization every second into data points. They always know what the current memory usage is, but what about last night at 3am? Can he get paged if the number is ever 100%?

A data historian solves these problems.

Now, we did not invent the concept of a data historian. In fact, I spent 10 years in the chemical industry working with historians that logged changes in chemical plant components - the ppm of a chemical in a vat, the vibrations of a pump - all feeding calculations to chemical engineers.

These systems cost 100's of thousands of $ and I always wondered what it would be like to hook my aquarium up to one and monitor my temp and PH from my IPhone.

I believe Nimbits is a brand new type of data historian. First, all of the data is fed directly into a web service and stored on Google's Cloud Computing Environment. This means virtually limitless storage and scalability. Further, since your data is already on the internet, sharing that data with other interested parties or even posting on social sites like Facebook and Twitter becomes a no-brainer.

Once again, imagine two lab techs on either side of the world with their changing data feeding nimbits data points on their accounts. They can both share points with each other and view real time changes in one shared spreadsheet.

Nimbits provides SDK's and Web Services so Software Engineers can plug into it any time their software needs to store a changing timer series value. Another one of my favorite uses of Nimbits is to store the time it takes for a long running function to complete. Drop the execution time in milliseconds into a Nimbits Data Point and at any time, view averages or changes to help optimize your program. Perhaps see how enhancements improved your performance. "Hey boss, that memory purchase improved performance by 300% - and here's the Nimbits data to prove it"

Any values that change over time can be stored and retrieved in the Nimbits System.

In my Blog, you'll see how to pull the current values of your data points into spreadsheets and graphical representations of your systems. View your data real time on mobile devices, and even tie into Wolfram Alpha which gives you an incredibly powerful tool to analyze your data using predictive algorithms and mathematics I will never understand.

There's so much more....but i hope this gives you enough insight to get as excited about Nimbits. Nimbits is currently in Beta mode and is due to release in early 2010. Check out www.nimbits.com for more updates!

Friday, October 23, 2009

One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for C# developers. If you want to do the same thing in JAVA, check out this post.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a C# .net class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue

using System;
using System.Collections.Generic;
using System.Text;
using System.Net;
using System.IO;

Update 7/2010 - this code is out of date now because we're no longer supporting the SOA key. Check out my more recent postings that have the Java code you need to authenticate directly to Nimbits with your Google Account.

One of the most common tasks with Nimbits is to write a value to a data point, and request the most recent value. So much so, there is an entire web service dedicated to this task.

You can call these services from any platform or programing language. This example is for Java developers. If you want to do the same thing in C# check out this post. I would love to see someone post a variation in another language like python.

A developer can read and write data to nimbits by calling our REST based web services. In order to do this, you need to log into app.nimbits.com and create your data points in advance. Then get your SOA key. Learn more about SOA keys here in my blog.

With a SOA key, you can access your Nimbits data from other applications you develop without being logged into Nimbits with your Google Account.

Here is a java class you can add to any project to read and write data to and from your data points. It's just two functions, getCurrentValue and recordValue

the magic happens when you go to nimbits and set up calculations and alerts to trigger every time you write to a data point. Other users around the world who you share your data with will see your changes in real time.

Please note we are phasing out SOA Key's in favor of OAuth. Please contract me if you have any questions - bsautner@ google's free mail service.

This blog posting will help you understand how to access your private Nimbits historical data using a SOA key instead of your google account. Please note you can always make your points public so anyone can see the data and make it much easier to access. See my other posts on sharing your data if that interests you.

When you log into Nimbits with your Google Account, you have access to all of your historical data and applications. However, we need to provide you with a way to tell the nimbits system who you are when your using or developing an application that dosn't have the ability to provide Google account information. The solution is your SOA key.

When you log into app.nimbits.com you can create a SOA key which is a random 64 character string like this:

drGcVC5mJOLiCDEFAIKf288yCDighMKtq02mzy0oT0aSiP1vscLlaHABT4GInFmC

just click the Manage SOA Key Menu and you'll be prompted to either rest your key to a new one or to have your current key emailed to you. Your key will be emailed to the Google account you're logged in with.

You should keep this key as safe as you keep your google account password since it can be used to read and write values to your data points. In order to do more advanced things like delete data or create points, you need to log into the application normally.

If you feel you need a new key, just use the menu to generate a new one. The old key will no longer work. Keep this in mind since any apps or spreadsheets you have will need to have their key updated.

Using a SOA key extends your ability to access your Nimbits data from apps outside the Google environment. Read more in this blog about making your data points public so they can be accessed without a key by anyone anywhere (for those of you who don't care who knows what the current humidity level of your humidor is but still want to give your facebook friends hourly updated :-)

Read more on this blog about using your SOA Key in a .net or java application to use nimbit.com as a data source.

Saturday, September 26, 2009

Update: A more recent article has a more generic Java Class for reading and writing to Nimbits or any App Engine Project that uses REST web services. Check it our here or enjoy the more rugged code below.

In this post, I'm going to show you how to use a temperature probe connected to your computer and feed that temperature into Nimbits so you can access that reading anywhere in the world. Each time the Fahrenheit temperature is read into the system, we'll automatically convert it to Celsius using the formula engine and set up an alert to email us automatically if the temp goes beyond a certain range.

I'm not going to go into any great detail on setting up the development environment. One should have Eclipse up and running and follow the instructions for installing RXTX on linux - Thankfully, you can skip to the bottom of these instructions and just do Method 3.

Step 1. Create your project

Launch Eclipse and select File - New Java Project

Name the project TempProbe

Click Finish

Right click on the TempProbe Project in Package Explorer and Select New - Class

Name the Class runProbe and Check to create a public static void main

Right click the project again and select Build Path and then Configure Build Path

On the Libraries tab, select to Add An external JAR and browse to where you dowloaded the RXTXcomm.jar file.

Step 2. Gather information

When you connected your Temp Probe USB adapter and new tty device was added to your system. You'll need that port information. Open a terminal and type:

ls /dev/tty*

when i connected my temp probe, i saw a new device called /dev/ttyUSB0 yours may be different.

You'll also need your nimbits.com SOA Key. Using a key allows you to feed data into nimbits from a system that's not logged into google with a google account.

When you log into nimbits you're authenticated using your google account and can access all of your information. When your not logged into google, you need to provide your personal key which is a 64 character string the application assigns to you. It's important you keep this key as private as you keep your google password since anyone who has it can access your Nimbits data via the web services (but not any other information related to your google account)

Log onto Nimbits and select Manage SOA Key at the top menu. Here, you can have your key emailed to the google account your logged in as. Be careful if you choose to get a new key. You can do this at anytime but any software you already have using the nibmits service will need to be updated.

While your logged into Nimbits, let's also create two new Data points under any catagory you like. Call the first point TempF and the next Point TempC

Click on the TempF point. You'll see an empty chart, and above it, the menu for the many ways you can work with this point. Select the point name from the menu to edit it's properties.

Select the calculations tab to edit the calulation. Read more about how calculations work in Nimbits. When your probe writes to the TempF point, it will trigger the calculation, use the new TempF value as the x parameter and send the result into the TempC point.

Save, you should be good to go. In face, if you click test you should get -17.777 because your curret value for the new TempF point is 0.

You'll also want to set the properties of the point. You can enter C and F for the units of measer. Also, note the compression setting. Compression means you'll only record values that are outside the compression range. So if your reading 77 degrees and your compression is set to 0.5, a reading of 77.1 will get ignored but 77.6 will be recorded. This is usefull for filtering out noise and small changes your not interested in. Let's give our temp points a compression of 0.5