Tuesday, October 1, 2013

If you have to deal with a large number of documents when doing queries against a Couchbase cluster it is important to use pagination to get rows by page. You can find some information in the documentation in the chapter "Pagination", but I want to go in more details and sample code in this article.

For this example I will start by creating a simple view based on the beer-sample dataset, the view is used to find brewery by country:

Thursday, July 18, 2013

Introduction

Developers are often asking me how to "version" documents with Couchbase 2.0. The short answer is: the clients and server do not expose such feature, but it is quite easy to implement.

In this article I will use a basic approach, and you will be able to extend it depending of your business requirements.

Design

The first thing to do is to select how to "store/organize" the versions of your document, and for this you have different designs:

copy the versions the document into new documents

copy the versions of the document into a list of embedded documents

store the list of attributes that have been changed into a embedded element (or new documents)

store the "delta"

…

You will have to chose the design based on your application requirements (business logic, size of the dataset, ...). For this article, let's use one of the most simplistic approach: create new document for each version with the following rules for the keys:

The current version is is a simple Key/Document, no change to the key.

The version is a copy of the document, and the version number is added to the key.

This looks like:

Current Version

mykey

Version 1

mykey::v1

Version 2

mykey::v2

...

...

With this approach, existing applications will always use the current version of the document, since the key is not changed. But this approach creates new documents that will be indexed by existing views.

For example, in the Beer Sample application, the following view is used to list the beer by name:

It is quite simple to "support" versioning without impacting the existing code, except the view itself. The new view needs to emit keys,value only for the current version of the document. This is the new view code:

The application can access the document using the key, but also get one version or the list of all versions, this is one of the reasons why it is interesting to create a key (mykey_version), and use it also to delete documents and related versions.

Use versioning

As an example, I have created a small library available on GitHub https://github.com/tgrall/couchbase-how-to-versioning, this library extends the Couchbase Client and overrides some of the operations : set, replace and delete. (the basic one: no TLL, no durability) As I said before this is just an example.

Line 5: instead of using the CouchbaseClient, the application uses the extended CouchbaseClientWithVersioning class.

Line 7: create a new entry

Line 9: create a new version, the boolean value to "true" force the versioning of the document

The application use other methods such as get a specific version (line 11), get all versions (line 13), delete a specific version (line 14), and finally delete the key and all versions (line 16).

So using this approach the developer controls explicitly when to create a version, since he has to add the boolean parameter in the set operation. In this small sample library it is also possible to do auto versioning, in this case all set and replace calls will create a version, to achieve that the developer just needs to call the setAutoVersioning(true) method. Something like:

With this approach you can provide versioning to your application with minimal code change. You can test it in the Beer Sample application, just do not forget to change the views as documenter above to only return current version of the documents.

Conclusion

As you can see doing versioning in Couchbase is not that complicated, but it is something that must be done by your application based on its requirements and constraints. You have many different solution and none of these options is perfect for all use cases.

In this specific sample code, I am working with a simple design where I create a copy of the documents for each version. With this approach also, it is interesting to mention that you can version "anything", not only JSON document but also any values. As I said before, this is one possible approach, and like any design, it has some impact on the application or database, in this case most the database:

Increase the number of keys and documents

Double - or more- the number of operations, for example when updating a document, the application needs to get the current value, create a version, save the current version.

Consistency management when adding new version and incrementing the version number (need to deal with errors when creating a new version, deleting the versions and counter....)

Thursday, July 11, 2013

Introduction

Clever Cloud is the first PaaS to provide Couchbase as a service allowing developers to run applications in a fully managed environment. This article shows how to deploy an existing application to Clever Cloud.

Clever Cloud provides support for various databases MySQL, PostgreSQL, but also and this is most important for me Couchbase. No only Clever Cloud allows you to use database services but also you can deploy and host your application that could be developed in the language/technology of your choice : Java, Node, Scala, Python, PHP, … and all this in a secure, scalable and managed environment.

The size of the RAM Quota for your bucket will have an impact on performance but also on the pricing.

5. Click “Add this Service”

You are done, you should receive an email with all the information to access your newly created bucket.

The mail from Clever Cloud contains the following information:

db_host = xxxxxxxx.couchbase.clvrcld.net

Location of the database, this is where the endpoint is located.

db_name = yyyyyyyy

Name of the Couchbase bucket

db_username = xxxxxxxx

Not used in Couchbase context

db_password = zzzzzzzz

Password to connect to the Couchbase Bucket

So you are now ready to use your bucket.

Note: In the current version of the Clever Cloud Couchbase Service you do not have access to a management console. If you want to get some information about the database or create views you need to do it from you application code.

Connect your Application to Couchbase@Clever-Cloud

The first step is to get some code, so let’s clone the “Couchbase Ideas Sample Application”, and install the dependencies, using the following commands:

Your application is now using Couchbase on the cloud powered by Clever Cloud. Let’s now deploy the application itself on Clever Cloud

Deploy your application on Clever Cloud

The easiest way to deploy an application to Clever Cloud is using git. The first thing to do is to add your SSH public key into Clever Cloud Console. If you do not have any SSH yet, follow the steps described on Github : “Generating SSH Keys”.

Add your SSH key

Note: As you can guess this should be done only once
Open the id_rsa.pub file with a text editor. This is your SSH key. Select all and copy to your clipboard.

Go to the Clever Cloud Console

Click on “Profile” entry in the left menu

Click on “SSH Keys”

Click on “Add a SSH Key”

Enter a name (anything you want) and paste your key

Click “Add” button

You are now ready to deploy applications to Clever Cloud. The next thing to do, is to create a new node application in Clever Cloud.

Create your Application

Click “Add an app” in the Application menu in the top menu.

Give a name and description to this application

Select the Instance type, in this case “Node.js”

Configure your instances, you can keep the default values for now, click “Next”

Check the configuration, and click “Create”

Your application is created, you are redirected to the generic information page, where you can find a Git URL that we will use to deploy the application.
You can navigate into the entries in the left menu to see more information about your application. In addition to the Information page, you can look at the following entries:

“Domain Names” to configure the URL to access your application

“Logs” to view the application logs

Deploy the Application

So we are almost there!
The deployment to Clever Cloud is done using a Git push command, so you need to add the deployment URL as a remote repository to your application, using the following command:

Once you have added the application as remote repository you can commit and push your application.

The last command pushes the application to Clever Cloud. It is important to note that Clever Cloud will always deploy the application on the “master” branch on the remote repository. The notation mybranch:master is used to mention it. If you work locally on your master branch just use “master”.

You can now go to the Clever Cloud console and look in the log and click on the URL in the “Domain Names” section to test your application.

You should be able to see your application, that is running on the Clever Cloud PaaS.

When you update your application, you just need to do a git push and git commit.

Conclusion

In this tutorial you have learned how to:

Create your Clever Cloud account

Create a Couchbase instance

Create and deploye a Node.js application

Feel free to test this yourself, with Node or other technology, as you can see it is quite easy to setup.

Wednesday, July 3, 2013

Introduction

During my last interactions with the Couchbase community, I had the question how can I easily import my data from my current database into Couchbase. And my answer was always the same:

Take an ETL such as Talend to do it

Just write a small program to copy the data from your RDBMS to Couchbase...

So I have written this small program that allows you to import the content of a RDBMS into Couchbase. This tools could be used as it is, or you can look at the code to adapt it to your application.

The Tool: Couchbase SQL Importer

The Couchbase SQL Importer, available here, allows you with a simple command line to copy all -or part of- your SQL schema into Couchbase. Before explaining how to run this command, let's see how the data are stored into Couchbase when they are imported:

Each table row is imported a single JSON document

where each table column becomes a JSON attribute

Each document as a key made of the name of the table and a counter (increment)

The following concrete example, based on the MySQL World sample database, will help you to understand how it works. This database contains 3 tables : City, Country, CountryLanguage. The City table looks like:

You see that here I am simply taking all the rows and "moving" them into Couchbase. This is a good first step to play with your dataset into Couchbase, but it is probably not the final model you want to use for your application; most of the time you will have to see when to use embedded documents, list of values, .. into your JSON documents.

In addition to the JSON document the tool create views based on the following logic:

a view that list all imported documents with the name of the "table" (aka type) as key

Friday, May 31, 2013

Introduction

When I was looking for a more effective way to create my cluster I asked some sysadmins which tools I should use to do it. The answer I got during OSDC was not Puppet, nor Chef, but was Ansible.

This article shows you how you can easily configure and create a Couchbase cluster deployed and many linux boxes...and the only thing you need on these boxes is an SSH Server!

Thanks to Jan-Piet Mens that was one of the person that convinced me to use Ansible and answered questions I had about Ansible.

You can watch the demonstration below, and/or look at all the details in the next paragraph.

Ansible

Ansible is an open-source software that allows administrator to configure and manage many computers over SSH.

I won't go in all the details about the installation, just follow the steps documented in the Getting Started Guide. As you can see from this guide, you just need Python and few other libraries and clone Ansible project from Github. So I am expecting that you have Ansible working with your various servers on which you want to deploy Couchbase.

Also for this first scripts I am using root on my server to do all the operations. So be sure you have register the root ssh keys to your administration server, from where you are running the Ansible scripts.

Create a Couchbase Cluster

So before going into the details of the Ansible script it is interesting to explain how you create a Couchbase Cluster. So here are the 5 steps to create and configure a cluster:

The group [couchbase-main] group is just one of the node that will drive the installation and configuration, as you probably already know, Couchbase does not have any master... All nodes in the cluster are identical.

Initialization of the cluster using the Couchbase CLI, on line 06 and 07

Then the system needs to ask all other server to join the cluster. For this the system needs to get the various IP and for each IP address execute the add-server command with the IP address. As far as I know it is not possible to get the IP address from the main playbook YAML file, so I ask the system to generate a shell script to add each node and execute the script.
This is done from the line 09 to 13.

I am adding the -vv parameter to allow you to see more information about what's happening during the execution of the script.

This will execute all the commands described in the playbook, and after few seconds you will have a new cluster ready to be used! You can for example open a browser and go to the Couchase Administration Console and check that your cluster is configured as expected.

As you can see it is really easy and fast to create a new cluster using Ansible.

I have also create a script to uninstall properly the cluster.. just launch

Tuesday, May 28, 2013

Already 6 months! Already 6 months that I have joined Couchbase as Technical Evangelist. This is a good opportunity to take some time to look back.

So first of all what is a Developer/Technical Evangelist?
Hmm it depends of each company/product, but let me tell you what it is for me, inside Couchbase. This is one of the most exciting job I ever had. And I think it is the best job you can have when you are passionate about technology, and you like to share this passion with others. So my role as Technical Evangelist is to help the developers to adopt NoSQL technologies in general, and as you can guess Couchbase in particular.

Let's now see in more details what I have done during these past six months and why I am so happy about it. I have organized the different activities in three types:

Outbound activities : meet the developers

Online activities : reach even more developers

Inbound Activities : make the product better !

Outbound activities : meet the developers !

A large part of my activities for this first semester was made of conferences and meetups. All these events are great opportunities for me to talk about NoSQL and get more people to use Couchbase Server 2.0, here a short list of what I have done:

participated to many Couchbase Developer Days in various cities (Portland, Seattle, Vancouver, Oslo, Copenhagen, Stockholm, Munich, Amsterdam, Barcelona, Paris, ...), these are one day workshops where I am helping developers to get their hands dirty on Couchbase

participated to Couchconf Berlin and Couchbase [UK] our main European events where I met many Customer and key members of the community

Yes! Be a Technical Evangelist means, at least for me, be on the road. It is very nice to meet developers from various countries, different cultures, languages, and… this also means tasting many different types of food!

Another interesting thing when you work on a database/infrastructure layer is the fact that it is technology agnostic; you can access Couchbase with multiple programming languages: Java, .Net,Javascript/Node, Ruby, PHP, Python, C, … and even Go. So with this job I met developers with different backgrounds and views about application development. So yes when I am at a conference or meetup, I am suppose to "teach" something to people, but I have also learned a lot of things, and still doing it.

Online activities : reach even more developers!

Meeting developers during conferences is great but it, it is also very important to produce content to reach even more people, so I have :

written blog post about Couchbase usage, most of them based on feedback/questions from the community

This task is quite interesting because it is the moment where you can reach many developers and also get feedback from users, and understand how they are using the product. I have to say that I was not as productive as I was expected, mainly because I was traveling a lot during this period.

Another important thing about online activities, is the "Couchbase Community" itself, many users of Couchbase are creating content : blog posts, samples, new applications, or features - for example I am talking with a person that is developing a Dart Client for Couchbase, so as Technical Evangelist I am also working closely with the most active contributor.

Inbound Activities : make the product better !

So the ultimate goal of a Technical Evangelist at Couchbase is to "convert" developers to NoSQL/Couchbase and get them to talk about Couchbase. Meeting them online or during events is a way of achieving this; but it is also great to do it directly with the product. This means participating to the "development" of the product or its ecosystem. Here some of the things that I have done on this topic:

talked a lot with the development team, core developers, product managers, architects, … Quite exciting to work with so much smart people and have access to them. During this discussions I was able to comment the roadmap, influence features, but also it is all the time an opportunity to learn new things about Couchbase - and many other things around architecture, programming languages, take a look for example to this nice post from Damien Katz .

contributed some code, yes remember Couchbase is an open source project and it is quite easy to participate to the development. Obviously based on my skills I have only help a little bit with the Java and the Javascript SDK. So if like me you are interested to contribute to the project, take a look to this page: "Contributing Changes"

but the biggest contributions to the products are such like doc reviews, testing and writing bug reports, and this is very important and interesting, since once again it helps a lot with the product adoption by the developers.

So what?

As you can see the Technical Evangelist job is a quite exciting job, and one of the reason I really love it, it is simply because it allows me to do many different things, that are all related to the technology. Six months is still a very short period, I still have many things to learn and to with the team to be successful, such as be more present online (blog, sample code, technical article, screencast, ..), be accepted in more conferences, and code a little more (I have to finish for example the Couchbase Data Provider for Hibernate OGM, and many other ideas around application development experience)

Finally, Couchbase needs you ! This is a good opportunity to say that Couchbase is always looking for talents, especially in the Technical/Developer Evangelist team, so do not hesitate to look at the different job openings and join the team !

Wednesday, March 6, 2013

Note : This article has been written in March 2013, since Couchbase and its drivers have a changed a lot. I am not working with/for Couchbase anymore, with no time to udpate the code.

A friend of mine wants to build a simple system to capture ideas, and votes. Even if you can find many online services to do that, I think it is a good opportunity to show how easy it is to develop new application using a Couchbase and Node.js.

So how to start?
Some of us will start with the UI, other with the data, in this example I am starting with the model. The basics steps are :

Monday, February 18, 2013

I read this question on Twitter, let me answer the question in this short article.

First of all you need to be sure your documents have an attribute that contains a date ;), something like :

To get the "latest hired employee" you need to create a view, and emit the hire date as key. The important part is to check that this date is emitted in a format that is sorted properly, for example an array of value using dateToArray function, or the time as numerical value. In the following view I am using the date as an array like that I will be able to do some grouping but this is another topic. The view looks like the following:

Wednesday, February 13, 2013

Most of the applications have to deal with "master/detail" type of data:

breweries and beer

department and employees

invoices and items

...

This is necessary for example to create application view like the following:

With Couchbase, and many of the document oriented databases you have different ways to deal with this, you can:

Create a single document for each master and embed all the children in it

Create a master and child documents and link them using an attribute.

In the first case when all the information are stored in a single document it is quite easy to use the entire set of data and for example create a screen that shows all the information, but what about the second case?

In this post I am explaining how it is possible to use Couchbase views to deal with that an make it easy to create master/detail views.

As an ex-Oracle employee, I am using the infamous SCOTT schema with the DEPT and EMP tables, as the first example. Then at the end I will extend this to the beer sample data provided with Couchbase.

The Data

Couchbase is a schema-less database, and you can store
“anything you want” into it, but for this you need to use JSON documents and
create 2 types of document : “department” and “employee”.

The way we usually do that is using a technical attribute
to type the document. So the employee and department document will look as follow :

This shows just the document, in Couchbase you have to associate a document to a key. For this example I am using a simple pattern : type__id , for these documents the keys will look like the following:

dept__10

emp__20

You can use any pattern to create a key, for example for the employee you could chose to put an email.

Note the “dept_id” attribute in the employee document. This
is the key of the department; you can see that as the “foreign key”. But
remember, the relationship between the department and employee documents
are managed entirely by the application, Couchbase Server does not enforce it.

I have created a Zip file that contains all the data, you can download it from here; and import the data into Couchbase using the cbdocloader utility. To import the data run the following command from a terminal window:

The View

Queries inside Couchbase are based on views; and views build indexes, so we have to create a view, a "collated view" to be exact.

The idea behing a collated view is to produce an index where the keys are ordered so that a parent id appears first followed by its children. So we are generating an index that will look like:
DEPT_10, Accounting
DEPT_10, Blake
DEPT_10, Miller
DEPT_20, Research
DEPT_20, Adams
DEPT_20, Ford
...

This is in fact quite easy to do with Couchbase views. The only trick here is to control the order and be sure the master is always the first one, just before its children.

So to control this we can create an compound key that contains the department id, a "sorting" element and the name (beer or brewery)

So the map function of the view looks like the following:

The key is composed of:

the department id extracted from the department document itself or from the employee document depending of the type of document

an arbitrary number that is used to control the ordering. I put 0 for the department, 1 for the employee

the name of the department or the employee, this also allows to sort the result by name

In addition to the key, this view is used to emit some information about the salary of the employees. The salary is simply the sum of the salary plus the commission when exists. The result of the view looks like:

With this view you can now use the result of the view to build report for your application. It is also possible to use parameters in your query to see only a part of the data, for example by departement, using for example startkey=["dept__20",0]&endkey=["dept__20",2] to view only the data -Department and Employees- of the deparment 20-Research.

The Beer Sample Application

You can create an equivalent view for the beer sample application where you print all the breweries and beers in the same report. The view is called "all_with_beers" in the design document "brewery". The view looks like:

Once you have publish it in production you can use it in the Beer Sample application, for this example I have modified the Java sample application.

Create a servlet to handle user request and on the /all URI.

The "BreweryAndBeerServlet" that calls the view using the following code :

The result of the query is set into the HttpRequest and the all.jsp page is executed. The JSP uses JSTL to print the information using the following code:

The JSP gets the items from the HTTP Request and loops on each items, then based on the type of the item the information is printed. The final result looks like :

Install Node

It is quite easy to install Node.js using the Windows installer provided at http://nodejs.org.

Once you have installed node, you can test is using the command line interface:

c:\Users\tgrall>node
> console.log(process.version);
v0.8.16

Node is installed. So far so good !

Install Couchnode

Couchnode, the Couchbase Client Library for Node.js, is a native module. The tool used to install native modules is node-gyp. So to install couchnode you need to install :

node-gyp

python

Visual Studio to have access to a C/C++ compiler

Install node-gyp

The node-gyp module is easy to install and you can install it using npm using the following command:

npm install -g node-gyp

The -g parameter indicates that this module will be installed globally and added to your %PATH%.

Install Python

GYP uses Python to generate the project, so you need to install it on your environment. I have installed Python 2.7.3 using the Windows installer.

Install Visual Studio

Finally you need a C/C++ compiler, the best way to get it is to install Visual Studio. As you probably know I am not a Windows expert and I do not know a lot about Microsoft development tools. I have downloaded Visual Studio Express the free development tools from here; it was sufficient.

Install Libcouchbase for Windows

Couchnode uses libcouchbase the C client library, so before running the npm install for Couchbase, you need to install libcouchbase itself.

You can download it from Couchbase site. The Windows versions are located in the left menu of the page. Download the zip file, that matches your environment. I have downloaded the "Windows, 64-bit MSVC 10".

Node-gyp will look for all the dependencies (DLL, library headers) into c:\couchbase directory, so you need to unzip the file in this folder. This location comes from the binding.gyp file of the couchnode project.

Install and Test Couchnode itself!

Let's check what we have done so far; we have installed:

Node

node-gyp

Python

Visual Studio

Libcouchbase

We are now ready to install and use couchnode itself. For this we can create a new node project.

mkdir my-app
cd my-app
npm install couchbase

The install command will:

Create a node_modules folder and put couchbase client library in it

When installing/building couchnode on Windows I had the following warning :