Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

NoSQL matters in Recognition Service
Catchoom technology recognizes an object by searching through a large collection of images in a fraction of a second. Catchoom targets application developers and integrators.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Visual Recognition: “Identify an object in front of the camera by comparing it to a huge collection of reference images”David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Minimum requirements: - a public API for the final users to perform Visual Recognition - a private API for the customer to manage the Collections and get statistics - a nice website for the customer, providing the functionality of both APIsDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Must be flexible: - A customer who does Augmented Reality, and needs a 3D model (binary format) in the item - Another one who needs just the item id - Our data model needs to allow everything (structured and unstructured data)David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Must be reliable: - Images or data should never be lost - Avoid single points of failure - We need redundancyDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Must be very fast: “Layar has been using Catchoom’s Visual Search technology since the launch of Layar Vision, allowing users to quickly view the AR content placed on top of images by just pointing their camera to the image. We’ve benchmarked Catchoom’s technology in 2011 against 3 of their main competitors and found they had the best results both on speed and on successful matches (including lowest false positives)” Dirk Groten – CTO of LayarDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Data models: - Collection: a set of items. Has at least one token. - Item: has at least one Image. Has metadata. - Image: you want several images if the item has different sides, logos, flavours... - Token: for authenticating the requests.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Scalability: - Need to scale different components, separately - Load balancing using Redis Lists: BLPOP: Remove and get the first element in a list, or block until one is available - But focus on the bottlenecks!David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Unstructured data: metadata - Metadata is optional and unstructed, can be from a json to a binary blob - Cant fit in a SQL table, and would be too slow - Serve the data from Redis, and use S3 as a backup - Warning: in the future, if we have huge metadata files, Redis will get out of memory. Well improve this approachDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Availability: - Avoid single points of failure. Replicate everything! - Replicating a SQL server is painful - Redis instances configured as Master/Slave - When the master dies: - promote a slave to be the new master - reconfigure the other slaves to use this new master - Redis Sentinel does this (beta)David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Sorted Sets: - To create indexes and filters - In example, “Most recognized images” (sorted by hits) - Updating the Sorted Set, no need to reconsolidate: ZADD Add one or more members to a sorted set, or update its score if it already existsDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Django apps compatibility: - we use Django and several contrib and external apps. - (“Standing in the shoulder of giants”) - but no support for NoSQL in Django ORM - dropping SQL is not an option! - we use MySQL. South migrations.David Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Summary: - We use a combination of SQL and NoSQL - Using NoSQL was necessary to meet the requirements - There are a lot of different uses for NoSQLDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom

NoSQL matters in Recognition Service
Recommendations: - There is no silver bullet - Use the best tool for each task - But avoid unneeded complexity! - Try Redis. Dont do a migration, just add it to your stackDavid Arcos | @DZPM Catchoom | http://catchoom.com | @catchoom