Category: AWS

I’m entering the final stages of a figure search engine, a nice wrapper for the new API method discussed below. It’s also a chance to properly release data mined directly from arxiv figures, and take advantage of the lambda + S3 processing pipeline I developed when pushing the p2t algorithms to cloud initially. Attached isContinue Reading figure meta data

As mentioned, after doing some experiments for the KDnuggets article, I bundled some of the existing API methods into a new one, which will extract from a page figures that have x/y scaling information. The JSON output is well suited to elasticsearch or your favorite flavor of NoSQL eg., an extract of the response: [{“input”:”tmp/quant-ph0002044-9.png”,Continue Reading new api method for data image search

Some time ago I launched a little project, mining data from arxiv; you can read about it in other blog posts. Specifically, I modeled figures from about 500k figures as Gaussian mixture models, in order to create some features, so figures might be ultimately represented as graphs for comparison. More ordinary methods might suffice tooContinue Reading arxiv mining

At the time of writing, AWS API gateway doesn’t support gzip requests, so I’ve been handling this at the lambda function itself and client side. Obviously compression makes a dramatic difference w.r.t performance, just ask the guys at Pied Piper 🙂 Another curious absence is support for multipart form data; attached a screen grab fromContinue Reading API Gateway Perf

We attended the AWS San Francisco Summit in April and were really impressed with the breadth and depth of the presentations. A stand out included this talk from Adam Boeglin on optimization. As we prepare to finally commercialize RESTful APIs, it’s clear a few lambda functions need a little tuning; this talk as some greatContinue Reading EC2 Optimization

One of the distinct advantages of devOps philosophy as implemented using AWS is the ability to rapidly kick out a new service. The new template-match API is a good example. Knowing the algorithm, I was able to quickly pull something together from the p2t C++ library, an example input image attached. The algorithm determines whereContinue Reading DevOps with AWS

Back in 2015 I began working on porting native p2t code to lambda functions. I dedicated the first couple of blogs on coming to grips with AWS and microservices; much has happened since. In the last couple of months I was excited to learn that AWS added binary support to API gateway. This is tremendousContinue Reading AWS API Gateway Binary Support

I’ve put some notes together, summarizing the steps involved in taking the p2t backend to AWS. Overall a very intuitive and relatively straightforward process, with very few gotchas; I cover these in these notes (recording below), and in prior blogs. As we prepare for a soft launch/ beta test, and then the collision conference inContinue Reading Soft Launch Prep

The day job continues to keep me busy, however, I managed to connect the dots with the compute infrastructure. Input document collections now happily cascade through a chain of processes, wrapped up as lambda functions. Meta data is pushed into dynamoDB tables, storing input and output state, allowing for browsing and downloading of outputs usingContinue Reading Divide & Conquer