README.md

findme 🕵️‍♀️🕵️‍♂️

findme is a serverless application to find unlabelled photos of you on twitter using machine learning.

Users provide a search query to retrieve tweets from the Twitter API. Face recognition is used to compare all faces found in the search results against the user's twitter profile image. Tweets with matching faces are shown in the client-side web application.

dependencies

If you want to deploy this project you will need an instance of the Apache OpenWhisk platform, access to a Redis database and credentials for Twitter and Auth0 applications.

install project dependencies

build custom runtime image (optional)

If you don't want to manually build a custom runtime image, you can use the following pre-existing image: jamesthomas/action-nodejs-v8:tfjs-faceapi. The serverless.yml is already configured to use this runtime image. If you need to make changes to the runtime image, follow these steps...

run web server for client-side app

architecture

This application has four serverless functions (two API handlers and two backend services) and a client-side application from a static web page.

Users log into the client-side web page using Auth0 and a valid Twitter account. This provides the backend application with the twitter profile image and API credentials.

When the user invokes a search query, the client-side application invokes the API endpoint for the register_search function with the query terms and twitter credentials. This function registers a new search job in Redis and fires a search_request trigger with the query and job id. This job identifier is returned to the client to poll for real-time status updates.

The twitter_search function is connected to the search_request trigger and invoked for each event. When this function is invoked, it uses the Twitter Search API to retrieve all tweets for the search terms. If the tweets contains photos, each tweet and photo url is fired as a separate tweet_image trigger event.

The compare_images function is connected to the search_request trigger. When this function is invoked, it downloads the user's twitter profile image along with the tweet image and runs face extraction against both images. If any faces in the tweet image match the face from the user's profile image, tweet ids are written to Redis before exiting.

The client-side web page polls for real-time search results by calling the API endpoint for the search_status function with the job identifier for the search. If tweet ids are returned from the search results, those tweets are displayed on the web page using the Twitter JS library.