Matthew D. Billenstein
matt@vazor.com - @mattbillenstein - github.com/mattbillenstein - 408-761-2626
Education
University of Cincinnati; Cincinnati, OH
BS in Computer Engineering with VLSI Minor June 2001
Awarded with the Professional Practice Outstanding Co-op Award for the
Computer Engineering class of 2001
Work Experience
UnitedMasters, San Francisco, CA 05/2016 - 11/2017
Staff Infrastucture Engineer
Built backend infrastructure from "Initial commit" to launch of first
user-facing product. Technologies used:
Nginx / OpenResty ssl termination / load balancing
Flask / gunicorn wsgi application server
Mongodb database storage
Redis cache / task queue (Python rq)
AWS EC2, VPC, Route53, S3 Hosting / Networking / DNS / Cloud Storage
Saltstack config management and deployment
Supervisord Process management
Airflow / Python Data task management and ingestion
GCP / BigQuery / GCS data warehouse
NSQ event streaming queue
Metabase Data dashboarding and visualization
Fluent Log aggregation
Squid HTTP(s) proxy
Ubuntu LTS Production deploy target
Mac OSX Dev deploy target
Buildkite Build / CI frontend
Rollbar Exception monitoring (Python and Javascript)
Server Density Server and service health monitoring
I have fairly deep knowledge of these technologies and have deployed and
managed them as a one-person infra team while also delivering on various
backend product features including managing the key data schemas, data
ingestion and export code, timeseries data storage (~100B datapoints), and
in-office IT including network rack and switch management.
OSS contributions mainly to Python libraries we used including fixing
issues in requests (urllib3 https://github.com/shazow/urllib3/pull/1174),
flask-classful, and a couple other smaller libraries.
Led weekly engineering focused meeting looking to improve tools and
processes used within the team as well as teaching engineers more about the
various software, systems, and design patterns that make for building
robust engineering solutions.
Apptimize, Menlo Park, CA 09/2015 - 03/2016
Contract infrastructure engineer
Reworked infrastructure hosting and deployment using AWS VPC and Ansible
for a microservices based Java backend. Automated provisioning,
deployment, configuration managment, and DNS. Analyze cloud costs and ways
to reduce overall spend. Rearchitected loadbalancing and failover
strategy. Improve overall system security leveraging best practices in AWS
VPC.
Leveraged infrastructure automation to improve developer environment for
working on backend services and event pipeline supporting both Linux and
OSX greatly reducing time for onboarding engineers.
Wrote CI build and deployment automation for mobile SDKs covering Android
and iOS and improved overall robustness of build system and artifact
hosting.
Homejoy; San Francisco, CA 05/2014 - 04/2015
Principal software engineer focused on infrastructure, systems,
provisioning, data engineering, continuous integration, test automation,
and process.
Built new backend infrastructure to host and migrate the website from
Google AppEngine to Amazon EC2 using open-source solutions. The AppEngine
datastore was replaced with Postgres, Redis for caching, and beanstalkd for
task queues. Custom components were developed in Python to maintain
backwards compatibility with AppEngine in order to facilitate a smooth
transition to the new backend including tools to emulate cron and task
queue workers. A software shim was used to translate the GAE APIs to work
with the new backend services. Deployment and configuration management was
implemented with Ansible.
Created a development environment for native development under OSX and
Ubuntu for the new system with automated installation and very close parity
to the tooling used in the production environment.
Deployed and administered Jenkins for CI including Github pull request
testing plugins.
Developed a new ETL and data export system to support data
science/engineering with the goal of centralizing data in disparate systems
into a single service for custom SQL analytics using Google BigQuery.
Deployed Tableau as the frontend of choice for data visualization.
Gave several talks and tutorials to the engineering team in order to get
them up to speed with a myriad of topics such as Unicode, systems and
software architecture, and data engineering.
Blueshell Games; San Francisco, CA 03/2014 - 05/2014
05/2015 - 06/2015
07/2015 - 09/2015
Contract software engineer
Implemented and tested an event-based data pipeline into Google BigQuery
for custom analytics and reporting. Wrote tools for schema management,
bulk and incremental data loading, and data archival. Migrated dozens of
terabytes of data from an existing Hadoop cluster to Amazon S3 for archival
and later into Google Cloud Storage for bulk data loading into BigQuery.
Refined bidding and pacing strategies in a real-time bidder (RTB, Mopub)
system focused on user acquisition in the mobile games market. Tested
varying bidding strategies as well as approaches using machine learning
techniques (python sklearn).
Infrastructure and deployment migration from Rackspace to Linode and
Ansible.
Flingo.tv (now Samba.tv); San Francisco, CA 03/2010 - 03/2014
Senior Engineer focused on backend services, database, deployment,
analytics, networking, data center, Linux system administration, et al.
Heavy use of Python and a myriad of other tech:
http://vazor.com/media/tech.png
Architected and built service stack supporting video fingerprinting using
Python/C++. Spec'd and racked server/network hardware to host this
application as well as legacy applications coming from EC2. Deployed
monitoring (nagios) / metrics (graphite) and other custom backend services
to support this product.
Rearchitected backend video streaming database from MySQL/SQLAlchemy to
MongoDB with a custom object mapper. Overall this solution is much more
efficient for serving video content metadata through the frontend API
service while providing a convenient programming interface. Refactored
various frontend/backend services which used the old MySQL backend to use
the new MongoDB backend.
Built EC2 deployment tools and rearchitected deployed infrastructure to be
more robust and provide better system monitoring. Deployed haproxy and
migrated from apache2 to nginx. Removed several single points of failure
from the system by developing scalable solutions for hosting content and
caching services for the connected TVs.
Designed and developed beta versions of internal metadata editor, analytics
dashboard with map/reduce backend, and application exception logging and
reporting. These tools provice real-time content management and analytics
services to the rest of the company.
Migrated development environment from subversion to git including importing
all projects, branches, and source code history. Deployed a gitosis server
with Trac for project/issue managment.
Slide; San Francisco, CA 06/2008 - 03/2010
Technical lead on a new Facebook game (http://apps.facebook.com/topfish)
driving an aggressive six-week schedule with a small focused team. Key
technologies used were Python, AS3, Javascript, AMF for data transport and
object reflection between the server and the client, and a custom ORM
between our backend storage and backend application software layer. Wrote
custom tools for data analytics and logging including funnels for A/B
testing the new user and payment flows.
Technical lead on http://superpokepets.com launch focusing first on
building a custom forum with high game integration. Later the site was
extended to bring the key game experience to the destination site while
also offering some features not present on the Facebook or Myspace version
of the game.
Developed new features in SuperPoke! focusing on user retention,
engagement, and monetization though virtual goods.
BitTorrent; San Francisco, CA 03/2007 - 06/2008
Developed large portions of a web analytics application targeted at p2p CDN
customers utilizing BitTorrent's DNA (http://www.bittorrent.com/dna)
infrastructure. Implemented large portions of the backend services (MySQL
DB, Skytide OLAP Business Intelligence server, and Apache Tomcat Java
servlet for serving AJAX JSON requests for OLAP server data), middleware
(SQLAlchemy ORM on top of Python MySQLdb), webapp in Pylons (Rails inspired
Python MVC framework using Mako templates). Integrated/implemented frontend
javascript widgets using the excellent Ext javascript library
(http://extjs.com). Also wrote Hadoop (Google GFS/BigTable inspired cluster
computing software) map/reduce log parser/aggregator for ingestion of data
into OLAP server.
Supported and maintained an internal webapp for managing BitTorrent.com's
content catalog. This is also a Pylons webapp using a Python ORM
(SQLObject) on top of MySQL and talking with several internal backend
services (search, trackers, etc.) for changing content deployment status.
Implemented features in BitTorrent.com's store for organizing TV content
into collections by series and season. This required new features in the
middleware/DB and scripts for migrating existing data into new DB schema.
Wrote backend scripts for content metadata translation (XML, spreadsheets,
and other flat-file formats to an internal standard) and ingestion as well
as running backend ingestion process for the entire BitTorrent.com content
catalog.
LSI Logic; Milpitas, CA 06/2001 - 03/2007
RapidChip Slice Design Engineer 06/2004 - 03/2007
Developed and refined slice design flow and key automation components
(heavy use of Python and GNU Make for this) which greatly improved overall
design efficiency across the slice design organization.
Extreme2 family slice engineer working on executing several of the slices
in the family from specification, bonding, layout, and final verification
(DRC/LVS).
Extreme1 family slice engineer supporting three key slices in LSI's
portfolio targeting several instances ongoing at LSI.
ASIC Customer Engineer - SVDC Bay Area Team 06/2001 - 06/2004
Developed and supported a number of special-purpose software tools across
the Design Center/Physical Design Center organization addressing a number
of shortcomings in the LSI FlexStream toolkit. This included a new
methodology and tool for addressing routing congestion in SOC ASIC designs
in Python.
Executed portions of a number of live designs in the design process
including presales estimates, routing/timing closure, timing driven
placement/resynthesis, power drop/electromigration analysis, and cross-talk
analysis. Also acted as technical lead on two large ASIC projects being the
main interface between LSI and the customer’s engineering staff for
multi-million dollar ASIC programs.
Developed and deployed Primetime SDC constraint translation tool which
translates customer SDC constraints to work with LSI backend
placement/optimization tools. Implemented in C/C++ using relevant
netlist/technology APIs and SDC parser API.