Ten years in software

Last updated March 5th, 2016

I have been working professionally as a Software Engineer for the past 10 years. In that time, I've learned a huge
amount, gained a bit of confidence, and largely ignored the social nature of our field. I haven't given back to the
community and now feel like it's a good time to change that. I've been very lucky in my career thus far and want to
share the broad lessons that I've learned along the way.

This is part one of a series of pieces written reflecting on my career:

My first job as a programmer was when I was a student at UCI working at a small voice-over-IP company my friend was
founding. None of us really knew what we were doing, and as a result I ended up sort of aimlessly tinkering with
Asterisk and building a configuration language to handle our routing rules. The job earned
me a paycheck, but not much else. Due to the lack of direction and not being personally invested, I ended up parting
ways with the business in order to focus on school. I slowly lost contact with the founders, who stuck it out and
eventually ended up forming the company FlowRoute.

In some ways I felt like I missed out on an opportunity to be part of a "real startup" from the get go, but I just
didn't feel personally invested. I took away the feeling of wanting to build something I was passionate about as well as
able to work with people who I could really learn from.

As I approached the end of my degree, I found many of my peers bitterly working at jobs they were thankful for only due
to their paycheck. I didn't want to share that fate.

In my senior year of college, I took a course where a small group of students would work with a local business to build
software for them: identify the requirements, scope and plan the work, and regularly demonstrate progress made on the
project to the clients. This class paired me with Sendio, a small company which made an
anti-spam email appliance (sold as a physical 1U or 2U rackable host) that would plug into your network and act as your
SMTP gateway/proxy/"smarthost". It used a challenge-response system to defend against SPAM. It worked very, very well. I
ended up working with a few classmates to build a terminal configuration UI to allow non-technical users to plug a
monitor and keyboard into the host to configure the server's network and perform other maintenance tasks. The project
itself was well defined, well managed, and my future boss always made himself available to help. After enjoying working
with the project and the people, I accepted their offer and started working once the class finished.

Looking back, I was incredibly lucky to have this as my first job out of college. For the first year or so, the
engineering staff was two people: myself and my wonderful boss, Cameron Brown. He took the time to show me the ropes by
educating me on how the entire system worked, showing me how to approached diagnosing problems, and giving me
challenging tasks that forced me to deeply understand the system and its protocols. While my other college friends were
working on small features at large banks or building mind-numbing network driver installation UIs, I was lucky enough to
dive headfirst into the process of comprehending, building, maintaining, and growing an entire system.

During my time there, I was exposed to and ultimately felt responsible for the development and uptime of a system which
ran on hundreds of hosts in extremely varied network environments. There was a lot to do. I needed to learn about the
internals of RHEL flavored Linux;
the architecture of qmail; how the various protocols which power the
Internet (IP, TCP, DNS, SMTP, HTTP) and how the host OS can be configured and misconfigured;
how to configure SELinux correctly;
how to build RPMs;
how to perform remote software and security updates to minimize losing hosts in the process;
how PostgreSQL replication works and
how to use it in a master/slave configuration;
how cryptography and public data published in DNS is used to make mail delivery more reliable and trustworthy via
DKIM and SPF; how to recover from data loss/corruption in our
software stacks, etc etc. Exposure to these tasks was both exciting and daunting. There was a never-ending supply of
things to do, which was incredibly useful as my first job out of college. Today, I only vaguely remember the (now likely
out-of-date) details of each of these systems and tasks, but thankfully I still hold on to one invaluable skill: how to
read and verify documentation.

The best (and most challenging) part of working at a small company is that it forces you to find things out for
yourself. If you have a problem and the only other engineer you work with doesn't know the solution, you're on your own
to figure it out. This is the job. If you're lucky, there is documentation that describes how a tool is used, what all
of its internal components do, and lays out the pre- and post- conditions of its interface. If you're unlucky, you've
got to rely on outdated blog posts with misinformation or worse, rely on your own observations to develop hypothesis for
how the tools should operate. And if you're really unlucky, you've got only the source code or binaries. At the end of
the day, regardless of how much (or how little) documentation you have, it's critical to verify the documented behavior
and prove that the hypotheses you develop are correct.

Sometimes documentation is wrong, sometimes "experts" (strangers on the Internet) share misinformation, and often
generalizations made from your own observations are incorrect. The only thing to do is to just do the fact checking work
yourself. And the process to do this is simple and proven: use the scientific method.

Gather data: read documentation, ask questions, and make observations

Make a hypothesis: when X is true and Y happens, Z should become true

Verify with a test: prove or disprove the hypothesis by running a manual or automated test

No matter how big and complex software is, its behavior can be observed, its dependencies researched, its unknowns
broken down, and every assumption made can be verified. If you keep at it, you can wrap your head around anything.

Do you want to learn more? Was something confusing? Was something insightful? In the NYC area
and want to grab a coffee? Feel free to drop me an email at
sufian@gmail.com or send a tweet my way
@sufianrhazi

Disclaimer: Unless stated otherwise, the above words are my own and do not represent the
opinions of any person or business but myself.