Week 12 - Hacked?

Software Engineering (CS 373) Spring 2017

What did you do this past week?

This week, I started work on Phase 3 of the IDB project. The two majors tasks of this phase are implementing search for
all attributes and all models in our database and creating a visualization of another group's project using
D3. So far, we have finished implementing search. We started off using
Flask-WhooshAlchemy which integrates well with
Flask-SQLAlchemy. One issue with this library is that it
is not very actively maintained and it only indexes rows as they are added or deleted. They do not index anything already
in the database. So, we switched to using
Flask-WhooshAlchemyPlus, a fork of the original library but one that has a function to index all rows already in the
database. It does not, however, support terms, something that
Whoosh on its own does.

What's in your way?

While the consequences of our web page going down right now are low, this week we realized the tremendous difficulty of
true site reliability. On Thursday, at 5:43pm, our page suddenly went down. I was SSH'd into the EC2 instance at the time
and I was messing with our search. Right at that moment though, our entire website went down and we got a popup
saying something along the lines of 'Your machine has been infected.' and a mailcious redirect to compliance-jessica.xyz/a.php.
Immediately, I posted on our group chat and as expected, every one was seeing the same issue on our page.

My first guess was
that we were hacked. We certainly had not taken proper security measures but at the same time, it did not make sense for someone
to want to hack into our website. I first checked Namecheap to see if it was now somehow pointing to a different IP but that
was not the case. Next, I checked our AWS EC2 instance and directly visited the Public DNS/IP of the instance but again,
I got the popup and redirect. Being SSH'd on the EC2 instance, I knew that the major files were not changed but just to
confirm I performed a git status but as expected, the only changes were the ones I had made in the last 10
minutes. I checked what Apache was pointing to and the contents of our wsgi file but everything was exactly as it should
have been. Google was not immediately very helpful either. Just to be safe, I changed all of our security settings to be
as strict as possible and I disallowed POST requests to our Public API. Still, the problem was far from fixed. At 6:03pm,
a full 20 minutes from the start of the problem, the website was back up.

We hadn't done anything but the website
was back. It was still not over, though. Javascript was not running on the model pages. I checked the console and
apparantly, our grid was missing the isotope() function. Instantly, we knew what the issue was.
Isotope is the plugin we use to display our grid system. It was also
used in initial phases for sorting and filtering but that functionality has since been moved to the back-end in order
to optimize speed. If Isotope's library was malfunctioning, it would explain why Javascript was not running and why
our grid was not looking right on the desktop site. We opened the link to the Isotope JS file and *boom*, there was
compliance-jessica again. We looked up the issue again but by this time, we knew that we were not directly hacked and that
this issue was affected more people that just us. There was now a
post on Hacker News. Unpkg
had pushed a bad nameserver update and sites using their CDN were all experiencing problems. Unpkg is a major CDN, recommended
by Facebook for use of the React library and it also hosts the Isotope library. Yet, one faulty update by someone somewhere later, our website was down. There
are certainly many lessons for us here. Especially for a site in production, the less dependencies, the better. If you need
an external library, host it locally or use subresource integrity. Test updates before modifying the production version. Not all of these safeguards may be
*necessary* or possible for a short-term project like this one but when creating something truly impactful in the future, these are issues that we must keep in mind.

What will you do next week?

I have an Algo problem set due on Tuesday. IEEE Robotics and Automation also has officer elections on Tuesday. While
I would love to run (I am a current officer), I am waiting for word from a few companies so that I can figure out
my Fall 2017 status. I also need to fix a bug with our search's highlighting. When users request a multi-word search,
right now, the second word search will search the html tags added by the first word's iteration of search and
replace as well. It should not be too difficult of a fix with RegEx.

What’s my experience of the class?

We are now talking about refactoring in class. This is following the assigned class reading of the
Refactoring book. It was a good book but my one complaint was that, given the content of the book, it would have
been better suited as a video lecture. It is much easier to follow code changes when seeing it being done than when
flipping through before and after code spanning multiple pages on a book. Professor Downing is going through the
examples in class now and it really is a much better way to teach refactoring.