The Need for Speed: A Toptal JavaScript Coding Challenge Retrospective

Toptal's Latin America Community leader, Eva specializes in AI and speaks at events worldwide about game theory and data science.

SHARE

Toptal started the JavaScript coding challenge web app as a way to attract people to our conference booths. Seeing how successful it was, we decided to make a pilot on the web, open for everyone in our community and their networks.

At launch time, we had encouraged motivated developers to come up with creative ways to score high on the overall JavaScript coding challenge. Some of the key success factors of great independent freelancers are the ability to think outside the box and find creative ways to work within a set of constraints.

JavaScript Coding Challenge Questions

The gauntlet consisted of a number of JavaScript questions—akin to those that might be used as interview questions—ranging from really basic JavaScript challenge questions:

box.double = function double (x) {
//return x doubled
};

…to more intermediate ones:

box.dateRank = function dateRank (x) {
// x is a date in 2019 as string (example: "06/30/2019")
// return the rank of the day in 2019 (i.e., "09/01/2019" translates to 244)
};

We wanted both beginners and advanced developers to have fun and invite their friends and colleagues to compete with them for the highest scores. The questions were sorted by points, so that even junior developers could score some points. The order for questions having the same amount of points was random, so the experience was slightly different every time, while the full set of questions stayed the same throughout the week.

The goal was to complete as many tasks as possible within the time limit of three minutes. In case someone found a way to complete all the tasks in the JavaScript coding challenge, 10 points would be awarded for each second that was left. We allowed multiple attempts to complete the challenge.

The first thing that we anticipated would happen is that people would take some time to solve the questions at a leisurely pace, and then copy and paste the answers into the web application.

After one hour and 40 minutes of launching the challenge, the first person followed this approach and got the 1445 maximum points that the application allowed, plus some extra points that we awarded for every second left.

A Turning Point in the JavaScript Coding Challenge

With the copy-and-paste approach, top contestants had nothing more to gain from focusing on coding the answers themselves. Instead, they turned their attention to bringing the speed to their automation skills.

The easiest approach at this point was to write some JavaScript that would solve each task while waiting on a loop until the buttons were ready, and copy-paste it into the browser console:

One thing to note is that, in this version of the speedcoding challenge, the code was tested only on the client side. Because of this, it was possible to simply send out the answers to the test cases instead of the code. This allowed for an optimization and to cut some milliseconds on the client side.

For three days, the scores remained the same. Some people were writing micro optimizations for their code, and several people were submitting their solutions in a loop, hoping the server would become less crowded so they could get a few points ahead. We were due for a big surprise.

Breaking the JavaScript Coding Challenge’s Maximum Score

First, let’s do some quick math: We had a total of 1445 points awarded for the completion of all tasks, and a time allowance of 180 seconds. If we award 10 points per second left in the application, the maximum theoretical achievable score would be 3245—in the case of submitting all answers instantly.

One of our users achieved a score of over 6000, which kept steadily growing over time.

How could someone get such a high score?

After a brief review, we found what had been happening. Our top contestant, a professional full-stack developer and Toptaler with more than 15 years of competitive programming experience, found a loophole in the setup of the coding challenge. He spawned multiple bots, which slowed down the server; meanwhile, he could complete the same task (the one that awarded the most points) as many times as possible and assign their scores to a single entry, continuously adding to that entry’s score.

This wasn’t against the rules, as we allowed for creative solutions; however, the particular method that he was using caused the server to be considerably busier, making network requests slower for everyone else. The first thing that we did was increase our server power, which only made him go from 56,000 to 70,000 points and stay in the first place.

While we didn’t want to intervene in the way people were interacting with the challenge, these attempts slowed down the server to the extent that the challenge was hard to use for other users, so we decided to fix the loophole.

The fix made it impossible for other people to achieve the same score during the last day of the JavaScript coding challenge. Because of this, we decided to extend the number of awards given to the top contestants. Originally, the top prize—a pair of AirPods—was supposed to go to the one top contestant only. In the end, AirPods were given to those holding the first six places.

Even our highest scorers had some difficulty starting out. In fact, of all the people who made five attempts or more, the top score for anyone’s first attempt was 645, and the median score for first attempts in that group was just 25 points.

By the end of the contest, one could guess the strategy being attempted from the total number of attempts. While some were more efficient than others, the top contestant far and away had the highest attempt count.

Moving Forward

What does the future hold?

This was only the first JS coding challenge iteration. We want to do many more JavaScript programming challenges in the future, learn lessons from the previous runs, and make it even more exciting. The first thing we want to do is to implement attempt throttling to limit the number of submissions. We also want to expand beyond coding challenges in JavaScript, making them available in a wide range of programming languages.

Lastly, while we are taking measures to make the JavaScript coding challenge more secure, we plan to keep allowing the use of bots and other creative approaches for future challenges.

Understanding the basics

How do I practice coding challenges?

To become an expert at solving coding challenges, you should constantly improve by selecting increasingly harder problems. Toptal's JavaScript challenge contains a wide array of task difficulty levels. Don't skip problems that look tedious: The only way to improve is to focus on solving these too.

What is a JavaScript challenge?

A JavaScript challenge is designed to test your programming knowledge in a timed environment. To complete one successfully, you should be able to intelligently select data structures and design algorithms quickly.

Eva is a skilled back-end developer and machine learning engineer with experience in scalability issues, system administration, and more. She has a flair for well-structured, readable, and maintainable applications and excellent knowledge of Python, Ruby, and Go. She is a quick learner and has worked in teams of all sizes.

Comments

Michael Hays

You're taking this silly competition a bit too seriously. Relax, it's all in good fun -- the top prize was a pair of AirPods! Besides, other top scorers were using scripts to submit answers; I'm sure they'd have exploited the bug if they'd known how.

Andrey

The data visualization graph above shows that only a couple of guys shared that belief that "anything goes". I would have said nothing if everyone was playing by the same rules but clearly other participants considered exploiting bugs, DDOSing and botnets to be dubious practices and not the "creative approach".

Shogo

How did you fix the loophole?

Michael Hays

Personally, I think it's great to say that anything goes. It gives us the opportunity to be truly creative, and beating the system is fun :)
I do think it would be fun to have "true" speed-programming challenges too, though. Seems like the only way to prevent people from knowing the questions ahead of time is to have a small window (~1 minute) where everyone has to begin their single attempt. Otherwise, people will always be able to collaborate.

Eva Bojorges

I wouldn't call this cheating. We announced from the beginning that we allowed for creativity in the solutions, so any way of solving it was allowed.

Andrey

Let me get it straight. Some guys cheated and hacked your app but you are saying it's ok and they got the top places. This kind of creativity when interacting with some real world apps can get you into legal trouble.

Tiago Chilanti

It is so amazing how people get creative when you challenge them. I´m really proud to have taken part on the development of this challenge.