Scribd is a
top 100 site on the web and one of the largest
sites built using Ruby on Rails. As one of the first rails sites
to reach scale, we’ve built a lot of infrastructure and solved a
lot of challenges to get Scribd to where it is today. We actively
try to push the envelope and have contributed substantial work
back to the open source community.

Scribd has an agile, startup culture and an unusually close
working relationship between engineering and ops. You’ll
regularly find cross-over work at Scribd, with ops people writing
application-layer code and engineers figuring out
operations-level problems. We think we’re able to make that work
because of the uniquely talented people we have on the team.

To allow us to keep scaling, we’re now looking to add a strong,
experienced operations guru to the …

Since in our setup Squid server uses one-process model (with an
asynchronous requests processing) there was no point in ordering
multi-core CPUs for our boxes and since we have a lots of pages
on the site and the cache is pretty huge all the servers ended up
being highly I/O bound. Considering these facts we’ve decided to
use the following hardware specs for the servers:

As we can see, it consists of a document-specific part
(/doc/1) and a non-unique human-readable slug
part (/Improved-Statistical-Test). When a user
comes to the site with a wrong slug in the document URL, we need
to make sure we send the user to the correct …

loops is a small and lightweight framework for
Ruby on Rails and Merb created to support simple background loops
in your application which are usually used to do some background
data processing on your servers (queue workers, batch tasks
processors, etc).

Originally loops plugin was created to make our (Scribd.com) own loops
code more organized. We used to have tens of different modules
with methods that were called with script/runner and then used
with nohup and other not so convenient backgrounding techniques.
When you have such a number of loops/workers to run in background
it becomes a nightmare to manage them on a regular basis
(restarts, code upgrades, status/health checking, etc).

Few days ago we were chatting in our corporate Campfire room and one of the
guys asked me what do I think about Rails developers hiring
process, what questions I’d ask a candidate, etc… This question
started really long and interesting discussion and I’d like to
share my thoughts on this question in this post.

So, first of all I would like to explain what kind of interviews
I really hate Ever since I was thinking of myself as of a
developer (many years ago) and was going to “software developer
position” interviews I really hated questions like “What is the
name and possible values of the third parameter of the function
some_freakin_weird_func() from some_weird.h” or “How to declare a
virtual destructor and when it could be useful?”… All my life I
had pretty practical thinking and never bothered to learn APIs or
some really deep language concepts that are useful in 1% of …

It’s been a few months since we‘ve started actively using ActiveMQ queue
server in our project. For some time we had pretty weird problems
with it and even started thinking about switching to something
else or even writing our own queue server which would comply with
our requirements. The most annoying problem was the following:
some time after activemq restart everything worked really well
and then activemq started lagging, queue started growing and all
producer processes were stalling on push() operations. We rewrote
our producers from Ruby to JRuby, then to Java and still – after
some time everything was in a bad shape until we restarted the
queue server.

So, long story short, after a lots of docs and source code
reading we’ve found really interesting thing. There is a
“feature” added in the recent ActiveMQ release …

Few months ago I’ve switched one of our internal projects
from doing synchronous database saves of analytics data to an
asynchronous processing using starling + a pool of workers. This was the day
when I really understood the power of specialized queue servers.
I was using database (mostly, MySQL) for this kind of tasks for
years and sometimes (especially under a highly concurrent load)
it worked not so fast… Few times I worked with some queue
servers, but those were either some small tasks or I didn’t have
a time to really get the idea, that specialized queue servers
were created just to do these tasks quickly and efficiently.

All this time (few months now) I was using starling noticed
really bad thing in how it works: if workers die (really die, or
lock on something for a long time, or just start lagging) …

Since the day one when I joined Scribd, I was thinking about the fact
that 90+% of our traffic is going to the document view pages,
which is a single action in our documents controller. I was
wondering how could we improve this action responsiveness and
make our users happier.

Few times I was creating a git branches and hacking this action
trying to implement some sort of page-level caching to make
things faster. But all the time results weren’t as good as I’d
like them to be. So, branches were sitting there and waiting for
a better idea.

Few months ago my
good friend has joined Scribd and we’ve started thinking on
this problem together. As the result of our brainstorming we’ve
managed to figure out what were the problems preventing us from
doing efficient caching: …

Bounces-handler package is a simple set of scripts to
automatically process email bounces and ISP‘s feedback loops
emails, maintain your mailing blacklists and a Rails plugin to
use those blacklists in your RoR applications.

This piece of software has been developed as a part of more
global work on mailing quality improvement in Scribd.com, but it
was one of the most critical steps after setting up reverse DNS
records, DKIM and SPF.

Content reproduced on this site is the property of the respective copyright holders.
It is not reviewed in advance by Oracle and does not necessarily represent the opinion
of Oracle or any other party.