Menu

His idea to build a more abstract testing framework was not stupid. It also wasn’t unique.

My experience is the desire to build highly abstract systems is very popular. As a Software Engineer specializing in Test Automation over the last 15 years, I’ve found it common. Avoiding mundane work (like writing test cases) is a keen focus for devs, especially for the smartest software engineers — and those earliest in their careers.

The problem in this case, is that I’ve seen exactly what he was suggesting – and I’ve tried it myself – at least a dozen times. I’ve seen it absorb at least 3 years of software engineering salary in the last decade alone. In fact, avoiding activities like this is exactly why companies pay me – to avoid wasting their money.

I had 2 options in responding to him:

1) Tell him it won’t work, likely demotivating him and encouraging him to ignore me from now on — he was going to run with this idea anyway OR

2) Find a way to make sure he had my experience in mind while attempting his idea

“Actually, I think it is a very smart idea,” I said — and I meant it. What I didn’t say is that sometimes the smartest ideas make the least business sense.

There are many, many smart people out there focused on abstracting test automation frameworks. Some have spent their entire careers on it. So what sense would it make to use the time and salary of an application developer to focus on creating yet another piece of software for testing?

Sure, he may create something that succeeds in its purpose — although I find it more likely that he’ll end up using the time to learn all the ways not to do it. Further, someone is going to have to maintain that software over time.

And during this effort of a smart idea, who will be writing the application code the company uses to pay his salary?

My experience is the best test cases are those that are concrete. If I can look at the name of a failing test case and know what part of the system is causing trouble, it saves me the time of having to debug my system, learn the test framework, and verify the environment is setup correctly.

I told him as much – my experience – the commodity his superiors were purchasing from me. I encouraged him to try it as a quick proof of concept and hoped it would force him toward experiencing the warning I raised.

I felt our relationship as teammates and his motivation were worth a day or two of creating a proof of concept. My experience tells me it’s not worth pursuing after that, but that’s not my call.

Smart ideas don’t always make business sense. They can be headed off quickly – but sometimes there’s as much of a hidden cost to mishandling a smart idea as there is to pursuing it when it doesn’t make business sense.

Paul Merrill (@dpaulmerrill) is a Software Engineer who specializes in Test Automation. He likes writing code when it makes sense and then he uses Java, Javascript and Python – sometimes with a sprinkling of Ruby on top. He’s built the company he founded in 2009, Beaufort Fairmont, as a solution for companies that want help automating or learning to automate testing. He uses a variety of open source tools (like Cucumber & RobotFramework) to decrease the length of test cycles and provide quality feedback to devs and stakeholders. Learn more at http://beaufortfairmont.com or https://www.linkedin.com/in/dpaulmerrill

Another option (that required a little bit of work) is to install the rails/rails_xss plugin. The plugin will change the default behavior of strings:

Strings now have a notion of “html safe”, which is false by default. Whenever rails copies a string into the response body it checks whether or not the string is safe, safe strings are copied verbatim into the response body, but unsafe strings are escaped first.

“File Access” is another warning category generated by Brakeman. “File Access” refers to a security vulnerability where attackers may gain access to files otherwise inaccessible.

Imagine your Rails application does this:

file = File.open(@somemodel.someattribute)

or perhaps

file = File.open(params[:somepath])

In the both cases you are opening a file based on the completely arbitrary value supplied by a client. Depending on what you do with the file, such as displaying it in a view, you might expose data to an attacker.

The danger here is that your application is not checking the validity of the input. An attacker can easily probe your system using your application just by experimenting with different inputs.

The important thing to note: the application opens an HTTP request to whatever value the attacker supplies as a parameter for params[:file].

In this case the params[:file] value is assumed to be a valid URL. Maybe it’s one that is supplied another controller in the same Rails application. Or the client is supposed to behave reasonably, sending a URL to a resource that the Rails application expects to work with.

Unfortunately, that’s not always going to be the case.

Let’s pretend that someone wants to make mischief for your application. What if they supply a URL for a very, very large file (perhaps digital elevation model data sets in 80MB range)?

If you application naively accepts any input, you are asking for trouble. In this scenario, an attacker could easily drown your application with requests that tie up your application’s request threads. Soon your web application will no longer be able to service real requests – they’ll be be busy fetching bogus URLs.

A “Mass Assignment” warning refers to the assignment of values to an object from a “naked” input. In this case, it looks like the input is actually a params hash value. What wrong with that? Isn’t that darned convenient, slapping together a record without explicitly assigning values for each field? The problem is the model itself. Let’s take a closer look at our model!

You can see the “admin” field at the bottom. You probably have the admin field act as a flag to the system to indicate if a user can do, well, admin things, like create or delete users. Or look at orders. Or addresses and phone numbers. The sort of personally identifiable information that, if leaked, would be a problem. Use your imagination.

When you allow mass assignment of fields, an attacker may send you anything in the parameter hash.

In a previous article, you read how random strangers were invited to run whatever command they wanted on your production systems.

Today, you will learn how you invite them to administer your database.

Look again in your Brakeman output and you might see:

High OrdersController show SQL Injection Possible SQL injection near line 87 ...

First things first – what does “SQL Injection” mean?

When an attacker uses “SQL Injection” they attempt run SQL statements or alter SQL statements in your code. They do this by manipulating input parameters that your code reads and then uses in ActiveRecord methods.

If you allow unchecked parameters into your ActiveRecord queries, you also allow anyone to commit untold horrors to and with your data.

HOW DO YOU FIX THIS?

A simple remedy is to use a conditional array. The Ruby Guide for 2.3.11 says
“Putting the variable directly into the conditions string will pass the variable to the database as-is.”
In a conditional array the “naked” parameter input will properly handle the value of params[:user], for example:

User.find(:first, :conditions => ["id = ?", params[:user]])

generates

SELECT * FROM `users` WHERE (id = '23 OR 1=1') LIMIT 1

In this case, we’re using MySQL, so the condition

(id = '23 OR 1=1')

does not evaluate the “OR 1=1″ as a WHERE condition, but as extraneous string data.

What does “Command Injection” mean, anyways? Command injection means that an attacker takes advantage of your Rails code to run commands they choose on your system.

If that doesn’t cause a shiver to run up your spine, you aren’t sufficiently paranoid. Maybe you are running commands in controller code that looks like

system(cmd)

or maybe

logger.info %x[#{cmd}]

At first glance this doesn’t look dangerous. You probably wrote the logger.info code so you can log the output of the “%x[…]” command (which returns the output as a string), something that system doesn’t do.

Your Rails application will attempt to run the cmd with the same permissions as the owner of the Rails process. (For Rails 2.3.x this is the owner of the config/environment.rb file. This is one reason why running Rails as a super-user privileged account – like root – is discouraged.)

The problem is that by running an arbitrary command, you make it possible that the string you pass to system or the %x won’t be what you think it will be.

What happens if you are building a command string from parts of a parameters hash?

cmd = "ls -al #{params[:user_id]}"
logger.info %x[#{cmd}]

What happens if params[:user_id] is something really sneaky?

Let’s pretend that an attacker provides the value “|rm -rf /” for the user_id parameter? The command will be executed as

ls -al | rm -rf /

Deleting all the files on your production host would be inconvenient, at the least. It could potentially end your business if you are not prepared for a total data loss.

What can you do to protect yourself?

One way is to not allow untrusted inputs as parameters to Ruby system calls. Instead of using

system(cmd)

try

system(cmd, arg1, arg2, ...)

The arguments after the command are not interpreted as arbitrary commands. The |rm -rf / is not executed.

What if you are using logger.info %x[...] you need to do a little more work.

If you are looking to record the output of a command you can use the open3 library.

stdin, stdout, stderr = Open3.popen3(cmd, args)
logger.info stdout

The point is don’t assume anything about what other clients will supply to your controller. If you must use parameters to build command strings, don’t open yourself to unlimited abuse with open-ended system shell commands.

A while ago I looked around for software business opportunities. I considered what was “good” or “bad” differently than I do now. I knew that I did not know what to look for. I wanted to “figure out” that part of my life.

I reached out to someone I worked for several years ago. I’ll call this person “Steve”. When I worked for Steve I wrote software. I repaired and enhanced an existing software product. The company was bought and sold a couple of times. The final owner decided to shutter the business, but most of the orignal employees enjoyed buy-outs of their shares.

Since that time Steve resurrected the product under a new corporation he owned as a partner. Steve sold the company to a well-known, privately owned corporation. Steve’s track record participating in software startups is quite unusual – almost as many “hits” as attempts. I value Steve’s insight and advice.

Steve and I ate lunch, and I began to explain what I was looking for. I was not looking for a job. I was far more interested in building a real business. Steve gave me a great piece of advice.

“If I were to do this over again, I would look for the squares.”

Steve meant look for the businesses that no one else considers.

Many startups that the technology press glorifies focus on hacker culture or consumer vanity. Pictures of your dinner, social media tools, computer games – the things that intoxicate the public. These business have one goal – grow fast and reach as many people as possible. They rely on network effects and customer acquisition models. The business of making money comes next, after they gorge themselves on millions of users.

On the other hand, there is what DHH called the “Fortune 5 Million” – the businesses providing a living for their owners. These are the “squares” – the unhip, real, businesses relentlessly centered on making money. Businesses like dry cleaners, doctors, financial planners, and gutter installers.

There is a sea of people who are in real pain who no one pays attention to. I called a dozen of “square” businesses over 2 days. Three called me back and agreed to talk to me. One owner offered to buy me lunch just because he wanted to talk about things in his work that frustrated him.

Because the market opportunity is not enough to sustain the interest of the “hip” software scene, most startups ignore these people. They won’t stoop down to pick up the dollar they see floating by in the street. But every moment there’s another dollar floating by.

For the last 3 weeks I’ve sent out hundreds of emails to fee-only financial advisors.

A typical email looks like this:

I’m doing a research project on fee-only financial planners and advisors.

I’m interested in the struggles and pain points you have running your business.

I’m not selling anything and I’m not marketing anything. I really just want to talk to you about the most painful things that cost you the most time and money in your practice.

This particular iteration of my contact email is not very different from the first. The differences are minor. For example, I added information that answered frequently asked questions. I don’t want to eliminate every question. If someone is curious enough, they’ll write back.

The goal of the email contact is a scheduled phone call. First the recipient has to open the email. Only 40% to 50% of the recipients open the emails. 26% of the recipients click on links in the email to direct them to this blog and my personal LinkedIn account. The funnel narrows with each step.

5% of all emails I’ve sent result in a phone conversation. I’ve done almost 12 hours of interviews with advisors and planners from Florida to Wisconsin and Portland.

If it takes 10,000 hours of practice to truly master a skill, then I’m clearly just beginning to crawl down this path.

Most phone calls don’t go as well as I’d like. When I’m talking to someone about their business, I want to hear about the things that make them grind their teeth. I want to hear about frustration. I want to hear about rage or despair. I want to know the tedious, boring, and wasteful things they have to deal with every day.

I’ve noticed that more than once a business owner will proclaim that they have no pain. They will say that they love every part of their job. I don’t believe them.

I do believe that they enjoy the meaningful outcomes they give to their clients. It’s the reason I decided to start with fee-only financial advisors. They care about helping their clients safeguard and grow their wealth.

Even on days when software development sucked me into a hyper-productive mode I had pain. I had to deal with buggy tools. I had to deal with broken build machines. I had to configure security settings for source code control systems. I could rationalize the pain as a necessary hurdle to get to the real work of writing working software.

So, what do I do when someone tells me that they have no pain? I try to understand their world. I try to put myself in their shoes and understand why they feel that way.

Next, I ask them to get as detailed as possible about what they do every day, step by step. Sometimes people “discover” a pain they didn’t realize they had. One person realized they didn’t really enjoy preparing quarterly billing for their clients.

This tactic doesn’t always work. Sometimes the call feels like a dud.

I try to remind myself that I’m consciously unskilled. Developing a skill takes practice and reflection. Each stumble brings me further down the road.

It’s been a week of contacting hundreds of businesses from Wisconsin to Florida to Louisiana. It’s time to ask “What have I learned?”

Not everyone is happy to see you.

I’ve sent over 400 emails to fee-only financial planners. Most of the people I sent emails to did not respond. Only about a third of the people who saw the emails opened them. Less than 5% responded to the email.

I am not surprised I didn’t get more responses. Sales and marketing emails bombard these people every day. My message isn’t a sales or marketing email. Without anything more than a casual glance my email appears to be spam. Most of the emails were probably ignored or deleted.

Responses were not uniform. Here are some of the summarized responses:

“Are you a reporter?”

“If you looking for a job, send us your resume.”

“Are you selling something?”

“If you want my time, I charge $300/hour, with a minimum fee of $500.”

“Who are you working for? Who is this research for?”

“I just don’t have the time to talk. This is the busiest time of the year for me.”

“Please remove me from your list.”

Luckily, several people did agree to talk to me. Even after they agreed to talk, and agreed to spare time out of their day for a phone call, they were skeptical.

Don’t feed the trolls.

For every negative response I wanted to write back and explain myself. Suppressing the need to be right is a skill. I felt like I had to wipe away the misconceptions and make things right.

I don’t think it mattered what they thought. I was not going to convince someone who I really did want to help them with an email. I have a limited number of hours every day to talk to the people who did want to talk. I had to focus on those people and move on.

I did however attempt to apologize to any one who felt I was spamming them. I also told them I would never contact them again. If someone asks you to not email them be polite and move on.

It’s not about you.

The whole point of doing these interviews is to work out their pain. If you decide in the interview to talk about yourself, it’s very easy to lose focus. Let them talk. In fact, they should talk most of the time.

I will ask a few questions. My favorite is “What’s the most painful part of your day?” Occasionally I will ask “What else?” and sometimes I will prompt them with “Tell me more.”

When things are really on a roll, they will begin to dig into the pain themselves. It seems cathartic. I don’t need to “sell” them that they have frustrations. Only after they talk about the pain do I (sometimes) begin to ask explicit questions about the wasteful actions, frustrations, etc, of that pain.

Get used to rejection.

People ignore or reject 98% of my emails. If being ignored hurts, give up now. It will only get worse. The old fairy-tale aphorism “You have to kiss a lot of frogs to meet a prince” also fits here. People will reject you.

Don’t dwell on rejection. Keep looking for the people who want to talk to you. They exist. You must find them.

Be honest.

People want to know why you are trying to call them. If someone called me out of the blue, I’d be curious, too. And skeptical. And on my guard. Don’t complicate things by trying to “spin” your mission as something that it’s not.

I tell people I’m looking for pain. I say that because it’s true. I make no effort to hide my hope that I hope to make a product. Of course, I explain that without real pain there is no solution to worry about.

While I don’t people distracted by my goals, I feel that being open and honest is best. In some cases, people became more interested in what I was going. They seemed genuinely supportive of my efforts.

It is almost 2 years to the day that I was accepted, along with a partner, into a technology accelerator program. I’ll make a long story very short — we burned through $20,000 and had nothing to show for it.