And thanks to your logfiles not being able to be viewed in real time (as they are owned by root), this leaves web developers that actually have a clue very few options for forensically backtracking the vector.

I would like to know what Rackspace is doing to help developers isolate these issues? Are logfiles being programmatically reviewed for malicious traffic? Without SSH access and the ability to tail apache logs, we cannot do this ourselves within any kind of timeframe that will be useful in preventing or mitigating an attack. If I am going to continue hosting with Rackspace, I want to be assured that Rackspace is actually doing something to help us protect ourselves other than send emails that overstate the obvious.

Your support staff, at least most of the level 1 techs, are completely and utterly incapable of handling anything relating to hacks. They are slow and under-educated, regardless of how well meaning they might be.

Lack of transparency and lack of talent. Harsh words but it comes straight to the point of trust in a provider will only get you so far before you need to step in and verify that they have the security capabilities you need.

I bring this up as FreshBooks recently spammed me with a “we’re secure” message, which created the following thread with a comical ending. First, here’s the excerpt from their message that caught my attention.

We want you to save time every month by using FreshBooks so you can focus on what you love to do. […] If you…need a nudge, here are some nuggets:

If youâ€™re thinking: â€œI donâ€™t know if my data is safe on the cloudâ€

Ok, well done. I’m paying attention to a message I would have otherwise tossed into the spam bucket. I wrote a quick reply.

My concern is with security/compliance. What are the extra steps?

I received a response from someone with this signature

xxxxxx from FreshBooks
(very) Small Business Consultant

I suspect the “(very)” is supposed to be humorous. It would be much more humorous if they put “non-VIP”, “n00b” or perhaps even “peasant” in their sig to reinforce a lack of support I should expect. Howdy, I have been assigned to your really tiny and unimportant issues. Now, how may I be of (very little) help? Hilarious.

Here is the actual response they sent me:

I’m not sure I understand. Extra steps to what, exactly? Are you talking about PCI compliance, or the security we have on our servers, or?

Yes, I actually was talking about or. What are the extra steps to or? But that is not what I responded. Instead I simply wrote the following reply to try to get back to their original statement in the email they sent me:

Hi, I was just quoting your email message. I don’t know what steps you meant.

That seemed to help as they then sent back the following response with URLs

Ah, I understand. You can see our security measures here: http://www.freshbooks.com/security-safeguards.php

We also use RackSpace for our server hosting, and you can see their info here: http://www.rackspace.com/

I hope this helps! Let me know if there is anything else I can help with :)

The rackspace URL is the generic front-page. Not a good sign, per the start of this post. I asked about extra steps. So I dig into the Freshbooks security page and it raises far more questions than answers. Here are some examples:

Any unusual behaviour is analyzed by AlertLogic’s CISSP-certified security experts, and responses are coordinated between them, Rackspace, and our system administration team.

Odd. They hold up the CISSP as a qualification for monitoring network traffic? I find that discouraging — indicates a lack of understanding about both the CISSP certification and network monitoring. Responses are coordinated by their system administration team, which suggests no security team. That would explain why they have to delegate. Still looking for the extra steps.

Who gets the keys? How are keys setup and managed? Nothing extra here either. So little information on such a critical issue reads like a Drobbox catastrophe just waiting to happen. Speaking of lessons learned, I then read this section:

FreshBooks has chosen Rackspace for our hosting needs. With clients like General Electric, Hershey, Cisco, Pfizer, EMI Music, Scott’s, Hilton, Sony Music, Columbia House and the US Marines, we know Rackspace provides the hardware, service and expertise you expect.

What are the chances that FreshBooks is going to be able to get good customer support/service while stuck behind a list of giants like Sony who are probably taking up every minute of Rackspace support time during their breaches?

And what are the chances that FreshBooks will be adequately protected from a mess like Sony? Have they verified segmentation? Transparency comes directly to mind. So, of course, I had to ask for clarification again but by this point I confess I was losing patience in finding any extra steps, which their original spam promised me.

your page does not mention compliance standards or third party assessments. are there any? CISSP-certification does not mean anything for analysis of vulnerabilities or threats. it is a general knowledge test, like a bachelor degree does not mean you are qualified to be a doctor.

rackspace disallows physical audits of their datacenter. how do you verify their security? the list of their clients only means you are all going to be competing for lifeboats when that ship sinks, not that it is well run. have you had any audits of your equipment there?

Then came the reply, short and to the point, which confirmed to me that there are no extra steps. I could even make the case that their security page is lacking important details and so they are in fact missing steps. They delegate their security and they simply hope that you will too. Here is their reply:

I spoke to my IT team about your questions, and I’ll quote a response: “If they don’t trust RackSpace, then they probably don’t want to use us”.

Doesn’t look like we’ll be the right fit for you. Better to find out earlier than later :)

Good thing I asked. Thought others might want to know. And with a nod to Alison Gianotto, here is my cranky haiku:

Freshbooks to Davi;
Security extra steps
can’t be verified

Update: An old video has surfaced that shows a trivial exploit of FreshBooks. The attacker logs in as a client who received an invoice and then deletes the invoice simply by changing the SetAction “print” command to “delete” in their browser.

6 Responses

You may be surprised to read this but as the Head of Magic of FreshBooks I am actually pleased that you wrote this blog post. Sure I wish it took more of a tone of praise than negativity but the post as is exposes something that is very valuable to me and that is that we need to be a little more clear in the language we choose to use in our emails.

You commented in your post that we made you feel unappreciated in the way we addressed you and for that I am sorry. We at FreshBooks try to have a little fun in everything we do and while we may not take ourselves too seriously we do take our business and our security very seriously.

I appreciate the analysis of our security you submitted and have to say you really know your stuff. We reviewed all your points and felt I should correct you on a few but realizing that correct is perhaps too strong a word I will just bring to light a few things worth noting on our end and please realize we usually donâ€™t ever go this in depth on the inner workings of our company so if I am vague in some spots please understand.

We go on record all the time that we are proud of our Rackspace partnership and have been happy with them for many years. We lease hardware, bandwidth and rack space from them, and do all of the system administration ourselves. That’s very different from their Managed Hosting product which essentially involves outsourcing system administration and it’s true that we don’t have a team dedicated to security, but we do have a crack sysadmin team that is trained in and looks after out security issues.
We’re still a small company with a little more than 80 staff. Most people here, if not everyone wears multiple hats and that is not that different from most companies our size.

We only use Rackspace’s system administrators when we need remote hands because our IT team are the administrators, and while Rackspace has sudo access on our servers, they can only use it when we enable their accounts, which we only do when we expect them to be doing maintenance. That means we view our logs (and they don’t), for instance — and we ship them off via syslog to AlertLogic for threat analysis and tamper-resistant archiving as well. Their analysis is automated, of course, but one of the value-added services we take advantage of there is to have their (human) analysts investigate alerts prior to passing them along, which for us means fewer false positives.

But at the same time, we’ve got our own copy, which get reviewed in-house regularly too.
I could go on and on (and will if you wish) but I think the core of what I am trying to say is (and this is not an excuse) our staff are some of the smartest people around and if pressed on a very technical question there may be a few people who may not give the best answer imaginable since tech may not be their strong suit, honestly I had to get help with understanding your analysis because some of it was over my head but our tech guys knew their stuff and the same applies to all our departments.

Oh and since you brought up that video I should tell you an interesting story about it. It is at least two years old and when it was originally posted (on a Friday evening) it was accurate. We did have that issue. But that issue only existed for 12 hours because one of our developers came in on the weekend and plugged that hole as soon as it was discovered. I only bring this up to highlight again that we take our platform and security seriously.

All this doesn’t change the fact that I am sorry and will apologize again for anything we said that seemed condescending towards you and if you wish to discuss this further I would be happy to pick up this discussion with you one on one at any time.

I am curious, after having a direct conversation with Saul – or looking back on this two years later, what now is your position on Freshbooks’ security. I have been trying out their demo, and thinking about connecting my bank account when I saw the above post.

Continuing the Discussion

[…] as they had claimed. Anyway if interested you can read the post by Davi Ottenheimer here: http://www.flyingpenguin.com/?p=13739 This entry was posted on September 27th, 2011 at 3:36 pm and is filed under security, […]

Subscribe

About flyingpenguin

flyingpenguin, a security consultancy, designs and assesses risk mitigation, compliance and response solutions, as well as delivers strategic and competitive knowledge to security software and hardware vendors. Innovation, integrity and transparency are hallmarks of our services. Davi Ottenheimer, President of flyingpenguin, has more than twenty years’ experience managing global security operations and assessments, including a […]more →