Log in

Dave Andersen's Blog

babble, tech, politics

Aug. 3rd, 2012

Hi, all - I've decided to move my future blog posts over toDave's Data on Blogger. LiveJournal has been great, but I'm craving something with a bit more 2012 and a bit less 1999. :) Please check there for updates, etc. And, of course, I post more frequent updates on my Google+ page.

Feb. 13th, 2011

This week started a bit off with a sore throat (my two highest-ever mileage weeks combined with the SIGCOMM deadline, or coincidence? You decide), but I took it easy for a few days and then flew to the bay area for an ISAT meeting. The meeting was interesting from my side of things (I learned things), but arguably the best treat was getting in 13 mile runs to lovely, hilly trails in the mornings. Run details from Garmin Connect.

Oct. 21st, 2010

This is really a preface to a longer post involving mildly surprising disclosures like me deciding to eat mostly vegan for the last 4 months (and counting...), but for now, I'll skip to some of the fun. Below is the "workout" part (not counting another 4 miles of warmup/cooldown) from a lactate threshold workout (eg, "comfortably hard", but decently harder than just normal daily runs -- I usually need a rest day after these) done today and a month ago:

In other words, a month ago I ran 4 miles at a 6:56 average pace with an average heart rate of 153 beats per minute; today, I ran 4 miles at a 6:43 average pace with an average heart rate of 151.5. In fairness, it was cold out today, but that's still a lovely improvement for only a month. The relation to the eating habits is that I've never before been able to sustain the mileage I'm running now. Wish I'd figured out earlier that eating better is good for running. :)

May. 4th, 2010

I was at the UCSD workshop on Non-Volatile Memories about three weeks ago, and had a surprisingly great time. I say "surprisingly" because I showed up at the reception the first night, realized I didn't know a single person there, and thought "uh-oh." That "uh-oh" turned into "ooh!" the next day -- I learned a surprising amount about the lower levels of contemporary nonvolatile memory technology and met some very cool folks.

Many of the slides from the talks are online (though, as in all things, the hallway conversations were both unrecorded and perhaps as or more useful). But one of the stand-out talks isn't -- Al Borchers talk about Google's experiences with Flash memory. I've jotted down some highlights from the talk that jumped out at me. Caveat: These are filtered through my own interest, and a lot of what really jabbed in my head echos our own experiences with FAWN, and several reinforced things I said in my talk the day before so if something seems odd, it's probably my fault, and not Al's.

Al Borchers is in the platforms group developing system software for Ggogle's server platforms, and has been working on high performance storage devices. Ph.D. in theoretical CS from Minnesota (1996), has been hacking unix and linux device drivers and systems software in industry. Much of the talk he gave involved work
with Naveen Upta, Tom Keane, Kyle Nesbit.

Looking at HW devices and how SW could be modified to take advantage of Flash, if necessary:
"It has been a rocky experience with flash. We've had difficulties with performance and reliability of devices, and figuring out where we can apply flash in a cost-effective way. Many applications... some obvious some not. Without forcing apps to change too radically.

As CPU overhead, with highest perf devices, we use one core to do I/O, and then we get NUMA effects as things communicate over the system bus.

Al then gave some numbers on the relative performance of, e.g., PCI vs SAS vs SATA drives they'd measured. I didn't write all of these down well enough to be confident reposting them. The gist was that access over PCI incurred less CPU overhead than SAS and SATA. NUMA access - when you had to go through a different core - hurt just as much.

CPU use for async/multithread: At high BW, sync multithreaded model uses 2-3x the CPU. They didn't really see that in SATA because it was limited to 31 outstanding requests by NCQ.

if you just have a few files, you can do a very simple userspace FS...

NUMA 20% to 40% more overhead

Interface:

IO size constrained by flash page/block sizes

Doesn't mean flash has to be a block device in the storage stack

New level in storage hierarchy needs a new interface

Next gen high speed NAND devices will have higher IOPS

Research opportunities:

Optimize storage stack for flash performance characteristics

Other interfaces for flash -- for example, user space IO

Problem 3: Error rates

Read error rate higher than predicted by bit error rate

Block, plane, and die failures seem to dominate

Errors seem concentrated in a few bad chips

(haven't watched the devices to see errors through wearout and aging)

Small impact to caching apps -- cache miss

Large impact to DB apps -- data loss

Looking at RAID, but w/traditional RAID drivers, perf is terrible. Adds another layer of CPU overhead. Looking at optimized RAIDs for flash, would like to see more...

research op: fault tolerance in flash storage devices, long term, large scale failure rates and mechanisms.

Q: Which drives did you use?
A: Doesn't matter. Can't say. But all of the devices suffer perf overhead.

Q: Can you comment SLC vs MLC on reliability?
A: Our initial reliability of SLC seemed a little bit better, but we haven't taken them to life and worn them out, but for both we saw a lot of early-life failures.

We feel like we're forced to MLC...

encouraged that mfgs are talking about enterprise MLC... discouraging looking at commodity flash chips that have shorter and shorter lifetimes

Q: Comment on pci-express as interface?
A: We like it better, it seems to perform better, lower overhead, ...

.. more about high overhead of going through block layer to get to SSDs at high IOPS.

All in all, it was an excellent talk, and shows that Google has been taking a very serious look at Flash in their datacenters. We're seeing a lot of indicators that Flash is poised -- but not completely ready yet -- to start making huge inroads into the DC.

Apr. 26th, 2010

The result of playing "what the heck do we have around the house to eat with this lettuce?" turned out surprisingly tasty last night. So I'm inflicting it on my blog as a way to write it down. Ingredients are a rough estimate - this was a purely seat-of-the-pants recipe:

Onion: Saute onion in a splash (I'm guessing under 1T) of olive oil. Past soft, but not quite to fully caramelized - maybe 15 minutes. Remove from heat and set aside.

Add a few T of water to small nonstick pan, warm, add sugars, bring to gentle simmer. Keep careful eye on and stir frequently. Midway to caramelization, add cashews. Continue to cook until you get a light brown caramel coating the cashews. Be careful: The syrup will caramelize fast, and you want to take it off the heat immediately afterwords.

Remove cashews from heat. Mix with onions. Break the cashews into smaller chunks -- they'll be glued together with the caramel -- but don't burn yourself. The caramel is hot and sticky.

Dec. 15th, 2009

I just had to, with some embarrassment, leave the following explanation in closing out my service ticket with my (patient and wonderful) ISP, Speakeasy.

The context: My internet service has been bouncing up and down every hour and a half for the last day or so. Very frustrating.

I believe I've diagnosed it, and owe you a thanks for putting up with a customer-caused problem. When on the most recent phone call, I accidentally bumped into the power cable for my NAT box, rebooting the machine, which solved the mystery:

A curtain was brushing against the power cord. Over the last few months, the constant bumping had worked the plug loose to the point where it was just barely in. When the furnace would turn on, about every hour and a half, the forced air would cause the curtain to move -- which would glitch the power to the NAT box. After about 20 minutes, the house would be warm again, and the furnace would shut off... only to repeat an hour and a half later.

Thanks for your patience and help with this one. I'm going to go bonk the owner of said NAT box on the head for not looking at the uptime on his machine before crying wolf. I believe you can close the ticket."