19 April 2006

Today I'll stop my 3rd personperspective. I'm going to write a little bit more about what I do to help MacBU ship great software and provide some color around what's it's like to work on Mac software at Microsoft. Often when we have press events or special visits from our MVPs I'll give them a tour of the Mac Lab and explain what we do. They've always found it very interesting and so I thought I'd share a virtual tour of our Mac Lab. Let's get started:

Lab Door

We start with the door. The Mac Lab is about 2000 square feet of solid computers. The calendars you see down the side of the wall we use to mark team birthdays as well as special events. The CD on the door is an old Apple Software Restore CD from the last beige Mac Apple produced, the Power Macintosh G3.

The "Sandbox" and Plasma display

The first area in the Mac Lab is what we call the Sandbox. This is where we keep all significant hardware configurations Apple has released that run our products. We'll use the Plasma display to, watch DVDs and play games, uh er, I mean, do important training presentations. ;-) It's actually very useful because everyone can be in front of a computer and still see the main screen and follow along. Often other groups at Microsoft (the games group, hardware drivers group and even the Windows media group) will come and schedule time in the Mac Lab to test their software on the different hardware configurations.

Old iMacs and iBooks in the distance

More Macs

We have lots of Apple hardware. You can see here the old colorful iMacs along with some of the old iBooks. You can also see two of our Lab Technicians working on the backup systems, but more about that later. Up until a few months ago we had every significant hardware revision Apple has ever released since the dawn of time. We even had a section of the Lab we affectionately called the "Mac Nursery" where we kept all the older Macs going. We even had an old SE/30 and IIci and super expensive Mac II all connected via PhoneNet running Spectre, just for fun. It's always super fun to boot Word 1.0 or Excel 1.0 on these old machines and see how much things have changed. Due to lack of space in the Lab we had to put all of these older machines into storage and recycled the following:

Macintosh (original)

Macintosh SE

Macintosh SE/30

Macintosh Classic

Macintosh Centris 610

Macintosh IIci

Macintosh IIsi

Power Macintosh 7100/66

Power Macintosh 7100/80

Power Macintosh 7500/100

Quadra 650

Power Macintosh G3

Duo Dock with Powerbook Duo 2300c

Power Computing PowerCenter Pro 210

May they rest in peace.

ADIC Backup Robots

One of the realities of working with computers is that things fail. More often then you might think. We've used different backup robots, but ever since we moved to ADIC we've never had a robot failure. They just make great stuff. We have 3 robots and use Veritas Backup Exec. It works pretty well, as you can see:

Lots of backup tapes

We also do offsite backups just incase "The Big One" hits, but for regular use these tapes work just great, except when they don't. Recently we had a failure and lost 400 GB of data! We restored it from the tapes and then discovered that the Mac version of the Backup Exec agent had a corruption bug causing the restores to be compromised! Veritas folk were super responsive and they should have a new Mac agent out soon. Backup software and file systems are in the class of software that simply must work, all the time. Alas, this is not always the case.

I'm going to skip the "Build Lab" section of our Lab since it's very much in transition. Maybe I'll post about that later. For now, on to our automation system!

Mac Office is one of those "software in the large" projects. There's really no way a team of our size would be able to adequately test all of Office without the use of automated testing. Every day we get a new build of Office from the build machines, we copy it to our Xserve RAID connected to our dual G5 Xserve for access by our 249 automation machines. We then run thousands and thousands of tests on the new build. Typically we get 4 builds of Office each day: English Ship, English Debug, Japanese Ship and Japanese Debug. We run our entire battery of tests against all the builds and then report any failures to testers via email. The testers investigate the failures, log any bugs and then move on to their other duties as testers. This turns out to be very effective, if used properly, and over time it allows testers to focus on things humans do best, while letting computers verify the repetitious and mundane, but necessary, testing. It all started with our Blue and White G3s years ago. At first when testers would upgrade their test machines, instead of recycling the machines, "The Lab" would get them to add them to our automation machine pool. I think we had about 20 machines to begin with.

Venerable Old Bluies

After some time we started getting Gray G4s...

Speedy G4s

Then we upgraded to some dual proc machines...

Dual G5s

Then Apple give us a special gift. :-) You'd be probably be very surprised at the cost of running all these machines. There's the obvious electricity costs, but also cooling costs and even the physical space costs. Additionally, our system scales, not with CPU horsepower, but with quantity of machines. Most of the tests we run don't run significantly faster on a dual G5 vs. a single G4. So when Apple announced the Mac mini it wasn't minutes before we were considering how to use it for our automation system. The Mac mini has all the perfect qualifications:

Low power

Low heat

Small

Easy to pack together

Inexpensive

So we got a few to test things out...

Mixed Automation Rack

And then we made the big purchase:

Rack A, B and C = 150 Mac minis!

These work extraordinarily well. You might wonder how we control all these Macs. We use two methods: KVM switch box and Apple Remote Desktop. Thanks to our Lab Manager's great relationship with the IOGear folk we have a very reliable solution these days. It seemed like it took for ever to find a USB KVM switch box that didn't leave the machines "headless" after random reboots. The 8 port USB KVM from IOGear has been rock solid. So what does it look like to sit in front of 64 Mac minis? Like this:

64 Mac minis: 1 keyboard, monitor and mouse

This works very well when you must access the machines physically. Even so, just scanning each Mac for 1 second gets very old, very fast and Apple Remote Desktop comes to the rescue! When we need to see all the machines at once we just select them and BOOM! they're there. It also gives us what I believe is the one true reason Apple invented the 30 inch Display. ;-)

The 30 Inch Display: Fulfilling the measure of its creation.

ARD displays 50 machines at a time and when you have a capable machine, it uses the "cube rotation" effect to move from one group of 50 to the next. I got a picture of the effect mid rotation below:

Rotates like butter!

So how does it all work? Like this: On each machine we have two volumes: ChangeOS and Mac OS X. The Mac OS X volume is where we install the different versions of the OS. We boot to the ChangeOS volume to free up the Mac OS X volume for modification. When we trigger an automation run we specify the OS version and language. Each machine then reboots to the ChangeOS partition, caches the OS .dmg locally and uses the asr command line tool to restore the image. The tool that does this work is one I wrote (in AppleScript Studio no less!) called Lab Assistant. We have images of the Mac OS from 8.1 all the way up to 10.4.6 in all the languages our products support. It's a lot of data which brings me to the backbone of our automation system:

Our Xserve RAID and XSAN

1 TB

Right now we've just been testing out the XSAN stuff to see how we want to use it. That's why you see all the Xserves. Just one side of the top Xserve RAID is 1 TB of data. For a fun comparison this whole rack which is about 7 ft tall is full of old RAID arrays is also 1 TB of data storage. We call it the Big Mac Daddy.

Big Mac Daddy

Other groups at Microsoft have hardware retention policies that force hardware upgrades every so often, but instead of just "recycling" these server machines, our Lab Manager intercepts them on the way out, and we use them for various things, storage, SQL server etc. We actually have some of the old MSN servers in our Mac Lab!

Hallways

When you have so many machines to maintain, being able to get behind the machines is very important.

Lots o' cables!

Top View

We like to pack in those Mac minis and the cords get pretty dense when we do. The hanging Mac mini box moves if the HVAC is working. If it's not working, we've got to turn off the machines until it's fixed.

Our main automation Xserve has a habit of failing in some serious way once a year, always around Christmas time. :/ For the last 2 years I've been in charge of fixing it and getting it back to operational. Most of our server racks are generic white enclosures, but we do have 1 black Dell rack. As punishment for bad behavior, we put the Xserve in the Dell rack. That'll teach it. ;-) This is what it looks like from the inside of the rack looking out on the world. Poor caged servers...

From the server's perspective

Of course our iWork/iLife balance is just fine as you can see by the following:

Tornado Foosball Table

One of our team members bought this awesome Tornado foosball table which we use along with and XBox and XBox 360 to relax after a hard days work.

Published!

A while back the Seattle PI actually did a front page story on the Mac Business Unit and you can see from the picture in the Lab it was when we had only the G3s.

Gotta have the Mission Statement

As you enter the kitchen we have our MacBU mission statement to remind us what it's all about. :)

Free drinks of course!

Just like everywhere at Microsoft, we get all-you-can-drink beverages.

Fun Times!

Part of our team mantra is that we work hard, and play hard. So we do lots of fun morale events. We just take time off work and do stuff. We're good friends and enjoy "just hanging out" together.

The Mac Library

This is just a pretty picture to represent what is really a much bigger collection of 3rd Party software we use to test with Office. Most if it is stored on file servers, but this gives you an idea. (There's some old WWDC DVDs if you can find them!)

Printer Lab

More Printers

All connected for printer testing

They're beautiful

A big part of Office functionality is printing, and we do loads of print testing. We work really closely with the printer vendors and make sure the printed page looks great. WYSIWYG is fundamental to Mac ethos. All these printers are connected via USB hubs and Ethernet to a Mac OS X Server 10.4 which is the printer server.

I hope that gives you a better idea about what the Mac Lab is like and what it's like to work in the Macintosh Business Unit at Microsoft.

That seems like a productive lab also sounds like a fun company to work for. I'm a PC freak myself last time I used a MAC was in grade 4. I'd like to learn more about the systems to add to my PC certifications. Can you recommend a good training facility in Ontario Canada?

Awesome post! So I have three questions...1) what is your electricity bill for all those Macs ;) 2) do you have any insider info on when MS going to release a universal binary for Office and 3) do you do have any Powerbooks, iBooks or Macbooks in the lab?

That place is heaven! A MacMini cluster is just about the coolest thing ever :P

Thanks so much for this great tour. I hope Microsoft support MacIntels officially for Vista as I'm currently running XP on my Macbook and it's very good. Sadly, it's a great shame that Windows Media Player was dropped. This is causing issues for intel mac owners. Could you not release just the codec on its own so that Mac users could continue to view WMV's via Quicktime/VLC.

I notice you don't have an Apple ][ integer machine like the one I have. For a year and a half, from August '77 til January '79 the only way to load or store data was an audio cassette tape. If you wanted to do decimal numbers you had to buy a floating point card.

Will we ever see Microsoft Money for OS X? Please make it so! There is no worthy Personal Finance Manager (yet) for OS X and we really need one - one with a great UI and feature parity with the Windows version.

to Kroc Camen: a plug-in to play WMV in QuickTime already exists, and it's made in close cooperation with Microsoft, which is why they dropped the WM Player. Take a look at http://www.flip4mac.com/ It's free!

awsum read, its like being in the room, because you give a lot of pics from all perspectives, and your comments are not n00by and tell what people wanna read.thanks a lot dude, you saved me going there LOL joke, hope ill manage to see it in real some day :Dseeing mac-geeks working for microsoft (not that i ever doubted that they exist) gives ms a sympathic image ^^

David, What does the Mac BU think about the ability to now run Windows on Intel Macs with Boot Camp or Parallels - in regards to the argument that says this will allow developers to abondon native OS X development and just run a Windows app on OS X through virtualization, etc. I hope you don't see this as a valid speculation.. as I would like to see more and more OS X-native apps rather than virtualized apps. Any thoughts to share on that?

I am also quite interested in the general results of the testing. For example, do any/most issues you find just effect a certain version/language of OS X, or the issues are common across the lots of versions etc? (ie is the "test under every version of OS X" strategy uncover many issues, or just a couple here and there?)

Have you considered using NetBoot instead of bringing the OS down onto the computer?

Does Apple give MS access to OS X releases any earlier than other ADC members (I would guess not?).

Have you had any cases where you have changed your testing methodology due to issues that _didn't_ get picked up?

PS: Since this has been shown on slashdot I am surprised (and glad) that the usual MS hating crowd hasn't left a few comments

Man, if you guys are every hiring let me know. I wish I could hangup my job as a system engineer doing mostly all pc networks. I am mac addict to the core. And I have to say, you have an awesome job. I am jealous.

I have wanted a mac for some time now and I love bugging the tech at school for information and he always pleeds ignorance. I am only 13 and I have grown up with microsoft. I got my first desktop at age 2. Right now I have a dell running xp pro and it just barly meats the memory requierments. If you are looking for a kid to do internship here's my e-mail. nicholas.tehrani@gmail.com. If anneybodey reading this wants a gmail address just ask I am giving away invites.

I gotta say this is a very impressive setup of pictures. Thanks for the information and pictures about your lab. The one area of MS I respect really the most is the mac section of it - out of all the software released by MS, the mac software is by far the best in my opinion. Nice job testing, and please keep up the good work.

Wow, great post! I'm really impressed with the MacBU, you guys make some pretty great software up there! Say, if some of you guys are up for a roadtrip/flight/whatever, it would be absolutely awesome to have a few of you guys come down to the Elk Grove Apple Campus (near Sacramento). I'm sure that we'd all love some insight as to what Microsoft is doing for it's Mac users. If you guys are interested, email me at tf5_bassist@yahoo.com (i'd put my Apple address, but I'd rather not get spam there haha). Thanks!

Have you guys ever tried Xgrid on all those systems? Specifically all those minis, wow, I bet you could do some monster number crunching with all that. Major thanks to you for this post!Does Bill ever go down there just for fun?

I interned for MacBU in Summer 2001 and we were convinced that Apple had us doing just that. I remember submitting tickets for about as many OS X problems as I did for Word, XL, PPT and Erage (Entourage).

But I found an issue in Word X for Mac:A dokument with a picture, created on a PC with office 2k, replaced this picture on a Mac, will be print sharp and clear on the PC, but unsharp and poor on the Mac.

It's the same dokument, from the same network drive printed on the same network-printer with the same ink and paper (photo-settings)!

In a short sentence:Create with PCChange with MacPrint with PC: sharpPrint with Mac: unsharp (low Resolution - even in preview)

Nice setup. Though I'm having trouble finding the iBooks in the pictures. Did you perhaps mean iMac? Because those are right in the front. Then again, I didn't see the two techs you mention working in the background, so maybe it's my eyes.

Well, the only time I was ever at Microsoft (for a Distance Learning standards group meeting), I was expecting a tour of the Halls of Darkness. Instead it was full of various flavors and colors of Apples, with lots of "think different" posters. Turns out our meeting was in a MacBU building (is there more than one?)

You people at microsoft think you're so cool with your fee soda's. I work at an un-named tech company. We may not get free soda's, but we get them for a quarter, but the trade off is free beer each and every friday.

That's a pretty sweet lab! Thanks for the pics. We use the same philosphy at the company I work for, intercept the older Macs for large scale automation testing to burn in new apps, network gear and servers.

Now if you guys could release a true MS Outlook client for OS X that uses MAPI like the old OS 9 client, then I will be truely impressed.

That's a pretty sweet lab! Thanks for the pics. We use the same philosphy at the company I work for, intercept the older Macs for large scale automation testing to burn in new apps, network gear and servers.

Now if you guys could release a true MS Outlook client for OS X that uses MAPI like the old OS 9 client, then I will be truely impressed.

What is the average life-span of that hardware and what happens to it when you are done with it? Does it get refurbished and resold? Does it get donated to low-income families or schools? Do you ever donate it to some unfortunate soul who is still running a 500Mhz G3 iBook with 256MB of RAM? Or is it so burnt-out by the time y'all are done with it that you just have to discard it?

So, I'm guessing you didn't keep the boxes? Seriously though, thanks for peek behind closed doors. I do have two questions. How large is the Mac Business Unit in terms of floor space and how many are employed by the unit? I'm trying to get an idea of scale relative to Microsoft as a whole.

I have always had the image in my mind of Bill Gates having a Mac on his desk, because he appreciates the elegance of the Mac experience.

What I'm interested in is the progress of Virtual PC for MacIntel. I don't want to reboot my Mac and have it taken over with XP. I want my Windows-on-MacIntel experience in virtualization, with Windows securely sandboxed away from any possible corruption of Mac OS X. Parallels for X is almost ready, and I wouldn't be surprised if the Win 4 Lin folks are looking at doing a similar product for Mac OS X. And there's always the 100% F/OSS QEMU project.

MacBU had better get cracking on Virtual PC or their lunch will be eaten by a small competitor.

This is the best PR that MS has ever done! Many thanks. It's good to see that MS *is* really serious about Macs. Now one wonders, if they do this for Macs - what on earth do they do for PCs!!! Great tour.

For those of you who have been asking about open positions, we do have several open positions right now. Go to Microsoft Careers (http://www.microsoft.com/careers/), search for a job, and select 'Mac Office' from the products list.

For those of you who have been suggesting features for our existing products or ports of other Microsoft applications, you can send us feedback through Mactopia (http://www.microsoft.com/mac/default.aspx?pid=feedback).

For the guy who said that MacBU isn't in Redmond, that's not true. We're split between Redmond and Mountain View, California. The rest of us are enjoying the sun down in the Bay Area. :)

I am in love with the Microsoft Mac BU. I want a job there now for the first time EVER. I thought Microsoft was mostly windows, and the MBU was just a small part on the side. Do you have any internships for high school students? I would love the experience.

i hate MAcs but it was an awesome tour. Thanks for showing me how the evil enemy snuck inside Microsoft. But I suppose it is useful to see what the enemy is up to. At least u r making some money off of them. At least you get free drinks, awesome!

Thank you for writing this great article. I have a whole new respect for your team and what you do. There is so much testing to do just for the Mac. Does Microsoft have a stadium where they keep the PC's they use to test the Windows OS?

Wow!Does Bill Gates come and play late at night over there? [Maybe THAT's why Vista looks soooooo much like OSX....]

Ahhhhhh, but I digress. Thanks so very much for sharing all of this. VERY inspiring, lots of fun to contemplate (and be jealous of), and oddly CUTE. It sure took me back, seeing the lists of "retired Macs" (made me miss my original dual floppy Mac SE with no internal HD). And it has me appreciate all the Mac Gurus (some visible, some invisible) who make things possible for users like me!

Wow, seems like a cool place to work.I'm an Apple freak since 1980 (started with an Apple ][+) and switched to Mac in 1990.At the moment I have an iBook G4, mac mini (G4) and a old dual G4 (AGP).At the moment I don't use any MS software (don't have the money to pay for Office).For most things Appleworks 6.0 is enough.I used to work for PcM.That's a large Dutch newspaperpublisher (Volkskrant, Trouw, NRC, AD, etc.).They have around 335 mac's which they use for layout.At the moment they are upgrading from G3's and G4's to G5's.All with MacOS X.Sadly, my contract expired and I had to leave for another job.

Your recount is a reminder to all that backup tapes must be read back on a regular basis and verified before a disaster occurs. And once your original is gone your backup becomes your new original. Back it up! Another great format is VXA, btw.

Please, please, tell us that when you write "recycled" you really mean someone took the antiques home or gave them to a collector or a museum. Even to those who are not the biggest fans of Microsoft this stuff is true history. Sadly, those close to it don't always understand it as such. An "original" Mac (whether it be a 128K Mac or a Mac XL) once owned by Microsoft would be very valuable indeed. The birth of Excel 1.0, not to mention its predecessor, Microsoft's first GUI spreadsheet known as Multiplan, quite a significant milestone in human history if only because of the number of lives eventually touched. Oh, please, please say those antiques didn't actually get recycled... That would be really bad Karma. Could have even auctioned them off for the Bill and Melinda Gates foundation. There must be a lot of readers out there just shuddering to think of the tragic waste.