I’ve got some news to announce.
Hollee and I are going traveling for a while … until July 2012!

Let me explain.

Hollee is currently applying for Law School – and probably will start in August 2012.
I’m in the fortunate position of being able to work wherever I have my laptop, illness a quiet room, and a stable internet connection.
I love Los Angeles; but it’s definitely time to do some exploring and try living elsewhere for a while; new cultures, activities, and all that jazz.

So, we’ve decided to take advantage of Hollee’s ‘year off’ and do just that!

First, we’re off to New Orleans early November. NOLA’s Hollee’s favorite city; we’ve got some good friends there, and we thought it’d be a good test of this whole ‘sabbatical’ thing. There’s a magic about the city that we both love; and we’re very excited about this change. We’re looking for apartments there right now and Hollee’s looking for a job out there.

After three months in NOLA, we’re off to our ultimate destination. We haven’t decided for sure — but we’re thinking Hong Kong — for six months.

Why Hong Kong? It’s the center of Asia, and means we can take lots of short trips across Asia; it’s not cold in the Winter (a definite requirement for Hollee), and it’s one of the major telecom business hubs in the world (definitely a bonus for my work.) HK isn’t set in stone; we’re mulling over our options – we figure we’ll be able to think about it more clearly once we’ve arrived in New Orleans… one thing at a time!

We’re going to miss all of you – and hope to keep in touch the ways we do in this modern age — Skype Video, email, IM. Leaving all of you – my family and friends in LA, is definitely the most difficult part of this trip, but I find consolation in the knowledge that we’ll be back.

“It’s not enough to be busy, so are the ants. The question is, what are we busy about?” — Thoreau

“Let yourself be silently drawn by the strange pull of what you really love” — Rumi

Note : I’m a preparedness guy. I’d rather be over-provisioned than under-provisioned, cheap because asking things of others because I didn’t prepare makes me feel like a mooch. This is both very in theme for the event – Radical Self-Reliance – but being emotionally open to relying upon others is also important. Find your middle ground; but items with a (#) are simply mandatory for anyone.

This may sound daunting — but this is an amazing experience if you allow it to be. Prepare as best as you need – trust the universe and your community to take care of you when you missed something; or in your ability to persevere without it. We are amazingly hardy and impressive creations – a week in the dust and heat is nothing; and our ancestors called that everyday. Remember that.

Attitude :

(#) Be who you aren’t usually; be your inner nature; explore something new – just don’t be what you are at home.

I’m a planner – learning to allow myself to accept and rely upon the gifts and kindness of others is a real challenge- and this is a great place for it. I’d suggest this attitude for almost everyone. Center Camp is the best place to hang out if you’re feeling lost/lonely/overwhelmed. You can refund your center there.

When they talk about compression and decompression – they mean it. It takes a while to adjust to being there [try to get in the mindset on the drive up by reading materials that speak to you] — and it definitely takes effort to adapt back to the busy, clinic planned, monetary, expectation-oriented world of home. It takes a few days – and this is how it should be. Make space for it.

You won’t see everything. A Friend of a Friend, Kathi Fuentevilla, posted this on FB – wisest thing I’ve seen to a friend who was planning intently.
“we all did it. We all marked the things we’d like to see and the things we absolutely had to see in the guide – I’ve done it for 6 years. I’ve never made it to a single one, but that doesn’t stop me from researching it & marking it all down.”
Time doesn’t operate usually out there. Most people don’t carry watches – and people leap into experiences when they’re exposed to them. Everyone is flaky about meeting up on time, events just don’t happen; and that’s just the way things are. Accept it and love it.

Burning Man is an excellent introduction into living in the Now. The environment is harsh enough that you can’t ignore it – so you’re always aware of it. The things happening around you are interesting enough that you can’t ignore it – so you’re always aware of it. This is beautiful. Take some time and just watch – and do nothing. You aren’t checking off a todo list, you’re living at it’s purest.

Leave your rational mind and judgement at home. There is a lot to be said about being a rational impartial observer – but do it after you’ve left. Empty your Cup. http://www.c2.com/cgi/wiki?EmptyYourCup

But into the insanity, and reflect non-judgmentally on the drive home. You’ll never been the same.

Culturally:

This is unlike anything you’ve experienced, not just because of the culture but the environment. The environment is punishing for those that live in the first-world; large temperature fluctuations; dust storms of corrosive sand; and very little ‘indoors’ space.

The city changes from day to day – early in the week its far more spiritual, naked, and empty; later in the week it’s far more of a spectacle. Both are wonderful, but the longer you can stay and watch the city grow around you; the better. I’m told by those that go in pre-event that the week before the event starts in magical. Some year after your first, pick a camp; get involved; and try going early.

There are spectators and there are participants. One of the goals of the city is to be totally-participatory, but dressing in the clothes out of you closet, showering daily, watching events and photographing them rather than sharing in them; these are a spectator mindset. Become a participant as often as you can; dress like the locals, live like the locals. You’re not an anthropologist, and you’re not writing a academic paper when you get home.

When wandering the Playa – you don’t need a wallet, ID [unless you're likely to be carded at the bars], etc. Keep those things locked in your car. (#) Bring a Bike Lock. There is theft – it’s tragic, but it happens, and often. Keep anything expensive locked in your car.

Environment :

(#) Read the Survival Guide. Reading and know it. Grab the PDF of this years, and at least skim the Recommended Reading section. http://www.burningman.com/preparation/event_survival/recommended_reading.html
Know the ten principals – they say a lot about our temporary city and it’s ideals.

The weather is flickle. We’ve had snow just before the event started; we’ve had rain on multiple occasions, and sudden bouts of wind that last anywhere from two minutes to two days are not uncommon. You don’t have to plan much for snow or rain, don’t worry, but keep in mind that weather is dynamic.

If you aren’t in a climate-controlled place [or an advanced shade structure/yurt] – don’t expect to sleep past 10am, regardless of what time you went to bed. It gets really hot; and tents get humid from water lost through the night. A shade structure over your tent will give you a few more hours.

For clothing in the daytime, bring light, flowing clothing (or nothing at all.) Lots of sunscreen, no matter how dark you are. You can wear street clothes, safari clothes; but I personally think that dressing normally takes away from the experience.

For clothing at night, layers. The weather can dip into the 40s, and if there’s a solid wind, it gets even colder. If you need to wear street clothes here; this is the time; but preferably, have a few costumes. Scan the internet, or ask me for ideas — there’s a million creative ideas out there; and Goodwill + Halloween store is your friend.

The stories you’ve heard about the dust is for real. Think finer than talcum powder. It’s very hard to remove permanently from fabrics, particularly mattresses; so keep that in mind. The dust is alkali and somewhat corrosive; treat your hands and feet with a wash (1 vinegar:4 water) once a day to prevent ‘Playa Foot.’ Keep your feet out of playa as much as possible – socks are your friend. When the wind starts, the Playa hits the wind, and visibility and breathability of the air goes down dramatically. If the wind picks up, you end up with a white-out – visibility as little as 6 feet. For these reasons, (#) ALWAYS carry Eye Protection, A Breathing Mask, and Water. Eye Protection can range from fancy goggles to cheap googles; but have some. Breathing Masks are painters or surgeons masks; although a bandana covering your mouth works in a pinch.

(#) Water is it’s own special thing. Carry some with you ALWAYS – Camelbaks are exceedingly popular – it’s very easy to get dehydrated; and very easy not to notice it when you’re sweating constantly. You might also get trapped somewhere during a whiteout – having water is a life-saver.

Bring a GOOD LED Headlamp and Flashlight, and spare batteries. Keep them with you at night at all times. They are required at night – if you’re walking in remote areas, bicyclists are libel to hit you if they can’t see you — we carry some extra glow sticks and zip-tie them to my backpacks.

Emergencies :

Communication off-playa, while improving these past years, is still sketchy. The best shot your family has for getting a hold of you in a catastrophe is to email 911@burningman.com with your exact camp location, and a Ranger will try to find you – but this is sketchy at best. Emergency Services also has a similar service – http://www.brcesd.com/?pg=contact.

There is wifi at Center Camp – it’s optimized for Skype traffic; but as the event progresses it gets sketchier and sketchier. It is not a reliable method; so don’t plan on it, and you’ll be pleasantly surprised when it does work.

The Black Rock Rangers are your friends. They come off a bit gruff– but the responsibility for keeping everyone alive falls on them; and they take it seriously. If you’re simply overwhelmed – you’re feeling majorly ill, having a bad drug experience, having heat issues – find a ranger; and they’ll get you to the hospital, or to special places for helping people down from bad trips.

Supplies :

Once you leave Reno, you’re on your own. There’s a tiny store in Empire that’s usually overwhelmed; and there is no commerce on Playa. Get your supplies, substances, waters, everything beforehand. Carry a tiny bit of cash – Camp Arctica sells Ice every day, and Center Camp has Lemonade, Coffee, Hot Chocolate, etc. [No water.] Spending days without spending anything is a novel experience, and definitely one that changes your perspective when you get home.

Bring all your own water. You will eat less than you think; but bring your food. Involved cooking that requires many steps or lots of cleaning is far less likely to get done. It’s hard to cook when you can’t just discard of waste water easily — you can’t pour it on the Playa.

If you’re taking substances – hide them well on the drive. The police know what’s going on. If you’re trying them on Playa for the first time; have a trusted sitter.

I bring gifts. It’s a huge part of what I love about the event. Give wholeheartedly and whimsically – it’s good for the soul. If you bring gifts; avoid brining stuff that is MOOPy (read the Survival Guide), or easily discarded – we find either mementos or functional gifts are well received.

Pack street clothes for your drive home. Keep them sealed in a Ziploc bag and do not touch them until you’re WAY off Playa and have found a place to shower – there’s a truck stop in Sparks, NV that’s pretty popular for that.

From Richard :

Richard Becker, a friend of mine from Enigma, gave me this sage advice on 6/29/05. I agree with almost all of it, and have included it in it’s entirety.
(He’s also a far better writer than I.)

1. Bring plenty of water.
2. Drink plenty of water, especially when you’re not thirsty.
3. You’ll never see everything, so read the maps and lists of things and pick what you really want to see.
4. If you see something cool going on while you’re on your way to your planned site/event at BM, GO TO THE THING YOU SEE RIGHT NOW. It may never happen again. It may be better than where you were going. Don’t miss the good thing you can see right now in favor of the thing you read about!
5. Wear sunscreen.
6. Bring canned, dried, pickled foods, and eat them straight out of the container with the one or two utensils you bring with you and wash after each meal.
7. Bring goggles and a cloth to wrap around your face. There are dustclouds and duststorms every year, without fail. You will face them.
8. Bring warm clothing for the nighttime. It can get quite cold. The weather can also turn.
9. Watch the skies. I’ve seen some very weird things. Things that don’t require any drugs or alcohol, because they’re real. Things your friends will doubt. You might see them too.
10. The Black Rock Desert is not made of sand. It is made of dust — talcum powder, to be specific. It will be in your food, drink, washing water, clothing, everything. Get used to it.
11. The gender ratio is 2/3 male to 1/3 female. This will affect your experience somewhat, particularly depending on your gender orientation. If you prefer females, you should anticipate that you will not meet one on the Playa. If you want a woman to be there for you at BM, plan ahead and bring one.
12. Bring colorful clothing and wild costuming. That, or skin, is the normal garb for BM.
13. Wear comfortable boots with everything. If you have costume shoes that you’ve *got* to wear, fine. Otherwise, don’t bother with any other footgear.
14. Bring a bicycle that you don’t mind messing up. It will get you around everywhere. But — BUT — make SURE you have a VERY comfortable seat, such as a gel seat, for it. And good shocks, if it’s built with shocks. And excellent tires. And make sure it’s visible at night.
15. Do not partake of any illegal substance in public, ever. What you do in privacy is your business. If anyone did take anything illegal at the Burn, I’d advise them to make sure they trusted the person giving it to them just as they should do so anywhere else.
16. Art cars are there to be seen and ridden on. Do this. But beware of them at night.
17. Your ticket to Burning Man says words to the effect of, “You could DIE if you go to BM.” Guess what: It’s true. Several people die every year at the Burn. *Take danger seriously.* You are at risk. But on the other hand, don’t be a worrier. Live smart and have fun at the Burn.
18. Don’t miss the burning of the Man on Saturday night.
19. Don’t miss the burning of the Temple on Sunday night.
20. Bring things to give as gifts to strangers — trinkets, doo-dads, cool little things. Only trade with people who insist on it. BM isn’t supposed to be a barter economy, it’s supposed to be a gift economy. Don’t encourage the trend to barter.
21. To barter, though, make sure you bring cigarettes, liquor, and drink mixers.
22. There will be a lot of nudity. It’s normal at the Burn, and it’s not intended sexually — it’s freedom. Just get used to it; before you know it, you’ll not even notice it.
23. Be kind to others.
24. Be generous to others.
25. Clean up your campsite and truck your own garbage out of the Playa — and take as much of other people’s trash with you as you can.
25. To coin the cliche: “What happens at Burning Man, stays at Burning Man.” Wherever you think your friends and companions do or act out something that is a different aspect of themselves, and shows a (possibly startling) new facet of their personality, if it does no harm — let it be. And let them be. Don’t publicize what is meant to be their free expression. And rely on others not to publicize yours.

There’s certainly much more to say: Take siestas by day, don’t overdo it, bring a foam pad to sleep on if you camp in a tent, bring a shade structure to keep cool, etc.

But ultimately, enjoy the Burn.

From Hollee :

My friend put it beautifully this evening for a virgin we were talking to.

You’re brain fills in new experiences with old information, every time you eat a sandwich, every time you drive to work, you brain is constantly filling in all the familiar things from memory so it does less work. The first time you experience this its going to be it’s most vivid and interesting and intense, bask in that. This is going to be the newest thing you’ve done in years, take in every sight and smell, and taste and voice because it’ll mean more this year than every other time.

Burning man is an Arts festival, pack what you need to survive, but make sure you enjoy what everyone has come there to contribute.
Things might be silly, you should be silly, this is a place for self expression in all it’s outlets, pick your vices and respect others’.

Some things that might be in the guides but are worth mentioning twice:
Baby Wipes are a god send, you can even pack a separate set and pour vinegar into them as an appendage remedy,
Make spare keys, not being able to drive home and waiting for an $8 an mile tow truck from reno is a crappy way to end your trip

As an alternative for a sleeping foam pad, the 99 cent store pool mattresses are great too.

I recieved an email from someone on the asterisk-biz list who reacted to my current quotes in my sig.

“I think you learn more from disappointment. But I think that success loses its sheen after a while and you begin to see that there’s no such thing as absolute success, generic that there’s always failure and there’s always disappointment and there’s always loss. But the secret is learning from the loss and realizing that none of those holes are vacuums. They’ll be filled up. And if you try to fill them up with ego or a kind of immediate gratification, you’re going to short-shrift yourself and you’re going to lose the opportunity to find out what happens when you give the loss space to fill itself and let life kind of come in and fill the cracks. ” — Michael J. Fox

“If we did all the things we are capable of, we would literally astound ourselves.” — Thomas Edison

“Let yourself be silently drawn by the strange pull of what you really love” — Rumi

He checked out this site and asked if I had made any progress on the workflow posts I had made.

My reply to him was lengthy; and ended up becoming as much of a state of the Rizwan as I’ve written recently. It’s incomplete, missing important details, entire parts of philosophy, citations, and editing. Certain people aren’t credited nearly as much as they should be.

It would take me days to properly write this — and it’s a worthy task, but not today. I’d spend too much time over the specific word choice, and how it would be interpreted, and how it might affect other’s thoughts of me. Today’s writing was flow, and heartful — and is included below.

I’m working on more of a focus, spiritual workflow right now than a strict task workflow; if only that until I establish a better workflow, no combination of GTD and Omnifocus will help. There’s a little bit more written at my Livejournal blog; but most of it is similar to below; such as http://rizwank.livejournal.com/2010/08/27/

I’ll take an opportunity to riff for a moment as I wait for this DNS server to reboot. (Going to probably repurpose this for the blog anyhow – thanks for prompting it. =) )

Reading GTD introduced me to the concept of ‘mind like water.’ Responding with the appropriate amount of energy and focus.

Inquiries into Inbox Zero led me to reading Merlin Mann. At one point, he notes that if you’re struggles to organize tons of X, the problem may not be your organizational system but that fact that you have tons of X. (And that you have to keep expending energy trying to find new organizational systems.)

The idea of “doing less” is patently fearsome to me. I’m 29; and I can still quote Breakfast Club – “When you grow up, your heart dies.” The idea of not accepting every invite, being everyone’s friend, and taking every buisness opportunity caused chills.

And then I read “Better”. http://www.merlinmann.com/better – my take away : Do less, better. I’m an engineer, photographer, gamer, entrepreneur, costumer, mild non-conformist, and a born philomath – but the thing that enables all the other parts of me was my engineering; and I could do better for myself by being better at a few core things, (and a jack of some trades) rather than being a jack of all trades. This is also the change I feared most about getting older; assuming it was made out of fear, inadequacy, or apathy – rather than a considered decision of a human being realizing that days are finite.

Add in Seth Godin’s ‘Dip’, (Or Merlin Mann’s ‘Creative Suck’, or the ‘Resistance’ in the Artist’s Way, perspectives from ‘The War of Art.’) – and the idea that there are spectacular returns for actually being focused; diminishing returns happen at the middle of the time series; it takes off again on the far end.

Add in a practice of meditation, particularly Mindfulness Meditation – integration of mind and body; an acknowledgement of their innate interconnection; and the fabulous benefits of forcing the mind to not think… even for just a moment. From that comes more Eastern writings – Rumi in particular. (Muhammed and the Huge Eater, as an example); and realizing the beauty in metaphor and the wisdom from past centuries that I’ve simple ignored — and the value of feeling alongside of thinking. My psychiatrist introduced me to the ideas, and it’s been hugely influential.

The Michael J Fox interview on NPR that led to that quote came at just the right moment, when I’d been spending lots of time thinking about expectations and it reinforced my attitude that adversity is best treated as a source of learning – not even just to avoid it in the future, but to know myself better.

Watching ‘Dharma and Greg’ with my girlfriend led me to observe Chuck Lorre’s Buddhist transformation (http://www.chucklorre.com/index.php?p=2), and led to a lot of reflection on my behavior; and an appreciation of Zen. New thoughts on awareness, suffering, judgement.

Scads of articles on focus and scientific studies on multi-tasking remind me that we’re just bad at it – even myself; who once prided myself as being a very efficient multi tasker. The best resource (beyond all the above mentioned) is http://zenhabits.net/archives/ – particularly focus.

So where have I ended up – not having executed on these growing ideals as well as I would have liked; but compassionate with myself for the attempt. I struggle to learn to learn to say ‘No,’ to stop seeking the approval of others, to stop letting a trained rational and logical sense from keeping me from listening to what I actually feel rather than what I think, to take time for myself; not for entertainment, but for quiet and peace, and allow the intuitive mind space to breathe. My RSS Reeder still is overwhelmed with flagged entries and unread entries; but I’m learning to slowly let go and just delete feeds. My email is still a nightmare. My task list is still a nightmare. My Instapaper account is filled with articles about focus that I simply have to make the time for; but I haven’t learned how to do that yet.

But I write occasionally, take notes after appointments, and skim my musings to myself to remind me of what’s truly important and who I *want* to be.

I’m building a list. The things that are important. The people that are important. And learning to say no to the rest. Until then – obligations and projects that make me feel alive will continue to get pushed aside by petty wastes of time; and the people that truly enrich me will continue to get crowded out by everyone else. I’m going to try to be stingy; not with my wealth, but with my focus and attention. And to take a Sabbath day.

So; I haven’t come to any conclusions as to a workflow; but have realized that the system has to change before a workflow can be erected.

It’s been a crazy time in my head, price reading tons of Merlin Mann, somnology and thinking bit thoughts as to who I want to be, dysentery what kind of work I want to do, and how I want to do it over this next five years.

It’s still all a jumble, but I’m slowly getting it down, written, for examination.

One workflow that has to be managed somehow : My links.

Each browser on my Macbook Pro (yippie! New MBP!) has at least thirty links saved. Firefox, with it’s impressive profile support, has seven profiles, most of which have a flood of links of some sort. My increasing desperation at opening a browser group should explain the names of my later profiles, including “Lean & Mean”, “Don’t keep any tabs open” and my favorite “Jesus!”. (This last profile is my current one, and is numbering about 60 tabs in 7 windows.)

Tab explosions come from the following :

Visiting Daring Fireball, Hacker News, Slashdot, 43 Folders. Gizmodo and Lifehacker add to the linkspam as well, but I do a decent job of ignoring those sites, especially after reading Mann’s essay Better. [This essay, I think, was the final push to actually focus my thinking on the big change.]

Any Google attempt of a problem I’m trying to solve. (Since I switch tasks so often, when a task is *somewhat done,* I’ll often move to something else, and stale tabs are left over)

Email processing.

Any Google attempt on anything interesting to me at the time.

One of the effects of the ‘Next Step,’ as I’m calling whatever set of discrete changes I’m contemplating now, is that I’m going to try to be a lot more disciplined about my curiousity, and stopping the constant indulging of “Oh, that looks neat!” and “Ooh, I want to know that!” (There are reasons, and this isn’t as simple of an idea as it sounds, perhaps more on that later.)

My tab workflow is a symptom of my overall workflow problems, and investing a lot of time fixing this symptom rather than the major problem (now that I’m focusing on that problem) is a distraction, not an advantage. So I’m not going to. But I do need a few more endpoints for the tabs that *do* get opened. Below, a chart of the types of data in my tabs, and what I’ve decided to do with them. (EDIT : I resisted the temptation to make a flowchart here. It’d be nice and all, I could put it up on my wall, but it’d be next to my GTD Workflow map, which is the domintant flowchart in my office, and I want to keep it this way. This data is fine in a table.)

(Hmm. I left this draft at 3:56pm. I return to it at 6:20pm. I’ve sent ten emails, upgraded the backup system for a client, done network debugging, uninstalled a few plugins (and done browser restarts), added plugins to WordPress to allow table editing, called family and purchased Choosy and iStat Menus. *headdesk*)

I used to use delicious. It’s gone stale, and I fear the data that’s in there now, I kind of feel like starting for scratch. The social functions of it worry me however – it’s a great way of having me be exposed to *more* new links – and I really just don’t need that.

I used to use browser bookmarks all synced by XMarks, but when I dumped in my backlogs of links to sort, and tried to start sorting them, the taxonomy was … massively draining, as well as annoyed at the hybrid tree/tag structure of most browsers bookmarks.

I’m going to start throwing away more links. While I think it’s *awesome* that there’s an HTML5 VNC client, and while I’m never going to remember it’s name, Googling those two words got me there pretty quickly. (I fear that I’ll forget that there’s an HTML5 VNC client in the long term – but I can’t imagine any workflow that has me going to my bookmarks *before* Google … I may have to get used to “if I forget it, I forget it” – a throughly disturbing idea personally, but one that might just be necessary.

Some of my Asterisk servers (against all reason, viagra it seems) are AsteriskNOW servers – which recently updated their asterisk16 branch to 1.6.2.6.

chan_gtalk, hygiene and res_jabber – modules for installing Jabber and Gtalk access are missing. As I’ve learned, Asterisk Modules have to be built on the exact same platform/compile time switches as the asterisk executable, so the only way I could get these modules was to recompile by source.

I had planned to ditch the AsteriskNOW packages, but I ended up modifying the packages to include these modules instead. These should install (somewhat seamlessly) over your current AsteriskNOW release. (The biggest issue is if you have other modules installed – they’ll have to be rebuilt. For Skype, install the -devel package, then rebuilt Skype from source direct from Digiums site.)

It was 2pm. Traffic in LA becomes a nightmare somewhere around 3:30pm. Jumped in the car to crash cart the server (because the tech on site didn’t know what a crash cart was and when I described it – said ‘that sounds complicated.’ Not happy with our colo right now.)

Arrived to find the server paused — not hung, just waiting, on a “Starting openwsmand…” message. Turns out, there’s a new service in the latest version of OpenManage that requires a new OpenSSL certificate for functioning. This, on it’s own, is okay – but two problems :

The cert is generated by the boottime init.d script if it’s missing.

The cert is generated using /dev/random as the entropy source

The latter is a security concern, but should never be paired with the top. I’ve had servers hang for hours waiting for /dev/random to generate enough entropy.

This was easily enough fixed, but the stress of driving like a madman, (and back) to make it before traffic shut down all routes to West LA was something I could do without. I’m going to patch against the project and see if they’ll fix this ridiculous behavior, and I’ve started looking into a remote KVM solution like kvm2ethernet – just call the colo and ask them to plug into a particular server. Thanks to this post for cutting the debugging time massively.

The other issue was that customers weren’t able to sign up for bits of today because there was a lock residing on one of our DB tables. We purge our Database (about 50G) monthly, but the cruft of leftover billing records take up huge amounts of space and deleting them can be a problem – long table scans, and Microsoft SQL Server does a table lock – and that’s the ball game. The credit card server can’t track that a valid charge was placed, so it terminates instead. We started with a query from our Marketer/Data Analysis guy looking like :

DELETE from billing where
and start_date_time >= '18-JUL-2009'
and start_date_time < '24-JUL-2009'
and node_type not in (3,4,5)

(He wanted to delete up until 30-AUG, but was slicing it up in the hopes of avoiding this problem.)

The above is about 300,000 rows.

A few issues, however :

billing has a clustered index around account_id. This makes perfect sense, the data is almost always referenced with respect to a particular customer, and ensuring that those records are all adjacent to each other on storage is common sense. However, the above query would be running all around the 9Gb table removing rows.

billing doesn't have an index on node_type, meaning each row has to be fetched before it can be selected for deletion.

The latter is less of a problem if the select and the delete were separated, but the table lock existed throughout the query.

In looking to solve it, one approach was to force MSSQL to use ROWLOCK (and disable escalation from ROWLOCK to TABLELOCK), but this was going to be a performance hit. I considered trying NOLOCK, but I wasn't sure of what the ramifications would be, and I really didn't want to spend hours fixing a crash database/corrupted data.

The final solution was to carve up the deletes into more manageable bits. SQL Server Interactive can be set to execute on a limited number of rows - so we could delete, say, 1000 rows, pause (let other things have access to the table), and then continue. And, now that we know about the clustering, why make the DELETE run across the entire table - why not allow it to trim one section at a time? And thusly we have...
(I'm not really familiar with SQL Server Syntax, so this is a first effort. It's lousy code, but a decent query)

It was 2pm. Traffic in LA becomes a nightmare somewhere around 3:30pm. Jumped in the car to crash cart the server (because the tech on site didn’t know what a crash cart was and when I described it – said ‘that sounds complicated.’ Not happy with our colo right now.)

Arrived to find the server paused — not hung, just waiting, on a “Starting openwsmand…” message. Turns out, there’s a new service in the latest version of OpenManage that requires a new OpenSSL certificate for functioning. This, on it’s own, is okay – but two problems :

The cert is generated by the boottime init.d script if it’s missing.

The cert is generated using /dev/random as the entropy source

The latter is a security concern, but should never be paired with the top. I’ve had servers hang for hours waiting for /dev/random to generate enough entropy.

This was easily enough fixed, but the stress of driving like a madman, (and back) to make it before traffic shut down all routes to West LA was something I could do without. I’m going to patch against the project and see if they’ll fix this ridiculous behavior, and I’ve started looking into a remote KVM solution like kvm2ethernet – just call the colo and ask them to plug into a particular server. Thanks to this post for cutting the debugging time massively.

The other issue was that customers weren’t able to sign up for bits of today because there was a lock residing on one of our DB tables. We purge our Database (about 50G) monthly, but the cruft of leftover billing records take up huge amounts of space and deleting them can be a problem – long table scans, and Microsoft SQL Server does a table lock – and that’s the ball game. The credit card server can’t track that a valid charge was placed, so it terminates instead. We started with a query from our Marketer/Data Analysis guy looking like :

DELETE from billing where
and start_date_time >= '18-JUL-2009'
and start_date_time < '24-JUL-2009'
and node_type not in (3,4,5)

(He wanted to delete up until 30-AUG, but was slicing it up in the hopes of avoiding this problem.)

The above is about 300,000 rows.

A few issues, however :

billing has a clustered index around account_id. This makes perfect sense, the data is almost always referenced with respect to a particular customer, and ensuring that those records are all adjacent to each other on storage is common sense. However, the above query would be running all around the 9Gb table removing rows.

billing doesn't have an index on node_type, meaning each row has to be fetched before it can be selected for deletion.

The latter is less of a problem if the select and the delete were separated, but the table lock existed throughout the query.

In looking to solve it, one approach was to force MSSQL to use ROWLOCK (and disable escalation from ROWLOCK to TABLELOCK), but this was going to be a performance hit. I considered trying NOLOCK, but I wasn't sure of what the ramifications would be, and I really didn't want to spend hours fixing a crash database/corrupted data.

The final solution was to carve up the deletes into more manageable bits. SQL Server Interactive can be set to execute on a limited number of rows - so we could delete, say, 1000 rows, pause (let other things have access to the table), and then continue. And, now that we know about the clustering, why make the DELETE run across the entire table - why not allow it to trim one section at a time? And thusly we have...
(I'm not really familiar with SQL Server Syntax, so this is a first effort. It's lousy code, but a decent query)

Usage

Just install the RPMs for the version that you want. You’ll need at least the base python version package and the libs package.
To start Python 2.6, type python26 at your commandline rather than python. (Your original Python 2.4 is still installed.)
If you’re installing packages with setuptools, make sure to use the correct python version. (i.e. python26 setup.py install)
If you’d like to build an x86_64, call rpmbuild –rebuild with the appropriate arch setting.

Just called sir at the airport parking lot – by the sort of person who calls any stranger sir (I'm one of them). Disturbing nonetheless. #

Usage

Just install the RPMs for the version that you want. You’ll need at least the base python version package and the libs package.
To start Python 2.6, type python26 at your commandline rather than python. (Your original Python 2.4 is still installed.)
If you’re installing packages with setuptools, make sure to use the correct python version. (i.e. python26 setup.py install)
If you’d like to build an x86_64, call rpmbuild –rebuild with the appropriate arch setting.

Just called sir at the airport parking lot – by the sort of person who calls any stranger sir (I'm one of them). Disturbing nonetheless. #

Usage

Just install the RPMs for the version that you want. You’ll need at least the base python version package and the libs package.
To start Python 2.6, type python26 at your commandline rather than python. (Your original Python 2.4 is still installed.)
If you’re installing packages with setuptools, make sure to use the correct python version. (i.e. python26 setup.py install)
If you’d like to build an x86_64, call rpmbuild –rebuild with the appropriate arch setting.

Just called sir at the airport parking lot – by the sort of person who calls any stranger sir (I'm one of them). Disturbing nonetheless. #