Server Error in '/' Application.
Runtime Error
Description: An exception occurred while processing your request. Additionally,
another exception occurred while executing the custom error page for the first
exception. The request has been terminated.

I'm curious what happened... I couldn't even post a question to meta during the error condition...

1 Answer
1

We're looking into why you saw an error, but the event was caused by SQL coming online in New York.

We're using SQL 2012 Always On Availability groups as part of our high availability architecture. How this works is the primary database server for that group (these are currently moved to Oregon due to Sandy) streams changes to the replicas.

Due to a lack of power in New York (status updates on our NY datacenter can be found here, here and here), we are keeping our equipment there off as much as we can. However, the replicas for SQL being down mean the backlog of changes needed to sync to them builds up in Oregon, at the rate of about 40GB that needs sending every 6 hours.

To relieve these logs (which we could tolerate for up to ~4 days) we spin up the New York SQL boxes to receive changes and let the logs clear out on the primary. On this latest spin up, the combination of no domain controller in New York online and the SQL boxes coming up simultaneously caused the windows clustering service to fail on NY-SQL01, and while the SQL cluster on top of this windows cluster assessed what was wrong, the databases were unavailable.

The app should have thrown a nice error page during this at least, we'll look into that.

We're by no means sitting around idle after our failover to Oregon. There are many, many things still happening, but we'll try and keep the interruptions to a minimum.

@NullUserExceptionอ_อ the unicorns that power the system insist on somewhere near the sea - they like boats.
–
Marc Gravell♦Oct 31 '12 at 19:46

1

Lack of transparency typically starts with abundance of unwarranted criticism, or an abundance of technical buffoonery.
–
Mike PenningtonOct 31 '12 at 19:47

How far down the “5 Whys” do you get before global warming and the size of cars in the US?
–
Ian RingroseOct 31 '12 at 22:51

1

Keep Oregon as your primary - it's closer to where I live so the bits get to me faster.
–
slugsterOct 31 '12 at 22:57

@NullUserExceptionอ_อ: Corvallis is well protected by the Oregon Coast Mountain Range from the Pacific Ocean which is 40 miles away—unlike NYC which is basically at sea level with no protection. There is a theoretical possibility of Corvallis flooding from the Willamette River, but dozens of large flood control dams built in the Cascades have, so far, adequately controlled that peril. The only other dangers are earthquakes, which we are pretty well prepared for, and volcanoes, which are of more danger to Portland 70 miles to the north.
–
wallykNov 1 '12 at 18:24

I am getting this error since a month. Not been able to resolve. I tried multiple browsers thinking could be a problem in browser but all fail with same error. Interestingly this happens only on my system. No such error on other systems
–
ShashiFeb 26 '13 at 11:39