Bullet-proof facilities

While writing this article, Hurricane Ike is blowing into the Gulf of Mexico, just 400 miles south of me and making its way to the Texas coastline. While Ike had no impact on my location, it certainly underlines the importance of being prepared for these types of events.

As part of the group charged with maintaining a broadcast facility, engineers have the added responsibility to ensure the stations continue operating through all types of emergencies. Having the ability to keep a usable signal on the air to disseminate information to the public is the primary reason licenses are issued. Serving in the public interest should be enough of a reason to keep a facility operational during these events; but the station is also a business and it is in the best interest of the owners to ensure continuity in the entire business operation. With so much of a typical station's data residing on the hard drives of a server or PC, you can lose in an instant what may have taken years to create. The key is to have a plan, make sure every employee knows the plan and his role, and create an appropriate due-diligence checklist.

What is an emergency?

The term "emergency" doesn't necessarily need to be a catastrophic situation. The Federal Emergency Management Agency (FEMA) defines an emergency as, "any unplanned event that can cause deaths or significant injuries to employees, customers or the public; or that can shut down your business, disrupt operations, cause physical or environmental damage, or threaten the facility's financial standing or public image."

The due-diligence checklist

If your station doesn't already have corporate-wide plans in place, there are several approaches to creating a comprehensive plan.

Your checklist should take into account two general situations: 1) Planning for the emergency — This will serve as the guideline and standard operating procedure for the company's plan to mitigate a potential situation. 2) Creating a disaster recovery plan — This will be the roadmap to getting the business returned to a state of normalcy.

Things to consider

As with most businesses, broadcast facilities are a collection of different systems and functions that work together to create the final end result. You will want to determine and list those functions and the proper personnel to address the issues.

First, creating a proper plan is an entire group effort; the planning should ultimately fall on each core function, i.e. engineering (technical facilities), IT (data), programming and operations (traffic, billing, sales).

Each group should document its respective functions, infrastructure requirements (i.e. networking, applications, data files) personnel and other needs, in as much detail as possible; this will form the basis for establishing a plan.

Lessons learned from past events, whether good or bad, can be extremely valuable. Many companies will routinely perform root cause analysis (RCA) when something goes wrong, which can be a powerful tool when it comes to predicting possible future problems.

Here are some items to consider:

Data storage — When/how often is data backed-up? Where do the back-ups reside? How is the back-up verified? Who are the personnel responsible for managing back-ups? Is data backed-up at sufficient intervals as to not disrupt the business operation should a failure occur? How will data get restored and who is responsible? Are network servers available offsite? How is the data accessed from the remote location? Since stored data plays such a large role in any business, you should always treat the location where the servers are located as a data center.

Back-up power systems — Have you determined the maximum run time that the studio(s), transmitter(s) will operate on back-up power? Are the back-up systems tested under load routinely? Are they under a current maintenance contract? Are the generator(s) sized adequately for the application? Is the fuel supply adequate and filled-up? If generators are diesel, has the fuel been tested and polished if needed? Are network servers and other critical equipment powered from back-up battery systems? Are the batteries routinely tested and/or changed? Is the battery back-up properly sized for the application? Have you run a scheduled disaster scenario to see how the system operates?

HVAC systems — Is there an alternate method of cooling/heating critical equipment? Is there a list of and a plan to shut down non-essential equipment?

Microwave radio systems — Is there a back-up method to transport data? If the station operates multiple hop sites, have you evaluated the impact of losing a donor site? Are these systems tied to back-up power? In the case of multiple T1 systems, what other services are affected by a loss of equipment? How do you control equipment at the remote site(s)?

Transmitter sites — Is there an alternate method to deliver programming to the site? Is there an alternate method to control the site? Is there a sufficient level of monitoring to detect smoke, fire, water, etc.? Are back-up transmitter systems in place and operational? Are all antenna switching systems routinely tested? Is there a sufficient level of monitoring of transmission line (VSWR) and other antenna parameters? Are proper fail safes in place and tested?

Studios — Is there a reciprocal plan in place with other stations to support back-up operations in the event of a total studio loss? Can the back-up studio be connected to the transmitter site remotely? If audio servers are out of commission, how will programming occur?

Personnel — Have you identified essential personnel? Does each employee understand his role and responsibility in the event of an emergency? Does the company have a plan to house personnel if travel is restricted?

Other considerations — Be aware of potential hazards such as flood zones, earthquake areas, proximity to chemical/nuclear plants, high lightning areas, fire prone zones, high wind areas, etc. Much of the information you will need to identify hazards specific to a particular location can be found at www.fema.org and other websites. While it is nearly impossible to predict those unforeseen situations with 100 percent accuracy, the implementation and follow-through of a written plan before a disaster strikes is essential.