Power Outages and Electrical Blackouts

August 1st, 2006

A quick guide provided by our disaster recovery and IT management team to cope with power outages and blackouts.

A word from our disaster recovery team
Summertime is here, and thanks to global warming, it might actually be here to stay longer and come around to visit us more often than in the past. As a result, electrical outages and blackouts are back again, thought this is not new to California, thanks to Enron's ever-cherished contributions, it has been awhile, Spring 2001 to be exact, since Businesses had to deal with this debilitating predicament. We wanted to seize the opportunity to transmit some simple guidelines to avoid the worst consequences possible for your business.

Overview
Outages are caused by failing equipment and cables, and as such are random, but rolling blackouts to alleviate the overall power grid might be around the corner. In this latter case, blackout zones are based on zone numbers. Like most local utility, your PG&E bill lists in which blackout zone number your facility resides. Most likely blackouts would last for 90 minutes and
proceed by numbered zone. For example, if zone 06 was the last zone to experience a blackout (say the prior day) when the next blackout is scheduled it will be zone (zone 07) that is the first to be darkened---then 90 minutes later power is returned to zone 07 and then zone 08 is blacked out for 90 minutes---And so on. If you become aware that blackouts are in effect you may be able to anticipate if and when your facility may experience a blackout.

If you are able to anticipate an imminent blackout:
* Instruct all of your staff to save any open data files, then close all applications and shutdown their workstations (i.e. turn the power switch off).
* Then have appropriate staff power down servers using the proper procedure (defined below).
* Finally, power off all printers and other peripherals such as scanners, modems, and the like. DO NOT TURN OFF the power switch on Internet or network hubs.

If an outage or blackout occurs without prior warning:
* All workstations will of course shut down. Instruct your staff to turn the power switches on their computers to the OFF position ? this is because you will want to turn systems back on in a specific sequence defined below. No need to rush ? you?ve got between 60 and 90 minutes to accomplish this. 

* If you're an ActivSupport customer, your servers will likely still be running as most servers at most Activsupport clients have an Uninterruptible Power Supply (UPS) attached. Recognize that the primary function of a UPS is to protect critical systems like servers from both spikes and dips in AC power. Note that surge protector only protect systems from power spikes, not from power dips. UPS devices are not designed to allow systems to run for significant periods of time without an AC power supply. A typical UPS offers between 5 and 15 minutes of power to a typical server. Use this available time to have an appropriate staff member shut down each server suing the proper shutdown procedure. Once the server(s) is connected to a given UPS device switch the power off on that UPS. The power switch is on the back of most UPS units.
Note: Most UPSs emit a loud alarm tone when AC power fails. Ignore this tone. Once you switch the UPS power off the alarm will stop!
* Finally, power off all printers and other peripherals such as scanners, modems, and the like. Note: DO NOT TURN OFF the power switch on Internet routers or network hubs. 

Restoring the network after power is turned back on:
When power is restored wait about 3-5 minutes before reestablishing your network as power is typically a bit unstable during the first couple of minutes following power restoration.
Follow these steps in order. The order in which you do these steps is important in order to restore all network services to all users:
* Turn all UPS devices back on (these are typically only used and attached to servers)
* Turn on your servers beginning with the Primary Domain Controller (PDC) first. When it has booted up fully then power on all other servers in any order you wish.
* When (and only when) all servers are fully booted up and running, instruct staff members to turn on their workstations and log in as normal.
* Finally turn on all printers and all other peripherals (such as scanners and modems).

Proper Shutdown Technique All Windows Based PCs (Doing this properly is especially important when turning off a Windows based server).
* If you are shutting down a server---first login with the administrator?s user name and password. (If you are shutting down a workstation this step is not typically required).
* Click on START button at the lower left hand corner of the screen. Now click on SHUTDOWN. A small dialogue box appears on the screen asking if you want to Restart, Logout User, or Shutdown the system. Select SHUTDOWN and click on the OK button.
* Be sure to wait until the message appears ?It is now safe to turn off your system?. This can take up to 5 minutes on an especially large or slow server. When you see this message turn the computer power off. 

Some tips to minimize data loss in the case of an unanticipated blackout:
Many programs such as the Microsoft Office Suite (Word, Excel, Access, PowerPoint, etc.) can be set to automatically save your work every X minutes.Use this option and set it to a short time frame like every 3 or 5 minutes. Unless you are working with an especially large data file these automatic saves will have little if any noticeable effect on the applications performance. Proper data backups to tape or other appropriate media are always important. But in the current environment in which blackouts are especially likely its wise to be all the more prudent concerning data backup. If the power suddenly fails in the midst of saving a new or updated file to disk, there is a high possibility that this file will be damaged, often irreversibly.
Consider adding a UPS unit to servers and to workstations on which essential data is regularly accessed and manipulated.