House Servers Operate From Backup Facility

Posted November 3, 2008 at 6:44pm

When a power outage disabled House e-mail and BlackBerry services about three weeks ago, a backup power supply should have taken up the slack.

Instead, Members and staffers were without official e-mail for almost a day, while House officials waited for a California company to send a replacement part.

In a recent internal report, the Architect of the Capitol’s office pinpoints an overloaded circuit breaker as the problem.

Employees of the Chief Administrative Officer connected new equipment to one power panel without ensuring it could handle the extra load, according to the report.

But the incident also reveals a bigger issue: The House’s primary servers are run out of the backup computer facility. That means that when the facility has a power outage — or a blown circuit breaker — there’s no second location to run services such as e-mail while the problem is fixed.

It wouldn’t be an issue if the primary facility, housed in the Ford Building, was fully functional. But it’s undergoing renovations, according to several sources.

On Monday, the ranking member of the House Administration Committee called for those renovations to be sped up.

“The CAO must work to expedite the restoration of redundant IT services in Ford, which are currently behind schedule so that we have a fail-safe system that ensures critical House services remain available without interruption,” Rep. Vernon Ehlers (R-Mich.) said in a statement.

The Alternate Computer Facility was acquired after Sept. 11, 2001, to ensure continuity if an emergency or power outage shut down servers at the primary location.

But at some point, some of the main House servers were moved to the ACF — transforming it into a primary facility and eliminating the only backup option.

CAO officials wouldn’t comment on the Ford renovations or where servers are housed, citing security concerns. House Administration spokesman Kyle Anderson also declined to talk about the situation, only stating that the committee “is strongly focused on ensuring that House Information Systems are secure and uninterrupted.”

But CAO spokesman Jeff Ventura said the Oct. 9 outage did reveal that the main circuit breaker did not have “electrical redundancy,” or a backup power supply.

CAO officials say that problem is fixed, at least for the moment.

“We have engineered a short-term solution to the problem, one which provides increased fail-safe integrity to our electrical systems,” Deputy CAO Ali Qureshi said in an e-mail. “And we are now going to work on the longer term solution with the AOC.”

The AOC’s report seems to lay the blame on the CAO’s IT staff, though not overtly.

Addressing the “root causes,” AOC officials wrote: “The House Data Center did not communicate with AoC when new equipment was added, which did not permit an analysis to determine if the circuit could handle the load.”

As a result, the new equipment pushed one of three circuit breakers over its limit. That main breaker was handling 635 amps, when it was designed to handle only 600.

The AOC is in charge of the physical building, meaning it measures the power being used. But the CAO handles what’s inside the building, such as the equipment that uses that power.

The two didn’t communicate.

“The AoC had no prior knowledge of what equipment was being installed and or tied in the House data center,” the report says, adding that House Data Center staff has previously “denied AoC’s requests to perform the needed preventive maintenance on any of the data center electrical systems.”

Qureshi said he disagrees with the report’s findings.

“We are not in complete agreement with the AOC’s findings, which is why we are scheduled to discuss the issue with them this week,” he said. “We do feel, however, that as a result of this upcoming meeting, we will ultimately come to an agreement about what caused the problem and, together, make sure it does not happen again.”