Some state government computer service restored

5:20 p.m.

Service is being restored to state government computers knocked out by an equipment failure at Virginia’s information-technology headquarters in Chesterfield County.

Samuel A. Nixon Jr., chief information officer, says that nearly 230 servers out of a total of 3,600 crashed Wednesday afternoon, disrupting such services as issuing drivers licenses and processing jobless claims. 

 The massive outage disrupted  services for thousands of member of the public at several agencies, including the Department of Motor Vehicles and the Virginia Employment Commission.

 Nixon said this afternoon that 38 of the affected servers are running again, and that the state’s computer agency and its contractor, Northrop Grumman, are working to get the others back online.

"We’re disappointed to have a failure, an outage of this magnitude," said Nixon. "No matter what you do, it’s going to happen on occasion."

Nixon said the interruption was of insufficient magnitude to activate a backup system in Southwest Virginia. He said believes state computer data is largely intact.

This incident comes about four months after the state rewrote its IT contract with Northrop Grumman, pledging more money to the company in return for improved service.

Nixon said it’s too early to determine whether Northrop Grumman will be hit with financial penalties because of the interruption.

"It depends on how the outage remains," said Nixon.

Megan Mitchell, a spokeswoman for Northrop Grumman, said, "Knowledgeable and dedicated staff at the agencies, VITA and Northrop Grumman are working together to respond appropriately to the impacted systems.

"It is our priority to minimize these impacts and restore services as quickly as possible."

1:30 p.m.

While officials are largely silent on computer outages across Virginia government, there are signs this afternoon that service is being restored.
 
An e-mail update circulated at midday within the state's computer department -- a copy was obtained by the Richmond Times-Dispatch -- indicates that repairs are underway.
 
"The ... technicians continue to work on the manual errors," said the Virginia Information Technologies Agency. "There are approximately 21 servers that have been returned to service; other servers are in various stages of recovery."
 
Still no word, however, from executives at VITA or its contractor, Northrop Grumman. The systems collapse apparently is the biggest to hit VITA and Northrop Grumman since 2007.

(This has been a breaking news update. Check back for more details as they become available. Read a complete account in tomorrow's Richmond Times-Dispatch.)

10:56 a.m.

Computer service across Virginia government is being interrupted today because of equipment problems at the state's embattled info-tech agency. The outage could be the largest for the state since 2007.
 
Outages popped up yesterday afternoon and are continuing this morning, forcing, for example, the Department of Motor Vehicles to suspend issuing drivers licenses at its 74 offices across the state.

VITA confirmed that failures in its systems caused the problems. 

"We have a system that stores data for several agencies and it had a hardware failure," said VITA spokeswoman Marcella Williamson.

"There was a redundant memory board but that failed also," she said. "And because of that, some servers can't get access to the data that’s stored on them."

VITA had sent out bulletins overnight indicating outages at more than two dozen agencies with computer servers at the  Chesterfield headquarters VITA shares with the state's IT contractor, Northrop Grumman.

"The. ... technicians are still troubleshooting the issue, attempting to recover the data on the storage-area network," a VITA update said. "Multiple agencies throughout the state are currently experiencing server failures. This is impacting the agency's ability to access applications, shared folders and other data stored on the servers to perform daily job duties."
 
The state recently rewrote its contract with Northrop Grumman, assuring the company even more money in return for improving service. The privatization contract, exceeding $2.3 billion, is the biggest ever issued by Virginia government.
 
In addition to DMV, affected agencies include the Department of Alcoholic Beverage Control, the Virginia Employment Commission, the governor's office and the departments of Emergency Management, Transportation, Taxation, Criminal Justice Services, Environmental Quality, Social Services, Juvenile Justice, Planning and Budget and Health.

DMV customers who can renew licenses through another service option are encouraged to go online at www.dmvNOW.com, or use the automated telephone service at 1-888-337-4782.

The outage began at 3 p.m. Wednesday. VITA and its Northrop Grumman partner's rapid response team worked through the night on the outage, which is affecting agencies in different ways.
 
At DMV, all other types of transactions, including vehicle decals and titles, transcripts, etc. are being processed normally.  
       
"The work we have done on the infrastructure has made our systems more secure and reliable," a VITA representative said, "and when something like this does happen we have people and processes in place to respond."

VITA's own website is among those affected: "An unexpected error has caused this application to quit," it said. "All processes have terminated."