Lack of backup foils Va.‘s new IT system
TIMES-DISPATCH
In just five weeks this fall, the Virginia DMV suffered 12 computer system outages, putting individual offices out of business for a total of more than 100 hours. One outage lasted 29 hours, another 17.
RICHMOND, Va. -- Virginia's expensive new state IT system does not have network backups in case connections between its computers fail.
In just five weeks this fall, the Virginia Department of Motor Vehicles suffered 12 computer system outages, putting individual offices out of business for a total of more than 100 hours. One outage lasted 29 hours, another 17.
"The problem of no-redundancy . . . accounts for 90 percent of our outages," said David W. Burhop, the DMV's chief information officer.
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
"That means people cannot do their work," said Murali Rao, VDOT's information technology director.
Problems with the state's outsourced computer and communica tions systems can have far-ranging impacts, affecting members of the public trying to do business with state agencies and government employees struggling to do their jobs in the face of computer crashes, lost e-mails and slow repairs.
"We've been literally having problems since July keeping the driver's-license system up," DMV Commissioner D.B. Smit said yesterday in a rare criticism of another state agency. "It's putting my people at risk because customers are angry.
"Every time we're down for an hour, that's about 2,500 people inconvenienced," Smit said. "They're blaming my people for it and [state IT officials] have an obligation to fix it."
In a unique public-private venture, Virginia agreed in 2005 to let the giant defense and information contractor Northrop Grumman run nearly all the state's IT systems.
The 10-year, $2.3 billion project aims to modernize 85 state government agencies' computer networks, PCs, phones, servers and e-mail systems, while holding down costs. The deal also provides IT services to about 1,000 local government customers.
Overseen by the Virginia Information Technologies Agency, the troubled outsourcing deal is the first and largest of its kind in the nation.
George F. Coulter took over as the state's chief information officer in August.
"The first thing I noticed was that the network that Northrop Grumman rolled out didn't have redundancy, backup," Coulter said yesterday. "The contract does not call for redundancy in carriers . . . in the network.
"Why that wasn't put into the network, I don't know," Coulter said. "This is a service we have to have."
Coulter is calling for an emergency meeting of the state's Information Technology Investment Board in the first week of December to deal with the lack of backup in the state's IT system.
"We don't have the pricing on it yet," he said.
"We're going to go after the critical agencies and the critical locations as a priority," Coulter said, pointing to VDOT, DMV and the state Department of Corrections as needing urgent attention.
"It's a major issue," he said. "We've got to get it fixed."
Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.
But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.
"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.
"It took VITA four hours to open the help ticket" and begin to solve the problem.
Contact Peter Bacqué at (804) 649-6813 or
.
Advertisement
Reader Reactions
yep. VITA/NG hasn’t suprised me…the Agencies have. While I understand we are driven by “law”...i’ve been suprised how the agencies have had to “buckle” under. VDOT had redundant circuits…we had working processes in place..and we continue to watch NG put in place intitiatives that don’t work…then when we tell them they don’t work..show them how they are suppose to work…we pay for it. The state is hostage. We don’t have the money to get out of this fiasco. I wish that we had the NG people that are “presented” in the propoganda (Ng’s plan to fix this problem)...my dealings have been..once I find a NG person that UNDERSTANDS what needs to be done and wants to work with us…they are “let go..downsizing ya know”...I’ve seen DIT mentions…they failed…sorry guys…you guys were suppose to be the “original” VITA and you couldn’t handle it…so you did telecommunications…its was what you could handle…VITA came in…without a clear understanding of what worked and…what didn’t…you were doomed for failure by the politicians. Those of us who live it wonder how we will ever get thru it… the public needs to understand…it ain’t 400million to get out of the contract…even if NG said..ya know what…we leave…never mind the 400 million…the state doesn’t have the money to re-build their IT base…WE don’t own our network anymore…we aren’t even a “customer” of Verizon anymore…every cirucit is owned by NG.. we can’t even get voice mail changed. those of us in the IT world back at the beginning of this fiasco tried to tell people…no one listened.
skippypeanut- I would bet you a pay check not ONE SINGLE agency manager would agree with you.
VITA isn’t the reason the transformations are late and when they are being worked the prelim work has been done poorly or not at all. Jobs that should take a day are taking a week and how that or the network outages are VITA’s fault is beyond me.
I can tell you VDOT’s pre NG network would run circles around what you have created and I promise the down times were far, short a few between.
And one of the many reasons was because WE ALREADY HAD BACKUP CIRCUITS !!
Wait and watch if NG doesn’t get paid to replace for they had no business taking away.
revnhoj, I hear you. The sooner the better.
Don’t spend another dollar on the mess.
Oh yeah, I did not mean to exclude VITA, this whole joke was DIT being control freaks, inept ones at that and trying to save their jobs, that goes for DTP (Project Manglement) as well.
I’ll go one better for you, NGArchitect; dump VITA and NG altogether and return IT to the individual agencies.
Start the disentanglement process NOW.
As always Charles Haugue is absolutely correct. He is much more eloquent in his posting than I will ever be or hope to be.
Anonymous,
DIT was NOT a failure. I was there, you obviously were not or you would know better.
Charles, please keep posting the facts and Happy Thanksgiving to you sir.
Another anonymous.
IMO - DIT was not a failure - it did what it was supposed to do and did it well. However, VITA was created to do something different than DIT did. It was supposed to have been the central agency to maximize the economies of scale so that agencies could leverage the buying power of the Commonwealth as oppposed to each individual agency. In order to do this, VITA would have needed to have a major emphasis on customer service. In other words, great idea, terrible execution. NG was hired to be the delivery arm to VITA and VITA was the interface to the agencies. Except, neither NG nor VITA had ever been trained to be able to provide quality customer service.
Despite this, VITA has accomplished much. They have delivered on legalizing the software that existed on PCs throughout the state. They have standardized on antivirus software and made tremendous progress on having one central email system. They have consolidated and simplified the hardware infrastructure. They have a better understanding of the overall IT assets that exist. This is all good. However, when it comes to responding to agency requests for IT, they have clearly fumbled the ball. NG severely underestimated the amount of work required to meet the contractual obligations. All parties knew that the rule of thumb was “equal or better service” for the same cost. NG thought they could do that for much less and was wrong. Now, they are trying to leverage their large presence in the state, the fact that the Commonwealth has to either pay up or declare them in default of their contract. Clearly, no Governor wants to sue the largest employer of Virginians in the Commonwealth. So, there is a stare down. Who will blink first? My guess is that McDonnell will come in and strip VITA of all authority related to IT procurement and give it to DGS, move VITA PMD, VEAD, and all other departments to DGS, and let George Coulter manage NG and do nothing else. They will continue this dance until the contract expires and then VITA will need to figure out what to do next. It will cost much more to get a new vendor to take over the contract and even more to bring it back in house. Who loses? The citizens of the Commonwealth. All because VITA chose a company that lacked the capabilities and experience to get the job done and buckled to a company that had significant political leverage. This was very predictable.
Part of Anonymous’ comment, “just like DIT, VITA/NG is a failure…“ is accurate. However, characterizing DIT, and by association the agencies, as a failure was one of the false assumptions that resulted in the creation of the vita/NG fiasco. If you believe DIT was a failure, provide facts to support your opinion.
The partnership of the agencies and DIT provided the state with innovative, cost effective, award winning technology.
NGArchitect: If you think assuming redundancy is funny, you shouldn’t be allowed to architect anything!
Dear NGArchitect (which seems to me to be in contradiction in terms). VITA is not the problem. NG is. They bid on a project that they had absolutely no idea how to do. See problems in Indy and SD. The only thing VITA did wrong was not picking IBM who understood what they were bidding on. NG quite simply underbid the work and thought that they would have enough leverage to either increase billing or get localities and other businesses to utilize the big buildings that you built to offset the costs. You overestimated the savings and now you are stuck. You want more money. I am glad that my comment made you laugh quite hard, but I am sure that you won’t be laughing when the GA decides to throw NG out on its backside ... Unless your company decides to just continuing losing money and reputation by providing crappy services. There is a trainwreck in your future.
Thanks, NGArchitect for displaying the typical attitude of NG: “We know what’s best and everyone else is just stupid”. It’s too bad that your work doesn’t reflect your self-described genius, because you have yet to demonstrate any successes or even basic understanding of the requirements of the agencies’ processes.
Perhaps if you could demonstrate that you could implement one single thing correctly without causing myriad problems and failures to fix breakages, the VITA personnel wouldn’t feel the need to micromanage so much.
Provisions in the contract include phrases like “current level of service” in many places. Maybe that’s why Somethingstinks thinks you should have assumed that systems with redundant links should continue to have redundant links, rather that just providing what your superior intellect tells you is all that’s needed.
There are some folks around you at NG that are actually competent, and I’m sure the agencies would prefer that VITA was out of the way so that they could deal with these people directly. But not if the billing issues aren’t fixed. Agencies are now stuck between getting lousy or non-existent services, or paying through the nose and dealing with excessive billing for services that they used to have by default.
Post a Comment(Requires free registration)
- Please avoid offensive, vulgar, or hateful language.
- Respect others.
- Use the "Flag Comment" link when necessary.
- See the Terms and Conditions for details.
Advertisement