State, IBM share blame for data contract problems, insiders say

The $863 million IBM Corp. contract to consolidate the computer data centers of 27 state agencies offered the promise of doing more for less.

IBM and its partners — the Team for Texas — would modernize and streamline how the agencies store and protect their data. And the state would save $178 million over the seven-year term of the contract, which began in April 2007 .

But many of the agencies involved say they are actually paying more to get less, as indicated by IBM's customer satisfaction score in August that hovered just above poor.

In the past few weeks, the agencies have won a powerful ally in Gov. Rick Perry, who called for work to stop after IBM's failure to back up some critical data came to light. IBM now faces the possibility of losing the contract if the company does not quickly fix the backup problems that have plagued the mammoth project.

As state officials wait to see whether IBM delivers on its promise, the question remains: How did this much-ballyhooed project go so wrong?

Interviews with industry experts and state officials suggest that the state and IBM share the blame.

State officials, including legislators, were unrealistically ambitious by requiring the contractor to move quickly to centralize all the functions of the data centers, which all had different equipment and applications, said Jim Moreno , who retired as a system administrator in May after 17 years with the Texas State Library and Archives Commission.

And IBM overreached with an aggressive plan that did not reflect the true cost of the work, critics say.

In a Nov. 3 letter to the governor's office, IBM appears to acknowledge that it bit off more than it could chew by assuming responsibility for existing technological conditions that are inadequate, inconsistent and not sustainable.

"The state and IBM will have to evaluate the investments required to remediate the current deficiencies, develop a funding model, and prioritize accordingly," wrote two IBM officials.

IBM spokesman Jeff Tieszen said he could not comment on the letter.

Hurdles from the start

The consolidation effort was prompted by the problematic conditions at the agencies' existing data centers:

• Sixty percent of the equipment in the data centers had exceeded its recommended working life.
• Fewer than half of the applications were protected by a disaster recovery plan.
• Critical environmental requirements at the specialized facilities, such as fire suppression, backup power and ventilation, were lacking.

The state also was missing an opportunity to use its size to reduce costs because the data centers were managed by the individual agencies.

Northrop Grumman Corp. had been operating the state's San Angelo data center since 1998 , when it took over for the original contractor, IBM.

That facility was meant to provide computer operations and disaster recovery services to state agencies as a way to reduce costs. Few agencies used it, though, so the state never saw those savings.

In 2005, those issues led the Legislature to mandate the merger of the 27 agencies' data centers into two , including a new facility in Austin and the existing operation in San Angelo.

Northrop Grumman, as an existing contractor, touted its "unequalled familiarity" with the state's technological infrastructure and offered a $1.6 billion proposal.

IBM's $863 million deal looked like the best value.

A major part of the price difference stemmed from IBM's plan to reduce the number of servers from more than 5,300 to about 1,300 so that 95 percent of the agencies' data would be on shared servers in the third year of the contract.

Northrup Grumman's plan kept more servers distributed across the state at the remote locations, and that approach cost $358 million more.

Service a concern

"We sold the consolidation plan on savings, but we will sustain it on service delivery," Texas Department of Information Resources Director Brian Rawson said at a conference in September, according to a report on the technology Web site, Channelweb.

Nineteen months into this contract, many of the customer agencies have been asking: What service?

The agencies cite lengthy service backlogs, ineffective communication, unqualified staff and inconsistent data backups as some reasons for IBM's poor performance.

The Texas secretary of state's office gave the IBM team, which includes Pitney Bowes, Unisys and Xerox, a customer satisfaction score of 0 on a scale of 1 to 5.

"Team for Texas seems unable or unwilling to correct shortcomings that have been evident for months," the secretary of state's assessment said.

The Texas Facilities Commission said, "August was a bad month for IBM performance," citing one example when a 15-minute task took one month to perform.

The Texas Department of Transportation contends that the IBM employees have no urgency about or ownership of the department's business. Problems that used to take state employees less than an hour to fix now take as long as a week, according to the department's most recent assessment of IBM.

Ed Serna , the Transportation Department's assistant executive director for support operations, said he has not seen any real value in the contract so far.

"We didn't gain anything \u2026 that we didn't have," Serna said.

He said he hopes that the attention on IBM's performance leads to an improvement of the system and that other persistent problems, such as slow procurement and inadequate staffing, also get addressed.

Chad Kissinger , president of a company that operates a data center in Austin , said it was bad public policy "to force one solution down all their throats" given the agencies' geographic disparity and varied functions.

Smaller agencies and remote locations will probably not get the level of attention that they want, Kissinger said.

Tieszen said IBM is committed to addressing the agencies' service concerns.

Cost savings uneven

At the time the contract was signed, the state and IBM said taxpayers would save $25 million over the first two years.

An early cost analysis by the accounting firm Grant Thornton showed the project had saved the state overall about $500,000 through February. A more recent cost-savings analysis is not available, said a spokesman for the Department of Information Resources, the state agency overseeing the contract.

But many agencies say cost savings have not materialized for them. Half of the agencies have requested additional money for the 2010-11 budget on top of their current allocations to pay for rising costs under the data center contract.

For example, the Texas Higher Education Coordinating Board has asked the Legislature for an additional $536,000 in its 2010-11 budget to cover its rising data center costs, which have just about doubled compared with its costs before the contract.

Information Resources attributes the increased costs at the coordinating board to unusually high networking costs because its headquarters is outside of the Capitol complex network.

"We've had concerns, and we've had bumps along the road," said Deputy Commissioner Arturo Alonzo.

But the agency supports the project and sees great opportunities through the consolidation, said Darla Fent , the agency's assistant commissioner for information technology services.

Some of the carping might be attributed to institutional turf wars.

A 2007 survey by the National Association of State Chief Information Officers found that work force resistance to change and agencies' desire to remain autonomous were by far the top obstacles to state government data center consolidation efforts across the country.

But there is no escaping that IBM has repeatedly failed to do the job it was hired to do.

In Tyler, IBM did not back up data on a server at the attorney general's Medicaid fraud division, as required by the contract.

When the server crashed, eight months of data files from the office's fraud investigations were initially lost because the office had stopped using its previous file backup software in November. Ninety percent of the data have since been recovered, but it is unclear whether all of that data were usable.

Federal food stamp funding was put at risk after 17 incidents at the IBM data center that had a "direct, negative impact" on recipients' ability to access their benefits, according to a warning letter to the Texas Health and Human Services Commission. One incident involved a 2?-hour outage after a lightning strike at a data center, and IBM's disaster recovery plan proved insufficient.

Information Resources had consistently notified IBM that it was not fulfilling all of its obligations, and it has penalized the company $5.2 million for missing certain service goals. But the warnings had no heft until Perry's office intervened in late October.

On Nov. 4, Information Resources gave IBM an official 30-day warning that could lead to termination for cause if the team's persistent failure to back up the agencies' data is not remedied. The department is also developing a contingency plan for ensuring data center services for the agencies without IBM.

kalexander@statesman.com; 445-3618