|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
|
|
As ERP implementations falter and fail, many people think the answer is more training. They're wrong.
|
|
In reality the ERP industry has hype as one of its cornerstones and it hasn't been performing like the marvel it was first made out to be. And that is the fact of the last two decades.
In the beginning in pre-Y2K years, plunging sales revenues and falling stock values were more common for SAP implementers then not. Now with much more powerful hardware and better networks they are less common. They tend "kind of work". But after 2000 came another interesting realization -- that all that hard work implementing an ERP system didn't actually guarantee business benefits. Or a positive return on investment. Still fashion, inertia and brainwashing ensures the continuation of SAP implementation, but companies learned to swipe the problem under the carpet more effectively.
Take old and pretty damning Meta Group's finding, for instance. It still is as valid today as when it was released: The average ERP implementation takes 23 months, has a total cost of ownership of $15 million and negative rewards to the business with an average negative net present value of $1.5 million. And for larger organization typically the news gets worse. Compaq was essentially ruined by SAP implementation.
An alarmingly long list of top-drawer integrators have fallen flat on their faces. Compared to these disasters, merely spending a lot of money on preparing to the ERP implementation and then abandoning the idea and reaping some benefits from house cleaning looks like not a bad business strategy.
Stable and profitable Pa.-based Hershey Foods, for example, issued two profit warnings after SAP/R3 implementation. Why? Massive distribution problems following a flawed implementation of SAP's R/3 ERP system, which affected shipments to stores in the peak Halloween and pre-Christmas sales periods. In a booming stock market, Hershey shares ended the year down 27% from its year's high. And Hershey wasn't alone in its misery. In November 1999, domestic appliance manufacturer Whirlpool of Benton Harbor, Mich., blamed shipping delays on difficulties associated with its SAP R/3 implementation. Like Hershey, Whirlpool's share price dove south on the news, falling from well over $70 to below $60.
While these two are know the highest-profile implementation debacles, many such disasters are just swiped under the carpet and never reported by the press. Companies as diverse as Scottsdale, Ariz.-based trash processor Allied Waste Industries; Newark, Del.-based high-tech fabric maker W.L. Gore & Associates; and industrial supplies distributor W.W. Grainger of Lake Forest, Ill., have all reported serious difficulties. So it is very difficult to see the real scope of SAP/R3 implementation disasters.
And if "serious difficulties" sounds bad, rest assured it can get much worse. When it launched a $35 million enterprise resource planning (ERP) project way in 1993, FoxMeyer Corp. was a $5 billion drug distributor in Carrollton, Texas. Now it's bankrupt. Texas-based pharmaceutical distributor actually collapsed following an SAP R/3 implementation, its bankruptcy trustees filed a $500 million lawsuit in 1998 against the German ERP giant, and another $500 million suit against co-implementer Andersen Consulting.
That actually tells you something about those sharks who implement SAP. They are dream of your competitors and without tough control can really devour the business as they do not care much about the results. Only money you pay to them. Consider them to be a Trojan horse you allowed into you fortress and put necessary guards to prevent mischief. If possible try to get a second opinion on key decisions. Among alternatives they consistently chose what is convinient for them, not what is the optimal for the business.
So what's going on? The good news and as most experts agree such failures are not systemic. "Very rarely are there instances when it's the ERP system itself — the actual software — that fails," says Jim Shepherd, senior vice president of research at Boston-based AMR Research. Public pronouncements by both SAP and Hershey, he notes, have acknowledged that the software does what it is supposed to and that no big fixes or patches are planned. The key problem is that it does not fit well to the business. It's too rigid and many implicit assumptions made are iether incorrect of hurt the current business processes. With predictable results. Also maintaining SAP/R# system is an expensive proposition. Many large companies hire on-site engineers from the hardware vendor to ensure that downtime is minimized.
Also for large companies SAP implementation usually corresponds with the drive for centralization. Company management likes the idea of a single central SAP system. And such systems are often horribly slow: you click the button go for the cup of coffee and only when you return the screen is updated ;-). Tremendous progress of hardware from early 90th greatly helped here to resolve to make this problem less acute. Still for large multinationals with central SAP instance this is the problem of architecture not particular implementation.
Like with other systems implementation we also need to differentiate between real implementation failures and a conscious chess-like move by the management to make the system implementation a scapegoat (although probably not without the reason).
"Blaming the failure on a system implementation has become a convenient excuse for companies that have missed their quarter-end [earnings] target."
As for blame, other ERP implementation face not better. SAP implementations are no more likely to go down the tubes than ERP systems from other vendors: W.L. Gore's system, for example, came from Pleasanton, Calif.-based PeopleSoft.
"When an ERP project unravels, or is seen not to perform well, one of the suppliers is usually chosen as the culprit,"
says David Duray, London-based global partner responsible for the SAP implementation business at PricewaterhouseCoopers.
"In my experience, this is usually more of a political decision than a proper problem-source identification exercise—and SAP, over the last few years, has been a popular target."
Furthermore, adds Roger Phillips, an IT analyst at specialist investment bank Granville in London, which tracks the global ERP market, there is no evidence that geography is a significant differentiator in the success stakes. Disasters, he believes, "simply go with the ERP territory." There are, he says, "no cultural or managerial foibles that make American ERP implementations any more predisposed to disasters than any other country's implementations."
So what does lie behind ERP disasters? And behind the rather longer list of costly-but-underwhelming implementations typified by that now-infamous Meta Group report? Increasingly, experts reckon that they've found the smoking gun: poor training. Not the technical training of the core team of people who are installing the software, but the education of the broad user community of managers and employees who are supposed to actually run the business with it.
In an effort to understand the technology issues faced by online brokers, the Attorney General's Office examined the growing frequency of mission critical failures at firms that installed major new information systems.
Both the growth of e-commerce and the attention paid to firms that experience mission critical failures has elevated the industry's awareness of the need to adopt risk assessment and mitigation strategies. "E-commerce will be the next area for risk management," says Lynn Edelson, head of PriceWaterhouse Coopers' operational and systems risk-management practice. "Organizations are starting to understand that e-commerce is not only a financial proposition, but an enterprise-wide system that needs sophisticated assessment to identify which risks can be controlled and which can't."173 In the following sections we identify business processes that foster quality software and reduce risks for technology dependent firms.
Five Lessons from AT&T Wireless's Project Failure:
Introduction
Enterprise Resources Planning (ERP) is an outgrowth of Material Requirements Planning (MRP) initiated in the 1970's as a new computer-based approach to planning and scheduling of material requirements and inventory, featuring the time-phased order point. MRP evolved to MRP II (Material Resources Planning) the "closed loop" process, to Business Requirements Planning (BRP) and eventually to ERP. As MRPII came into vogue in the late 1970's and early 1980's, software companies began to develop software packages around MRPII concepts.
At the same time, research of integrated data bases was in progress at a university, and out of that research emerged data base management systems (DBMS). One of the earliest successful commercially-produced data base management systems was IDMS (for IBM-based systems) and DBMS (for DEC-based systems) produced by Cullinane, who's company name was later changed to Cullinet. IMS, a structured data base management system for high transactions, was another data base management system produced by IBM.
The idea of the integrated data base as the engine for fully integrated software was probably one of the greatest outgrowths of Ollie Wight's and Dave Goddard's MRP. Eventually, the acronym ERP was conceived to represent what had already been developed by software companies.
The early software packages were developed by way of a transactional approach, and were highly unfriendly to a user. With the advent of the personal computers, the development of Microsoft's Windows NT, and the mid-range IBM AS/400 computer, client-server systems began to emerge. Windows, used as the base operating system, allowed software packages to become more and more user-friendly.
Today, ERP systems have proliferated entensively, and have reached a stage where development has become industry specific. Thus it is plausible to search for an ERP package developed for one's specific industry idiosyncracies.
The Issues
The biggest single issue in ERP is the failure of a successful implementation. It is mind-boggling to continually encounter companies who make major ERP gaffes in this day and age, especially since most of the trials and tribulations of MRPII implementation were suffered and learned from in the early 1980's with alpha, beta and gamma releases.
So what constitutes failure? Several thing come to mind: (1) Not making the promised
return on investment,
(2) Inordinately extending the implementation schedule and start-up date,
(3) Running over budget by unconscionable variances,
(4) Grinding the organization to a crawl pace, or the severest of all consequences,
(5) Stopping production and/or not delivering orders to your customers.
Industry statistics show that >60% of ERP implementation starts historically fail. Does this
mean that you are doomed from the start? Of course not, if you learn from the mistakes of others.
So the pertinent question is what are the main causes of ERP failure and what can be done to prevent
this from happening to you?
There are twelve major reasons for why companies get bogged down or fail in implementing ERP.
(1) Lack of Top Management Commitment
The propensity of top management to delegate the oversight of an ERP implementation to lower management levels often results in (1) being "out of touch" with critical events, or (2) the lack of understanding of the size, scope, and technical aspects of the project, and subsequently, the lack of proper commitment of time and resources required for a successful implementation. The result is a failure waiting to happen.(2) Inadequate Requirements Definition
Surveys have shown that inadequate definition of functional requirements accounts for nearly 60% of ERP implementation failures. This is simply a matter of not comprehensively and systematically developing a quality set of functional requirements definitions. This leads to the second greatest cause of ERP implementation failures: poor package selection.(3) Poor ERP Package Selection
Poor package selection occurs when a company has inadequately developed functional requirements definitions. It also occurs when staff members assigned to ERP projects do not take the time to run the screens of the new system, as they would during their daily work tasks, to find out if the software package features are adequate for their needs.Another reason we have found is executives, familiar with an ERP system from a last job they held, implement the same system in their new company without defining functional requirements. We have also encountered companies who made major gaffes by selecting a package at the top levels of a company without intimately knowing its characteristics. What often results from this is the ERP package doesn't fit the organizational needs, or that the package selected takes longer to process daily work tasks.
(4) Inadequate Resources
The third greatest reason for ERP implementation failures is inadequate resources. Many companies will attempt to "save dollars" by doing everything on an overtime basis, whether or not there are adequate skills within the company, extending individual work loads to 150%. This approach can be a "kiss of death" for the program. Time and time again we run across this mistake in ERP implementations. The financial and emotional drain of what seems sometimes to be perpetual extensions, reschedules and delays of implementations takes its toll. People burn out after having put in extensive hours over a long period of time.(5) Resistance to Change/Lack of Buy-in
The lack of a change management approach as part of the program can prevent a program from succeeding. Resistance to change is quite often caused by (1) A failure to build a case for change, (2) Lack of involvement by those responsible for working with changed processes (3) Inadequate communication (4) Lack of visible top management support and commitment, and (5) Arrogance. A lack of buy-in often results from not getting end-users involved in the project from the very start, thereby negating their authorship and ownership of the new system and processes.(6) Miscalculation of Time and Effort
Another cause of ERP implementation failure is the miscalculation of effort and time it will take to accomplish the project. Companies who treat an ERP selection, evaluation and implementation comparable to buying a washing machine are doomed to failure.(7) Misfit of Application Software with Business Processes
One of the main causes of ERP implementation failure is the misfit of application software with the company business processes. This failure -- to examine underlying business process flaws, and integrate the applications with the business processes, causes loss of productivity and time, and ultimate benefits.
(8) Unrealistic Expectation of Benefits and ROI
Another significant cause for ERP implementation failure is the unrealistic expectation of benefits and return on investment. Software providers are notorius for overstating the benefits in terms of ROI, when the total costs of the project have been understated. Often left out of the total costs are costs of planning, consulting fees, training, testing, data conversions, documentation, replacement staffing, and the learning curve performance drop. When this happens, a company doesn't stand a chance of achieving the ROI it anticipated.
(9) Inadequate Training and Education
Another of the biggest causes of ERP implementation failure is inadequate education and training, which are almost always underestimated. ERP-related training is crucial as most employees must learn new software interfaces and business processes which affect the operation of the entire enterprise. The corporate culture is impacted by changes in the company’s business processes, and shortchanging this part of the ERP implementation leads to much pain and suffering downstream.(10) Poor Project Design and Management
A major mistake is to short-cut critical events in the project plan, such as time for documentation, redefining and integrating processes, or testing before going live. Another common mistake is made when a company leaves out the self-examination of business processes and uses ERP to cover-up weaknesses. It is easier to buy software than to perform the more difficult task of identifying weaknesses and opportunities for improvement.
(11) Poor Communications
One of the causes of ERP implementation failure is poor project communications, beginning with a failure to announce the reason for the up and coming effort, and continuing to advise the organization of the progress and importance of the ERP implementation to the company. Poor communications prevent different parts of the organization from assessing how they will be impacted by changes in processes, policies, and procedures. Communications are a vital part of managing change in a corporate environment.
(12) Ill-advised Cost Cutting
Another of the key causes of ERP implementation failure is ill-advised cost cutting. In an effort to avoid temporary conversion costs, some companies take a very risky route and go live at multi-plant sites simultaneously, subjecting all plants or some plants to a total shutdown should there be a false start-up. This is suicidal. Others attempt to unrealistically compress the schedule in order to save on expenses, only to eventually overrun both schedule and budget. We feel that ROI should take a "back seat" when upgrading an important part of a company's infrastructure: the information system. Instead the implementation should be treated as an upgrade necessary to maintain or gain a strategic and competitive advantage.
|
|
Switchboard | ||||
| Latest | |||||
| Past week | |||||
| Past month | |||||
 |
 |
 |
November 9, 2012 | Computerworld
IDG News Service - IBM has been slapped with a multimillion-dollar lawsuit by chemical products manufacturer Avantor Performance Materials, which alleges that IBM lied about the suitability of an SAP-based software package it sells in order to win Avantor's business.
In 2010, Avantor decided to upgrade its ERP (enterprise resource planning) platform to SAP software, according to the lawsuit, filed Thursday in U.S. District Court for the District of New Jersey.
"Seizing upon Avantor's decision -- and fully aware that, given the competitive pressures of Avantor's industry, and the specialized demands of its customers, Avantor could not tolerate any disruptions in customer service -- IBM represented that IBM's 'Express Life Sciences Solution' ... was uniquely suited to Avantor's business," the lawsuit states. "The Express Solution is a proprietary IBM pre-packaged software solution that runs on an SAP platform."
But Avantor discovered a different truth after signing on with IBM, finding that Express Life was "woefully unsuited" to its business and the implementation brought its operations to "a near standstill," according to the suit.
IBM also violated its contract by staffing the project with "incompetent and reckless consultants" who made "numerous design, configuration and programming errors," it states.
In addition, IBM "intentionally or recklessly failed" to tell Avantor about risks to the project and hurried towards a go-live date, the suit alleges.
"To conceal the System's defects and functional gaps, IBM ignored the results of its own pre-go-live tests, conducted inadequate and truncated testing and instead recommended that Avantor proceed with the go-live as scheduled -- even though Avantor had repeatedly emphasized to IBM that meeting a projected go-live date was far less important than having a fully functional System that would not disrupt Avantor's ability to service its customers," the suit states.
The resulting go-live, which occurred in May, "was a disaster," with the system failing to process orders properly, losing some orders altogether, failing to generate need paperwork for U.S. Customs officials and directing "that dangerous chemicals be stored in inappropriate locations," the suit states.
Avantor has suffered tens of millions of dollars in monetary damages, as well as taken a hit to its reputation among partners and customers, the suit states.
For example, before the go-live, IBM and Avantor met with one of Avantor's biggest customers, which voiced worry that its EDI (electronic data interchange) with Avantor for product ordering wouldn't work after the changeover, and IBM assured the customer it would, according to the suit. "In fact, the EDI interface immediately failed upon go-live."
"IBM, meanwhile, has already pocketed over $13 million in fees from Avantor for a systems implementation project it mismanaged and was unable to perform properly," the lawsuit states. "Incredibly, IBM is now seeking to profit from its misconduct by demanding millions of dollars in additional fees to redesign and rebuild the defective System it implemented."
IBM said it disagreed with the claims and will defend itself against them vigorously. "We believe the allegations in the complaint are exaggerated and misguided and are surprised that Avantor chose to file suit," a spokesman said via email. "IBM met its contractual obligations and delivered a solution that Avantor continues to use in its operations."
Avantor's suit does state that IBM made efforts to right the project's course, albeit ultimately ineffective ones, following a June meeting with Avantor's then-CEO, Rajiv Gupta.
IBM "began to acknowledge the severity of the situation" and replaced many of the original consultants, according to the suit. These workers did extensive redesign and programming.
In July, "IBM told Avantor to cancel every pending order and reset the entire System in light of pervasive warehouse problems," it states. "IBM said this was necessary to discover the root cause of the problem. Ultimately, IBM acknowledged that it had to engage in extensive remedial efforts to redesign and rebuild the System that Avantor hired it to deliver."
"Numerous" IBM workers have told Avantor personnel that IBM failed to manage the project correctly and use SAP "best practices," according to the complaint.
IBM workers even called the project the worst SAP implementation they'd ever seen, it adds.
Avantor is seeking assorted damages in an amount to be determined at trial.
In many respects, the lawsuit reflects other high-profile litigation over SAP projects.
But Avantor broke from tradition by calling attention to its lawsuit via a press release, as companies suing systems integrators and ERP vendors rarely actively seek publicity.
One famous exception came several years ago when Waste Management sued SAP over an allegedly failed project. Waste Management won a sizable settlement from SAP after a public bout of ugly back-and-forth allegations.
It wasn't immediately clear Friday whether Avantor also plans to initiate legal action against SAP, or even has grounds to do so.
An SAP spokesman said Friday he hadn't been aware of Avantor's suit against IBM and declined comment.
Chris Kanaracus covers enterprise software and general technology breaking news for The IDG News Service. Chris' email address is [email protected]
|
|
|
An alarmingly long list of top-drawer integrators have fallen flat on their faces. Compared to these disasters, merely spending a lot of money on an ERP implementation that achieves very little is a consummation devoutly to be wished.
Hershey, Pa.-based Hershey Foods, for example, issued two profit warnings in as many months in the run-up to last Christmas. Why? Massive distribution problems following a flawed implementation of SAP's R/3 ERP system, which affected shipments to stores in the peak Halloween and pre-Christmas sales periods. In a booming stock market, Hershey shares ended the year down 27 percent from its year's high.
And Hershey wasn't alone in its misery. In November 1999, domestic appliance manufacturer Whirlpool of Benton Harbor, Mich., also blamed shipping delays on difficulties associated with its SAP R/3 implementation. Like Hershey, Whirlpool's share price dove south on the news, falling from well over $70 to below $60. While these two have (so far) been the highest-profile implementation debacles, companies as diverse as Scottsdale, Ariz.-based trash processor Allied Waste Industries; Newark, Del.-based high-tech fabric maker W.L. Gore & Associates; and industrial supplies distributor W.W. Grainger of Lake Forest, Ill., have all reported serious difficulties.
And if "serious difficulties" sounds bad, rest assured it can get much, much worse. After Carrollton, Texas-based pharmaceutical distributor FoxMeyer Drug actually collapsed following an SAP R/3 implementation, its bankruptcy trustees filed a $500 million lawsuit in 1998 against the German ERP giant, and another $500 million suit against co-implementer Andersen Consulting. (Both cases were unresolved at the time of writing.)
So what's going on? The good news-if that's the right word-is that most experts agree that such failures are not systemic. "Very rarely are there instances when it's the ERP system itself-the actual software-that fails," says Jim Shepherd, senior vice president of research at Boston-based AMR Research. Public pronouncements by both SAP and Hershey, he notes, have acknowledged that the software does what it is supposed to and that no big fixes or patches are planned. What's more, he adds, the prudent observer will differentiate between real implementation failures and not-so-real failures. "Blaming the failure on a system implementation has become a convenient excuse for companies that have missed their quarter-end [earnings] target."
As for blame, it is evenly spread. SAP implementations are no more likely to go down the tubes than ERP systems from other vendors: W.L. Gore's system, for example, came from Pleasanton, Calif.-based PeopleSoft. "When an ERP project unravels, or is seen not to perform well, one of the suppliers is usually chosen as the culprit," says David Duray, London-based global partner responsible for the SAP implementation business at PricewaterhouseCoopers. "In my experience, this is usually more of a political decision than a proper problem-source identification exercise-and SAP, over the last few years, has been a popular target."
Furthermore, adds Roger Phillips, an IT analyst at specialist investment bank Granville in London, which tracks the global ERP market, there is no evidence that geography is a significant differentiator in the success stakes. Disasters, he believes, "simply go with the ERP territory." There are, he says, "no cultural or managerial foibles that make American ERP implementations any more predisposed to disasters than any other country's implementations."
So what does lie behind ERP disasters? And behind the rather longer list of costly-but-underwhelming implementations typified by that now-infamous Meta Group report? Increasingly, experts reckon that they've found the smoking gun: poor training. Not the technical training of the core team of people who are installing the software, but the education of the broad user community of managers and employees who are supposed to actually run the business with it.
|
|
|
The Hidden Costs of ERP
Although different companies will find different land mines in the budgeting process, those who have implemented ERP packages agree that certain costs are more commonly overlooked or underestimated than others. Armed with insights from across the business, ERP pros vote the following areas as most likely to result in budget overrun.1. Training
Training is the near-unanimous choice of experienced ERP implementers as the most elusive budget item. It's not so much that this cost is completely overlooked as it is consistently underestimated. Training expenses are high because workers almost invariably have to learn a new set of processes, not just a new software interface.2. Integration and Testing
Testing the links between ERP packages and other corporate software links that have to be built on a case-by-case basis is another often underestimated cost. A typical manufacturing company may have add-on applications for logistics, tax, production planning and bar coding. If this laundry list also includes customization of the core ERP package, expect the cost of integrating, testing and maintaining the system to skyrocket.As with training, testing ERP integration has to be done from a process-oriented perspective. Instead of plugging in dummy data and moving it from one application to the next, veterans recommend running a real purchase order through the system, from order entry through shipping and receipt of payment-the whole order-to-cash banana-preferably with the participation of the employees who will eventually do those jobs.
3. Data conversion
It costs money to move corporate information, such as customer and supplier records, product design data and the like, from old systems to new ERP homes. Although few CIOs will admit it, most data in most legacy systems is of little use. Companies often deny their data is dirty until they actually have to move it to the new client/server setups that popular ERP packages require. Consequently, those companies are more likely to underestimate the cost of the move. But even clean data may demand some overhaul to match process modifications necessitated-or inspired-by the ERP implementation.4. Data analysis
Often, the data from the ERP system must be combined with data from external systems for analysis purposes. Users with heavy analysis needs should include the cost of a data warehouse in the ERP budget-and they should expect to do quite a bit of work to make it run smoothly. Users are in a pickle here: Refreshing all the ERP data in a big corporate data warehouse daily is difficult, and ERP systems do a poor job of indicating which information has changed from day to day, making selective warehouse updates tough. One expensive solution is custom programming. The upshot is that the wise will check all their data analysis needs before signing off on the budget.5. Consultants Ad Infinitum
When users fail to plan for disengagement, consulting fees run wild. To avoid this, companies should identify objectives for which its consulting partners must aim when training internal staff. Include metrics in the consultants' contract; for example, a specific number of the user company's staff should be able to pass a project-management leadership test-similar to what Big Five consultants have to pass to lead an ERP engagement.6. Replacing Your Best and Brightest
It is accepted wisdom that ERP success depends on staffing the project with the best and brightest from the business and IS. The software is too complex and the business changes too dramatic to trust the project to just anyone. The bad news is, a company must be prepared to replace many of those people when the project is over. Though the ERP market is not as hot as it once was, consulting firms and other companies that have lost their best people will be hounding yours with higher salaries and bonus offers than you can afford-or that your HR policies permit. Huddle with HR early on to develop a retention bonus program and to create new salary strata for ERP veterans. If you let them go, you'll wind up hiring them-or someone like them-back as consultants for twice what you paid them in salaries.7. Implementation Teams Can Never Stop
Most companies intend to treat their ERP implementations as they would any other software project. Once the software is installed, they figure, the team will be scuttled and everyone will go back to his or her day job. But after ERP, you can't go home again. You're too valuable. Because they have worked intimately with ERP, they know more about the sales process than the salespeople do and more about the manufacturing process than the manufacturing people do. Companies can't afford to send their project people back into the business because there's so much to do after the ERP software is installed. Just writing reports to pull information out of the new ERP system will keep the project team busy for a year at least. And it is in analysis-and, one hopes, insight-that companies make their money back on an ERP implementation. Unfortunately, few IS departments plan for the frenzy of post-ERP installation activity, and fewer still build it into their budgets when they start their ERP projects. Many are forced to beg for more money and staff immediately after the go-live date, long before the ERP project has demonstrated any benefit.8. Waiting for ROI
One of the most misleading legacies of traditional software project management is that the company expects to gain value from the application as soon as it is installed; the project team expects a break, and maybe a pat on the back. Neither expectation applies to ERP. Most don't reveal their value until after companies have had them running for some time and can concentrate on making improvements in the business processes that are affected by the system. And the project team is not going to be rewarded until their efforts pay off.9. Post-ERP Depression
ERP systems often wreak cause havoc in the companies that install them. In a recent Deloitte Consulting survey of 64 Fortune 500 companies, one in four admitted that they suffered a drop in performance when their ERP systems went live. The true percentage is undoubtedly much higher. The most common reason for the performance problems is that everything looks and works differently from the way it did before. When people can't do their jobs in the familiar way and haven't yet mastered the new way, they panic, and the business goes into spasms.
CIO - Enterprise Resource Planning Research Center
http://www.cio.com/research/erp/
The front page of CIO.com's ERP Research Center not only supplies the standard articles and resources. You will also find ERP related metrics, Q&A's and a feature known as CIO Radio, which consist of online interviews with industry experts.ERPCentral
http://www.erpcentral.com/
This portal to ERP issues links to trade publication articles as well as B2B, CRM, Wireless and e-Business information.ERP Fan Club
http://www.erpfans.com
Yes, ERP does have its own fan-club site on the Web, with links to news, vendor information and the like.The IT Industry Portal - ERP
http://www.erphub.com
Even though this site is a subsection of EarthWeb, another IT industry portal, non-IT executives will still find useful feature articles and case studies on topics like ERP implementation strategies, infrastructure and system performance.IT Toolbox - ERP
http://www.erpassist.com
At this Yahoo-style portal, extensive links to ERP books, white papers, articles and other resources are organized by vendor and by broad topic themes.A Three-Step Assessment
http://www.darwinmag.com/read/090103/value.html
Two crucial elements are often missing from a solid foundation for your investment analysis. We fill in the gaps.Finding the Strategy Gaps
http://www.darwinmag.com/read/060103/strategygap.html
To get from where your business is to where it wants to go
|
|
|
Following is an article from today's East Bay Express that sheds additional light on the acquisition of the City's SAP system and the ongoing challenges of making it perform. The article can also be found at http://www.eastbayexpress.com/issues/2004-06-16/bottomfeeder.html.From eastbayexpress.com
Originally published by East Bay Express Jun 16, 2004
©2004 New Times, Inc. All rights reserved.System Failure
Richmond finance officials compiled most of this year's budget by hand, even though the city has spent $4.5 million on software designed to make such things more efficient.
BY WILL HARPERIn April 2000, the city of Richmond began installing a state-of-the-art business software package that city leaders claimed would fulfill their technology needs and make local government function more efficiently. Top city officials have since touted the software's virtues to other public agencies and in the promotional materials of its computer consultant.
SAP, the software manufacturer, even flew former City Manager Isiah Turner to a users' conference in Orlando to deliver a positive PowerPoint presentation in which he described Richmond's experience as "a true story, with a happy ending."
But four years and about $4.5 million into the scheduled seven-year transition, the story is neither complete nor looking particularly happy. With Richmond now in the midst of a widely publicized financial crisis, some city leaders are now pointing fingers at SAP's wares as a costly failure. Frustrated department heads complain that the fancy new system -- originally designed to meet the needs of the business world -- doesn't work and should be upgraded or replaced. Instead of making things run more efficiently, at least two department heads say, the system has actually created extra work.
Finance director Pat Samsell, for instance, says his people have worked hundreds of extra hours preparing this year's 893-page draft budget because they were forced to do much of the work manually. All the extra prep time, he says, has limited his ability to do fiscal projections and analysis. Considering that Richmond has recently found itself grappling with a "surprise" $35 million deficit, this is a city that needs as much financial analysis as possible to prevent future surprises. The ostensibly sophisticated SAP system is "not intuitive, nor is it user-friendly," he says.
Meanwhile, planning director Barry Cromartie grouses that he can't use the system to perform "cost recovery," in which his planners bill applicants on an hourly basis for their project-related work. Cromartie says he needs to track those work hours to show his bosses on the city council that his department is generating enough money to cover his staff -- and thus protect them from layoffs amid the current budget crunch. The fed-up director created a spreadsheet last year with Microsoft Excel, using a basic consumer program to do the job rather than fuss with the troublesome, multimillion-dollar SAP system. Cromartie says he's been promised an operational billing and revenue-tracking component for two years. "As of this day, we do not have a workable system," he says.
City leaders began looking for a new software system at the height of the dot-com boom in 1999, when everyone and his grandmother was hyping this so-called systems integration software. Information technology director Sue Hartman recalls that the previous setup was so old that no one offered tech support for it anymore. After a competitive process in which Oracle and PeopleSoft also submitted proposals, Hartman says the selection committee picked SAP, with Denver-based consulting firm Solbourne to do the implementation.
In June 2002, more than a year after the new rig went live, SAP flew Turner to Orlando to give a presentation at its annual Sapphire conference. He was accompanied by Hartman, whose expenses were paid by the city. She provided Feeder with a video of Turner's 26-minute presentation, in which he described both the benefits and challenges of the SAP system. Turner boasted that the software-conversion project had come in on budget and without any political fallout. "We think thus far it's a happy ending," he told the audience. Also present, Hartman recalls, were Department of Defense representatives in the market for similar software. They later recruited Turner to give a presentation on SAP for their bosses in DC.
Turner, who retired at the end of last year, wasn't the only Richmond heavy giving the system premature kudos. Solbourne, the contractor hired to make it all work properly, listed Richmond as a "success story" in its literature, and quoted ex-finance director Anna Vega crediting the company for its contribution "to the success of this project." In July 2001, IT director Hartman bragged to a trade publication about how the then-new system empowered its users.
So what does Hartman think of it now? She defends it vigorously and insists it's a clear improvement over what the city had before. "We're functioning. We issue payroll checks, we pay vendors," she insists. "This is not a failed system." She knows there have been complaints from city staff, but says that happens with any major systems change, as people get accustomed to the new technology.
Another municipality still getting used to its new SAP software is the city of Tacoma, Washington. A couple years ago, Tacoma officials came to Richmond and received a presentation on the system from Hartman and others. They must have liked what they heard -- Tacoma has since spent $50 million for its own SAP system, assistant finance director Joe Delaney says. It went online eight months ago and the bugs are still being worked out. "It was a difficult transition due to the complexity of the system," he says.
Miffed Richmond leaders say a key reason for their problems is that the software the city bought was first designed for the corporate world. At the time of the sale, SAP had only recently gotten into the public sector, so programmers and consultants had to rejigger the product to make it work in a government setting. Finance chief Samsell says a midsize city like Richmond (pop. 100,000) could have bought a cheaper and more effective system from a smaller firm that specialized in government software. Over the next few months, Samsell says, he will shop around to see whether it would be cheaper to buy a new system or upgrade to SAP's new government-specific software.
Meanwhile, the city council has directed Everett Jenkins, the acting city attorney, to explore the possibility of a lawsuit against SAP and the implementation consultants who sold them a system tailored for the wrong market. "The question is," Councilman Gary Bell says, "were we used as guinea pigs?"
(December 1999)
Duing the last three months of 1999 a number of companies were reported as having operational difficulties associated with new ERP systems. In some cases the companies openly blamed SAP (and other ERP vendors), and one company has even filed a lawsuit against PeopleSoft and their implementors. Here's our take on the story:
Hershey's – Just prior to Halloween 1999, Hershey's reported a 19% drop in 3rd quarter net earnings, and placed part of the blame on "computer problems". The problems facing the maker of Hershey's chocolate bars and Kisses, Reese's Peanut Butter Cups and Kit Kats could continue through the high margin Christmas season and on through to Easter 2000. Analysts have estimated that the loss in sales could be $100 in the fourth quarter alone, and that they could wind up losing 0.5% market share in the US. According to Hershey's, the problems lie within their new order-and distribution system that uses software from both SAP and Siebel Systems. Since the system went live in July 1999, Hershey's have been unable to fill orders 100% and get their products onto the shelves. SAP have been working with Hershey's to fix the problems, and - as far as we know - Hershey's have not actually blamed anyone for their troubles.
Whirlpool – In early November 1999, the Wall Street Journal ran a story about Whirlpool blaming delays in shipping it's appliances in part to it's SAP implementation which went live two months prior. Apparently, orders for quantities smaller than one truckload had faced snags in the areas of order processing, tracking, and invoicing. CNET have reported that SAP "gave Whirlpool the red light" twice prior to their go-live, saying that their supply chain was not ready.
Other companies (including Allied Waste) have joined in the chorus, but with little detail.
One company – W.L. Gore and Associates have even filed a lawsuit. Bloomberg News reported at the end of October that the makers of Gore-Tex (a waterproof, Teflon-based fiber used in outdoor wear such as Timberland boots) had filed suit against PeopleSoft and Deloitte & Touche over an allegedly botched attempt to install PeopleSoft's HR module. Gore apparently ended up paying D&T twice D&T's original estimate, and in the end had to bring in another implementor to "re-implement" the system. PeopleSoft are being sued as well because – it seems – they recommended D&T. According to the report, "Gore seeks compensation in the millions of dollars for damages it suffered because of PeopleSoft's and D&T's scheme to defraud and failure to perform as promised". Harsh words indeed.
This is all on top of the still unresolved $500 million lawsuit filed by the bankruptcy trustee for FoxMeyer (a former $5 billion drug distribution company) against Andersen Consulting, "alleging the company's botched implementation of SAP's R/3 software helped send FoxMeyer into bankruptcy in 1996." It is interesting to note that, according to CNET, at the time Andersen called the claims "outlandish and totally at odds with facts".
So. What does this all mean? Who is at fault? How many more are going to come out of the woodwork? Is this Y2K related? What can be done to prevent these problems in the future? So many questions …
Of course no one has all the answers, but we would suggest that it is critical for customers, consultants and SAP (and other ERP vendors) to hold open, honest dialogue at the start of the project, and nail down the critical success factors of the implementation, including:
- The customer's expectations
- SAP product capabilities, and gaps
- The level of change the customer has to go through to make the system fit
- The level of commitment within the customer organization to see the project through to completion
- The customers organization and culture, and the fit between the customers organization and culture and the project organization and culture
- The risks presented by politics within the customer organization, and
- The consultant's capabilities, responsibilities and role (if applicable)
|
|
|
CIO Magazine
Project Management | AT&T Wireless AT&T Wireless Self-Destructs
The story of a botched CRM upgrade that cost the telco thousands of new customers and an estimated $100 million in lost revenue. Hard lessons learned. Executive Summary Last fall, AT&T Wireless frantically tried to complete a CRM upgrade to Siebel 7. It had to be done in time to handle the customer service challenges accompanying a Federal Communications Commission deadline for allowing customers to change carriers without changing their phone numbers. The effort was a failure. Systems crashed and stayed down. Customer reps could not keep up, and angry customers abandoned the carrier. The weakened company was ultimately bought out by rival Cingular. Through exclusive interviews with company insiders, we piece together what went wrong and highlight the key project management lessons, including the pitfalls of poor planning and the deleterious effects of an ill-timed outsourcing push.
Too bad AT&T Wireless couldn't get it right. Even the $3 trillion worldwide construction industry is adopting IT to manage its immensely intricate projects. And a small publisher in Salt Lake City could have taught the erstwhile wireless pro about project prioritization. We covered both recently: Cures for Complexity and First Things First, respectively.
JACK LEE MARKEDNov. 24, 2003, on his calendar because that was the day he would finally be able to change his cell phone carrier without losing his phone number-thanks to a Federal Communications Commission ruling. But Lee, president of Tangara Technologies, a company that develops software for forms, decided to wait a day before switching to AT&T Wireless to let the chaos of "number porting" die down a little. Little did he know that the chaos was just beginning.
Lee ordered his new phone on Nov. 25. When he went to the AT&T Wireless website to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold-with the line often going dead when AT&T Wireless's customer service lines became overloaded.
After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers.
The breakdown couldn't have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $100 million in lost revenue. But that wasn't all. The failure so damaged AT&T Wireless's reputation that many analysts believe it hastened its sale to Cingular in February for $41 billion, or $15 per share, which was just under half the value of AT&T Wireless's shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do," the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company.
AT&T Wireless's mistakes offer valuable lessons for CIOs. For one, it's unwise to freight major system upgrades with external complications. AT&T Wireless's CRM upgrade was hamstrung from almost the very beginning by rumors of outsourcing deals and future layoffs. These rumors generated pervasive morale problems that hurt the productivity of project staff.
Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline-the federally mandated Nov. 24 number portability date.
Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time.
Playing Catch-Up with the Competition
When AT&T Wireless began its Siebel CRM system upgrade in 2003, it was a company that had slipped from unquestioned market leader to middle of the pack. Its overall market share had slid from an industry-leading 25 percent at the end of 2001 to 17 percent in 2003, third behind Verizon and Cingular.Worse, AT&T Wireless was playing catch-up on its most important technology asset, its phone network. One of the older wireless companies, AT&T Wireless made an early bet on a technology called TDMA that could not handle data transfer over cell phones-the next big thing for business customers. Even before AT&T Wireless was spun out from its parent AT&T in 2001, it had begun a furious buildout of an expensive new network-global system for mobile communications, or GSM-that could not only handle data but had the added advantage of global compatibility with overseas providers. Only one other major carrier, Cingular, was saddled with the challenge of building out a new GSM network while still servicing the old one. The other carriers had all chosen network technologies that could handle data transfers.
GSM was a great opportunity for AT&T Wireless, but it was also a huge CRM challenge. The company had to convince its old customers to move off TDMA, which worked as well as most other carriers' networks for voice calls, and onto GSM, which had poorer voice quality, according to Morgan Stanley. It also had to convince new customers that GSM was the wave of the future, that they would soon be shipping data over their phones instead of their laptops. But customers didn't buy the pitch. By 2003, AT&T Wireless's percentage of customers on GSM was hovering at 15 percent, according to analysts, while Cingular had 35 percent of its customers moved over (thanks, in part, to its acquisition of a cellular provider with an existing GSM network). And things weren't improving. In the third quarter of 2003, less than half of AT&T Wireless's new customers were choosing GSM, while Cingular was signing up 75 percent, according to Morgan Stanley.
Everyone at AT&T Wireless agreed that the company would keep its existing customers and add more new ones if its customer representatives could handle more calls and get customers up and running faster. Customer service representatives needed about 20 minutes, on average, to work through five or six screens that fetched information from about 15 legacy systems, say former employees. Slow access to customer information records was just one of the reasons why AT&T Wireless possessed the second-highest cost per subscriber (behind Sprint PCS) of the top six national carriers in 2003, according to financial services company UBS.
Why AT&T Needed to Upgrade
AT&T Wireless installed Siebel's CRM system in 2001 to be the front end of its customer service process. The back end, however was a complex mishmash of systems, say former employees. Telco billing systems, for example, were stuffed full of different rate plans and arcane metering processes. Systems that tracked calls and set up new phone numbers (provisioning) communicated with hundreds of thousands of different telephone switches around the country and the world. To work for AT&T Wireless, Siebel's version 6 had to be highly customized. Though the software came with integration tools, consultants usually resorted to writing point-to-point scripts to hook the systems together. Policing the overall integration in a scenario like this is difficult at best. Indeed, a former AT&T Wireless employee who worked on the project recalls the test system crashing and remaining down for six weeks during the summer of 2002 when AT&T Wireless began preparing Siebel version 6 to deal with number portability. And when Siebel 6 was finally up and running, it still couldn't handle all the information that customer service representatives needed.The next release of Siebel, version 7, was much more powerful. Siebel developed an industry-specific version of the new software that had many more features and could capture more telco-specific information. And new Web capabilities meant that AT&T Wireless could potentially reduce the number of customer service screens to one by building a Web portal that put all the different systems and information sources together in one place. "That was the point of the upgrade," says a former employee. "They wanted to get everything on one screen so they could sign up customers faster."
Marc Siegel, a spokesman for AT&T Wireless, agrees:"As we needed to handle more transactions, more customers, we needed a more robust system. So we upgraded. [The upgrade] was going to make it easier for our representatives to get access to information and give them a fuller array of information on the customer."
In the spring of 2003, the company decided to upgrade to version 7.5 for the roughly 3 million customers on the GSM and general packet radio service (the data portion of the new network). TDMA customers would continue to be serviced through AT&T Wireless's legacy Axys CRM system until they could be moved over to GSM and the new Siebel system. The project was called Odyssey.
The Upgrade Begins, Badly
Though the upgrade promised gains in speed and simplicity, getting there was neither speedy nor simple. AT&T Wireless's IT staff had to rip apart old links from the different legacy systems to the client/server-based Siebel 6 system and rewrite them to work over the Web. The back-end systems-integration work was so complex that it wasn't unusual to see teams of 20 or more people assigned to write connections for a single system, says a former employee. Coordination between the teams-the responsibility of the lead integrator on the project, Deloitte and Touche-quickly got out of hand. (Deloitte and Touche did not respond to repeated requests for an interview.)"Everything was siloed among the different groups, and we all worked independently of each other," says a project team member. Teams would work on a revision to their piece of Odyssey, for example, only to find that when they finished testing, code had changed elsewhere in the system, rendering the testing meaningless. "In other projects I've worked on, the project managers would freeze code while the teams did revisions to their pieces so you could test everything against the same code base," he says. "I didn't hear of that happening on this project."
Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."
Meet the New Boss
On April 10, 2003, Mike Benson, a 15-year AT&T veteran, sent a memo to employees announcing that he was leaving after three years as CIO. AT&T Wireless's Siegel says the 48-year-old Benson "decided to retire." Christopher Corrado, the head of the security solutions practice for Wipro, an Indian offshore outsourcing company, was named executive vice president and CIO. Before joining AT&T Wireless, Corrado had been CTO at two divisions of Merrill Lynch, where he presided over the offshore outsourcing of portions of Merrill Lynch's IT. Employees speculated that it was just a matter of time before offshoring began at AT&T Wireless.
"Corrado was the hatchet man-everyone knew it," says a former employee. "We'd see our people going into these long meetings with people from Indian companies. We'd see whiteboards that had questions like, 'What opportunity do we have to offshore/outsource?'"
CIO Christopher Corrado never directly addressed rumors that AT&T Wireless planned to outsource much of the support and maintenance for its CRM system.
Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee. "We all came away saying, 'Who is this arrogant jerk?'" Corrado could not be reached for comment.
The Deadline Looms
As November approached, AT&T Wireless was one of the last industry holdouts opposing the Federal Communications Commission's new rule on number portability, along with Cingular and Alltel. The industry had succeeded in delaying the change since 1998 on legal technicalities, and AT&T Wireless was counting on another postponement, say industry sources. But on Halloween, the U.S. Court of Appeals for the District of Columbia Circuit rejected the appeal to move back the FCC's Nov. 24 deadline. An AT&T Wireless spokesman told Bloomberg News that the company was ready for porting.It wasn't.
Things were going so badly with the Siebel CRM upgrade that there was talk on the project of trying to roll back to the old Siebel 6 system before the porting date. "Two weeks before launch there was rollback talk," a former employee says. But project managers hadn't preserved enough of the old system to make it feasible. There was no plan B. Project members estimated that the team needed two or three more months of bug fixing and testing to get the system running reliably.
And the Siebel system wasn't the only thing AT&T Wireless needed to get working before the deadline. Wireless number porting required new systems that had to be integrated with the different carriers' CRM systems. Five of the six top providers outsourced the administration of the number changes to TSI Communications and used software built by Telcordia. But in June 2002 (before the other carriers had announced they were going with TSI), AT&T Wireless chose its longtime software and administration provider, NeuStar. Ultimately, this would create serious interoperability problems for AT&T Wireless.
AT&T Wireless defends its decision by saying that NeuStar was the most experienced provider in the market, having worked on landline portability in the early '80s. It also says NeuStar's software offering was more complete than TSI's at the time AT&T made its decision.
No Treat, Just Trick
On Halloween evening, the project team tried to bring up the new Siebel system for the first time. It crashed. "It went down and stayed down," says a former employee. "They couldn't get it working again." Project staffers were brought in to work on the system for three straight days to try to rescue it, with no luck, say former employees. The system was unstable.Though observers could not trace specific causes for the crash, a few factors played a role. As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system. Rather than freeze the code for the system once problems began, teams continued to add new pieces to the project in an attempt to get it all working. But the new pieces simply added more errors to an already bug-ridden system, which complicated the process of finding root problems and fixing them.
"[Integration was] far more technical and far more difficult to test in advance and also fix problems once they appeared," acknowledged Michael Keith, president of AT&T Wireless Mobility Operations, in a conference call on the company's fourth quarter results.
Compounding these difficulties was the hit employee morale had taken on Nov. 16, when AT&T Wireless President and CEO John Zeglis confirmed the layoff rumors. Zeglis told analysts in a third quarter conference call that the company would lay off 1,900 workers. He did not say where the cuts would come or when. But IT managers started telling employees that layoffs would begin in February 2004, according to former employees.
While employees struggled to fix glitches in the CRM upgrade, CEO John Zeglis confirmed the layoff rumors, saying AT&T Wireless would soon lay off 1,900 workers.
Some AT&T Wireless IT employees had gotten hold of a report prepared by Tata Consulting Services of India outlining plans for offshoring pieces of AT&T Wireless's IT infrastructure. The report, obtained by CIO, also lays out plans for offshoring a portion of the support and maintenance for the new Siebel system. "You had to wonder why you were working so hard if the whole thing was going to go away anyway," says a former project employee.
On Nov. 19, The Wall Street Journal ran a story on planned layoffs and outsourcing at AT&T Wireless, and CIO Corrado responded in a memo to the staff: "We are in the awkward position of having reports in the press about a contract with HP before we have communicated with the employees who will be affected. We have just today signed a letter of intent with HP to deliver the services provided by the following teams: Desktop Services; Retail Field Services; Business Office WAN/Telecom; MSI Packaging; Service Desk-password reset and desktop calls." But as for bigger questions, namely that support and maintenance for the Siebel system would be outsourced to offshore provider Wipro and Tata, Corrado did not address the rumors directly. Instead, he concluded: "We currently outsource work throughout the company, including work within both Customer Services and IT. We will continue evaluating the best mix of internal and external resources as we work to achieve best-in-class margins."
In fact, AT&T Wireless was planning to move overseas more than 3,000 positions in its computer operations and customer service.
The Second Crash
The Odyssey team was still struggling to get the Siebel system up and keep it up when, on Nov. 24, customers began trying to port their numbers between carriers. That's when the software for handling number portability crashed.TSI and NeuStar had arrived at "differing interpretations" of the industry's data standard for exchanging and verifying porting requests, known as Wireless Intercarrier Communications Interface Specifications (WICIS). The Yankee Group's Entner says that TSI modified its version of the WICIS standard without telling NeuStar. "They made changes in fields and reporting that didn't impact all their five customers but no longer made it possible to communicate with [NeuStar]," says Entner. TSI's Michael O'Brien, vice president of marketing, denies that changes were made without telling NeuStar but acknowledges that the two vendors had different interpretations of WICIS. (NeuStar declined to be interviewed for this story.) AT&T Wireless says that TSI crashes prior to the deadline prevented NeuStar from being able to test the process adequately, and on Nov. 24, the electronic flow of porting requests between NeuStar and TSI broke down right away.
The industry had estimated that automated ports would require 2.5 hours to complete, so the porting messages were programmed with time and date limits. If the other carrier's system didn't respond within the allotted time, the port that failed was supposed to drop into an electronic bin to be handled manually.
NeuStar's software could not respond to TSI's systems before time ran out. And the error messages weren't all getting through either. When AT&T Wireless's Porting Administration Group tried to track and resolve errors using a workflow management tool, it crashed or froze. That meant AT&T Wireless customer service representatives were in the dark when their phones lit up. Combined with their continued problems accessing Siebel, there was little they could do to help their customers. In that first week of porting, more than half of the complaints filed with the FCC singled out AT&T Wireless, according to The Associated Press. The FCC sent a letter to AT&T Wireless on Dec. 4, asking for an explanation, triggering a torrent of bad publicity.
Five Lessons from AT&T Wireless's Project Failure It did a lot wrong. Here's what you can do right. "There were 50,000 [AT&T] customers per week that didn't have service because of this," says Phil Cusick, a telecom industry analyst with Bear Stearns. And many of them, like independent consultant Ian Drake, didn't stick around for a fix. After three weeks of trying to track his newly purchased AT&T Wireless phone, Drake gave up and signed on with Verizon.
A Company Fire sale
As word of AT&T Wireless's difficulties spread, customers began deserting at a rapid clip. The problems continued through December, according to AT&T Wireless President and CEO Zeglis. "It took most of December to expand the systems capacity so that the care reps could have as much access as they needed in order to handle customer calls at the same time the salespeople were processing new orders," he said in a conference call with analysts on Jan. 22. "Frankly, our customer service went pretty far south."For independent cell phone retailers, which account for about 60 percent of new sales in the industry, there was no choice. "The indirect sellers pushed other brands when they heard that AT&T was having problems," says Greg Teets, telecom analyst for A.G. Edwards. AT&T Wireless was forced to bump up commissions to independents and lower prices to try to stem the tide. It didn't work. After adding 229,000 customers in the third quarter of 2003, fifth out of the top six providers, AT&T Wireless added just 128,000 in the fourth quarter, last among the top providers and less than one-tenth the number of industry leader Verizon, which added 1.5 million new subscribers.
By January, AT&T Wireless was in the advanced stages of plans to move overseas more than 3,000 positions in its computer operations and customer service, according to The Wall Street Journal. (It backed off from the plan only after agreeing to be purchased.) Some employees became part of AT&T Wireless's "buddy program," in which consultants from Indian outsourcing companies Tata Consultancy Services and Wipro were assigned to AT&T Wireless employees to learn their jobs.
In February, 220 AT&T Wireless IT employees were told that they would be released from the company in March. According to one of the terminated employees, another round of 250 layoffs is planned for June and the same number for September. "It's tough," an employee said in February. The Indian workers "basically follow people around all day and pepper them with questions." Other former employees say some staffers resisted helping the consultants. "People would make project decisions when the Indians weren't around," a former employee says.
On Feb. 17, Cingular announced that it was buying AT&T Wireless.
That same day, Deborah Sawyer's voice-mail box began filling up with messages. Sawyer, a partner for executive search firm Morgan Howard Worldwide, says she received inquiries from 12 different AT&T Wireless IT executives looking to move on.
But for their company, the odyssey was over.
Executive Editor Christopher Koch can be reached at [email protected].
|
|
|
New submitter yeshuawatso writes I work for one of the largest HVAC manufacturers in the world. We've currently spent millions of dollars investing in an ERP system from Oracle (via a third-party implementor and distributor) that handles most of our global operations, but it's been a great ordeal getting the thing to work for us across SBUs and even departments without having to constantly go back to the third-party, whom have their hands out asking for more money. What we've also discovered is that the ERP system is being used for inputting and retrieving data but not for managing the data.Managing the data is being handled by systems of spreadsheets and access databases wrought with macros to turn them into functional applications. I'm asking you wise and experienced readers on your take if it's a better idea to continue to hire our third-party to convert these applications into the ERP system or hire internal developers to convert these applications to more scalable and practical applications that interface with the ERP (via API of choice)?
We have a ton of spare capacity in data centers that formerly housed mainframes and local servers that now mostly run local Exchange and domain servers. We've consolidated these data centers into our co-location in Atlanta but the old data centers are still running, just empty.
We definitely have the space to run commodity servers for an OpenStack, Eucalyptus, or some other private/hybrid cloud solution, but would this be counter productive to the goal of standardizing processes. Our CIO wants to dump everything into the ERP (creating a single point of failure to me) but our accountants are having a tough time chewing the additional costs of re-doing every departmental application. What are your experiences with such implementations?
Computerworld
Companies Don't Learn From Previous IT Snafus
Hazy goals, culture of hiding problems lead to more multimillion-dollar disasters
OCTOBER 30, 2000
(COMPUTERWORLD) - Computer projects have failed for as long as there have been computers. But now that most companies are only as stable as their bits and bytes, the consequences of information technology screwups aren't easily disguised - they show up in earnings reports.When IT goes bad, high-growth rocket ships like Oxford Health Plans Inc. and Ben & Jerry's Homemade Inc. report their first-ever financial losses. Others crater and run for bankruptcy protection, as did drug distribution giant FoxMeyer Corp.
In a Computerworld study of multimillion-dollar IT disasters, the following two not-so-surprising themes emerged:
- User companies like FoxMeyer often file tough-to-win lawsuits against the vendors or consultants involved. Nonetheless, collectively, users rarely seem to learn much from the episodes or apply the lessons to future projects.
- All of the botched projects in Computerworld's Top 10 disasters list were big and richly complex; many were the toughest IT projects the users had ever tried. Five were hideously difficult enterprise resources planning (ERP) system implementations.
Root Causes Remain the Same
The root causes of IT failures haven't changed a bit over the years.
Top 10 Failures See Top 10 Corporate Information Technology Failures in a PDF chart. (Requires Adobe Acrobat Reader)
Miscommunication, hazy goals, "scope creep," inept leadership, pitiful project management - you've heard, if not heeded, it all before.
"We may be neck-deep in the New Economy and Internet time, but you still have the same factors and the same failings," said Bruce Webster, a director at PricewaterhouseCoopers in Washington.
Webster recently studied 120 IT lawsuits filed since 1976, and he said he's convinced that most flops could be avoided if people simply knew the time-honored best practices of systems development.
Dishonorable Mentions Some information technology disasters of the past decade aren't included in the Top 10 chart (at right) because their financial costs weren't clear or weren't quite as high as those that made the list. Others were quasigovernmental. But they were significant bungles nonetheless. Here's a sampling:
∙ High hopes for a high-tech airport in Denver were dashed in 1994 when a baggage-sorting system misplaced and damaged countless suitcases. The object-oriented system, which was built by BAE Automated Systems Inc. in Carrollton, Texas, ran on IBM's OS/2.
To fix it, primary sponsors Chicago-based United Air Lines Inc. and the city of Denver ended up paying at least $86 million more than the original $193 million price tag. The airport opened almost three years late.
∙ Ben & Jerry's Homemade Inc., a small Vermont ice cream maker that hit the big time in the early 1990s, took a $4.1 million write-off in 1995 when it canceled a project to build a fully automated packing plant. As a result, the company posted its first quarterly loss ever.
∙ Early this year, Thomas & Betts Corp., a $2.5 billion electrical parts maker in Memphis, blamed problems with a new Internet-based order-management system for a 50% drop in profits in the fourth quarter of last year.
The homegrown mainframe system also cost the company another $42 million in order and shipping disruptions. In the following quarter, the company spent $11 million on customer service, extra freight costs and other measures to make up for ongoing system crashes and backlogs.
In a lawsuit still pending, shareholders say Thomas & Betts misled them about the success of the new system.
∙ Just in time for the Memorial Day holiday crush in 1998, Avis Inc.'s Wizcom reservation system crashed. The outage blocked the rental car company, as well as many travel agencies and hotels also linked to the system, from taking bookings for 30 hours.
∙ In the summer of 1994, an error in a routine file update crashed the automated teller machine (ATM) system of Chemical Banking Corp. in New York. ATMs at Chemical, which was then one of the five largest banks in the U.S., were down for five hours.
∙ In 1993, Fidelity Brokerage Services Inc. in Boston started a multimillion-dollar project to build a Windows-based trading application for customers with home PCs. "Jamaica" still wasn't ready by mid-1996, reportedly because of political clashes between quirky programmers and staid investment bankers.
The application wasn't a failure; it worked when it was finally rolled out. But the numerous development delays put Fidelity far behind rivals such as Charles Schwab & Co. in San Francisco.
"I don't know how many IT managers, team leaders, directors and CIOs have actually sat down and read The Mythical Man-Month, The Psychology of Computer Programming and Death March," he said, referring to three books that amount to the software development canon. "The causes of disasters are all well documented. They're fundamental."
Still, warning lights are easy to overlook when the whole room is spinning.
"There's a natural tendency to get overly committed to something, especially when there are no clear signals telling you you are off course," said Mark Keil, an associate professor at Georgia State University in Atlanta.
The infamously buggy baggage-handling system at the Denver International Airport is one case that offered unambiguous proof of technology glitches: shredded luggage.
But tests of most questionable IT projects don't yield such graphic evidence.
In large systems integration or ERP deals, "there's no torn suitcase sitting at your feet to wake you up," said Keil, who has studied IT disasters for nine years. "So it's a lot easier to delude yourself into thinking things aren't that bad."
Project Euthanasia
Euthanasia for the project might be the best course, but people often have too much heart and money invested to end it.
One technique for preventing a disaster is to add some humility to the endeavor. Invite a third party to review your work - a reliable consultant, an academic or a buddy CIO.
An outsider can "walk into the project setting for 20 minutes, talk to a few people and come to the conclusion that things have run amok. But people inside may not even be aware," Keil said.
Greyhound Lines Inc. in Dallas, for example, seemingly didn't know anything was wrong with its new reservations and logistics system - until it went live and 12% of its customers went away mad in one month.
Though specific individuals might learn from their own mistakes, those lessons aren't transferred to any collective IT consciousness.
"The people with that [failure] experience aren't always the people in authority the next time that situation arises," observed Kevin Hickey, a former head of IT at Trumbull, Conn.-based Oxford Health Plans Inc. "The fact is, hubris will always be with us."
And then there's what Webster calls "the thermocline of truth." Swimmers know that lake water separates into warm and cold horizontal bands. The area between is a thermocline.
In IT groups, everyone below Webster's "thermocline of truth" knows the project is sinking, while everyone above it thinks things are fine. Senior executives can be oblivious. They aren't involved enough, they don't want to have to face a failure, or underlings are afraid to tell them, he explained.
"You can see that persist almost until the point where the project is supposed to be delivered," he said. "Then, suddenly, it's, 'What do you mean this will take another six months?' "
That was part of the sorry plight of Fort Worth, Texas-based AMR Corp.'s Confirm reservations system. Confirm managers are even said to have orchestrated a cover-up.
Overall, IT culture is such that problems, especially expensive ones (which hold the most valuable lessons), are hidden. Programmers write around buggy code rather than tear it apart. Managers revise project specifications to reflect what they did instead of what they should have done. Senior IT leaders neglect to tell their bosses the bad news.
Most companies are too embarrassed to analyze their failures, said Effy Oz, an associate professor of management and IT at Pennsylvania State University in Great Valley.
"People will say, 'There's no time, and we're not paid to have these discussions,' " Oz said. "The CEO has to be a very confident person to say, 'These things happen. Let's learn from it.' "
The average loss in an abandoned project is $4.2 million, according to Oz. The blowups in Computerworld's top 10 list cost much more than that. And, if history is any indication, they will happen again.
Travel System Is Confirm-ed Disaster
On the heels of its hugely successful Sabre airline reservation system, Fort Worth, Texas-based AMR Corp., the parent company of American Airlines Inc., in the late 1980s formed a joint venture with Marriott International Inc., Hilton Hotels Corp. and Budget Rent A Car Corp. to build a similar system for the travel industry. But Confirm, as the project was called, wasn't to be. In fact, the effort is viewed as one of the worst IT failures ever for its mismanagement, questionable ethics and unworkable software.
AMR's information systems unit in Dallas was the lead developer on Confirm, which was originally due in June 1992 for no more than $55.7 million. Yet Confirm started to miss deadlines and cost estimates several months after development began.
Specifications were unclear, unskilled programmers were assigned to the project and mainframe-based transaction-processing software couldn't be integrated with a related mainframe decision-support system. Moreover, one year into design and development, Confirm had already fallen one year behind schedule.
Bethesda, Md.-based Marriott and Lisle, Ill.-based Budget started asking questions in 1990 but were assured that Confirm would work and that programmers would make up time and meet the deadline.
In April 1992, just three months before it was slated to go live, Confirm failed tests at Los Angeles-based Hilton. AMR also told its partners in a letter that it needed another 15 to 18 months.
"The individuals whom we gave responsibility for managing Confirm have proven to be inept [and] concealed a number of important technical and performance problems," the AMR letter said.
The legendary Max Hopper, who had been instrumental in Sabre's development, was vice chairman of AMR's IT unit at the time, though not directly involved in the daily work of Confirm. However, he acknowledged in a note to his employees that some Confirm managers "did not disclose the true status of the project in a timely manner. This has created more difficult problems - of both ethics and finance - than would have existed if those people had come forward with accurate information" [News, May 22, 1995]. After consuming almost four years and $125 million, Confirm was effectively dead.
In September 1992, AMR sued Budget, Hilton and Marriott; Marriott then sued AMR. The suits were settled out of court for undisclosed terms. Hopper recently declined to discuss Confirm, citing the secret settlements.
AMR was mainly to blame, but all sides acted unprofessionally, said Effy Oz, an associate professor of management and IT at Pennsylvania State University.
Executives at AMR "simply lied to their client-partners," said Oz, who has studied Confirm. "The partners were irresponsible in not asking more questions more often and in accepting at face value all that [AMR] told them."
Budget, Hilton and Marriott today use their own - separate - reservation systems.
A Really Bad Bet for Drug Distributor
When it launched a $35 million enterprise resource planning (ERP) project way in 1993, FoxMeyer Corp. was a $5 billion drug distributor in Carrollton, Texas.
Now it's bankrupt.
It wasn't the fumbled IT project alone that destroyed FoxMeyer, but that was a critical contributor, according to lawsuits FoxMeyer filed against SAP AG, SAP America Inc. and Chicago-based Andersen Consulting in 1998.
SAP lied repeatedly about R/3's capabilities, and Andersen's inexperienced staff used FoxMeyer as a training camp, claims the drug company, which seeks damages of $500 million from SAP and $500 million from Andersen.
FoxMeyer was one of the first big users to take a chance on ERP, which was a hot new idea at the time. Perhaps the company should have been more cautious. Legal documents show that FoxMeyer knew that SAP's R/3 software hadn't yet been used at distribution companies - just at manufacturers.
Still, FoxMeyer was jazzed. Then-CIO Robert Brown told Computerworld in 1994, "We are betting our company on this."
Big problems started to emerge later that year. For example, the new R/3 software miscounted inventory, which in turn screwed up customer orders. Outright crashes were routine.
SAP declined to talk specifically about FoxMeyer. But an SAP spokesman said users who install R/3 are usually changing basic business processes at the same time, which "is typically where most of the pain and challenges of implementation come from."
FoxMeyer also charges that R/3 performed worse than the company's proprietary Unisys Corp. system. R/3 could process just 10,000 invoice lines per night, compared with 420,000 for the Unisys setup.
SAP misled FoxMeyer with faulty benchmarks, according to the suit. Other users have also questioned SAP's benchmarks [News, Sept. 4, 1995]. SAP doesn't misrepresent benchmarks, the company's spokesman said.
As for Andersen, its people "were neophytes," said Mark Ressler, a lawyer from New York firm Kasowitz, Benson, Torres & Friedman LLP who represents FoxMeyer.
According to FoxMeyer, many Andersen workers were recent college graduates, and others lacked R/3 experience. "There's no better way to overcome that than [by] experimenting on a live patient," Ressler said.
They bungled data conversions by using incorrect drug product codes, for example, and built faulty interfaces between the old and new systems, the lawsuit charges.
As a result, most R/3 modules were rolled out to just six of 23 warehouses by the time the company filed for bankruptcy protection in August 1996.
Andersen didn't respond to requests for comments. The suits are slated for trials next May.
Meanwhile, FoxMeyer itself is being sued by stockholders, in part for allegedly hiding timely information about the computer problems and their effects on business.
High-Flying HMO Modernizes, Crashes
Oxford Health Plans Inc. was the Netscape of health maintenance organizations. It seemed to burst from nowhere, captivate customers and force competitors to change the way they operated.
Then Oxford decided to modernize its information technology.
It was 1995 and Oxford's old turnkey, Pick-based billing and membership tracking system would no longer do. A complete overhaul, using more modern Unix technology, is what the company wanted - and fast.
The project included many custom applications that ran with Oracle databases and other software. But key was a claims processing system, dubbed Pulse, that Oxford's internal IT people built with Oracle tools.
Trouble hit the Trumbull, Conn.-based HMO almost as soon as the rollout started in late 1996. Customers suddenly got claims laden with errors - when they got claims at all. The company paid bills it shouldn't have and denied claims it should have paid.
All the late and inaccurate paperwork caused New York state to fine Oxford $3 million for violating insurance laws.
Overall, the new software overestimated revenue by $392 million for 1997 and 1998 while also underestimating medical costs. That awful combination led to Oxford's $291 million loss in 1997.
Angry doctors and patients abandoned the HMO. Membership dropped 20% from 1997 to 1999, partly because of the systems problems and partly because Oxford withdrew from four of the seven states it did business in.
Ultimately, top executives left, and Oxford hired a new head of operations - Kevin Hickey, then an operations manager at Aetna Inc. - to help with an IT cleanup already under way.
Hickey immediately faced down the billing mess, which "wasn't just an inconvenience. This was a survival issue," he said in an interview.
First, he shelved Pulse and returned to the old Pick application. Pulse was never fully integrated with the Oracle software, he said.
Cambridge Technology Partners Inc. and Diamond Technology Partners Inc. came in for quick-hit assignments to help fix claims processing.
Oxford also shut down its advanced technology unit. Studying artificial intelligence software for possible future systems was frivolous now that "emergency" IT problems threatened to incapacitate the HMO, Hickey explained.
Oxford hired Computer Sciences Corp. to create a plan for outsourcing its entire IT operation.
But in 1998, a new CEO swept in and swept away that idea, along with most remaining legacy executives. Hickey, too, was replaced after just a year at the company.
Today, Oxford is smaller and smarter. It wrote off $5 million for hardware and software in 1998. Late last year, system fixes even took precedence over customer acquisition, according to documents filed with the Securities and Exchange Commission (SEC).
Oxford has since completed major systems fixes and put in place new quality assurance and other testing programs. But SEC documents warn that unexpected sales calculations could still turn up.
Oracle's Rotten Idea
Tri Valley Growers got caught in an Oracle Corp. software scheme that Oracle CEO Larry Ellison later admitted was a bad idea.
In November 1996, the $800 million agriculture cooperative in San Ramon, Calif., bought $6 million worth of services and software from Oracle. The core of the deal was Oracle CPG, ERP software for consumer packaged goods companies.
While the financial and manufacturing pieces of the CPG suite were from Oracle, order processing, production planning and other packages were from other vendors. Oracle consultants were hired to integrate it all.
The new software was expected to make Tri Valley more efficient, improve customer service and save it $5 million a year.
In a press release trumpeting the Tri Valley deal in 1998, for example, Oracle said Tri Valley customers would be able to file a single purchase order, no matter how the cooperative managed its many fruit product lines internally.
But the system never got that far. Oracle couldn't get most of its applications to work with the non-Oracle software.
Tri Valley claims to have spent more than $22 million before halting the project and turning to SAP AG software.
The Oracle software "never worked, has not and cannot be integrated and could not even be installed on Tri Valley's computers," according to a lawsuit the cooperative filed in February.
Last year, Ellison called the Oracle CPG bundle "a huge mistake" [News, April 21, 1999]. Oracle no longer sells it.
In July, Tri Valley filed for bankruptcy protection. So far, it hasn't blamed its poor financial shape on the failed Oracle project - but it might, and it might ask for more money, to boot, said Peter Sipkins, Tri Valley's lawyer, with Dorsey & Whitney LLP in Minneapolis.
Oracle has denied all allegations. Tri Valley IT officials didn't return calls seeking comment. But a former IT manager there called the situation "very tragic."
A trial is scheduled for next June.
FoxMeyer Trustee Sues SAP, Deloitte; $500 Million is Sought From Each Firm
News Summary
August 27, 1998
The Wall Street JournalLawsuits filed by KBT&F on behalf of the bankruptcy trustee for FoxMeyer Drug Company, a drug distributer, against German software manufacturer SAP AG and FoxMeyer's former auditor, Deloitte & Touche LLP are featured in an article in the August 27, 1998 edition of The Wall Street Journal. The lawsuit against SAP alleges that the failure of its software to perform caused FoxMeyer's demise. The complaint against Deloitte & Touche alleges that the firm was negligent in failing to stop a $750 million refinancing of FoxMeyer.
Bankruptcy trustee sues SAP AG By Reuters,
>ZDNN
>August 27, 1998 5:45 AM PT
>
>
>WILMINGTON, Del, Aug 26 (Reuters) - The bankruptcy trustee appointed to
>oversee the liquidation of FoxMeyer Corp. and Foxmeyer Drug Co. has sued the
>companies' software supplier, SAP AG for $500 million for alleged "gross
>negligence."
>
>In papers filed Monday in the U.S. District Court in Delaware and made
>available to Reuters today, trustee Bart Brown said SAP's alleged fraud and
>negligence, "led to the demise of FoxMeyer, a once-thriving $5 billion
>wholesale drug distribution company."
>
>According to the lawsuit, although SAP's had historically made software for
>manufacturing, the Walldorf, Germany-based company, "assured FoxMeyer that
>its R/3 system was well suited to the needs" of the high volume and complex
>price structure of FoxMeyer's distribution business.
>
>Couldn't Handle Volume
>When the R/3 system was installed, its volume limitations made it usable at
>only six of FoxMeyer's 23 distribution warehouses, court papers said.
>
>"The failure of the R/3 system to perform as SAP had represented...was a
>significant factor contributing to Foxmeyer's August 1996 bankruptcy and
>subsequent liquidation."
>
>According to court papers, on September 30, 1993, FoxMeyer agreed to pay
>more
>than $5 million to license SPA's R/3 software system.
>
>Forced Back
>The software's first use was set for early 1995 at six warehouses built by
>FoxMeyer to handle a new contract. Under the contract, which the lawsuit
>says
>was worth $500 million to $1 billion in annual revenues, FoxMeyer was to
>distribute drugs for the University HealthSystem Consortium, a nationwide
>network of teaching hospitals.
>
>Instead, the R/3 system's alleged failure to meet FoxMeyer's volume needs
>forced the drug company to revert to its Unisys system.
>
>The lawsuit charges SAP and its U.S. unit, SAP America Inc., with breach of
>contract and of express warranties and with fraudulent and negligent
>misrepresentation and concealment.
>
>"(SAP) made misrepresentations of material facts regarding its ability and
>intention to correct problems with the R/3" software's capacity to process
>FoxMeyer's orders," court papers say.
>
>In addition to actual damages of $500 million, the suit seeks a jury trial
>and unspecified punitive damages.
Read More
