Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Idealization of "in the cloud" computing model

Prev | Contents | Next

Carr's fundamentalist ambition to exterminate local datacenter infidels and establish the dominance of "in the cloud computing" in it most "pure" form (100% on remote datacenter) is pretty funny, but combination of this naive zeal with good PR and excellent writing abilities is outright dangerous. In any case vision of  all powerful remote datacenters with minimum local client functionality is misguided. There are several important concerns that dooms Carr's simplistic vision. Here are some of them

Incompetent and simplistic description by Carr of capabilities of an important but not so new (when he wrote his paper, the technology was already more then ten years old ;-) technology, actually (at least for professional audience) serves more to discredit the idea ("tell me who is your friend...")  then to promote it. 

First of all I will be the first to admit that "in the cloud" computing is a really interesting new development of distributed computing. It recently gained some traction due to availability of high speed fiber WAN networks.  But this technology can be implemented in many flavors: can be a pure play (everything done on the server; this that's the approach Carr advocates) or mixed approach with a significant part of the processing done at the client (more common and more practical approach). In no way it presuppose simplistic variant preached by Carr  in which the everything should be done of the server while for some strange reasons rich client considered to be not kosher.

In no way "in the cloud" software services model presuppose simplistic variant preached by Carr  in which the everything should be done of the server and that rich client is not kosher.

Actually term used by Carr is not mainstream and more appropriate technical term for "in the cloud" software services provider model is Software as a Service (SaaS):

Software as a service (SaaS, typically pronounced 'Sass') is a model of software deployment where an application is hosted as a service provided to customers across the Internet. By eliminating the need to install and run the application on the customer's own computer, SaaS alleviates the customer's burden of software maintenance, ongoing operation, and support. Using SaaS also can reduce the up-front expense of software purchases, through less costly, on-demand pricing. From the software vendor's standpoint, SaaS has the attraction of providing stronger protection of its intellectual property and establishing an ongoing revenue stream. The SaaS software vendor may host the application on its own web server, or this function may be handled by a third-party application service provider (ASP).

... ... ...

The concept of "software as a service" started to circulate prior to 1999 and was considered to be "gaining acceptance in the marketplace" in Bennett et al., 1999 paper on "Service Based Software" [1].

Whilst the term "software as a service" was in common use, the camelback acronym "SaaS" was allegedly not coined until several years later in a white paper called "Strategic Backgrounder: Software as a Service" [2] by the Software & Information Industry's eBusiness Division published in Feb. 2001, but written in fall of 2000 according to internal Association records.

Webmail, web hosting and other new "groupware" technologies (blogs, wikis, web-conferencing, etc) based on posting user-generated content that need to be distributed to the "world" proved to be suitable for SaaS model and they are the main drivers of this model.  Web hosting providers emerged as a strong, growing industry that essentially pioneered the commercialization of SaaS model and convincingly proved its commercial viability. As Kismore Swaminathan aptly noted in his response to Alastair McAulay’s article

Cloud computing has received plenty of attention of late. Its colorful name notwithstanding, it promises tangible benefits to its users such as sourcing flexibility for hardware and software, a pay-as-you-go rather than fixed-cost approach, continuous upgrades for mission-critical enterprise software, and the centralized management of user software.

Yet, as presently constituted, cloud computing may not be a panacea for every organization. The hardware, software and desktop clouds are mature enough for early adopters. Amazon, for example, can already point to more than 10 billion objects on its storage cloud; Salesforce.com generated more than $740 million in revenues for its 2008 fiscal year; and Google includes General Electric and Procter & Gamble among the users of its desktop cloud. However, several issues must still be addressed and these involve three critical matters: where data resides, the security of software and data against malicious attacks, and performance requirements.

I would argue that "in the cloud" paradise looks more like software demo then reality ;-). It turns out that for the last five years there was relatively little demand for "in the cloud" service providers and this technology competes with several other, first of all with virtual appliances and "datacenter in the box".  

Now about the joy of keeping all your data "in the cloud". From an end user perspective, it doesn’t make a difference if a server glitch is caused by a web host or a software service provider. Your data is in the cloud, and not on your local PC. If the cloud evaporates you can’t get to your data. If the service or site goes down, you can’t get to your data.

While software as a service might represent a licensing savings, there might be a better way to achieve the middle ground and lot to lose all advantages of local storage of data. For example by using some kind P2P infrastructure automatically synchronized so that you can view/edit data locally.  Data should not be held hostage until the user can get back to the cloud.  In this sense Microsoft's Microsoft Live Mesh might be a step in right direction as it provides useful middle grown by synchronizing data across multiple computers belonging to a users (home/office or office1/office2, etc).

But the key problem with "in the cloud" delivery of services is that only some services, for example Web and email related and those with limited I/O (both ways) are suitable for this deployment mode. Among "limited I/O" services we can mention payroll and enterprise benefits services -- another area where "in the cloud" computing definitely makes sense.  But most I/O intensive enterprise processing like file sharing is more efficiently done on a local level. That includes most desktop Office related tasks, ERP tasks to name just a few.  They are more difficult and more expensive to move into the cloud and economic justification for such a move is open for review. So in a way it is a complementary technology to the local datacenter not an alternative one. Moreover "in the cloud" approach can be implemented on a local level over LAN  instead of WAN ("mini cloud" or "cloud in the box").

From the historical standpoint "in the cloud"  providers" represent the return to the mainframe era on a new technology level and we all remember how much IBM was hated in 60th and 70th and how much energy and money people have spent trying to free themselves from this particular "central provider of services". And to what extent rapid adoption of PC were driven by the desire to send a particular "central service provider" to hell.  That raise a legitimate question of the user resistance.

And like any new service along with solving old problems it creates new.  Proponents often exaggerate positive features and  underestimate problems and costs.  Carr's vision of the IT future is based on a remote centralized and outsourced datacenter that provides services via "cloud" using high speed fiber links.  It is very true that fiber optic lines made "in the cloud" computing more acceptable. But that does not mean that this technology is a "universal bottle opener".  I would like to stress that this is just another example of distributed computing model (with usage of WAN instead of LAN) and as such it has its limits. According to Wikipedia The Fallacies of Distributed Computing are a set of common but flawed assumptions made by programmers in development of  distributed applications. They were originated by Peter Deitch   and his "eight classic fallacies" can be summarized as following:

  1. The network is reliable.
  2. Latency is zero.
  3. Bandwidth is infinite.
  4. The network is secure.
  5. Topology doesn't change.
  6. There is one administrator.
  7. Transport cost is zero.
  8. The network is homogeneous.

Actually services itself are not that cheap so cost savings in switching to the "in the cloud" service provider for large organizations might simply be not sufficient. As for May, 2008 the number of providers are in dozens with only few established vendors (WEB site providers, Amazon Web Services, Goggle App Engine and soon Microsoft Live Mesh, which is a slightly different animal, and a more realistic "mixed" approach ). Spectrum of available services is very limited. Currently is far too limited to replace a real datacenter except small startups. The most commercially viable part is represented by Web hosting and rack hosting providers.  For web hosting providers the advantages are quickly diminishing with the complexity of website, though. Among Amazon Web services only S3 storage currently can be called a successful, viable service. Microsoft Live Mesh is mostly a data synchronization service. It presuppose existence of local computers and initially provides syncing files between multiple instances of local computers belonging to the same user. This is an important service but this not a pure "in the cloud" provider model. It is more realistic "mixed" approach.

Again, with the current prices of servers and network equipment, most existing services are not cheap and became expensive as you scale them up.  We will discuss this issue later

All-in-all it is not very clear what market share "in the cloud" services deserve and but it is clear that Carr hypothesis of the universal move to "in the cloud" service providers is nothing but a modern variant of Japanese 5th generation computing.  Currently, there are a lot more places to run software than there used to be, thanks to the proliferation of powerful laptops, mobile phones like iPhone and other devices. Actually iPhones represent another interesting development that runs in the direction opposite to the move to the "in the cloud" computing. It is a powerful local device with wireless network connectivity. The key here is substantial processing power and storage capacity available on the device which is increasing with each new model.

From social standpoint Carr views also looks very unconvincing. Service providers mind their own interests first. also large service provider is the same bureaucratic monster as a typical datacenter that share with the latter all the spectrum of Dilbertalization problems. Large customers experience "vendor lock-in" working with service providers as it involves significant effort to adapt on both sides. So walk-out is a less viable option that one can assume on pure theoretical backgrounds.  It is also possible that outsourcing to "software service providers" like any outsourcing can be used as a negotiating tool (aka "method of blackmail"), to force wage concessions from workers, tax breaks from governments, etc., even when the actual savings would be small, or even  when they are negative and moving makes no business sense whatsoever.

Bandwidth communism

Promoting remote outsourced datacenter, Carr forgot to calculate the cost of bandwidth.  Or, more correctly, he assumes that free Internet connectivity is adequate for all purposes.  Yes, fiber-optic changed WAN landscape making remote services more viable and Internet tremendously more powerful.  But the devil is in details. For example file sharing for a large company over WAN is still bad idea as public bandwidth is insufficient and private is costly. Also any enterprise making bet of 24x7 availability of public bandwidth for vital corporate services looks slightly suicidal because of the "tragedy of commons" problem, which already demonstrated itself in repressions against P2P enthusiasts by large ISPs.  All-in-all this "grand utility computing" vision ("bandwidth communism") is a trap in which Nicholas Carr falls due to the lack of understanding of networking. 

Fiber networks increased both Internet and LAN bandwidth substantially and revitalized distributed computing. But there is a big difference whether you distribute over LAN or WAN.  The latter is much tougher case. With all the tremendous progress of Internet available bandwidth does not increase as quickly as computing power. Nowhere close, and it never has. If anything, due to increased scrutiny and "shaping" by ISPs (they are not a charity and need to be profitable)  bandwidth "per user" might recently start decreasing as such resource hogs as YouTube and video programming distribution services (movies on demand) are becoming more and more prominent. Ability of video streams and P2P services to clog the Net in the most inopportune moment  now is well established fact and is  a real curse for  university networks.

For i/o intensive tasks, unless you pay for the quality of service,  "in the cloud" computing model stands on a very shaky ground. Reliable 24x7 bandwidth cannot be free for all users in all circumstances and for all protocols. Contrary to Carr's views substantial amount of traffic with the remote datacenter is not that easy to transmit reliably and with minimal latency via public channels in rush hours.  But buying private links to remote datacenters can be extremely expensive: for mid-side companies it is usually as expensive as keeping everything in house. For multinationals it is more expensive, so only "other considerations" (like "jurisdiction over the data") can sustain the centralization wave to the large remote datacenters.  For many multinationals SOX was the last straw that made move of datacenters out of the USA desirable, costs be damned.  Cost of private high speed links limits cost efficiency of  the "in the cloud" approach to any service where disruptions or low bandwidth in certain times of the day cannot lead to substantial monetary losses.  For critical business services such as ERP public data channel can be too brittle.

But it is fair to state that the situation is different for different services. For example for SMTP mail outsourcers like Google/Postini, this problem is not relevant due to the properties of the SMTP protocol: they can communicate via regular Internet. The same is true to DNS services providers, webmail and instant messaging. CRM is also pretty close. But for ERP, file sharing and WAN based backup the situation is very different:  providing high speed networking services over WAN is a very challenging engineering task to say the least. The cost of bandwidth also puts natural limits on service providers growth as local networks are usually much cheaper and much faster. Achieving 1Gbit/s speed on LAN is extremely cheap (even laptops now have 1Gbit adapters) while it is quite expensive over WAN. There are also other limiting factors:  

  1. The possibility of creation of local LAN backbone with speeds higher the 1 Gbit/s.  10Gbit/s backbones are becoming more and more common.
     
  2. Limited bandwidth at the point of connection of provider to the Internet. Every provider is connected to the Internet via a pipe and that pipe is only so big. For example OC-1 and OC-3 have their upper limit of 51.84Mbit/s and 155.2 Mbit/s  correspondingly.  Funny the upper speed of OC-3 (which is pretty expensive) is only slightly higher that 100Mbit/s which long ago became the lowest common denominator for LANs.  Large service providers typically use OC-46 with speed up to 2488.32 Mbit/s which is similar to the speed of gigabit Ethernet.  10 Gigabit Ethernet is the fastest commonly available network standard for LANs.  It is still emerging technology with only 1 million ports shipped in 2007 but it is quickly growing in importance.   It might be eventually used in modified form for WANs too.   Anyway as WAN bandwidth is limited and shared between multiple customers the spike in activity of one customer might negatively affect others.  Networking problems at the provider level affect all its customers and recovery period might lead to additional spikes of activity.
     
  3. Local vs. remote storage of data. Recent enterprise level hardrives (Cheetah 15K)  have speed up to 164 MB/sec (Megabytes, not megabits).  From the speed and cost point of view the ability to keep data/programs local is a big technological advantage.  For I/O intensive applications it might be that the only viable role for remote providers is synchronization with local data ;-). Example of this approach is Microsoft's Live Mesh
     
  4. Interdependence of customers on the transport level. This is jut another incarnation of "tragedy of commons" problem. Bandwidth hogs like game, P2P, music and video enthusiasts  do not care a dime about your SLA and can easily put a company that uses public links into disadvantage any time of the day if and when something new and exiting like a new HD movie was released. Also  providers are not willing to sacrifice their revenue to accommodate "free-riders.": as soon as usage of bandwidth cuts into profits it is punishable and no amount of rhetoric about "Internet freedom" and "Net neutrality"  can change that. That means that enterprise customers relying on public bandwidth can suffer from the effort of providers to manage free-riding. That means the corporation which moved services to the cloud competes with various bandwidth hogs who do not want to scarifies any ground and ready to go quite far to satisfy their real or perceived needs.  My university experience suggest that corporate users can suffer from Internet clogging in the form of sluggish download speeds, slow response times and frustration with i/o intensive services that become much less useful and/or enjoyable. See for example Time Warner Cable Vs. The Horde.
     
  5. Competition for the resources at remote datacenter level.  For any successful service providing all the necessary bandwidth is costly and cuts into margins.  Recently Amazon faced the situation when bandwidth required for its Elastic Compute Cloud (EC2) proved to be higher then by all of Amazon.com’s global websites combined. You can read between lines how that affect profitability:

Adoption of Amazon Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) continues to grow. As an indicator of adoption, bandwidth utilized by these services in fourth quarter 2007 was even greater than bandwidth utilized in the same period by all of Amazon.com’s global websites combined.

Web services providers which offer customers unlimited bandwidth are banking on the fact that the majority of their customers will not use much of their bandwidth. This is essentially a marketing trick.  As soon as you exceed a fraction of what is promised they may well kick you out.  People who tried to implement software , mp3 or video sharing services on low cost ISP accounts realized that very soon. See for example references that I collected under "Unlimited bandwidth myth".  Web neutrality does not mean the tragedy of commons is not applicable.   As Bernardo A. Huberman, Rajan M. Lukose noted:

Because the Internet is a public good and its numerous users are not charged in proportion to their use, it appears rational for individuals to consume bandwidth greedily while thinking that their actions have little effect on the overall performance of the Internet. Because every individual can reason this way, the whole Internet's performance can degrade considerably, which makes everyone worse off. An analysis of the congestions created by such dilemmas predicts that they are intermittent in nature with definite statistical properties leading to short-lived spikes in congestion. Internet latencies were measured over a wide range of conditions and locations and were found to confirm these predictions, thus providing a possible microscopic mechanism for the observed intermittent congestions of the Internet.

So a company which will try to implement Web based streaming of say corporate video conference via cloud is up to nasty surprises unless it paid "arm and leg" for dedicated lines  to its headquarters and other major locations (which make the whole idea much less attractive in comparison with the local datacenter). The ability to stream video of any considerable quality in real-time between two (or more!) arbitrary points in the network is not really something that can be easily done over the current Internet.

The main point to make is that a reliable WAN network connectivity cost a lot of money  is difficult to achieve. This problem is unavoidable if your major components are "in the cloud" (in WAN). Also in the "free internet" enterprises are starting to compete for bandwidth with streaming media (films over Internet). The latter proved to be a huge resource hog and quality of a typical Internet connection now fluctuates widely during the day. That means that in order to achieve respectable quality of service for bandwidth intensive applications enterprises need to buy dedicated WAN connections. That is a very expensive habit to say the least. In typical for multinationals moves, say, relocation of  SAP/R3 instance from USA to Europe (just from one continent to another) to achieve reasonable latency for requests coming from the USA is not that easy and definitely not cheap.  The cost of  high bandwidth transatlantic connection is the major part of additional costs and eats all savings from the centralization. The same effect is true about any WAN connection: reliable high-bandwidth WAN connections are expensive.  Moreover the reliability needs to be carefully monitored and that also cost money as anybody who was responsible for the company WAN SLA can attest.

Sweeping under the carpet bureaucratization and mainframe-style problems
 with external providers of IT services

Centralisation, or centralization (see spelling differences), is the process by which the activities of an organization, particularly those regarding decision-making, become concentrated within a particular location and/or group.

Wikipedia

There is a shadow of mainframes over the whole idea of "in the cloud" software services providers.  Excessive centralization has its own perils, perils well known from the days of IBM360/370 and "glass datacenters".  While at the beginning growth of "cloud services"  providers increase profit margins and may even increase the quality of the service eventually each organization hits "size" wall.

That's why businesses tend to perceive such providers as inflexible, and, due to natural pre-occupation with profit margins, hostile to their business needs: exactly like their perceived "grass datacenter" in not so distant past.  So much for the improvement of the business model in comparison with traditional local datacenters, as problematic as they are (and I would be the last to deny that they have their own set of problems).  

As one reader of Calculated Risk blog noted in his comments to  the post Popular Google Product Suffers Major Disruption

Lucifer (profile) wrote on Sat, 6/20/2009 - 12:06 pm

They started employing more MBAs at Google? Seriously, any creative company that "grows up" and starts taking advice from bean counters/ entrail readers and sociopaths is doomed.

Microsoft's greatest strength was that its founders never lost control.. so even though their products were inferior (at least initially), their competitors took advice from MBAs and f**ked themselves up, leaving MS as the only survivor.

It's very hard to scale service-based tech firm and keep the mojo that made the startup successful in the first place, especially via acquisitions or employing 'professional managers' to operate the company. Basically I think the story is simple -- Parkinson's law -- bureaucracies naturally grow without limit. That includes the management of large "cloud-services" providers including Google.  Excessive layers of middle managers and affiliated cronies ("shadow secretaries") count in the denominator of labor productivity. Everyone knows that many middle managers (often called PHBs) are mainly "inventing" work for themselves and other middle managers writing memos and calling meetings and stuff.  Middle management is all about sustaining internal information flow; technology makes good middle management more efficient since it enables better information flow but it makes bad middle manager even more harmful as they "invent" useless information flow (spam) and block useful information in order to survive.

Issues of quality, loyalty, knowledge of the business are automatically surfaced and as a result customers suffer. Mainframe-style utility model encourages excessive bureaucratization with rigid procedures, stupid customer service and power concentrated at the top. That means that the issues of sociopathic leadership and sycophant managers replacing founders is even more acute then in regular IT firms.  Corporate predators  prefer large companies. As a result demoralization surface and IT personnel, that is cubicle serfs, now spend a large fraction of their time in the office, surfing the internet.

Contrary to simplistic description and assumptions typical for writers like Carr mainframe warts are very difficult, if not impossible to overcome.  And they can be amplified by low cost of free services with no reliability guarantee.   Disappearance of data usually is not covered. There is a  danger of relying of semi-free (advertisement supported) services too much.

For example anybody who used low cost Web-hosting provider can attest that interests of providers run contrary to the interests of advanced users and as such often stifle advanced technology adoption even if they are supported by a particular provider because they, for example, might increase the load on the server.  Also provision of the adequate bandwidth for multiple users (and decent responses times) can be a shaky area. Especially during  rush period like 8-11 EST.  Typically customer service is far from being responsive to any complains.

Security is another important (and costly to resolve) challenge: break-in into a large provider affects all its customers. There were several high profile break-ins into large Web hosting providers during the last two years, so this is not a theoretical threat. Claiming that Web provider are a total solution for any organization is like saying that just because the Big 4 accounting firms (with their the army of  accountants, tax specialists and so on) exist, organizations can do away with internal accountants altogether.  The hybrid platforms, such as Saleforce.com's application upgrades and quality of service issue still are of major concern. 

Relying on advertisement is another mine field. Many people hesitate to send anything important to a Gmail address, knowing that the mail will be scanned (by what is an advertising company) in transit.

Still there is a lot of disappointments with this model as exemplified with the following characterization of "cloud-based" services and outsourcing:

'We thought that we are like a farmer shooing a fat rabbit, but it turned out that the rabbit is shooing at us."

This quote suggests that  providers of such services as any  outsourcers in the future might have difficulties to shake money loose from the organizations,  as customers discover that the interests are highly divergent. IBM  already discovered that this is an inherent limit of their push to the "'service oriented organization".  Recently they even experienced a backlash.

Because we need to bridge interests of two or more different organization, there are several significant problems in interaction with "in the cloud" providers. Among them

  1. Problem of loyalty. It cuts both ways. First of all in case you use external providers loyalty of staff disappears and for any complex service you face "all or nothing situation": If service works everything is wonderful, but if it does not troubleshooting is extremely difficult and fixing the problem is almost impossible even  if you understands what is happening -- infrastructure belongs to other organization. Anybody who used Web hosting (the most successful example of such services) can attest that this is a wonderful service as long as it works. But if you have a problem you have a big problem: local staff has no loyalty to the particular organization and competition does not work as another provider can be as bad or even worse; so switching brings you nothing but additional headache and losses.  Even elementary problem can take several months to be resolved and I am not joking I experience it myself.  Oversubscription which leads to highly loaded servers and insufficient network bandwidth is another common problem.  There are also problems related to the "race to the bottom" in such services: the main differentiator becomes price and to attract new customers Web providers often make claims that are untrue (unlimited bandwidth is one typical example). As a result naive customers who believe in such claims are burned. 
     
  2. Forced upgrades. In case of external providers that platform is owned by the provider and if his financial or technological interests dictate that upgrade is necessary it will be done. That means that customers have to adapt or leave.  And upgrade for a provider with large number of customers can be huge mess that cost clients dearly.  I experienced this situation myself and can attest that the level of frustration involved is substantial and the mess can easily last several months.
     
  3. Compatibility problems.  As provider uses specific technology the level of interoperability of this technology with other important for the company technologies is not under the control of the user of such services. It can lead to significant costs due to luck of interoperability. In the simplest example lack of interoperability with Microsoft Office is a serious problem for Sun which uses Open Office (Star Office to be exact). 
     
  4. The loss of operational flexibility When switch to "in the cloud" provider is done on cost grounds alone, that usually creates new (and initially  mostly unrecognized) dependencies that deprive the company from much of the operational flexibility.  Typically security departments are direct beneficiary as security-related spending tend to grow dramatically as a defensive reaction of the organism. The main consequence of the bureaucratization is the loss of flexibility, and sooner or later this lack of flexibility can come back and haunt the  company.

    In the recent interview Bill Gates noted that "The IT systems are your brain. If you take your brain and outsource it then any adaptability you want (becomes) a contract negotiation". After you negotiated the contact with the "in the cloud" provider any flexibility you used to have is lost as every change explicitly or implicitly become a change of the contact negotiation.  Moreover, if the company lost its key IT staff and key architectural issues are decided by outsourcers, then essentially the company  becomes a hostage of outsourcers as it no longer has brain power to access the options and the architecture (and thus the costs) are by-and-large controlled by forces outside your control. That is much more serious risk that many PHB assume: the line between architectural decisions and implementation decisions is pretty fuzzy.  There is also associated brain-drain risk – if you outsource important functions, you irrevocable erode the capabilities within your firm.  When problems arise, internal IT staff can often resolve the problem more quickly, with less bureaucratic overhead inherent when two corporations are involved. 

    The fist category of factors that lead to loss of flexibility is connected with additional standards, policies and procedures that are usually introduced in external service provider situation.

    The other important aspect of the loss of  remnants of competitive advantage, if any, as the service provider is now the place were the critical mass of know-how and talent pool reside. That somewhat reverses "client-provider" relationship: it is service provider who now can dictate the rules of the game and who is the only party who understands the precise nature of the tasks involved and can conceal this knowledge for his own advantage from the other party. That usually is helped by the demoralization of the remaining staff in the company
     

  5. Amplification of the management missteps.

    Their first and most important desire of service provider is to keep the client, even if client's situation changed and no more lend itself easily to the services provided. That can lead to huge architectural blunders. Those things are not limited to offshoring but happened often with complex projects like ERP were external consultants are involved, especially in case big consultant firms are involved. Several large companies went out of business or were bought as a result. Among example we can list FoxMeyer, AT&T Wireless.  Several companies were severely hurt (Herschel, etc). This is might actually include Compaq.

    As for Compaq, the story is more complex then simplistic "competition with Dell" story which Carr tries to present in his first book. Dot-com bust hurt "value added" companies like HP and Compaq disproportionally and increased appetite for  mergers and acquisitions activities. Dogged determination of Carly Fiorina about the merger (which served both for self-promotion and as a smokescreen for her own problems at struggling HP) and the ability of former Compaq boss, a certified public accountant Michael Capellas, to understand personal benefits from Carly Fiorina proposition proved to be a winning combination.  Capellas who proved to be  a "serial CEO", actually, was a big fan of SAP and that probably was a contributed factor is Compaq troubles.  When he was appointed, he promised to turn around struggling Compaq in 180 days. Now we know that after just 84 he found a much better financial solution, at least for himself,  one big shortcut for a turnaround ;-). It is interesting to note that in 2000, based on iPAQ success, he declared[ ComputerWeekly.com,Feb 2000] :

    "There will be an explosion in devices, all attached to the Web. We are going to simplify the PC radically." Capellas promises that future desktop computers will be easier to manage than today's machines.

Large companies have become very adept at establishing remote teams in places like India, Russia, etc. Due to their ability to offer higher wages and better environment they are able to attract local talent and run challenging research and development projects. Often, though, these outposts become disconnected from their parent companies because they fail to establish rapport with key participants in the central office. Foreign prophets are usually ignored.  There is something fundamental in this "tragedy of local talent" and communication is still a problem even in the age of videoconferences. Without "warm-body" personal contacts it is difficult to build long-term trust based relationships.

Many organizations who thought outsourcing IT was the key to success miserably failed. Whose who did not failed lost competitive advantage, experienced the demoralizing effect of  vendor lockdown and QOC hokey-pokey which just simply made no sense.  Some in-sourced the IT back, some recreated it from scratch, are still in denial.

That means that client of service providers will be implicitly pushed to lowest common denominator and cannot utilize local expertise, even if such exists. They face "helpdesk level" people and instead of benefitting from specialized provider are often proposed wrong solutions to misdiagnosed problems.  my experience with WEB providers suggests that trivial problems like an error in DNS record or wrong permissions can became real production issues.

Service providers can evolve and upgrade software independently of wishes of some or all of the customers. That means that customers who are not satisfied with the direction taken need either to adapt or abandon the service.

At the same time Carr is a gifted writer and astute observer who in his writings helped to reveal the level of degradation and bureaucratization (what is often called "dilbertization of IT") that exists in current datacenters today. Here is an apt quote from one Amazon review:

The key question is, "is it as bad as Carr reports?" I can share my experience as a consultant who has worked some of the largest US and global corporations by answering "unfortunately, yes". I've seen the symptoms Carr cites in one engagement after another. It is not the fault of technology. I've witnessed the implementation of technical solutions that should have added value to business operations, yet were so mismanaged by IT that the solutions never came close to the projected ROI that justified their acquisition and implementation. Indeed, this book is similar to a collection of anti-patterns -- common bad practices -- which, sadly, reflect a typical IT department.

When the cloud evaporates: performance issues and dealing with WAN and provider outages

Public internet is unsuitable for handling large volume of transaction with stringent performance criteria. That means that it is dangerous to put databases at "in the cloud providers" : the more successful "in the cloud" providers are ( or if there just are too many P2P  and or multiplayer videogames enthusiasts in the same subnet), the slower your  WAN connection will be ("tragedy of commons").

Moreover, in comparison with LAN, WAN-based provision of software services is more complex system and as such is less reliable especially at bottlenecks (which are service provider "entry points" and associated infrastructure (DNS,  routers, switches, etc).  With WAN outage the situation can became  a lot worse then when just when spreadsheets or MS Word documents suddenly are inaccessible on the local server due to LAN outage (but you can still download then into USB stick directly from the server and work with the local copy until network connectivity is restored, because your departmental file server is just several dozens of yards away and friendly administrator probably can help you to get to the data. In case of WAN there is no way to put a USB stick on the server or use other shortcut to avoid effects of network downtime: if WAN connection is down you are really down.  Generally not only you can do nothing about the outage, its effects might be amplified by the fact that there are many customers affected.  All you will get is the message like this:

The service is experiencing technical difficulties. We apologize for the inconvenience. Please try again later .

That means that in some cases the effect on organization of external outage might be such that the head of the person who enthusiastically recommended company to move "into the cloud" rolls down independent of his position, real or faked IQ and technical competence. Recently both Gmail and Amazon services experienced multiple outages. As Brad Stone noted in NYT:

There is plenty of disappointment to go around these days. Such technology stalwarts as Yahoo, Amazon.com and Research in Motion, the company behind the BlackBerry, have all suffered embarrassing technical problems in the last few months.

About a month ago, a sudden surge of visitors to Mr. Payne’s site began asking about the normally impervious Amazon. That site was ultimately down for several hours over two business days, and Amazon, by some estimates, lost more than a million dollars an hour in sales.

The Web, like any technology or medium, has always been susceptible to unforeseen hiccups. Particularly in the early days of the Web, sites like eBay and Schwab.com regularly went dark.

But since fewer people used the Internet back then, the stakes were much lower. Now the Web is an irreplaceable part of daily life, and Internet companies have plans to make us even more dependent on it.

Companies like Google want us to store not just e-mail online but also spreadsheets, photo albums, sales data and nearly every other piece of personal and professional information. That data is supposed to be more accessible than information tucked away in the office computer or filing cabinet.

The problem is that this ideal requires Web services to be available around the clock — and even the Internet’s biggest companies sometimes have trouble making that happen.

Last holiday season, Yahoo’s system for Internet retailers, Yahoo Merchant Solutions, went dark for 14 hours, taking down thousands of e-commerce companies on one of the busiest shopping days of the year. In February, certain Amazon services that power the sites of many Web start-up companies had a day of intermittent failures, knocking many of those companies offline.

The causes of these problems range widely: it might be system upgrades with unintended consequences, human error (oops, wrong button) or even just old-fashioned electrical failures. Last month, an electrical explosion in a Houston data center of the Planet, a Web hosting company, knocked thousands of Web businesses off the Internet for up to five days.

“It was prolonged torture,” said Grant Burhans, a Web entrepreneur from Florida whose telecommunications- and real-estate-related Web sites were down for four days, costing him thousands of dollars in lost business.

 I was actually surprised how much posts each "in the cloud" service outage generates and how significant were losses reported by some users; in addition to Official Gmail Blog one of the best places to track Gmail outages proved to be Twitter; there is also a site http://downforeveryoneorjustme.com/ which provides a free check for frustrated Gmail and other "in the cloud" services users). Some reactions are pretty funny:

As any self-respecting obsessive business e-mail checker could tell you, each outage is a real shock and fails on the most inopportune moment. In reality most email outages does not make users less productive, they just deprive them from their favorite tool of wasting own and other time and  procrastination ;-)

Here is a short but pretty sobering list for 2008. It neither complete not contains the most important posts about particular outage.

[Jan 28, 2008] Gmail OUTAGE, relying on Google too much - TECH.BLORGE.com

Google services are usually rock solid reliable, but earlier today some Gmail users lost service for a couple of hours. That begs the question, are we relying on Google too much?

Gmail outages are peppered over the last few years and every time it creates a flurry of activity as businesses and individuals realize how much they rely on a single business and platform for all of their messaging needs.

The same concept is mirrored with search engine traffic, where many people don’t even realize there are search engines out there other than Google.

[Feb 16, 2008] Amazon explains its S3 outage Between the Lines ZDNet.com

Amazon has issued a statement that adds a little more clarity to its Web services outage on Friday.

Here’s Amazon’s explanation of the S3 outage, which wreaked havoc on startups and other enterprises relying on Amazon’s cloud.

Early this morning, at 3:30am PST, we started seeing elevated levels of authenticated requests from multiple users in one of our locations. While we carefully monitor our overall request volumes and these remained within normal ranges, we had not been monitoring the proportion of authenticated requests. Importantly, these cryptographic requests consume more resources per call than other request types.

Shortly before 4:00am PST, we began to see several other users significantly increase their volume of authenticated calls. The last of these pushed the authentication service over its maximum capacity before we could complete putting new capacity in place. In addition to processing authenticated requests, the authentication service also performs account validation on every request Amazon S3 handles. This caused Amazon S3 to be unable to process any requests in that location, beginning at 4:31am PST. By 6:48am PST, we had moved enough capacity online to resolve the issue.

As we said earlier today, though we’re proud of our uptime track record over the past two years with this service, any amount of downtime is unacceptable. As part of the post mortem for this event, we have identified a set of short-term actions as well as longer term improvements. We are taking immediate action on the following: (a) improving our monitoring of the proportion of authenticated requests; (b) further increasing our authentication service capacity; and (c) adding additional defensive measures around the authenticated calls. Additionally, we’ve begun work on a service health dashboard, and expect to release that shortly.

Sincerely,
The Amazon Web Services Team

[Jun 9, 2008] Surviving gMail 502 outages SKFox

I use mint to see live stats for the traffic to skfox.com and Google Analytics for the long term statistics. I’ve seen a resurgence today of traffic to my two blog posts (in March and again in May) about my gMail account being down. The most common for me is a “Temporary Error (502)” and it would seem to be happening to other users too. I hate to be the bearer of bad news, but there isn’t a whole lot you can do about it. On the short side, most of the outages only last 30 minutes or so with the longest being 3.5 hours. It can be terribly frustrating, but there are a few things you can do to alleviate the pain.

Some users have reported success getting to their mail during outages by using any number of alternative links to gmail. Your mileage may vary, but here they are:

Today’s outage is a known issue, and I hope for all of your coming here from search engines, that it comes back up quickly for you.

[Jul 20, 2008] Amazon's S3 Storage Outage Busts Web Sites, Crashes iPhone Apps

What happened? Sometime this morning, Amazon's (AMZN) S3 storage service went down. Or, according to Amazon's official note, S3 is "experiencing elevated error rates." A lot of companies -- Twitter, 37signals, etc. -- rely on S3 to host static files like images, style sheets, etc. for their Web apps. So when S3 goes down, those sites lose some/most/all functionality, depending on what the company uses S3 for.

So how'd Amazon's storage service bork my iPhone? Tapulous relies on S3 for images like Twinkle user icons. And they must not have included a "plan B" in their code to handle an image server outage. So when S3 hiccuped, and Twinkle couldn't download images, the app crashed, taking me back to the iPhone home screen. (And, hours later, it's still crashing.)

[Aug 11, 2008]  TheStar.com Business Google Gmail users having troubles

SEATTLE–Google Inc said Monday that many users of its Gmail service are having trouble accessing their online e-mail accounts due to a temporary outage in a contacts system used by Gmail.

The problems began at about 5 p.m., and a company spokesman said Google is starting to implement a fix for the problem right now.

"We are starting to roll out a fix now and hope to have the problem resolved as quickly as possible," said a Google spokesman.

[Aug 25, 2008] FOXNews.com - Amazon's Site Outage Raises Scalability Questions - Science News Science & Technology Technology News

Amazon.com has built a reputation for being an aggressive, take-no-prisoners kind of company, but it showed its more cooperative side this week.

Late last week, Internet traffic monitoring firm Keynote Systems issued a report that said many of the largest e-commerce players will face major load-handling challenges for the holidays if they don't make big changes by the fourth quarter.

However, after so many years of holiday shopping seasons, some in the industry scoffed that the majors could be caught so short.

To help out, Amazon (AMZN) generously knocked out its own servers for almost 2 hours on Aug. 21.

OK, OK. Amazon probably didn't crash just to help Keynote make a point. But its timing was impeccable nonetheless.

It's interesting to note that within a few days this month, three of the industry's most powerful retail forces — Wal-Mart (WMT), Amazon and Google (GOOG) — all suffered problems that, one way or the other, can be classified as scalability-related.

While there is nothing strange about such outages as large distributed datacenters are notoriously difficult to operate, please note that in  case the organization uses both Amazon and Gmail they have multiple noticeable outages during the first half of 2008. At the same time it is impossible to deny that such outages definitely make fragility of in the cloud model visible even for the most "pro-clouds" observer such as Mr. Carr although I doubt that he'll reproduce the statistics listed above in his blog.

But the most important difference between "in the cloud" services and local services is not duration or frequency of the outages. The most important is the level and quality of information available about the situation. In case of local services all information about the situation is readily available and thus some countermeasures can be taken. In military world such difference is of paramount important. IT is not the different in this respect from military world.  In case of  "in the cloud" the amount and quality of information about the outage are very low; services customers are essentially in the dark. Services just abruptly seizes to exists and then magically comes back.  This lack of information has its own costs.

The most important difference between "in the cloud" services and local services is not duration or frequency of the outages. The most important is the level and quality of information available about the situation. In case of local services all information about the situation is readily available and thus some countermeasures can be taken. In military world such difference is of paramount important. IT is not the different in this respect from military world.

Virtualization generally magnifies failures. In the physical world, a server failure typically would mean that backup servers would step in to prevent downtime. In the virtual world, depending on the number of virtual machines residing on a physical box, a hardware failure could impact multiple virtual servers and the applications they host. As a result failures have a much larger impact, effecting multiple applications and multiple users. Little fires turn into big fires. Virtualization might also increase two other problems:

All-in-all the more a particular company relies on "in the cloud computing", the more severe will be effect of the outages. And in comparison with traditional local LAN-based  "physical server" deployment there are several sources of them:

The first thing you notice during large outage of "in the cloud" service provider is that customer service is overloaded and you need to wait a long time to get to the human voice. There are usually just too many angry and frustrated customers before you. Also bad is that the nature of the problem and the estimate of the time needed to resolve is usually are not communicated, keeping you in the dark.  Moreover even after you get to a support person the information you get will be sketchy. Here is one example:

I was working on a client webinar using WebEx Event Center. I tried to log onto our site and got a cryptic message saying the site was unavailable. Whammo. I got on the line to technical support and waited my turn in queue, along with what must have been quite a few other panicked or angry customers. My support person was surprisingly calm and good natured, given the call volume they must have been processing. He confirmed that there were network problems on their side and said that while he didn't know details of the problem, he was certain it would be all right soon. I had a little time before our event, so I agreed to try again later.

Sure enough, our site came up, but on a backup network. This wasn't completely clean, as one of our scheduled future events had disappeared from our registration page, meaning people couldn't sign up. But at least we could carry on with today's show. Unfortunately, performance was quite a bit below normal standards, with slides and annotations taking inconsistent and sometimes very long time lags to appear on participants' computers.

After the event, strange behavior continued, with an inability to access post-event reports through the WebEx administrator interface. I called technical support again and got agreement that it certainly was strange behavior and we should all hope it would correct itself once the system was back to normal again. Whenever that might be.

Now, I'm not singling out WebEx for any particular acrimonious treatment here. I happened to be using them when this problem occurred. Any provider can have a similar problem. At least WebEx had a backup network plan in place and we were technically able to carry on with our scheduled event. But it's worth noting that there is a frustrating sense of powerlessness while a problem like this is going on. You can't prod anybody for more details, because you don't have access to the people trying to fix the problem as you might if a similar situation occurred with your own business network. You can't get much in the way of progress updates. And you can't put your own backup plan into effect.

One interesting difference in "horror stories" between loacl and "in the cloud" datacenter was aptly outlined in the following Computerworld  quote:

Just after midnight on Jan. 22, 2006, Con Edison began telling the Internet that it was Martha Stewart. That is, Con Edison erroneously began sending out routing information to redirect Internet traffic intended for Martha Stewart Living Omnimedia to its own servers.

The publishing company was a Con Edison customer. So were other businesses and Internet providers whose routing information Con Edison hijacked at the same time. The result was a mess that wasn't completely cleared up for 18 hours — and some businesses were offline for most of that time.

But not Martha Stewart, whose CIO, Mike Plonski, wrote to me to clarify what happened at his company.

Plonski's secret sauce? No big secret — just network monitoring and redundancy.

Plonski said: "While one of the Internet connections at our corporate offices was impacted by the ConEd issue you describe, we, as a company, are smart enough to have employed redundancy, both by location and carrier, for our network operations. As a result, during this time frame, we simply flipped all of our Internet traffic to run over our secondary line until ConEd resolved their issue."

OK, it was a little more complicated than that. Plonski said his staff spotted the problem through routine network monitoring. There was clearly something wrong with network traffic coming to the corporate office over the Con Edison line. Thanks to the monitoring, the company knew about the problem about 30 minutes after it started.

Because of the type of outage, an IT staffer had to connect and manually switch over to a redundant line. That took another hour.

Total time for the outage: about 90 minutes in the wee hours of a Sunday morning. Total impact: minimal.

An outage? Yes. A knockout? No way. And handling the problem didn't require rocket science — just monitoring, redundancy and sharp IT staff work.

Repeat after me: [in case of local datacenters "handling the problem didn't require rocket science — just monitoring, redundancy and sharp IT staff work."

Interoperability problems

While individual "in the cloud" service provider can do a decent job in providing the required functionality, the problem arise when you need to use several such providers and issues of interoperability arise.

The lack of interoperability between different SaaS applications is one of the most well known and, at the same time, largest problems with SaaS. In a way companies who are using several different providers spending way too much money with the side effect of creating a problematic,  cumbersome IT architecture that lack flexibility and efficiency and as such is inferior to the integrated datacenter architecture. The focus of vendors is on the adaptation of the SaaS model to enterprise requirements (enterprise-level integration), but there is a growing need for vendor-to-vendor integration. How and when those needs will be addressed remains to be seen.  

In a way SaaS application emphasize too much GUI portion of functionality of application at the expense of the ability of smoothly exchange data with other, often similar applications. 

The emergence of a several classes of enterprise SaaS (for email, CRM, supply chain management, benefits and other HR-related applications, etc ) creates problems of similar or identical data existing in various formats in different providers and their synchronization and timely updates. It providers does not share common "hardware cloud" we have a huge problems as not only protocols of data exchange, but reliability and security issues are pretty complex.

Also the existing interoperability can be broken anytime by software updates by one of several for existing "in the cloud" providers.

Prev | Contents | Next

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

 You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019