...In 1990 Lucy Patrick commented: "Although we live in a democracy, cult behavior
manifests itself in our unwillingness to question the judgment of our leaders, our tendency to
devalue outsiders and to avoid dissent. We can overcome cult behavior, he says, by recognizing
that we have dependency needs that are inappropriate for mature people, by increasing
anti-authoritarian education, and by encouraging personal autonomy and the free exchange of
Groupthink is a psychological
that occurs within a
people in which the desire for harmony or conformity in the group results in an
irrational or dysfunctional
decision-making outcome. Group members try to minimize conflict and reach a
consensus decision without
critical evaluation of alternative viewpoints by actively suppressing dissenting
viewpoints, and by isolating themselves from outside influences.
requires individuals to avoid raising
issues or alternative solutions, and there is loss of individual creativity,
uniqueness and independent thinking. The dysfunctional
dynamics of the "in-group"
produces an "illusion of invulnerability" (an inflated certainty that the right
decision has been made). Thus the "in-group" significantly overrates its own abilities
in decision-making and significantly underrates the abilities of its opponents (the "out-group").
Furthermore, groupthink can produce dehumanizing actions against the "out-group".
Antecedent factors such as
group cohesiveness, faulty group structure, and situational context (e.g.,
community panic) play into the likelihood of whether or not groupthink will impact
the decision-making process.
You all read typical advantages sited for DevOps methodology: It helps you deliver products faster, improves your
profitability, ensures continuous integration, removes roadblocks from your releases and gives you a
competitive advantage. Those claims alert all seasoned IT professionals because they are very similar claims about previous IT fads that now are
semi-forgotten. There were at least half dozen previous attempts to achieve nirvana in the
datacenter and, especially, enterprise software development, starting from "verification revolution" initiated
by Edsger W. Dijkstra, who probably
was one of first "cult leader" style figures in the history of programming.
Still large and successful companies as Netflix and Amazon supposedly practice DevOps (and in case of Netflix heavily
advertize). While there are multiple definition of DevOps, the definition usually includes the usage (typically in
the cloud environment) of the following methodologies ( DevOps - Wikipedia )
Most of those are "no-brainers" and should be used by any decent software development organization, but some are questionable: more
frequent releases is a good idea only for minor releases, bug fixes.
Continues delivery is even more questionable idea and sticking to it undermines efforts to revise the
architecture of the packages. It is essentially "froze" the current architecture "forever". As such it is
a very bad thing, a fallacy. Moreover continues delivery is nothing new as most software development project now provide
access to beta versions of the software. The same id true to automated testing which now is sold under the umbrella of
"continues testing". Perl developers used it decades ago.
Most constructive ideas associated with DevOps were used in computational clusters for a decade or so. And even earlier
they were reflected in Sun concept of the "grid" which originated with their purchase In 2000 Gridware,
Inc. a privately owned commercial vendor of advanced computing resource management software with offices in San Jose, Calif.,
and Regensburg, Germany. Later
that year, Sun offered a free version of Gridware for Solaris and Linux, and renamed the product Sun Grid Engine.(Sun Grid as cloud
service was launched in March 2006).
Those idea center around Sun earlier idea "network is a computer" and including the idea of using central management unit for a
large group of servers or the whole datacenter (like "headnode" in computational cluster), central monitoring, logging and parallel
execution tools to deliver changes to multiple server simultaneously. Similar ideas were developed by Tivoli under the name
"system management software services" starting from 1989( IBM acquired Tivoli in 1996). Accounting to Wikipedia (Tivoli
The most dangerous aspect of DevOps is the attempt the attempt of elimination or, at least, a diminished role of sysadmins
in the datacenter
The other important (and rather dangerous) aspect of DevOps is the attempt of elimination or, at least, a diminished role of
sysadmins in the datacenter. The slogan is "all power to the developers ;-)". With the complexity of modern
operating systems such as Red Hat 7 this is completely unrealistic goal. And no amount of automation can help to achieve it.
Even in the cloud, as soon as you try to do something more or less complex you need a specialist completely devoted to
learning this part of infrastructure. That's why Certified AWS specialist and
similar certification started to proliferate and holders of those certification are in demand.
Attempt of DevOps reverse the history and return us to the easly days of computing when programmer was simultaniously a system
administrator (and sometimes even hardware engineer ;-) are doomed to be a failure.
The costs of moving everything to the cloud are often exorbitant and this fiasco is covered with fake PowerPoint presentations,
destined to hide the fact that after top management got their bonuses the enterprise IT entered the state of semi-paralysis. Yes,
cloud has its uses but it is far for being panacea.
The other aspect here is the question of loyalty. By outsourcing datacenter as DevOps recommends you essentially rely of kindness
of strangers in case of each more or less significant disaster. Strangers that have zero loyalty to your particular
firm/company no matter what CLA are told. In other way their drama of outsourcing helpdesk is now replayed on a higher
level and more dangerous consequences.
Software engineering proved to be very susceptible to various fads
and pseudo-religious movements (with high levels of religious fervor).
Another warning sign is lofty goals. Nobody in his sound mind would object to
achieving stated DevOps goals
The goals of DevOps span the entire delivery pipeline. They include:
Faster mean time to recovery (in the event of a new release crashing or otherwise disabling the current
The question is only: Does DevOps really achieves them or is mostly a hype?
Software engineering proved to be very susceptible to various fads and pseudo-religious movements (with high levels of religious
fervor). Prophets emerge and disappear with alarming regularity (say, each ten years or so -- in other words the period long enough
to forget about the previous fiasco). We already mentioned the "verification revolution", but we can dig even deeper
and mention also "structured programming revolution" with its pseudo-religious crusade against goto statements (while misguided
it, at least, has a positive effect on introducing additional high level control structures into programming languages; see important historically article by Donald
Knuth Structured programming with goto statements
Verification hoopla actually destroyed careers of several talented computer scientists such as
David Gries (of
Compiler Construction for Digital Computers fame; who also participated in creation of amazing
PL-C compiler designed by Richard W. Conway and
Thomas R. Wilcox ). It
also damaged long term viability of achievements of
Niklaus Wirth ( a very talented language
developer, who participated in creation of Algol 60 and later developed Pascal and Modula). He also
is known for his Wirth's law:
"Software is getting slower more rapidly than hardware becomes faster."
Dismal quality of Netflix web interface actually instantly razes several question about
"What those guys are doing?" Like Amazon Web site this is essentially an index to the product database with some added
blog features (reviews.)
But if Amazon at least tries to implement several most relevant types of searches, Netflix does not.
For rare movies it is rather challenging to find what you interested in, unless you know the exact
title ;-). Movies are not consistently indexed by director and major stars. The way they
deal with reviews is actually sophomoric. But the colors are nice, no question about it ;-)
Amazon is much better, but in certain areas it still has "very average" or even below average quality. One particularly bad oversight is that
the reviews are indexed only in several basic dimensions (number of stars, popularity (up votes) and chronological order), not by "reviewer reputation"
(aka karma), number of previous reviews (first time reviewers are often fake reviewers), and other more advanced criteria.
You also can't combine criteria in the search request.
result it requires additional, often substantial, work to filter our "fake" reviews (bought, or "friends reviews" produced by people who are not actual customers).
They might even dominate Amazon rating for some products/books. In any case that "effect" diminishes the value of Amazon rating.
In some categories of
products considerably. Recently Amazon developers tried to compensate this
by "verified purchase" criteria, but without much success, as, for example, in most cases the book can be
returned. While some
fake reviews are detectable by the total number of reviews posted (often this number is one ;-). or their style, in
many cases it is not possible easily to distinguish "promotional campaign" of the author of the book (or vendor of the
product) from actual reviews.
Shrewd companies can subsidized purchases in exchange for positive reviews. In this sense Amazon interface sucks and sucks
Amazon cloud has
its uses and was generally a useful innovation, so all the DevOps hoopla about it is not that bad, but is rather expensive and is suitable mostly for loads with huge
short term peaks and deep,
long valleys. For example genomic decoding.
As large enterprise brass now is hell-bent on deploying DevOps (for obvious, mostly greedy reasons connected with outsource, see
below ;-) it would be stupid to
protest against this techno-cult. In most cases your honestly will not be appreciated, to say the least. So you need to adapt
somehow and try to make lemonade out of lemons. There are two technologies that you can adpot under Dev Ops umrella
The first thing you can do is to negotiated some valuable training. And I am not talking about this junk coursers in DevOp
(although off-site, say in NYC, utilized as extra vacation they also have some value ;-). I am talking, for example, getting couse
in Docker and Ancible/Puppet/chet. And if you use Puppet you can legitimately ask for Python classes. So you can navigate this
landscape from the point of view of getting additional training and try to squeezed something from stone: corporations now are notoriously tight as for training expenses.
One way to play this game is to equate DevOps with some interesting for you technology. Docker has it value in both sysadmin area
and applications area. So you can claim that this is a prerequisite for DevOps and site some books on the subject. Of
course, this is not a panacea from many
of the existing enterprise datacenter ills, but the technology is interesting, implementation is elegant and the approach has merits,
especially if you in proxy envoronment. In case of research applications a developer often can make them run quicker in Docker
environment, epscially if applicatin is availble on Ubuntu or Debian in packaged form, but not on CentOS or RHEL. There are
some useful collections of prepackaged application, mainly for bioscience. See for example BioContainers (the list of available containers can
be found here .)
Esoteric applications that are difficult to install what which require specific libraries with version different that,
say, used by your RHEL version can really benefit from Docker, as it allow your to compartmentalize "library hell" and use the
most suitable for the particular application flavor of Linux (which is of course the flavor in which the application was
developed). For example, many scientific application are native to Debian, which is not an enterprise linux distribution.
Docker allow you to run instances of applications installed from Debian packages on a Red Hat server ("run anywhere" meme). And if Docker is not enough you can always fall back to regular VM
such as XEN.
In any case Docker is not a fad. It is an interesting and innovative "light-weight VM" technology which was first commercially introduced by Sun in Solaris
10 in 2005 (as Solaris zones) and replicated in Linux approximately 10
years later. The key idea that this is essentially packaging technology -- all "VM" share the kernel with the base OS. It is a variant of paravitualization, which produces minimal overhead in comparison with running application as a task
on a real server. Much more efficient then VMware. Linux containers like Solaris zones are essentially extension of the concept of jail ( the idea of extending jail concept to more "full" VM
environment originated in FreeBSD.) It is definitely worth learning, and in some cases to be widely deployed. Anyway,
this is something real, not the typical shaman-style rituals that "classic" DevOps try to propagate (
Lysenkoism first played out are a real tragedy, now it degenerated into a farce ;-)
As most DevOps propagandists and zealots are technologically extremely stupid, it is not that difficult to deceive them.
Sell the Docker as the key part of DevOps 2.0 toolkit, the quintessence of the cloud technology (they like the word "cloud) which
lies in the core of DevOps 2.0 -- a hidden esoteric truth about DevOps that only real gurus know about :-) They will eat
On higher, datacenter level you can try to push the adoption of Red Hat OpenShift, which kind of in-house cloud and
is cheaper and more manageable then Amazon elastic cloud, or Azure. And in many cases makes sense to deploy. That
might allow to get to extract from management payment for
RHEL Learning Subscription. Try to equate the term "hybrid cloud" with use of
OpenShift within enterprise. You can also point out, that unless you have short peak and long no load periods both Azure and AWS are
pretty expensive, and it is not wise to put all eggs into one basket.
While modern configuration management tools are excessively complex some of them try to ride DevOps hype. And an important part
of this hype is Infrastructure as Code(IaC) hoopla. So you
can get training and support in installation and configuration of one of such tools, if you want to. They are overly complex,
duplicate many existing tools functionality, are overhyped, but they are not that bad. See
Unix Configuration Management Tools for more "in-depth" discussion.
As I mentioned above adopting Puppet (or pretending to adopt it ;-) might allow your to get some valuable Python
training. Puppet was most loud in DevOps hoopla ( they even advertise Agile on their
pages ;-). They also pretend to be continues delivery tool, the pretence that might perfectly suit your needs:
No one wants to spend the day waiting for someone to manually spin up a VM or twiddle their thumbs while infrastructure
updates happen. Freedom comes when you get nimble, get agile, and tackle infrastructure management in minutes, not hours. Puppet
Enterprise uniquely combines the industry-standard model-driven approach we pioneered with the flexibility of task-based
automation. Welcome back, lunch break.
... ... ...
Continuous delivery isn’t just for developers. Build, test, and promote Puppet code, high-five your teammates, and get back to
doing what you love with Continuous Delivery for Puppet Enterprise. Instead of wondering if you have the latest code running, now
you can streamline workflows between Dev and Ops, deliver faster changes confidently and do it all with the visibility your teams
For example Ansible; please note that you can sell
as a configuration management tool even Pdsh to those idiots. Mention that it is
well integrates with GIT (another magic word that you need to know and use). GIT is not that impressive for Linux configuration
management, but sometimes can be used. See for example etckeeper package; if you find it useful and want to use it just
claim that this is DevOps 2.0 too, and most probably you will be blessed to deploy it. And while it has drawbacks, it allows
to record actions of multiple sysadmin on the server that result in changes of files in /etc directory, treating it as a
software project with multiple components. So this is far from perfect but still a usable tool for solving the problem of multiple
cooks in the same kitchen.
Use GitLab as a proxy for continue integration
In some, rear cases you might even wish to play "continues integration" game. If corporate brass demands continuous
integration to be implemented you might try to adapt software like GitLab
for this purpose and use GitLab "pipelines", which provides some interesting opportunities for several scripting languages such as
Python and Ruby. Automated testing part if probably the most useful. While you can always write independent scripts for
automated testing, integration of them into GitLab framework often is a better deal.
But we should be aware that the key problem with
"continuous integration" and closely related concept of "continuous testing" is that for testing scripts the most difficult part is
the validation of the output of the test: the road to hell is always paid with good intention. And while the idea of automated
testing is good, for many scripts and software packages the implementation is complex and man power consuming. Actually this is one area were outsourcing might help. So far
there was not breakthrough in this area and much depends on your own or developers abilities as programmer. Regular expression
can help to check output, but they are not a panacea.
Also the most complex is your testing script the more fragile it is and has less chances to survive changes in the codebase. So
complex testing scripts tend to freeze the current architecture as as such as harmful.
In its essence, continuous
delivery is an overhyped variation of the idea of night build, used for ages. If you want to play a practical joke on your
developers tell them to use Jenkins as an automated tests tool. Jenkins is a very complex tool, but it can be installed through native system packages, Docker
container, or even run standalone by
any machine with a Java Runtime Environment (JRE) installed. AWS
has a "ready made" project How to
set up a Jenkins build server which can save you from troubles and waisting your time;-). But often you
do not need to install it; just associate the term Jenkins with "continuous integration" and provide for them a
Docker container with Jenkins. Your advantage is that developers usually do not have deep understanding
of Linux, and, especially, virtualization. issues. And usually they do not want to learn too much new staff which gets them too far
away from their favorite programming language. So there are strong chance that they will ignore this technology and stick to GIT and
their old custom testing scripts ;-)
Try to improves your bargaining position
While I mentioned two useful tools that suit DevOps banner, there might be more. Summarizing, I think that "creative" ad hoc interpretation of DevOps might improve your bargaining position, and can serve as a
draft of the plan of your fight against the stupid aspects of the DevOps onslaught.
It is important to try to dictate the terms of
the game using your superiority in understanding of Linux and control of the data center servers. Depending on your
circumstances and the level of religious zeal for DevOps in your organization that might be a viable
strategy against unreasonable demands of developers. In any case do not surrender without a fight and please understand that a direct frontal attack on this fad
is a bad strategy.
Now if some brainwashed with DevOps developer tries to enforce his newly acquired via DevOps hoopla rights to manage
the servers ("we are one team"), you can instantly put him in his place pointing our that those weaklings do not know DevOps 2.0
technology and as such does not belong to this Brave New World. It is pretty effective IT political line that gives you better
chances for survival then the direct attack on this fad.
Still, you might also need to visit a couple of brainwashing session (aka "DevOps Learning Bundle") to demonstrate your
You probably heard a little bit about so called high demand cults. They exist in techno-sphere too, and while features are
somewhat water downed they are still recognizable (High Demand Cults):
Remember ... A group does not have to be religious to be cultic in behavior. High
demand groups can be commercial, political and psychological. Be aware, especially if you are a
bright, intelligent and idealistic person. The most likely person to be caught up in this type of
behavioral system is the one who says “I won’t get caught. It will never happen to me. I am too
intelligent for that sort of thing.”
The following statements, compiled by Dr.
Michael Langone, editor of
Cultic Studies Journal, often characterize manipulative groups. Comparing these
statements to the group with which you or a family member is involved may help you determine if
this involvement is cause for concern.
Fake display of excessively zealous, unquestioning commitment and
adulation to new members. Expensive gift, etc. This is a typical practice of sociopaths and many
cult leaders are sociopaths.
Isolation: The group leader instills a polarized, "we-they" mentality and try to
isolate members from their previous contacts. Members are encouraged or required to work with
and/or socialize only with group members. Severing of Ties with Past, Family, Friends, Goals,
and Interests - Especially if they are negative towards or impede the goals of the group.
High demand on members time, intensity of contacts: Intense involvement in cult
activities work along with isolation from others. Behaviour is closely prescribed and
carefully supervised. Members are expected to devote inordinate amounts of time to the group
and group activities.
Manipulation: The group's leadership induces guilt feelings in members in order to
control them. The pity play is high in their list of manipulation techniques. It's
okay to pity someone who has gone through difficult times, but if you find yourself feeling
sorry for someone's sad story, make sure the story is true. The pity play should serves as a
valuable warning sign that you are dealing with a manipulator, in this case a female
Brainwashing: Questioning, doubt, and dissent are discouraged or even punished with
a bouts of anger or (fake) withdrawal. Special techniques are used to suppress doubts about
illegal or questionable or amoral practices in the group or its leader(s) and put dissidents
in line. Special literature is distributed and indoctrination campaign is launched. Much like
pedophiles "groom" children by pushing or encouraging them to watch porno movies and books
with explicit content. If group leader requires special favors from members of the group (for
example sex with female members) this all is masked under some artificial pretext like
"exercise in liberation" or "exercise in compassion".
Dictate and micromanagement: The group's leader practice micromanagement and
dictates – sometimes in great detail – how members should think, act, what to wear and
Instilling amorality, end justifies the means mentality: Any action or behaviour
is justifiable as long as it furthers the group's goals. The group (leader) becomes
absolute truth and is above all man-made laws.
My initial impression that former cultists come face with a multiplicity of losses,
accompanied by a deep, and sometimes debilitating, sense of anguish. See for example interviews
with defector from Mormonism on YouTube
... ... ....
My hope upon initiating this research was to provide a link between cult leaders
and corporate psychopath and demonstrate that cult leaders practices (that are more
or less well understood and for which extensive literature exists) have a strong
predictive power for the behavior of a corporate psychopath. We should not focus
just on the acute and long-term distress accompanying reporting to corporate
Here are some psychological mechanisms used:
Control of the Environment and Communication The control of
human communication is the most basic feature of the high demand cult
environment. This is the control of what the individual sees, hears,
reads, writes, experiences and expresses. It goes even further than that, and
controls the individuals communication with himself - his own thoughts.
The Mystique of the Organization. This seeks to provoke specific
patterns of behaviour and emotion in such a way that these will appear to have
arisen spontaneously from within the environment. For the manipulated person
this assumes a near-mystical quality. This is not just a power trip by the manipulators.
They have a sense of “higher purpose” and see themselves as being
the “keepers of the truth.” By becoming the instruments of their own mystique,
they create a mystical aura around the manipulating institution
- the Party, the Government, the Organization, etc. They are the chosen agents
to carry out this mystical imperative.
Everything is black & white. Pure and impure is defined by
the ideology of the organization. Only those ideas, feelings and actions
consistent with the ideology and policy are good. The individual conscience
is not reliable. The philosophical assumption is that absolute purity is attainable
and that anything done in the name of this purity is moral. By defining and
manipulating the criteria of purity and conducting an all-out war on impurity
(dissension especially) the organization creates a narrow world of guilt and
shame. This is perpetuated by an ethos of continuous reform, the demand that
one strive permanently and painfully for something which not only does not exist
but is alien to the human condition.
Absolute “Truth” . Their “truth” is the absolute truth. It is sacred
- beyond questioning. There is a reverence demanded for the leadership. They
have ALL the answers. Only to them is given the revelation of “truth”.
Thought terminating clichés. Everything is compressed into brief,
highly reductive, definitive-sounding phrases, easily memorized and easily expressed.
There are “good” terms which represents the groups ideology and “evil” terms
to represent everything outside which is to be rejected. Totalist language is
intensely divisive, all-encompassing jargon, unmercifully judging. To those
outside the group this language is tedious - the language of non-thought. This
effectively isolates members from outside world. The only people who understand
you are other members. Other members can tell if you are really one of them
by how you talk.
Hubbard called Dianetics "a milestone for man comparable to his discovery of fire and superior to his invention of the wheel
and the arch". It was an immediate commercial success and sparked what
Martin Gardner calls "a nationwide cult of
By August 1950, Hubbard's book had sold 55,000 copies, was selling at the rate of 4,000 a week and was being translated into
French, German and Japanese. Five hundred Dianetic auditing groups had been set up across the United States.
... ... ...
The manuscript later became part of Scientology mythology.
An early 1950s Scientology publication offered signed "gold-bound and locked" copies for the sum of $1,500 apiece (equivalent to
$15,282 in 2017). It warned that "four of the first
fifteen people who read it went insane" and that it would be "[r]eleased only on sworn statement not to permit other readers to
read it. Contains data not to be released during Mr. Hubbard's stay on earth."
... ... ...
In October 1984 Judge Paul G. Breckenridge ruled in Armstrong's favor, saying:
The evidence portrays a man who has been virtually a
pathological liar when it comes to his history, background and achievements. The writings and documents in
evidence additionally reflect his egoism, greed, avarice, lust for power, and vindictiveness and aggressiveness
against persons perceived by him to be disloyal or hostile. At the same time it appears that he is charismatic and
highly capable of motivating, organizing, controlling, manipulating and inspiring his adherents. He has been
referred to during the trial as a "genius," a "revered person," a man who was "viewed by his followers in awe."
Obviously, he is and has been a very complex person and that complexity is further reflected in his alter ego, the
Church of Scientology.
The key indicator
is greedy, control oriented leadership, ministers who enrich themselves at the expense of followers (one minister of Prosperity
Theology asked followers to donate to him money for his new private jet). Attempts to extract money from the followers
requiring payment for some kind of training, "deep truth", sacred manuscripts reading.
who raises uncomfortable questions or does not "get with the program" is ostracized. Questioning of the dogma is
In reality nothing is new under the sun in software development. DevOps rehashed the ideas many
of which are at least a decade old,
with some that are at
least 30 years old (Unix configuration management, version control). And the level of
discussion is often lower than the level on which those ideas are discussed in
The Mythical Man-Month, which was published
in 1975 (another sign of "junk science").
What is more important that all
under the surface of all those lofty goals is the burning desire of companies brass to use DevOps as another
smoke screen for outsourcing.
Yet another justification for "firing a lot of people."
Under the surface of all those lofty goals is the burning desire of companies brass to use DevOps as another
smoke screen for outsourcing.
Yet another justification for "firing a lot of people."
As in any cult, there are some grains of rationality in DevOps, along with poignant
critique of the status quo. Which works well to attract the followers. Overhyping some ideas about how to cope with the
current, unsatisfactory for potential followers
situation in enterprise IT is another sign of a technocult. Some of those ideas are, at least on superficial level, pretty attractive. Otherwise such a technocult
can't attract followers. As a rule, techno-cults generally emerge in time of huge dissatisfaction and strive by proposing
"salvation" from the current difficulties:
...think this is one of the main reason why we see this DevOps movement, we are many that see this
happen in many organizations, the malfunctioning organization and failing culture that can’t get
operation and development to work towards common goals. In those organizations some day development
give up and take care of operation by them self and let the operation guys take care of the old stuff.
Nobody can deny that there a lot of problems in corporate datacenters those days.
bureaucratization is rampant and that stifle few talented people who still did not escape this
environment, or switched to writing open source software during working hours, because achieving
anything within the constrain of the existing organization is simply impossible ;-)
But along the this small set of rational (and old) ideas there a set of
completely false, even bizarre ideas and claims, which make it a cult.. There is also fair share of
creep. Again, it is important to understand that part of the promotion
campaign success (with money for it mostly from companies who benefit from outsourcing) is connected
with the fact that corporate IT brass realized than DevOps can serve well as a smote screen for another round of
outsourcing of "ops". (Ulf Månsson
about infrastructure )
Pretention that this is a new technology is artfully created by inventing a new language ripe with new
obscure terms. Another variant of Newspeak. And this is very important to understand. It is this language that allow to package bizarre and unproven
ideas into the cloak of respectability.
Primitivism of thinking and unfounded claims of this new IT fashion (typical half-life of IT fad
is less then a decade; for example, who now remembers all this verification hoopla and
books published on the topic) are clearly visible in advocacy
papers such as
DevOps to traditional IT Eight key differences - DevOps.com. Some of the claims are clearly suspect,
and smell of "management consultant speak" (an interesting variety of
corporate bullshit). For example:
Traditional IT is fundamentally a risk averse organization. A CIO’s first priority is to
do no harm to the business. It is the reason why IT invests in so much red tape, processes,
approvals etc. All focused on preventing failure. And yet despite all these investments,
IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements
are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.
A DevOps organization is risk averse too but they also understand that failure is inevitable.
So instead of trying to eliminate failure they prefer to choose when and how they fail. They
prefer to fail small, fail early, and recover fast. And they have built their structure and
process around it. Again the building blocks we have referred to in this article – from test
driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation
etc. all reinforce this mindset.
Note a very disingenuous claim "IT has a terrible track record – 30% of new projects are delivered late,
50% of all new enhancements are rolled back due to quality issues, and 40% of the delay is caused by
infrastructure issues." I think all previous fashionable methodologies used
the same claim, because the mere complexity of large software projects inevitably leads to overruns
and doubling estimated time and money is a good software engineering methodology known since
publishing of Mythical man month :-)
As for another disingenuous claim "So instead of trying to eliminate failure they prefer
to choose when and how they fail. They prefer to fail small, fail early, and recover fast."
this is not a realistic statement.
It is a fake. In case of a SNAFU you can't predict the size of the failure. Just look at history of Netflix,
Amazon or Google Gmail failures. Tell this mantra "They prefer to fail
small, fail early, and recover fast" to customers of Google or Amazon
during the next outage
Note also such criticism is carefully swiped under the carpet and definition of DevOps evolves with time to preserve
its attractiveness to new members (
DevOps is Dead! Long Live
DevOps! - DevOps.com ). In other words the correct definition of DevOps is "it is a very good
thing" ;-). For example:
Some are seekers on the quest for the one, true DevOps. They were misled. I’m here to say: Give
it up. Whatever you find at the end of that journey isn’t it. That one true DevOps is dead. Dead
and buried. The search is pointless or, kind of worse: The search misses the point altogether.
DevOps began as a sort of a living philosophy: about inclusion rather than exclusion, raises
up rather than chastises, increases resilience rather than assigns blame, makes smarter and more
awesome rather than defines process and builds bunkers. At any rate, it was also deliberately never
In 2010, it seemed largely about
Release Automation (ARA), monitoring, configuration management and a lot of beginning discussion
about culture and teams. By 2015, a lot of CI/CD, containers and APIs had been added. The dates are
rough but my point is: That’s all still there but now DevOps discussions today also include
service design and microservices. Oh, and the new shiny going by the term “serverless.” It is about
all of IT as it is naturally adapting.
Junk science is and always was based on cherry-picked evidence which has carefully been
selected or edited to support a pre-selected "truth". Facts that do not fit the
agenda are suppressed (Groupthink). Apocalyptic
yelling are also very typical. Same for
Pollyanna creep. Deployment is typically top-down. Corporate management is used as
an enforcement branch (corporate Lysenkoism). Here are some signs of "junk science" (Seven
Eight Warning Signs of Junk Science):
Here is a non-exclusive list of seven eight symptoms to watch out for:
Science by press release. It’s never, ever a good sign when ‘scientists’
announce dramatic results before publishing in a peer-reviewed journal. When this
happens, we generally find out later that they were either self-deluded or
functioning as political animals rather than scientists. This generalizes a bit;
one should also be suspicious of, for example, science first broadcast by
congressional testimony or talk-show circuit.
Rhetoric that mixes science with the tropes of eschatological panic.
When the argument for theory X slides from “theory X is supported by evidence” to
“a terrible catastrophe looms over us if theory X is true, therefore we cannot
risk disbelieving it”, you can be pretty sure that X is junk science. Consciously
or unconsciously, advocates who say these sorts of things are trying to panic the
herd into stampeding rather than focusing on the quality of the evidence for
Rhetoric that mixes science with the tropes of moral panic. When the
argument for theory X slides from “theory X is supported by evidence” to “only
bad/sinful/uncaring people disbelieve theory X”, you can be even more sure that
theory X is junk science. Consciously or unconsciously, advocates who say these
sorts of things are trying to induce a state of preference falsification in which
people are peer-pressured to publicly affirm a belief in theory X in spite of
Consignment of failed predictions to the memory hole. It’s a sign of
sound science when advocates for theory X publicly acknowledge failed predictions
and explain why they think they can now make better ones. Conversely, it’s a sign
of junk science when they try to bury failed predictions and deny they ever made
Over-reliance on computer models replete with bugger factors that aren’t
causally justified.. No, this is not unique to climatology; you see it a lot
in epidemiology and economics, just to name two fields that start with ‘e’. The
key point here is that simply fitting historical data is not causal justification;
there are lots of ways to dishonestly make that happen, or honestly fool yourself
about it. If you don’t have a generative account of why your formulas and
coupling constants look the way they do (a generative account which itself makes
falsifiable predictions), you’re not doing science – you’re doing numerology.
If a ‘scientific’ theory seems tailor-made for the needs of politicians or
advocacy organizations, it probably has been. Real scientific results have a
cross-grained tendency not to fit transient political categories. Accordingly, if
you think theory X stinks of political construction, you’re probably right. This
is one of the simplest but most difficult lessons in junk-science spotting! The
most difficult case is recognizing that this is happening even when you agree with
Past purveyers of junk science do not change their spots. One of the
earliest indicators in many outbreaks of junk science is enthusiastic endorsements
by people and advocacy organizations associated with past outbreaks. This
one is particularly useful in spotting environmental junk science, because
unreliable environmental-advocacy organizations tend to have long public pedigrees
including frequent episodes of apocalyptic yelling. It is pardonable to be taken
in by this the first time, but foolish by the fourth and fifth.
Refusal to make primary data sets available for inspection. When
people doing sound science are challenged to produce the observational and
experimental data their theories are supposed to be based on, they do it. (There
are a couple of principled exceptions here; particle physicists can’t save the
unreduced data from particle collisions, there are too many terabytes per second
of it.) It is a strong sign of junk science when a ‘scientist’ claims to have
retained raw data sets but refuses to release them to critics.
If we are talking about DevOps as a software development methodology it is similar to
Agile. The latter was a rather successful attempt
to reshuffle a set of old ideas for fun and profit (some worthwhile, some not so much) into attractive marketable technocult
and milk the resulting "movement" with books, conferences, consulting, etc.
In 2017 only few seasoned software developers believe that Agile is more then self-promotion campaign of group of unscrupulous
and ambitious people, who appointed themselves as the high priests of this cult. Half-life of such "cargo cult" programming
methodologies is usually around a decade, rarely two (Agile became fashionable around 1996-2000). Now it looks like Agile is
well past the "hype stage" in the software methodology life cycle and attempts to resurrect it with DevOps will fail.
Another relevant concept as for discussion of DevOps is Cargo Cult science. Cargo cult science comprises practices that
have the semblance of being scientific, but do not in fact follow the scientific method. The term was first used by physicist Richard Feynman during his
1974 commencement address at the California Institute of Technology. Software development provides a fertile ground for cargo cult
science. For example, talented software developers
are often superstars who can follow methods not suitable for "mere mortals" like continuous
delivery. The came can happen with organization, which have some unique circumstances that make continues delivery
successful. For example if "development" consists of just small patches and bug fixes, while most of the codebase remains static.
I think Bill Joy (of BSD Unix, csh, NFS, Java, vi editor fame ) was such a software
superstar when he produced BSD tapes. He actually created vi editor using terminal over 1200 baud
modem (Bill Joy's
greatest gift to man – the vi editor • The Register) -- which is excruciatingly slow. The feat which is difficult if not
impossible for "mere mortal", if just for the lack of patience (transmission speed of 1200 baud modem is close to the speed with
mechanical typewriters can print). The same
is true for Donald Knuth who created singlehandedly a
Fortran complier during a summer, when he was still a student. And
Ken Thompson--who is the father of Unix.
And Larry Wall who created Perl while being
almost blind on one eye. But that does not mean that there practices are scalable.
Brooks book "Mythical man-month" is still relevant today as it was at the moment of publication.
The biggest mistake you can make as a manager or a large and important software
project is to delegate the key functions in the design to mediocre people. You can't replace
software talent with the organization, although organization helps. any attempt to claim otherwise
are cargo cult science. In other word tin software engineering there is no replacement for
Like Agile before DevOps emphasizes a culture of common goals (this time between
"operations" and "developers" with the idea of merging them like in good old times -- the
name was NoOps; which is pretty questionable idea taking into account the complexity of the current IT infrastructure) and get
things done together, presenting itself as new IT culture.
Also details are fuzzy and contradictory. They vary from "prophet" to another. That strongly suggests that like many similar IT
"fashions" before it DevOps just mean "a good idea" (remember an attributed to Mahatma Gandhi acerbic
remark to a self-confident Western journalist: on the question “What do you think of Western civilization?,” he reportedly
have answered, “I think it would be a good idea” :-)
While it seems the IT
management is rushing to embrace
the concept of DevOps (because it justifies further outsourcing under the smokescreen of new terms), nobody agrees on what it actually means. And that creates some skepticism.
DevOps paints a picture of two cultures ("operations" vs. "developers" ) once at odds ("glass datacenter"), now
miraculously working together in harmony. But
first of all there are weak and strong developers. There are weak and strong Unix sysadmins )and strong sysadmin are often are not
bad software developers in their own set of languages, mostly scripting languages.)
The problem of huge, excessive complexity of modern IT infrastructure
can't be changed with some fashionable chair reshuffling. What actually happen is that more talented
members of the team get additional workload. That's why some critics claim that DevOps kills
developers. Meaning "talented developers". And it certainly
can be done but it is easier said then done.
The fact that DevOps is somehow connected with Agile is pretty alarming and suggests that might well be yet another "snake oil"
initiative. With a bunch of
talented an unprincipled salesmen who benefit from training courses, consulting gigs, published books, conferences, and other legal ways to extract
money from lemmings.
The key objection to DevOps is that reliance on super-platforms such as
Amazon cloud or Microsoft Azure could, in
the future potentially intellectually capture the organization and remaining sysadmins (and IT staff
in general), who are increasingly distant from
“nuts and bolts" of operating system and hardware and operate in what essentially is proprietary
of the particular vendor. That converts Unix sysadmin into a flavor on Windows sysadmin
with less nice GUI. In other words they need put all trust in the platforms and detached
themselves from "nuts and
bolts" levels. That means that, in a way, they became as dependent on those platforms as
opiates addicts on their drags.
The also creates a set of DevOps promoters such as cloud providers, who
want to become "gatekeepers", binding users to their technology. Those gatekeepers once they became
non-displaceable makes sure that the organization lost the critical mass of technical IQ
in "low level" (operating system level) infrastructure and can't
abandon them without much pain.
At this point they start to use this dependency to their advantage.
Typically they try to estimate
customers "willingness to pay" and, as a result,
gradually increase the price of their services. IBM was a great practitioner of this fine art in the
past. That's why everybody hated it. VMware is also proved to be quite adept in this art.
Not that Amazon cloud is cheap. It is not,
even is we calculate not only hardware and electricity savings, but also (and rather generously as in one sysadmin
for 50 servers) manpower saving they provide. It is often cheaper to run your own hardware within the
internal cloud then Amazon unless you have a huge peaks.
The same is even more true for VMware. If in the particular organization VMware used as virtualization platform
for Linux (which due to the being open source allows para- virtualization, instead of full
virtualization that VMware implements) , then to talk about savings is possible
only when one is sufficiently drunk. Not that such savings do not exist. They do, but lion
share of them goes directly to VMware, not to the organization which deploy this platform.
The danger is that you are basically willingly allowed to capture yourself and
willing to be part of this ecosystem that is controlled by one single "gatekeeper." Such a
decision creates an
environment in which switching costs can be immense. That's why there is such a
competition amount three top players in cloud provider space for the new enterprise customers. The
first who grab a particular customer is the one to control and can milk such a customer
for a long, long time. Classic DevOps advocates response that "you should have
negotiated better" is false because people not have enough information when they enter
negotiations, and it's too late when they finally will "get it."
DevOps is presented by adherents as all singing, all dancing universal solution to all problems
of mankind, or, at least, of current problems such as the overcomplexity, alienation of developers. paralysis via excessive security,
and red tape that exists in the modern data center.
But the problem is that the level of complexity of modern It is such that the division of labor between sysadmins and developers
is not only necessary it is vital for the success.
Such hope is ignoring the fact that there is no "techno
cure" for large datacenter problems, because those problem are not only technological in nature, but
also reflect complex mix of sociological (the curse of overcomplexity is one of those; see
The Collapse of Complex Societies; neoliberal transformation of the enterprise with switch to outsourcing and contractor labor
is another) and, especially, balance of power between various groups
within the data center. Such as corporate management, developers and operation staff.
Which create pretty
interesting mix from sociological point of view and simultaneously creates a set of internal
conflict and constant struggle for power between various strata of the datacenter ecosystem. From
this point of view DevOps clearly represents political victory for developers, and management
at the expense of other players and first of all system administrators.
In a way,
this idea can be viewed as a replay (under a new name) old Sun idea "network is computer". That does not mean that there
is a rational element in DevOps, as the trend to merge individual servers into some kind of computational superstructure. As
exemplified by Amazon cloud and Azure with their complete switch to virtual instances and attempt to diminish the role of
This tend exist for quite a long time. For example Sun grid was just one and early successful attempts in this direction which
led to creation of the whole class of computer environment which now are known as computational clusters. DevOps can be viewed as
an attempt to create "application clusters".
to the extent that it is connected with advances in virtualization such as Solaris zones and Linux containers it is not that bad
if solid ideas get some marketing push.
But the key question here is: can we really eliminate the sysadmin role in such a complex environment as, for example
modern Linux (as exemplified by RHEL 7) ? Does cloud environment such as Azure creates for developer the possibility to get
rip of sysadmin and do it all by himself/herself.
No, but it is true that with virtualization the level of dependency can be less that in classic datacenter with real servers.
One hugely positive thing from the point of view of developers that in virtual instances in cloud environment they have access to
root. That really free them for shackles imposed by the fact that of "real servers" only sysadmin has access to root, and it
can be grated to the developer only temporary and with some red tape involved.
But for important production servers that is a double-edge sword. If the developer does not fully understand the ropes (and with
the complexity of RHEL 7 he just can't) he/she can bring the server down in no time by a reckless action, repaying various
Sysadmin Horror Stories on a new level. So the level of defense in the form of sysadmin is
now absent. And the environment is no less complex that in classical datacenter. Especially with RHEL 7.
There is another aspect in cloud technology. An organization can not productively use technology platform that is far above technical
capabilities of the staff. And if you try at the end of the day you will suffer unpredictable set
of negative consequences. The elimination of sysadmins lower the level of technical competence in the eEnterprise
dramatically. The sooner the management realize that this is a false gold the better. After all people have only one head and
companies like Red Hat ignore this limitation with the monster complexity of RHEL 7 (which added the systemd daemon)
That effects are only to cost overruns (which usually can be swiped under the carpet). There are additional effects which are
somewhat similar to those predicted by the first two
Softpanorama Laws of Computer Security:
In a long run the level of security of any large enterprise Unix environment
can not be significantly different from the average level of qualification
of system administrators responsible for this environment...
If a large discrepancy between the level of qualification of system
administrators and the level of Computer Security of the system or network
exists, the main trend is toward restoring equilibrium at some, not so distant,
In other words elimination of sysadmin level means that technology can't be utilized fully because of lack of understanding,
including lack of understand of security by the developers.
As DevOps carries certain political dimensions (connection to the neoliberal transformation of the
society, with its fake "cult of creativity", rise of role (and income) of the top 1% and decline of
IT "middle class"). Outsourcing and additional layoffs are probably the most prominent result of introduction of DevOps into real datacenters.
So DevOp often serves as a Trojan horse for the switch to outsourcers and contract labor.
As DevOps carries certain political dimensions (connection to the neoliberal transformation of the
society, with its fake "cult of creativity", rise of top 1% and decline of middle class),
outsourcing and additional layoffs are probably the most prominent result of introduction of DevOps into real datacenters.
The whole idea that by adding some tools VM-run virtual instances and some additional management capabilities introduced by
tools like Puppet you can by successfully computerized and transferred the IT to outsourcers and contractors is questionable.
No amount of ideological
brainwashing you can return datacenter to the good old days in Unix minicomputers when a single person was a masters of all trades -- a
developer, a system administrator and a tester. This is impossible due that current complexity of environment: there is a large gap
in the level of knowledge of (excessively complex) OS by a typical sysadmin (say with RHCE certification) and a typical developer. Attempts to
narrow this gap via tools (and outsourcers) which is the essence fo DevOps movement can go only that far.
But the most alarming tendency is that DevOps serve as a smoke screen for further outsourcing and moving from a traditional data center
to the cloud deployment model with contractors as the major element of the work force. In other words DevOps are used by
corporate brass as another way to cut costs (and the costs of IT in most large manufacturing corporations are already about 1% or
less; so there is not much return achievable for this cost cutting anyway).
From 1992 to 2012 Data Centers already experienced a huge technological reorganization, which might be
called Intel revolution. Which dramatically increased role
of Intel computers in the datacenter, introduced new server form factors such as blades and new
storage technologies such as SAN and NAS. And make Linux the most poplar OS in the datacenter, displacing Solaris, AIX and
In addtiona virtualization became common in Windows world due to
proliferation of VMware instances.
Faster internet and wireless
technologies allowed more distributed workforce and ability for people work part of the week from
home. Smartphones now exceed the power of 1996 desktop. Moreover there was already a distinct trend
of the consolidation of datacenters within the
As the result in multinationals (and all large companies) companies many services, such as email and to lesser extent file storage are already provided
via internal company cloud from central servers. At the same time it became clear that along with
technical challenges "cloud services" create a bottleneck on WAN level and present huge threat to security and privacy. The
driving force behind the cloud is the desire to synchronize and access data from several devices
that people now own (desktop, laptop, smartphone, tablets. In other words to to provide the access to user data
from multiple devices (for example email can be read on smartphone and laptop/desktop). The first such application, ability to view corporate e-mail from the
cell phone, essentially launched Blackberry smartphones into prominence.
In view of those changes managing datacenter remotely became a distinct possibility. That's why
DevOps serves for higher management as a kind of "IT outsourcing manifesto". But with
outsourcing the problem o loyalty comes into forefront.
But there are also other reasons why this mode of work is not reproducible in the in the modern datacenter. The maximum you can achieve (and that's pretty complex undertaking) is to remove some artificial bureaucratic
barriers and make developer more productive because he can control more of the whole process not to
bump with his head into bureaucratic obstacles when he need to make some changes in the environment
in addition to his application.
But it is very important to understand that there are good developers and bad developers. Reckless
power hungry developer, who will utilize those new opportunities 100%, is a great danger. By removing bureaucratic obstacles you also remove
some safeguards against
such developers. There is no free lunch.
Reckless developer is a great danger. By removing
bureaucratic obstacles you also remove safeguards against such developers. There is no free
In other words the idea of even partial elimination of specialization between developers and operation staff is a
dangerous illusion taking into account the complexity of modern operating systems.
Giving a talented developer his own sysadmin and tester makes sense. But making him his own sysadmin
and tester is borderline idiotism. Talented developers is a very rare commodity and their use should be
highly optimized. Re-reading
Mythical Man-month helps to get proper line
of thinking about this problem.
The idea of removing extra red tape also has some validity, but
here you are facing uphill battle against entrenched IT management interests, which try to isolate
itself from people in the trenches with additional management layers (and those "redundant"
management layers is kind of cancer of IT departments, because those middlemen tend to replicate
like rabbits in uncontrollable fashion). High level IT management want technoserfs
to work like slaves on this techno plantation to get their bonuses. And they want to achieve
that without direct contact with those "badly smelling people" creating those middle
management positions that isolate them. But in order to pay those people the herd of technoserfs
needs to thinned, as private profits and bonuses for the brass here comes
before efficiency :-) Generally the question of parasitic rents within IT organization is a
very interesting topic, but it is beyond the scope of this paper.
As large datacenters and large software development problem are
inherently complex. So going over budget and taking twice longer than it should in large software project is more or
less typical course of events. A larger danger here is compete fiasco, like in the past happened
with too enthusiastic idiots who implemented SAP in their companies. This type of failures actually
can crash the whole company and put it out of business.
Any claims that those fundamental issues can be somehow eliminated with some cosmetic
measures ("magic dust") is a typical snake oil salesmen staff. Software development talent, especially high level
software development talent that encompass also strong architectural vision, is a very scarce commodity. In you have no such developers, that chances of successful completion of a complex project are slim. No superficial remedies can help to solve this tremendously difficult
problem. Huge blunders of incompetent high level IT managers is also a reality.
In short the key problem of modern datacenter and modern software applications is lack of
top software talent both on the level of software developers and high IT management. As well as mind boggling complexity
of both the operating systems and applications.
They are probably the most complex artifacts created by mankind. And that means that it is unclear whether the
better solution lies in increasing the complexity or decreasing it. I tend to think that the
latter might be a better path forward (KISS principle). Which puts me on the collision course
with DevOps hoopla. Your mileage may vary.
Like any repackaging effort ,DevOps presents old, exiting technologies as something
revolutionary. They are not. Among measures that DevOps "movement" advocates several old
ideas make sense if implemented without fanaticism. And they could be implemented before the
term DevOps (initially NoOps) was invented. That includes:
The move of applications to VM, including but not limited to light weight virtual machines (Docker). And generally idea of wider adoption of
light weight virtual machines in enterprise environment, inread or in addition to traditional
(and expensive) VMware (I do not understand advantages of running Linux under VMware -- this is
VM tuned to Windows that implements heavy virtualization (virtualization of the CPU), while Linux
being open source requires para-virtualization (virtualization of system calls), actually
available in XEN and derivatives such as Oracle virtual manager).
Wider adoption of scriptable configuration tools and configuration management system (
Infrastructure as code -- IaC ). Truth be told this not necessary to do
via Puppet and similar
complex Unix configuration management packages) as well as continued delivery schemes. Although Puppet is a bad choice as it
is too complex, has
unanticipated quirks and rather low reliability it is a step forward. But it might be two steps back as it contributes to creating "Alice in wonderland"
environment when nobody can troubleshoot the problem because of complex subsystems involved. See
Unix configuration management for details.
Wider usage of version management for configuration files. Such systems as GIT and
Subversion is definitely underutilized in enterprise environment and it can be implemented both
wider and better. None of them fits Unix configuration management requirements perfectly. But some
intermediate tools can be created that compensate deficiencies
Wider use of scripting and scripting languages such as Python, Ruby, shell and (to a much lesser
extent) good old Perl. Of them only Python spans both Unix system administration and
software development areas and as such is preferable (despite Perl being a stronghold of
Unix sysadmins). That actually includes de-emphasizing Java development in favor of Python.
Attempts to cut development cycle to smaller more manageable chunks (although idea
of "continuous delivery" is mostly bunk). Here a lot of discretion is needed as overdoing
this destroys the development process and, what is even more dangerous, de-emphasizes the value
of architectural integrity and architecture as a concept (this is a heritage of Agile, which is
mostly a snake oil).
Some of those innovations ( like Docker) definitely makes more sense than others because preoccupation with
VMware as the only solution for virtualization in enterprise environment is unhealthy from multiple
points of view.
But all of them are adding, not removing complexity. Just the source of it
different (in case of Docker you also can somewhat compensate increase of the complexity due to the switch to
virtual environment by the fact that you can compartmentalize most critical applications in the separate
"zones (using Solaris terminology -- Docker is essentially reimplementation of Solaris zones in
May be the truth lies somewhere in between and selected and "in moderation"
implementation of some of those
technology in the datacenter can be really beneficial. But excessive zeal can hurt not help. In this
sense presence of newly converted fanatics is
a recipe for additional problems. if not a disaster.
I actually saw the situations, when implementation of DevOps
brought corporate IT to a screeching halt, almost complete paralysis when nothing but maintaining
(with great difficulties) of the status quo was achieved for the first two years. Outsourcing in
those cases played major negative role as new staff needs large amount of time and effort to understand the
infrastructure and processes and attempt to "standardize" some servers and services. Often failing dismally due to complex interrelations between them and applications in place.
While the current
situation in the typical large datacenter is definitely unsatisfactory from many points of view, the first principle should be
"do no harm". In many cases it might sense to announce the switch to DevOps, force
people to prepare to it (especially discard old servers and unused applications in favor of virtual instances on private or public
cloud) and then cancel the
implementation. You probably can probably achieve 80% of positive effect this way, avoiding 80% of
negative effects. Moving datacenter to a new location can also tremendously help and can be
used instead of DevOps implementation :-)
While the current situation in a typical large datacenter is definitely unsatisfactory from many points of
view, the first principle should be "do no harm".
Again, the main problem with DevOps hoopla is that, if implemented in full scale, it substantially increases the complexity of environment.
Both virtual machines and configuration management tools provide additional "levels of indirection"
which makes troubleshooting more complex and causes of failures more varied and complex. That
represents problems even for seasoned Unix sysadmins to say nothing for poor developers who are
thrown into this water and asked to swim.
DevOps methodology implemented in modest scope does provide some of the claimed benefits (and some
level of configuration management in a large datacenter is a must). The question is what if the
optimal solution for this and how not to overdo it. But in a typical DevOps implementation this
question somehow does not arise and it just degenerates into another round of
centralization and outsourcing which make the situation worse. Often much worse to the level when It
really became completely dysfunctional. I saw this affect of DevOps implementation in large
So it should be evaluated on case by case basic, not
as panacea. As always much depends of the talent of the people who try to implement it. Also
change in a large datacenter is exceedingly difficult and often degenerates into what can be called
one step forward, two steps back". For example, learning such tools as Puppet or Chief requires quite a lot of effort for rather questionable
return on investment as complexity precludes full utilization of the tool and it is downsized to basic
staff. So automation using them is a mixed blessing.
Similarly move to lightweight VM (and virtual servers in general) which is a part of DevOps
hype are easier to deploy, but load management of multiple servers running on the same box is an additional
and pretty complex
task. Also VMware that dominates VM scene is expensive (which mean that the lion share of saving are going to VMware,
not to the enterprise which deploys it ;-) and is a bad VM for Linux. Linux needs para-virtualization not
full CPU virtualization that VMware, which was designed for Windows, offers (with some interesting optimization tweaks). Docker, which is a rehash of the idea of Solaris zones is a better deal, but
is a pretty new for Linux technology and it has its own
limitations. Often severe as this is a light weight VM.
Cost wise DevOps (typically implemented with the help of outsourcers) is a rather expensive proposition which provides both higher complexity and lower reliability.
Outsourcing costs are low usually only at the beginning as most vendors perform "bait and switch"
underbidding their services and then try to compensate this by inflating operating costs, whenever
possible, especially in areas were local "know how" disappeared.
So benefits, is they exist, are not automatic, but are dependent of the qualification of people
implementing this initiative.
VM problems while relatively rare still are unwelcome addition to the set of existing problem; also if hypervisor or server goes down all VMs go down
Using VM is very similar to replacing enterprise servers with desktops, but at a higher cost (BTW "disposable desktops" is an interesting
alternative to VMware VMs cost wise). The cost of "decent" desktop ($500-$800) now is one tens of the cost
of a "decent" enterprise server ($6K-$8K). but ten desktops can definitely do more computation wise.
Adding to the mix management tools such as Puppet works for multiple desktops too.
DevOps was a reaction to the problem in companies who now need to maintain thousands of
servers, such as Netflix, Amazon, Facebook and Google.
So DevOps is
intrinsically connected with attempts of merging servers into some computational superstructure and automation of system administration using Unix configuration
management tools. BTW that does not mean that Puppet or Chief are the proper way to do that (the
king might still be naked ;-). They just
bask in DevOps glory. And try to sustain and maintain the DevOps hype for fun and profit.
The second part is an attempt to break isolated and fossilized silos in which developers and
operation staff live in large corporation. Sometimes with almost without interaction. So in a
way this as an over-reaction to a dysfunctional ops organization (as
Adrian Cockcroft admitted in his
comment.), which actually clearly demonstrates that he does not understand how complex
organization operated (his idea of abolishing operations is just an attempt to grab the power by
"developer class") and never read the book by Charles Perrow (Complex
Organizations A Critical Essay). In complex organization technological issues re always
intersect with power struggle and effects of neoliberalism on IT and corporate environment in
general. Blowback of excessive outsourcing of IT is hitting large corporations and is a part of
problem that IT organizations experience now. DevOps in this sense is a part of the problem and not
a part of the solution. You will never read about this this in "DevOps hoopla" style of books
that proliferate (Amazon lists around hundred of books on this topic; most junk). In reality
globalization while solving one set of problem creates another and her IT is both the victim
and the part of this "march of neoliberalism" over globe:
Information and communications technologies are part of the infrastructure of globalization in
finance, capital mobility and transnational business. Major changes in the international economic
landscape are intertwined and contemporary accelerated globalization is in effect a package deal
that includes informatization (applications of information technology), flexibilization (changes
in production and labour associated with post-Fordism), financialization (the growing importance
of financial instruments and services) and deregulation or liberalization (unleashing market
forces). This package effect contributes to the dramatic character of the changes associated with
globalization, which serves as their shorthand description.
A good early overview why DevOps emerged (when it was still called NoOps) was made in 2012 by
Mile Loukidis, who wrote historically important paper of the subject.
Adrian Cockcroft’s article about
Netflix ignited a controversy that has been smoldering for some months. John Allspaw’s
detailed response to Adrian’s article makes a key
point: What Adrian described as “NoOps” isn’t really. Operations doesn’t go away.
Responsibilities can, and do, shift over time, and as they shift, so do job descriptions. But no
matter how you slice it, the same jobs need to be done, and one of those jobs is operations. What
Adrian is calling NoOps at Netflix isn’t all that different from Operations at Etsy. But that
just begs the question: What do we mean by “operations” in the 21st century? If NoOps is a
movement for replacing operations with something that looks suspiciously like operations, there’s
clearly confusion. Now that some of the passion has died down, it’s time to get to a better
understanding of what we mean by operations and how it’s changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no
distinction between dev and ops. If you developed, you operated. You mounted the tapes, you
flipped the switches on the front panel, you rebooted when things crashed, and possibly even
replaced the burned out vacuum tubes. And you got to wear a geeky white lab coat. Dev and ops
started to separate in the ’60s, when programmer/analysts dumped boxes of punch cards into
readers, and “computer operators” behind a glass wall scurried around mounting tapes in response
to IBM JCL. The operators also pulled printouts from line printers and shoved them in labeled
cubbyholes, where you got your output filed under your last name.
The arrival of minicomputers in the 1970s and PCs in the ’80s broke down the wall between
mainframe operators and users, leading to the system and network administrators of the 1980s and
’90s. That was the birth of modern “IT operations” culture. Minicomputer users tended to be
computing professionals with just enough knowledge to be dangerous. (I remember when a new
director was given the root password and told to “create an account for yourself” … and promptly
crashed the VAX, which was shared by about 30 users). PC users required networks; they required
support; they required shared resources, such as file servers and mail servers. And yes, BOFH (“Bastard
Operator from Hell”) serves as a reminder of those days. I remember being told that “no one”
else is having the problem you’re having — and not getting beyond it until at a company meeting
we found that everyone was having the exact same problem, in slightly different ways. No wonder
we want ops to disappear. No wonder we wanted a wall between the developers and the sysadmins,
particularly since, in theory, the advent of the personal computer and desktop workstation meant
that we could all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important
websites. As companies and computing facilities grew larger, the fire-fighting mentality of many
system administrators didn’t scale. When the whole company runs on one 386 box (like O’Reilly in
1990), mumbling obscure command-line incantations is an appropriate way to fix problems. But that
doesn’t work when you’re talking hundreds or thousands of nodes at Rackspace or Amazon. From an
operations standpoint, the big story of the web isn’t the evolution toward full-fledged
applications that run in the browser; it’s the growth from single servers to tens of servers to
hundreds, to thousands, to (in the case of Google or Facebook) millions. When you’re running at
that scale, fixing problems on the command line just isn’t an option. You can’t afford letting
machines get out of sync through ad-hoc fixes and patches. Being told “We need 125 servers online
ASAP, and there’s no time to automate it” (as
Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn’t surprising. One of the
themes of O’Reilly’s Velocity Conference is
“Infrastructure as Code.” If you’re going to do operations reliably, you need to make it
reproducible and programmatic. Hence virtual machines to shield software from configuration
issues. Hence Puppet and
Chef to automate configuration, so you know every machine has an identical software
configuration and is running the right services. Hence Vagrant
to ensure that all your virtual machines are constructed identically from the start. Hence
automated monitoring tools to ensure that your clusters are running properly. It doesn’t matter
whether the nodes are in your own data center, in a hosting facility, or in a public cloud. If
you’re not writing software to manage them, you’re not surviving.
Furthermore, as we move further and further away from traditional hardware servers and
networks, and into a world that’s virtualized on every level, old-style system administration
ceases to work. Physical machines in a physical machine room won’t disappear, but they’re no
longer the only thing a system administrator has to worry about. Where’s the root disk drive on a
virtual instance running at some colocation facility? Where’s a network port on a virtual switch?
Sure, system administrators of the ’90s managed these resources with software; no sysadmin worth
his salt came without a portfolio of Perl scripts. The difference is that now the resources
themselves may be physical, or they may just be software; a network port, a disk drive, or a CPU
has nothing to do with a physical entity you can point at or unplug. The only effective way to
manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming
code as early as the late ’80s; indeed, Perl was designed as a programming language for
automating system administration. It didn’t take long for leading-edge sysadmins to realize that
handcrafted configurations and non-reproducible incantations were a bad way to run their shops.
It’s possible that this trend means the end of traditional system administrators, whose jobs are
reduced to racking up systems for Amazon or Rackspace. But that’s only likely to be the fate of
those sysadmins who refuse to grow and adapt as the computing industry evolves. (And I suspect
that sysadmins who refuse to adapt swell the ranks of the BOFH fraternity, and most of us would
be happy to see them leave.) Good sysadmins have always realized that automation was a
significant component of their job and will adapt as automation becomes even more important. The
new sysadmin won’t power down a machine, replace a failing disk drive, reboot, and restore from
backup; he’ll write software to detect a misbehaving EC2 instance automatically, destroy the bad
instance, spin up a new one, and configure it, all without interrupting service. With automation
at this level, the new “ops guy” won’t care if he’s responsible for a dozen systems or 10,000.
And the modern BOFH is, more often than not, an old-school sysadmin who has chosen not to adapt.
James Urquhart nails it when
he describes how modern applications, running in the cloud, still need to be resilient and fault
tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But he notes
that those features, formerly provided by the IT/operations infrastructures, now need to be part
of the application, particularly in “platform as a service” environments. Operations doesn’t go
away, it becomes part of the development. And rather than envision some sort of uber developer,
who understands big data, web performance optimization, application middleware, and fault
tolerance in a massively distributed environment, we need operations specialists on the
development teams. The infrastructure doesn’t go away — it moves into the code; and the people
responsible for the infrastructure, the system administrators and corporate IT groups, evolve so
that they can write the code that maintains the infrastructure. Rather than being isolated, they
need to cooperate and collaborate with the developers who create the applications. This is the
movement informally known as “DevOps.”
Amazon’s EBS outage last year demonstrates
how the nature of “operations” has changed. There was a marked distinction between companies that
suffered and lost money, and companies that rode through the outage just fine. What was the
difference? The companies that didn’t suffer, including Netflix, knew how to design for
reliability; they understood resilience, spreading data across zones, and a whole lot of
reliability engineering. Furthermore, they understood that resilience was a property of the
application, and they worked with the development teams to ensure that the applications could
survive when parts of the network went down. More important than the flames about Amazon’s
services are the testimonials of how intelligent and careful design kept applications running
while EBS was down.
ChaosMonkey is an excellent, if extreme, example of a tool to ensure that a complex
distributed application can survive outages; ChaosMonkey randomly kills instances and services
within the application. The development and operations teams collaborate to ensure that the
application is sufficiently robust to withstand constant random (and self-inflicted!) outages
On the other hand, during the EBS outage, nobody who wasn’t an Amazon employee touched a single
piece of hardware. At the time, JD Long tweeted
that the best thing about the EBS outage was that his guys weren’t running around like crazy
trying to fix things. That’s how it should be. It’s important, though, to notice how this differs
from operations practices 20, even 10 years ago. It was all over before the outage even occurred:
The sites that dealt with it successfully had written software that was robust, and carefully
managed their data so that it wasn’t reliant on a single zone. And similarly, the sites that
scrambled to recover from the outage were those that hadn’t built resilience into their
applications and hadn’t replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to
the application itself, we’re also seeing a redistribution of costs. It’s a mistake to think that
the cost of operations goes away. Capital expense for new servers may be replaced by monthly
bills from Amazon, but it’s still cost. There may be fewer traditional IT staff, and there will
certainly be a higher ratio of servers to staff, but that’s because some IT functions have
disappeared into the development groups. The bonding is fluid, but that’s precisely the point.
The task — providing a solid, stable application for customers — is the same. The locations of
the servers on which that application runs, and how they’re managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds
like Amazon’s, private clouds, traditional colocation, and building their own infrastructure.
It’s hard to beat Amazon if you’re a startup trying to conserve cash and need to allocate or
deallocate hardware to respond to fluctuations in load. You don’t want to own a huge cluster to
handle your peak capacity but leave it idle most of the time. But Amazon isn’t inexpensive, and a
larger company can probably get a better deal taking its infrastructure to a colocation facility.
A few of the largest companies will build their own datacenters. Cost versus flexibility is an
important trade-off; scaling is inherently slow when you own physical hardware, and when you
build your data centers to handle peak loads, your facility is underutilized most of the time.
Smaller companies will develop hybrid strategies, with parts of the infrastructure hosted on
public clouds like AWS or Rackspace, part running on private hosting services, and part running
in-house. Optimizing how tasks are distributed between these facilities isn’t simple; that is the
province of operations groups. Developing applications that can run effectively in a hybrid
environment: that’s the responsibility of developers, with healthy cooperation with an operations
The use of metrics to monitor system performance is another respect in which system
administration has evolved. In the early ’80s or early ’90s, you knew when a machine crashed
because you started getting phone calls. Early system monitoring tools like HP’s OpenView
provided limited visibility into system and network behavior but didn’t give much more
information than simple heartbeats or reachability tests. Modern tools like DTrace provide
insight into almost every aspect of system behavior; one of the biggest challenges facing modern
operations groups is developing analytic tools and metrics that can take advantage of the data
that’s available to predict problems before they become outages. We now have access to the data
we need, we just don’t know how to use it. And the more we rely on distributed systems, the more
important monitoring becomes. As with so much else, monitoring needs to become part of the
application itself. Operations is crucial to success, but operations can only succeed to the
extent that it collaborates with developers and participates in the development of applications
that can monitor and heal themselves.
Success isn’t based entirely on integrating operations into development. It’s naive to think
that even the best development groups, aware of the challenges of high-performance, distributed
applications, can write software that won’t fail. On this two-way street, do developers wear the
beepers, or IT staff? As Allspaw points out, it’s important not to divorce developers from the
consequences of their work since the fires are frequently set by their code. So, both developers
and operations carry the beepers. Sharing responsibilities has another benefit. Rather than
finger-pointing post-mortems that try to figure out whether an outage was caused by bad code or
operational errors, when operations and development teams work together to solve outages,
a post-mortem can
focus less on assigning blame than on making systems more resilient in the future. Although we
used to practice “root cause analysis” after failures, we’re recognizing that finding out the
single cause is unhelpful. Almost every outage is the result of a “perfect storm” of normal,
everyday mishaps. Instead of figuring out what went wrong and building procedures to ensure that
something bad can never happen again (a process that almost always introduces inefficiencies and
unanticipated vulnerabilities), modern operations designs systems that are resilient in the face
of everyday errors, even when they occur in unpredictable combinations.
In the past decade, we’ve seen major changes in software development practice. We’ve moved
from various versions of the “waterfall” method, with interminable up-front planning, to “minimum
viable product,” continuous integration, and continuous deployment. It’s important to understand
that the waterfall and methodology of the ’80s aren’t “bad ideas” or mistakes. They were
perfectly adapted to an age of shrink-wrapped software. When you produce a “gold disk” and
manufacture thousands (or millions) of copies, the penalties for getting something wrong are
huge. If there’s a bug, you can’t fix it until the next release. In this environment, a software
release is a huge event. But in this age of web and mobile applications, deployment isn’t such a
big thing. We can release early, and release often; we’ve moved from continuous integration to
continuous deployment. We’ve developed techniques for quick resolution in case a new release has
serious problems; we’ve mastered A/B testing to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations
staff. Operations groups are adopting, and in many cases, leading in the effort to implement
these changes. They’re the specialists in resilience, in monitoring, in deploying changes and
rolling them back. And the many attendees, hallway discussions, talks, and keynotes at O’Reilly’s
Velocity conference show us that they are adapting. They’re learning about adopting approaches to
resilience that are completely new to software engineering; they’re learning about monitoring and
diagnosing distributed systems, doing large-scale automation, and debugging under pressure. At a
recent meeting, Jesse Robbins described scheduling EMT training sessions for operations staff so
that they understood how to handle themselves and communicate with each other in an emergency.
It’s an interesting and provocative idea, and one of many things that modern operations staff
bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and
bleeding-edge; now, it’s expected. But we haven’t taken it far enough. We’re still learning how
to monitor systems, how to analyze the data generated by modern monitoring tools, and how to
build dashboards that let us see and use the results effectively. I’ve joked about “using a
Hadoop cluster to monitor the Hadoop cluster,” but that may not be far from reality. The amount
of information we can capture is tremendous, and far beyond what humans can analyze without
techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient
protocols for the web, like SPDY. Operations is
involved, more than ever, in tuning the performance of operating systems and servers (even ones
that aren’t under our physical control); a lot of our “best practices” for TCP tuning were
developed in the days of ISDN and 56 Kbps analog modems, and haven’t been adapted to the reality
of Gigabit Ethernet, OC48* fiber, and their descendants. Operations groups are responsible for
figuring out how to use these technologies (and their successors) effectively. We’re only
beginning to digest IPv6 and the changes it implies for network infrastructure. And, while I’ve
written a lot about
building resilience into applications, so far we’ve only taken baby steps. There’s a lot
there that we still don’t know. Operations groups have been leaders in taking best practices from
older disciplines (control systems theory, manufacturing, medicine) and integrating them into
And what about NoOps? Ultimately, it’s a bad name, but the name doesn’t really matter. A group
practicing “NoOps” successfully hasn’t banished operations. It’s just moved operations elsewhere
and called it something else. Whether a poorly chosen name helps or hinders progress remains to
be seen, but operations won’t go away; it will evolve to meet the challenges of delivering
effective, reliable software to customers. Old-style system administrators may indeed be
disappearing. But if so, they are being replaced by more sophisticated operations experts who
work closely with development teams to get continuous deployment right; to build highly
distributed systems that are resilient; and yes, to answer the pagers in the middle of the night
when EBS goes down. DevOps.
Photo: Taken at IBM’s headquarters in Armonk, NY. By Mike Loukides.
There are many types of members, each with its own motivation.
First of all there are the ones who have been moved to do so out of a genuine spiritual aspiration,
prepared to sacrifice.
Other members may have been searching for a truth behind the dreadful aspects of life around
them and an escape therefrom.
People who wrestle with a psychological problem seek (unconsciously) a key or solution to
it in a particular teaching or discipline.
Then there are those that did so out of a feeling of isolation and loneliness. Lack of true
companionship may have made them long to be taken up in a fraternity of kindred souls.
People will be drawn to a brotherhood whose aims are within their reach of comprehension and
nearest to their hearts. A religious background may make them veer towards a sect that accomodates
nostalgia, like the Jehova's Witnesses.
For younger people it may be an escape from having to enter into a harsh materialistic society.
They are in an idealistic life-cycle, prepared to give up their study and career to a worthy cause.
Some older people can never make up their minds and wander from one cause to another.
Finally there are those who wrestle with a father/mother complex. The childhood's need for
such archetypal figure may persist in later life. It is transferred subsequently to the master,
who then holds sway over him/her.
The weaker the individual's independance, the more will he be tied to the group. Members who understand
group-mechanisms, prepared to cope with them in order to direct their attention to the spirit, will
benefit most as they are selective in picking up the cream of what is given and taking the rest with
a grain of salt.
New religious movements arise usually around a father/mother figure who has gained authority after
receiving a special revelation, communication, truth or insight. His charisma will vouchsafe loyal
followers, even if his lifestyle may give rise to severe doubts to some. He may boost his prestige
by claiming to follow the footsteps of a an esteemed spiritual teacher, represent an esoteric tradition,
be of noble descent, or channel the wisdom of a great mind. (Eckankar's Paul Twitchell is
the last in the lineage of 970 "Eckmasters")
He/she represents an archetype in members' subconscious minds. That of a wise father, or mother.
As such he/she will have a compelling influence on followers who project their father/mother complex
Alternatively women may fall in love with the leader, worship him, exert themselves to cater after
his wishes and whims. They will try to stay in his viscinity, make themselves indispensable and slowly
take control of the movement. Jealousy amongst them will make things even worse and split the ranks.
The psychological make-up of a guru may be generalized as follows:
Difficult youth. Self-chosen isolation at childhood. Introvert. Therefore not used to share
inner life with friends.
Narcissistically absorbed in inner mental processes.
Undergone a traumatic experience.
Authoritarian attitude. Élitist and anti-democratic.
Attracting disciples rather than friends on account of the fact that they never learnt to
exchange thoughts in childhood.
Jeffrey Masson (see below) has this to say about gurus:
Every guru claims to know something you cannot know by yourself or through ordinary channels.
All gurus promise access to a hidden reality if only you will follow their teaching, accept their
authority, hand your life over to them. Certain questions are off limits. There are things you cannot
know about the guru and the guru's personal life. Every doubt about the guru is a reflection of your
own unworthiness, or the influence of an external evil force. The more obscure the action of the
guru, the more likely it is to be right, to be cherished. Ultimately you cannot admire the guru,
you must worship him. You must obey him, you must humble yourself, for the greater he is, the less
you are - until you reach the inner circle and can start abusing other people the way your guru abused
you. All this is in the very nature of being a guru.
Nature seems to instill in a person, faced with a mission, great task, or challenge, a feeling
of superiority, unsurmountable optimism, and enormous self-esteem, bordering on an inflated ego,
to accomplish what is needed. This drive is reminiscent of the reckless impetus of the adolescent.
Having reached maturity a person may feel "chosen" - impelled to forge ahead with vigor and inspire
others. Undaunted in the face of obstacles and criticism, it is as if a cloak of invulnerability
is laid on his/her shoulders.
Similarly an artist may be driven by a compulsion to express an inner content. He will be prepared
to sacrifice everything to give way to his creative impulse. Fortunately his sacrifice does not involve
more than the people immediately around him.
Not so with the leader. The number of his followers may grow to considerable proportions. Nature
is not concerned whether his sense of superiority has any real foundation. The inflated ego is more
or less instinctively driven towards a goal.
Although attaining heights no one would have thought conceivable of that person, when the hour of
truth has come events may prove that he has overreached himself, disregarded good advice, or lost
complete sense of reality. The result may be either catastrophe, or the uncritical followers may
be saddled up with a heritage built on quicksand - on a flight of fancy without actual foundation.
This applies to many fields of human endeavour (Hitler), but specially in the treacherous
domain of the spirit.
Discipline - nausea
The teacher may come to the conclusion that unless his followers change fundamentally - undergo a
catharsis, or transformation - they will never be able to move forward. He/she regards them as being
"asleep" (Jesus, Gurdjieff). Unless drastic measures are employed they will not wake up. To
jolt them out of their complacency great sacrifices are demanded. Jesus asked a rich young
man to give up all his worldly possessions (S.Matthew 19:21) before following him. Masters in
Zen Buddhism, or Gurdjieff, made novices undergo a harsh regime in order to crack open and attain
a different state of mind.
This I can have no quarrel with, if it is done against a background
of compassion. If the unselfish motive disappears, or commercial considerations become dominant,
the harsh discipline may become morbid and degrading. Having lost his dedication the teacher may
become nauseated by the mentality and sheepishness of his followers, and in cases derive a sadistic
delight in tormenting them.
In recent years reports are brought out about sexual violation of members by guru's, leaders
and....bishops! Another example of authority being abused.
The path of a guru is like that of a razor's edge. He may so easily succumb to the temptation
of exploiting the power he has attained over his followers. Financial irresponsibility, abuse of
followers, reprehensible sexual behaviour......... mass suicide, it is all within his reach once
he has overstepped boundaries.
During his lifetime the leader will act as a moderator and steer the movement. He will re-interprete
his teachings as he sees fit from the responses he receives. The death of the founder marks a turning
point. His teachings will become inflexible, as no one dares to temper with them as he did himself.
The élan disappears, rigidity takes over, unless another figure arises that leads the movement in
a different direction, for better or for worse (St.Paul).
The more secret(ive) the leader's sayings the better. Pronouncements are characterized by great
certainty and authority as if it were the word of God. In some cases it is presented as such. By
his special way of delivery and presentation it may escape the audience that similar wisdom may be
found in any book on spirituality nowadays found in the bookshop around the corner.
Whether the guru bases his wise words on actual experience or on hearsay is difficult to ascertain.
In general it may be said that the more mystifying his teachings the stronger their appeal. After
all it is beyond reason and should appeal only to the heart.
An exception should be made for true mystical literature based on inner experience which can hardly
be expected to appeal to the intellect, but be appreciated intuitively, especially by those who had
Members may adopt fresh meanings to words, talk to each other in a jargon that the outsider can
hardly follow (group-speak). The result being an inability to relate in speech, or explain
new concepts to the outsider (Fourth Way).
(This may be best understood in other fields: help-programs of software, pop-up windows, warning-messages,
not to speak of manuals for installing hardware, drawn up by boffins, are a nightmare to most users!)
Another characteristic is to lift out of context one aspect of religious truth and make it absolute.
Such key truth will overshadow all other aspects of faith.
It may be:
prophecy, as we have seen with Jehova's Witnesses, Adventists and countless sects;
baptism at an adult age, important to Baptists;
charity in the case of the Salvation Army (but who could blame them for this?);
channeling in New Age/Spiritualistic movements,
etc. When this occurs other significant facets of faith are pushed to the background.
Such partial truths are often heralded as the result of a search for knowledge. The motto "Knowledge
is power" is used to suggest that the statements are objective, scientific, or historical facts.
Actually they cannot stand the touchstone of the merest critical scrutiny.
Authorities may be paraded to back-up such claims. They have either never been heard of, cannot be
considered impartial, or their pronouncements have been lifted out of context. The discussion about
the veracity of evolution is full of such red herrings.
4. Uniqueness of the movement
Movements will extol usually their superiority over others. After all there should be a strong
reason to select that particular group. Some present themselves as being the sole way towards salvation,
being God's chosen people. Others make a promise of a benefit that is only reserved for members of
that sect. To avert attention some pride themselves of not having a teaching, or for their openness
and democratic rules.
In short new movements will advance a variety of reasons for their uniqueness. Herewith a few:
Our holy book is the word of God.
Never before has mankind been offered this discipline/interpretation/insight of our leader.
Do not pour new wine in old bottles.
God's, or a celestial, new message to mankind for this particular time.
Impending disaster (pollution, atomic explosion, collision) calls for drastic measures. Salvation
only reserved for faithful members.
Fresh interpretation of holy book thanks to fresh insight/revelation of founder.
Esoteric tradition, once accessible to adepts only, now revealed.
New channel for teachings of esoteric lodge/brotherhood.
New doctrine/insight based on latest scientific discoveries, reveals truth.
Only those following this particular work on self, discipline, or belief, will reach eternity,
be released from earth's satanic attraction, cycle of rebirths, etc.
Preparatory group to make way for the coming of the new messiah/world-teacher/avatar
Noteworthy is the vehemency with which groups stress differences between each other. The closer
movements share an outlook the more virulent the attacks on their rivals become, seemingly more than
on groups which follow a completely different belief.
Eric Hoffer writes in his 'The True Believer': "true believers of various hues ....view
each other with mortal hatred and are ready to fly at each other's throat..."
This manifests itself specially when groups split. In Christianity one could not steep low enough
to attack other followers of Christ, who held a slightly different opinion. It resulted in persecution
of heretics, burning of early Christian literature, and disastrous wars.
Despite their peaceful appearance relatively new spiritual movements like Theosophy, Rosicrucianism,
etc., following splits, exert themselves in accusations against former comrades.
Attacks against belief in paranormal phenomena, for instance by CSICOP, are reminiscent
of the zeal of a Christian crusade, be it that they have their roots in humanism and its desperately
clinging to a rationalistic/materialistic outlook on life current at the beginning of this century.
Consequently the groups of these 'evangelists of rational enlightenment' have similar behavioural
patterns and vehemency as sects.
5. Probation and conversion
Certain sects are only too eager to accept individuals. They may have high entrance fees. Or their
members are swayed by zeal to convert.
Many movements will put up a barrier by means of an initiation to test the applicant's fitness
to become part of the group. Henceforth they will play an important pioneer-part in the foretold
future. Having reached such coveted stage members will not fail to follow what they are being told
for fear of expulsion.
The new member may undergo a conversion, gaining a completely new insight in the meaning
of life, see it in a way the sect does. His previous life with all its relationships has become meaningless.
He may have turned himself inside out by a confession of his previous "sins". His conversion
is marked by a feeling of peace, happiness and transcendence.
6. Failure of predictions
Common belief in a prophecy will be a strong binding force. One of the principal attractions of
the first Christian sects was that they offered salvation from a threatening disaster. That being
the end of the world. Only the baptized would await a glorious future. Sects like the Jehova's Witnesses
have taken over this succesful formula.
Christians have had to come up with all sorts of arguments to explain away the unfulfilled prediction
of their founder regarding the end of the world: "This generation shall not pass away, till all these
things be accomplished." (S.Matthew 24:34). One of the lame excuses being that this prediction concerns
the fall of Jersusalem only. However, all prophecies in the New Testament in this respect suggest
that the impending doom was to be expected in their lifetime.
Jehova's Witnesses have taken the risk of being more specific in their predictions. Older
members, who built their faith on them, have had the humiliating experience of having had to explain
away various times in their lives the failure of the outcome of their forewarnings.
But predictions are not limited to the religious faiths. The New Age movements use this
shared belief in portents as well. For more than sixty years an imminent landing of UFO's has been
predicted. Various cults claimed in vain to be their first contactees.
In other movements the second coming of Christ was a main feature (Benjamin Creme). In Theosophy
a Messenger was expected from 1975 onward.
The uncritical believers in Edgar Cayce's trance sayings put weight on his predictions
of cataclysms (photo Edgar Cayce).
Nostradamus' (photo) obscure astrological foresayings have captured the minds of
people for centuries. Each time his verses were interpreted again to suit the circumstances. In hindsight
some of his quatrains seem to have relevance to the catastrophe of the destroyed World Trade Center.
Quatrains I, 87 - IX, 92 and X, 59 may refer to skyscrapers in New York involved in a terrible explosion.
Sociologists have observed that, failure of prediction results in quite the opposite effect on
believers. Contrary to what one would expect it may cause a rally amongst members. Failure is blamed
on a misunderstanding, or a faux pas by members. To counteract ridicule they tend to stick together
more than ever.
Of course there is a limit. According to a social survey, when predictions fail to materialize
three times in a row members are bound to stop, reflect and draw conclusions.
The shattering of such false hopes comes as a severe blow and may mark the beginning of the end of
One wonders in this respect how many members of the People Forever International sect promoting
physical immortality for its followers would have to die before their groups would break up in disappointment.
(Since I wrote this ten years ago I have been informed that indeed members have died and the movement
broke up in 1998!)
Yet, we see from the Jehova's Witnesses that skilful manoeuvring may off-set unfulfilled prophecies.
To what extremes such believes can lead shows the mass suicidal action of the Heaven's Gate
sect and later in Uganda. Such tragic endings are the result of various contributing factors, which
are beyond the scope of this article.
7. Belief versus intellect/Secrets
Often disciplines followed in spiritual movements have the effect of a lowering of the threshold
to the unconscious mind. Suggestion will begin to play an important part. Precepts are being experienced
as the truth, sacrosanct and sure. There is no element of doubt anymore about assumptions and speculation,
although actually they lack any factual foundation.
Absolute belief that the Bible is God's word is the cornerstone of most orthodox Christian sects.
In Islam the Koran is supposed to contain the word of Allah.
Intellectual analysis of faith is tentamount to heresy.
The ideal breeding ground for convictions are mass-gatherings. During mass gatherings, such as
congresses, members are stirred up to an euphoria, the effect of which may linger on for weeks. This
is the precise period of time for leaders, or committees, to announce fresh sectarian measures, postulate
incredible notions/prophecies, call for further sacrifices, etc. etc. It will all be accepted unquestioningly.
Only at a later date, when the euphoria has worn off, will one start to wonder about what was decided.
Spiritual movements often hide a corpse in their closet. It may be a part of the history of the
movement, details about the hidden life of the leader, or a once revered figure. Things may have
been written by them one does not like to be reminded of. A fight, quarrel, full of vehemence and
hatred, may have led to a split.
There are so many examples that a long list could be drawn up of the many concealed secrets of
Whereas in most movements the works of the leader are almost known by heart, Jehova's Witnesses
hardly know of the existence of the seven volumes of writings Studies in the Scriptures of
their founder Charles Taze Russell (1852-1916). Some of his opinions are such cause of embarrassment
that they are not deemed worth reading nowadays.
Eventually a renegate member will reveal such secrets in writing. Frantic denials and counter
accusations by those in charge presently will follow almost automatically. These are usually accepted
in gratitude by devotees, who cannot get over the shock of such revelations.
8. Common practice, work and ritual
Communal singing, ritual and (incomprehensible) practices (Freemasonry) are strong binding factors.
The more irrational they are, the better. Others are a special food regime, the change of name, clothing,
or a common aversion.
Joint work for the benefit of the group gives the feeling of a common endeavour and unites the participants.
So does proselytization in the streets, menial work of construction and renovation of premises. There
is a thin line between true participation and exploitation, however.
Dubious was the practice, common in the seventies, to incite members to criticize one of them
to an extent that he/she would break down under the weight of often absurd allegations and insults,
resulting in a brainwash effect.
9. Sacrifices, financial secrecy, favours to the rich.
Finances are always a ticklish matter. Human groups always wish to grow. Finances are important.
Accountability is often not considered appropriate. Danger arises that members of the inner circle
become lax in expenditure of members' contributions. Ambitious schemes call for a constant need for
funding. This is the ideal breeding ground for favours to wealthy members. Those who contribute generously
stand more chance to be taken in confidence and admitted to the inner circles. Often, as a proof
of loyalty, extraordinary sums of money are demanded.
Degrees of initiation may be dependent on one's years of loyalty to the group. In Eckankar
up to 8 degrees are given. However, if one fails to pay membership's fees for some time, degrees
of initiation may be stripped off again.
Next to financial contributions members will often be expected to offer services to the group.
However, if they also have to work for practically nothing in commercial enterprises it becomes dubious.
Movements that gather wealth at the expense of their members are questionable. Seldom or never requests
for return of contributions/investments are honoured.
10. Unquestioning leadership, reprehensible behaviour amongst members
Man in a herd may not show the best side of his nature. Unconscious drives may reign his/her behaviour.
This is applicable especially in circumstances that man strives for the spiritual. Heshe may tend
to show split-personality behaviour. On one hand the spiritual personality which is supposed to have
come to terms with his animal nature. It is wise, friendly and compassionate on the outside. In the
shadows lurks the personality that has been forced into the background, still ridden with all the
expulsed human frailties. In moments of weakness it will see its chance to play hideous tricks. It
will do so without being noticed by the person involved. The result being: uncharitable behaviour,
envy, malicious gossip, hypocrisy, harsh words, insensitivity, unfounded criticism and even worse,
not expected from such charismatic figure. It is one of the main reasons for people leaving a particular
group in great disappointment.
It is not often realized that, like other human groups, spiritual movements behave like organisms.
Group-psychological processes manifest which are sometimes not unlike those in primitive societies.
There is the pecking order, the alpha members, and also the group-instinct directed against similar
groups. Aggression goes unnoticed and is tolerated when an acceptable common goal is provided. For
instance hostility against an individual outside the group, or a critical member inside. This has
the effect of strengthening ties within the group like in the animal world.
If leadership loses contact with its members it will have to exert greater discipline. Deviating
opinions cannot be tolerated anymore. Persons who hold them are seen as traitors. Acting against
them, preferably in secret, is the only way out for the leadership to avert this danger. Members
may disappear suddenly without the reasons becoming known, much to the surprise of those left behind.
For such machinations in Theosophy read Emily Lutyens: "Candles in the Sun".
Spiritual newsgroups on Internet provide illustration of (un)concious nastiness being ventilated
under the veil of anonimity. Messages are often rife with diatribe, personal attacks and misunderstanding.
Many of such contributors have no interest at all in the matters discussed. Yet even in closed newsgroups,
only open to subscribers, complaints about the tone of communications are being aired.
11. Fear of exclusion
The stronger members are tied to a group, the more the fear of exclusion lurks. They may have
invested their life's savings in the work (Scientology), paid a percentage of their income, failed
to conclude their study, or make a career, or sacrificed a succesful one.
In many cases a member will have alienated himself from family and friends. He has been told to cut
ties with the past. (In the Attleboro cult followers are advised to burn photographs that remind
them of bygone days). No wonder his or her sudden conversion, accompanied by fanatism and urge to
proselytize, has shied away former friends and relatives.
There is no way left but to seek comfort and understanding with members of the spiritual group.
Isolation is sometimes intentionally sought. Formerly, in the Bhagavan Shri Rajneesh movement,
members went about in red/orange dresses and wore mala's with a photo of their master, so setting
themselves aside from the mundane world.
The Hare Krishna movement goes even further. Groups of members go out into the streets in
their oriental dresses for song and dance routines. However, in most movements the alienation is
far more subtle and the natural outcome of an adverse attitude towards the materialism of society.
The true nature of the so-called friendships within the group will only be revealed after a devotee
has left the fold. Members have seen this happen, but did not give it a thought at the time, because
it happened to someone else. But when they undergo the same fate themselves they will feel the humiliation
of being ignored, not being greeted anymore, marriage gone - even not being recognized by one's own
The outcast feels thrown in an abyss. He is cut off from social contacts, his life in pieces.
The magnitude of this desperate experience should not be under-estimated. The renegade will feel
deep shame. He may have confessed in the group intimate secrets, which are now being ridiculed by
his former so-called friends.
The expulsee, deeply hurt, may become embittered and even enter into a suicidal mental state.
Those readers who have been a member of a movement may recognize some of the above psychological
mechanisms. The first reaction of non-members may be to vow never to enter a group. Let us bear in
mind, however, that it should be considered a challenge to face these obstacles for the benefit that
may result from association with kindred spirits.
A prerequisite is that these conditions are being noticed, looked in the eye, and not denied.
The closer people live together, the more group-tensions will build up. Even in reputable circles
as Freudian psycho-analytical associations they occur. Few communes are granted a long life as a
result of one or more of the pitfalls summarized above. Headquarters, contrary to expectations, are
known to be hotbeds of gossip, mutual repulsion and cynism.
So, do not be disheartened and join a group of your liking. After all people who marry also see
wrecked marriages all around them, yet go ahead intent on a happy union in mutual trust, without
regard to the outcome.
Involvement with other people will lead to personal growth if the consequences are anticipated. The
more one stands on one's own feet the more benefit will arise from cooperating with others. It should
be borne in mind that the saying "It is better to give, than to receive" is not merely a moral precept.
(Read my precepts for living)
Please remember that there are hundreds of movements and that it has not been my intention to
summarize them all, or to level any form of criticism at one of them. Indicating the psychological
mechanisms operative in some, or all of them, has been my main theme.
On a separate page I have gone into the mysterious
between people who meet in harmony.
In conclusion one may take heed of Krishnamurti's words in 1929 when he refused to become a 'World
Teacher' of an organisation set up for him:
I maintain that Truth is a pathless land, and you cannot approach it by any path whatsoever,
by any religion, by any sect. Truth being limitless, unconditioned, cannot be organised, nor should
any organisation be formed to lead or coerce people along any particular path. If you first understand
that, then you will see how impossible it is to organize a belief. A belief is purely an individual
matter, and you cannot and must not organize it. If you do, it becomes dead, crystallised; it becomes
a creed, a sect, a religion, to be imposed on others.
"... Having a customer always available on site would mean that the customer in question is probably a small, expendable fish in his organization and is unlikely to have any useful knowledge of its business practices. ..."
"... Unit testing code before it is written means that one would have to have a mental picture of what one is going to write before writing it, which is difficult without upfront design. And maintaining such tests as the code changes would be a nightmare. ..."
"... Programming in pairs all the time would assume that your topnotch developers are also sociable creatures, which is rarely the case, and even if they were, no one would be able to justify the practice in terms of productivity. I won't discuss why I think that abandoning upfront design is a bad practice; the whole idea is too ridiculous to debate ..."
"... Both book and methodology will attract fledgling developers with its promise of hacking as an acceptable software practice and a development universe revolving around the programmer. It's a cult, not a methodology, were the followers shall find salvation and 40-hour working weeks ..."
"... Two stars for the methodology itself, because it underlines several common sense practices that are very useful once practiced without the extremity. ..."
"... The second is the dictatorial social engineering that eXtremity mandates. I've actually tried the pair programming - what a disaster. ..."
"... I've also worked with people who felt that their slightest whim was adequate reason to interfere with my work. That's what Beck institutionalizes by saying that any request made of me by anyone on the team must be granted. It puts me completely at the mercy of anyone walking by. The requisite bullpen physical environment doesn't work for me either. I find that the visual and auditory distraction make intense concentration impossible. ..."
"... One of the things I despise the most about the software development culture is the mindless adoption of fads. Extreme programming has been adopted by some organizations like a religious dogma. ..."
To fairly review this book, one must distinguish between the methodology it presents and the
actual presentation. As to the presentation, the author attempts to win the reader over with emotional
persuasion and pep talk rather than with facts and hard evidence. Stories of childhood and comradeship
don't classify as convincing facts to me.
A single case study-the C3 project-is often referred to, but with no specific information (do
note that the project was cancelled by the client after staying in development for far too long).
As to the method itself, it basically boils down to four core practices:
Always have a customer available on site.
Unit test before you code.
Program in pairs.
Forfeit detailed design in favor of incremental, daily releases and refactoring.
If you do the above, and you have excellent staff on your hands, then the book promises that
you'll reap the benefits of faster development, less overtime, and happier customers. Of course,
the book fails to point out that if your staff is all highly qualified people, then the project
is likely to succeed no matter what methodology you use. I'm sure that anyone who has worked in
the software industry for sometime has noticed the sad state that most computer professionals
are in nowadays.
However, assuming that you have all the topnotch developers that you desire, the outlined methodology
is almost impossible to apply in real world scenarios. Having a customer always available
on site would mean that the customer in question is probably a small, expendable fish in his organization
and is unlikely to have any useful knowledge of its business practices.
Unit testing code before it is written means that one would have to have a mental picture
of what one is going to write before writing it, which is difficult without upfront design. And
maintaining such tests as the code changes would be a nightmare.
Programming in pairs all the time would assume that your topnotch developers are also sociable
creatures, which is rarely the case, and even if they were, no one would be able to justify the
practice in terms of productivity. I won't discuss why I think that abandoning upfront design
is a bad practice; the whole idea is too ridiculous to debate.
Both book and methodology will attract fledgling developers with its promise of hacking
as an acceptable software practice and a development universe revolving around the programmer.
It's a cult, not a methodology, were the followers shall find salvation and 40-hour working weeks.
Experience is a great teacher, but only a fool would learn from it alone. Listen to what the
opponents have to say before embracing change, and don't forget to take the proverbial grain of
Two stars out of five for the presentation for being courageous and attempting to defy the
standard practices of the industry. Two stars for the methodology itself, because it underlines
several common sense practices that are very useful once practiced without the extremity.
Maybe it's an interesting idea, but it's just not ready for prime time.
Parts of Kent's recommended practice - including aggressive testing and short integration cycle
- make a lot of sense. I've shared the same beliefs for years, but it was good to see them clarified
and codified. I really have changed some of my practice after reading this and books like this.
I have two broad kinds of problem with this dogma, though. First is the near-abolition of documentation.
I can't defend 2000 page specs for typical kinds of development. On the other hand, declaring
that the test suite is the spec doesn't do it for me either. The test suite is code, written for
machine interpretation. Much too often, it is not written for human interpretation. Based on the
way I see most code written, it would be a nightmare to reverse engineer the human meaning out
of any non-trivial test code. Some systematic way of ensuring human intelligibility in the code,
traceable to specific "stories" (because "requirements" are part of the bad old way), would give
me a lot more confidence in the approach.
The second is the dictatorial social engineering that eXtremity mandates. I've actually
tried the pair programming - what a disaster. The less said the better, except that my experience
did not actually destroy any professional relationships. I've also worked with people who
felt that their slightest whim was adequate reason to interfere with my work. That's what Beck
institutionalizes by saying that any request made of me by anyone on the team must be granted.
It puts me completely at the mercy of anyone walking by. The requisite bullpen physical environment
doesn't work for me either. I find that the visual and auditory distraction make intense concentration
I find revival tent spirit of the eXtremists very off-putting. If something works, it works
for reasons, not as a matter of faith. I find much too much eXhortation to believe, to go ahead
and leap in, so that I will eXperience the wonderfulness for myself. Isn't that what the evangelist
on the subway platform keeps saying? Beck does acknowledge unbelievers like me, but requires their
exile in order to maintain the group-think of the X-cult.
Beck's last chapters note a number of exceptions and special cases where eXtremism may not work
- actually, most of the projects I've ever encountered.
There certainly is good in the eXtreme practice. I look to future authors to tease that good
out from the positively destructive threads that I see interwoven.
A customer on May 2, 2004
A work of fiction
The book presents extreme programming. It is divided into three parts:
(1) The problem
(2) The solution
(3) Implementing XP.
The problem, as presented by the author, is that requirements change but current methodologies
are not agile enough to cope with this. This results in customer being unhappy. The solution is
to embrace change and to allow the requirements to be changed. This is done by choosing the simplest
solution, releasing frequently, refactoring with the security of unit tests.
The basic assumption which underscores the approach is that the cost of change is not exponential
but reaches a flat asymptote. If this is not the case, allowing change late in the project would
be disastrous. The author does not provide data to back his point of view. On the other hand there
is a lot of data against a constant cost of change (see for example discussion of cost in Code
Complete). The lack of reasonable argumentation is an irremediable flaw in the book. Without some
supportive data it is impossible to believe the basic assumption, nor the rest of the book. This
is all the more important since the only project that the author refers to was cancelled before
Many other parts of the book are unconvincing. The author presents several XP practices. Some
of them are very useful. For example unit tests are a good practice. They are however better treated
elsewhere (e.g., Code Complete chapter on unit test). On the other hand some practices seem overkill.
Pair programming is one of them. I have tried it and found it useful to generate ideas while prototyping.
For writing production code, I find that a quiet environment is by far the best (see Peopleware
for supportive data). Again the author does not provide any data to support his point.
This book suggests an approach aiming at changing software engineering practices. However the
lack of supportive data makes it a work of fiction.
I would suggest reading Code Complete for code level advice or Rapid Development for management
A customer on November 14, 2002
Not Software Engineering.
Any Engineering discipline is based on solid reasoning and logic not on blind faith. Unfortunately,
most of this book attempts to convince you that Extreme programming is better based on the author's
experiences. A lot of the principles are counterintuitive and the author exhorts you just try
it out and get enlightened. I'm sorry but these kind of things belong in infomercials not in s/w
The part about "code is the documentation" is the scariest part. It's true that keeping the
documentation up to date is tough on any software project, but to do away with documentation is
the most ridiculous thing I have heard.
It's like telling people to cut of their noses to avoid colds. Yes we are always in search
of a better software process. Let me tell you that this book won't lead you there.
The "gossip magazine diet plans" style of programming.
This book reminds me of the "gossip magazine diet plans", you know, the vinegar and honey diet,
or the fat-burner 2000 pill diet etc. Occasionally, people actually lose weight on those diets,
but, only because they've managed to eat less or exercise more. The diet plans themselves are
worthless. XP is the same - it may sometimes help people program better, but only because they
are (unintentionally) doing something different. People look at things like XP because, like dieters,
they see a need for change. Overall, the book is a decently written "fad diet", with ideas that
are just as worthless.
A customer on August 11, 2003
Hackers! Salvation is nigh!!
It's interesting to see the phenomenon of Extreme Programming happening in the dawn of the
21st century. I suppose historians can explain such a reaction as a truly conservative movement.
Of course, serious software engineering practice is hard. Heck, documentation is a pain in the
neck. And what programmer wouldn't love to have divine inspiration just before starting to write
the latest web application and so enlightened by the Almighty, write the whole thing in one go,
as if by magic? No design, no documentation, you and me as a pair, and the customer too. Sounds
like a hacker's dream with "Imagine" as the soundtrack (sorry, John).
The Software Engineering struggle is over 50 years old and it's only logical to expect some resistance,
from time to time. In the XP case, the resistance comes in one of its worst forms: evangelism.
A fundamentalist cult, with very little substance, no proof of any kind, but then again if you
don't have faith you won't be granted the gift of the mystic revelation. It's Gnosticism for Geeks.
Take it with a pinch of salt.. well, maybe a sack of salt. If you can see through the B.S. that
sells millions of dollars in books, consultancy fees, lectures, etc, you will recognise some common-sense
ideas that are better explained, explored and detailed elsewhere.
Kent is an excellent writer. He does an excellent job of presenting an approach to software
development that is misguided for anything but user interface code. The argument that user interface
code must be gotten into the hands of users to get feedback is used to suggest that complex system
code should not be "designed up front". This is simply wrong. For example, if you are going to
deploy an application in the Amazon Cloud that you want to scale, you better have some idea of
how this is going to happen. Simply waiting until your application falls over and fails is not
an acceptable approach.
One of the things I despise the most about the software development culture is the mindless
adoption of fads. Extreme programming has been adopted by some organizations like a religious
Engineering large software systems is one of the most difficult things that humans do. There
are no silver bullets and there are no dogmatic solutions that will make the difficult simple.
Maybe I'm too cynical because I never got to work for the successful, whiz-kid companies; Maybe
this book wasn't written for me!
This book reminds me of Jacobsen's "Use Cases" book of the 1990s. 'Use Cases' was all the rage
but after several years, we slowly learned the truth: Uses Cases does not deal with the architecture
- a necessary and good foundation for any piece of software.
Similarly, this book seems to be spotlighting Testing and taking it to extremes.
'the test plan is the design doc'
Not True. The design doc encapsulates wisdom and insight
a picture that accurately describes the interactions of the lower level software components
is worth a thousand lines of code-reading.
Also present is an evangelistic fervor that reminds me of the rah-rah eighties' bestseller,
"In Search Of Excellence" by Peters and Waterman. (Many people have since noted that most of the
spotlighted companies of that book are bankrupt twenty five years later).
- in a room full of people with a bully supervisor (as I experienced in my last job at
a major telco) innovation or good work is largely absent.
- deploy daily - are you kidding? to run through the hundreds of test cases in a large
application takes several hours if not days. Not all testing can be automated.
- I have found the principle of "baby steps", one of the principles in the book, most useful
in my career - it is the basis for prototyping iteratively. However I heard it described in
1997 at a pep talk at MCI that the VP of our department gave to us. So I dont know who stole
it from whom!
Lastly, I noted that the term 'XP' was used throughout the book, and the back cover has a blurb
from an M$ architect. Was it simply coincidence that Windows shares the same name for its XP release?
I wondered if M$ had sponsored part of the book as good advertising for Windows XP! :)
The change madness is getting
worse with every passing year .
The demands for change being placed on corporate IT are
plain ridiculous. As a consequence we are breaking IT. In pursuit of absurd project commitments
we are eating
And the hysteria reaches fever pitch as people extrapolate trends into the future
linearly or worse still exponentially. This is such bad scientific thinking
that it shouldn't be worthy of debate, but the power of critical thought is a scarce
A broken management and governance system, a broken value system, and a broken
But even in the best and healthiest organisations, there are plenty of rogues; psychopaths
(and milder sociopaths) who are never going to care about anyone but themselves. They soar in
management (and they're drawn to the power); they look good to all measures and controls
except a robust risk management system - it is the last line of defense.
...I'm saying there is a real limit to how fast humans can change: how fast we can change our
behaviours, our attitudes, our processes, our systems. We need to accept that the technology
is changing faster than society, our IT sector, our organisations, our teams, ourselves can
I'm saying there is a social and business backlash already to the pace of change. We're
standing in the ruins of an economy that embraced fast change.
I'm saying there are real risks to the pace of change, and we currently live in a culture
that thinks writing risks down means you can then ignore them, or that if you can't ignore
them you can always hedge them somehow.
We have to slow down a bit. perhaps "Slow IT" is the wrong name but it was catchy. I'm not
saying go slooooow. We've somehow sustained a pretty impressive pace for decades. But clearly
it can't go much faster, if at all, and all these demands that it must go faster are plain
silly. It just can't. There's bits falling off, people burning out, smoking shells of
I'm not saying stop, but I am saying ease off a little, calm down, stop panicking, stop
this desperate headlong rush. You are right Simon that mindfulness is a key element: we all
need time to think. Let the world keep up.
Much of the zeitgeist that drives the frenzy you describe is generated by vendors
especially those with software in their portfolio. Software has more margin that hardware or
service. As a result they have more marketing budget. With that budget they invest and spend
a lot of time and effort to figure out exactly how to generate the frenzy with a new thing
that you must have. They have to do this to keep market interest in the products. That is
actually what their job is.
The frenzy is deliberately and I would say almost scientifically engineered by very very
bright marketing people in software vendors. Savvy IT organizations are aware of that
distinction and maintain their focus on enabling their business to be successful. IT as
Utility, On Demand, SOA, Cloud, ..... Software vendors will not and should not stop doing
that - that is what keeps them in business and generates profits that enable new innovation.
The onus is on the buyer to understand that whatever the latest technology is, does not
provide the answer for how they will improve business performance. Improving business
performance is the burden that only the organization can bear.
And now example buss words infused nonsense: ""DevOps is a concept, an idea, a life philosophy," says Gottfried Sehringer, chief marketing
, a software delivery
automation company. "It's not really a process or a toolset, or a technology." And another one:
..." "In an ideal world, you would push a button to release every few seconds," Sehringer says.
But this is not an ideal world, and so people plug up the process along the way."... "
I want to see sizable software product with the release every few seconds.
Even for a small and rapidly evolving web site scripts should be released no more frequently then
"... Even if you're a deity of software bug-squashing, how can you -- or any developer or operations specialist -- deliver high-quality, "don't break anything" code when you have to build and release that fast? Everyone has their own magic bullet. "Agile -- " cries one crowd. " Continuous build -- " yells another. " Continuous integration -- " cheers a third. ..."
"... Automation has obvious returns on investment. "You can make sure it's good in pre-production and push it immediately to production without breaking anything, and then just repeat, repeat, repeat, over and over again," says Sehringer. ..."
"... In other words, you move delivery through all the steps in a structured, repeatable, automated way to reduce risk and increase the speed of releases and updates. ..."
quickie guide to continuous delivery in DevOps
In today's world, you have to develop and
deliver almost in the same breath. Here's a quick guide to help you figure out which continuous
delivery concepts will help you breathe easy, and which are only hot air. Developers are always
under pressure to produce more and release software faster, which encourages the adoption of
new concepts and tools. But confusing buzzwords obfuscate real technology and business
benefits, particularly when a vendor has something to sell. That makes it hard to determine
what works best -- for real, not just as a marketing phrase -- in the continuous flow of build and
deliver processes. This article gives you the basics of continuous delivery to help you sort it
To start with, the terms apply to different parts of the same production arc, each of which
are automated to different degrees:
Continuous integration means frequently merging code into a central repository
"Frequently" means usually several times a day. Each merge triggers an automated "build and
test" instance, a process sometimes called continuous build . But by either name, continuous
integration and continuous build do nothing in terms of delivery or deployment. They're about
code management, not what happens to the code afterward.
Continuous delivery refers to the automation of the software release process
includes some hands-on effort by developers. Usually, developers approve or initiate the
deployment, though there can be other manual steps as well.
Continuous deployment is continuous delivery with no manual steps for developers. The
whole thing is automated, and it requires not so much as a nod from humans.
With continuous deployment, "a developer's job typically ends at reviewing a pull request
from a teammate and merging it to the master branch," explains Marko Anastasov in a
. "A continuous integration/continuous deployment service takes over from there
by running all tests and deploying the code to production, while keeping the team informed
about [the] outcome of every important event."
However, knowing the terms and their definitions isn't enough to help you determine when and
where it is best to use each. Because, of course, every shop is different.
It would be great if the market clearly distinguished between concepts and tools and their
uses, as they do with terms like DevOps. Oh, wait.
"DevOps is a concept, an idea, a life philosophy," says Gottfried Sehringer, chief marketing
, a software delivery
automation company. "It's not really a process or a toolset, or a technology."
But, alas, industry terms are rarely spelled out that succinctly. Nor are they followed with
hints and tips on how and when to use them. Hence this guide, which aims to help you learn when
to use what.
Choose your accelerator according to your need for speed
That's not the end of it; some businesses push for software updates to be faster still. "If
you work for Amazon, it might be every few seconds," says Sehringer.
Even if you're a deity of software bug-squashing, how can you -- or any developer or operations
specialist -- deliver high-quality, "don't break anything" code when you have to build and release
that fast? Everyone has their own magic bullet. "Agile -- " cries one crowd. "
-- " yells another. "
cheers a third.
Let's just cut to the chase on all that, shall we?
"Just think of continuous as 'automated,'" says Nate Berent-Spillson, senior delivery
, a software
services provider. "Automation is driving down cost and the time to develop and deploy."
Well, frack, why don't people just say automation?
Add to the idea of automation the concepts of continuous build, continuous delivery,
continuous everything, which are central to DevOps, and we find ourselves talking in circles.
So, let's get right to sorting all that out.
... ... ...
Rinse. Repeat, repeat, repeat, repeat (the point of automation in
Automation has obvious returns on investment. "You can make sure it's good in pre-production
and push it immediately to production without breaking anything, and then just repeat, repeat,
repeat, over and over again," says Sehringer.
In other words, you move delivery through all the steps in a structured, repeatable,
automated way to reduce risk and increase the speed of releases and updates.
In an ideal world, you would push a button to release every few seconds," Sehringer says.
But this is not an ideal world, and so people plug up the process along the way.
A company may need approval for an application change from its legal department. "Some
companies are heavily regulated and may need additional gates to ensure compliance," notes
Sehringer. "It's important to understand where these bottlenecks are." The ARA software should
improve efficiencies and ensure the application is released or updated on schedule.
"Developers are more familiar with continuous integration," he says. "Application release
automation is more recent and thus less understood."
... ... ...
Pam Baker has written hundreds of articles published in leading technology, business and
finance publications including InformationWeek, Institutional Investor magazine, CIO.com,
NetworkWorld, ComputerWorld, IT World, Linux World, and more. She has also authored several
analytical studies on technology, eight books -- the latest of which is Data Divination: Big
Data Strategies -- and an award-winning documentary on paper-making. She is a member of the
National Press Club, Society of Professional Journalists and the Internet Press Guild.
"... It's basically a bullshit bingo post where someone repeats a buzzword without any knowledge of the material behind it ..."
"... continuous delivery == constant change ..."
"... This might be good for developers, but it's a nightmare for the poor, bloody, customers. ..."
"... However, I come at it from the other side, the developers just push new development out and production support is responsible for addressing the mess, it is horrible, there is too much disconnect between developers and their resulting output creating consistent outages. The most successful teams follow the mantra "Eat your own dog food" , developers who support the crap they push ..."
"... But do you know who likes Continuous Delivery? Not the users. The users hate stuff changing for the sake of change, but trying to convince management seems an impossible task. ..."
"... some of us terrified what 'continious delivery' means in the context of software in the microcontroller of a health care device, ..."
"... It is a natural consequence of a continuous delivery, emphasis on always evolving and changing and that the developer is king and no one can question developer opinion. Developer decides it should move, it moves. No pesky human testers to stand up and say 'you confused the piss out of us' to make them rethink it. No automatic test is going to capture 'confuse the piss out of the poor non-developer users'. ..."
"... It's amazing how common this attitude has become. It's aggressively anti-customer, and a big part of the reason for the acceleration of the decline of software quality over the past several years. ..."
"... All I know is that, as a user, rapid-release or continuous delivery has been nothing but an enormous pain in the ass to me and I wish it would die the horrible death it deserves already. ..."
Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop the wrong
thing that doesn't even work they way they thought it should 4. The project leader is canned 5.
The software is implemented by an outside vendor, with all the flaws. 6. The software operates
finally after 5 years of modifications to both the software and the workflows (to match the flaws
in the software). 7. As soon as it's all running properly and everyone is trained, a new project
is launched to redo it, "the right way". 8. Goto 1
Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop
the wrong thing that doesn't even work they way they thought it should 4. The project leader
is canned 5. The software is implemented by an outside vendor, with all the flaws. 6. The software
operates finally after 5 years of modifications to both the software and the workflows (to
match the flaws in the software). 7. As soon as it's all running properly and everyone is trained,
a new project is launched to redo it, "the right way". 8. Goto 1
You just accurately described a 6 year project within our organization....and it made me cry
Does this model have a name? an urban dictionary name? if not it needs one.
Yeah, maybe there's something useful in TFA, but I'm not really inclined to go looking based
on what was in the summary. At no point, did the person being quoted actually say anything of
It's just buzzword soup with a dash of new technologies thrown in.
Five years ago they would have said practically the same words, but just talked about
utilizing the cloud instead of AI.
I'm also a little skeptical of any study published by a company looking to sell you what the
study has just claimed to be great. That doesn't mean its a complete sham, but how hard did they
look for other explanations why some companies are more successful than others?
I notice the targets are all set from the company's point of view... including customer satisfaction.
However it's quite easy to meet any goal, as long as you set it low enough.
Companies like Comcast or Qwest objectively have abysmal customer satisfaction ratings; but
they likely meet their internal goal for that metric. I notice, in their public communications,
they always use phrasing along the lines of "giving you an even better customer service experience"
- again, the trick is to set the target low and
This might be good for developers, but it's a nightmare for the poor, bloody, customers.
Any professional outfit will test a new release (in-house or commercial product) thoroughly
before letting it get anywhere close to an environment where their business is at stake.
This process can take anywhere from a day or two to several months, depending on the complexity
of the operation, the scope of the changes, HOW MANY (developers note: not if any ) bugs
are found and whether any alterations to working practices have to be introduced.
So to have developers lob a new "release" over the wall at frequent intervals is not useful,
it isn't clever, nor does it save (the users) any money or speed up their acceptance. It just
costs more in integration testing, floods the change control process with "issues" and means that
when you report (again, developers: not if ) problems, it is virtually impossible to describe
exactly which release you are referring to and even more impossible for whoever fixes the bugs
to produce the same version to fix and then incorporate those fixes into whatever happens to be
the latest version - that hour. Even more so when dozens of major corporate customers are ALL
reporting bugs with each new version they test.
Any professional outfit will test a new release (in-house or commercial product) thoroughly
before letting it get anywhere close to an environment where their business is at stake. This
process can take anywhere from a day or two to several months, depending on the complexity of
the operation, the scope of the changes, HOW MANY (developers note: not if any) bugs are found
and whether any alterations to working practices have to be introduced.
I wanted to chime in with a tangible anecdote to support your
I can sympathize with that few, of it appearing to have too many developers focused upon deployment/testing
then actual development.
However, I come at it from the other side, the developers just push new development out
and production support is responsible for addressing the mess, it is horrible, there is too much
disconnect between developers and their resulting output creating consistent outages. The most
successful teams follow the mantra "Eat your own dog food" , developers who support the crap they
The crux of the problem is that we (in these discussions and the analysts) describe *all* manner
of 'software development' as the same thing. Whether it's a desktop application, an embedded microcontroller
in industrial equipment, a web application for people to get work done, or a webapp to let people
see the latest funny cat video.
Then we start talking past each other, some of us terrified what 'continious delivery'
means in the context of software in the microcontroller of a health care device, others t
Well, 'continuous delievery' is a term with a defined meaning. And releasing phone apps with
unwanted UI/functionality in rapid succession is not part of that definition. Continuous delievery
basically only is the next logical step after continuous integration. You deploy the new functionallity
automatically (or with a click of a button) when certain test criteria are met. Usually on a subset
of your nodes so only a subset of your customers sees it. If you have crashes on those nodes or
customer complaints you
You deploy the new functionallity automatically (or with a click of a button) when certain
test criteria are met. Usually on a subset of your nodes so only a subset of your customers
sees it. If you have crashes on those nodes or customer complaints you roll back.
Why do you consider this to be a good thing? It's certainly not for those poor customers who
were chosen to be involuntary beta testers, and it's also not for the rest of the customers who
have to deal with software that is constantly changing underneath them.
'continuous delievery' is a term with a defined meaning. And releasing phone apps with unwanted
UI/functionality in rapid succession is not part of that definition.
It is a natural consequence of a continuous delivery, emphasis on always evolving and changing
and that the developer is king and no one can question developer opinion. Developer decides it
should move, it moves. No pesky human testers to stand up and say 'you confused the piss out of
us' to make them rethink it. No automatic test is going to capture 'confuse the piss out of the
poor non-developer users'.
If you have crashes on those nodes or customer complaints you roll back.
Note that a customer with a choice is likely to just go somewhere else rather than use your
I recently interviewed at a couple of the new fangled big data marketing startups that correlate
piles of stuff to help target ads better, and they were continuously deploying up the wazoo. In
fact, they had something like zero people doing traditional QA.
It was not totally insane at all. But they did have a blaze attitude about deployments -- if
stuff don't work in production they just roll back, and not worry about customer input data being
dropped on the floor. Heck, they did not worry much about da
You want your deployment system to be predictable, and as my old AI professor used to say,
intelligent means hard to predict. You don't want AI for systems that just have to do the exact
same thing reliably over and over again.
...At Duke University's Office of Information Technology (OIT), we began looking at containers as
a way to achieve higher density from the virtualized infrastructure used to host websites. Virtual
machine (VM) sprawl had started to become a problem. We favored separating each client's website
onto its own VM for both segregation and organization, but steady growth meant we were managing more
servers than we could handle. As we looked for ways to lower management overhead and make better
use of resources, Docker hit the news, and we began to experiment with
for our web applications.
For us, the initial investigation of containers mirrors a shift toward a DevOps culture.
Where we started
When we first looked into container technology, OIT was highly process driven and composed of
monolithic applications and a monolithic organizational structure. Some early forays into automation
were beginning to lead the shift toward a new cultural organization inside the department, but even
so, the vast majority of our infrastructure consisted of "pet" servers (to use the
vs. cattle analogy). Developers created their applications on staging servers designed to match
production hosting environments and deployed by migrating code from the former to the latter. Operations
still approached hosting as it always had: creating dedicated VMs for individual services and filing
manual tickets for monitoring and backups. A service's lifecycle was marked by change requests, review
boards, standard maintenance windows, and lots of personal attention.
A shift in culture
As we began to embrace containers, some of these longstanding attitudes toward development and
hosting began to shift a bit. Two of the larger container success stories came from our investigation
into cloud infrastructure. The first project was created to host hundreds of R-Studio containers
for student classes on Microsoft Azure hosts, breaking from our existing model of individually managed
servers and moving toward "cattle"-style infrastructure designed for hosting containerized applications.
The other was a rapid containerization and deployment of the Duke website to Amazon Web Services
while in the midst of a denial-of-service attack, dynamically creating infrastructure and rapidly
The success of these two wildly nonstandard projects helped to legitimize containers within the
department, and more time and effort was put into looking further into their benefits and those of
on-demand and disposable cloud infrastructure, both on-premises and through public cloud providers.
It became apparent early on that containers lived within a different timescale from traditional
infrastructure. We started to notice cases where short-lived, single-purpose services were created,
deployed, lived their entire lifecycle, and were decommissioned before we completed the tickets created
to enter them into inventory, monitoring, or backups. Our policies and procedures were not able to
keep up with the timescales that accompanied container development and deployment.
In addition, humans couldn't keep up with the automation that went into creating and managing
the containers on our hosts. In response, we began to develop more automation to accomplish usually
human-gated processes. For example, the dynamic migration of containers from one host to another
required a change in our approach to monitoring. It is no longer enough to tie host and service monitoring
together or to submit a ticket manually, as containers are automatically destroyed and recreated
on other hosts in response to events.
Some of this was in the works for us already-automation and container adoption seem to parallel
one another. At some point, they become inextricably intertwined.
As containers continued to grow in popularity and OIT began to develop tools for container orchestration,
we tried to further reinforce the "cattle not pets" approach to infrastructure. We limited login
of the hosts to operations staff only (breaking with tradition) and gave all hosts destined for container
hosting a generic name. Similar to being coached to avoid naming a stray animal in an effort to prevent
attachment, servers with generic names became literally forgettable. Management of the infrastructure
itself became the responsibility of automation, not humans, and humans focused their efforts on the
services inside the containers.
Containers also helped to usher continuous integration into our everyday workflows. OIT's Identity
Management team members were early adopters and began to build Kerberos key distribution centers
(KDCs) inside containers using Jenkins, building regularly to incorporate patches and test the resulting
images. This allowed the team to catch breaking builds before they were pushed out onto production
servers. Prior to that, the complexity of the environment and the widespread impact of an outage
made patching the systems a difficult task.
Embracing continuous deployment
Since that initial use case, we've also embraced continuous deployment. There is a solid pattern
for every project that gets involved with our continuous integration/continuous deployment (CI/CD)
system. Many teams initially have a lot of hesitation about automatically deploying when tests pass,
and they tend to build checkpoints requiring human intervention. However, as they become more comfortable
with the system and learn how to write good tests, they almost always remove these checkpoints.
Within our container orchestration automation, we use Jenkins to patch base images on a regular
basis and rebuild all the child images when the parent changes. We made the decision early that the
images could be rebuilt and redeployed at any time by automated processes. This meant that any code
included in the branch of the git repository used in the build job would be included in the image
and potentially deployed without any humans involved. While some developers initially were uncomfortable
with this, it ultimately led to better development practices: Developers merge into the production
branch only code that is truly ready to be deployed.
This practice facilitated rebuilding container images immediately when code is merged into the
production branch and allows us to automatically deploy the new image once it's built. At this point,
almost every project using the automatic rebuild has also enabled automated deployment.
Today the adoption of both containers and DevOps is still a work in progress for OIT.
Internally we still have to fight the entropy of history even as we adopt new tools and culture.
Our biggest challenge will be convincing people to break away from the repetitive break-fix
mentality that currently dominates their jobs and to focus more on automation. While time is
always short, and the first step always daunting, in the long run adopting automation for day-to-day
tasks will free them to work on more interesting and complex projects.
Thankfully, people within the organization are starting to embrace working in organized or ad
hoc groups of cross-discipline members and developing automation together. This will definitely become
necessary as we embrace automated orchestration and complex systems. A group of talented individuals
who possess complementary skills will be required to fully manage the new environments.
"... All of the hype around software and developers, which tends to significantly skew even the
DevOps discussion, runs the risk of creating the perception that IT ops is just a necessary evil. Indeed,
some go so far as to make the case for a 'NoOps' world in which the public cloud magically takes care
of everything downstream once developers have 'innovated' and 'created'. ..."
"... This kind of view comes about from people looking through the wrong end of the telescope. Turn
the thing around and look up close at what goes on in the world of ops, and you get a much better sense
of perspective. Teams operating in this space are not just there to deploy the next custom software
release and make sure it runs quickly and robustly - in fact that's often a relatively small part of
what they do. ..."
"... And coming back to operations, you are sadly mistaken if you think that the public cloud makes
all challenges and requirements go away. If anything, the piecemeal adoption of cloud services has made
things more complex and unpredictable from an integration and management perspective. ..."
"... There are all kinds of valid reasons to keep an application sitting on your own infrastructure
anyway - regulation, performance, proximity to dependent solutions and data, etc. Then let's not forget
the simple fact that running things in the cloud is often more expensive over the longer term. ..."
Get real – it's not all about developers and DevOps
Listen to some DevOps evangelists talk, and you would get the impression that IT operations teams
exist only to serve the needs of developers. Don't get me wrong, software development is a good competence
to have in-house if your organisation depends on custom applications and services to differentiate
As an ex-developer, I appreciate the value of being able to deliver something tailored to a specific
need, even if it does pain me to see the shortcuts too often taken nowadays due to ignorance of some
of the old disciplines, or an obsession with time-to-market above all else.
But before this degenerates into an 'old guy' rant about 'youngsters today', let's get back to
the point that I really want to make.
All of the hype around software and developers, which tends to significantly skew even the
DevOps discussion, runs the risk of creating the perception that IT ops is just a necessary evil.
Indeed, some go so far as to make the case for a 'NoOps' world in which the public cloud magically
takes care of everything downstream once developers have 'innovated' and 'created'.
This kind of view comes about from people looking through the wrong end of the telescope.
Turn the thing around and look up close at what goes on in the world of ops, and you get a much better
sense of perspective. Teams operating in this space are not just there to deploy the next custom
software release and make sure it runs quickly and robustly - in fact that's often a relatively small
part of what they do.
This becomes obvious when you recognize how much stuff runs in an Enterprise IT landscape - software
packages enabling core business processes, messaging, collaboration and workflow platforms keeping
information flowing, analytics environments generating critical business insights, and desktop and
mobile estates serving end user access needs - to name but a few.
There's then everything required to deal with security, data protection, compliance and other
aspects of risk. Apart from the odd bit of integration and tailoring work - the need for which is
diminishing with modern 'soft-coded', connector-driven solutions - very little of all this has anything
to do with development and developers.
A big part of the rationale for modernising your application landscape and migrating to the latest
flexible and open software packages and platforms is to eradicate the need for coding wherever you
can. Code is expensive to build and maintain, and the same can often be achieved today through software
switches, policy-driven workflow, drag-and-drop interface design, and so on. Sensible IT teams only
code when they absolutely have to.
And coming back to operations, you are sadly mistaken if you think that the public cloud makes
all challenges and requirements go away. If anything, the piecemeal adoption of cloud services has
made things more complex and unpredictable from an integration and management perspective.
There are all kinds of valid reasons to keep an application sitting on your own infrastructure
anyway - regulation, performance, proximity to dependent solutions and data, etc. Then let's not
forget the simple fact that running things in the cloud is often more expensive over the longer term.
Against this background, an 'appropriate' level of custom development and the selective use of
cloud services will be the way forward for most organisations, all underpinned by a well-run data
centre environment acting as the hub for hybrid delivery. This is the approach that tends to be taken
by the most successful enterprise IT teams, and the element that makes particularly high achievers
stand out is agile and effective IT operations.
This isn't just to support any DevOps agenda you might have; it is demonstrably a key enabler
across the board. Of course if you work in operations, you will know already intuitively know all
this. But if you want some ammunition to spell it out to others who need enlightenment, take a look
at our research report entitled
IT Ops and a Digital
Business Enabler; more than just keeping the lights on . This is based on input from 400 Senior
European IT professionals. ®
I think this is one fad that has run its course. If nothing else, the one thing that cloud has
brought to the software world is the separation of software from the environment it runs in, and
since the the Ops side of DevOps is all about the integration of the platform and software, what
you end up with in a cloudy world is a lot of people looking for a new job.
For decades developers have been ignored by infrastructure vendors because the decision makers
buying infrastructure sit in the infrastructure teams. Now with the cloud etc vendors realize
they will lose supporters within these teams.
So instead - infrastructure vendors target developers to become their next fanboys.
E.g. Dear developer, you won't need to speak to your infrastructure admins anymore to setup
a development environment. Now you can automate, orchestrate the provisioning of your containerized
development environment at the push of a button. Blah blah blah, but you have to buy our storage.
I remember the days when every DBA wanted RAID10 just because thats what the whitepaper recommended.
By that time storage technology had long moved on, but the DBA still talked about Full Stripe
Now with DevOps you'll have Developers influencing infrastructure decisions, because they just
learned about snapshots. And yes - it has to be all flash - and designed from the ground up by
millenials that eat avocado.
Re: DevOps was never supposed to replace Operations
Yes, DevOps isn't about replacing Ops. But try telling that to the powers that be. It is sold
and seen as a cost cutting measure.
As for devs learning Ops and vice versa, there are very few on both sides who really understand
what it takes to do the others job. I have a very high regard for Devs, but when it comes to infra,
they are, as a whole, very incompetent. Just like I'm incompetent in Dev. can't have one without
the other. I feel that in time, the pendulum will swing away from cloud as execs and accountants
realize how it isn't really saving any money.
The real question is: Will there be any qualified operations engineers available or will they
all have retired out or have found work elsewhere. It isn't easy to be an ops engineer, takes
a lot of experience to get there, and qualified candidates are hard to come by. Let's face it,
in today's world, its a dying breed.
Nice of you to point out what us in Ops have known all along. I'm afraid it will fall on deaf
ears, though. Until the executives who constantly fall for the new shiny are made to actually
examine business needs and processes and make business decisions based on said.
Our laughable move to cloud here involved migrating off of on prem Exchange to O365. The idea
was to free up our operations team to allow us to do more in house projects. Funny thing is, it
takes more management of the service than we ever did on premises. True, we aren't maintaining
the Exchange infra, but now we have SQL servers, DCs, ADFS, etc, to maintain in the MS cloud to
allow authentication just to use the product. And because mail and messaging is business critical,
we have to have geographically disparate instances of both. And the cost isn't pretty. Yay cloud.
Linus Torvalds believes the technology industry's celebration of innovation is smug, self-congratulatory,
and self-serving. The term of art he used was more blunt: "The innovation the industry talks about
so much is bullshit," he said. "Anybody can innovate. Don't do this big 'think different'... screw
that. It's meaningless.
In a deferential interview at the
Open Source Leadership Summit in California on Wednesday, conducted by Jim Zemlin, executive
director of the Linux Foundation, Torvalds discussed how he has managed the development of the Linux
kernel and his attitude toward work.
"All that hype is not where the real work is," said Torvalds. "The real work is in the details."
Torvalds said he subscribes to the view that successful projects are 99 per cent perspiration,
and one per cent innovation.
As the creator and benevolent dictator of the
open-source Linux kernel , not to mention the
inventor of the Git distributed version control system, Torvalds has demonstrated that his approach
produces results. It's difficult to overstate the impact that Linux has had on the technology industry.
Linux is the dominant operating system for servers. Almost all high-performance computing runs on
Linux. And the majority of mobile devices and embedded devices rely on Linux under the hood.
The Linux kernel is perhaps the most successful collaborative technology project of the PC era.
Kernel contributors, totaling more than 13,500 since 2005, are adding about 10,000 lines of code,
removing 8,000, and modifying between 1,500 and 1,800 daily, according to Zemlin. And this has been
going on – though not at the current pace – for more than two and a half decades.
"We've been doing this for 25 years and one of the constant issues we've had is people stepping
on each other's toes," said Torvalds. "So for all of that history what we've done is organize the
code, organize the flow of code, [and] organize our maintainership so the pain point – which is people
disagreeing about a piece of code – basically goes away."
The project is structured so people can work independently, Torvalds explained. "We've been able
to really modularize the code and development model so we can do a lot in parallel," he said.
Technology plays an obvious role but process is at least as important, according to Torvalds.
"It's a social project," said Torvalds. "It's about technology and the technology is what makes
people able to agree on issues, because ... there's usually a fairly clear right and wrong."
But now that Torvalds isn't personally reviewing every change as he did 20 years ago, he relies
on a social network of contributors. "It's the social network and the trust," he said. "...and we
have a very strong network. That's why we can have a thousand people involved in every release."
The emphasis on trust explains the difficulty of becoming involved in kernel development, because
people can't sign on, submit code, and disappear. "You shoot off a lot of small patches until the
point where the maintainers trust you, and at that point you become more than just a guy who sends
patches, you become part of the network of trust," said Torvalds.
Ten years ago, Torvalds said he told other kernel contributors that he wanted to have an eight-week
release schedule, instead of a release cycle that could drag on for years. The kernel developers
managed to reduce their release cycle to around two and half months. And since then, development
has continued without much fuss.
"It's almost boring how well our process works," Torvalds said. "All the really stressful times
for me have been about process. They haven't been about code. When code doesn't work, that can actually
be exciting ... Process problems are a pain in the ass. You never, ever want to have process problems
... That's when people start getting really angry at each other." ®
"... Most of us use some form of desired state solution already. Desired state solutions basically
involve an OS agent that gets a config from a centralized location and applies the relevant configuration
to the operating system and/or applications. ..."
12 May 2017 at 14:56, Trevor
Pott Infrastructure as code is a buzzword frequently thrown out alongside DevOps and continuous
integration as being the modern way of doing things. Proponents cite benefits ranging from an amorphous
"agility" to reducing the time to deploy new workloads. I have an argument for infrastructure as
code that boils down to "cover your ass", and have discovered it's not quite so difficult as we might
... ... ...
None of this is particularly surprising. When you have an environment where each workload is
pet , change is slow, difficult, and requires a lot of testing. Reverting changes is equally
tedious, and so a lot of planning goes into making sure than any given change won't cascade and cause
knock-on effects elsewhere.
In the real world this is really the result of two unfortunate aspects of human nature. First:
everyone hates doing documentation, so it's highly unlikely that in an unstructured environment every
change from the last refresh was documented. The second driver of chaos and problems is that there
are few things more permanent than a temporary fix.
When you don't have the budget for the right hardware, software or services you make do. When
something doesn't work you "innovate" a solution. When that breaks something, you patch it. You move
from one problem to the next, and if you're not careful, you end up with something so fragile that
if you breathe on it, it falls over. At this point, you burn it all down and restart from scratch.
This approach to IT is fine - if you have 5, 10 or even 50 workloads. A single techie can reasonably
be expected to keep that all in their head, know their network and solve any problems they encounter.
Unfortunately, 50 workloads is today restricted to only the smallest of shops. Everyone else is juggling
too many workloads to be playing the pets game any more.
Most of us use some form of desired state solution already. Desired state solutions basically
involve an OS agent that gets a config from a centralized location and applies the relevant configuration
to the operating system and/or applications. Microsoft's group policy can be considered a really
primitive version of this, with System Center being a more powerful but miserable to use example.
The modern friendly tools being Puppet, Chef, Saltstack, Ansible and the like.
Once you have desired state configs in place we're no longer beating individual workloads into
shape, or checking them manually for deviation from design. If all does what it says on the tin,
configurations are applied and errors thrown if they can't be. Usually there is some form of analysis
software to determine how many of what is out of compliance. This is a big step forward.
The Technocult, also known as the Machine cult is the semi-offical name given by
of the Crossed Heart to followers of the Mechanicum faith who supply and maintain virtually all
of the church's technology, engineering and industry.
Although they serve with the Church of the Crossed Heart they have their own version of worship
that differs substantially in theology and ritualistic forms from that of
The Twelve Angels .
Instead the Technocult worships a deity they call the Machine god or Omnissiah. The Technocult believes
that knowledge is divine and comes only form the Omnissiah thus making any objects that demonstrate
the application of knowledge , i.e machinery, or contain it (books) holy in the eyes/optical implants
of the Techcult. The Technocult regard organic flesh as weak and imperfect, with
the Rot being veiwed as a divine
message from the Omnissah demonstrating its weakness, thus making its removal and replacement by
mechanical, bionic parts a sacred process that brings them closer to their god with many of its older
members having very little of their original bodies remaining.
The date of the cults formation is unknown, or a closely guarded secret...
1. Saying you're doing Agile just cos you're doing daily stand-ups. You're not doing agile. There
is so much more to agile practices than this! Yet I'm surprised how often I've heard that story.
It really is remarkable.
... ... ....
3. Thinking that agile is a silver bullet and will solve all your problems. That's so naiive,
of course it won't! Humans and software are a complex mix with any methodology, let alone with an
added dose of organisational complexity. Agile development will probably help with many things, but
it still requires a great deal of skill and there is no magic button.
... ... ...
8. People who use agile as an excuse for having no process or producing no documentation. If documents
are required or useful, there's no reason why an agile development team shouldn't produce them. Just
not all up-front; do it as required to support each feature or iteration. JFDI (Just F'ing Do It)
is not agile!
David, 23 February 2010 at 1:21 am
So agree on number 1. Following "Certified" Scrum Master training (prior to the exam requirement),
a manager I know now calls every regular status meeting a "scrum", regardless of project or methodology.
Somehow the team is more agile as a result.
Ironically he pulled up another staff member for "incorrectly" using the term retrospective.
Andy Till, 23 February 2010 at 9:28 am
I can think of far worse, how about pairing with the guy in the office who is incapable of
Steve Watson, 13 May 2010 at 10:06 am
I like number 9 as I find with testing people think that they no longer need to write proper
test cases and scripts – a list of confirmations on a user story will do. Well, if its a simple
change I guess you can dispense with test scripts, but if its something more complex then there
is no reason NOT to write scripts. If you have a reasonably large team of people who could execute
the tests, they can follow the test steps and validate against the expected results. It also means
that you can sensibly lump together test cases and cover them with one test.
If you dont think about how you will execute them and just tackle them one by one off the confirmations
list, you miss the opportunity to run one test and cover many separate cases, saving time.
I always find test scripts useful if someone different re-runs a test, as they then follow
the same process as before. This is why we automate regression so the tests are executed the same
John Quincy, 24 October 2011 at 12:02 am
I am not a fan of agile. Unless you have a small group of developers who are in perfect sync
with each other at all times, this "one size fits all" methodology is destructive and downright
dangerous. I have personally witnessed a very good company go out of business this year because
they transformed their development shop from a home-grown iterative methodology to SCRUM. The
team was required to abide by the SCRUM rules 100%. They could not keep up with customer requirements
and produced bug filled releases that were always late. These developers went from fun, friendly,
happy people (pre-SCRUM) [who NEVER missed a date] to bitter, sarcastic, hard to be around 'employees'.
When the writing was on the wall a couple of months back, the good ones got the hell out of there,
and the company could not recover.
Some day, I'm convinced that Beck through Thomas will proclaim that the Agile Manifesto was
all a big practical joke that got out of control.
This video pretty much lays out the one and only reason why management wants to implement Agile:
It's a cycle of violence when a project claims to be Agile just because of standups and iterations
and don't think about resolving the core challenges they've had to begin with. People are left
still battling said challenges and then say that Agile sucks.
"... while DevOps is appealing to startups, there are important stumbling blocks in the path of DevOps adoption within the enterprise. ..."
"... The tools needed to implement a DevOps culture are lacking. While some of the tools can be provided by vendors and others can be created within the enterprise, a process which takes a long period of time, "there is a marathon of organizational change and restructuring that must occur before such tools could ever be bought or built." ..."
While acknowledging that large companies such as Google and Facebook benefit from implementing
DevOps, and that "there is no lack of appetite to experiment with DevOps practices" within "Fortune
500s and specifically financial services firms", Shannon-Solomon remarks that "there are few true
change agents within enterprise IT willing to affect DevOps implementations."
Shehas come to this conclusion basedon "conversations with startup founders, technology incumbents
offering DevOps solutions, and technologists within large enterprises."
Shannon-Solomon brings four arguments to support her position:
Siloed structures and organizational change . According to Shannon-Solomon, enterprises
create siloed structures between teams because that's how large organizations maximize value results,
and one of DevOps'main purpose is to bring thosesilos down. Sometimes a large company relies on
another vendor for operational support, and such vendors "tend to bring their own degree of siloing."
Also, it is expensive for an enterprise to invest into "holistic solutions to facilitate DevOps"
after they "invested heavily on integration for existing solutions."
Buy vs. build.The tools needed to implement a DevOps culture are lacking. While
some of the tools can be provided by vendors and others can be created within the enterprise,
a process which takes a long period of time, "there is a marathon of organizational change and
restructuring that must occur before such tools could ever be bought or built."
Vendors of DevOps solutions acknowledge that when selling to the enterprise, they are trying
to sell a cultural revolution. According to Shannon-Solomon, it is hard to introduce DevOps
to development teams because Vendors need to first win over individual developers with the efficiency
of their solution, for example, by initially eschewing procurement and offering a sandbox environment
directly to developers to test out the environment.
Selling DevOps toolkits to the enterprise means facing the well-documented challenges of navigating
procurement and a mindset currently more primed to build than buy DevOps tools.
Return on investment . Shannon-Solomon cites a senior IT professional working at an
investment bank saying that his company "has not been very successful at tracking DevOps projects
occurring within individual business units using homegrown tools, and any evaluation of a project's
success has been limited to anecdotal assessments and perceived results."
Shannon-Solomonends her post wondering "how long will it be until enterprises are forced to accept
that they must accelerate their experiments with DevOps" and hoping that "more individual change
agents within large organizations may emerge" in the future.
No methodology can substitute good engineers who actually talk to and work with each other. Good
engineers can benefit from a better software development methodology, but even the best software development
methodology is powerless to convert mediocre developers into stars.
"... disorganized and never-ending ..."
"... Agile is to proper software engineering what Red Bull Flugtag is to proper aeronautic engineering.... ..."
"... As TFA points out, that always works fine when your requirements are *all* known an are completely static. That rarely happens in most fields. ..."
"... The problem with Agile is that it gives too much freedom to the customer to change their mind late in the project and make the developers do it all over again. ..."
"... If you are delivering to customer requests you will always be a follower and never succeed. You need to anticipate what the customers need. ..."
"... It frequently is. It doesn't matter what methodology you use -- if you change major features/priorities at the last minute it will cost multiple times as much. Yet frequently customers expect it to be cheap because "we're agile". And by accepting that change will happen you don't push the customers to make important decisions early, ensuring that major changes will happen, instead of just being possible. ..."
"... The problem with all methodologies, or processes, or whatever today's buzzword is, is that too many people want to practice them in their purest form. Excessive zeal in using any one approach is the enemy of getting things done. ..."
"... On a sufficiently large project, some kind of upfront design is necessary. ..."
"... If you insist on spinning back every little change to a monstrously detailed Master Design Document, you'll move at a snail's pace. As much as I hate the buzzword "design patterns", some pattern is highly desirable. ..."
"... there is no substitute for good engineers who actually talk to and work with each other. ..."
"... If you don't trust those people to make intelligent decisions (including about when things do have to be passed up) then you've either got the wrong people or a micromanagement fetish ..."
"... The problem the article refers to about an upfront design being ironclad promises is tough. Some customers will work with you, and others will get their lawyers and "systems" people to waste your time complaining about every discrepancy, ..."
"... In defense everything has to meet spec, but it doesn't have to work. ..."
"... There is absolutely no willingness to make tradeoffs as the design progresses and you find out what's practical and necessary and what's not. ..."
"... I'll also admit that there is a tendency to get sloppy in software specs because it is easier to make changes. Hardware, with the need to order materials, have things fabbed, tape out a chip, whatever, imposes a certain discipline that's lacking when you know you can change the source code at anytime. Being both, I'm not saying this is because hardware engineers are virtuous and software engineers are sloppy, but because engineers are human (at least some of them). ..."
"... Impressive stuff, and not unique to the space shuttle. Fly-by-wire systems are the same way. You're talking DO-178B [wikipedia.org] Level A stuff. It works, and it's very very expensive. If it was only 10x the cost of normal software development I'd be amazed. I agree that way too much software is poorly planned and implemented crap, and part of the reason is that nobody wants realistic cost estimates or to make the difficult decisions about what it's supposed to do up-front. But what you're talking about is aerospace quality. You couldn't afford a car or even a dishwasher made to those standards. ..."
This article discusses how some experienced developers have changed that perception. '... She's
been frustrated by her Agile experiences - and so have her clients.
"There is no process. Things fly all directions, and despite SVN [version control] developers
overwrite each other and then have to have meetings to discuss why things were changed. Too many
people are involved, and, again, I repeat, there is no process.' The premise here is not that Agile
sucks - quite to the contrary - but that developers have to understand how Agile processes can make
users anxious, and learn to respond to those fears. Not all those answers are foolproof.
For example: 'Detailed designs and planning done prior to a project seems to provide a "safety
net" to business sponsors, says Semeniuk. "By providing a Big Design Up Front you are pacifying this
request by giving them a best guess based on what you know at that time - which is at best partial
or incorrect in the first place." The danger, he cautions, is when Big Design becomes Big Commitment
- as sometimes business sponsors see this plan as something that needs to be tracked against.
"The big concern with doing a Big Design up front is when it sets a rigid expectation that
must be met, regardless of the changes and knowledge discovered along the way," says Semeniuk.' How
do you respond to user anxiety from Agile processes?"
Agile summed up (Score:5, Funny)
Agile is to proper software engineering what Red Bull Flugtag is to proper aeronautic engineering....
Re: doesn't work
As TFA points out, that always works fine when your requirements are *all* known an are
completely static. That rarely happens in most fields.
Even in the ones where it does it's usually just management having the balls to say "No, you
can give us the next bunch of additions and changes when this is delivered, we agreed on that".
It frequently ends up delivering something less than useful.
Re: doesn't work (Score:5, Insightful)
The problem with Agile is that it gives too much freedom to the customer to change their
mind late in the project and make the developers do it all over again.
Re: doesn't work (Score:4, Insightful)
...but they can be trusted to say what is most important to them at the time.
No they can't. If you are delivering to customer requests you will always be a follower
and never succeed. You need to anticipate what the customers need. As with the I guess made
up quote attributed to Henry Ford, "If I listened to my customers I'd have been trying to make
faster horses." Whether he said it or not, the statement is true. Customers know what they have
and just want it to be faster/better/etc you need to find out what they really need.
Re: doesn't work (Score:5, Insightful)
It frequently is. It doesn't matter what methodology you use -- if you change major features/priorities
at the last minute it will cost multiple times as much. Yet frequently customers expect it to
be cheap because "we're agile". And by accepting that change will happen you don't push the customers
to make important decisions early, ensuring that major changes will happen, instead of just being
Re: doesn't work (Score:5, Interesting)
"Proper software engineering" doesn't work.
You're right, but you're going to the other extreme. The problem with all methodologies,
or processes, or whatever today's buzzword is, is that too many people want to practice them in
their purest form. Excessive zeal in using any one approach is the enemy of getting things done.
On a sufficiently large project, some kind of upfront design is necessary. Spending too much
time on it or going into too much detail is a waste though. Once you start to implement things,
you'll see what was overlooked or why some things won't work as planned. If you insist on
spinning back every little change to a monstrously detailed Master Design Document, you'll move
at a snail's pace. As much as I hate the buzzword "design patterns", some pattern is highly desirable.
Don't get bent out of shape though when someone has a good reason for occasionally breaking that
pattern or, as you say, you'll wind up with 500 SLOC's to add 2+2 in the approved manner.
Lastly, I agree that there is no substitute for good engineers who actually talk to and
work with each other. Also don't require that every 2 bit decision they make amongst themselves
has to be cleared, or even communicated, to the highest levels. If you don't trust those people
to make intelligent decisions (including about when things do have to be passed up) then you've
either got the wrong people or a micromanagement fetish. Without good people you'll never
get anything decent done, but with good people you still need some kind of organization.
The problem the article refers to about an upfront design being ironclad promises is tough.
Some customers will work with you, and others will get their lawyers and "systems" people to waste
your time complaining about every discrepancy, without regard to how important it is. Admittedly
bad vendors will try and screw their customers with "that doesn't matter" to excuse every screw-up
and bit of laziness. For that reason I much prefer working on in-house projects, where "sure we
could do exactly what we planned" gets balanced with the cost and other tradeoffs.
The worst example of those problems is defense projects. As someone I used to work with said:
In defense everything has to meet spec, but it doesn't have to work. In the commercial
world specs are flexible, but it has to work.
If you've ever worked in that atmosphere you'll understand why every defense project costs
a trillion dollars. There is absolutely no willingness to make tradeoffs as the design progresses
and you find out what's practical and necessary and what's not. I'm not talking about meeting
difficult requirements if they serve a purpose (that's what you're paid for) but being unwilling
to compromise on any spec that somebody at the beginning of the project pulled out of their posterior
and obviously doesn't need to be so stringent. An elephant is a mouse built to government specifications.
Ok, you can get such things changed, but it requires 10 hours from program managers for every
hour of engineering. Conversely, don't even think about offering a feature or capability that
will be useful and easy to implement, but is not in the spec. They'll just start writing additional
specs to define it and screw you by insisting you meet those.
As you might imagine, I'm very happy to be back in the commercial world.
Re: doesn't work (Score:2, Interesting)
You've fallen into the trap of using their terminology. As soon as 'the problem' is defined
in terms of 'upfront design', you've already lost half the ideological battle.
'The problem' (with methodology) is that people want to avoid the difficult work of thinking
hard about the business/customer's problem and coming up with solutions that meet all their needs.
But there isn't a substitute for thinking hard about the problem and almost certainly never will
The earlier you do that hard thinking about the customer's problems that you are trying to
solve the cheaper, faster and better quality the result will be. Cheaper? Yes, because bugfixing
that is done later in the project is a lot more expensive (as numerous software engineering studies
have shown) Faster? Yes, because there's less rework. (Also, since there is usually a time = money
equivalency, you can't have it done cheap unless it is also done fast. Higher quality? Yes, because
you don't just randomly stumble across quality. Good design trumps bad design every single time.
... ... ...
Re: doesn't work (Score:4, Interesting)
Until the thing is built or the software is shipped there are many options and care should
be taken that artificial administrative constraints don't remove too many of them.
Exactly, and as someone who does both hardware and software I can tell you that that's better
understood by Whoever Controls The Great Spec in hardware than in software. Hardware is understood
to have physical constraints, so not every change is seen as the result of a screw-up. It's a
I'll also admit that there is a tendency to get sloppy in software specs because it is easier
to make changes. Hardware, with the need to order materials, have things fabbed, tape out a chip,
whatever, imposes a certain discipline that's lacking when you know you can change the source
code at anytime. Being both, I'm not saying this is because hardware engineers are virtuous and
software engineers are sloppy, but because engineers are human (at least some of them).
This is my evidence that "proper software engineering" *can* work. The fact that most businesses
(and their customers) are willing to save money by accepting less from their software is not the
fault of software engineering. We could and did build buildings much faster than we do today,
if you are willing to make more mistakes and pay more in human lives. If established industries
and their customers began demanding software at that higher standard and were willing to pay for
it like it was real engineering, then maybe it would happen more often.
Impressive stuff, and not unique to the space shuttle. Fly-by-wire systems are the same way.
You're talking DO-178B [wikipedia.org] Level A stuff. It works, and it's very very expensive.
If it was only 10x the cost of normal software development I'd be amazed. I agree that way too
much software is poorly planned and implemented crap, and part of the reason is that nobody wants
realistic cost estimates or to make the difficult decisions about what it's supposed to do up-front.
But what you're talking about is aerospace quality. You couldn't afford a car or even a dishwasher
made to those standards.
Re: doesn't work (Score:3)
260 people maintaining 420,000 lines of code, written to precise externally provided specifications
that change once every few years.
This is fine for NASA, but if you want something that does roughly what you need before your competitors
come up with something better, you'd better find some better programmers.
In light of all the hype, we have created a DevOps parody Series – DevOps: Fact or Fiction .
those of you who did not see, in October we created an entirely separateblog(inspired by
this ) – however decided that
it is relevant enough to transform into a series on the
AppDynamics Blog . The series will point
out the good, the bad, and the funny about IT and DevOps. Don't take anything too seriously – it's
nearly 100% stereotypes : ).
Stay tuned for more DevOps: Fact or Fiction to come. Here we go
"... Start-ups taught us this. Good developers can be passable DBAs, if need be. They make decent testers, "deployment engineers", and whatever other ridiculous term you'd like to use. Their job requires them to know much of the domain of "lower" roles. There's one big problem with this, and hopefully by now you see it: It doesn't work in the opposite direction. ..."
"... An example will make this more clear. My dad is a dentist running his own practice. He employs a secretary, hygienist, and dental assistant. Under some sort of "DentOps" movement, my dad would be making appointments and cleaning people's teeth while trying to find time to drill cavities, perform root canals, etc. My dad can do all of the other jobs in his office, because he has all the specialized knowledge required to do so. But no one, not even all of his employees combined, can do his job. ..."
"... Such a movement does a disservice to everyone involved, except (of course) employers. What began as an experiment aimed at increasing software quality has become a farce, where the most talented employees are overworked (while doing less, less useful work) and lower-level positions simply don't exist. ..."
"... you're right. Pure DevOps is no more efficient or sensible than pure Agile (or the pure Extreme Programming or the pure Structured Programming that preceeded it). The problem is purists and ideological zealotry not the particular brand of religion in question. Insistence on adherence to dogma is the problem as it prevents adoption of flexible, 'fit for purpose' solutions. Exposure to all the alternatives is good. Insisting that one hammer is ideal for every sort of nail we have and ever will have is not. ..."
"... There are developers who have a decent set of skills outside of development in QA, Operations, DB Admin, Networking, etc. Equally so, there are operations engineers who have a decent set of skills outside of operations in QA, Development, DB Admin, networking, etc. Extend this to QA and other disciplines. What I have never seen is one person who can perform all those jobs outside of their main discipline with the same level of professionalism, experience and acumen that each of those roles require to do it well at an Enterprise/World Class level. ..."
"... I prefer to think of DevOps as more of a full-stack team concept. Applying the full-stack principle at the individual levels is not sustainable, as you point out. ..."
"... DevOps roles are strictly automation focused, at least according to all job specifications I see on the internet. They don't need any development skills at all. To me it looks like a new term for what we used to call IT Operations, but more scripting/automation focused. DevOps engineer will need to know Puppet, Chef, Ansible, OS management, public cloud management, know how to set up monitoring, logging and all that stuff usual sysadmin used to do but in the modern world. In fact I used to apply for DevOps roles but quickly changed my mind as it turned out no companies need a person wearing many hats, it has absolutely nothing to do with creating software. Am I wrong? ..."
There are two recent trends I really hate: DevOps and the notion of the "full-stack" developer.
The DevOps movement is so popular that I may as well say I hate the x86 architecture or monolithic
kernels. But it's true: I can't stand it. The underlying cause of my pain? This fact: not every
company is a start-up, though it appears that every company must act as though they were.
"DevOps" is meant to denote a close collaboration and cross-pollination between what were previously
purely development roles, purely operations roles, and purely QA roles. Because software needs to
be released at an ever-increasing rate, the old "waterfall" develop-test-release cycle is seen as
broken. Developers must also take responsibility for the quality of the testing and release environments.
The increasing scope of responsibility of the "developer" (whether or not that term is even appropriate
anymore is debatable) has given rise to a chimera-like job candidate: the "full-stack" developer.
Such a developer is capable of doing the job of developer, QA team member, operations analyst, sysadmin,
and DBA. Before you accuse me of hyperbole, go back and read that list again. Is there any role in
the list whose duties you wouldn't expect a "full-stack" developer to be well versed in?
Where did these concepts come from? Start-ups, of course (and the Agile methodology). Start-ups
are a peculiar beast and need to function in a very lean way to survive their first few years. I
don't deny this . Unfortunately, we've taken the multiple technical roles that engineers at start-ups
were forced to play due to lack of resources into a set of minimum qualifications for the
role of "developer".
Imagine you're at a start-up with a development team of seven. You're one year into development
of a web applications that X's all the Y's and things are going well, though it's always a frantic
scramble to keep everything going. If there's a particularly nasty issue that seems to require deep
database knowledge, you don't have the liberty of saying "that's not my specialty," and handing it
off to a DBA team to investigate. Due to constrained resources, you're forced to take on the role
of DBA and fix the issue yourself.
Now expand that scenario across all the roles listed earlier. At any one time, a developer at
a start-up may be acting as a developer, QA tester, deployment/operations analyst, sysadmin, or DBA.
That's just the nature of the business, and some people thrive in that type of environment. Somewhere
along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up
developer had to take on different roles he or she should actually be all those things
If such people even existed , "full-stack" developers still wouldn't be used as they should.
Rather than temporarily taking on a single role for a short period of time, then transitioning
into the next role, they are meant to be performing all the roles, all the time . And here's what
really sucks: most good developers can almost pull this off.
The Totem Pole
Good developers are smart people. I know I'm going to get a ton of hate mail, but there is
a hierarchy of usefulness of technology roles in an organization. Developer is at the top, followed
by sysadmin and DBA. QA teams, "operations" people, release coordinators and the like are at the
bottom of the totem pole. Why is it arranged like this?
Because each role can do the job of all roles below it if necessary.
Start-ups taught us this. Good developers can be passable DBAs, if need be. They make decent
testers, "deployment engineers", and whatever other ridiculous term you'd like to use. Their job
requires them to know much of the domain of "lower" roles. There's one big problem with this, and
hopefully by now you see it: It doesn't work in the opposite direction.
A QA person can't just do the job of a developer in a pinch, nor can a build-engineer do the job
of a DBA. They never acquired the specialized knowledge required to perform the role. And
that's fine. Like it or not, there are hierarchies in every organization, and people have different
skill sets and levels of ability. However, when you make developers take on other roles, you don't
have anyone to take on the role of development!
An example will make this more clear. My dad is a dentist running his own practice. He employs
a secretary, hygienist, and dental assistant. Under some sort of "DentOps" movement, my dad would
be making appointments and cleaning people's teeth while trying to find time to drill cavities, perform
root canals, etc. My dad can do all of the other jobs in his office, because he has all the specialized
knowledge required to do so. But no one, not even all of his employees combined, can do his job.
Such a movement does a disservice to everyone involved, except (of course) employers. What
began as an experiment aimed at increasing software quality has become a farce, where the most talented
employees are overworked (while doing less, less useful work) and lower-level positions simply don't
And this is the crux of the issue. All of the positions previously held by people of various levels
of ability are made redundant by the "full-stack" engineer. Large companies love this, as it means
they can hire far fewer people to do the same amount of work. In the process, though, actual development
becomes a vanishingly small part of a developer's job . This is why we see so many developers
that can't pass FizzBuzz: they never really had to write any code. All too common a question now,
can you imagine interviewing a chef and asking him what portion of the day he actually devotes to
Jack of All Trades, Master of None
If you are a developer of moderately sized software, you need a deployment system in place. Quick,
what are the benefits and drawbacks of the following such systems: Puppet, Chef, Salt, Ansible, Vagrant,
Docker. Now implement your deployment solution! Did you even realize which systems had no business
being in that list?
We specialize for a reason: human beings are only capable of retaining so much knowledge. Task-switching
is cognitively expensive. Forcing developers to take on additional roles traditionally performed
by specialists means that they:
aren't spending their time developing
need to keep up with an enormous domain of knowledge
are going to burn out
What's more, by forcing developers to take on "full-stack" responsibilities, they are paying their
employees far more than the market average for most of those tasks. If a developer makes 100K
a year, you can pay four developers 100K per year to do 50% development and 50% release management
on a single, two-person task. Or, simply hire a release manager at, say, 75K and two developers
who develop full-time. And notice the time wasted by developers who are part time release-managers
but don't always have releases to manage.
Don't Kill the Developer
The effect of all of this is to destroy the role of "developer" and replace it with a sort of
"technology utility-player". Every developer I know got into programming because they actually enjoyed
doing it (at one point). You do a disservice to everyone involved when you force your brightest people
to take on additional roles.
Not every company is a start-up. Start-ups don't make developers wear multiple hats by choice,
they do so out of necessity. Your company likely has enough resource constraints without you inventing
some. Please, don't confuse "being lean" with "running with the fewest possible employees". And for
God's sake, let developers write code!
Some background... I started life as a dev (30years ago), have mostly been doing sysadmin and
project tech lead sorts of work for the last 15. I've always assumed the DevOps movement was resulting
in sub-par development and sub-par sysadmin/ops precisely because people were timesharing their
But what it does bring to the party is a greater level of awareness of the other guys problems.
There's nothing quite like being rung out of bed at 3am to motivate a developer to improve his
products logging to make supporting it easier. Similarly the admin exposed to the vagaries of
promoting things into production in a supportable, repeatable, deterministic manner quickly learns
to appreciate the issues there. So DevOps has served a purpose and has offered benefits to the
organisations that signed on for it.
But, you're right. Pure DevOps is no more efficient or sensible than pure Agile (or the
pure Extreme Programming or the pure Structured Programming that preceeded it). The problem is
purists and ideological zealotry not the particular brand of religion in question. Insistence
on adherence to dogma is the problem as it prevents adoption of flexible, 'fit for purpose' solutions.
Exposure to all the alternatives is good. Insisting that one hammer is ideal for every sort of
nail we have and ever will have is not.
I'm very disappointed to see this kind of rubbish. It's this type of egocentric thinking and generalization
that the developer is an omniscient deity requiring worshiping and pampering that prevents DevOps
from being successful. Based on the tone and your perspective it sounds like you've been doing
A developer role alone is not the linchpin that keeps DevOps humming - instead it's the respect
that each team member holds for each discipline and each team member's area of expertise, the
willingness of the entire team to own the product, feature delivery and operational stability
end to end, to leverage each others skills and abilities, to not blame Dev or Ops or QA for failure,
and to share knowledge.
There are developers who have a decent set of skills outside of development in QA, Operations,
DB Admin, Networking, etc. Equally so, there are operations engineers who have a decent set of
skills outside of operations in QA, Development, DB Admin, networking, etc. Extend this to QA
and other disciplines. What I have never seen is one person who can perform all those jobs outside
of their main discipline with the same level of professionalism, experience and acumen that each
of those roles require to do it well at an Enterprise/World Class level.
If you're a developer doing QA and operations, you're doing it because you have to, but there
should be no illusion that you're as good in alternate roles as someone trained and experienced
in those disciplines. To do so is a disservice to yourself and your organization that signs your
paycheck. If you're in this situation and you'd prefer making a difference rather than spewing
complains, I would recommend talking to your manager and above about changing their skewed vision
of DevOps. If they aren't open to communication, collaboration, experimentation and continual
improvement, then their DevOps vision is dysfunctional and they're not supporting DevOps from
the top down. Saying your DevOps and not doing it is *almost* more egregious than saying the developer
is the top of a Totem Pole of existence.
he prefaced it with 'crybabies please ignore' It's his opinion. That everyone but the lower totem
pole people agree with so.. agree to disagree. I also don't think being at the bottom of the totem
pole is a big f'in deal. If you're getting paid.. embrace it! So many other ways to enjoy life!
The top dog people have all the pressure and die young! 99% of the people on earth dont know the
difference between one nerd and another. And other nerds are always going to be egomaniacs who
will find some way to justify their own superiority no matter what your achievements. So this
kind of posturing is a waste of time.
I think there's a problem with your definition of DevOps. It doesn't mean developers have to be
"full-stack" or do ops stuff. And it doesn't mean "act like a startup." It simply means, at its
basis, that Developers and Operations work well together and do not have any communication barriers.
This is why I hate DevOps as a title or department, because DevOps is a culture.
Let's take your DentOps example. The dentist has 3 support staff. What if they rarely spoke
to the dentist? What if they were on different floors of the building? What if the dentist wrote
an email about how teeth should be cleaned and wasn't available to answer questions or willing
to consider feedback? What if once in a while the dentist needed to understand enough about the
basics of appointment scheduling to point out problems with the system? Maybe appointments are
being scheduled too close together. Would the patients get backed up throughout the day because
that's the secretary's problem? Of course not. Now we'd be getting into a more accurate analogy
to DevOps. If anything a dentist's office is ALREADY "DentOps" and the whole point of DevOps is
to make the dev/ops interaction work in a logical culture that other industries (like dentists)
I would tend to agree with some of that. Being able to trouble shoot network issues using monitoring
tools like Fiddler is a good thing to be aware of. I can also see a lot of companies using it
as a way to make one person do everything. Moreover, there are probably folks out there that perpetuate
that behavior by taking on the machismo argument.
By saying that if I can do it that you should be able to do it too or else you're not as good
of a developer as I am. I have never heard anyone outright claim this, but I've seen this attitude
time and time again from ambitious analysts looking to get a leg up, a pay raise, and a way to
template their values on the rest of the team. One of the first things that you're taught as a
dev is that you can't hope to know it all.
Your responsibility first and foremost as a developer is the stability and reliability of your
code and the services that you provide. In some industries this is literally a matter of life
and death(computers in your car, mission critical medical systems). It doesn't work for everyplace.
I wouldn't want to pay a receptionist 200k a year like a dentist though. Learn to hire better
receptionists. Even a moderately charming woman can create more customer loyalty, and cheaper,
than the best dentist in the world. I want my dentist to keep quiet and have a steady hand. I
want my receptionist to engage me and acknolwedge my existence.
I want my secretary to be a multitasking master. I want my dentist not to multitask at all
Good points, I tend to agree. I prefer to think of DevOps as more of a full-stack team concept.
Applying the full-stack principle at the individual levels is not sustainable, as you point out.
The full-stack DevOps team will have team members with primary skills in either of the traditional
specialties, and will, over time, develop decent secondary skills. But the value is not in people
constantly content switching - that actually kills efficiency. The value is in developers understanding
and developing an open relationship with testing and operations - and vice versa. And this cooperation
is inhibited by putting people in separate teams with conflicting goals. DevOps in practice is
not a despecialization. It's bringing the specialists together.
The more isolated or silo'd developers become, the less they realize what constitutes delivering
software, and the more problems are contributed to the IT process of test/build/release/scale/monitor,
etc. Writing code is a small fraction of that delivery process. I've written about the success
of devops and microservices that touches on this stuff because they're highly related. The future
success of devops/microservices/cloud/etc isn't related to technology insofar as it is culture:
Great article and you're definitely describing one form of dysfunctional organisation where DevOps,
Agile, Full Stack, and every other $2 word has been corrupted to become a cost cutting justification;
cramming more work onto people who aren't skilled for it, and eho end up not having any time to
do what they were hired as experts for!
But I'd also agree with other posters that it's a little developer centric. I'm a terrible
programmer and a great DBA. I can tell you most programmers who try to be DBAs are equally terrible.
It's definitely not "doing the job of the receptionist" 😄
And we shouldn't forget what DevOps is meant to be about; teams making sure nobody gets called
at night to fix each other's messes. That means neither developers with shitty deployments straight
to production nor operations letting the disks silently fill because "if it ain't C: it ain't
I know of 0 developers that can manage a network of any appreciable scale.
In cloud and large enterprise networks, if there were a totem (which there isn't) using your
methodology would place the dev under the network engineer. Their software implements the protocol
and configuration intent of the NE. Good thing the whole concept is a pile of rubbish. I think
you fell into the trap you called out which is thinking at limited scale.
It's true. We can all create LAN's at home but I wouldn't dare f with a corporate network and
risk shutting down amazon for a day. Which seems to happen quite a bit.... maybe they're DEVOPPING
a bit too much.
Jeff Knupp is to one side of the spectrum. DevOps Reaper is to the other side.
Enno is more
attune to what is really going on. So I won't repeat any of those arguments.
However I will ask you to put me in a box. What am I?
I graduated as a Computer Engineer (hybrid between Electrical Engineering and Computer Science).
I don't say that anymore as companies have no idea as to what that means. So I called myself a
Digital Electronics and Software Engineer for a while. The repeated question was all too often:
"So what are you, software or hardware?"
I spent my first few years working down from board design, writing VHDL and Verilog, to embedded
software in C and C++, then algorithms in optimization with the CUDA framework in C, with C++
wrappers and C# for the logic tier. Then worked another few year in particle physics with C++
compute engines with x86 assembly declarations for speed and C# for WPF UIs.
After that I went to work for a wind turbines company as system architect where is was mostly
embedded and programming ARM Cortex microprocessors, high power electronics controls, custom service
and diagnostics tools in C#. Real-time web based dashboards with Angular, Bootrap, and the likes
for a good looking web app.
Nowadays I'm working with mobile first web applications that have a massive backend to power them.
It is mostly a .NET stack form Entity Framework, to .NET WebAPI, to Angular power font ends. This
company is not a start up but it is a small company. Therefore I wear the many hats. I introduced
the new software life cycle with includes continuous integration and continuous deployment. Yes,
I manage build servers, build tools, I develop, I'm QA, I'm a tester, I'm a DBA., I'm the deployment
and configuration manager.
If you are wondering I have resorted to start calling a full stack developer. It has that edgy
sound that companies like to hear. I'm still a young developer. I've only been developing for
In my team we are all "Jack of all Trades" and "Masters of Many". We switch tasks and hats
because it is fun and keep everyone from getting bored/stuck. Our process is called "Best practices
that work for this team".
So, I think of myself as a software engineer. I think I'm a developer. I think I'm DevOps,
I think I'm QA.
Lets start with that DevOps didn't come from startups. It came from Boeing mainly, and a few other
major blue chip IT shops, investing heavily in systems management technology around the turn of
the century. The goal at the time was simply to change the ratio of servers to IT support personnel,
and the re-thinking and re-organizing of development and operations into one organization with
one set of common goals. The 'wearing many hats' thing you discuss is a feature of startups, but
that feature is independent of siloed or integrated organizations.
I prefer the 'sportzing' analogy of basketball and football. Football has specialist teams
that are largely functionally independent because they focus on distinct goals. Basketball has
specialist positions, but the whole team is focused on the same goals. I'm not saying one sport
is better than the other. I am saying the basketball mentality works better in the IT environment.
Delivering the product or service to the customer is the common goal that everyone should be thinking
about, and how the details of their job fits into that overall picture. It sounds like to me you
are really saying "Hey, its my job and only my job to think about how it all fits together and
Secondly, while it is pretty clear that the phrase 'full stack engineer' is about as useful
as "Cloud Computing", your perspective that somehow developers are the 'top' of the tree able
to do any job is very mistaken. There are key contributors from every specialty who have that
ability, and more useful names for them are things like "10x", or "T-shaped". Again, you are describing
a real situation, but correlating it with unrelated associations. It is just as likely, and just
as valuable, to find an information architect who can also code, or a systems admin that can also
diagnose database performance, or an electrician that can also hang sheetrock. Those people do
fit your analogy of 'being on top', because they are not siloed and stovepiped into just their
The DevOps mindset fosters this way of thinking, instead of the old and outdated specialist
way of thinking you are defending. Is it possible your emotional reaction is fear based against
the possibility that your relative value will decrease if others start thinking outside their
Interesting to note that Agile also started at Boeing, but 10 years earlier. I live in the
startup world of Seattle, but know my history and realize that much of what appears new is actually
just 'new to you'(or me), and that most of cutting edge technology and thinking is just combining
ideas from other industries in new ways.
The problem is that developers are trained to crank out code and hope that QA teams will find
problems, many times not even sure how to catch holes. DevOps trains people to think critically
and do both. It isn't killing developers, it is making them look like noobs while phasing them
Yeah, good luck with that attitude. Your company's gonna have a good'ole time looking for and
keeping new developer talent. Because as we all know, smart people love working with dummies.
I'd love to see 'your QA' team work on our 'spatial collision algorithm' and make our devs "look
like noob". You sound like most middle management schmucks.
Funniest article so far on full stack. It's a harsh reality for devs, because were asked to do
everything, know everything, so how can you really believe QA or DBA can do the job of someone
like that? There is a crazy amount of hours a fullstack dev invests in aquiring that kind of knowledge,
not to mention some people are also talented at their job. Imagine trying to tell the QA to do
that? Maybe for a few hours someone can be a backup just in case something happens, but really
it's like replacing the head surgeon.
The best skill you can learn in your coding career is your next career. Noone wants a 45 year
I see so much time wasted learning every new thing when you should just be plugging away to
get the job done, bank the $$, and move on. All your accumulated skills will be worthless in a
decade or so, and your entire knowledge useless in 2 decades. My ability to turn a wrench is what's
keeping me from the poor house. And I have a engineering degree from UIUC! I also don't mind.
Think about a 100 week as a plumber with OT in a reasonably priced neighborhood, vs a coder. Who
do you think is making more? Now I'm not saying you cant survive into your 50's programming, but
typically they get retired forcefully, and permanently.. by a heart attack!
But rambling aside.. the author makes a good point and i think is the future of big companies
in tech. The current model is driven by temporary factors. Ideally you'd have a specialized workforce.
But I think that as a programmer you are in constant fear of being obsolete so you don't want
to be pigeon-holed. It's just not mathematically possible to have that 10,000 hour mastery in
50 different areas.. unless you are Bill Murray in Groundhog Day.
A developer who sees himself at the top of a pyramid. Not surprising, your myopic and egotistical
view. I laugh at people who code a few SELECT statements and think they can fill the DBA role.
HA HA HA. God, the arrogance. "Well it worked on my machine." - How many sys admins have heard
this out of a developers mouth. Unfortunately, projects get stuck with supporting such issues
because that very ego has led the developer too far down the road to turn back. They had no common
sense or modesty to call on the knowledge of their Sys Ops team to help design the application.
I interview jobs candidates all the time calling themselves full stack simply because they compliment
their programming language of choice with a mere smattering of knowledge in client-side technologies
and can write a few SQL queries. Most developers have NO PERCEPTION of the myriad intricacies
it takes to get an application from their unabated desktop with its FULL ADMIN perms and "unlimited
resources", through a staging/QA environment, and eventually to the securely locked down production
system with limited and perhaps shared or hosted limited resources. Respect of your support teams,
communication and coordination, and the knowledge that you do not know it all. THAT'S being Full
Stack and DevOps sir.
there's always that one query that noone can do in a way that takes less than 2 hours until u
pass it off to a real DBA.. its the 80/20 rule basically. I truly dont believe 'full stack' exists.
It's an illusion. There's always something that suffers.
The real problem is smart people are in such demand we're forced to adapt to this tribal pre-civilization
hodgepodge. Once the industry matures, it'll disappear. Until then they will think they re-invented
'm confused here. DevOps roles are strictly automation focused, at least
according to all job specifications I see on the internet. They don't need any development skills
at all. To me it looks like a new term for what we used to call IT Operations, but more scripting/automation
focused. DevOps engineer will need to know Puppet, Chef, Ansible, OS management, public cloud management,
know how to set up monitoring, logging and all that stuff usual sysadmin used to do but in the modern
world. In fact I used to apply for DevOps roles but quickly changed my mind as it turned out no companies
need a person wearing many hats, it has absolutely nothing to do with creating software. Am I wrong?
It depends on what you mean with development skills. Have you ever tried to automate the deployment
of a large web application? In fact the scripts that automate the deployment of large scalable
web applications are pretty complex softwares which require in-depth thinking and should follow
all the important principles a good developers should know: components isolation, scalability,
maintainability, extensibility, etc..
Successful DevOps doesn't mean a full stack developer does it all, that's only true for a broken
company that succeeded despite bad organization. For example, Twitter's Dev only culture is downright
sick, and ONLY works because they are in the tech field. Mind you, I still believe personally
that it works for them DESPITE its unbalanced structure. In other words, bad DevOps means the
Dev has no extra resources and just more requirements, yea that sucks!....
BUT, on the flip,
Infrastructure works with QA/Build to define supportable deployment standards, they gotta learn
all the automatic bits and practice using them. Now Devs have to package all their applications
properly, in the formats supported by QABuild's CI and Repositories (that 'working just fine'
install script definitly doesn't count). BUT the Dev's get pre-made CI-ready examples, and if
needed, code-migration assistance from the QA/Build team. Pretty soon they learn how to package
that type of app, like a J2EE Maven EAR, or a Webdeploy to IIS.... and the rest should be hadled
for them, as automaticlly as possible by the proactive operations teams.
Make sense? This is how its supposed to work, It sounds like your left alone in a terrible
Dev Only/heavy world. The key to DevOps that is great, and everybody likes, vs. more work... is
having a very balanced work flow between the teams, and making sure the pass-off points are VERY
well defined. Essentially it requires management that cut the responsibility properly, so they
have a shared interest in collaborating. In a Dev heavy organization, the Devs can just throw
garbage over the wall, and operations has to react to constant problems... they start to hate
each other and ..... Dev managers get the idea that they can cut out ops if they do "DevOps",
so then they throw it all at you like right now.
I see in this post so much rubbish and narrow mindedness, so much of the exact stuff that is
killing any type of companies. In the last 10 years I had many roles that required of me, as a
system engineer, to come in and straighten out all kind of really bad compromises developers did
just to make stuff work.
The role never shows the level of intelligence or capabilities. I've seen so many situations
in the last 10 years when smart people with the wrong attitude and awareness are too smart for
anyone's good and limited people still providing more value than a very smart ones acting as if
he is too smart to even have a conversation about anything.
This post is embarrassing for you Jeff, I am sorry for you man.... you just don't get it!
A developer do not have to do full stack, the developer can continue with development, but has
to adopt some things for packaging,testing, and how it is operated.
Operations can continue with operations, but has to know how things are built and packaged.
Developers and operations needs to share things like use the same application server for example.
Developer needs to understand how it is operated to make sure that the code is written in a proper
way. Operations needs to adopt in the need for fast delivery and be able to support a controlled
way of deploying daily into production.
Here is a complementing post I have around the topic
I will share my experience , I started off my career teaching Programming which included database
programming (Oracle-pl/sql, SQL Server-transact sql) which gave good insights into database internals
which landed me into DBA world for last 10 years . During these 10 years where i have worked in
technology companies regarded as top-notch , I have seen very smart Developers writing excellent
application codes but missing out on writing optimized piece to interact with the database. Hence,
I think each Job has a scale and professionals of any group can not do what the top professionals
of other group can do. I have seen Developers with fairly good database internals knowledge and
I have dbas writing code for their automation which can compares well with features of some commercial
database products like TOAD. So , generalization like this does not hold.
The idea that there is a hierachy of usefulness is bunk. Most developers are horrible at operations
because they dislike it. Most sysadmins and DBAs are horrible at coding because they dislike it.
People gravitate to what interests them and a disinterested person does a much poorer job than
an interested one. DevOps aims to combine roles by removing barriers, but there are costs to quality
that no one likes to talk about. Using your hierarchy example most doctors could obtain their
RN but they would not make good nurses.
This is an excellent article on the general concepts of DevOps and the DevOps movement. It helps
to identify the cultural shifts required to facilitate proper DevOps implementations. I also write
about DevOps.. I authored a book on implementing CI, CD and DevOps related functions within an
organization and it was recently published. The book is aptly titled Mastering Jenkins (
http://www.masteringjenkins... ) and aims to codify not only the architectural implementations
and requirements of DevOps but the cultural shift needed to propery advocate for the proper adoption
of DevOps practices. Let me know what you think.
I agree. Although I'm not in the business (yet), I will be soon. What I've noticed just playing
around with Vagrant and Chef, Puppet, Ansible is the great amount of time to try and master just
one of these provisioners. I can't imagine being responsible for all these roles you spoke of
in the article. How can one possibly master all of them, and be good at any of them?
hmmm.... users & business see as one application.... for them how it was developed, deployed does
not matter.... IT is an enabler by definition... so DevOps is mostly about that... giving one
view to the customer; quick changes, stable changes, stable application....
Frankly, DevOps is not about developers or testers... it is about the right architecture, right
framework... developers/testers anyways do what is there in the script... DevOps is just a new
scripts to them.
For right DevOps, you need right framework. architecture for the whole of the program; you
need architecture which is built end to end and not in silos...
Software Developer write code that business/customer use
Test Developer write test code to test SUT
Release Developer write code to automate release process
Infrastructure developer write code to create infrastructure automatically.
Performance Developer writes code to performance test the SUT
Security Developer writes code to scan the SUT for security
Database Developer write code for DB
So which developer are you thinking DevOps going to kill?
Today's TDD world, a developer (it could be anyone above) needs to get out of their comfort
zone to makes sure they write a testable, releasable, deployable, performable, security complaint
and maintainable code.
DevOps brings all this roles together to collaborate and deliver.
Why wouldn't they be? What are the basic responsibilities that make for a passable DBA and which
of those responsibilities cannot be done by a good developer? Say a good developer has just average
experience writing stored procs, analyzing query performance, creating (or choosing not to create,
for performance reasons) indexes, constraints and triggers, configuring database access rights,
setting up regular backups, regular maintenance (ex. rebuilding indexes to avoid fragmentation)...
just to name a few.
I'm sure there's several responsibilities that DBA's have that developers
would have very little to no experience in, but we're talking about making for a passable DBA.
Developers may not be as good at the job as someone who specializes in it for a living, but the
author's wording seems to have been chosen very carefully.
Yup, I see lots of people trying to defend the DBA as a thing, just like people keep
trying to defend the traditional sysadmin as a thing. I started my career as a sysadmin in the 90s,
but times have changed and I don't call myself a sysadmin anymore, because that's not what I do.
Now I'm a Systems Engineer/SRE. My mode of working isn't slamming software together, but engineering
automation to do it for me.
But I also do QA, Data storage performance analysis, networking, and [have a] deep knowledge of the applications
"... The Emergence of the "DevOps' DevOp", a pseudo intellectual loudly spewing theories about distantly unrelated fields that are entirely irrelevant and are designed to make them feel more intelligent and myself more inadequate ..."
"... "The Copenhagen interpretation certainly applies to DevOps" ..."
"... "I'm modeling the relationship between Dev and Ops using quantum entanglement, with a focus on relative quantum superposition - it's the only way to look at it. Why aren't you?" ..."
"... Enterprise Architects. They used to talk about the "Enterprise Continuum". Now they talk about "The Delivery Continuum" or the "DevOps Continuum". ..."
DevOps and I sort of have a love/hate relationship. DevOps is near and dear to our heart here at
UpGuard and there are plenty
of things that I love about it . Love it or hate it, there is little doubt that it is here to
stay. I've enjoyed a great deal of success thanks to agile software development and DevOps methods,
but here are 10 things I hate about DevOps!
#1 Everyone thinks it's about Automation.
#2 "True" DevOps apparently have no processes - because DevOps takes care of that.
#3 The Emergence of the "DevOps' DevOp", a pseudo intellectual loudly spewing theories about
distantly unrelated fields that are entirely irrelevant and are designed to make them feel more intelligent
and myself more inadequate:
"The Copenhagen interpretation certainly applies to DevOps"
"I'm modeling the relationship between Dev and Ops using quantum entanglement, with a focus
on relative quantum superposition - it's the only way to look at it. Why aren't you?"
#4 Enterprise Architects. They used to talk about the "Enterprise Continuum". Now they talk
about "The Delivery Continuum" or the "DevOps Continuum". How about talking about the business
#5 Heroes abound with tragic statements like "It took 3 days to automate everything.. it's great
now!" - Clearly these people have never worked in a serious enterprise.
#6 No-one talks about Automation failure...it's everywhere. i.e Listen for the words "Pockets
of Automation". Adoption of technology, education and adaptation of process is rarely mentioned (or
#7 People constantly pointing to Etsy, Facebook & Netflix as DevOps. Let's promote the stories
of companies that better represent the market at large.
#8 Tech hipsters discounting, or underestimating, Windows sysadmins. There are a lot of them and
they better represent the Enterprise than many of the higher profile blowhards.
#9 The same hipsters saying their threads have filled up with DevOps tweets where there were none
#10 I've never heard of a Project Manager taking on DevOps. I intend on finding one.
What do you think - did I miss anything? Rants encouraged ;-) Please add your comments.
"... DevOps. The latest software development fad. ..."
"... Continuous Delivery (CD), the act of small, frequent, releases was defined in detail by Jez Humble and Dave Farley in their book – Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. The approach makes a lot of sense and encourages a number of healthy behaviors in a team. ..."
"... The problem is we now have teams saying they're doing DevOps. By that they mean is they make small, frequent, releases to production AND the developers are working closely with the Ops team to get things out to production and to keep them running. ..."
"... Well, the problem is the name. We now have a term "DevOps" to describe the entire build, test, release approach. The problem is when you call something DevOps anyone who doesn't identify themselves as a dev or as Ops automatically assumes they're not part of the process. ..."
DevOps. The latest software development fad. Now you can be Agile, use Continuous Delivery, and believe
Continuous Delivery (CD), the act of small, frequent, releases was defined in detail by Jez Humble
and Dave Farley in their book – Continuous Delivery: Reliable Software Releases Through Build, Test,
and Deployment Automation. The approach makes a lot of sense and encourages a number of healthy
in a team. For example, frequent releases more or less have to be small. Small releases are easier
to understand, which in turn increases our chances of building good features, but also our chances
of testing for the right risks. If you do run into problems during testing then it's pretty easy
to work out the change that caused them, reducing the time to debug and fix issues.
Unfortunately, along with all the good parts of CD we have a slight problem. The book focused
on the areas which were considered to be the most broken, and unfortunately that led to the original
CD description implying "Done" meant the code was shipped to production. As anyone who has ever worked
on software will know, running code in production also requires a fair bit of work.
So, teams started adopting CD but no one was talking about how the Ops team fitted into the release
cycle. Everything from knowing when production systems were in trouble, to reliable release systems
was just assumed to be fully functional, and unnecessary for explanation.
To try to plug the gap DevOps rose up.
Now, just to make things even more confusing. Dave Farley later said that not talking about Ops
was an omission and CD does include the entire development and release cycle, including running in
production. So DevOps and CD have some overlap there.
DevOps does take a slightly different angle on the approach than CD. The emphasis for DevOps is
on the collaboration rather than the process. Silos should be actively broken down to help developers
understand systems well enough to be able to write good, robust and scalable code.
So far so good.
The problem is we now have teams saying they're doing DevOps. By that they mean is they make small,
frequent, releases to production AND the developers are working closely with the Ops team to get
things out to production and to keep them running.
Sounds good. So what's the problem?
Well, the problem is the name. We now have a term "DevOps" to describe the entire build, test,
release approach. The problem is when you call something DevOps anyone who doesn't identify themselves
as a dev or as Ops automatically assumes they're not part of the process.
Seriously, go and ask your designers what they think of DevOps. Or how about your testers. Or
Product Managers. Or Customer Support.
And that's a problem.
We've managed to take something that is completely dependant on collaboration, and trust, and
name it in a way that excludes a significant number of people. All of the name suggestions that arise
when you mention this are just ridiculous. DevTestOps? BusinessDevTestOps? DesignDevOps? Aside from
just being stupid names these continue to exclude anyone who doesn't have these words in their title.
So do I hate DevOps? Well no, not the practice. I think we should always be thinking about how
things will actually work in production. We need an Ops team to help us do that so it makes total
sense to have them involved in the process. Just take care with that name.
Is there a solution? Well, in my mindwe're still talking about collaboration above all else. Thinking
about CD as "Delivery on demand" also makes more sense to me. We, the whole team, should be ready
to deliver working software to the customer when they want it. By being aware of the confusion, and
exclusion that some of these names create we can hopefully bring everyone into the project before
it's too late.
DevOps initiatives include a range of technologies and methodologies spanning the software delivery
process. IT leaders and DevOps practitioners should proactively understand the readiness and capabilities
of technology to identify the most appropriate choices for their specific DevOps initiative.
Table of Contents
What You Need to Know
The Hype Cycle
The Priority Matrix
On the Rise
DevOps Toolchain Orchestration
User and Entity Behavior Analytics
Application Release Automation
At the Peak
Web-Scale Application Architecture
Enterprise-Class Agile Development
Software-Defined Data Center
Continuous Configuration Automation
Sliding Into the Trough
Application Testing Services
Climbing the Slope
Application Performance Monitoring Suites
Test Data Management
Hype Cycle Phases, Benefit Ratings and Maturity Levels
The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (2013) is the third
book by Gene Kim. The business novel tells the story of an IT manager who has ninety days to rescue
an over-budget and late IT initiative, code-named The Phoenix Project. The book was co-authored by
Kevin Behr and George Spafford and published by IT Revolution Press in January 2013.
The novel is thought of as the modern day version of The Goal by Eliyahu M. Goldratt. The novel
describes the problems that almost every IT organization faces, and then shows the practices of how
to solve the problems, improve the lives of those who work in IT and be recognized for helping the
business win. The goal of the book is to show that a truly collaborative approach between IT and
business is possible.
The novel tells the story of Bill, the IT manager at Parts Unlimited. The company's new
IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the
project is massively over budget and very late. The CEO wants Bill to report directly to him and
fix the mess in ninety days or else Bill's entire department will be outsourced. With the help of
a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that
IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking,
Bill must organize work flow, streamline interdepartmental communications, and effectively serve
the other business functions at Parts Unlimited.
The book has been called a "must read" for IT professionals and quickly reached #1 in its Amazon.com
categories. The Phoenix Project was featured on 800 CEO Reads Top 25: What Corporate America
Is Reading for June, 2013. InfoQ stated, "This book will resonate at one point or another with
anyone who's ever worked in IT." Jeremiah Shirk, Integration & Infrastructure Manager at Kansas
State University, said of the book: "Some books you give to friends, for the joy of sharing a great
novel. Some books you recommend to your colleagues and employees, to create common ground. Some books
you share with your boss, to plant the seeds of a big idea. The Phoenix Project is all three."
Other reviewers were more skeptical, including the IT Skeptic "Fictionalising allows you to paint
an idealised picture, and yet make it seem real, plausible... Sorry but it is all too good to be
true... none of the answers are about people or culture or behaviour. They're about tools and techniques
and processes."  Jez Humble (author of Continuous Delivery) said "unlike real life, there aren't
many experiments in the book that end up making things worse..."
In a recent webinar, XebiaLabs VP of DevOps Strategy Andrew Phillips sat down with Atos Global
Thought Leader in DevOps Dick van der Sar to separate the facts from the fiction. Their findings:
most myths come attached with a small piece of fact and vice versa.
1. DevOps Is Developers Doing Operations: Myth
An integral part of DevOps' automation component involves a significant amount of code. This causes
people to believe Developers do most of the heavy lifting in the equation. In reality, what ends
up happening is due to the amount of infrastructure as Code, Ops begin to look a lot like Dev.
2. Projects Are Dead: Myth
Projects are an ongoing process of evolving systems and failures. To think they can just be handed
off to maintenance forever after completion is simply incorrect. This is only true for tightly scoped
software needs, including systems built for specific events. When you adopt DevOps and Agile, you
are replacing traditional project-based approaches with a focus on product lifecycles.
3. DevOps Doesn't Work in Complex Environments: Myth
DevOps is actually made to thrive in complex environments. The only instance in which it doesn't
work is when unrealistic and/or inappropriate goals are set for the enterprise. Complex environments
typically suffer due to lack of communication about the state of, and changes to, the interconnected
systems. DevOps, on the other hand, encourages communication and collaboration that prevent these
issues from arising.
4. It's Hard to Sell DevOps to the Business: Myth
The benefits to DevOps are closely tied benefiting the business. However, that's hard to believe
when you pitch adopting DevOps as a plan to "stop working on features and sink a lot of your money
into playing with shiny new IT tech." Truth is, DevOps is going to impact the entire enterprise.
This may be the source of resistance, but as long as you find the balance between adoption and disruption,
you will experience a successful transition.
5. Agile Is for Lazy Engineers: Myth
DevOps prides itself on eliminating unnecessary overhead. Through automation, your enterprise
can see a reduction in documentation, meetings, and even manual tasks, giving team members more time
to focus on more important priorities. You know your team is running successfully if their productivity
Nonetheless, DevOps does not come without its own form of "boring" processes, including test plans
or code audits. Agile may eliminate waste but that doesn't include the tedious yet necessary aspects.
6. If You Can't Code, You Have No Chance in DevOps: Fact
This is only afact because the automation side of DevOps is all Infrastructure as Code (IaC).
This typically requires some sort of software development skill such as modularization, automated
testing, and Continuous Integration (CI) as IaC. Regardless of scale, automating anything will require,
at the very least, software development skills.
7. Managers Disappear: Myth
Rather than disappear, managers take a different role with DevOps. In fact, they are still a necessity
to the team. Managers are tasked with the responsibility of keeping the entire DevOps team on track.
Classic management tasks may seem to disappear but only because the role is changing to be more focused
8. DevOps or Die: Fact!
Many of today's market leaders already have some sort of advanced DevOps structure in place. As
industries incorporate IT further into their business, we will begin to see DevOps as a basic necessity
to the modern business and those that can't adapt will simply fall behind.
That being said, you shouldn't think of DevOps as the magic invincibility potion that will keep
your enterprise failure free. Rather, DevOps can prevent many types of failure, but there will always
be environment specific threats unique to every organization that DevOps can't rescue you from.
Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure,
brought to you in partnership with
Sauce Labs .
Out of these misunderstandings several common myths
have been created. Acceptance of these myths misleads
Here are some of the most common myths and the facts
that debunk them.
Myth 1: DevOps needs agile.
Although DevOps and agile are terms frequently used
together, they are a long way away from being
synonymous with one another. Agile development refers
to a method of software delivery that builds software
incrementally, whereas DevOps refers not only to a
method of delivery but to a culture, which when
adopted, results in
many business benefits
, including faster software
DevOps processes can help to compliment agile
development, but it is not reliant on agile and can
support a range of operation models such as
Waterfall – where build processes can be optimised and accelerated, and automation can be
Agile – where heightened communication between
development and operations increases end-product
Hybrid approach – where speed, quality and
compliance are all increased.
For optimum results, full adoption of the DevOps
philosophy is necessary.
Myth 2: DevOps can't work with legacy.
DevOps is often regarded as a modern concept that
helps forward-thinking businesses innovate. Although
this is true, it can also help those organisations with
long-established, standard IT practices. In fact, with
legacy applications there are usually big advantages to
Managing legacy care and bringing new software to
market quickly; blending stability and agility, is a
frequently encountered problem in this new era of
digital transformation. Bi-modal IT is an approach
where Mode 1 refers to legacy systems focussed on
stability, and Mode 2 refers to agile IT focussed on
rapid application delivery. DevOps principles are often
included exclusively within Mode 2, but automation and
collaboration can also be used with success within Mode
1 to increase delivery speed whilst ensuring stability.
Myth 3: DevOps is only for continuous delivery.
DevOps doesn't (necessarily) imply continuous
delivery. The aim of a DevOps culture is to increase
the delivery frequency of an organisation, often from
quarterly/monthly to daily releases or more, and
improve their ability to respond to changes in the
While continuous delivery relies heavily on
automation and is aimed at
agile and lean thinking
DevOps it is not reliant on a shared culture which
enhances collaboration. Gartner summed up the
distinction with a report that stated that: "DevOps is
not a market, but a tool-centric philosophy that
supports a continuous delivery value chain."
Myth 4: DevOps requires new tools.
As with the implementation of any new concept or
idea, a common misconception about
is that new toolsets, and skills are
required. Though the provision of appropriate and
relevant tools can aid adoption, organisations are by
no means required to replace tools and processes they
use to produce software.
DevOps enables organisations to deliver new
capabilities more easily, and bring new software into
production more rapidly in order to respond to market
changes. It is not strictly reliant on new tools to get
this job done.
Myth 5: DevOps is a skill.
The rapid growth of the DevOps movement has resulted
in huge demand for professionals who are skilled within
the methodology. However, this fact is often
misconstrued to suggest that DevOps is itself a skill –
this is not the case.
DevOps is a culture – one that needs to be fully
adopted throughout an entire organisation for optimum
results, and one that is best supported with
appropriate and relevant tools.
Myth 6: DevOps is software.
Understanding that DevOps adoption can be better
facilitated with software is important, however, maybe
more so is understanding that they are not one and the
same. Although it is true that there is a significant
amount of DevOps software available on the market
today, purchasing a specific ad-hoc DevOps product, or
even suite of products, will not make your business
The DevOps methodology is the communication,
collaboration and automation of your development and
operations functions, and as described above, is
required to be adopted by an entire organisation to
achieve optimum results. The software and tools
available will undoubtedly reduce the strain of
adoption on your business but conscious adoption is
required for your business to fully reach the potential
that DevOps offers.
Like any new and popular term, people have somewhat
confused and sometimes contradictory or partial
impressions of what DevOps is and how it works.
DevOps is a philosophy which enables businesses to
automate their processes and work more collaboratively
to achieve a common goal and deliver software more
help organisations to successfully implement DevOps,
to learn how we made DevOps a reality at
TSB bank Sabadell
These are just a few things that jumped out at me (and annoyed me)
However, there are teams at Netflix that do traditional Operations, and teams that do DevOps
Ops is ops is ops. No matter what you call it, Operations is operations.
Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime
There's no such thing as a "DevOps tool". People were using CFengine, Puppet and Chef long before
DevOps was even a term. These are configuration management tools. In fact Adrian has even said they
use Puppet in their legacy datacenter:
yet he seems to make the distinction between the ops guys there and the "devops" guys (whatever
There is no ops organization involved in running our cloud…
Just because you outsourced it, doesn't mean it doesn't exist. Oh and it's not your cloud. It's
Reading between the lines
Actually this doesn't take much reading between the lines. It's out there in plain sight:
In reality we had the usual complaints about how long it took to get new capacity, the lack
of consistency across supposedly identical systems, and failures in Oracle, in the SAN and the
networks, that took the site down too often for too long.
We tried bringing in new ops managers, and new engineers, but they were always overwhelmed
by the fire fighting needed to keep the current systems running.
This is largely because the people making decisions are development managers, who have been
burned repeatedly by configuration bugs in systems that were supposed to be identical.
The developers used to spend hours a week in meetings with Ops discussing what they needed,
figuring out capacity forecasts and writing tickets to request changes for the datacenter.
There is no ops organization involved in running our cloud, no need for the developers to interact
with ops people to get things done, and less time spent actually doing ops tasks than developers
would spend explaining what needed to be done to someone else.
I'm glad to see this spelled out in such detail. This is what I've been telling people semi-privately
for a while now. Because Netflix had such a terrible experience with its operations team, they
went to the opposite extreme and disintermediated them.
Imagine you were scared as a kid by a clown. Now imagine you have kids of your own. You hate clowns.
You had a bad experience with clowns. But it's your kid's birthday party so here you are making baloon
animals, telling jokes and doing silly things to entertain the kids.Just because you aren't wearing
makeup doesn't make you any less of a clown. You're doing clown shit. Through the eyes of the kids,
you're a clown. Deal with it. Netflix is still doing operations. What should be telling and frightening
to operations teams everywhere is this:
The Netflix response to poorly run operations that can't service the business is going to
become the norm and not the exception. Evolve or die.
Please note that I don't lay all the blame on the Netflix operations team. I would love to hear
the flipside of this story from someone who was there originally when the streaming initiative started.
It would probably be full of stories we've heard before - no resources, misalignment of incentives
and a whole host of others.
Adrian, thank you for writing the blog post. I hope it serves as a warning to those who come.
Hopefully someday you'll be able to see a clown again and not get scared ;)
Adrian Cockcroft's article about
NoOps at Netflix ignited a controversy that has been smouldering for some months. John Allspaw's
detailed response to Adrian's article
makes a key point: What Adrian described as "NoOps" isn't really. Operations doesn't go away. Responsibilities
can, and do, shift over time, and as they shift, so do job descriptions. But no matter how you slice
it, the same jobs need to be done, and one of those jobs is operations. What Adrian is calling NoOps
at Netflix isn't all that different from Operations at Etsy. But that just begs the question: What
do we mean by "operations" in the 21st century? If NoOps is a movement for replacing operations with
something that looks suspiciously like operations, there's clearly confusion. Now that some of the
passion has died down, it's time to get to a better understanding of what we mean by operations and
how it's changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no distinction
between dev and ops. If you developed, you operated. You mounted the tapes, you flipped the switches
on the front panel, you rebooted when things crashed, and possibly even replaced the burned out vacuum
tubes. And you got to wear a geeky white lab coat. Dev and ops started to separate in the '60s, when
programmer/analysts dumped boxes of punch cards into readers, and "computer operators" behind a glass
wall scurried around mounting tapes in response to IBM JCL. The operators also pulled printouts from
line printers and shoved them in labeled cubbyholes, where you got your output filed under your last
The arrival of minicomputers in the 1970s and PCs in the '80s broke down the wall between mainframe
operators and users, leading to the system and network administrators of the 1980s and '90s. That
was the birth of modern "IT operations" culture. Minicomputer users tended to be computing professionals
with just enough knowledge to be dangerous. (I remember when a new director was given the root password
and told to "create an account for yourself" … and promptly crashed the VAX, which was shared by
about 30 users). PC users required networks; they required support; they required shared resources,
such as file servers and mail servers. And yes, BOFH ("Bastard
Operator from Hell") serves as a reminder of those days. I remember being told that "no one"
else is having the problem you're having - and not getting beyond it until at a company meeting we
found that everyone was having the exact same problem, in slightly different ways. No wonder we want
ops to disappear. No wonder we wanted a wall between the developers and the sysadmins, particularly
since, in theory, the advent of the personal computer and desktop workstation meant that we could
all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important websites.
As companies and computing facilities grew larger, the fire-fighting mentality of many system administrators
didn't scale. When the whole company runs on one 386 box (like O'Reilly in 1990), mumbling obscure
command-line incantations is an appropriate way to fix problems. But that doesn't work when you're
talking hundreds or thousands of nodes at Rackspace or Amazon. From an operations standpoint, the
big story of the web isn't the evolution toward full-fledged applications that run in the browser;
it's the growth from single servers to tens of servers to hundreds, to thousands, to (in the case
of Google or Facebook) millions. When you're running at that scale, fixing problems on the command
line just isn't an option. You can't afford letting machines get out of sync through ad-hoc fixes
and patches. Being told "We need 125 servers online ASAP, and there's no time to automate it" (as
Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn't surprising. One of the
themes of O'Reilly's Velocity Conference is
"Infrastructure as Code." If you're going to do operations reliably, you need to make it reproducible
and programmatic. Hence virtual machines to shield software from configuration issues. Hence
Chef to automate configuration, so you know
every machine has an identical software configuration and is running the right services. Hence
Vagrant to ensure that all your virtual machines
are constructed identically from the start. Hence automated monitoring tools to ensure that your
clusters are running properly. It doesn't matter whether the nodes are in your own data center, in
a hosting facility, or in a public cloud. If you're not writing software to manage them, you're not
Furthermore, as we move further and further away from traditional hardware servers and networks,
and into a world that's virtualized on every level, old-style system administration ceases to work.
Physical machines in a physical machine room won't disappear, but they're no longer the only thing
a system administrator has to worry about. Where's the root disk drive on a virtual instance running
at some colocation facility? Where's a network port on a virtual switch? Sure, system administrators
of the '90s managed these resources with software; no sysadmin worth his salt came without a portfolio
of Perl scripts. The difference is that now the resources themselves may be physical, or they may
just be software; a network port, a disk drive, or a CPU has nothing to do with a physical entity
you can point at or unplug. The only effective way to manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming
code as early as the late '80s; indeed, Perl was designed as a programming language for automating
system administration. It didn't take long for leading-edge sysadmins to realize that handcrafted
configurations and non-reproducible incantations were a bad way to run their shops. It's possible
that this trend means the end of traditional system administrators, whose jobs are reduced to racking
up systems for Amazon or Rackspace. But that's only likely to be the fate of those sysadmins who
refuse to grow and adapt as the computing industry evolves. (And I suspect that sysadmins who refuse
to adapt swell the ranks of the BOFH fraternity, and most of us would be happy to see them leave.)
Good sysadmins have always realized that automation was a significant component of their job and
will adapt as automation becomes even more important. The new sysadmin won't power down a machine,
replace a failing disk drive, reboot, and restore from backup; he'll write software to detect a misbehaving
EC2 instance automatically, destroy the bad instance, spin up a new one, and configure it, all without
interrupting service. With automation at this level, the new "ops guy" won't care if he's responsible
for a dozen systems or 10,000. And the modern BOFH is, more often than not, an old-school sysadmin
who has chosen not to adapt.
James Urquhart nails
it when he describes how modern applications, running in the cloud, still need to be resilient
and fault tolerant, still need monitoring, still need to adapt to huge swings in load, etc. But he
notes that those features, formerly provided by the IT/operations infrastructures, now need to be
part of the application, particularly in "platform as a service" environments. Operations doesn't
go away, it becomes part of the development. And rather than envision some sort of uber developer,
who understands big data, web performance optimization, application middleware, and fault tolerance
in a massively distributed environment, we need operations specialists on the development teams.
The infrastructure doesn't go away - it moves into the code; and the people responsible for the infrastructure,
the system administrators and corporate IT groups, evolve so that they can write the code that maintains
the infrastructure. Rather than being isolated, they need to cooperate and collaborate with the developers
who create the applications. This is the movement informally known as "DevOps."
Amazon's EBS outage last year
demonstrates how the nature of "operations" has changed. There was a marked distinction between companies
that suffered and lost money, and companies that rode through the outage just fine. What was the
difference? The companies that didn't suffer, including Netflix, knew how to design for reliability;
they understood resilience, spreading data across zones, and a whole lot of reliability engineering.
Furthermore, they understood that resilience was a property of the application, and they worked with
the development teams to ensure that the applications could survive when parts of the network went
down. More important than the flames about Amazon's services are the testimonials of how intelligent
and careful design kept applications running while EBS was down.
Netflix's ChaosMonkey is an excellent, if extreme, example of a tool to ensure that a complex
distributed application can survive outages; ChaosMonkey randomly kills instances and services within
the application. The development and operations teams collaborate to ensure that the application
is sufficiently robust to withstand constant random (and self-inflicted!) outages without degrading.
On the other hand, during the EBS outage, nobody who wasn't an Amazon employee touched a single
piece of hardware. At the time, JD Long
tweeted that the best thing about the EBS outage was that his guys weren't running around like crazy
trying to fix things. That's how it should be. It's important, though, to notice how this differs
from operations practices 20, even 10 years ago. It was all over before the outage even occurred:
The sites that dealt with it successfully had written software that was robust, and carefully managed
their data so that it wasn't reliant on a single zone. And similarly, the sites that scrambled to
recover from the outage were those that hadn't built resilience into their applications and hadn't
replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to the
application itself, we're also seeing a redistribution of costs. It's a mistake to think that the
cost of operations goes away. Capital expense for new servers may be replaced by monthly bills from
Amazon, but it's still cost. There may be fewer traditional IT staff, and there will certainly be
a higher ratio of servers to staff, but that's because some IT functions have disappeared into the
development groups. The bonding is fluid, but that's precisely the point. The task - providing a
solid, stable application for customers - is the same. The locations of the servers on which that
application runs, and how they're managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds like
Amazon's, private clouds, traditional colocation, and building their own infrastructure. It's hard
to beat Amazon if you're a startup trying to conserve cash and need to allocate or deallocate hardware
to respond to fluctuations in load. You don't want to own a huge cluster to handle your peak capacity
but leave it idle most of the time. But Amazon isn't inexpensive, and a larger company can probably
get a better deal taking its infrastructure to a colocation facility. A few of the largest companies
will build their own datacenters. Cost versus flexibility is an important trade-off; scaling is inherently
slow when you own physical hardware, and when you build your data centers to handle peak loads, your
facility is underutilized most of the time. Smaller companies will develop hybrid strategies, with
parts of the infrastructure hosted on public clouds like AWS or Rackspace, part running on private
hosting services, and part running in-house. Optimizing how tasks are distributed between these facilities
isn't simple; that is the province of operations groups. Developing applications that can run effectively
in a hybrid environment: that's the responsibility of developers, with healthy cooperation with an
The use of metrics to monitor system performance is another respect in which system administration
has evolved. In the early '80s or early '90s, you knew when a machine crashed because you started
getting phone calls. Early system monitoring tools like HP's OpenView provided limited visibility
into system and network behavior but didn't give much more information than simple heartbeats or
reachability tests. Modern tools like DTrace provide insight into almost every aspect of system behavior;
one of the biggest challenges facing modern operations groups is developing analytic tools and metrics
that can take advantage of the data that's available to predict problems before they become outages.
We now have access to the data we need, we just don't know how to use it. And the more we rely on
distributed systems, the more important monitoring becomes. As with so much else, monitoring needs
to become part of the application itself. Operations is crucial to success, but operations can only
succeed to the extent that it collaborates with developers and participates in the development of
applications that can monitor and heal themselves.
Success isn't based entirely on integrating operations into development. It's naive to think that
even the best development groups, aware of the challenges of high-performance, distributed applications,
can write software that won't fail. On this two-way street, do developers wear the beepers, or IT
staff? As Allspaw points out, it's important not to divorce developers from the consequences of their
work since the fires are frequently set by their code. So, both developers and operations carry the
beepers. Sharing responsibilities has another benefit. Rather than finger-pointing post-mortems that
try to figure out whether an outage was caused by bad code or operational errors, when operations
and development teams work together to solve outages,
can focus less on assigning blame than on making systems more resilient in the future. Although we
used to practice "root cause analysis" after failures, we're recognizing that finding out the single
cause is unhelpful. Almost every outage is the result of a "perfect storm" of normal, everyday mishaps.
Instead of figuring out what went wrong and building procedures to ensure that something bad can
never happen again (a process that almost always introduces inefficiencies and unanticipated vulnerabilities),
modern operations designs systems that are resilient in the face of everyday errors, even when they
occur in unpredictable combinations.
In the past decade, we've seen major changes in software development practice. We've moved from
various versions of the "waterfall" method, with interminable up-front planning, to "minimum viable
product," continuous integration, and continuous deployment. It's important to understand that the
waterfall and methodology of the '80s aren't "bad ideas" or mistakes. They were perfectly adapted
to an age of shrink-wrapped software. When you produce a "gold disk" and manufacture thousands (or
millions) of copies, the penalties for getting something wrong are huge. If there's a bug, you can't
fix it until the next release. In this environment, a software release is a huge event. But in this
age of web and mobile applications, deployment isn't such a big thing. We can release early, and
release often; we've moved from continuous integration to continuous deployment. We've developed
techniques for quick resolution in case a new release has serious problems; we've mastered A/B testing
to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations staff.
Operations groups are adopting, and in many cases, leading in the effort to implement these changes.
They're the specialists in resilience, in monitoring, in deploying changes and rolling them back.
And the many attendees, hallway discussions, talks, and keynotes at O'Reilly's Velocity conference
show us that they are adapting. They're learning about adopting approaches to resilience that are
completely new to software engineering; they're learning about monitoring and diagnosing distributed
systems, doing large-scale automation, and debugging under pressure. At a recent meeting, Jesse Robbins
described scheduling EMT training sessions for operations staff so that they understood how to handle
themselves and communicate with each other in an emergency. It's an interesting and provocative idea,
and one of many things that modern operations staff bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and
bleeding-edge; now, it's expected. But we haven't taken it far enough. We're still learning how to
monitor systems, how to analyze the data generated by modern monitoring tools, and how to build dashboards
that let us see and use the results effectively. I've joked about "using a Hadoop cluster to monitor
the Hadoop cluster," but that may not be far from reality. The amount of information we can capture
is tremendous, and far beyond what humans can analyze without techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient protocols
for the web, like SPDY. Operations
is involved, more than ever, in tuning the performance of operating systems and servers (even ones
that aren't under our physical control); a lot of our "best practices" for TCP tuning were developed
in the days of ISDN and 56 Kbps analog modems, and haven't been adapted to the reality of Gigabit
Ethernet, OC48* fiber, and their descendants. Operations groups are responsible for figuring out
how to use these technologies (and their successors) effectively. We're only beginning to digest
IPv6 and the changes it implies for network infrastructure. And, while I've written a lot about
building resilience into applications, so far we've only taken baby steps. There's a lot there
that we still don't know. Operations groups have been leaders in taking best practices from older
disciplines (control systems theory, manufacturing, medicine) and integrating them into software
And what about NoOps? Ultimately, it's a bad name, but the name doesn't really matter. A group
practicing "NoOps" successfully hasn't banished operations. It's just moved operations elsewhere
and called it something else. Whether a poorly chosen name helps or hinders progress remains to be
seen, but operations won't go away; it will evolve to meet the challenges of delivering effective,
reliable software to customers. Old-style system administrators may indeed be disappearing. But if
so, they are being replaced by more sophisticated operations experts who work closely with development
teams to get continuous deployment right; to build highly distributed systems that are resilient;
and yes, to answer the pagers in the middle of the night when EBS goes down. DevOps.
There has been a sometimes heated discussion on twitter about the term NoOps recently,
and I've been quoted extensively as saying that NoOps is the way developers work at Netflix. However,
there are teams at Netflix that do traditional Operations, and teams that do DevOps as well. To try
and clarify things I need to explain the history and current practices at Netflix in chunks of more
than 140 characters at a time.
When I joined Netflix about five years ago, I managed a development team, building parts of the web
site. We also had an operations team who ran the systems in the single datacenter that we deployed
our code to. The systems were high end IBM P-series virtualized machines with storage on a virtualized
Storage Area Network. The idea was that this was reliable hardware with great operational flexibility
so that developers could assume low failure rates and concentrate on building features. In reality
we had the usual complaints about how long it took to get new capacity, the lack of consistency across
supposedly identical systems, and failures in Oracle, in the SAN and the networks, that took the
site down too often for too long.
At that time we had just launched the streaming service, and it was still an experiment, with little
content and no TV device support. As we grew streaming over the next few years, we saw that we needed
higher availability and more capacity, so we added a second datacenter. This project took far longer
than initial estimates, and it was clear that deploying capacity at the scale and rates we were going
to need as streaming took off was a skill set that we didn't have in-house. We tried bringing in
new ops managers, and new engineers, but they were always overwhelmed by the fire fighting needed
to keep the current systems running.
Netflix is a developer oriented culture, from the top down. I sometimes have to remind people that
our CEO Reed Hastings was the founder and initial developer of Purify, which anyone developing serious
C++ code in the 1990's would have used to find memory leaks and optimize their code. Pure Software
merged with Atria and Rational before being swallowed up by IBM. Reed left IBM and formed Netflix.
Reed hired a team of very strong software engineers who are now the VPs who run developer engineering
for our products. When we were deciding what to do next Reed was directly involved in deciding that
we should move to cloud, and even pushing us to build an aggressively cloud optimized architecture
based on NoSQL. Part of that decision was to outsource the problems of running large scale infrastructure
and building new datacenters to AWS. AWS has far more resources to commit to getting cloud to work
and scale, and to building huge datacenters. We could leverage this rather than try to duplicate
it at a far smaller scale, with greater certainty of success. So the budget and responsibility for
managing AWS and figuring out cloud was given directly to the developer organization, and the ITops
organization was left to run its datacenters. In addition, the goal was to keep datacenter capacity
flat, while growing the business rapidly by leveraging additional capacity on AWS.
Over the next three years, most of the ITops staff have left and been replaced by a smaller team.
Netflix has never had a CIO, but we now have an excellent VP of ITops Mike Kail (@mdkail), who now
runs the datacenters. These still support the DVD shipping functions of Netflix USA, and he also
runs corporate IT, which is increasingly moving to SaaS applications like Workday. Mike runs a fairly
conventional ops team and is usually hiring, so there are sysadmin, database,, storage and network
admin positions. The datacenter footprint hasn't increased since 2009, although there have been technology
updates, and the over-all size is order-of-magnitude a thousand systems.
As the developer organization started to figure out cloud technologies and build a platform to support
running Netflix on AWS, we transferred a few ITops staff into a developer team that formed the core
of our DevOps function. They build the Linux based base AMI (Amazon Machine Image) and after a long
discussion we decided to leverage developer oriented tools such as Perforce for version control,
Ivy for dependencies, Jenkins to automate the build process, Artifactory as the binary repository
and to construct a "bakery" that produces complete AMIs that contain all the code for a service.
Along with AWS Autoscale Groups this ensured that every instance of a service would be totally identical.
Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime. This
is largely because the people making decisions are development managers, who have been burned repeatedly
by configuration bugs in systems that were supposed to be identical.
By 2012 the cloud capacity has grown to be order-of-magnitude 10,000 instances, ten times the capacity
of the datacenter, running in nine AWS Availability zones (effectively separate datacenters) on the
US East and West coast, and in Europe. A handful of DevOps engineers working for Carl Quinn (@cquinn
- well known from the Java Posse podcast) are coding and running the build tools and bakery, and
updating the base AMI from time to time. Several hundred development engineers use these tools to
build code, run it in a test account in AWS, then deploy it to production themselves. They never
have to have a meeting with ITops, or file a ticket asking someone from ITops to make a change to
a production system, or request extra capacity in advance. They use a web based portal to deploy
hundreds of new instances running their new code alongside the old code, put one "canary" instance
into traffic, if it looks good the developer flips all the traffic to the new code. If there are
any problems they flip the traffic back to the previous version (in seconds) and if it's all running
fine, some time later the old instances are automatically removed. This is part of what we call NoOps.
The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring
out capacity forecasts and writing tickets to request changes for the datacenter. Now they spend
seconds doing it themselves in the cloud. Code pushes to the datacenter are rigidly scheduled every
two weeks, with emergency pushes in between to fix bugs. Pushes to the cloud are as frequent as each
team of developers needs them to be, incremental agile updates several times a week is common, and
some teams are working towards several updates a day. Other teams and more mature services update
every few weeks or months. There is no central control, the teams are responsible for figuring out
their own dependencies and managing AWS security groups that restrict who can talk to who.
Automated deployment is part of the normal process of running in the cloud. The other big issue is
what happens if something breaks. Netflix ITops always ran a Network Operations Center (NOC) which
was staffed 24x7 with system administrators. They were familiar with the datacenter systems, but
had no experience with cloud. If there was a problem, they would start and run a conference call,
and get the right people on the call to diagnose and fix the issue. As the Netflix web site and streaming
functionality moved to the cloud it became clear that we needed a cloud operations reliability engineering
(CORE) team, and that it would be part of the development organization. The CORE team was lucky enough
to get Jeremy Edberg (@jedberg - well know from running Reddit) as its initial lead engineer, and
also picked up some of the 24x7 shift sysadmins from the original NOC. The CORE team is still staffing
up, looking for Site Reliability Engineer skill set, and is the second group of DevOps engineers
within Netflix. There is a strong emphasis on building tools too make as much of their processes
go away as possible, for example they have no run-books, they develop code instead,
To get themselves out of the loop, the CORE team has built an alert processing gateway. It collects
alerts from several different systems, does filtering, has quenching and routing controls (that developers
can configure), and automatically routes alerts either to the PagerDuty system (a SaaS application
service that manages on call calendars, escalation and alert life cycles) or to a developer team
email address. Every developer is responsible for running what they wrote, and the team members take
turns to be on call in the PagerDuty rota. Some teams never seem to get calls, and others are more
often on the critical path. During a major production outage con call, the CORE team never make changes
to production applications, they always call a developer to make the change. The alerts mostly refer
to business transaction flows (rather than typical operations oriented Linux level issues) and contain
deep links to dashboards and developer oriented Application Performance Management tools like AppDynamics
which let developers quickly see where the problem is at the Java method level and what to fix,
The transition from datacenter to cloud also invoked a transition from Oracle, initially to SimpleDB
(which AWS runs) and now to Apache Cassandra, which has its own dedicated team. We moved a few Oracle
DBAs over from the ITops team and they have become experts in helping developers figure out how to
translate their previous experience in relational schemas into Cassandra key spaces and column families.
We have a few key development engineers who are working on the Cassandra code itself (an open source
Java distributed systems toolkit), adding features that we need, tuning performance and testing new
versions. We have three key open source projects from this team available on github.com/Netflix.
Astyanax is a client library for Java applications to talk to Cassandra, CassJmeter is a Jmeter plugin
for automated benchmarking and regression testing of Cassandra, and Priam provides automated operation
of Cassandra including creating, growing and shrinking Cassandra clusters, and performing full and
incremental backups and restores. Priam is also written in Java. Finally we have three DevOps engineers
maintaining about 55 Cassandra clusters (including many that span the US and Europe), a total of
600 or so instances. They have developed automation for rolling upgrades to new versions, and sequencing
compaction and repair operations. We are still developing our Cassandra tools and skill sets, and
are looking for a manager to lead this critical technology, as well as additional engineers. Individual
Cassandra clusters are automatically created by Priam, and it's trivial for a developer to create
their own cluster of any size without assistance (NoOps again). We have found that the first attempts
to produce schemas for Cassandra use cases tend to cause problems for engineers who are new to the
technology, but with some familiarity and assistance from the Cloud Database Engineering team, we
are starting to develop better common patterns to work to, and are extending the Astyanax client
to avoid common problems.
In summary, Netflix stil does Ops to run its datacenter DVD business. we have a small number of DevOps
engineers embedded in the development organization who are building and extending automation for
our PaaS, and we have hundreds of developers using NoOps to get their code and datastores deployed
in our PaaS and to get notified directly when something goes wrong. We have built tooling that removes
many of the operations tasks completely from the developer, and which makes the remaining tasks quick
and self service. There is no ops organization involved in running our cloud, no need for the developers
to interact with ops people to get things done, and less time spent actually doing ops tasks than
developers would spend explaining what needed to be done to someone else. I think that's different
to the way most DevOps places run, but its similar to other PaaS enviroments, so it needs it's own
name, NoOps. [Update: the DevOps community argues that although it's different, it's really just
a more advanced end state for DevOps, so lets just call it PaaS for now, and work on a better definition
Thanks John. I agree with some of what you point out. Netflix in effect
over-reacted to a dysfunctional ops organization. I think there are several other organizations
who would recognize our situation, and would also find a need to over-react to make the solution
Your definition of DevOps seems far broader than the descriptions and definitions I can find by
googling or looking on Wikipedia. I don't recognize what we do in those definitions - since they
are so focused on the relationship between a Dev org and an Ops org, so someone should post an
updated definition to Wikipedia or devops.com.
Until then maybe I'll just call it NetflOps or
I have a loaded NoOps question for you :) I am very interesting in understanding
how a decentralized environments he said-she said issues get solved. For example, I know netflix
uses horizontally scalable rest layers as integration points.
Suppose one team/application is having an intermittent problem/bug with another team/application.
Team 1 opens a issue. Team 2 reads investigates closes the issue as not a problem. Team 1 double
checks and reasserts the issue is Team 2.
In a decentralized environment how is this road block cleared?
As an ops person I spend a good deal of time chasing down problems very external to me. I accept
this as an ops person. Since developers are pressed into ops how much time will a developer spend
on another teams reported problems. Will team 1 forgo there own scrum deadlines this week because
team 2 sucks up all their time reporting bogus problems?
We aren't decentralized. So in the scenario you mention everyone gets in a
room and figures it out, or we just end up with a long email thread if it's less serious. APM tools
help pinpoint what is going on at the request level down to Java code. Once we have root cause someone
files a Jira to fix the issue. There is a manager rotation for centrally prioritizing and coordinating
response to major outages. (I'm on duty this week, it comes up every few months.)
We have a few people
who have "mad wireshark skills" to debug network layer problems, but that's infrequent and I'm hopeful
that boundary.com will come up with better tools in this space.
We don't follow a rigid methodology or fixed release deadlines, we ship code frequently enough that
delays aren't usually a big issue, and we have a culture of responsible adults so we communicate
and respect each others needs across teams. The infrequent large coordinating events like a new country
launch are dealt with by picking one manager to own the overall big picture and lead the coordination.
Adrian - Great article! I'm always fascinated to read & hear how folks
have solved problems that plague a lot of IT organizations.
I've got a question about something in your reply above: "we have a culture of responsible adults
so we communicate and respect each others needs across teams".
I've found that, time and time again, the most difficult thing about organizational change is
the people. How does one go about hiring "responsible adults"? I know that it might sound like
a silly or a flippant question, but seriously -- I've lost count of how many times grown up folks
act like childish, selfish, spoiled brats.
My views on culture may not be much help - read http://perfcap.blogspot.com/2011/12/how-netflix-gets-out-of-way-of.html
to see my explanation of how Netflix does things differently.
Culture is very hard to create or modify but easy to destroy. This is because everyone has to
buy into it for it to be effective, and then every manager has to hire only people who are compatible
with the culture, and also get rid of people who turn out not to fit in, even if they are doing
So the short answer is start a new company from scratch with the culture you want, and pay a lot
of attention to who you hire. I don't think it is possible to do a culture shift if there are
more than a roomful of people involved.
Part of getting a "culture of responsible adults" together is partly down
to "culture" - although it helps to have mature sensible individuals, fostering that also means avoiding
finger-pointing and blame.
The more defensive people are made to feel, the more likely they are to start throwing tantrums when
under pressure. A culture where you can put your hand up and say: "I got that wrong, how can we put
it right?" gets better results in the long term than one where you might be fired or disciplined
for a genuine mistake.
I always wondered how evil geniuses like Ernst Blofeld recruit when getting it wrong means you might
end up in the shark tank...
Yes, incident reviews that don't involve blame and finger-pointing are
also key. Making the same mistake several times, trying to hide your mistakes, or clear lapses
of judgement can't be tolerated though.
Great article Adrian. I have a question. Is a consecutive IP space important?
Since AWS EIP doesn't guarantee consecutive addresses, I've wondered if this mattered to app developers.
Anything that could have been done by subnet is out the window. For example if you wanted to do port
sweeps of your network blocks for an audit, perform penetration testing, or parsing logs by IPs.
I suppose this could be done pragmatically but was curious about your experiences. Does it matter?
If the network topology matters you can use VPC to manage it. Also if you
are a big enough customer to have an enterprise level support contract with AWS and use a lot
of EIPs it is possible to get them allocated in contiguous blocks.
I'm curious about your statement "Notice that we didn't use the typical DevOps
tools Puppet or Chef to create builds at runtime. This is largely because the people making decisions
are development managers, who have been burned repeatedly by configuration bugs in systems that were
supposed to be identical."
If systems are built from the same puppet manifests, what kind of configuration bugs can occur? Also,
how is the alternative method you choose any less likely to cause the same problems?
Puppet is overkill for what we end up doing. We have a base AMI that is
always identical, we install an rpm on it or untar a file once, rebake the AMI and we are done.
That AMI is then turned into running systems using the AWS autoscaler. It's more reliable because
there are fewer things to go wrong at boot time, no dependencies on repo's or orchestration tools,
we know that the install had completely succeeded the same way on every instance before the instance
Adrian, Last time we talked you mentioned that you were not 100% transitioned
to AWS, but were - looks like this was achieved, with the exception for the core DVD Business.
As many companies are talking about Hyrbrid cloud as a target state, and Netflix went through
the transition from Pvt/Managed Ops to Public-AWS/NoOps, can you talk about the interim state
you were in and what development and operations use cases you optimized around during transition
– ala, what were the Hybrid models and principles you all followed – to move from Ops to NoOps.
Did you attempt to keep these teams and tools separate or did you all try to create a transition
strategy that allowed you to hedge the bets with AWS and the 'all-in strategy" to possibly
come back if needed etc..
What Ops Governance and Dev Tooling approach did you take in the state? Specifically around
cloud abstraction layers to ease the Access management, support, tooling, elasticity needs.
Can you shed some light on the thinking and approach you took while you were in mid-state of
Also can you comment on the how much do you govern and drive the development and deployment
approach so you can unify the continuous Integration and Continuous deployment tools so that
you can reduce the chaos in this space?
If you look at the presentations on slideshare.net/adrianco I have discussed
in some detail what the transition strategy and tools looked like. We continued to run the
old code in the datacenter as we gradually moved functionality away from it. The old DC practices
were left intact as we moved developers to the cloud one group at a time.
Dev and admin teams struggle these days with keeping up with agile development.
DevOps helps by breaking down some of the walls. But one of the biggest challenges is getting
the entire development team involved and not just 1 or 2 people who help do deployments.
entire team needs visibility to the production server environments to help support and troubleshoot
FAIR USE NOTICEThis site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
You can use PayPal to make a contribution, supporting development
of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with.We do not warrant the correctness
of the information provided or its fitness for any purpose.