...In 1990 Lucy Patrick commented: "Although we live in a democracy, cult behavior manifests itself in our unwillingness to
question the judgment of our leaders, our tendency to devalue outsiders and to avoid dissent. We can overcome cult behavior, he
says, by recognizing that we have dependency needs that are inappropriate for mature people, by increasing anti-authoritarian
education, and by encouraging personal autonomy and the free exchange of ideas."[108]
Groupthink is a psychological phenomenon that occurs
within a group of people in which the desire for harmony
or conformity in the group results in an irrational or dysfunctional
decision-making outcome. Group members try to minimize
conflict and reach a consensus decision without critical
evaluation of alternative viewpoints by actively suppressing dissenting viewpoints, and by isolating themselves from outside
influences.
Groupthink requires individuals to avoid raising
controversial issues or alternative solutions, and there
is loss of individual creativity, uniqueness and independent thinking. The dysfunctional
group dynamics of the "in-group"
produces an "illusion of invulnerability" (an inflated certainty that the right decision has been made). Thus the "in-group" significantly
overrates its own abilities in decision-making and significantly underrates the abilities of its opponents (the "out-group").
Furthermore, groupthink can produce dehumanizing actions against the "out-group".
Antecedent factors such as group cohesiveness,
faulty group structure, and situational context (e.g., community panic) play into the likelihood of whether or not groupthink
will impact the decision-making process.
Fad: “an intense and widely shared enthusiasm for something, especially one that is short-lived.”
Hoopla: "excitement surrounding an event or situation, especially when considered to
be unnecessary fuss."
Companies often spend a lot of money on questionable IT technology. It happens for all sorts of reasons. More often than not they
are chasing the latest shiny new fad. If management bonuses are involved the fad is almost irresistible and in this case very few organizations
are tracking what they’re getting in terms of business outcomes for their investment in the fad. Are those people wasting money?
Yes, but that's company money, and the size of executive bonuses outweighs other considerations. Maximization of executive bonuses is
always win-win strategy for higher management under neoliberalism. For a neoliberalized company outsourcing of IT is the
most reliable way to achieve that. DevOps provides a nice smokescreen to hide the sad truth that the adoption is more about outsourcing
than the technology in question.
And in case of dramatic failure you can always blade external consultant for the fiasco. Everybody knows the benefits of the cloud—lower
personnel costs and the technology still is gradually improving. But many companies end up spending way too much to make the move online—because
they don’t realize what the shift entails and what a hit it means for for the hardware "knowledge sphere". Without people with critical
knowledge of hardware (which can be achieved only by running own datacenter) are often fall pray of unscrupulous consultants.
For such companies cloud means vendor lock-in which they can't escape.
But many companies end up spending way too much to make the move online—because they don’t realize what the shift entails.
You all read about typical advantages typically listed for DevOps methodology: it supposedly helps you deliver products faster, improves
company profitability (of course ;-), ensures continuous integration (without asking the key question: is this a good idea?),
removes roadblocks from your releases (magically, without changing the level of incompetence of the management ;-), and gives you a
competitive advantage (without changing the quality of developers). Those are all signs that this is yet another software development
methodology fad (DevOps – Fad or Here to Stay- - DZone DevOps):
In software development, as trends gain more popularity, they gain new adopters at ever-increasing rates. Often, it is simply
because that’s what everybody seems to be doing and there is a Fear Of Missing Out (FOMO). This leads many organizations
to get swept up in fads, some of which are so short-lived that the fad has passed before any real benefits are recognized. Then on
to the next development craze.
Today, DevOps is the trend that is grabbing headlines and attracting all the attention. While some organizations have resisted
making a DevOps transition – either due to confusion about what DevOps entails, concerns that it may just be a passing fad, or a
simple aversion to change – other organizations have jumped in with both feet
Those claims alert all seasoned IT professionals because they are very similar to claims of previous IT fads that now are completely
or semi-forgotten (DevOps is a Myth):
In the same way we see this happening with DevOps and Agile. Agile was all the buzz since its inception in 2001. Teams were
moving to Scrum, then Kanban, now SAFE and LESS. But Agile didn’t deliver on its promise of better life. Or rather – it became so
commonplace that it lost its edge. Without the hype, we now realize it has its downsides. And we now hope that maybe this new
DevOps thing will make us happy.
Typically fads like DevOps run for a decade or slightly more; after that they are rejected and forgotten. We are probably in the
middle of this path right now as all this DevOps hoopla started around 2015. So a reasonable guess is that nowhere after 2025
DevOps will be open for a critical review. In this sense this pager is premature and as such the acceptance of my augments will be low.
Typically fads like DevOps run for a decade or slightly more; after that they are rejected and forgotten. We are probably
in the middle of this path right now as all this DevOps hoopla started around 2015
If we are to believe DevOps advocates, we should accept that DevOps magically invalidates
Peter Principle, Parkinson
law and several other realistic assessment of corporate environment (and specifically IT environment which was aptly illustrated
by Dilbert cartoons) and creates software paradise of Earth, kind of "software communism" as in "From
each according to his ability, to each according to his needs - Wikipedia" :-)
In reality, it often creates the environment aptly depicted in The Good Soldier
Svejk (the book that NYT put in the category of Greatest Books Never Read -- or hardly read by Americans; it served an
inspiration for "Catch 22" and Kurt Vonnegut's "Slaughterhouse-Five")
For US readers I only want to note that there is some Svejkian qualities/traits in Heller's Yossarian, the ordinary guy trapped in
the twisted logic of army regulations. Yossarian is smart enough to want to escape war by pretending he's crazy -- which makes him sane
enough to be kept in the army.
Hasek's piercing satire describes the confrontation of this ordinary man with the chaos of overcentralized organizations, even if
that chaos is cloaked in military discipline. And to certain extent this situation in a small scale replicates in large datacenters.
A similar tragedy is replayed as a farce in large corporate environments. Including identity craze -- quite similar to the tangled
politics of the Czechs, Austrians and Hungarians in Hapsburg empire ;-) And the problem of incompetence of management still
looms large. Under neolibneral high level managemnt is occupied by bean counters instead of engenners. The results can be sizing in
fate of IBM and HP. top mane a few companies.
The Good Soldier Svejk environment is always to some extent present in
large datacenter, were absurdities loom large and like Peter Pniciple suggersts it is often easer and with less discussions to get approval
on $10M project (move to Azure in one example here) then 100K (for example, upgrading outdated networking equipment). To make the situation
worse often project leaders selected in such a way that they correspond to a justification given to Svejk: "He may be an idiot
but he's our superior and we must trust that our superiors know what they are doing; there's a war on." The case of an idiot manager
(often an MBA or a bean counter), or worse a sociopath in a corner office, or a control freak in management role in the large datacenter
is a topic that is waiting for the new Hasek.
There were at least half-dozen previous attempts to achieve nirvana in the corporate datacenter and, especially, enterprise software
development, starting from "verification revolution" initiated by Edsger
W. Dijkstra, who probably was one of first "cult leader" style figure in the history of programming. It all basically comes
down to this: some IT companies (for some, often short, period of time) somehow achieve better results than others in software development.
They promote their success as the sign that they adopted some new magic methodology. Later such a company often flops, but
methodology Hoopla still linger in the air for some time, like the Cheshire cat smile.
Religious myth and religious thinking plays a trmendous role in IT. It is rarely discussed. Methodologies like DevOps, Agile, etc
don’t reflect reality. They are sets of carefully constructed religious myths. But they are useful as any region to management
as it creates a way to attribute any accidental and temporary success to the management and to avoid sacking of management. It also
helps to soothe customer frustration when we forced to admit that the software delivered is a mess and does not behave as expected by
the customer. It’s a pretty ingenious neoliberal way of pushing the responsibility down from the most powerful to the least powerful
-- every time( "they did not get DevOps and that's why we are having problems"). It is an easier path than to admit that the quality
of management was abysmally low. That technical competence of managers is gone and that software was developed in chaotic and
unprofessional manner.
Due to this capability, DevOps is an important secular religion (techno-cult), which changes the way of delivery software in corporation
(often by completely botching the whole process), but as a religion it greatly helps to ignore real reasons for the failures and
concentrates of myths. Emergence of DevOps clergy greatly helps.
OK, containers are neat (pioneered by FreeBSD in 1990th and later adopted and extended to the modern form by Sun in Solaris 10 (2000)),
putting your stuff in the cloud can, for certain types of workloads (mainly "episodic" workloads with rare high peaks and long
low intensity valleys, such as genomic decoding, social sites, store fronts, etc) be neat (pioneered by Amazon, two decades ago). And
even Ansible, which in a way can be viewed as a reimplementation of IBM JCL on a new level can be very useful for the same reasons
as JCL was and remains useful: many sysadmin tasks are simple sequential tasks (waterfall model) and mechanism when failure of one step
of the task means the failure of all the task is often the only control mechanism needed.
But the key reason of all this hoopla with DevOps is that cutting down infrastructure costs and outsourcing is a honey for higher
management (as it drives up their bonuses). And this pretty mercantile consideration is the real driver of DevOps: it is prudent to
view it as yet another smoke screen for outsourcing (in accounting terms, this is shifting CapEx to OpEx, not much more) hidden
in religious zeal of adopting new technology.
Like is the case with many cults, the details of this "ideology" are fuzzy, exact definition is non-existent, and the only
solid agreement within the DevOps community is that this is "a good thing". Nobody agrees about the details, or on the precise definition.
This should be clear for anyone who read articles about DevOps. First of all the typical level of such articles is quite mediocre,
or worse. They just do not contain any bright ideas. By-and-large they are nothing but marketing materials for certain companies
and unscrupulous academics who want to ride outsourcing bandwagon under the smoke screen of IT.
Still several large and successful companies such as Netflix and Amazon supposedly practice DevOps (and, in case of Netflix, heavily
advertize it). The question is why? In case of Netflix probably because the company is nothing more then a special portal for
viewing movies, and claiming that they practice some new shiny technology probably can help in marketing their product. They have
a very average, I would say mediocre portal, which in now way represents anything like the state of the art. But for them ability to
run on cloud servers is critical as they have peak loads which are concentrated in few hours of the day and particular days. In
case of Amazon they use the cloud for their needs too, so the ability to sell parts of its helps the bottom line. It is nice and profitable
side business for them. And the main Amazon datacenter role is to run the storefront, which by definition has its peak hours, so the
ability to expand computational facilities during those hours is critical and here the cloud fits the bill.
While there are multiple definition of DevOps, a typical definition usually includes the usage (typically in the cloud environment)
of the following methodologies ( DevOps - Wikipedia )
Code – code development and review,
source code management tools, code merging.
NOTE: source management tool originated in early 70th. GIT (which is based on ideas of BITKEEPER) in 2005
Build – continuous
integration tools, build status. NOTE: Used from early 70th. Originated on mainframe as overnight builds.
Test – continuous testing
tools that provide feedback on business risks. NOTE: Fuzzy concept .
Package – artifact
repository, application pre-deployment staging. NOTE: Software packaging became popular in Solaris since 1990.
Red Hat RPM format is available since 1997.
Release – change management, release approvals,
release automation. NOTE: Change management was in place in large mainframe software project since late 60th.
Configure – infrastructure configuration and management,infrastructure as codetools.
NOTE: Good tools definitely has value, but it is difficult to do them right and so far DevOps does
not produce any innovative ideas in this area. Parallel execution tools originated in Unix in late 70th. CFEengine was written in 1993 as a tool used for managing
large numbers of Unix workstations that run heterogeneous operating systems, e.g. Solaris, Linux, AIX, Tru64 and HP-UX.
Monitor – applications
performance monitoring, end-user experience. NOTE: Performance monitoring and optimization was in place since at
least late 60th. Knuth groundbreaking article An empirical
study of FORTRAN programs was published in 1971. Importance of end-user experience was discussed in Fred
Brooks The Mythical Man-Month published in 1975.,
were he point out on importance of fighting the "second system" effect: the tendency of small, elegant, and successful systems
to be succeeded by over-engineered, bloated systems, due to inflated expectations and overconfidence ( feature creep). Mainly
due to the desire to provide the consumer with a more useful or desirable product, in order to increase sales or distribution
without understanding real needs of the consumer. Once the product reaches the point at which it does everything customer needs
additional bell and whistles increases the cost of development, lessen the customer satisfaction and efficiency. The fear of losing
customers due to sting to the old version and a perceived lack of improvement also play some role in feature creep.
Most of those are "no-brainers" and should be used by any decent software development organization, but some are questionable: more
frequent releases is a good idea only for minor releases and bug fixes, never for major releases. They tend to freeze
the current (often deficient) architecture and prevent the necessary architectural changes.
More frequent releases is a good idea only for minor releases and bug fixes, never for major releases. They
tend to freeze the current (often deficient) architecture and prevent the necessary architectural changes.
A very important tip here is that DevOps definition is fuzzy by design as this is mainly marketing campaign for outsourcing
IT infrastructure (and a very successful one), not a technological revolution (although technological progress, especially in
hardware and communications, makes some things possible now that were unthinkable before 2000; progress in servers remote control, web
and smartphones alone were the real game changers, much bigger then all DevOps Hoopla)
That means that you as a sysadmin affected by this marketing campaign have a certain flexibility with what to call DevOps ;-) In
other words you can include something that you like and claim that this is essential for DevOps. It's a bad cook who can't name
the same cutlet with 12 different names ;-) And you can even teach this thing to lemmings as a part of DevOps training, joining the
cult. You can't even imagine how gullible are DevOps enthusiasts and how badly they want additional support
TIP: DevOps definition is quote fuzzy. It is mostly "this is a good thing" type of definitions. So you can include
something that you like and claim that this is essential for DevOps. It's bad cook who can't name the same cutlet with 12
different names ;-) And you can even teach this thing to lemmings as a part of DevOps training.
Out of those seven "sure" things listed above, the continuous delivery is probably most questionable idea. As sticking to it undermines
efforts to revise the architecture of the large software packages. It essentially tend to "freeze" the current architecture "forever".
As such it is a very bad thing, a fallacy.
Moreover "continuous delivery" is nothing new as most software development projects now provide access to beta versions of
the software. Providing beta versions is just another name for continues delivery :-) Also most organization for ages practice
night builds. Which helps to keep developers deviating too much from the common core.
But while the release of beta versions is a useful tool that helps to prepare better for the eventual rollout and provides additional
feedback form beta users, only "beta addicts" use them in production environment. Continues delivery makes all users beta testers.
If the goal of this whole exercise is to engage users in testing and detecting the bugs you need to assume that the userd are loyal
to your company and the product and really want to become the beta testers, because after a while they can defect.
But it has one useful affect: as te product constantly changes users are forced to kee[ up with modifications binding them to this
modern variant of Red Queen Race. Few would eve quetion the absurdity of the whole idea.
Night builds and providing beta versions are just another names for continues delivery :-) You can use this
fact to your advantage
The same reservation are true for automated testing which now is sold under the umbrella of "continuous testing". Compiler developers
used this since 1960th (IBM/360 compiler development efforts), and Perl developers created automated suit for this purpose decades ago.
So again, nothing new under the sun.
The key observation here is that while it is "a very good thing", it is not that easy to implement properly outside several narrow
domains, such as programming languages, web interfaces and such. And again, in compiler development this technology is used since
60th. First tools to use command output for driving further processing were developed in 70th (where this approach was pioneered
by Expect), automatic testing of web interfaces stated in mid 1990th (where several powerful commercial tools were created and marketed),
etc.
Most constructive ideas associated with cloud computing were used in computational clusters since late 1990, or so. They
were reflected in Sun concept of the "grid" which originated with their purchase in 2000
Gridware, Inc. a privately owned commercial vendor of advanced computing resource management software with offices in San Jose,
Calif., and Regensburg, Germany.[29] Later that year, Sun
offered a free version of Gridware for Solaris and Linux, and renamed the product Sun Grid Engine( Sun Grid as cloud service was launched
in March 2006 or 14 years ago)
Those ideas revolve around Sun earlier idea "network is a computer" and including the idea of using central management unit for a
large group of servers or the whole datacenter (like "headnode" in computational cluster), central monitoring, logging and parallel
execution tools to deliver changes to multiple servers simultaneously.
Similar ideas were developed by Tivoli under the name "system management software services" starting from 1989 ( IBM acquired
Tivoli in 1996). Accounting to Wikipedia (Tivoli Software )
The other important (and rather dangerous) aspect of DevOps is the attempt of elimination or, at least, a diminishing role of sysadmins
in the datacenter. Of course, developers always dreamed of getting root privileges. That simplifies many things and cuts a lot of red
tape ;-). But the problem here is with the overcomplexity of modern Linux one person can't be a master of two trades. He
can barely be a master of one. Major enterprise Linux distributions such as Red Hat and Suse are just tar pits of overcomplexity
and can swallow naive developers who think that the can cross them alive ;-)
The idea of a DevOps engineer who is wearing two hats: of sysadmin and of a programmer is a fallacy. Each field
now is complex enough to require specialization. There’s no way to measure if one person is more of a DevOps engineer or not,
because the balance between knowledge of a particular programming toolset and the knowledge of Linux as an operating system is very
tricky and depends on jobs responsibilities.
Most sysadmins worth its name are quite proficient in scripting (several key developers/promoters of Perl were sysadmin; now
many know Python on a pretty decent level). Therefore they can instantly be renamed DevOps engineers. That's allows you to join
the techno-cult, but changes nothing. The problem is that most sysadmins do not have enough time to study a favorite scripting
language in depth and operate with a small subset necessary for to the tool they create or maintian.
The situation with programmers is more even worse. In most cases they only imagine that they know Linux. In reality they know such
a small subset of sysadmin knowledge that many of them are unable to pass even basic sysadmin certification exam such as Red Hat
RHCSA. This fact can be profitably exploited
giving you an opportunity of a meaningful activity (teaching developers RHCSA course) under the flag of OOP ("if you can't beat
them, join") instead of useless and humiliating beating the drum and marching with the DevOps banner -- participating in a senseless
DevOps training sessions. I once taught elements of Bash scripting to a group of researchers under the flag of DevOps training
:-)
Here is one tip on how to deal with ambitious (and usually reckless) developers, who try to obtain root access to the servers under
the flag of DevOps. You should say that yes, you are glad to do this, but only after passing such an elementary for them test
as RHCSA certification, and you are confident
that for such a great specialist it is not a big deal. Usually requests ends at this point ;-) If you strongly do not like
the person you can add Bash test from LinkedIn to the mix, or something similar :-).
TIP: If ambitious (and usually reckless) developers try to obtain root access to the servers under the flag of DevOps.
You should say yes, you will glad to do this, but only after passing
RHCSA certification
In reality what kills the idea once and for all is the complexity of modern operating systems. With the current complexity of RHEL
(as of RHEL 7), the idea that a regular software developer can learn this level of complexity to function as full fledged sysadmin is
completely fallacious unless you can implant to the guy the second head ( spending substantial money of classes and training can help,
but a lot of sysadmin skills are based on raw experience including painful blunders and such. )
This is especially true for handing of the disasters in the data center, of SNAFU in military jargon. This were ou need all
the skills you have, as Red hat support is often useless and nothing more that a human assisted query into there Knowledgebase.
Of course VMs and containers create new oppostunities to go back to the previous version but still this is a the situation when you
typically need to find a root cause. and this is not an easy task and often is connected with a blunder committed by one of the sysadmins
or developers, acting on incomplete or erroneous information, or assumption.
Also idea that sysadmin can raise their programming skill to the level of professional developers in, say, Python does not take into
account that many sysadmin became sysadmin because they do not want to become a developers, and are happy with writing small scripts
in bash and AWK.
Additional layers of complexity in a form of Ansible or Puppet help only in situation when everything works OK. Please note that
you can use Ansible as "parallel ssh" for execution of "ad hoc" scripts so you can adopt it without much damage.
Any tool like Ansible does add a level of indirection. As soon as serious problem arise you need people capable to go down
to the level of kernel and see what is happening. How can a person writing scripts in Python do this is anybody guess. As
Red Hat support was by-and-large destroyed, and now by default, is not much more then a query to Red Hat tech database, you face
serious problem with downtime. As an experiment create an LVM volume consisting of two disk arrays (PV1 and PV2). Then fail two disks
in RAID 5 array of PV2(by simply removing them.) Now open ticket with Red Hat and see how Red Hat will help you to restore the
data on LVM volume (PV1 is still intact).
Switching to the use of VM solves certain problem with maintenance and troubleshooting (availability of the baseline
to which you can always return) while creating other problems due to shared networking and memory in such servers (which make VM mostly
glorified workstation as for computer power, but with current capabilities of workstations this often is more than enough).
So this trend can help (although VMware will appropriate all your savings in the process;-), but here again the problem with the
increased level of complexity might bite you sooner or later. The same is true for Azure, Amazon cloud, etc. Also,
for anything than rare high peak low troth workloads and experimentation they are prohibitively expensive. Even for large corporations,
who can negotiate special deals with those providers.
As a side note I would say that with version 7 RHEL is over the head even for many sysadmins as for troubleshooting serious problems.
IMHO it was a major screw up -- adding systemd (clearly
Apple inspired solution) means that many old skills (and books) can't be reused. New skills need to be acquired by sysadmin and this
is neither cheap, nor quick process. They now try to mask this complexity with Ansible and Web console, but that does not solve
problems, only swipe them under the carpet.
Naive developers who think that system administration is "nothing special" and eagerly venture into this extremely complex minefield
based on this college course in Ubuntu and running a Ubuntu box at home very soon learn the meaning of the terms SNAFU and "horror story".
As in wiping, say a couple of terabytes of valuable corporate data with one click (or with one simple script.) In this case DevOps converts
to Opps...
In case of wiping, say a couple of terabytes of valuable corporate data with one click (or with one simple script.) by
newly minted sysadmins converted from the developers DevOps converts to Opps...
You can just cut the red tape by providing the most ambitious and capable developers (and only them, you need to be selective and
institute kind of internal exam for that) access to root in virtual instances as crashing virtual machine is less serious event then
crashing a physical server -- you always have (or should have) a previous version of VM ready to be re-launched in minutes.
But beyond that point, God forbid. Root access to real medium size or large server (say 2 sockets or 4 sockets server with
32GB-128GB of RAM and 16TB or more of storage) running important corporate application should be reserved to people who have at
least entry level Linux-based admin certification such as
RHCSA, and (which is especially important)
hands-on expertise with backups.
During his career each sysadmin goes though his own set of horror stories, but "missing
backup" (along with creative uses or rm) is probably the
leitmotiv on the most. Creating new horror stories in the particular organization is probably not what higher management
with their quest for bonuses and neoliberalization of IT ( neoliberal "shareholder value" mantra means converting IT staff into contractors
and outsourcing large part of the business to low cost countries like, say, Eastern European countries) meant by announcing their new
plush DevOps initiative.
Each sysadmin goes though his own set of horror stories, but "missing
backup" (along with creative uses or rm) is probably the
leitmotiv on the most. Creating new horror stories in the particular organization is probably not what higher management
with their quest for bonuses and neoliberalization of IT ( neoliberal "shareholder value" mantra means converting IT staff into
contractors and outsourcing large part of the business to low cost countries like, say, Eastern European countries) meant by announcing
their new plush DevOps initiative.
Good understanding of the Linux environment now requires many years of hand on 10 hours a day work experience (exactly how many years,
of course, depends on the person). The minimum for reaching "master" level of a given skill is estimated to be around
10,000 hours, the earlier you start the better. Please note that many sysadmin
came from hobbyists background and start tinkering with hardware in high school or earlier. So, a couple years after graduating from
colleague they often have almost ten year experience. And taking into account Byzantium tendencies of mainstream programming languages
(and those days you need to know several of them, say bash, Python and Javascript or Java ) 30,000 hours is a more reasonable estimate
(one year is approximately 3000 working hours). Which means formula 4+6 (four years of college and 5-6 years on the job self-education
) to get to speed in any single specialty (iether programmer or system administrator). When you need to learn the second craft the process,
of course can go faster (especially for programmers) but 10K hours rules still probably applies (networking staff along probably
need that amount of hours).
The idea to give root access to non-trained developer who never passes, say,
RHCSA Certification is actually much bigger
fallacy that people assume. If you want to be particular nasty in
BOFH fashion, you can give root access to business critical server
to several developers (of course, only under pressure and with written approval from your management, if you are not suicidal).
If you survive fallout from the subsequent SNAFU, for example wiping out 10TB of genomic data with one rm command (and you will
be blamed, so you need to preserve all the emails with approvals), then you not only can remove all administrative access from those
"victim of overconfidence," but get the management officially prohibit this practice. At least until the memory about this accident
fades and another datacenter administration decides to repeat old mistakes.
The key slogan of DevOps movement is "all power to the developers" ;-). But while idea is noble this goal is completely
unrealistic. No amount of automation can replace a role of specialist. Ansible and all other fashionable toys are good when system
is running smoothly. As soon as you hit major software or hardware problem you need to operate on the lower level of abstraction
and get into nitty-gritty details configuration, they are not only useless but harmful.
Try for a test add echo statement like echo "Hello DevOps" to .bashrc on your account and then ask somebody
who is DevOps zealot to help to troubleshoot problem with scp (scp stops working, but ssh to such a box still works).
But, at least, Ansible is useful for automating routine tasks.
Although even in this area it is overrated and in reality does not provide much gains over much simpler and more reliable
Pdsh. All those examples with creation of accounts a
In any case, as a smoke screen to protect yourself from fake changes (which can destroy any functioning datacenter) desired
by DevOps zealots, Ansible deployment makes certain sense. Formally it is clearly a part of DevOps toolkit. Initially Ancible can be
used strictly as a clone of Pdsh until the need for more complex functionality arise.
Even in the cloud, as soon as you try to do something more or less complex you need a specialist completely devoted to learning
this part of infrastructure. One problem that I noticed that most developers have a very weak (often close to zero) understanding of
networking. That's their Achilles spot and that's why they often suggest and sometimes even implement a crazy WAN based solution (aka
cloud solutions).
Most developers have a very weak (often close to zero) understanding of networking. That's their Achilles spot and that's
why they often suggest and sometimes even implement a crazy WAN based solution (aka cloud solutions) replacing tried and true
internal applications
According to Wikipedia The Fallacies of Distributed Computing are a set of common but flawed assumptions made by programmers
in development of distributed applications. They were originated by
Peter Deutsch (who was at the time at Sun Microsystems
), and his "eight classic fallacies" -- describing false assumptions that programmers new to distributed applications typically made.
(2) No matter how hard you push and no matter what the priority, you can't increase the speed of light.
(2a) (corollary). No matter how hard you try, you can't make a baby in much less than 9 months. Trying to speed this up *might*
make it slower, but it won't make it happen any quicker. Callon Informational [Page 1]
(3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where
they are going to land, and it could be dangerous sitting under them as they fly overhead.
(4) Some things in life can never be fully appreciated nor understood unless experienced firsthand. Some things in networking
can never be fully understood by someone who neither builds commercial networking equipment nor runs an operational network.
(5) It is always possible to aglutenate multiple separate problems into a single complex interdependent solution. In most cases
this is a bad idea.
(6) It is easier to move a problem around (for example, by moving the problem to a different part of the overall network architecture)
than it is to solve it.
(6a) (corollary). It is always possible to add another level of indirection.
(7) It is always something
(7a) (corollary). Good, Fast, Cheap: Pick any two (you can't have all three).
(8) It is more complicated than you think.
(9) For all resources, whatever it is, you need more.
(9a) (corollary) Every networking problem always takes longer to solve than it seems like it should.
(10) One size never fits all.
(11) Every old idea will be proposed again with a different name and a different presentation, regardless of whether it works.
(11a) (corollary). See rule 6a.
(12) In protocol design, perfection has been reached not when there is nothing left to add, but when there is nothing left to
take away.
Those WAN blunders costs corporation serious money with zero of negative return on the investment. It is easy to suggest to
transfer say 400TB over Atlantic for a person who do not understand what is the size of the pipeline in datacenter No.1 and datacenter
No.2 in the particular corporation. Or implement a monolithic 5 PB "universal storage system" which become single point of failure despite
all IBM assurances that GPFS is indestructible and extremely reliable; in this case a serious GPFS bug can produce a failure
after which you can kiss goodbye to several terabytes of corporate data. If you are lucky, most of them are useless or duplicated somewhere
else, but still...
The term "Black swans" is the name of a very rare outlier events that has a severe impact. In production systems, these are
problems with software or hardware that you do not suspect exits until it is way too late. When they strike they can't be fixed quickly
and easily by a rollback or some other standard response from your vendor tech-support playbook. They are the events you tell new sysadmins
years after the fact.
Additional and more complex automation increases probability of black swans not diminishes it -- "Complex systems are intrinsically
hazardous and brittle systems." Once you have multiple automation systems, their potential interactions become extremely complex
and impossible to predict. For example, your automation system can restart servers that were shut down for maintenance in the middle
of firmware update.
Additional and more complex automation increases probability of black swans not diminished it -- "Complex systems are intrinsically
hazardous and brittle systems." For example, your automation system can restart servers that were shut down for maintenance
in the middle of firmware update.
It is easy for a developer to buy a two socket server with Dell professional support configuration included and than, in four years
down the road, discover that RIAD5 need monitoring and failure of two disks in RAID5 configuration is fatal. As well as the fact that
Dell professional services did not include spare in RAID5 configuration because the developer if his naivety demanded as much
disk space as possible (high end Dell controllers are capable of supporting RAID 5 with a spare which is a more reliable configuration
then plain vanilla RAID5 as even failure of two disk does not lead to the destruction of the disk array and potential loss of data).
And if RAID5 lost two disks you are probably on your way to enrich data recovery companies such as OnTrack ;-)
Those "horror stories" can be continued indefinitely, but the point is that the level of complexity of modern IT infrastructure
is such that you do need to have different specialists responsible for different parts of IT. And while separation of IT roles into
developer and sysadmin has its flaws and sysadmins already benefit from leaning more programming and developer more about underlying
operating system, you can't jump over mental limitation of mere mortals. Still it is true that some, especially talented, sysadmins
are pretty capable to became programmers as they already know shell and/or Perl on the level that makes learning of a new scripting
language a matter of several months. And talented developers can learn OS on the level close to a typical sysadmin in quite
a short time. But they are exceptions, not a rule.
In any case, the good old days when a single person like Donald Knuth in late 50th early 60th was on a night shift at the university
datacenter a programmer, system operator and OS engineer simultaneously are gone. That's why certifications like
Certified AWS specialist and similar for other clouds like Azure started to
proliferate and holders of those certification are in demand.
Attempt of DevOps reverse the history and return us to the early days of computing when programmer was simultaneously a system administrator
(and sometimes even hardware engineer ;-) are doomed to be a failure. The costs of moving everything to the cloud are often exorbitant,
especially when the issues with WAN are not understood beforehand and this fiasco is covered with PowerPoint presentations,
destined to hide the fact that after top management got their bonuses the enterprise IT entered the state of semi-paralysis. Yes, cloud
has its uses but it is far for being panacea.
The other aspect here is the question of loyalty. By outsourcing datacenter as DevOps recommends, you essentially rely of kindness
of strangers in case of each more or less significant disaster. Strangers that have zero loyalty to your particular firm/company
no matter how SLA are written. In other words, the drama of outsourcing helpdesk is now replayed on a higher level, and
with more dangerous consequences.
Software engineering proved to be very susceptible to various fads
which resemble pseudo-religious movements (with high levels of religious fervor).
Another warning sign that DevOps adepts are not completely honest is lofty goals. Nobody in his sound mind would object
to achieving stated DevOps goals
The goals of DevOps span the entire delivery pipeline. They include:
Faster mean time to recovery (in the event of a new release crashing or otherwise disabling the current system).
The only question is: Can DevOps really achieves them or is mostly a hype? Listen to
this song to find out ;-)
Historically software engineering proved to be very susceptible to various fads which usually take the form of pseudo-religious
movements (with high levels of religious fervor). Prophets emerge and disappear with alarming regularity (say, each ten years or so
-- in other words the period long enough to forget about the previous fiasco). We already mentioned the "verification revolution", but
we can dig even deeper and mention also "structured programming revolution" with its pseudo-religious crusade against goto statements
(while misguided it, at least, has a positive effect on introducing additional high level control structures into programming languages;
see important historically article by Donald KnuthStructured programming with goto
statementscs.sjsu.edu).
Verification hoopla actually destroyed careers of several talented computer scientists such as
David Gries (of
Compiler Construction for Digital Computers fame; who also participated in creation of amazing, brilliant teaching
PL-C compiler designed by Richard W. Conway and Thomas R. Wilcox ). It also damaged
long term viability of achievements of Niklaus Wirth ( a very talented
language developer, who participated in creation of Algol 60 and later developed Pascal and Modula). He also is known for his
Wirth's law: "Software is getting slower more rapidly
than hardware becomes faster."
Dismal duct database with some added blog features (reviews.) But if Amazon at least tries to implement several most
relevant types of searches, Netflix does not. Fr rare movies it is rather challenging to find what you interested in, unless you know
the exact title ;-). Movies are not consistently indexed by director and major stars. The way they deal with reviews is
actually sophomoric. But the colors are nice, no question about it :-)
Amazon is much better, but in certain areas it still has "very average" or even below average quality. One particularly bad oversight
is that the reviews are indexed only in several basic dimensions (number of stars, popularity (up votes) and chronological order), not
by "reviewer reputation" (aka karma), number of previous reviews (first time reviewers are often fake reviewers), date of the fist review
by this review written, and other more advanced criteria. For example, I can't exclude reviewers who wrote less then 2 reviews
and who first wrote review less a year ago.
You also can't combine criteria in the search request. That create difficulties with detecting fake reviewers, that is
a real problem on Amazon review system. As a result it requires additional, often substantial, work to filter our "fake" reviews (bought,
or "friends reviews" produced by people who are not actual customers). They might even dominate Amazon rating for some products/books.
In any case that "effect" diminishes the value of Amazon rating and makes Amazon review system "second rate". Recently
Amazon developers tried to compensate this by "verified purchase" criteria, but without much success, as, for example, in most cases
the book can be returned and unopened product can be returned too. While some fake reviews are detectable by the total number
of reviews posted (often this number is one ;-). or their style, in many cases it is not possible easily distinguish "promotional campaign"
of the author of the book (or vendor of the product) from actual reviews. Shrewd companies can subsidize purchases in exchange for positive
reviews. In this sense Amazon interface sucks and sucks badly.
Amazon cloud has its uses and was generally a useful innovation, so all the DevOps hoopla about it is not that bad, but it
is rather expensive and is suitable mostly for loads with huge short term peaks and deep, long valleys. For example genomic decoding.
The cost of running the infrastructure of a medium firm (200-300 servers) on Amazon cloud is comparable with the cost of running private
datacenter with hardware support outsources and local servers providing local services (the idea of autonomous remote servers) by "local"
staff which definitely has higher loyalty to the corporation and which can be more specialized. In this case all servers also
can be managed from the central location which creates synergies similar to cloud and the cost of personnel is partially offset by lower
cost of WAN connections due to provision on several local services (email, file storage, local knowledgebase, website, etc) by remote
autonomous datacenters (using IBM terminology)/
WAN access is Achilles spot of the cloud infrastructure and costs a lot for medium and large firms with multiple locations. Remote
management tools now are so powerful that a few qualified sysadmin now can run pretty large distributed datacenter with comparable (or
higher) reliability to Amazon or Microsoft clouds.
Especially if servers are more or less uniform and can be organized in the grid. Of course you need to create a spreadsheet
comparing the costs of those two variants, but generally a server with two 24 core CPUs, 128 GB and several terabytes of local storage
can run a lot of services locally without any (or with very limited) access to WAN and it cost as much as one-two years of renting similar
computational capabilities on Amazon cloud or Azure, while having 5 year manufacturer warranty.
Although details are all that matter in such cases and are individual to each company, cloud is not the only solutions for the problems
that plague modern datacenter. And in many cases it is a wrong solution as outsourcing deprives the company the remnants of IT
talent and it became hostage of predatory outsourcing firms. You just need to wait a certain amount of time to present it as "new" and
revolutionary solution for problem caused by the DevOps and cloud ;-) Again DevOps is often a smoke screen for outsourcing and management
priorities change with time after facing the harsh reality of new incarnation of "glass datacenters." People forgot how strongly
centralized IBM mainframe environment (aka "glass datacenters") were hated in the past. Often executives that got the bonuses
for this "revolutionary transformation" is gone in a couple of years ;-). And the new management became more open to alternative
solutions as soon as they experience reality of cloud environment and the level of red tape, nepotism and cronyism rampant in
Indian outsourcing companies ("everything is a contract negotiation"). Some problems just can't be swiped under the carpet.
DevOS increase previous tendency to rate performance of system administrators in the same way they rate the performance of helpdesk
personnel via customer surveys. That creates additional and unnecessary pressure of system administrators and the desire to please the
user at all costs.
Which, in view that certain percent of users are ignorant jerks (losers in sysadmin terminology) creates problem. Now can't say directly
in user face that his request is absurd and dictate by his ignorance, even diplomatically, because that will put down your ratings.
Not that this will hurt you too much, but just understanding that this perverted feedback loop is in place creates problems. You
need to find other areas to press and possibly punish such users for their obnoxious behaviour. One way is to press violation of the
terms of serve. If you control the terms of service for particular set of servers (HPC cluster, WEb server farm, etc) you can always
put items that can be used against lusers. One trivial way is some kind of annual certification for security or knolwdge or DevOps or
operating system knowledge test, something like that. In other words welcome to the "Bastard operator from hell" on new level.
As large enterprise brass now is hell-bent on deploying DevOps (for obvious, mostly greedy reasons connected with outsourcing and
bonuses, see below ;-) it would be stupid to protest against this techno-cult.
In most cases your honestly will not be appreciated, to say the least. In other words direct opposition will be a typical "career limiting
move" even if you 100% right (and it is unclear whether you are: the proof of the pudding is in the eating)
So you need to adapt somehow, and try to make lemonade out of lemons. You can concentrate on what is positive on DevOps
agenda for now and see how the situation looks like when the dust settles. There are two promising technologies that you can adopt under
DevOps umbrella (we already mentioned Ansible):
Jira and Confluence. Both represent an improvement in their respective areas in comparison with typical products
used by the company.
Docker. While VM provide the same functionality Docker containers are more lightweight and Docker is a well engineered
application. It solves the problem of running various flavor of Linux and/or installing various complex applications in the most
efficient and non-intrusive way. In other words it is one of the rare tools that makes sense, despite the fact that it is promoted
by DevOps.
Git and GitLab. While GIT is overhyped and is not suitable as Unix configuration files management tool (although
attempts to manage /etc were made, see below). It is still good enough to use for managing your scripts.
Ansible. This is a more questionable choice but on the level zero. Essentially a reinvention of IBM JCL on a new level.
And as is the case with JCL it is adequate for performing/controlling many sysadmin tasks, which consists of a linear sequence of
steps with failure of one step leading to the failure of the whole task. It also can be used strictly for execution of ad hoc
scripts, much like a better version of Pdsh. So at least it is not harmful despite
introducing yet another domain specific language into the mix. If also is written in Python which is not a bad thing.
And if you want to learn this scripting language you can unscrupulously exploit this fact telling your managers that learning necessary
Python is a step to using Ansible. It uses "yet another domain specific language" YAML but it is easy to learn. And even it
is not necessary for using Ansible exclusively as "ad hoc execution of commands" tool (replacement for pdsh:-).
If you have NFS system available on the nodes that it can be even more useful. Ansible is very useful as a smoke screen to demonstrate
to higher ups that you are on the path of DevOps adoption. One advantage of Ansible is that it does not have clients you need
to install and maintain on all servers. Moreover it can be used as the replacement (an overkill) of pdsh, allowing you to reuse existing
skills and scripts (the same is true if you use parallel similar tools)
Creation of hybrid cloud. You can mix AWS (or Azure) with distributed domestic infrastructure. There are some tasks
that can benefit from running in the cloud and that can be run in the cloud cost-effectively as cloud is expensive (especially disk
storage). But short term task that do not excessive amount of storage and can be run using spot prices might make sense.
Internet facing servers in the cloud are perfect for escaping the stronghold of local proxy. This along probably will make
sysadmins a cloud enthusiast :-) Opportunity to create Docker containers on externally facing cloud severs and then import
then into internal (proxied environment) in invaluable.
Also some task naturally can benefit from the cloud capabilities. For example, this might well be true for bioinformatics
tasks in medium sized companies (large companies typically own a datacenter or several of them so calculations of benefits became
excessively difficult as networking, air conditioning and electricity is supplied a part of large infrastructure.) At the same time
any researcher in medium or large size company can have 64GB 4.5 GHz CPU workstation or, if you are working for a financial company,
a personal 4 sled server with similar parameters for each sled and say, 40 TB of storage. Even for bioinformatics with its
rare peak loads, which theoretically is ideally suited for the cloud, the problem of moving large amount of data to and from
the cloud is very annoying and difficult to solve satisfactory. Which means that local facilities can be competitive (What the cloud
allows management to do, is to shift capital budget into expenses and claim that they saved money)
And there are many tasks that are not suitable for the cloud iether due to high I/O or due to high "fast storage" (this those
days means PCIe NVMe class 40 or 50 SSD) requirements.
In any case, most of us probably need to "beat the drum and march with the banner" as your higher level management typically
adopted some stupid slogan like "Cloud first" (sounds a lot like "Clowns first"). If the game is played carefully and skillfully
you can improve life of peripheral sites dramatically as under this banner as you can improve bandwidth connecting them to the Internet
claiming that it is need to utilize the cloud computing. You can also call centrally managed local servers of peripheral sites "hybrid
cloud" (especially if they are organized as a grid with the central node) and as cloud concept is very fuzzy there is nothing wrong
with this slight exaggeration :-)
The first thing you can do in order to ride DevOps hoopla is to negotiated some valuable training. And I am not talking about this
junk courses in DevOp (although off-site, say in NYC, utilized as extra vacation they also have some value, but this opportunity closed
with COVID-19 epidemic ;-). I am talking, for example, getting at least a couple of courses in Docker and Ansible. Again, if you standardize
on Ansible, you can legitimately ask for Python classes and with some efforts get them.
That actually is in line DevOps philosophy as according to it sysadmins need to grow into developers and developers "downsize" into
sysadmins merging in one happy and highly productive family ;-) So you can navigate this landscape from the point of view of getting
additional training and try to squeezed some water from the stone: corporations now are notoriously tight as for training expenses.
Try to get some training out of DevOps hoopla. If, for example, you standardize on Ansible or Puppet you can legitimately
ask for Python classes. That actually is in line DevOps philosophy as according to it sysadmins need to grow into developers and
developers into sysadmins merging in one happy and highly productive family ;-)
Summarizing, one way to play this game is to equate DevOps with usage of some useful for you software (Docker) and system management
tool (say Ansible) or other interesting for you technology. Which you can safely claim belongs to DevOps. It is a fake movement so some
exaggeration, or deviation usually will not be noticed by the adepts.
For some reason two products of Austrian firm Atlassian ( Jira and Confluence ) are strongly
associated with DevOp. Unlike the majority of DevOps associated products they are definitely above average software (Jira more
so and Confluence less so; but they can be integrated which adds value) which typically is heads above what company is using. Replacing
the current Helpdesk system with Jira and documentation system with Confluence might improve those two important areas of your environment
and they are worth trying.
Another similar to Confluence product -- Microsoft Teams also makes sense to deploy as a part of DevOps hoopla. It is still
pretty raw but the level of integration of a web forum, file repository and wiki is interesting. It also integrates well
with Outlook which is the dominant email client in large corporations.
Docker has it value in both sysadmin area and in applications area. So you can claim that this is a prerequisite for DevOps
and site some books on the subject. Of course, this is not a panacea from many of the existing enterprise datacenter ills, but
the technology is interesting, implementation is elegant and the approach has merits, especially if you are in a proxy protected
environment, which means that the installation of non standard applications a huge pain.
In case of research applications a developer often can make them run quicker in Docker environment, especially if application
is available on Ubuntu or Debian in packaged form, but not on CentOS or RHEL. There are some useful collections of prepackaged
application, mainly for bioscience. See for example BioContainers (the list of available containers
can be found here .)
Esoteric applications that are difficult to install what which require specific libraries with version different that, say,
used by your RHEL version can really benefit from Docker, as it allow your to compartmentalize "library hell" and use the most
suitable for the particular application flavor of Linux (which is of course the flavor in which the application was developed). For
example, many scientific application are native to Debian, which is not an enterprise Linux distribution. Docker allow you to
run instances of applications installed from Debian packages on a Red Hat server ("run anywhere" meme). And if Docker is not enough
you can always fall back to regular VM such as XEN.
In any case Docker is not a fad. It is an interesting and innovative "light-weight VM" technology which was first commercially introduced
by Sun in Solaris 10 in 2005 (as Solaris zones) and replicated in Linux approximately
10 years later. The key idea that this is essentially packaging technology -- all "VM" share the kernel with the base OS. It is
a variant of paravitualization, which produces minimal overhead in comparison with running application as a task on a real server. Much
more efficient then VMware. Linux containers like Solaris zones are essentially extension of the concept of jail ( the idea of extending
jail concept to more "full" VM environment originated in FreeBSD.) It is definitely worth learning, and in some cases to be widely
deployed. Anyway, this is something real, not the typical shaman-style rituals that "classic" DevOps try to propagate (
Lysenkoism first played out as a real tragedy, now it degenerated into a farce ;-)
As most DevOps propagandists and zealots are technologically extremely stupid, it is not that difficult to deceive them. Sell
the Docker as the key part of DevOps 2.0 toolkit, the quintessence of the cloud technology (they like the word "cloud" ) which lies
in the core of DevOps 2.0 -- a hidden esoteric truth about DevOps that only real gurus know about :-) They will eat that...
On higher, datacenter level you can try to push the adoption of Red Hat OpenShift, which is a kind of in-house cloud
and is cheaper and more manageable then Amazon elastic cloud, or Azure. And in some cases makes sense to deploy. That
might allow you to extract from management payment for
RHEL Learning Subscription. Try to equate
the term "hybrid cloud" with use of OpenShift within enterprise. You can also point out, that unless you have short peak and long no
load periods both Azure and AWS are pretty expensive, and it is not wise to put all eggs into one basket.
GIT is not that impressive for Linux configuration management, but sometimes can be used. See, for example, etckeeper package;
if you find it useful and want to use it just claim that this is DevOps 2.0 too, and most probably you will be blessed to deploy
it. And while it has drawbacks, it allows to record actions of multiple sysadmin on the server that result in changes of files in
/etc directory, treating it as a software project with multiple components. So this is far from perfect but still a usable tool
for solving the problem of multiple cooks in the same kitchen.
The main value here is GitLab in that in many large enterprises it is now installed internally as a part of DevOps wave. If
so, you need to learn how to use it as it is useful tool for exchange of information between distributed teams of sysadmins.
In some, rare cases you might even wish to play "continues integration" game. If corporate brass demands
continuous integration to be
implemented you might try to adapt software like GitLab for this purpose and use GitLab
"pipelines", which provides some interesting opportunities for several scripting languages such as Python and Ruby. Automated
testing part is probably the most useful. While you can always write independent scripts for automated testing, integration of them
into GitLab framework often is a better deal.
But we should be aware that the key problem with "continuous integration" and closely related concept of "continuous testing"
is that for testing scripts the most difficult part is the validation of the output of the test: the road to hell is always paid with
good intention.
And while the idea of automated testing is good, for many scripts and software packages the implementation is complex
and man power consuming. Actually this is one area were outsourcing might help. So far there was not breakthrough in this area and much
depends on your own or developers abilities. Regular expression can help to check output, but they are not a panacea.
Also the most complex is your testing script the more fragile it is and has less chances to survive changes in the codebase. So complex
testing scripts tend to freeze the current architecture. And as such as harmful.
In its essence, continuous delivery is an overhyped variation of the idea of the night build, used for ages. If you want to play
a practical joke on your developers tell them to use Jenkins as an automated tests tool. Jenkins is a very complex tool (with
multiple security vulnerabilities), that you generally should avoid. But it can be installed via Docker container. AWS has a "ready
made" project How to set up a Jenkins
build server which can save you from troubles and wasting your time;-). But often you do not need to install it; just
associate the term Jenkins with "continuous integration" and provide for them a Docker container with Jenkins. Your advantage
is that developers usually do not have deep understanding of Linux, and, especially, virtualization issues. And usually they do not
want to learn too much new staff which gets them too far away from their favorite programming language.
So there is a strong, almost 100% chance, that after the period of initial excitement they will ignore this technology and stick
to GIT and their old custom testing scripts ;-)
Ansible is definitely less toxic solution then Puppet. And it offers possibility to ride the DevOps hoopla without doing any harm.
I do not see any innovative ideas in it (this is just a reimplementation of JCL on a new level), but it has some useful tidbits, is
a fashionable, and is heavily pushed by Red Hat (which now, BTW now is an IBM company; and any seasoned sysadmin knows what that means).
Unlike IBM JCL it does not invoke scripts and programs directly. It does this via special wrappers, which is understandable as it
needs to transport the scripts and to execute them on several different systems. Wrappers are called modules and some of them provide
capabilities beyond running scripts and copying files to remote servers (which is the scope of tools like pdsh). For example,
modifying crontab.
Also there are some side benefits here: centralization of reporting, creation of you own repository of playbooks (you need to find
good prototypes, so that you do not reinvent the bicycle ) with some possibilities of sharing playbooks for common operations (installation
of LAMP stack and its components, especially MySQL is one example)
As any JCL-style system, Ansible is perfect for implementing post-installation steps on freshly installed servers. But its capabilities
heavily depends on the quality of utilities which actually perform each steps. Wrappers are good but they are not "replacement for displacement."
You usually can adapt you old utilities to this new role with minimum investment of time and efforts. So it can get a boost
to your own sysadmin utilities development activities.
There are multiple posts on the Web about how to convert simple installation "step by step" shell scripts into Ansible playbooks.
If done without excessive zeal (for example, how often you install Apache server to justify tinkering with Ansible) some such
conversion even might be useful. I would recommend to ignore attempt to exaggerate the value of Idempotency and push
it down your throat; in most case simply sanity checks are more then enough. Browsing articles written about conversion of bash
scripts to playbook might provide a good initial step for learning Ansible. Start by using Ansible as glue to coordinate the scripts
and to do the ssh part. See potentially useful details in the following articles:
GitHub contains several collection some of playbooks some elements of which you probably can adapt and reuse. Try to avoid
complex modules at least the first year of working with Ansible. Limit yourself to very basics -- wrappers for executing commands and
scripts; plus probably yum module (or equivalent for Suse and Debian.) Browsing Web for tutorials also allow you to find some
reusable playbooks that serve as examples in corresponding tutorials.
Here are result of a quick search of GitHub. I can attest the quality and this is not much but something for the start.
What is important for us is that Ansible constitute an important part of
Infrastructure as Code(IaC) hoopla. So you might be able to
get some Python training on the wave of DevOps enthusiasm in your particular organization.
It is also can be used in a way Pdsh or parallel are used (Ansible
can be viewed as Pdsh on steroids), so all your previous scripts and skills
are still fully applicable. But please not that in comparison with pdsh Ansible adds a lot of complexity (unless you use it
as pdsh). It introduces its own JCL language (aka "system automation" language), some modules duplicates existing Unix tools
functionality, and like most tools promoted by DevOps it is overhyped. But is small dozes and without excessive enthusiasm it is
not that bad. It is a useful tool and investment in learning entry level of playbooks pays quickly off allowing to groups some
of your script into more manageable and flexible "ensembles". Again, especially convenient to use Ansible for series of steps needed
after kickstart installation (post installation). It is competitive with hand written drivers for the same. See
Unix Configuration Management Tools for more "in-depth" discussion.
Selling Ansible deployment to management it might be worth mentioning that it is well integrates with GIT (another magic word that
you need to know and use).
As I mentioned above, adopting Ansible (or pretending to adopt it ;-) might allow your to get some valuable Python training.
NOTE about Puppet:
Out of sense of self-preservation it makes sense to stay out of Puttet wave. While I am strongly prejudiced against Puppet (and
consider writing Puppet automation scripts as a new type of masturbation), Puppet also was/is the most loud in DevOps
hoopla ( you can judge what type of company you are dealing by the checking if they advertise Agile
on their pages; Puppet does do that and thus can be considered to be snake oil sellers product ;-). They also pretend to provide
"continues delivery" so formally you can classify Puppet as such a tool, the pretence that might perfectly suit your needs:
No one wants to spend the day waiting for someone to manually spin up a VM or twiddle their thumbs while infrastructure updates
happen. Freedom comes when you get nimble, get agile, and tackle infrastructure management in minutes, not hours. Puppet Enterprise
uniquely combines the industry-standard model-driven approach we pioneered with the flexibility of task-based automation. Welcome
back, lunch break.
... ... ...
Continuous delivery isn’t just for developers. Build, test, and promote Puppet code, high-five your teammates, and get
back to doing what you love with Continuous Delivery for Puppet Enterprise. Instead of wondering if you have the latest code
running, now you can streamline workflows between Dev and Ops, deliver faster changes confidently and do it all with the visibility
your teams need.
BTW the way many years ago one really talented system administrator created Perl for very similar (actually more broad) purposes.
Including the attempt to make life of sysadmins easier. And his solution is still preferable today.
While I mentioned two useful tools that suit DevOps banner, there might be more. Summarizing, I think that "creative"
ad hoc interpretation of DevOps might improve your bargaining position, and can serve as a draft of the plan of your fight against
the most stupid aspects of the DevOps onslaught.
It is important to try to dictate the terms of the game using your superiority in understanding of Linux and control of the
data center servers. Depending on your circumstances and the level of religious zeal for DevOps in your organization, that might
be a viable strategy against unreasonable demands of developers. In any case do not surrender without a fight and please understand
that a direct frontal attack on this fad is a bad strategy.
Now if some brainwashed with DevOps developer tries to enforce his newly acquired via DevOps hoopla rights to manage the servers
("we are one team"), you can instantly put him in his place pointing our that those weaklings do not know DevOps 2.0 technology and
as such does not belong to this "Brave New World." Send all such enthusiasts to the Docker exile, pointing out that control of physical
servers is so yesterday. It is pretty effective political line that gives you much better chances for survival then the
direct attack on this fad.
Still, you might also need to visit a couple of brainwashing session (aka "DevOps Learning Bundle") to demonstrate your loyalty.
While DevOps is notoriously fuzzy concept, I think there are three concepts very relevant to the discussion of DevOps that can be
legitimately criticized:
You probably heard a little bit about so called high demand cults. They exist in techno-sphere too, and while features
are somewhat watered down they are still recognizable (High
Demand Cults):
Remember ... A group does not have to be religious to be cultic in behavior. High demand groups can be commercial, political
and psychological. Be aware, especially if you are a bright, intelligent and idealistic person. The most likely person to be caught
up in this type of behavioral system is the one who says “I won’t get caught. It will never happen to me. I am too intelligent for
that sort of thing.”
The following statements, compiled by Dr. Michael Langone, editor
of Cultic Studies Journal,
often characterize manipulative groups. Comparing these statements to the group with which you or a family member is involved may
help you determine if this involvement is cause for concern.
Love bombing: Fake display of excessively
zealous, unquestioning commitment and adulation to new members. Expensive gift, etc. This is a typical practice of sociopaths
and many cult leaders are sociopaths.
Isolation: The group leader instills a polarized, "we-they" mentality and try to isolate members from their previous
contacts. Members are encouraged or required to work with and/or socialize only with group members. Severing of Ties with
Past, Family, Friends, Goals, and Interests - Especially if they are negative towards or impede the goals of the group.
High demand on members time, intensity of contacts: Intense involvement in cult activities work along with isolation
from others. Behaviour is closely prescribed and carefully supervised. Members are expected to devote inordinate amounts of time
to the group and group activities.
Manipulation: The group's leadership induces guilt feelings in members in order to control them. The pity
play is high in their list of manipulation techniques. It's okay to pity someone who has gone through difficult times, but
if you find yourself feeling sorry for someone's sad story, make sure the story is true. The pity play should serves as a valuable
warning sign that you are dealing with a manipulator, in this case a female sociopaths.
Brainwashing: Questioning, doubt, and dissent are discouraged or even punished with a bouts of anger or (fake) withdrawal.
Special techniques are used to suppress doubts about illegal or questionable or amoral practices in the group or its leader(s)
and put dissidents in line. Special literature is distributed and indoctrination campaign is launched. Much like pedophiles
"groom" children by pushing or encouraging them to watch porno movies and books with explicit content. If group leader requires
special favors from members of the group (for example sex with female members) this all is masked under some artificial pretext
like "exercise in liberation" or "exercise in compassion".
Dictate and micromanagement: The group's leader practice micromanagement and dictates – sometimes in
great detail – how members should think, act, what to wear and feel.
Instilling amorality, end justifies the means mentality: Any action or behaviour is justifiable as long as it furthers
the group's goals. The group (leader) becomes absolute truth and is above all man-made laws.
My initial impression that former cultists come face with a multiplicity of losses, accompanied by a deep, and sometimes debilitating,
sense of anguish. See for example interviews with defector from Mormonism on YouTube
... ... ....
My hope upon initiating this research was to provide a link between cult leaders and corporate psychopath and demonstrate that
cult leaders practices (that are more or less well understood and for which extensive literature exists) have a strong predictive
power for the behavior of a corporate psychopath. We should not focus just on the acute and long-term distress accompanying
reporting to corporate psychopath.
Here are some psychological mechanisms used:
Control of the Environment and Communication The control of human communication is the most basic feature of
the high demand cult environment. This is the control of what the individual sees, hears, reads, writes, experiences
and expresses. It goes even further than that, and controls the individuals communication with himself - his own thoughts.
The Mystique of the Organization. This seeks to provoke specific patterns of behaviour and emotion in such a way that
these will appear to have arisen spontaneously from within the environment. For the manipulated person this assumes a near-mystical
quality. This is not just a power trip by the manipulators. They have a sense of “higher purpose” and see themselves
as being the “keepers of the truth.” By becoming the instruments of their own mystique, they create a mystical aura around
the manipulating institution - the Party, the Government, the Organization, etc. They are the chosen agents to carry out
this mystical imperative.
Everything is black & white. Pure and impure is defined by the ideology of the organization. Only those
ideas, feelings and actions consistent with the ideology and policy are good. The individual conscience is not reliable. The philosophical
assumption is that absolute purity is attainable and that anything done in the name of this purity is moral. By defining and manipulating
the criteria of purity and conducting an all-out war on impurity (dissension especially) the organization creates a narrow world
of guilt and shame. This is perpetuated by an ethos of continuous reform, the demand that one strive permanently and painfully
for something which not only does not exist but is alien to the human condition.
Absolute “Truth” . Their “truth” is the absolute truth. It is sacred - beyond questioning. There is a reverence demanded
for the leadership. They have ALL the answers. Only to them is given the revelation of “truth”.
Thought terminating clichés. Everything is compressed into brief, highly reductive, definitive-sounding phrases, easily
memorized and easily expressed. There are “good” terms which represents the groups ideology and “evil” terms to represent everything
outside which is to be rejected. Totalist language is intensely divisive, all-encompassing jargon, unmercifully judging. To those
outside the group this language is tedious - the language of non-thought. This effectively isolates members from outside world.
The only people who understand you are other members. Other members can tell if you are really one of them by how you talk.
Hubbard called Dianetics "a milestone for man comparable to his discovery of fire and superior to his invention of the wheel and
the arch". It was an immediate commercial success and sparked what
Martin Gardner calls "a nationwide cult of incredible
proportions".[136]
By August 1950, Hubbard's book had sold 55,000 copies, was selling at the rate of 4,000 a week and was being translated into French,
German and Japanese. Five hundred Dianetic auditing groups had been set up across the United States.[137]
... ... ...
The manuscript later became part of Scientology mythology.[75]
An early 1950s Scientology publication offered signed "gold-bound and locked" copies for the sum of $1,500 apiece (equivalent to
$15,282 in 2017). It warned that "four of the first fifteen
people who read it went insane" and that it would be "[r]eleased only on sworn statement not to permit other readers to read it.
Contains data not to be released during Mr. Hubbard's stay on earth."[81]
... ... ...
In October 1984 Judge Paul G. Breckenridge ruled in Armstrong's favor, saying:
The evidence portrays a man who has been virtually a
pathological liar
when it comes to his history, background and achievements. The writings and documents in evidence additionally reflect his egoism,
greed, avarice, lust for power, and vindictiveness and aggressiveness against persons perceived by him to be disloyal or hostile.
At the same time it appears that he is charismatic and highly capable of motivating, organizing, controlling, manipulating and
inspiring his adherents. He has been referred to during the trial as a "genius," a "revered person," a man who was "viewed by
his followers in awe." Obviously, he is and has been a very complex person and that complexity is further reflected in his alter
ego, the Church of Scientology.[327]
The key indicator is greedy, control oriented leadership, ministers who enrich themselves at the expense of followers (one minister
of Prosperity Theology asked followers to donate to him money for his new private jet). Attempts to extract money from the followers
requiring payment for some kind of training, "deep truth", sacred manuscripts reading.
The person who raises uncomfortable questions or does not "get with the program" is ostracized. Questioning of the dogma is discouraged.
In reality nothing is new under the sun in software development. DevOps rehashed the ideas many of which are at least a decade old,
with some that are at least 30 years old (Unix configuration management, version control). And the level of discussion is often
lower than the level on which those ideas are discussed in The Mythical Man-Month,
which was published in 1975 (another sign of "junk science").
What is more important that all under the surface of all those lofty goals is the burning desire of companies brass to use DevOps
as another smoke screen for outsourcing. Yet another justification for "firing a lot of people."
Under the surface of all those lofty goals is the burning desire of companies brass to use DevOps as another smoke screen
for outsourcing. Yet another justification for "firing a lot of people."
As in any cult, there are some grains of rationality in DevOps, along with poignant critique of the status quo. Which works
well to attract the followers. Overhyping some ideas about how to cope with the current, unsatisfactory for potential followers situation
in enterprise IT is another sign of a technocult. Some of those ideas are, at least on superficial level, pretty attractive. Otherwise
such a technocult can't attract followers. As a rule, techno-cults generally emerge in time of huge dissatisfaction and strive
by proposing "salvation" from the current difficulties:
...think this is one of the main reason why we see this DevOps movement, we are many that see this happen in many organizations,
the malfunctioning organization and failing culture that can’t get operation and development to work towards common goals. In
those organizations some day development give up and take care of operation by them self and let the operation guys take care of
the old stuff.
Nobody can deny that there a lot of problems in corporate datacenters those days. bureaucratization is rampant and that stifle few
talented people who still did not escape this environment, or switched to writing open source software during working hours, because
achieving anything within the constrain of the existing organization is simply impossible ;-)
But along the this small set of rational (and old) ideas there a set of completely false, even bizarre ideas and claims, which make
it a cult.. There is also fair share of Pollyanna creep. Again, it is important to understand that part of the
promotion campaign success (with money for it mostly from companies who benefit from outsourcing) is connected with the fact that corporate
IT brass realized than DevOps can serve well as a smote screen for another round of outsourcing of "ops". (Ulf Månsson about infrastructure )
This creates new way of working, one good example is the cultural change at Nokia Entertainment UK, presented at the DevOps conference
in Göteborg, by inclusion going from 6 releases/year and 50 persons working with releases to 246 releases/year with only 4 persons,
see http://www.slideshare.net/pswartout/devopsorg-how-we-are-including-almost-everyone.
That story was impressive.
Pretention that this is a new technology is artfully created by inventing a new language ripe with new obscure terms. Another variant
of Newspeak. And this is very important to understand. It is this language that allow to package bizarre and unproven ideas into the
cloak of respectability.
Primitivism of thinking and unfounded claims of this new IT fashion (typical half-life of IT fad is less then a decade;
for example, who now remembers all this verification hoopla and books published on the topic) are clearly visible in advocacy
papers such as Comparing DevOps to
traditional IT Eight key differences - DevOps.com. Some of the claims are clearly suspect, and smell of "management consultant speak"
(an interesting variety of corporate bullshit). For example:
Traditional IT is fundamentally a risk averse organization. A CIO’s first priority is to do no harm to the business.
It is the reason why IT invests in so much red tape, processes, approvals etc. All focused on preventing failure. And
yet despite all these investments, IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements
are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues.
A DevOps organization is risk averse too but they also understand that failure is inevitable. So instead of trying to eliminate
failure they prefer to choose when and how they fail. They prefer to fail small, fail early, and recover fast. And they
have built their structure and process around it. Again the building blocks we have referred to in this article – from test
driven development, daily integration, done mean deployable, small batch sizes, cell structure, automation etc. all reinforce this
mindset.
Note a very disingenuous claim "IT has a terrible track record – 30% of new projects are delivered late, 50% of all new enhancements
are rolled back due to quality issues, and 40% of the delay is caused by infrastructure issues." I think all previous
fashionable methodologies used the same claim, because the mere complexity of large software projects inevitably leads to overruns and
doubling estimated time and money is a good software engineering methodology known since publishing of Mythical man month :-)
As for another disingenuous claim "So instead of trying to eliminate failure they prefer to choose when and how they fail.
They prefer to fail small, fail early, and recover fast." this is not a realistic statement. It is a fake. In case of a SNAFU
you can't predict the size of the failure. Just look at history of Netflix, Amazon or Google Gmail failures. Tell this mantra
"They prefer to fail small, fail early, and recover fast" to customers of Google
or Amazon during the next outage
Note also such criticism is carefully swiped under the carpet and definition of DevOps evolves with time to preserve its attractiveness
to new members ( DevOps is Dead! Long Live DevOps! - DevOps.com
). In other words the correct definition of DevOps is "it is a very good thing" ;-). For example:
Some are seekers on the quest for the one, true DevOps. They were misled. I’m here to say: Give it up. Whatever you find at the
end of that journey isn’t it. That one true DevOps is dead. Dead and buried. The search is pointless or, kind of worse: The search
misses the point altogether.
DevOps began as a sort of a living philosophy: about inclusion rather than exclusion, raises up rather than chastises, increases
resilience rather than assigns blame, makes smarter and more awesome rather than defines process and builds bunkers. At any rate,
it was also deliberately never strictly defined.
In 2010, it seemed largely about Application
Release Automation (ARA), monitoring, configuration management and a lot of beginning discussion about culture and teams. By
2015, a lot of CI/CD, containers and APIs had been added. The dates are rough but my point is: That’s all still there but now DevOps
discussions today also include service design and microservices. Oh, and the new shiny going by the term “serverless.” It
is about all of IT as it is naturally adapting.
I will be repeating myself here. Like any repackaging effort ,DevOps presents old, exiting technologies as something revolutionary.
They are not. Among measures that DevOps "movement" advocates several old ideas make sense if implemented without fanaticism.
And they could be implemented before the term DevOps (initially NoOps) was invented. That includes:
The move of applications to VM, including but not limited to light weight virtual machines (Docker). And generally
idea of wider adoption of light weight virtual machines in enterprise environment, instead or in addition to traditional (and expensive)
VMware (I do not understand advantages of running Linux under VMware -- this is VM tuned to Windows that implements heavy virtualization
(virtualization of the CPU), while Linux being open source requires para-virtualization (virtualization of system calls), actually
available in XEN and derivatives such as Oracle virtual manager).
Wider adoption of scriptable configuration tools and configuration management system ( Infrastructure as code -- IaC ).
Truth be told this not necessary to do via Puppet and similar complex Unix configuration management
packages) as well as continued delivery schemes. Although Puppet is a bad choice as it is too complex, has unanticipated
quirks and rather low reliability it is a step forward. But it might be two steps back as it contributes to creating "Alice
in wonderland" environment when nobody can troubleshoot the problem because of complex subsystems involved. See
Unix configuration management for details.
Wider usage of version management for configuration files. Such systems as GIT and Subversion is definitely underutilized
in enterprise environment and it can be implemented both wider and better. None of them fits Unix configuration management
requirements perfectly. But some intermediate tools can be created that compensate deficiencies
Wider use of scripting and scripting languages such as Python, Ruby, shell and (to a much lesser extent) good old Perl.
Of them only Python spans both Unix system administration and software development areas and as such is preferable (despite
Perl being a stronghold of Unix sysadmins). That actually includes de-emphasizing Java development in favor of Python.
Attempts to cut development cycle to smaller more manageable chunks (although idea of "continuous delivery" is mostly
bunk). Here a lot of discretion is needed as overdoing this destroys the development process and, what is even more dangerous,
de-emphasizes the value of architectural integrity and architecture as a concept (this is a heritage of Agile, which is mostly a
snake oil).
Some of those innovations ( like Docker) definitely makes more sense than others because preoccupation with VMware as the only solution
for virtualization in enterprise environment is unhealthy from multiple points of view.
But all of them are adding, not removing complexity. Just the source of it different (in case of Docker you also can somewhat
compensate increase of the complexity due to the switch to virtual environment by the fact that you can compartmentalize most critical
applications in the separate "zones (using Solaris terminology -- Docker is essentially reimplementation of Solaris zones in Linux).
May be the truth lies somewhere in between and selected and "in moderation" implementation of some of those technology in the datacenter
can be really beneficial. But excessive zeal can hurt not help. In this sense presence of newly converted fanatics is a recipe for additional
problems. if not a disaster.
I actually saw the situations, when implementation of DevOps brought corporate IT to a screeching halt, almost complete paralysis
when nothing but maintaining (with great difficulties) of the status quo was achieved for the first two years. Outsourcing in those
cases played major negative role as new staff needs large amount of time and effort to understand the infrastructure and processes and
attempt to "standardize" some servers and services. Often failing dismally due to complex interrelations between them and
applications in place.
While the current situation in the typical large datacenter is definitely unsatisfactory from many points of view, the first
principle should be "do no harm". In many cases it might sense to announce the switch to DevOps, force people to prepare to it (especially
discard old servers and unused applications in favor of virtual instances on private or public cloud) and then cancel the implementation.
You probably can probably achieve 80% of positive effect this way, avoiding 80% of negative effects. Moving datacenter to a new
location can also tremendously help and can be used instead of DevOps implementation :-)
While the current situation in a typical large datacenter is definitely unsatisfactory from many points of view, the
first principle should be "do no harm".
Again, the main problem with DevOps hoopla is that, if implemented in full scale, it substantially increases the complexity of environment.
Both virtual machines and configuration management tools provide additional "levels of indirection" which makes troubleshooting more
complex and causes of failures more varied and complex. That represents problems even for seasoned Unix sysadmins to say nothing
for poor developers who are thrown into this water and asked to swim.
Sometimes DevOps methodology implemented in modest scope does provide some of the claimed benefits (and some level of configuration
management in a large datacenter is a must). The question is what if the optimal solution for this and how not to overdo it. But in
a typical DevOps implementation this question somehow does not arise and it just degenerates into another round of centralization
and outsourcing which make the situation worse. Often much worse to the level when It really became completely dysfunctional. I saw
this affect of DevOps implementation in large corporations.
So it should be evaluated on case by case basic, not as panacea. As always much depends of the talent of the people who try to implement
it. Also change in a large datacenter is exceedingly difficult and often degenerates into what can be called one step forward, two steps
back". For example, learning such tools as Puppet or Chief requires quite a lot of effort for rather questionable return on investment
as complexity precludes full utilization of the tool and it is downsized to basic staff. So automation using them is a mixed blessing.
Similarly move to lightweight VM (and virtual servers in general) which is a part of DevOps hype are easier to deploy, but load management
of multiple servers running on the same box is an additional and pretty complex task. Also VMware that dominates VM scene is expensive
(which mean that the lion share of saving are going to VMware, not to the enterprise which deploys it ;-) and is a bad VM for Linux.
Linux needs para-virtualization not full CPU virtualization that VMware, which was designed for Windows, offers (with some interesting
optimization tweaks). Docker, which is a rehash of the idea of Solaris zones is a better deal, but is a pretty new for Linux technology
and it has its own limitations. Often severe as this is a light weight VM.
Junk science is and always was based on cherry-picked evidence which has carefully been selected or edited to support a pre-selected
"truth". Facts that do not fit the agenda are suppressed (Groupthink). Apocalyptic
yelling are also very typical. Same for Pollyanna creep.
Deployment is typically top-down. Corporate management is used as an enforcement branch (corporate
Lysenkoism). Here are some signs of "junk science" (Seven
Eight Warning Signs of Junk Science):
Here is a non-exclusive list of seven eight symptoms to watch out for:
Science by press release. It’s never, ever a good sign when ‘scientists’ announce dramatic results before publishing
in a peer-reviewed journal. When this happens, we generally find out later that they were either self-deluded or functioning as
political animals rather than scientists. This generalizes a bit; one should also be suspicious of, for example, science first
broadcast by congressional testimony or talk-show circuit.
Rhetoric that mixes science with the tropes of eschatological panic. When the argument for theory X slides from “theory
X is supported by evidence” to “a terrible catastrophe looms over us if theory X is true, therefore we cannot risk disbelieving
it”, you can be pretty sure that X is junk science. Consciously or unconsciously, advocates who say these sorts of things are
trying to panic the herd into stampeding rather than focusing on the quality of the evidence for theory X.
Rhetoric that mixes science with the tropes of moral panic. When the argument for theory X slides from “theory X
is supported by evidence” to “only bad/sinful/uncaring people disbelieve theory X”, you can be even more sure that theory X is
junk science. Consciously or unconsciously, advocates who say these sorts of things are trying to induce a state of preference
falsification in which people are peer-pressured to publicly affirm a belief in theory X in spite of private doubts.
Consignment of failed predictions to the memory hole. It’s a sign of sound science when advocates for theory X publicly
acknowledge failed predictions and explain why they think they can now make better ones. Conversely, it’s a sign of junk science
when they try to bury failed predictions and deny they ever made them.
Over-reliance on computer models replete with bugger factors that aren’t causally justified.. No, this is not unique
to climatology; you see it a lot in epidemiology and economics, just to name two fields that start with ‘e’. The key point here
is that simply fitting historical data is not causal justification; there are lots of ways to dishonestly make that happen, or
honestly fool yourself about it. If you don’t have a generative account of why your formulas and coupling constants look
the way they do (a generative account which itself makes falsifiable predictions), you’re not doing science – you’re doing numerology.
If a ‘scientific’ theory seems tailor-made for the needs of politicians or advocacy organizations, it probably has been.
Real scientific results have a cross-grained tendency not to fit transient political categories. Accordingly, if you think theory
X stinks of political construction, you’re probably right. This is one of the simplest but most difficult lessons in junk-science
spotting! The most difficult case is recognizing that this is happening even when you agree with the cause.
Past purveyers of junk science do not change their spots. One of the earliest indicators in many outbreaks of junk
science is enthusiastic endorsements by people and advocacy organizations associated with past outbreaks. This one is
particularly useful in spotting environmental junk science, because unreliable environmental-advocacy organizations tend to have
long public pedigrees including frequent episodes of apocalyptic yelling. It is pardonable to be taken in by this the first time,
but foolish by the fourth and fifth.
Refusal to make primary data sets available for inspection. When people doing sound science are challenged to produce
the observational and experimental data their theories are supposed to be based on, they do it. (There are a couple of principled
exceptions here; particle physicists can’t save the unreduced data from particle collisions, there are too many terabytes per
second of it.) It is a strong sign of junk science when a ‘scientist’ claims to have retained raw data sets but refuses to release
them to critics.
If we are talking about DevOps as a software development methodology it is similar to Agile. The
latter was a rather successful attempt to reshuffle a set of old ideas for fun and profit (some worthwhile, some not so much) into attractive
marketable technocult and milk the resulting "movement" with books, conferences, consulting, etc.
In 2017 only few seasoned software developers believe that Agile is more then self-promotion campaign of group of unscrupulous and
ambitious people, who appointed themselves as the high priests of this cult. Half-life of such "cargo cult" programming methodologies
is usually around a decade, rarely two (Agile became fashionable around 1996-2000). Now it looks like Agile is well past the "hype
stage" in the software methodology life cycle and attempts to resurrect it with DevOps will fail.
As DevOps carries certain political dimensions (connection to the neoliberal transformation of the society, with its fake "cult of
creativity", rise of role (and income) of the top 1% and decline of IT "middle class"). Outsourcing and additional layoffs are
probably the most prominent result of introduction of DevOps into real datacenters. So DevOp often serves as a Trojan horse for the
switch to outsourcers and contract labor.
As DevOps carries certain political dimensions (connection to the neoliberal transformation of the society, with its fake
"cult of creativity", rise of top 1% and decline of middle class), outsourcing and additional layoffs are probably the most prominent
result of introduction of DevOps into real datacenters.
The whole idea that by adding some tools VM-run virtual instances and some additional management capabilities introduced by
tools like Puppet you can by successfully computerized and transferred the IT to outsourcers and contractors is questionable.
No amount of ideological brainwashing you can return datacenter to the good old days in Unix minicomputers when a single person
was a masters of all trades -- a developer, a system administrator and a tester. This is impossible due that current complexity of environment:
there is a large gap in the level of knowledge of (excessively complex) OS by a typical sysadmin (say with RHCE certification) and a
typical developer. Attempts to narrow this gap via tools (and outsourcers) which is the essence of DevOps movement can go
only that far.
But the most alarming tendency is that DevOps serve as a smoke screen for further outsourcing and moving from a traditional data
center to the cloud deployment model with contractors as the major element of the work force. In other words DevOps are
used by corporate brass as another way to cut costs (and the costs of IT in most large manufacturing corporations are already about
1% or less; so there is not much return achievable for this cost cutting anyway).
From 1992 to 2012 Data Centers already experienced a huge technological reorganization, which might be called Intel revolution. Which
dramatically increased role of Intel computers in the datacenter, introduced new server form factors such as blades and new storage
technologies such as SAN and NAS. And make Linux the most poplar OS in the datacenter, displacing Solaris, AIX and HP-UX.
In addition virtualization became common in Windows world due to proliferation of VMware instances.
Faster internet and wireless technologies allowed more distributed workforce and ability for people work part of the week from home.
Smartphones now exceed the power of 1996 desktop. Moreover there was already a distinct trend of the consolidation of datacenters
within the large companies.
As the result in multinationals (and all large companies) companies many services, such as email and to lesser extent file storage
are already provided via internal company cloud from central servers. At the same time it became clear that along with technical challenges
"cloud services" create a bottleneck on WAN level and present huge threat to security and privacy. The driving force behind
the cloud is the desire to synchronize and access data from several devices that people now own (desktop, laptop, smartphone, tablets.
In other words to provide the access to user data from multiple devices (for example email can be read on smartphone and laptop/desktop).
The first such application, ability to view corporate e-mail from the cell phone, essentially launched Blackberry smartphones into prominence.
In view of those changes managing datacenter remotely became a distinct possibility. That's why DevOps serves for higher management
as a kind of "IT outsourcing manifesto". But with outsourcing the problem o loyalty comes into forefront.
Another relevant concept as for discussion of DevOps is Cargo Cult science. Cargo cult science comprises practices that have
the semblance of being scientific, but do not in fact follow the scientific method. The term was first used by physicist Richard Feynman
during his 1974 commencement address at the California Institute of Technology. Software development provides a fertile ground for cargo
cult science. For example, talented software developers are often superstars who can follow methods not suitable for "mere mortals"
like continuous delivery. The came can happen with organization, which have some unique circumstances that make continues
delivery successful. For example if "development" consists of just small patches and bug fixes, while most of the codebase remains static.
I think Bill Joy (of BSD Unix, csh, NFS, Java, vi editor fame ) was such a software superstar when he produced BSD tapes. He
actually created vi editor using terminal over 1200 baud modem (Bill
Joy's greatest gift to man – the vi editor • The Register) -- which is excruciatingly slow. The feat which is difficult if
not impossible for "mere mortal", if just for the lack of patience (transmission speed of 1200 baud modem is close to the speed with
mechanical typewriters can print). The same is true for Donald Knuth who created singlehandedly
a Fortran complier during a summer, when he was still a student. And Ken
Thompson--who is the father of Unix. And Larry Wall who created Perl while being
almost blind on one eye. But that does not mean that there practices are scalable. Brooks book "Mythical man-month" is still
relevant today as it was at the moment of publication.
The biggest mistake you can make as a manager or a large and important software project is to delegate the key functions
in the design to mediocre people. You can't replace software talent with the organization, although organization helps. any attempt
to claim otherwise are cargo cult science. In other word tin software engineering there is no replacement for displacement.
Like Agile before DevOps emphasizes a culture of common goals (this time between "operations" and "developers" with the idea
of merging them like in good old times -- the initial name was NoOps; which is pretty questionable idea taking into account the
complexity of the current IT infrastructure) and get things done together, presenting itself as new IT culture.
Also details are fuzzy and contradictory. They vary from "prophet" to another. That strongly suggests that like many similar IT "fashions"
before it DevOps just mean "a good idea" (remember an attributed to Mahatma Gandhi acerbic remark to a self-confident Western journalist:
on the question “What do you think of Western civilization?,” he reportedly have answered, “I think it would be a good idea” :-)
While it seems the IT management is rushing to embrace the concept of DevOps (because it justifies further outsourcing under the
smokescreen of new terms), nobody agrees on what it actually means. And that creates some skepticism.
DevOps paints a picture of two cultures ("operations" vs. "developers" ) once at odds ("glass datacenter"), now miraculously working
together in harmony. But first of all there are weak and strong developers. There are weak and strong Unix sysadmins )and strong sysadmin
are often are not bad software developers in their own set of languages, mostly scripting languages.)
The problem of huge, excessive complexity of modern IT infrastructure can't be changed with some fashionable chair reshuffling. What
actually happen is that more talented members of the team get additional workload. That's why some critics claim that DevOps kills developers.
Meaning "talented developers". And it certainly can be done but it is easier said then done.
The fact that DevOps is somehow connected with Agile is pretty alarming and suggests that might well be yet another "snake
oil" initiative. With a bunch of talented an unprincipled salesmen who benefit from training courses, consulting gigs, published books,
conferences, and other legal ways to extract money from lemmings.
When you read sentences like "DevOps as an environment where an agile relationship will take place between operations and
development teams. "(https://www.quora.com/What-is-devops-and-why-is-it-important
) you quickly understand what type of people can benefit from DevOps.
The key objection to DevOps is that reliance on super-platforms such as Amazon cloud or Microsoft Azure could, in the future potentially
intellectually capture the organization and remaining sysadmins (and IT staff in general), who are increasingly distant from “nuts
and bolts" of operating system and hardware and operate in what essentially is proprietary environment of the particular vendor.
That converts Unix sysadmin into a flavor on Windows sysadmin with less nice GUI. In other words they need put all trust
in the platforms and detached themselves from "nuts and bolts" levels. That means that, in a way, they became as dependent on
those platforms as opiates addicts on their drags.
The also creates a set of DevOps promoters such as cloud providers, who want to become "gatekeepers", binding users to their
technology. Those gatekeepers once they became non-displaceable makes sure that the organization lost the critical mass of technical
IQ in "low level" (operating system level) infrastructure and can't abandon them without much pain.
At this point they start to use this dependency to their advantage. Typically they try to estimate customers "willingness to
pay" and, as a result, gradually increase the price of their services. IBM was a great practitioner of this fine art in the past.
That's why everybody hated it. VMware is also proved to be quite adept in this art.
Not that Amazon cloud is cheap. It is not, even is we calculate not only hardware and electricity savings, but also (and rather generously
as in one sysadmin for 50 servers) manpower saving they provide. It is often cheaper to run your own hardware within the internal
cloud then Amazon unless you have a huge peaks.
The same is even more true for VMware. If in the particular organization VMware used as virtualization platform for Linux (which
due to the being open source allows para- virtualization, instead of full virtualization that VMware implements) , then to talk about
savings is possible only when one is sufficiently drunk. Not that such savings do not exist. They do, but lion share of them goes
directly to VMware, not to the organization which deploy this platform.
The danger is that you are basically willingly allowed to capture yourself and willing to be part of this ecosystem that is controlled
by one single "gatekeeper." Such a decision creates an environment in which switching costs can be immense. That's
why there is such a competition amount three top players in cloud provider space for the new enterprise customers. The first who
grab a particular customer is the one to control and can milk such a customer for a long, long time. Classic DevOps advocates response
that "you should have negotiated better" is false because people not have enough information when they enter negotiations, and it's
too late when they finally will "get it."
DevOps is presented by adherents as all singing, all dancing universal solution to all problems of mankind, or, at least, of current
problems such as the overcomplexity, alienation of developers. paralysis via excessive security, and red tape that exists in the modern
data center.
But the problem is that the level of complexity of modern It is such that the division of labor between sysadmins and developers
is not only necessary it is vital for the success.
Such hope is ignoring the fact that there is no "techno cure" for large datacenter problems, because those problem are not only technological
in nature, but also reflect complex mix of sociological (the curse of overcomplexity is one of those; see
The Collapse of Complex Societies; neoliberal transformation of the enterprise with switch to outsourcing and contractor labor is
another) and, especially, balance of power between various groups within the data center. Such as corporate management, developers and
operation staff.
Which create pretty interesting mix from sociological point of view and simultaneously creates a set of internal conflict and constant
struggle for power between various strata of the datacenter ecosystem. From this point of view DevOps clearly represents political victory
for developers, and management at the expense of other players and first of all system administrators.
In a way, this idea can be viewed as a replay (under a new name) old Sun idea "network is computer". That does
not mean that there is a rational element in DevOps, as the trend to merge individual servers into some kind of computational superstructure.
As exemplified by Amazon cloud and Azure with their complete switch to virtual instances and attempt to diminish the role of "real
hardware".
This tend exist for quite a long time. For example Sun grid was just one and early successful attempts in this direction which led
to creation of the whole class of computer environment which now are known as computational clusters. DevOps can be viewed as an attempt
to create "application clusters".
To the extent that it is connected with advances in virtualization such as Solaris zones and Linux containers it is not that bad
if solid ideas get some marketing push.
But the key question here is: can we really eliminate the sysadmin role in such a complex environment as, for example modern
Linux (as exemplified by RHEL 7) ? Does cloud environment such as Azure creates for developer the possibility to get rip of sysadmin
and do it all by himself/herself.
No, but it is true that with virtualization the level of dependency can be less that in classic datacenter with real servers.
One hugely positive thing from the point of view of developers that in virtual instances in cloud environment they have access to root.
That really free them for shackles imposed by the fact that of "real servers" only sysadmin has access to root, and it can be grated
to the developer only temporary and with some red tape involved.
But for important production servers that is a double-edge sword. If the developer does not fully understand the ropes (and with
the complexity of RHEL 7 he just can't) he/she can bring the server down in no time by a reckless action, repaying various
Sysadmin Horror Stories on a new level. So the level of defense in the form of sysadmin is
now absent. And the environment is no less complex that in classical datacenter. Especially with RHEL 7.
There is another aspect in cloud technology. An organization can not productively use technology platform that is far above technical
capabilities of the staff. And if you try at the end of the day you will suffer unpredictable set of negative consequences. The
elimination of sysadmins lower the level of technical competence in the eEnterprise dramatically. The sooner the management realize
that this is a false gold the better. After all people have only one head and companies like Red Hat ignore this limitation with the
monster complexity of RHEL 7 (which added the systemd daemon)
That effects are only to cost overruns (which usually can be swiped under the carpet). There are additional effects which are somewhat
similar to those predicted by the first two Softpanorama Laws of Computer
Security:
In a long run the level of security of any large enterprise Unix environment can not be significantly different from the
average level of qualification of system administrators responsible for this environment...
If a large discrepancy between the level of qualification of system administrators and the level of Computer Security of
the system or network exists, the main trend is toward restoring equilibrium at some, not so distant, point...
In other words elimination of sysadmin level means that technology can't be utilized fully because of lack of understanding, including
lack of understand of security by the developers.
But there are also other reasons why this mode of work is not reproducible in the in the modern datacenter. The maximum you can achieve
(and that's pretty complex undertaking) is to remove some artificial bureaucratic barriers and make developer more productive because
he can control more of the whole process not to bump with his head into bureaucratic obstacles when he need to make some changes in
the environment in addition to his application.
But it is very important to understand that there are good developers and bad developers. Reckless power hungry developer,
who will utilize those new opportunities 100%, is a great danger. By removing bureaucratic obstacles you also remove some safeguards
against such developers. There is no free lunch.
Reckless developer is a great danger. By removing bureaucratic obstacles you also remove safeguards against
such developers. There is no free lunch.
In other words the idea of even partial elimination of specialization between developers and operation staff is a dangerous illusion
taking into account the complexity of modern operating systems. Giving a talented developer his own sysadmin and tester
makes sense. But making him his own sysadmin and tester is borderline idiotism. Talented developers is a very rare commodity and
their use should be highly optimized. Re-reading Mythical Man-month helps
to get proper line of thinking about this problem.
The idea of removing extra red tape also has some validity, but here you are facing uphill battle against entrenched IT management
interests, which try to isolate itself from people in the trenches with additional management layers (and those "redundant" management
layers is kind of cancer of IT departments, because those middlemen tend to replicate like rabbits in uncontrollable fashion).
High level IT management want technoserfs to work like slaves on this techno plantation to get their bonuses. And they want to achieve
that without direct contact with those "badly smelling people" creating those middle management positions that isolate them.
But in order to pay those people the herd of technoserfs needs to thinned, as private profits and bonuses for the brass
here comes before efficiency :-) Generally the question of parasitic rents within IT organization is a very interesting topic,
but it is beyond the scope of this paper.
As large datacenters and large software development problem are inherently complex. So going over budget and taking twice longer
than it should in large software project is more or less typical course of events. A larger danger here is compete fiasco, like in the
past happened with too enthusiastic idiots who implemented SAP in their companies. This type of failures actually can crash the whole
company and put it out of business.
Any claims that those fundamental issues can be somehow eliminated with some cosmetic measures ("magic dust") is a typical snake
oil salesmen staff. Software development talent, especially high level software development talent that encompass also strong
architectural vision, is a very scarce commodity. In you have no such developers, that chances of successful completion of a complex
project are slim. No superficial remedies can help to solve this tremendously difficult problem. Huge blunders of incompetent
high level IT managers is also a reality.
In short the key problem of modern datacenter and modern software applications is lack of top software talent both on the level
of software developers and high IT management. As well as mind boggling complexity of both the operating systems and applications.
They are probably the most complex artifacts created by mankind. And that means that it is unclear whether the better solution
lies in increasing the complexity or decreasing it. I tend to think that the latter might be a better path forward (KISS principle).
Which puts me on the collision course with DevOps hoopla. Your mileage may vary.
Cost wise DevOps (typically implemented with the help of outsourcers) is a rather expensive proposition which provides both higher
complexity and lower reliability. Outsourcing costs are low usually only at the beginning as most vendors perform "bait and switch"
underbidding their services and then try to compensate this by inflating operating costs, whenever possible, especially in areas were
local "know how" disappeared.
So benefits, is they exist, are not automatic, but are dependent of the qualification of people implementing this initiative.
VM problems while relatively rare still are unwelcome addition to the set of existing problem; also if hypervisor or server goes
down all VMs go down with it). Using VM is very similar to replacing enterprise servers with desktops, but at a higher cost (BTW "disposable
desktops" is an interesting alternative to VMware VMs cost wise). The cost of "decent" desktop ($500-$800) now is one tens of
the cost of a "decent" enterprise server ($6K-$8K). but ten desktops can definitely do more computation wise.
Adding to the mix management tools such as Puppet works for multiple desktops too.
DevOps was a reaction to the problem in companies who now need to maintain thousands of servers, such as Netflix, Amazon, Facebook
and Google.
So DevOps is intrinsically connected with attempts of merging servers into some computational superstructure and automation of system
administration using Unix configuration management tools. BTW that does not mean that Puppet or Chief are the proper way to do that
(the king might still be naked ;-). They just bask in DevOps glory. And try to sustain and maintain the DevOps hype for fun and
profit.
The second part is an attempt to break isolated and fossilized silos in which developers and operation staff live in large corporation.
Sometimes with almost without interaction. So in a way this as an over-reaction to a dysfunctional ops organization (as
Adrian Cockcroft
admitted in his
comment.), which actually clearly demonstrates that he does not understand how complex organization operated (his idea of
abolishing operations is just an attempt to grab the power by "developer class") and never read the book by Charles Perrow (Complex
Organizations A Critical Essay). In complex organization technological issues re always intersect with power struggle and
effects of neoliberalism on IT and corporate environment in general. Blowback of excessive outsourcing of IT is hitting large corporations
and is a part of problem that IT organizations experience now. DevOps in this sense is a part of the problem and not a part of the solution.
You will never read about this this in "DevOps hoopla" style of books that proliferate (Amazon lists around hundred of books on this
topic; most junk). In reality globalization while solving one set of problem creates another and her IT is both the victim and
the part of this "march of neoliberalism" over globe:
Information and communications technologies are part of the infrastructure of globalization in finance, capital mobility and transnational
business. Major changes in the international economic landscape are intertwined and contemporary accelerated globalization is in
effect a package deal that includes informatization (applications of information technology), flexibilization (changes in production
and labour associated with post-Fordism), financialization (the growing importance of financial instruments and services) and deregulation
or liberalization (unleashing market forces). This package effect contributes to the dramatic character of the changes associated
with globalization, which serves as their shorthand description.
A good early overview why DevOps emerged (when it was still called NoOps) was made in 2012 by Mile Loukidis, who wrote historically
important paper of the subject.
Adrian Cockcroft’s article about
NoOps at Netflix ignited a
controversy that has been smoldering for some months. John Allspaw’s detailed
response to Adrian’s article makes a key point: What Adrian described as “NoOps” isn’t really. Operations doesn’t go away. Responsibilities
can, and do, shift over time, and as they shift, so do job descriptions. But no matter how you slice it, the same jobs need to be
done, and one of those jobs is operations. What Adrian is calling NoOps at Netflix isn’t all that different from Operations at Etsy.
But that just begs the question: What do we mean by “operations” in the 21st century? If NoOps is a movement for replacing operations
with something that looks suspiciously like operations, there’s clearly confusion. Now that some of the passion has died down, it’s
time to get to a better understanding of what we mean by operations and how it’s changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no distinction between dev and ops. If you
developed, you operated. You mounted the tapes, you flipped the switches on the front panel, you rebooted when things crashed, and
possibly even replaced the burned out vacuum tubes. And you got to wear a geeky white lab coat. Dev and ops started to separate in
the ’60s, when programmer/analysts dumped boxes of punch cards into readers, and “computer operators” behind a glass wall scurried
around mounting tapes in response to IBM JCL. The operators also pulled printouts from line printers and shoved them in labeled cubbyholes,
where you got your output filed under your last name.
The arrival of minicomputers in the 1970s and PCs in the ’80s broke down the wall between mainframe operators and users, leading
to the system and network administrators of the 1980s and ’90s. That was the birth of modern “IT operations” culture. Minicomputer
users tended to be computing professionals with just enough knowledge to be dangerous. (I remember when a new director was given
the root password and told to “create an account for yourself” … and promptly crashed the VAX, which was shared by about 30 users).
PC users required networks; they required support; they required shared resources, such as file servers and mail servers. And yes,
BOFH (“Bastard Operator from Hell”) serves as a reminder of those days. I
remember being told that “no one” else is having the problem you’re having — and not getting beyond it until at a company meeting
we found that everyone was having the exact same problem, in slightly different ways. No wonder we want ops to disappear. No wonder
we wanted a wall between the developers and the sysadmins, particularly since, in theory, the advent of the personal computer and
desktop workstation meant that we could all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important websites. As companies and computing
facilities grew larger, the fire-fighting mentality of many system administrators didn’t scale. When the whole company runs on one
386 box (like O’Reilly in 1990), mumbling obscure command-line incantations is an appropriate way to fix problems. But that doesn’t
work when you’re talking hundreds or thousands of nodes at Rackspace or Amazon. From an operations standpoint, the big story of the
web isn’t the evolution toward full-fledged applications that run in the browser; it’s the growth from single servers to tens of
servers to hundreds, to thousands, to (in the case of Google or Facebook) millions. When you’re running at that scale, fixing problems
on the command line just isn’t an option. You can’t afford letting machines get out of sync through ad-hoc fixes and patches. Being
told “We need 125 servers online ASAP, and there’s no time to automate it” (as
Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn’t surprising. One of the themes of O’Reilly’s
Velocity Conference is “Infrastructure as Code.” If you’re going to do operations
reliably, you need to make it reproducible and programmatic. Hence virtual machines to shield software from configuration issues.
Hence Puppet and Chef
to automate configuration, so you know every machine has an identical software configuration and is running the right services. Hence
Vagrant to ensure that all your virtual machines are constructed identically
from the start. Hence automated monitoring tools to ensure that your clusters are running properly. It doesn’t matter whether the
nodes are in your own data center, in a hosting facility, or in a public cloud. If you’re not writing software to manage them, you’re
not surviving.
Furthermore, as we move further and further away from traditional hardware servers and networks, and into a world that’s virtualized
on every level, old-style system administration ceases to work. Physical machines in a physical machine room won’t disappear, but
they’re no longer the only thing a system administrator has to worry about. Where’s the root disk drive on a virtual instance running
at some colocation facility? Where’s a network port on a virtual switch? Sure, system administrators of the ’90s managed these resources
with software; no sysadmin worth his salt came without a portfolio of Perl scripts. The difference is that now the resources themselves
may be physical, or they may just be software; a network port, a disk drive, or a CPU has nothing to do with a physical entity you
can point at or unplug. The only effective way to manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming code as early as the late ’80s;
indeed, Perl was designed as a programming language for automating system administration. It didn’t take long for leading-edge sysadmins
to realize that handcrafted configurations and non-reproducible incantations were a bad way to run their shops. It’s possible that
this trend means the end of traditional system administrators, whose jobs are reduced to racking up systems for Amazon or Rackspace.
But that’s only likely to be the fate of those sysadmins who refuse to grow and adapt as the computing industry evolves. (And I suspect
that sysadmins who refuse to adapt swell the ranks of the BOFH fraternity, and most of us would be happy to see them leave.) Good
sysadmins have always realized that automation was a significant component of their job and will adapt as automation becomes even
more important. The new sysadmin won’t power down a machine, replace a failing disk drive, reboot, and restore from backup; he’ll
write software to detect a misbehaving EC2 instance automatically, destroy the bad instance, spin up a new one, and configure it,
all without interrupting service. With automation at this level, the new “ops guy” won’t care if he’s responsible for a dozen systems
or 10,000. And the modern BOFH is, more often than not, an old-school sysadmin who has chosen not to adapt.
James Urquhart nails it when he describes how
modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt
to huge swings in load, etc. But he notes that those features, formerly provided by the IT/operations infrastructures, now need to
be part of the application, particularly in “platform as a service” environments. Operations doesn’t go away, it becomes part of
the development. And rather than envision some sort of uber developer, who understands big data, web performance optimization, application
middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams.
The infrastructure doesn’t go away — it moves into the code; and the people responsible for the infrastructure, the system administrators
and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they
need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as “DevOps.”
Amazon’s EBS outage last year demonstrates how the nature of
“operations” has changed. There was a marked distinction between companies that suffered and lost money, and companies that rode
through the outage just fine. What was the difference? The companies that didn’t suffer, including Netflix, knew how to design for
reliability; they understood resilience, spreading data across zones, and a whole lot of reliability engineering. Furthermore, they
understood that resilience was a property of the application, and they worked with the development teams to ensure that the applications
could survive when parts of the network went down. More important than the flames about Amazon’s services are the testimonials of
how intelligent and careful design kept applications running while EBS was down.
Netflix’s ChaosMonkey is
an excellent, if extreme, example of a tool to ensure that a complex distributed application can survive outages; ChaosMonkey randomly
kills instances and services within the application. The development and operations teams collaborate to ensure that the application
is sufficiently robust to withstand constant random (and self-inflicted!) outages without degrading.
On the other hand, during the EBS outage, nobody who wasn’t an Amazon employee touched a single piece of hardware. At the time,
JD Long tweeted that the best thing about the EBS outage was that his
guys weren’t running around like crazy trying to fix things. That’s how it should be. It’s important, though, to notice how this
differs from operations practices 20, even 10 years ago. It was all over before the outage even occurred: The sites that dealt with
it successfully had written software that was robust, and carefully managed their data so that it wasn’t reliant on a single zone.
And similarly, the sites that scrambled to recover from the outage were those that hadn’t built resilience into their applications
and hadn’t replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to the application itself, we’re also
seeing a redistribution of costs. It’s a mistake to think that the cost of operations goes away. Capital expense for new servers
may be replaced by monthly bills from Amazon, but it’s still cost. There may be fewer traditional IT staff, and there will certainly
be a higher ratio of servers to staff, but that’s because some IT functions have disappeared into the development groups. The bonding
is fluid, but that’s precisely the point. The task — providing a solid, stable application for customers — is the same. The locations
of the servers on which that application runs, and how they’re managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds like Amazon’s, private clouds, traditional
colocation, and building their own infrastructure. It’s hard to beat Amazon if you’re a startup trying to conserve cash and need
to allocate or deallocate hardware to respond to fluctuations in load. You don’t want to own a huge cluster to handle your peak capacity
but leave it idle most of the time. But Amazon isn’t inexpensive, and a larger company can probably get a better deal taking its
infrastructure to a colocation facility. A few of the largest companies will build their own datacenters. Cost versus flexibility
is an important trade-off; scaling is inherently slow when you own physical hardware, and when you build your data centers to handle
peak loads, your facility is underutilized most of the time. Smaller companies will develop hybrid strategies, with parts of the
infrastructure hosted on public clouds like AWS or Rackspace, part running on private hosting services, and part running in-house.
Optimizing how tasks are distributed between these facilities isn’t simple; that is the province of operations groups. Developing
applications that can run effectively in a hybrid environment: that’s the responsibility of developers, with healthy cooperation
with an operations team.
The use of metrics to monitor system performance is another respect in which system administration has evolved. In the early ’80s
or early ’90s, you knew when a machine crashed because you started getting phone calls. Early system monitoring tools like HP’s OpenView
provided limited visibility into system and network behavior but didn’t give much more information than simple heartbeats or reachability
tests. Modern tools like DTrace provide insight into almost every aspect of system behavior; one of the biggest challenges facing
modern operations groups is developing analytic tools and metrics that can take advantage of the data that’s available to predict
problems before they become outages. We now have access to the data we need, we just don’t know how to use it. And the more we rely
on distributed systems, the more important monitoring becomes. As with so much else, monitoring needs to become part of the application
itself. Operations is crucial to success, but operations can only succeed to the extent that it collaborates with developers and
participates in the development of applications that can monitor and heal themselves.
Success isn’t based entirely on integrating operations into development. It’s naive to think that even the best development groups,
aware of the challenges of high-performance, distributed applications, can write software that won’t fail. On this two-way street,
do developers wear the beepers, or IT staff? As Allspaw points out, it’s important not to divorce developers from the consequences
of their work since the fires are frequently set by their code. So, both developers and operations carry the beepers. Sharing responsibilities
has another benefit. Rather than finger-pointing post-mortems that try to figure out whether an outage was caused by bad code or
operational errors, when operations and development teams work together to solve outages,
a post-mortem can focus less on assigning
blame than on making systems more resilient in the future. Although we used to practice “root cause analysis” after failures, we’re
recognizing that finding out the single cause is unhelpful. Almost every outage is the result of a “perfect storm” of normal, everyday
mishaps. Instead of figuring out what went wrong and building procedures to ensure that something bad can never happen again (a process
that almost always introduces inefficiencies and unanticipated vulnerabilities), modern operations designs systems that are resilient
in the face of everyday errors, even when they occur in unpredictable combinations.
In the past decade, we’ve seen major changes in software development practice. We’ve moved from various versions of the “waterfall”
method, with interminable up-front planning, to “minimum viable product,” continuous integration, and continuous deployment. It’s
important to understand that the waterfall and methodology of the ’80s aren’t “bad ideas” or mistakes. They were perfectly adapted
to an age of shrink-wrapped software. When you produce a “gold disk” and manufacture thousands (or millions) of copies, the penalties
for getting something wrong are huge. If there’s a bug, you can’t fix it until the next release. In this environment, a software
release is a huge event. But in this age of web and mobile applications, deployment isn’t such a big thing. We can release early,
and release often; we’ve moved from continuous integration to continuous deployment. We’ve developed techniques for quick resolution
in case a new release has serious problems; we’ve mastered A/B testing to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations staff. Operations groups are adopting,
and in many cases, leading in the effort to implement these changes. They’re the specialists in resilience, in monitoring, in deploying
changes and rolling them back. And the many attendees, hallway discussions, talks, and keynotes at O’Reilly’s Velocity conference
show us that they are adapting. They’re learning about adopting approaches to resilience that are completely new to software engineering;
they’re learning about monitoring and diagnosing distributed systems, doing large-scale automation, and debugging under pressure.
At a recent meeting, Jesse Robbins described scheduling EMT training sessions for operations staff so that they understood how to
handle themselves and communicate with each other in an emergency. It’s an interesting and provocative idea, and one of many things
that modern operations staff bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and bleeding-edge; now, it’s expected.
But we haven’t taken it far enough. We’re still learning how to monitor systems, how to analyze the data generated by modern monitoring
tools, and how to build dashboards that let us see and use the results effectively. I’ve joked about “using a Hadoop cluster to monitor
the Hadoop cluster,” but that may not be far from reality. The amount of information we can capture is tremendous, and far beyond
what humans can analyze without techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient protocols for the web, like
SPDY. Operations is involved, more than ever, in tuning the performance
of operating systems and servers (even ones that aren’t under our physical control); a lot of our “best practices” for TCP tuning
were developed in the days of ISDN and 56 Kbps analog modems, and haven’t been adapted to the reality of Gigabit Ethernet, OC48*
fiber, and their descendants. Operations groups are responsible for figuring out how to use these technologies (and their successors)
effectively. We’re only beginning to digest IPv6 and the changes it implies for network infrastructure. And, while I’ve written a
lot about building resilience
into applications, so far we’ve only taken baby steps. There’s a lot there that we still don’t know. Operations groups have been
leaders in taking best practices from older disciplines (control systems theory, manufacturing, medicine) and integrating them into
software development.
And what about NoOps? Ultimately, it’s a bad name, but the name doesn’t really matter. A group practicing “NoOps” successfully
hasn’t banished operations. It’s just moved operations elsewhere and called it something else. Whether a poorly chosen name helps
or hinders progress remains to be seen, but operations won’t go away; it will evolve to meet the challenges of delivering effective,
reliable software to customers. Old-style system administrators may indeed be disappearing. But if so, they are being replaced by
more sophisticated operations experts who work closely with development teams to get continuous deployment right; to build highly
distributed systems that are resilient; and yes, to answer the pagers in the middle of the night when EBS goes down. DevOps.
Photo: Taken at IBM’s headquarters in Armonk, NY. By Mike Loukides.
There are many types of members, each with its own motivation.
First of all there are the ones who have been moved to do so out of a genuine spiritual aspiration, prepared to sacrifice.
Other members may have been searching for a truth behind the dreadful aspects of life around them and an escape therefrom.
People who wrestle with a psychological problem seek (unconsciously) a key or solution to it in a particular teaching or discipline.
Then there are those that did so out of a feeling of isolation and loneliness. Lack of true companionship may have made them
long to be taken up in a fraternity of kindred souls.
People will be drawn to a brotherhood whose aims are within their reach of comprehension and nearest to their hearts. A religious
background may make them veer towards a sect that accomodates nostalgia, like the Jehova's Witnesses.
For younger people it may be an escape from having to enter into a harsh materialistic society. They are in an idealistic
life-cycle, prepared to give up their study and career to a worthy cause.
Some older people can never make up their minds and wander from one cause to another.
Finally there are those who wrestle with a father/mother complex. The childhood's need for such archetypal figure may persist
in later life. It is transferred subsequently to the master, who then holds sway over him/her.
The weaker the individual's independence, the more will he be tied to the group. Members who understand group-mechanisms, prepared
to cope with them in order to direct their attention to the spirit, will benefit most as they are selective in picking up the cream
of what is given and taking the rest with a grain of salt.
2. Leader/founder/guru
New religious movements arise usually around a father/mother figure who has gained authority after receiving a special revelation,
communication, truth or insight. His charisma will vouchsafe loyal followers, even if his lifestyle may give rise to severe doubts
to some. He may boost his prestige by claiming to follow the footsteps of a an esteemed spiritual teacher, represent an esoteric
tradition, be of noble descent, or channel the wisdom of a great mind. (Eckankar's Paul Twitchell is the last in the lineage
of 970 "Eckmasters")
He/she represents an archetype in members' subconscious minds. That of a wise father, or mother. As such he/she will have a compelling
influence on followers who project their father/mother complex on him/her.
Alternatively women may fall in love with the leader, worship him, exert themselves to cater after his wishes and whims. They
will try to stay in his vicinity, make themselves indispensable and slowly take control of the movement. Jealousy amongst them will
make things even worse and split the ranks.
The psychological make-up of a guru may be generalized as follows:
Difficult youth. Self-chosen isolation at childhood. Introvert. Therefore not used to share inner life with friends.
Narcissistically absorbed in inner mental processes.
Undergone a traumatic experience.
Authoritarian attitude. Elitist and anti-democratic.
Attracting disciples rather than friends on account of the fact that they never learnt to exchange thoughts in childhood.
Jeffrey Masson (see below) has this to say about gurus:
Every guru claims to know something you cannot know by yourself or through ordinary channels. All gurus promise access to
a hidden reality if only you will follow their teaching, accept their authority, hand your life over to them. Certain questions are
off limits. There are things you cannot know about the guru and the guru's personal life. Every doubt about the guru is a reflection
of your own unworthiness, or the influence of an external evil force. The more obscure the action of the guru, the more likely it
is to be right, to be cherished. Ultimately you cannot admire the guru, you must worship him. You must obey him, you must humble
yourself, for the greater he is, the less you are - until you reach the inner circle and can start abusing other people the way your
guru abused you. All this is in the very nature of being a guru.
Sub-conscious drive
Nature seems to instill in a person, faced with a mission, great task, or challenge, a feeling of superiority, insurmountable
optimism, and enormous self-esteem, bordering on an inflated ego, to accomplish what is needed. This drive is reminiscent of the
reckless impetus of the adolescent. Having reached maturity a person may feel "chosen" - impelled to forge ahead with vigor and inspire
others. Undaunted in the face of obstacles and criticism, it is as if a cloak of invulnerability is laid on his/her shoulders.
Similarly an artist may be driven by a compulsion to express an inner content. He will be prepared to sacrifice everything to give
way to his creative impulse. Fortunately his sacrifice does not involve more than the people immediately around him.
Not so with the leader. The number of his followers may grow to considerable proportions. Nature is not concerned whether his
sense of superiority has any real foundation. The inflated ego is more or less instinctively driven towards a goal.
Although attaining heights no one would have thought conceivable of that person, when the hour of truth has come events may prove
that he has overreached himself, disregarded good advice, or lost complete sense of reality. The result may be either catastrophe,
or the uncritical followers may be saddled up with a heritage built on quicksand - on a flight of fancy without actual foundation.
This applies to many fields of human endeavour (Hitler), but specially in the treacherous domain of the spirit.
Discipline - nausea
The teacher may come to the conclusion that unless his followers change fundamentally - undergo a catharsis, or transformation -
they will never be able to move forward. He/she regards them as being "asleep" (Jesus, Gurdjieff). Unless drastic measures are employed
they will not wake up. To jolt them out of their complacency great sacrifices are demanded. Jesus asked a rich young
man to give up all his worldly possessions (S.Matthew 19:21) before following him. Masters in Zen Buddhism, or Gurdjieff,
made novices undergo a harsh regime in order to crack open and attain a different state of mind.
This I can have no quarrel with,
if it is done against a background of compassion. If the unselfish motive disappears, or commercial considerations become dominant,
the harsh discipline may become morbid and degrading. Having lost his dedication the teacher may become nauseated by the mentality
and sheepishness of his followers, and in cases derive a sadistic delight in tormenting them.
In recent years reports are brought out about sexual violation of members by guru's, leaders and....bishops! Another example
of authority being abused.
The path of a guru is like that of a razor's edge. He may so easily succumb to the temptation of exploiting the power he has attained
over his followers. Financial irresponsibility, abuse of followers, reprehensible sexual behaviour......... mass suicide, it is all
within his reach once he has overstepped boundaries.
Legacy
During his lifetime the leader will act as a moderator and steer the movement. He will re-interprete his teachings as he sees
fit from the responses he receives. The death of the founder marks a turning point. His teachings will become inflexible, as no one
dares to temper with them as he did himself. The élan disappears, rigidity takes over, unless another figure arises that leads the
movement in a different direction, for better or for worse (St.Paul).
3. Doctrine/teaching
The more secret(ive) the leader's sayings the better. Pronouncements are characterized by great certainty and authority as if
it were the word of God. In some cases it is presented as such. By his special way of delivery and presentation it may escape the
audience that similar wisdom may be found in any book on spirituality nowadays found in the bookshop around the corner.
Whether the guru bases his wise words on actual experience or on hearsay is difficult to ascertain. In general it may be said
that the more mystifying his teachings the stronger their appeal. After all it is beyond reason and should appeal only to the heart.
An exception should be made for true mystical literature based on inner experience which can hardly be expected to appeal to the
intellect, but be appreciated intuitively, especially by those who had similar experiences.
Group-speak
Members may adopt fresh meanings to words, talk to each other in a jargon that the outsider can hardly follow (group-speak).
The result being an inability to relate in speech, or explain new concepts to the outsider (Fourth Way).
(This may be best understood in other fields: help-programs of software, pop-up windows, warning-messages, not to speak of manuals
for installing hardware, drawn up by boffins, are a nightmare to most users!)
Another characteristic is to lift out of context one aspect of religious truth and make it absolute. Such key truth will overshadow
all other aspects of faith.
It may be:
prophecy, as we have seen with Jehova's Witnesses, Adventists and countless sects;
baptism at an adult age, important to Baptists;
charity in the case of the Salvation Army (but who could blame them for this?);
channeling in New Age/Spiritualistic movements,
etc. When this occurs other significant facets of faith are pushed to the background.
Such partial truths are often heralded as the result of a search for knowledge. The motto "Knowledge is power" is used to suggest
that the statements are objective, scientific, or historical facts. Actually they cannot stand the touchstone of the merest critical
scrutiny.
Authorities may be paraded to back-up such claims. They have either never been heard of, cannot be considered impartial, or their
pronouncements have been lifted out of context. The discussion about the veracity of evolution is full of such red herrings.
4. Uniqueness of the movement
Movements will extol usually their superiority over others. After all there should be a strong reason to select that particular
group. Some present themselves as being the sole way towards salvation, being God's chosen people. Others make a promise of a benefit
that is only reserved for members of that sect. To avert attention some pride themselves of not having a teaching, or for their openness
and democratic rules.
In short new movements will advance a variety of reasons for their uniqueness. Herewith a few:
Our holy book is the word of God.
Never before has mankind been offered this discipline/interpretation/insight of our leader.
Do not pour new wine in old bottles.
God's, or a celestial, new message to mankind for this particular time.
Impending disaster (pollution, atomic explosion, collision) calls for drastic measures. Salvation only reserved for faithful
members.
Fresh interpretation of holy book thanks to fresh insight/revelation of founder.
Esoteric tradition, once accessible to adepts only, now revealed.
New channel for teachings of esoteric lodge/brotherhood.
New doctrine/insight based on latest scientific discoveries, reveals truth.
Only those following this particular work on self, discipline, or belief, will reach eternity, be released from earth's satanic
attraction, cycle of rebirths, etc.
Preparatory group to make way for the coming of the new messiah/world-teacher/avatar
Noteworthy is the vehemency with which groups stress differences between each other. The closer movements share an outlook
the more virulent the attacks on their rivals become, seemingly more than on groups which follow a completely different belief.
Eric Hoffer writes in his 'The True Believer': "true believers of various hues ....view each other with mortal hatred and
are ready to fly at each other's throat..."
This manifests itself specially when groups split. In Christianity one could not steep low enough to attack other followers of
Christ, who held a slightly different opinion. It resulted in persecution of heretics, burning of early Christian literature, and
disastrous wars.
Despite their peaceful appearance relatively new spiritual movements like Theosophy, Rosicrucianism, etc., following splits, exert
themselves in accusations against former comrades.
Attacks against belief in paranormal phenomena, for instance by CSICOP, are reminiscent of the zeal of a Christian crusade,
be it that they have their roots in humanism and its desperately clinging to a rationalistic/materialistic outlook on life current
at the beginning of this century. Consequently the groups of these 'evangelists of rational enlightenment' have similar behavioural
patterns and vehemency as sects.
5. Probation and conversion
Certain sects are only too eager to accept individuals. They may have high entrance fees. Or their members are swayed by zeal
to convert.
Many movements will put up a barrier by means of an initiation to test the applicant's fitness to become part of the group. Henceforth
they will play an important pioneer-part in the foretold future. Having reached such coveted stage members will not fail to follow
what they are being told for fear of expulsion.
The new member may undergo a conversion, gaining a completely new insight in the meaning of life, see it in a way the sect
does. His previous life with all its relationships has become meaningless. He may have turned himself inside out by a confession
of his previous "sins". His conversion is marked by a feeling of peace, happiness and transcendence.
6. Failure of predictions
Common belief in a prophecy will be a strong binding force. One of the principal attractions of the first Christian sects was
that they offered salvation from a threatening disaster. That being the end of the world. Only the baptized would await a glorious
future. Sects like the Jehova's Witnesses have taken over this succesful formula.
Christians have had to come up with all sorts of arguments to explain away the unfulfilled prediction of their founder regarding
the end of the world: "This generation shall not pass away, till all these things be accomplished." (S.Matthew 24:34). One of the
lame excuses being that this prediction concerns the fall of Jersusalem only. However, all prophecies in the New Testament in this
respect suggest that the impending doom was to be expected in their lifetime.
Jehova's Witnesses have taken the risk of being more specific in their predictions. Older members, who built their faith
on them, have had the humiliating experience of having had to explain away various times in their lives the failure of the outcome
of their forewarnings.
But predictions are not limited to the religious faiths. The New Age movements use this shared belief in portents as well.
For more than sixty years an imminent landing of UFO's has been predicted. Various cults claimed in vain to be their first contactees.
In other movements the second coming of Christ was a main feature (Benjamin Creme). In Theosophy a Messenger was expected
from 1975 onward.
The uncritical believers in Edgar Cayce's trance sayings put weight on his predictions of cataclysms (photo Edgar Cayce).
Nostradamus' (photo) obscure astrological foresayings have captured the minds of people for centuries. Each time
his verses were interpreted again to suit the circumstances. In hindsight some of his quatrains seem to have relevance to the catastrophe
of the destroyed World Trade Center. Quatrains I, 87 - IX, 92 and X, 59 may refer to skyscrapers in New York involved in a terrible
explosion.
Sociologists have observed that, failure of prediction results in quite the opposite effect on believers. Contrary to what one
would expect it may cause a rally amongst members. Failure is blamed on a misunderstanding, or a faux pas by members. To counteract
ridicule they tend to stick together more than ever.
Of course there is a limit. According to a social survey, when predictions fail to materialize three times in a row members are
bound to stop, reflect and draw conclusions.
The shattering of such false hopes comes as a severe blow and may mark the beginning of the end of a movement.
One wonders in this respect how many members of the People Forever International sect promoting physical immortality for its
followers would have to die before their groups would break up in disappointment. (Since I wrote this ten years ago I have been informed
that indeed members have died and the movement broke up in 1998!)
Yet, we see from the Jehova's Witnesses that skilful manoeuvring may off-set unfulfilled prophecies.
To what extremes such believes can lead shows the mass suicidal action of the Heaven's Gate sect and later in Uganda. Such
tragic endings are the result of various contributing factors, which are beyond the scope of this article.
7. Belief versus intellect/Secrets
Often disciplines followed in spiritual movements have the effect of a lowering of the threshold to the unconscious mind. Suggestion
will begin to play an important part. Precepts are being experienced as the truth, sacrosanct and sure. There is no element of doubt
anymore about assumptions and speculation, although actually they lack any factual foundation.
Absolute belief that the Bible is God's word is the cornerstone of most orthodox Christian sects. In Islam the Koran is supposed
to contain the word of Allah.
Intellectual analysis of faith is tentamount to heresy.
The ideal breeding ground for convictions are mass-gatherings. During mass gatherings, such as congresses, members are stirred
up to an euphoria, the effect of which may linger on for weeks. This is the precise period of time for leaders, or committees, to
announce fresh sectarian measures, postulate incredible notions/prophecies, call for further sacrifices, etc. etc. It will all be
accepted unquestioningly. Only at a later date, when the euphoria has worn off, will one start to wonder about what was decided.
Secrets
Spiritual movements often hide a corpse in their closet. It may be a part of the history of the movement, details about the hidden
life of the leader, or a once revered figure. Things may have been written by them one does not like to be reminded of. A fight,
quarrel, full of vehemence and hatred, may have led to a split.
There are so many examples that a long list could be drawn up of the many concealed secrets of spiritual groups.
Whereas in most movements the works of the leader are almost known by heart, Jehova's Witnesses hardly know of the existence
of the seven volumes of writings Studies in the Scriptures of their founder Charles Taze Russell (1852-1916). Some of his
opinions are such cause of embarrassment that they are not deemed worth reading nowadays.
Eventually a renegate member will reveal such secrets in writing. Frantic denials and counter accusations by those in charge presently
will follow almost automatically. These are usually accepted in gratitude by devotees, who cannot get over the shock of such revelations.
8. Common practice, work and ritual
Communal singing, ritual and (incomprehensible) practices (Freemasonry) are strong binding factors. The more irrational they are,
the better. Others are a special food regime, the change of name, clothing, or a common aversion.
Joint work for the benefit of the group gives the feeling of a common endeavour and unites the participants. So does proselytization
in the streets, menial work of construction and renovation of premises. There is a thin line between true participation and exploitation,
however.
Dubious was the practice, common in the seventies, to incite members to criticize one of them to an extent that he/she would break
down under the weight of often absurd allegations and insults, resulting in a brainwash effect.
9. Sacrifices, financial secrecy, favours to the rich.
Finances are always a ticklish matter. Human groups always wish to grow. Finances are important. Accountability is often not considered
appropriate. Danger arises that members of the inner circle become lax in expenditure of members' contributions. Ambitious schemes
call for a constant need for funding. This is the ideal breeding ground for favours to wealthy members. Those who contribute generously
stand more chance to be taken in confidence and admitted to the inner circles. Often, as a proof of loyalty, extraordinary sums of
money are demanded.
Degrees of initiation may be dependent on one's years of loyalty to the group. In Eckankar up to 8 degrees are given. However,
if one fails to pay membership's fees for some time, degrees of initiation may be stripped off again.
Next to financial contributions members will often be expected to offer services to the group. However, if they also have to work
for practically nothing in commercial enterprises it becomes dubious. Movements that gather wealth at the expense of their members
are questionable. Seldom or never requests for return of contributions/investments are honoured.
10. Unquestioning leadership, reprehensible behaviour amongst members
Man in a herd may not show the best side of his nature. Unconscious drives may reign his/her behaviour. This is applicable especially
in circumstances that man strives for the spiritual. Heshe may tend to show split-personality behaviour. On one hand the spiritual
personality which is supposed to have come to terms with his animal nature. It is wise, friendly and compassionate on the outside.
In the shadows lurks the personality that has been forced into the background, still ridden with all the expulsed human frailties.
In moments of weakness it will see its chance to play hideous tricks. It will do so without being noticed by the person involved.
The result being: uncharitable behaviour, envy, malicious gossip, hypocrisy, harsh words, insensitivity, unfounded criticism and
even worse, not expected from such charismatic figure. It is one of the main reasons for people leaving a particular group in great
disappointment.
It is not often realized that, like other human groups, spiritual movements behave like organisms. Group-psychological processes
manifest which are sometimes not unlike those in primitive societies. There is the pecking order, the alpha members, and also the
group-instinct directed against similar groups. Aggression goes unnoticed and is tolerated when an acceptable common goal is provided.
For instance hostility against an individual outside the group, or a critical member inside. This has the effect of strengthening
ties within the group like in the animal world.
If leadership loses contact with its members it will have to exert greater discipline. Deviating opinions cannot be tolerated
anymore. Persons who hold them are seen as traitors. Acting against them, preferably in secret, is the only way out for the leadership
to avert this danger. Members may disappear suddenly without the reasons becoming known, much to the surprise of those left behind.
For such machinations in Theosophy read Emily Lutyens: "Candles in the Sun".
Spiritual newsgroups on Internet provide illustration of (un)concious nastiness being ventilated under the veil of anonimity.
Messages are often rife with diatribe, personal attacks and misunderstanding. Many of such contributors have no interest at all in
the matters discussed. Yet even in closed newsgroups, only open to subscribers, complaints about the tone of communications are being
aired.
11. Fear of exclusion
The stronger members are tied to a group, the more the fear of exclusion lurks. They may have invested their life's savings in
the work (Scientology), paid a percentage of their income, failed to conclude their study, or make a career, or sacrificed a succesful
one.
In many cases a member will have alienated himself from family and friends. He has been told to cut ties with the past. (In the Attleboro
cult followers are advised to burn photographs that remind them of bygone days). No wonder his or her sudden conversion, accompanied
by fanatism and urge to proselytize, has shied away former friends and relatives.
There is no way left but to seek comfort and understanding with members of the spiritual group.
Isolation is sometimes intentionally sought. Formerly, in the Bhagavan Shri Rajneesh movement, members went about in red/orange
dresses and wore mala's with a photo of their master, so setting themselves aside from the mundane world.
The Hare Krishna movement goes even further. Groups of members go out into the streets in their oriental dresses for song
and dance routines. However, in most movements the alienation is far more subtle and the natural outcome of an adverse attitude towards
the materialism of society.
The true nature of the so-called friendships within the group will only be revealed after a devotee has left the fold. Members
have seen this happen, but did not give it a thought at the time, because it happened to someone else. But when they undergo the
same fate themselves they will feel the humiliation of being ignored, not being greeted anymore, marriage gone - even not being recognized
by one's own children anymore.
The outcast feels thrown in an abyss. He is cut off from social contacts, his life in pieces. The magnitude of this desperate
experience should not be under-estimated. The renegade will feel deep shame. He may have confessed in the group intimate secrets,
which are now being ridiculed by his former so-called friends. The expulsee, deeply hurt, may become embittered and even enter into
a suicidal mental state.
Those readers who have been a member of a movement may recognize some of the above psychological mechanisms. The first reaction
of non-members may be to vow never to enter a group. Let us bear in mind, however, that it should be considered a challenge to face
these obstacles for the benefit that may result from association with kindred spirits.
A prerequisite is that these conditions are being noticed, looked in the eye, and not denied. The closer people live together,
the more group-tensions will build up. Even in reputable circles as Freudian psycho-analytical associations they occur. Few communes
are granted a long life as a result of one or more of the pitfalls summarized above. Headquarters, contrary to expectations, are
known to be hotbeds of gossip, mutual repulsion and cynism.
So, do not be disheartened and join a group of your liking. After all people who marry also see wrecked marriages all around them,
yet go ahead intent on a happy union in mutual trust, without regard to the outcome.
Involvement with other people will lead to personal growth if the consequences are anticipated. The more one stands on one's own
feet the more benefit will arise from cooperating with others. It should be borne in mind that the saying "It is better to give,
than to receive" is not merely a moral precept. (Read my precepts
for living)
Please remember that there are hundreds of movements and that it has not been my intention to summarize them all, or to level
any form of criticism at one of them. Indicating the psychological mechanisms operative in some, or all of them, has been my main
theme.
On a separate page I have gone into the mysterious presence-phenomenon
arising between people who meet in harmony.
In conclusion one may take heed of Krishnamurti's words in 1929 when he refused to become a 'World Teacher' of an organisation
set up for him:
I maintain that Truth is a pathless land, and you cannot approach it by any path whatsoever, by any religion, by any sect.
Truth being limitless, unconditioned, cannot be organised, nor should any organisation be formed to lead or coerce people along any
particular path. If you first understand that, then you will see how impossible it is to organize a belief. A belief is purely an
individual matter, and you cannot and must not organize it. If you do, it becomes dead, crystallised; it becomes a creed, a sect,
a religion, to be imposed on others.
Why? Wall Street will love it. They love macho "˜transformations'. By sheer executive fiat Things Will Change, for sure.
Throw in "˜technology' and it makes Wall Street puff up that little bit more.
The fact that virtually no analyst or serious buyer of stocks has the first idea of what's involved in such a transformation is
irrelevant. They will lap it up.
This is how capitalism works, and it indisputably results in the most efficient allocation of resources possible.
A Dash of Layoffs, a Sprinkling of Talent
These analysts and buyers will assume there will be reductions to employee headcount sooner rather than later, which of course will
make the transformation go faster and beat a quick path to profit.
Hires of top "˜industry experts' who know the magic needed to get all this done, and who will be able to pass on their wisdom without
friction to the eager staff that remain, will make this a sure thing.
In the end, of course, you don't want to come out of this looking too bad, do you?
So how best to minimise any fallout from this endeavour?
Leadership
The first thing you should do is sort out the leadership of this transformation.
Hire in a senior executive specifically for the purpose of making this transformation happen.
Well, taking responsibility for it, at least. This will be useful later when you need a scapegoat for failure.
Ideally it will be someone with a long resume of similar transformational senior roles at different global enterprises.
Don't be concerned with whether those previous roles actually resulted in any lasting change or business success; that's not the
point. The point is that they have a lot of experience with this kind of role, and will know how to be the patsy. Or you can get
someone that has
Dunning-Kruger
syndrome
so they can truly inhabit the role.
The
kind of leader you want.
Make sure this executive is adept at managing his (also hired-in) subordinates in a divide-and-conquer way, so their aims are never
aligned, or multiply-aligned in diverse directions in a 4-dimensional ball of wool.
Incentivise senior leadership to grow their teams rather than fulfil the overall goal of the program (ideally, the overall goal will
never be clearly stated by anyone "" see Strategy, below).
Change your CIO halfway through the transformation. The resulting confusion and political changes of direction will ensure millions
are lost as both teams and leadership chop and change positions.
With a bit of luck, there'll be so little direction that the core business can be unaffected.
Strategy
This second one is easy enough. Don't have a strategy. Then you can chop and change plans as you go without any kind of overall
direction, ensuring (along with the leadership anarchy above) that nothing will ever get done.
Unfortunately, the world is not sympathetic to this reality, so you will have to pretend to have a strategy, at the very least. Make
the core PowerPoint really dense and opaque. Include as many buzzwords as possible "" if enough are included people will assume you
know what you are doing. It helps if the buzzwords directly contradict the content of the strategy documents.
It's also essential that the strategy makes no mention of the "˜customer', or whatever provides
Vandelay's
revenue,
or why the changes proposed make any difference to the business at all. That will help nicely reduce any sense of urgency to the
whole process.
Try to make any stated strategy:
hopelessly optimistic (set ridiculous and arbitrary deadlines)
inflexible from the start (aka "˜my way, or the highway')
Whatever strategy you pretend to pursue, be sure to make it "˜Go big, go early', so you can waste as much money as fast as possible.
Don't waste precious time learning about how change can get done in your context. Remember, this needs to fail once you're gone.
Technology Architecture
First, set up a completely greenfield "˜Transformation Team' separate from your existing staff. Then, task them with solving every
possible problem in your business at once. Throw in some that don't exist yet too, if you like! Force them to coordinate tightly
with every other team and fulfil all their wishes.
Ensure your security and control functions are separated from (and, ideally, in some kind of war with) a Transformation Team that is
siloed as far as possible from the mainstream of the business. This will create the perfect environment for expensive white
elephants to be built that no-one will use.
All this taken together will ensure that the Transformation Team's plans have as little chance of getting to production as possible.
Don't give security and control functions any responsibility or reward for delivery, just reward them for blocking change.
Ignore the "˜decagon of despair'. These things are nothing to do with Transformation, they are just blockers people like to talk
about. The official line is that hiring Talent (see below) will take care of those. It's easy to exploit an organisation's
insecurity about its capabilities to downplay the importance of these
All politics about fake news aside (PLEASE!), I've heard a growing number of reports, sighs
and cries about Fake Agile. It's frustrating when people just don't get it, especially when
they think they do. We can point fingers and vilify those who think differently -- or we can
try to understand why this "us vs them" mindset is splintering the Agile
community....
The primary difference between converged infrastructure (CI) and
hyper-converged infrastructure is that in HCI, both the storage area network and the underlying
storage abstractions are implemented virtually in software (at or via the hypervisor) rather
than physically, in hardware. [ citation needed ]
Because all of the software-defined elements are implemented within the context of the
hypervisor, management of all resources can be federated (shared)
across all instances of a hyper-converged infrastructure. Expected benefits [
edit ]
Hyperconvergence evolves away from discrete, hardware-defined systems that are connected and
packaged together toward a purely software-defined environment where all functional elements
run on commercial, off-the-shelf (COTS) servers, with the convergence of elements enabled by a
hypervisor. [1][2] HCI
infrastructures are usually made up of server systems equipped with Direct-Attached Storage (DAS) .
[3] HCI
includes the ability to plug and play into a data-center pool of like systems. [4][5] All
physical data-center resources reside on a single administrative platform for both hardware and
software layers. [6]
Consolidation of all functional elements at the hypervisor level, together with federated
management, eliminates traditional data-center inefficiencies and reduces the total cost of
ownership (TCO) for data centers. [7][
need
quotation to verify ][8][9]
The potential impact of the hyper-converged infrastructure is that companies will no longer
need to rely on different compute and storage systems, though it is still too early to prove
that it can replace storage arrays in all market segments. [10] It
is likely to further simplify management and increase resource-utilization rates where it does
apply. [11][12][13]
Category : Tools(Practitioner's
Reflections on The DevOps Handbook)The Holy Wars of DevOps
Yet another argument explodes online around the 'true nature of DevOps', around 'what DevOps
really means' or around 'what DevOps is not'. At each conference I attend we talk about DevOps
culture , DevOps mindset and DevOps ways . All confirming one single truth – DevOps is a
myth .
Now don't get me wrong – in no way is this a negation of its validity or importance.
As Y.N.Harrari shows so eloquently in his book 'Sapiens' –
myths were the forming power in the development of humankind. It is in fact our ability to
collectively believe in these non-objective, imagined realities that allows us to collaborate
at large scale, to coordinate our actions, to build pyramids, temples, cities and
roads.
There's a Handbook!
I am writing this while finishing the exceptionally well written "DevOps Handbook" . If you really
want to know what stands behind the all-too-often misinterpreted buzzword – you better
read this cover-to-cover. It presents an almost-no-bullshit deep dive into why, how and what in
DevOps. And it comes from the folks who invented the term and have been busy developing its
main concepts over the last 7 years.
Now notice – I'm only saying you should read the "DevOps Handbook" if you want to
understand what DevOps is about. After finishing it I'm pretty sure you won't have any interest
in participating in petty arguments along the lines of 'is DevOps about automation or not?'.
But I'm not saying you should read the handbook if you want to know how to improve and speed up
your software manufacturing and delivery processes. And neither if you want to optimize your IT
organization for innovation and continuous improvement.
Because the main realization that you, as a smart reader, will arrive at – is just
that there is no such thing as DevOps. DevOps is a myth .
So What's The Story?
It all basically comes down to this: some IT companies achieve better results than others .
Better revenues, higher customer and employee satisfaction, faster value delivery, higher
quality. There's no one-size-fits-all formula, there is no magic bullet – but we can
learn from these high performers and try to apply certain tools and practices in order to
improve the way we work and achieve similar or better results. These tools and processes come
from a myriad of management theories and practices. Moreover – they are constantly
evolving, so we need to always be learning. But at least we have the promise of better life.
That is if we get it all right: the people, the architecture, the processes, the mindset, the
org structure, etc.
So it's not about certain tools, cause the tools will change. And it's not about certain
practices – because we're creative and frameworks come and go. I don't see too many folks
using Kanban boards 10 years from now. (In the same way only the laggards use Gantt charts
today) And then the speakers at the next fancy conference will tell you it's mainly about
culture. And you know what culture is? It's just a story, or rather a collection of stories
that a group of people share. Stories that tell us something about the world and about
ourselves. Stories that have only a very relative connection to the material world. Stories
that can easily be proven as myths by another group of folks who believe them to be
wrong.
But Isn't It True?
Anybody who's studied management
theories knows how the approaches have changed since the beginning of the last century.
From Taylor's scientific management and down to McGregor's X&Y theory they've all had their
followers. Managers who've applied them and swore getting great results thanks to them. And yet
most of these theories have been proven wrong by their successors.
In the same way we see this happening with DevOps and Agile. Agile was all the buzz since
its inception in 2001. Teams were moving to Scrum, then Kanban, now SAFE and LESS. But Agile
didn't deliver on its promise of better life. Or rather – it became so commonplace that
it lost its edge. Without the hype, we now realize it has its downsides. And we now hope that
maybe this new DevOps thing will make us happy.
You may say that the world is changing fast – that's why we now need new approaches!
And I agree – the technology, the globalization, the flow of information – they all
change the stories we live in. But this also means that whatever is working for someone else
today won't probably work for you tomorrow – because the world will change yet again.
Which means that the DevOps Handbook – while a great overview and historical document
and a source of inspiration – should not be taken as a guide to action. It's just another
step towards establishing the DevOps myth.
And that takes us back to where we started – myths and stories aren't bad in
themselves. They help us collaborate by providing a common semantic system and shared goals.
But they only work while we believe in them and until a new myth comes around – one
powerful enough to grab our attention.
Your Own DevOps Story
So if we agree that DevOps is just another myth, what are we left with? What do we at
Otomato and other DevOps consultants and
vendors have to sell? Well, it's the same thing we've been building even before the DevOps
buzz: effective software delivery and IT management. Based on tools and processes, automation
and effective communication. Relying on common sense and on being experts in whatever myth is
currently believed to be true.
As I keep saying – culture is a story you tell. And we make sure to be experts in both
the storytelling and the actual tooling and architecture. If you're currently looking at
creating a DevOps transformation or simply want to optimize your software delivery – give
us a call. We'll help to build your authentic DevOps story, to train your staff and to
architect your pipeline based on practice, skills and your organization's actual needs. Not
based on myths that other people tell.
"... Scrum is dead, long live Screm! We need to implement it immediately. We must innovate and stay ahead of the curve! level 7 ..."
"... First you scream, then you ahh. Now you can screm ..."
"... Are you saying quantum synergy coupled with block chain neutral intelligence can not be used to expedite artificial intelligence amalgamation into that will metaphor into cucumber obsession? ..."
We recently implemented DevOps practices, Scrum, and sprints have become the
norm... I swear to god we spend 60% of our time planning our sprints, and 40% of the time doing the work, and
management wonders why our true productivity has fallen through the floor...
level 5
Are you saying quantum synergy coupled with block chain neutral intelligence can not be
used to expedite artificial intelligence amalgamation into that will metaphor into cucumber obsession?
What is it with these people? Someone brought me an "urgent" request (of course there
wasn't a ticket), so I said no worries, I'll help you out. Just open a ticket for me so we can track the work and document
the conversation. We got that all knocked out and everyone was happy.
So a day or two later, I suddenly get an instant message for yet another "urgent"
issue. ... Ok ... Open a ticket, and I'll get it assigned to one of my team members to take a look.
And a couple days later ... he's back and I'm being asked for help troubleshooting an
application that we don't own. At least there's a ticket and an email thread... but wtf man.
What the heck man?
This is like when you get a free drink or dessert from your waiter. Don't keep coming
back and asking for more free pie. You know damn well you're supposed to pay for pie. Be cool. I'll help you out when
you're really in a tight spot, but the more you cry "urgent", the less I care about your issues.
Hey
r/Sysadmin
, please don't suffer in silence. I know the job can be very difficult at times, especially with competing
objectives, tight (or impossible) deadlines, bad bosses and needy end users, but please - always remember that there are
ways to manage that stress. Speaking to friends and family regularly to vent, getting a therapist, or taking time off.
Yes, you do have the ability to take personal leave/medical leave if its that bad. No,
it doesn't matter what your colleagues or boss will think..and no, you are not a quitter, weak, or a loser if you take
time for yourself - to heal mentally, physically or emotionally.
EDIT: Holy Cow! Thanks for the silver and platinum kind strangers. All i wanted to do
was to get some more awareness on this subject, and create a reminder that we all deserve happiness and peace of mind. A
reminder that hopefully sticks with you for the days and weeks to come.
We, sysadmins, have times when we don't really have nothing to do but maintenance. BUT,
there are times when it seems like chaos comes out of nowhere. When I have a lot of tasks to do, I tend to get slower. The
more tasks I have pending, the slower I become. I cannot avoid to start thinking about 3 or 4 different problems at the
same time, and I can't focus! I only have 2 years of experiences as sysadmin.
Have you ever wondered if your whole career will be related to IT stuff? I have, since
my early childhood. It was more than 30 years ago - in the marvelous world of an 8-bit era. After writing my first code
(10 PRINT "
my_name
" : 20 GOTO 10) I exactly knew what I wanted to do in the
future. Now, after spending 18 years in this industry, which is half of my age, I'm not so sure about it.
I had plenty of time to do almost everything. I was writing software for over 100K
users and I was covered in dust while drilling holes for ethernet cables in houses of our customers. I was a main network
administrator for a small ISP and systems administrator for a large telecom operator. I made few websites and I was
managing a team of technical support specialists. I was teaching people - on individual courses on how to use Linux and
made some trainings for admins on how to troubleshoot multicast transmissions in their own networks. I was active in some
Open Source communities, including running forums about one of Linux distributions (the forum was quite popular in my
country) and I was punching endless Ctrl+C/Ctrl+V combos from Stack Overflow. I even fixed my aunt's computer!
And suddenly I realised that I don't want to do this any more. I've completely burnt
out. It was like a snap of a finger.
During many years I've collected a wide range of skills that are (or will be) obsolete.
I don't want to spend rest of my life maintaining a legacy code written in C or PHP or learning a new language which is
currently on top and forcing myself to write in a coding style I don't really like. That's not all... If you think you'll
enjoy setting up vlans on countless switches, you're probably wrong. If you think that managing clusters of virtual
machines is an endless fun, you'll probably be disappointed. If you love the smell of a brand new blade server and the
"click" sound it makes when you mount it into the rack, you'll probably get fed up with it. Sooner or later.
But there's a good side of having those skills. With skills come experience, knowledge
and good premonition. And these features don't get old. Remember that!
My employer offered me a position of a project manager and I eagerly agreed to it. It
means that I'm leaving the world of "hardcore IT" I'll be doing some other, less crazy stuff. I'm logging out of my
console and I'll run Excel. But I'll keep all good memories from all those years. I'd like to thank all of you for doing
what you're doing, because it's really amazing. Good luck! The world lies in your hands!
254 comments
450
Posted by
u/remrinds
1 day ago
General Discussion
UPDATE: So our cloud exchange server was down for 17 hours on friday
my original post got deleted because i behaved wrongly and posted some slurs. I
apologise for that.
anyway, so, my companie is using Office365 ProPlus and we migrated our on premise
exchange server to cloud a while ago, and on friday last week, all of our user (1000 or so) could not access their
exchange box, we are a TV broadcasting station so you can only imagine the damage when we could not use our mailing
system.
initially, we opened a ticket with microsoft and they just kept us on hold for 12 hours
(we are in japan so they had to communicate with US and etc which took time), and then they told us its our network infra
thats wrong when we kept telling them its not. we asked them to check their envrionment at least once which they did not
until 12 hours later.
in the end, it was their exchange server that was the problem, i will copy and paste
the whole incident report below
Title: Can't access Exchange
User Impact: Users are unable to access the Exchange Online service.
Current status: We've determined that a recent sync between Exchange Online and Azure Active Directory (AAD) inadvertently resulted in access issues with the Exchange Online service. We've restored the affected environment and updated the Global Location Service (GLS) records, which we believe has resolved the issue. We're awaiting confirmation from your representatives that this issue is resolved.
Scope of impact: Your organization is affected by this event, and this issue impacts all users.
Start time: Friday, October 4, 2019, 4:51 AM
Root cause: A recent Service Intelligence (SI) move inadvertently resulted in access issues with the Exchange Online service.
they wont explain further than what they posted on the incident page but if anyone here
is good with microsofts cloud envrionment, can anyone tell me what was the root cause of this? from what i can gather, the
AAD and exchange server couldnt sync but they wont tell us what the actual problem is, what the hell is Service
intelligence and how does it fix our exchange server when they updated the global location service?
any insight on these report would be more than appreciated
I rolled out this patch this weekend to my test group and it appears that some of the
workstations this was applied to are having print spooler issues.
I hadn't found a use case for containers in my environment so I had put off learning
Docker for a while. But, I'm writing a rails app to simplify/automate some of our administrative tasks. Setting up my
different dev and test environments was definitely non trivial, and I plan on onboarding another person or 2 so they can
help me out and add it to their resume.
I installed Docker desktop on my Mac, wrote 2 files essentially copied from Docker's
website, built it, then ran it. It took a total of 10 minutes to go from zero Docker to fully configured and running. It's
really that easy to start using it. So, now I've decided to set up Kubernetes at work this week and see what I can find to
do with it.
Let me tell you about the Monday from hell I encountered yesterday.
I work for
xyz corp
which is an MSP for IT
services. One of the companies we support, we'll call them
abc corp
.
I come in to work Monday morning and look at my alerts report from the previous day and
find that all of the servers (about 12+) at
abc corp
are showing offline. Manager
asks me to go on site to investigate, it's around the corner so
nbd
.
I get to the client, head over to the server room and open up the KVM console. I switch
inputs and see no issues with the ESX hosts. I then switch over to the (for some reason, physical) vCenter server and find
this lovely little message:
Now, I've never seen this before and it looks sus but just in case, i reboot the
machine -
same message
. Do a
quick google search and found that the server was hit with an MBR level ransomware encryption. I then quickly switched
over to the server that manages the backups and found that
it's also encrypted -
f*ck.
At this point, I call
Mr. CEO
and account
manager to come on site. While waiting for them, I found that the SANs had also been logged in to and had all data deleted
and snapshots deleted off the datastores and the EQL volume was also encrypted -
PERFECT!
At this point, I'm basically freaking out.
ABC Corp
is owned by a parent company who apparently also got hit however we don't manage them *phew*.
Our only saving grace at this point is the offsite backups. I log in to the server and
wouldn't ya know it, I see this lovely message:
Last replication time: 6/20/2019 13:00:01
BackupsTech
had a script that ran to report on
replication status daily and the reports were showing that they were up to date. Obviously, they weren't so at this point
we're basically f*cked.
We did eventually find out this originated from
parentcompany
and that the accounts used were from the old IT Manager that recently left a few weeks ago.
Unfortunately, they never disabled the accounts in either domain and the account used was a domain admin account.
We're currently going through and attempting to undelete the VMFS data to regain access
to the VM files. If anyone has any suggestions on this, feel free to let me know.
Sick and tired of listening to these so called architects and full stack developers who
watch bunch of videos on YouTube and Pluralsight, find articles online. They go around workplace throwing words like
containers, devops, NoOps, azure, infrastructure as code, serverless, etc, they don't understand half of the stuff. I do some
of the devops tasks in our company, I understand what it takes to implement and manage these technologies. Every meeting is
infested with these A holes.
1.0k comments
91% Upvoted
What are your thoughts? Log in or Sign up
log
in
sign
up
Sort by
level 1
Your best defense against these is to come up with non-sarcastic and
quality questions to ask these people during the meeting, and watch them not have a clue how to answer them.
For example, a friend of mine worked at a smallish company, some manager
really wanted to move more of their stuff into Azure including AD and Exchange environment. But they had
common problems with their internet connection due to limited bandwidth and them not wanting to spend more. So
during a meeting my friend asked a question something like this:
"You said on this slide that moving the AD environment and Exchange
environment to Azure will save us money. Did you take into account that we will need to increase our internet
speed by a factor of at least 4 in order to accommodate the increase in traffic going out to the Azure cloud?
"
Of course, they hadn't. So the CEO asked my friend if he had the numbers,
which he had already done his homework, and it was a significant increase in cost every month and taking into
account the cost for Azure and the increase in bandwidth wiped away the manager's savings.
I know this won't work for everyone. Sometimes there is real savings in
moving things to the cloud. But often times there really isn't. Calling the uneducated people out on what they
see as facts can be rewarding.
level 2
my previous boss was that kind of a guy. he waited till other people were
done throwing their weight around in a meeting and then calmly and politely dismantled them with facts.
no amount of corporate pressuring or bitching could ever stand up to that.
level 3
This is my approach. I don't yell or raise my voice, I just wait. Then I
start asking questions that they generally cannot answer and slowly take them apart. I don't have to be loud
to get my point across.
level 4
This tactic is called "the box game". Just continuously ask them logical
questions that can't be answered with their stupidity. (Box them in), let them be their own argument against
themselves.
Not to mention downtime. We have two ISPs in our area. Most of our clients
have both in order to keep a fail-over. However, depending on where the client is located, one ISP is fast but
goes down every time it rains and the other is solid but slow. Now our only AzureAD customers are those who
have so many remote workers that they throw their hands up and deal with the outages as they come. Maybe this
works in Europe or South Korea, but this here 'Murica got too many internet holes.
level 3
Yup. If you're in a city with fiber it can probably work. If you have even
one remote site, and all they have is DSL (or worse, satellite, as a few offices I once supported were
literally
in the woods when I worked for a timber company) then even
Citrix becomes out of the question.
Definitely this, if you know your stuff and can wrap some numbers around it
you can regain control of the conversation.
I use my dad as a prime example, he was an electrical engineer for ~40
years, ended up just below board level by the time he retired. He sat in on a product demo once, the kit they
were showing off would speed up jointing cable in the road by 30 minutes per joint. My dad asked 3 questions
and shut them down:
"how much will it cost to supply all our jointing teams?" £14million
"how many joints do our teams complete each day?" (this they couldn't
answer so my dad helped them out) 3
"So are we going to tell the jointers that they get an extra hour and a
half hour lunch break or a pay cut?"
Room full of executives that had been getting quite excited at this awesome
new investment were suddenly much more interested in showing these guys the door and getting to their next
meeting.
level 3
I think they did 3 as that was the workload so being able to do 4 isn't
relevant if there isn't a fourth to do.
That's what I get out of his story anyway.
And also if it was going to be 14 million in costs to equip everyone, the
savings have to be there. If adding 1 unit of productivity per day didn't save 14 million in a year or two,
it's not really worth it.
level 5
That was basically what I figured was the missing piece - the logistical
inability to process 4 units.
As far as the RoI, I had to assume that the number of teams involved and
their operational costs had already factored into whether or not 14m was even a price anyone could consider
paying. In other words, by virtue of the meeting even happening I figured that the initial costing had not
already been laughed out of the room, but perhaps that's a bit too much of an assumption to make.
level 6
In my limited experience they slow roll actually pricing anything. Of
course physical equipment pricing might be more straightforward, which would explain why his dad got a number
instead of a discussion of licensing options.
level 5
And also if it was going to be 14 million in costs to equip everyone,
the savings have to be there. If adding 1 unit of productivity per day didn't save 14 million in a year or
two, it's not really worth it.
That entirely depends. If you have 10 people producing 3 joints a day, with
this new kit you could reduce your headcount by 2 and still produce the same, or take on additional workload
and increase your production. Not to mention that you don't need to equip everyone with these kits either, you
could save them for the projects which needed more daily production thus saving money on the kits and
increasing production on an as needed basis.
They story is missing a lot of specifics and while it sounds great, I'm
quite certain there was likely a business case to be made.
He's saying they get 3 done in a day, and the product would save them 30
minutes per joint. That's an hour and a half saved per day, not even enough time to finish a fourth joint,
hence the "so do i just give my workers an extra 30 min on lunch"? He just worded it all really weird.
level 4
Your ignoring travel time in there, there are also factors to do with
outside contractors carrying out the digging and reinstatement works and getting sign off to actually dig the
hole in the road in the first place.
There is also the possibility I'm remembering the time wrong... it's been a
while!
level 5
Travel time actually works in my favour. If it takes more time to go from
job to job, then the impact of the enhancement is magnified because the total time per job shrinks.
level 4
That seems like a poor example. Why would you ignore efficiency
improvements just to give your workers something to do? Why not find another task for them or figure out a way
to consolidate the teams some? We fight this same mentality on our manufacturing floors, the idea that if we
automate a process someone will lose their job or we wont have enough work for everyone. Its never the case.
However, because of the automation improvements we have done in the last 10 years, we are doing 2.5x the
output with only a 10% increase in the total number of laborers.
So maybe in your example for a time they would have nothing for these
people to do. Thats a management problem. Have them come back and sweep the floor or wash their trucks.
Eventually you will be at a point where there is more work, and that added efficiency will save you from
needing to put more crews on the road.
Calling the uneducated people out on what they see as facts can be
rewarding.
I wouldnt call them all uneducated. I think what they are is basically
brainwashed. They constantly hear from the sales teams of vendors like Microsoft pitching them the idea of
moving everything to Azure.
They do not hear the cons. They do not hear from the folks who know their
environment and would know if something is a good fit. At least, they dont hear it enough, and they never see
it first hand.
Now, I do think this is their fault... they need to seek out that info
more, weigh things critically, and listen to what's going on with their teams more. Isolation from the team is
their own doing.
After long enough standing on the edge and only hearing "jump!!", something
stupid happens.
level 3
There's little downside to containers by themselves. They're just a method
of sandboxing processes and packaging a filesystem as a distributable image. From a performance perspective
the impact is near negligible (unless you're doing some truly intensive disk I/O).
What can be problematic is taking a process that was designed to run on
exactly n dedicated servers and converting it to a modern 2 to n autoscaling deployment that shares hosting
with other apps on n a platform like Kubernetes. It's a significant challenge that requires a lot of expertise
and maintenance, so there needs to be a clear business advantage to justify hiring at least one additional
full time engineer to deal with it.
level 5
Containers are great for stateless stuff. So your webservers/application
servers can be shoved into containers. Think of containers as being the modern version of statically linked
binaries or fat applications. Static binaries have the problem that any security vulnerability requires a full
rebuild of the application, and that problem is escalated in containers (where you might not even know that a
broken library exists)
If you are using the typical business application, you need one or more
storage components for data which needs to be available, possibly changed and access controlled.
Containers are a bad fit for stateful databases, or any stateful component,
really.
Containers also enable microservices, which are great ideas at a certain
organisation size (if you aren't sure you need microservices, just use a simple monolithic architecture). The
problem with microservices is that you replace complexity in your code with complexity in the communications
between the various components, and that is harder to see and debug.
level 5
Containers are fine for stateful services -- you can manage persistence at
the storage layer the same way you would have to manage it if you were running the process directly on the
host.
Backing up containers can be a pain, so you don't want to use them for
valuable data unless the data is stored elsewhere, like a database or even a file server.
For spinning up stateless applications to take workload behind a load
balancer, containers are excellent.
The problem is that there is an overwhelming din from vendors. Everybody
and their brother, sister, mother, uncle, cousin, dog, cat, and gerbil is trying to sell you some
pay-by-the-month cloud "solution".
The reason is that the cloud forces people into monthly payments, which is
a guarenteed income for companies, but costs a lot more in the long run, and if something happens and one
can't make the payments, business is halted, ensuring that bankruptcies hit hard and fast. Even with the
mainframe, a company could limp along without support for a few quarters until they could get cash flow
enough.
If we have a serious economic downturn, the fact that businesses will be
completely shuttered if they can't afford their AWS bill just means fewer companies can limp along when the
economy is bad, which will intensify a downturn.
Our company viewed the move to Azure less as a cost savings measure and
more of a move towards agility and "right now" sizing of our infrastructure.
Your point is very accurate, as an example our location is wholly incapable
of moving moving much to the cloud due to half of us being connnected via satellite network and the other half
being bent over the barrel by the only ISP in town.
level 2
I'm sorry, but I find management these days around tech wholly inadequate.
The idea that you can get an MBA and manage shit you have no idea about is absurd and just wastes everyone
elses time for them to constantly ELI5 so manager can do their job effectively.
level 2
Calling the uneducated people out on what they see as facts can be
rewarding
Aaand political suicide in a corporate environment. Instead I use the
following:
"I love this idea! We've actually been looking into a similar solution
however we weren't able to overcome some insurmountable cost sinkholes (remember: nothing is impossible; just
expensive). Will this idea require an increase in internet speed to account for the traffic going to the azure
cloud?"
level 3
This exactly, you can't compromise your own argument to the extent that
it's easy to shoot down in the name of giving your managers a way of saving face, and if you deliver the facts
after that sugar coating it will look even worse, as it will be interpreted as you setting them up to look
like idiots. Being frank, objective and non-blaming is always the best route.
level 4
I think it's quite an american thing to do it so enthusiastically; to hurt
no one, but the result is so condescending. It must be annoying to walk on egg shells to survive a day in the
office.
And this is not a rant against soft skills in IT, at all.
level 4
We work with Americans and they're always so positive, it's kinda annoying.
They enthusiastically say "This is very interesting" when in reality it sucks and they know it.
Another less professional example, one of my (non-american) co-workers
always wants to go out for coffee (while we have free and better coffee in the office), and the American
coworker is always nice like "I'll go with you but I'm not having any" and I just straight up reply "No. I'm
not paying for shitty coffee, I'll make a brew in the office". And that's that. Sometimes telling it as it is
makes the whole conversation much shorter :D
level 5
Maybe the american would appreciate the break and a chance to spend some
time with a coworker away from screens, but also doesn't want the shit coffee?
Yes. "Go out for coffee" is code for "leave the office so we can complain
about management/company/customer/Karen". Some conversations shouldn't happen inside the building.
level 7
Inferior American - please compute this - the sole intention was
consumption of coffee. Therefore due to existence of coffee in office, trip to coffee shop is not sane.
Resistance to my reasoning is futile.
level 7
Coffee is not the sole intention. If that was the case, then there wouldn't
have been an invitation.
Another intention would be the social aspect - which americans are known to
love, especially favoring it over doing any actual work in the context of a typical 9-5 corporate job.
I've effectively resisted your reasoning, therefore [insert ad hominem
insult about inferior logic].
There is still value to tact in your delivery, but, don't slit your own
throat. Remind them they hired you for a reason.
"I appreciate the ideas being raised here by management for cost savings
and it certainly merits a discussion point. As a business, we have many challenges and one of them includes
controlling costs. Bridging these conversations with subject matter experts and management this early can
really help fill out the picture. I'd like to try to provide some of that value to the conversation here"
level 3
If it's political suicide to point out potential downsides, then you need
to work somewhere else.
Especially since your response, in my experience, won't get anything done
(people will either just say "no, that's not an issue", or they'll find your tone really condescending) and
will just piss people off.
I worked with someone like that would would always be super passive
aggressive in how she brought things up and it pissed me off to no end, because it felt less like bringing up
potential issues and more like being belittling.
level 4
Feels good to get to a point in my career where I can call people our
whenever the hell I feel like it. I've got recruiters banging down my door, I couldn't swing a stick without
hitting a job offer or three.
Of course you I'm not suggesting problem be combative for no reason and to
be tactful about it but if you don't call out fools like that then you're being negligent in your duties.
level 4
"I am resolute in my ability to elevate this collaborative,
forward-thinking team into the revenue powerhouse that I believe it can be. We will transition into a DevOps
team specialising in migrating our existing infrastructure entirely to code and go completely serverless!" -
CFO that outsources IT
level 2
We recently implemented DevOps practices, Scrum, and sprints have become
the norm... I swear to god we spend 60% of our time planning our sprints, and 40% of the time doing the work,
and management wonders why our true productivity has fallen through the floor...
level 5
If you spend three whole days out of every five in planning meetings, this
is a problem with your meeting planners, not with screm. If these people stay in charge, you'll be stuck in
planning hell no matter what framework or buzzwords they try to fling around.
level 6
My last company used screm. No 3 day meeting events, 20min of loose
direction and we were off to the races. We all came back with parts to different projects. SYNERGY.
level 7
Are you saying quantum synergy coupled with block chain neutral
intelligence can not be used to expedite artificial intelligence amalgamation into that will metaphor into
cucumber obsession?
What is it with these people? Someone brought me an "urgent" request (of course there
wasn't a ticket), so I said no worries, I'll help you out. Just open a ticket for me so we can track the work and document
the conversation. We got that all knocked out and everyone was happy.
So a day or two later, I suddenly get an instant message for yet another "urgent"
issue. ... Ok ... Open a ticket, and I'll get it assigned to one of my team members to take a look.
And a couple days later ... he's back and I'm being asked for help troubleshooting an
application that we don't own. At least there's a ticket and an email thread... but wtf man.
What the heck man?
This is like when you get a free drink or dessert from your waiter. Don't keep coming
back and asking for more free pie. You know damn well you're supposed to pay for pie. Be cool. I'll help you out when
you're really in a tight spot, but the more you cry "urgent", the less I care about your issues.
Hey
r/Sysadmin
, please don't suffer in silence. I know the job can be very difficult at times, especially with competing
objectives, tight (or impossible) deadlines, bad bosses and needy end users, but please - always remember that there are
ways to manage that stress. Speaking to friends and family regularly to vent, getting a therapist, or taking time off.
Yes, you do have the ability to take personal leave/medical leave if its that bad. No,
it doesn't matter what your colleagues or boss will think..and no, you are not a quitter, weak, or a loser if you take
time for yourself - to heal mentally, physically or emotionally.
EDIT: Holy Cow! Thanks for the silver and platinum kind strangers. All i wanted to do
was to get some more awareness on this subject, and create a reminder that we all deserve happiness and peace of mind. A
reminder that hopefully sticks with you for the days and weeks to come.
We, sysadmins, have times when we don't really have nothing to do but maintenance. BUT,
there are times when it seems like chaos comes out of nowhere. When I have a lot of tasks to do, I tend to get slower. The
more tasks I have pending, the slower I become. I cannot avoid to start thinking about 3 or 4 different problems at the
same time, and I can't focus! I only have 2 years of experiences as sysadmin.
Have you ever wondered if your whole career will be related to IT stuff? I have, since
my early childhood. It was more than 30 years ago - in the marvelous world of an 8-bit era. After writing my first code
(10 PRINT "
my_name
" : 20 GOTO 10) I exactly knew what I wanted to do in the
future. Now, after spending 18 years in this industry, which is half of my age, I'm not so sure about it.
I had plenty of time to do almost everything. I was writing software for over 100K
users and I was covered in dust while drilling holes for ethernet cables in houses of our customers. I was a main network
administrator for a small ISP and systems administrator for a large telecom operator. I made few websites and I was
managing a team of technical support specialists. I was teaching people - on individual courses on how to use Linux and
made some trainings for admins on how to troubleshoot multicast transmissions in their own networks. I was active in some
Open Source communities, including running forums about one of Linux distributions (the forum was quite popular in my
country) and I was punching endless Ctrl+C/Ctrl+V combos from Stack Overflow. I even fixed my aunt's computer!
And suddenly I realised that I don't want to do this any more. I've completely burnt
out. It was like a snap of a finger.
During many years I've collected a wide range of skills that are (or will be) obsolete.
I don't want to spend rest of my life maintaining a legacy code written in C or PHP or learning a new language which is
currently on top and forcing myself to write in a coding style I don't really like. That's not all... If you think you'll
enjoy setting up vlans on countless switches, you're probably wrong. If you think that managing clusters of virtual
machines is an endless fun, you'll probably be disappointed. If you love the smell of a brand new blade server and the
"click" sound it makes when you mount it into the rack, you'll probably get fed up with it. Sooner or later.
But there's a good side of having those skills. With skills come experience, knowledge
and good premonition. And these features don't get old. Remember that!
My employer offered me a position of a project manager and I eagerly agreed to it. It
means that I'm leaving the world of "hardcore IT" I'll be doing some other, less crazy stuff. I'm logging out of my
console and I'll run Excel. But I'll keep all good memories from all those years. I'd like to thank all of you for doing
what you're doing, because it's really amazing. Good luck! The world lies in your hands!
254 comments
450
Posted by
u/remrinds
1 day ago
General Discussion
UPDATE: So our cloud exchange server was down for 17 hours on friday
my original post got deleted because i behaved wrongly and posted some slurs. I
apologise for that.
anyway, so, my companie is using Office365 ProPlus and we migrated our on premise
exchange server to cloud a while ago, and on friday last week, all of our user (1000 or so) could not access their
exchange box, we are a TV broadcasting station so you can only imagine the damage when we could not use our mailing
system.
initially, we opened a ticket with microsoft and they just kept us on hold for 12 hours
(we are in japan so they had to communicate with US and etc which took time), and then they told us its our network infra
thats wrong when we kept telling them its not. we asked them to check their envrionment at least once which they did not
until 12 hours later.
in the end, it was their exchange server that was the problem, i will copy and paste
the whole incident report below
Title: Can't access Exchange
User Impact: Users are unable to access the Exchange Online service.
Current status: We've determined that a recent sync between Exchange Online and Azure Active Directory (AAD) inadvertently resulted in access issues with the Exchange Online service. We've restored the affected environment and updated the Global Location Service (GLS) records, which we believe has resolved the issue. We're awaiting confirmation from your representatives that this issue is resolved.
Scope of impact: Your organization is affected by this event, and this issue impacts all users.
Start time: Friday, October 4, 2019, 4:51 AM
Root cause: A recent Service Intelligence (SI) move inadvertently resulted in access issues with the Exchange Online service.
they wont explain further than what they posted on the incident page but if anyone here
is good with microsofts cloud envrionment, can anyone tell me what was the root cause of this? from what i can gather, the
AAD and exchange server couldnt sync but they wont tell us what the actual problem is, what the hell is Service
intelligence and how does it fix our exchange server when they updated the global location service?
any insight on these report would be more than appreciated
I rolled out this patch this weekend to my test group and it appears that some of the
workstations this was applied to are having print spooler issues.
I hadn't found a use case for containers in my environment so I had put off learning
Docker for a while. But, I'm writing a rails app to simplify/automate some of our administrative tasks. Setting up my
different dev and test environments was definitely non trivial, and I plan on onboarding another person or 2 so they can
help me out and add it to their resume.
I installed Docker desktop on my Mac, wrote 2 files essentially copied from Docker's
website, built it, then ran it. It took a total of 10 minutes to go from zero Docker to fully configured and running. It's
really that easy to start using it. So, now I've decided to set up Kubernetes at work this week and see what I can find to
do with it.
Let me tell you about the Monday from hell I encountered yesterday.
I work for
xyz corp
which is an MSP for IT
services. One of the companies we support, we'll call them
abc corp
.
I come in to work Monday morning and look at my alerts report from the previous day and
find that all of the servers (about 12+) at
abc corp
are showing offline. Manager
asks me to go on site to investigate, it's around the corner so
nbd
.
I get to the client, head over to the server room and open up the KVM console. I switch
inputs and see no issues with the ESX hosts. I then switch over to the (for some reason, physical) vCenter server and find
this lovely little message:
Now, I've never seen this before and it looks sus but just in case, i reboot the
machine -
same message
. Do a
quick google search and found that the server was hit with an MBR level ransomware encryption. I then quickly switched
over to the server that manages the backups and found that
it's also encrypted -
f*ck.
At this point, I call
Mr. CEO
and account
manager to come on site. While waiting for them, I found that the SANs had also been logged in to and had all data deleted
and snapshots deleted off the datastores and the EQL volume was also encrypted -
PERFECT!
At this point, I'm basically freaking out.
ABC Corp
is owned by a parent company who apparently also got hit however we don't manage them *phew*.
Our only saving grace at this point is the offsite backups. I log in to the server and
wouldn't ya know it, I see this lovely message:
Last replication time: 6/20/2019 13:00:01
BackupsTech
had a script that ran to report on
replication status daily and the reports were showing that they were up to date. Obviously, they weren't so at this point
we're basically f*cked.
We did eventually find out this originated from
parentcompany
and that the accounts used were from the old IT Manager that recently left a few weeks ago.
Unfortunately, they never disabled the accounts in either domain and the account used was a domain admin account.
We're currently going through and attempting to undelete the VMFS data to regain access
to the VM files. If anyone has any suggestions on this, feel free to let me know.
Sick and tired of listening to these so called architects and full stack developers who
watch bunch of videos on YouTube and Pluralsight, find articles online. They go around workplace throwing words like
containers, devops, NoOps, azure, infrastructure as code, serverless, etc, they don't understand half of the stuff. I do some
of the devops tasks in our company, I understand what it takes to implement and manage these technologies. Every meeting is
infested with these A holes.
1.0k comments
91% Upvoted
What are your thoughts? Log in or Sign up
log
in
sign
up
Sort by
level 1
Your best defense against these is to come up with non-sarcastic and
quality questions to ask these people during the meeting, and watch them not have a clue how to answer them.
For example, a friend of mine worked at a smallish company, some manager
really wanted to move more of their stuff into Azure including AD and Exchange environment. But they had
common problems with their internet connection due to limited bandwidth and them not wanting to spend more. So
during a meeting my friend asked a question something like this:
"You said on this slide that moving the AD environment and Exchange
environment to Azure will save us money. Did you take into account that we will need to increase our internet
speed by a factor of at least 4 in order to accommodate the increase in traffic going out to the Azure cloud?
"
Of course, they hadn't. So the CEO asked my friend if he had the numbers,
which he had already done his homework, and it was a significant increase in cost every month and taking into
account the cost for Azure and the increase in bandwidth wiped away the manager's savings.
I know this won't work for everyone. Sometimes there is real savings in
moving things to the cloud. But often times there really isn't. Calling the uneducated people out on what they
see as facts can be rewarding.
level 2
my previous boss was that kind of a guy. he waited till other people were
done throwing their weight around in a meeting and then calmly and politely dismantled them with facts.
no amount of corporate pressuring or bitching could ever stand up to that.
level 3
This is my approach. I don't yell or raise my voice, I just wait. Then I
start asking questions that they generally cannot answer and slowly take them apart. I don't have to be loud
to get my point across.
level 4
This tactic is called "the box game". Just continuously ask them logical
questions that can't be answered with their stupidity. (Box them in), let them be their own argument against
themselves.
Not to mention downtime. We have two ISPs in our area. Most of our clients
have both in order to keep a fail-over. However, depending on where the client is located, one ISP is fast but
goes down every time it rains and the other is solid but slow. Now our only AzureAD customers are those who
have so many remote workers that they throw their hands up and deal with the outages as they come. Maybe this
works in Europe or South Korea, but this here 'Murica got too many internet holes.
level 3
Yup. If you're in a city with fiber it can probably work. If you have even
one remote site, and all they have is DSL (or worse, satellite, as a few offices I once supported were
literally
in the woods when I worked for a timber company) then even
Citrix becomes out of the question.
Definitely this, if you know your stuff and can wrap some numbers around it
you can regain control of the conversation.
I use my dad as a prime example, he was an electrical engineer for ~40
years, ended up just below board level by the time he retired. He sat in on a product demo once, the kit they
were showing off would speed up jointing cable in the road by 30 minutes per joint. My dad asked 3 questions
and shut them down:
"how much will it cost to supply all our jointing teams?" £14million
"how many joints do our teams complete each day?" (this they couldn't
answer so my dad helped them out) 3
"So are we going to tell the jointers that they get an extra hour and a
half hour lunch break or a pay cut?"
Room full of executives that had been getting quite excited at this awesome
new investment were suddenly much more interested in showing these guys the door and getting to their next
meeting.
level 3
I think they did 3 as that was the workload so being able to do 4 isn't
relevant if there isn't a fourth to do.
That's what I get out of his story anyway.
And also if it was going to be 14 million in costs to equip everyone, the
savings have to be there. If adding 1 unit of productivity per day didn't save 14 million in a year or two,
it's not really worth it.
level 5
That was basically what I figured was the missing piece - the logistical
inability to process 4 units.
As far as the RoI, I had to assume that the number of teams involved and
their operational costs had already factored into whether or not 14m was even a price anyone could consider
paying. In other words, by virtue of the meeting even happening I figured that the initial costing had not
already been laughed out of the room, but perhaps that's a bit too much of an assumption to make.
level 6
In my limited experience they slow roll actually pricing anything. Of
course physical equipment pricing might be more straightforward, which would explain why his dad got a number
instead of a discussion of licensing options.
level 5
And also if it was going to be 14 million in costs to equip everyone,
the savings have to be there. If adding 1 unit of productivity per day didn't save 14 million in a year or
two, it's not really worth it.
That entirely depends. If you have 10 people producing 3 joints a day, with
this new kit you could reduce your headcount by 2 and still produce the same, or take on additional workload
and increase your production. Not to mention that you don't need to equip everyone with these kits either, you
could save them for the projects which needed more daily production thus saving money on the kits and
increasing production on an as needed basis.
They story is missing a lot of specifics and while it sounds great, I'm
quite certain there was likely a business case to be made.
He's saying they get 3 done in a day, and the product would save them 30
minutes per joint. That's an hour and a half saved per day, not even enough time to finish a fourth joint,
hence the "so do i just give my workers an extra 30 min on lunch"? He just worded it all really weird.
level 4
Your ignoring travel time in there, there are also factors to do with
outside contractors carrying out the digging and reinstatement works and getting sign off to actually dig the
hole in the road in the first place.
There is also the possibility I'm remembering the time wrong... it's been a
while!
level 5
Travel time actually works in my favour. If it takes more time to go from
job to job, then the impact of the enhancement is magnified because the total time per job shrinks.
level 4
That seems like a poor example. Why would you ignore efficiency
improvements just to give your workers something to do? Why not find another task for them or figure out a way
to consolidate the teams some? We fight this same mentality on our manufacturing floors, the idea that if we
automate a process someone will lose their job or we wont have enough work for everyone. Its never the case.
However, because of the automation improvements we have done in the last 10 years, we are doing 2.5x the
output with only a 10% increase in the total number of laborers.
So maybe in your example for a time they would have nothing for these
people to do. Thats a management problem. Have them come back and sweep the floor or wash their trucks.
Eventually you will be at a point where there is more work, and that added efficiency will save you from
needing to put more crews on the road.
Calling the uneducated people out on what they see as facts can be
rewarding.
I wouldnt call them all uneducated. I think what they are is basically
brainwashed. They constantly hear from the sales teams of vendors like Microsoft pitching them the idea of
moving everything to Azure.
They do not hear the cons. They do not hear from the folks who know their
environment and would know if something is a good fit. At least, they dont hear it enough, and they never see
it first hand.
Now, I do think this is their fault... they need to seek out that info
more, weigh things critically, and listen to what's going on with their teams more. Isolation from the team is
their own doing.
After long enough standing on the edge and only hearing "jump!!", something
stupid happens.
level 3
There's little downside to containers by themselves. They're just a method
of sandboxing processes and packaging a filesystem as a distributable image. From a performance perspective
the impact is near negligible (unless you're doing some truly intensive disk I/O).
What can be problematic is taking a process that was designed to run on
exactly n dedicated servers and converting it to a modern 2 to n autoscaling deployment that shares hosting
with other apps on n a platform like Kubernetes. It's a significant challenge that requires a lot of expertise
and maintenance, so there needs to be a clear business advantage to justify hiring at least one additional
full time engineer to deal with it.
level 5
Containers are great for stateless stuff. So your webservers/application
servers can be shoved into containers. Think of containers as being the modern version of statically linked
binaries or fat applications. Static binaries have the problem that any security vulnerability requires a full
rebuild of the application, and that problem is escalated in containers (where you might not even know that a
broken library exists)
If you are using the typical business application, you need one or more
storage components for data which needs to be available, possibly changed and access controlled.
Containers are a bad fit for stateful databases, or any stateful component,
really.
Containers also enable microservices, which are great ideas at a certain
organisation size (if you aren't sure you need microservices, just use a simple monolithic architecture). The
problem with microservices is that you replace complexity in your code with complexity in the communications
between the various components, and that is harder to see and debug.
level 5
Containers are fine for stateful services -- you can manage persistence at
the storage layer the same way you would have to manage it if you were running the process directly on the
host.
Backing up containers can be a pain, so you don't want to use them for
valuable data unless the data is stored elsewhere, like a database or even a file server.
For spinning up stateless applications to take workload behind a load
balancer, containers are excellent.
The problem is that there is an overwhelming din from vendors. Everybody
and their brother, sister, mother, uncle, cousin, dog, cat, and gerbil is trying to sell you some
pay-by-the-month cloud "solution".
The reason is that the cloud forces people into monthly payments, which is
a guarenteed income for companies, but costs a lot more in the long run, and if something happens and one
can't make the payments, business is halted, ensuring that bankruptcies hit hard and fast. Even with the
mainframe, a company could limp along without support for a few quarters until they could get cash flow
enough.
If we have a serious economic downturn, the fact that businesses will be
completely shuttered if they can't afford their AWS bill just means fewer companies can limp along when the
economy is bad, which will intensify a downturn.
Our company viewed the move to Azure less as a cost savings measure and
more of a move towards agility and "right now" sizing of our infrastructure.
Your point is very accurate, as an example our location is wholly incapable
of moving moving much to the cloud due to half of us being connnected via satellite network and the other half
being bent over the barrel by the only ISP in town.
level 2
I'm sorry, but I find management these days around tech wholly inadequate.
The idea that you can get an MBA and manage shit you have no idea about is absurd and just wastes everyone
elses time for them to constantly ELI5 so manager can do their job effectively.
level 2
Calling the uneducated people out on what they see as facts can be
rewarding
Aaand political suicide in a corporate environment. Instead I use the
following:
"I love this idea! We've actually been looking into a similar solution
however we weren't able to overcome some insurmountable cost sinkholes (remember: nothing is impossible; just
expensive). Will this idea require an increase in internet speed to account for the traffic going to the azure
cloud?"
level 3
This exactly, you can't compromise your own argument to the extent that
it's easy to shoot down in the name of giving your managers a way of saving face, and if you deliver the facts
after that sugar coating it will look even worse, as it will be interpreted as you setting them up to look
like idiots. Being frank, objective and non-blaming is always the best route.
level 4
I think it's quite an american thing to do it so enthusiastically; to hurt
no one, but the result is so condescending. It must be annoying to walk on egg shells to survive a day in the
office.
And this is not a rant against soft skills in IT, at all.
level 4
We work with Americans and they're always so positive, it's kinda annoying.
They enthusiastically say "This is very interesting" when in reality it sucks and they know it.
Another less professional example, one of my (non-american) co-workers
always wants to go out for coffee (while we have free and better coffee in the office), and the American
coworker is always nice like "I'll go with you but I'm not having any" and I just straight up reply "No. I'm
not paying for shitty coffee, I'll make a brew in the office". And that's that. Sometimes telling it as it is
makes the whole conversation much shorter :D
level 5
Maybe the american would appreciate the break and a chance to spend some
time with a coworker away from screens, but also doesn't want the shit coffee?
Yes. "Go out for coffee" is code for "leave the office so we can complain
about management/company/customer/Karen". Some conversations shouldn't happen inside the building.
level 7
Inferior American - please compute this - the sole intention was
consumption of coffee. Therefore due to existence of coffee in office, trip to coffee shop is not sane.
Resistance to my reasoning is futile.
level 7
Coffee is not the sole intention. If that was the case, then there wouldn't
have been an invitation.
Another intention would be the social aspect - which americans are known to
love, especially favoring it over doing any actual work in the context of a typical 9-5 corporate job.
I've effectively resisted your reasoning, therefore [insert ad hominem
insult about inferior logic].
There is still value to tact in your delivery, but, don't slit your own
throat. Remind them they hired you for a reason.
"I appreciate the ideas being raised here by management for cost savings
and it certainly merits a discussion point. As a business, we have many challenges and one of them includes
controlling costs. Bridging these conversations with subject matter experts and management this early can
really help fill out the picture. I'd like to try to provide some of that value to the conversation here"
level 3
If it's political suicide to point out potential downsides, then you need
to work somewhere else.
Especially since your response, in my experience, won't get anything done
(people will either just say "no, that's not an issue", or they'll find your tone really condescending) and
will just piss people off.
I worked with someone like that would would always be super passive
aggressive in how she brought things up and it pissed me off to no end, because it felt less like bringing up
potential issues and more like being belittling.
level 4
Feels good to get to a point in my career where I can call people our
whenever the hell I feel like it. I've got recruiters banging down my door, I couldn't swing a stick without
hitting a job offer or three.
Of course you I'm not suggesting problem be combative for no reason and to
be tactful about it but if you don't call out fools like that then you're being negligent in your duties.
level 4
"I am resolute in my ability to elevate this collaborative,
forward-thinking team into the revenue powerhouse that I believe it can be. We will transition into a DevOps
team specialising in migrating our existing infrastructure entirely to code and go completely serverless!" -
CFO that outsources IT
level 2
We recently implemented DevOps practices, Scrum, and sprints have become
the norm... I swear to god we spend 60% of our time planning our sprints, and 40% of the time doing the work,
and management wonders why our true productivity has fallen through the floor...
level 5
If you spend three whole days out of every five in planning meetings, this
is a problem with your meeting planners, not with screm. If these people stay in charge, you'll be stuck in
planning hell no matter what framework or buzzwords they try to fling around.
level 6
My last company used screm. No 3 day meeting events, 20min of loose
direction and we were off to the races. We all came back with parts to different projects. SYNERGY.
level 7
Are you saying quantum synergy coupled with block chain neutral
intelligence can not be used to expedite artificial intelligence amalgamation into that will metaphor into
cucumber obsession?
Everyone says this, meaning "the way
I
do Agile is good, the way everyone else does it sucks. Buy my Agile book! Or my Agile training course! Only
$199.99".
level 8
There are different ways to do Agile? From the past couple of places I
worked at I thought Agile was just standing in a corner for 5 minutes each morning. Do some people sit?
level 9
Wait until they have you doing "retrospectives" on a Friday afternoon with
a bunch of alcohol involved. By Monday morning nobody remembers what the fuck they retrospected about on
Friday.
level 10
No, that's what it's supposed to look like. A quick 'Is anyone blocked?
Does anyone need anything/can anyone chip in with X? Ok get back to it'
What it usually looks like is a round table 'Tell us what you're working
on' that takes at least 30, and depending on team size, closer to an hour.
level 10
our weekly 'stand-up' is often 60-90 minutes long, because they treat it
like not only a roundtable discussion about what you're working on, but an opportunity to hash out every
discussion to death, in front of C-levels.
To me agile is an unfortunate framework to confront and dismantle a lot of
hampering low value business processes. I call it a "get-er-done" framework. But yes theres not all Rose's and
sunshine in agile. But it's important to destroy processes that make delivering value impossible
Like all such things - it's a useful technique, that turns into a colossal
pile of wank if it's misused. This is true of practically every buzzword laden methodology I've seen
introduced in the last 20 years.
level 5
Fro me, the fact that my team is moderately scrummy is a decent treatment
for my ADHD. The patterns are right up there with ritalin in terms of making me less neurologically crippled.
level 5
I think I'm going to snap if I have one more meting that discusses seamless migrations
and seamless movement across a complex multi forest non-standardized network.
Props to the
artist who actually found a way to visualize most of this meaningless corporate lingo. I'm sure it wasn't easy to come up
with everything.
He missed "sea
change" and "vertical integration". Otherwise, that was pretty much all of the useless corporate meetings I've ever attended
distilled down to 4.5 minutes. Oh, and you're getting laid off and/or no raises this year.
For those too
young to get the joke, this is a style parody of Crosby, Stills & Nash, a folk-pop super-group from the 60's. They were
hippies who spoke out against corporate interests, war, and politics. Al took their sound (flawlessly), and wrote a song in
corporate jargon (the exact opposite of everything CSN was about). It's really brilliant, to those who get the joke.
"The company has
undergone organization optimization due to our strategy modification, which includes empowering the support to the
operation in various global markets" - Red 5 on why they laid off 40 people suddenly. Weird Al would be proud.
In his big long
career this has to be one of the best songs Weird Al's ever done. Very ambitious rendering of one of the most
ambitious songs in pop music history.
This should be
played before corporate meetings to shame anyone who's about to get up and do the usual corporate presentation. Genius
as usual, Mr. Yankovic!
There's a quote
it goes something like: A politician is someone who speaks for hours while saying nothing at all. And this is exactly
it and it's brilliant.
From the current
Gamestop earnings call "address the challenges that have impacted our results, and execute both deliberately and with
urgency. We believe we will transform the business and shape the strategy for the GameStop of the future. This will be
driven by our go-forward leadership team that is now in place, a multi-year transformation effort underway, a commitment
to focusing on the core elements of our business that are meaningful to our future, and a disciplined approach to
capital allocation."" yeah Weird Al totally nailed it
Excuse me, but
"proactive" and "paradigm"? Aren't these just buzzwords that dumb people use to sound important? Not that I'm accusing you
of anything like that. [pause] I'm fired, aren't I?~George Meyer
I watch this at
least once a day to take the edge of my job search whenever I have to decipher fifteen daily want-ads claiming to
seek "Hospitality Ambassadors", "Customer Satisfaction Specialists", "Brand Representatives" and "Team Commitment
Associates" eventually to discover they want someone to run a cash register and sweep up.
The irony is a
song about Corporate Speak in the style of tie-died, hippie-dippy CSN (+/- )Y four-part harmony. Suite Judy Blue Eyes
via Almost Cut My Hair filtered through Carry On. "Fantastic" middle finger to Wall Street,The City, and the monstrous
excesses of Unbridled Capitalism.
If you
understand who and what he's taking a jab at, this is one of the greatest songs and videos of all time. So spot on.
This and Frank's 2000 inch tv are my favorite songs of yours. Thanks Al!
hahaha,
"Client-Centric Solutions...!" (or in my case at the time, 'Customer-Centric' solutions) now THAT's a term i haven't
heard/read/seen in years, since last being an office drone. =D
When I interact
with this musical visual medium I am motivated to conceptualize how the English language can be better compartmentalized
to synergize with the client-centric requirements of the microcosmic community focussed social entities that I
administrate on social media while interfacing energetically about the inherent shortcomings of the current
socio-economic and geo-political order in which we co-habitate. Now does this tedium flow in an effortless stream of
coherent verbalisations capable of comprehension?
When I bought
"Mandatory Fun", put it in my car, and first heard this song, I busted a gut, laughing so hard I nearly crashed.
All the corporate buzzwords!
(except "pivot", apparently).
I agree with the sentiment. This talk of going serverless or getting rid of traditional
IT admins has gotten very old. In some ways it is true, but in many ways it is greatly exaggerated. There will always be a
need for onsite technical support. There are still users today that cannot plug in a mouse or keyboard into a USB port. Not
to mention layer 1 issues; good luck getting your cloud provider to run a cable drop for you. Besides, who is going to
manage your cloud instances? They don't just operate and manage themselves.
"... Having a customer always available on site would mean that the customer in question is probably a small, expendable fish in his organization and is unlikely to have any useful knowledge of its business practices. ..."
"... Unit testing code before it is written means that one would have to have a mental picture of what one is going to write before writing it, which is difficult without upfront design. And maintaining such tests as the code changes would be a nightmare. ..."
"... Programming in pairs all the time would assume that your topnotch developers are also sociable creatures, which is rarely the case, and even if they were, no one would be able to justify the practice in terms of productivity. I won't discuss why I think that abandoning upfront design is a bad practice; the whole idea is too ridiculous to debate ..."
"... Both book and methodology will attract fledgling developers with its promise of hacking as an acceptable software practice and a development universe revolving around the programmer. It's a cult, not a methodology, were the followers shall find salvation and 40-hour working weeks ..."
"... Two stars for the methodology itself, because it underlines several common sense practices that are very useful once practiced without the extremity. ..."
"... The second is the dictatorial social engineering that eXtremity mandates. I've actually tried the pair programming - what a disaster. ..."
"... I've also worked with people who felt that their slightest whim was adequate reason to interfere with my work. That's what Beck institutionalizes by saying that any request made of me by anyone on the team must be granted. It puts me completely at the mercy of anyone walking by. The requisite bullpen physical environment doesn't work for me either. I find that the visual and auditory distraction make intense concentration impossible. ..."
"... One of the things I despise the most about the software development culture is the mindless adoption of fads. Extreme programming has been adopted by some organizations like a religious dogma. ..."
To fairly review this book, one must distinguish between the methodology it presents and the
actual presentation. As to the presentation, the author attempts to win the reader over with emotional
persuasion and pep talk rather than with facts and hard evidence. Stories of childhood and comradeship
don't classify as convincing facts to me.
A single case study-the C3 project-is often referred to, but with no specific information (do
note that the project was cancelled by the client after staying in development for far too long).
As to the method itself, it basically boils down to four core practices:
Always have a customer available on site.
Unit test before you code.
Program in pairs.
Forfeit detailed design in favor of incremental, daily releases and refactoring.
If you do the above, and you have excellent staff on your hands, then the book promises that
you'll reap the benefits of faster development, less overtime, and happier customers. Of course,
the book fails to point out that if your staff is all highly qualified people, then the project
is likely to succeed no matter what methodology you use. I'm sure that anyone who has worked in
the software industry for sometime has noticed the sad state that most computer professionals
are in nowadays.
However, assuming that you have all the topnotch developers that you desire, the outlined methodology
is almost impossible to apply in real world scenarios. Having a customer always available
on site would mean that the customer in question is probably a small, expendable fish in his organization
and is unlikely to have any useful knowledge of its business practices.
Unit testing code before it is written means that one would have to have a mental picture
of what one is going to write before writing it, which is difficult without upfront design. And
maintaining such tests as the code changes would be a nightmare.
Programming in pairs all the time would assume that your topnotch developers are also sociable
creatures, which is rarely the case, and even if they were, no one would be able to justify the
practice in terms of productivity. I won't discuss why I think that abandoning upfront design
is a bad practice; the whole idea is too ridiculous to debate.
Both book and methodology will attract fledgling developers with its promise of hacking
as an acceptable software practice and a development universe revolving around the programmer.
It's a cult, not a methodology, were the followers shall find salvation and 40-hour working weeks.
Experience is a great teacher, but only a fool would learn from it alone. Listen to what the
opponents have to say before embracing change, and don't forget to take the proverbial grain of
salt.
Two stars out of five for the presentation for being courageous and attempting to defy the
standard practices of the industry. Two stars for the methodology itself, because it underlines
several common sense practices that are very useful once practiced without the extremity.
Maybe it's an interesting idea, but it's just not ready for prime time.
Parts of Kent's recommended practice - including aggressive testing and short integration cycle
- make a lot of sense. I've shared the same beliefs for years, but it was good to see them clarified
and codified. I really have changed some of my practice after reading this and books like this.
I have two broad kinds of problem with this dogma, though. First is the near-abolition of documentation.
I can't defend 2000 page specs for typical kinds of development. On the other hand, declaring
that the test suite is the spec doesn't do it for me either. The test suite is code, written for
machine interpretation. Much too often, it is not written for human interpretation. Based on the
way I see most code written, it would be a nightmare to reverse engineer the human meaning out
of any non-trivial test code. Some systematic way of ensuring human intelligibility in the code,
traceable to specific "stories" (because "requirements" are part of the bad old way), would give
me a lot more confidence in the approach.
The second is the dictatorial social engineering that eXtremity mandates. I've actually
tried the pair programming - what a disaster. The less said the better, except that my experience
did not actually destroy any professional relationships. I've also worked with people who
felt that their slightest whim was adequate reason to interfere with my work. That's what Beck
institutionalizes by saying that any request made of me by anyone on the team must be granted.
It puts me completely at the mercy of anyone walking by. The requisite bullpen physical environment
doesn't work for me either. I find that the visual and auditory distraction make intense concentration
impossible.
I find revival tent spirit of the eXtremists very off-putting. If something works, it works
for reasons, not as a matter of faith. I find much too much eXhortation to believe, to go ahead
and leap in, so that I will eXperience the wonderfulness for myself. Isn't that what the evangelist
on the subway platform keeps saying? Beck does acknowledge unbelievers like me, but requires their
exile in order to maintain the group-think of the X-cult.
Beck's last chapters note a number of exceptions and special cases where eXtremism may not work
- actually, most of the projects I've ever encountered.
There certainly is good in the eXtreme practice. I look to future authors to tease that good
out from the positively destructive threads that I see interwoven.
A customer on May 2, 2004
A work of fiction
The book presents extreme programming. It is divided into three parts:
(1) The problem
(2) The solution
(3) Implementing XP.
The problem, as presented by the author, is that requirements change but current methodologies
are not agile enough to cope with this. This results in customer being unhappy. The solution is
to embrace change and to allow the requirements to be changed. This is done by choosing the simplest
solution, releasing frequently, refactoring with the security of unit tests.
The basic assumption which underscores the approach is that the cost of change is not exponential
but reaches a flat asymptote. If this is not the case, allowing change late in the project would
be disastrous. The author does not provide data to back his point of view. On the other hand there
is a lot of data against a constant cost of change (see for example discussion of cost in Code
Complete). The lack of reasonable argumentation is an irremediable flaw in the book. Without some
supportive data it is impossible to believe the basic assumption, nor the rest of the book. This
is all the more important since the only project that the author refers to was cancelled before
full completion.
Many other parts of the book are unconvincing. The author presents several XP practices. Some
of them are very useful. For example unit tests are a good practice. They are however better treated
elsewhere (e.g., Code Complete chapter on unit test). On the other hand some practices seem overkill.
Pair programming is one of them. I have tried it and found it useful to generate ideas while prototyping.
For writing production code, I find that a quiet environment is by far the best (see Peopleware
for supportive data). Again the author does not provide any data to support his point.
This book suggests an approach aiming at changing software engineering practices. However the
lack of supportive data makes it a work of fiction.
I would suggest reading Code Complete for code level advice or Rapid Development for management
level advice.
A customer on November 14, 2002
Not Software Engineering.
Any Engineering discipline is based on solid reasoning and logic not on blind faith. Unfortunately,
most of this book attempts to convince you that Extreme programming is better based on the author's
experiences. A lot of the principles are counterintuitive and the author exhorts you just try
it out and get enlightened. I'm sorry but these kind of things belong in infomercials not in s/w
engineering.
The part about "code is the documentation" is the scariest part. It's true that keeping the
documentation up to date is tough on any software project, but to do away with documentation is
the most ridiculous thing I have heard.
It's like telling people to cut of their noses to avoid colds. Yes we are always in search
of a better software process. Let me tell you that this book won't lead you there.
The "gossip magazine diet plans" style of programming.
This book reminds me of the "gossip magazine diet plans", you know, the vinegar and honey diet,
or the fat-burner 2000 pill diet etc. Occasionally, people actually lose weight on those diets,
but, only because they've managed to eat less or exercise more. The diet plans themselves are
worthless. XP is the same - it may sometimes help people program better, but only because they
are (unintentionally) doing something different. People look at things like XP because, like dieters,
they see a need for change. Overall, the book is a decently written "fad diet", with ideas that
are just as worthless.
A customer on August 11, 2003
Hackers! Salvation is nigh!!
It's interesting to see the phenomenon of Extreme Programming happening in the dawn of the
21st century. I suppose historians can explain such a reaction as a truly conservative movement.
Of course, serious software engineering practice is hard. Heck, documentation is a pain in the
neck. And what programmer wouldn't love to have divine inspiration just before starting to write
the latest web application and so enlightened by the Almighty, write the whole thing in one go,
as if by magic? No design, no documentation, you and me as a pair, and the customer too. Sounds
like a hacker's dream with "Imagine" as the soundtrack (sorry, John).
The Software Engineering struggle is over 50 years old and it's only logical to expect some resistance,
from time to time. In the XP case, the resistance comes in one of its worst forms: evangelism.
A fundamentalist cult, with very little substance, no proof of any kind, but then again if you
don't have faith you won't be granted the gift of the mystic revelation. It's Gnosticism for Geeks.
Take it with a pinch of salt.. well, maybe a sack of salt. If you can see through the B.S. that
sells millions of dollars in books, consultancy fees, lectures, etc, you will recognise some common-sense
ideas that are better explained, explored and detailed elsewhere.
Kent is an excellent writer. He does an excellent job of presenting an approach to software
development that is misguided for anything but user interface code. The argument that user interface
code must be gotten into the hands of users to get feedback is used to suggest that complex system
code should not be "designed up front". This is simply wrong. For example, if you are going to
deploy an application in the Amazon Cloud that you want to scale, you better have some idea of
how this is going to happen. Simply waiting until your application falls over and fails is not
an acceptable approach.
One of the things I despise the most about the software development culture is the mindless
adoption of fads. Extreme programming has been adopted by some organizations like a religious
dogma.
Engineering large software systems is one of the most difficult things that humans do. There
are no silver bullets and there are no dogmatic solutions that will make the difficult simple.
Maybe I'm too cynical because I never got to work for the successful, whiz-kid companies; Maybe
this book wasn't written for me!
This book reminds me of Jacobsen's "Use Cases" book of the 1990s. 'Use Cases' was all the rage
but after several years, we slowly learned the truth: Uses Cases does not deal with the architecture
- a necessary and good foundation for any piece of software.
Similarly, this book seems to be spotlighting Testing and taking it to extremes.
'the test plan is the design doc'
Not True. The design doc encapsulates wisdom and insight
a picture that accurately describes the interactions of the lower level software components
is worth a thousand lines of code-reading.
Also present is an evangelistic fervor that reminds me of the rah-rah eighties' bestseller,
"In Search Of Excellence" by Peters and Waterman. (Many people have since noted that most of the
spotlighted companies of that book are bankrupt twenty five years later).
- in a room full of people with a bully supervisor (as I experienced in my last job at
a major telco) innovation or good work is largely absent.
- deploy daily - are you kidding? to run through the hundreds of test cases in a large
application takes several hours if not days. Not all testing can be automated.
- I have found the principle of "baby steps", one of the principles in the book, most useful
in my career - it is the basis for prototyping iteratively. However I heard it described in
1997 at a pep talk at MCI that the VP of our department gave to us. So I dont know who stole
it from whom!
Lastly, I noted that the term 'XP' was used throughout the book, and the back cover has a blurb
from an M$ architect. Was it simply coincidence that Windows shares the same name for its XP release?
I wondered if M$ had sponsored part of the book as good advertising for Windows XP! :)
Chasing recent fad and risking the organization assets (systems, processes, people, reputations) for the sake of advancing your
goals is a clear-cut characteristic of a broken ecosystem.
The change madness is getting
worse with every passing year .
The demands for change being placed on corporate IT are
plain ridiculous. As a consequence we are breaking IT. In pursuit of absurd project commitments
we are eating
ourselves .
And the hysteria reaches fever pitch as people extrapolate trends into the future
linearly or worse still exponentially. This is such bad scientific thinking
that it shouldn't be worthy of debate, but the power of critical thought is a scarce
resource
A broken management and governance system, a broken value system, and a broken
culture.
But even in the best and healthiest organisations, there are plenty of rogues; psychopaths
(and milder sociopaths) who are never going to care about anyone but themselves. They soar in
management (and they're drawn to the power); they look good to all measures and controls
except a robust risk management system - it is the last line of defense.
...I'm saying there is a real limit to how fast humans can change: how fast we can change our
behaviours, our attitudes, our processes, our systems. We need to accept that the technology
is changing faster than society, our IT sector, our organisations, our teams, ourselves can
change.
I'm saying there is a social and business backlash already to the pace of change. We're
standing in the ruins of an economy that embraced fast change.
I'm saying there are real risks to the pace of change, and we currently live in a culture
that thinks writing risks down means you can then ignore them, or that if you can't ignore
them you can always hedge them somehow.
We have to slow down a bit. perhaps "Slow IT" is the wrong name but it was catchy. I'm not
saying go slooooow. We've somehow sustained a pretty impressive pace for decades. But clearly
it can't go much faster, if at all, and all these demands that it must go faster are plain
silly. It just can't. There's bits falling off, people burning out, smoking shells of
projects everywhere.
I'm not saying stop, but I am saying ease off a little, calm down, stop panicking, stop
this desperate headlong rush. You are right Simon that mindfulness is a key element: we all
need time to think. Let the world keep up.
Yes, Rob, short-termism is certainly bad news, and rushing to achieve short-term goals
without thinking about them in the larger context is a good indication of disaster ahead.
Much of the zeitgeist that drives the frenzy you describe is generated by vendors
especially those with software in their portfolio. Software has more margin that hardware or
service. As a result they have more marketing budget. With that budget they invest and spend
a lot of time and effort to figure out exactly how to generate the frenzy with a new thing
that you must have. They have to do this to keep market interest in the products. That is
actually what their job is.
The frenzy is deliberately and I would say almost scientifically engineered by very very
bright marketing people in software vendors. Savvy IT organizations are aware of that
distinction and maintain their focus on enabling their business to be successful. IT as
Utility, On Demand, SOA, Cloud, ..... Software vendors will not and should not stop doing
that - that is what keeps them in business and generates profits that enable new innovation.
The onus is on the buyer to understand that whatever the latest technology is, does not
provide the answer for how they will improve business performance. Improving business
performance is the burden that only the organization can bear.
And now example buss words infused nonsense: ""DevOps is a concept, an idea, a life philosophy," says Gottfried Sehringer, chief marketing
officer at
XebiaLabs
, a software delivery
automation company. "It's not really a process or a toolset, or a technology." And another one:
..." "In an ideal world, you would push a button to release every few seconds," Sehringer says.
But this is not an ideal world, and so people plug up the process along the way."... "
I want to see sizable software product with the release every few seconds.
Even for a small and rapidly evolving web site scripts should be released no more frequently then
daily.
Notable quotes:
"... Even if you're a deity of software bug-squashing, how can you -- or any developer or operations specialist -- deliver high-quality, "don't break anything" code when you have to build and release that fast? Everyone has their own magic bullet. "Agile -- " cries one crowd. " Continuous build -- " yells another. " Continuous integration -- " cheers a third. ..."
"... Automation has obvious returns on investment. "You can make sure it's good in pre-production and push it immediately to production without breaking anything, and then just repeat, repeat, repeat, over and over again," says Sehringer. ..."
"... In other words, you move delivery through all the steps in a structured, repeatable, automated way to reduce risk and increase the speed of releases and updates. ..."
The
quickie guide to continuous delivery in DevOps
In today's world, you have to develop and
deliver almost in the same breath. Here's a quick guide to help you figure out which continuous
delivery concepts will help you breathe easy, and which are only hot air. Developers are always
under pressure to produce more and release software faster, which encourages the adoption of
new concepts and tools. But confusing buzzwords obfuscate real technology and business
benefits, particularly when a vendor has something to sell. That makes it hard to determine
what works best -- for real, not just as a marketing phrase -- in the continuous flow of build and
deliver processes. This article gives you the basics of continuous delivery to help you sort it
all out.
To start with, the terms apply to different parts of the same production arc, each of which
are automated to different degrees:
Continuous integration means frequently merging code into a central repository
.
"Frequently" means usually several times a day. Each merge triggers an automated "build and
test" instance, a process sometimes called continuous build . But by either name, continuous
integration and continuous build do nothing in terms of delivery or deployment. They're about
code management, not what happens to the code afterward.
Continuous delivery refers to the automation of the software release process
, which
includes some hands-on effort by developers. Usually, developers approve or initiate the
deployment, though there can be other manual steps as well.
Continuous deployment is continuous delivery with no manual steps for developers. The
whole thing is automated, and it requires not so much as a nod from humans.
With continuous deployment, "a developer's job typically ends at reviewing a pull request
from a teammate and merging it to the master branch," explains Marko Anastasov in a
blog post
. "A continuous integration/continuous deployment service takes over from there
by running all tests and deploying the code to production, while keeping the team informed
about [the] outcome of every important event."
However, knowing the terms and their definitions isn't enough to help you determine when and
where it is best to use each. Because, of course, every shop is different.
It would be great if the market clearly distinguished between concepts and tools and their
uses, as they do with terms like DevOps. Oh, wait.
"DevOps is a concept, an idea, a life philosophy," says Gottfried Sehringer, chief marketing
officer at
XebiaLabs
, a software delivery
automation company. "It's not really a process or a toolset, or a technology."
But, alas, industry terms are rarely spelled out that succinctly. Nor are they followed with
hints and tips on how and when to use them. Hence this guide, which aims to help you learn when
to use what.
Choose your accelerator according to your need for speed
That's not the end of it; some businesses push for software updates to be faster still. "If
you work for Amazon, it might be every few seconds," says Sehringer.
Even if you're a deity of software bug-squashing, how can you -- or any developer or operations
specialist -- deliver high-quality, "don't break anything" code when you have to build and release
that fast? Everyone has their own magic bullet. "Agile -- " cries one crowd. "
Continuous
build
-- " yells another. "
Continuous integration
-- "
cheers a third.
Let's just cut to the chase on all that, shall we?
"Just think of continuous as 'automated,'" says Nate Berent-Spillson, senior delivery
director at
Nexient
, a software
services provider. "Automation is driving down cost and the time to develop and deploy."
Well, frack, why don't people just say automation?
Add to the idea of automation the concepts of continuous build, continuous delivery,
continuous everything, which are central to DevOps, and we find ourselves talking in circles.
So, let's get right to sorting all that out.
... ... ...
Rinse. Repeat, repeat, repeat, repeat (the point of automation in
DevOps)
Automation has obvious returns on investment. "You can make sure it's good in pre-production
and push it immediately to production without breaking anything, and then just repeat, repeat,
repeat, over and over again," says Sehringer.
In other words, you move delivery through all the steps in a structured, repeatable,
automated way to reduce risk and increase the speed of releases and updates.
In an ideal world, you would push a button to release every few seconds," Sehringer says.
But this is not an ideal world, and so people plug up the process along the way.
A company may need approval for an application change from its legal department. "Some
companies are heavily regulated and may need additional gates to ensure compliance," notes
Sehringer. "It's important to understand where these bottlenecks are." The ARA software should
improve efficiencies and ensure the application is released or updated on schedule.
"Developers are more familiar with continuous integration," he says. "Application release
automation is more recent and thus less understood."
... ... ...
Pam Baker has written hundreds of articles published in leading technology, business and
finance publications including InformationWeek, Institutional Investor magazine, CIO.com,
NetworkWorld, ComputerWorld, IT World, Linux World, and more. She has also authored several
analytical studies on technology, eight books -- the latest of which is Data Divination: Big
Data Strategies -- and an award-winning documentary on paper-making. She is a member of the
National Press Club, Society of Professional Journalists and the Internet Press Guild.
"... It's basically a bullshit bingo post where someone repeats a buzzword without any knowledge of the material behind it ..."
"... continuous delivery == constant change ..."
"... This might be good for developers, but it's a nightmare for the poor, bloody, customers. ..."
"... However, I come at it from the other side, the developers just push new development out and production support is responsible for addressing the mess, it is horrible, there is too much disconnect between developers and their resulting output creating consistent outages. The most successful teams follow the mantra "Eat your own dog food" , developers who support the crap they push ..."
"... But do you know who likes Continuous Delivery? Not the users. The users hate stuff changing for the sake of change, but trying to convince management seems an impossible task. ..."
"... some of us terrified what 'continious delivery' means in the context of software in the microcontroller of a health care device, ..."
"... It is a natural consequence of a continuous delivery, emphasis on always evolving and changing and that the developer is king and no one can question developer opinion. Developer decides it should move, it moves. No pesky human testers to stand up and say 'you confused the piss out of us' to make them rethink it. No automatic test is going to capture 'confuse the piss out of the poor non-developer users'. ..."
"... It's amazing how common this attitude has become. It's aggressively anti-customer, and a big part of the reason for the acceleration of the decline of software quality over the past several years. ..."
"... All I know is that, as a user, rapid-release or continuous delivery has been nothing but an enormous pain in the ass to me and I wish it would die the horrible death it deserves already. ..."
Yeah, this is an incredibly low quality article. It doesn't specify what it means by what AI
should do, doesn't specify which type of AI, doesn't specify why AI should be used, etc. Junk
article.
It's basically a bullshit bingo post where someone repeats a buzzword without any knowledge
of the material behind it.
Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop the wrong
thing that doesn't even work they way they thought it should 4. The project leader is canned 5.
The software is implemented by an outside vendor, with all the flaws. 6. The software operates
finally after 5 years of modifications to both the software and the workflows (to match the flaws
in the software). 7. As soon as it's all running properly and everyone is trained, a new project
is launched to redo it, "the right way". 8. Goto 1
Here is the actual model, a model that will exist for the next 1,000 years.
1. Someone (or something) gathers requirement. 2. They get it wrong. 3. They develop
the wrong thing that doesn't even work they way they thought it should 4. The project leader
is canned 5. The software is implemented by an outside vendor, with all the flaws. 6. The software
operates finally after 5 years of modifications to both the software and the workflows (to
match the flaws in the software). 7. As soon as it's all running properly and everyone is trained,
a new project is launched to redo it, "the right way". 8. Goto 1
You just accurately described a 6 year project within our organization....and it made me cry
Does this model have a name? an urban dictionary name? if not it needs one.
Yeah, maybe there's something useful in TFA, but I'm not really inclined to go looking based
on what was in the summary. At no point, did the person being quoted actually say anything of
substance.
It's just buzzword soup with a dash of new technologies thrown in.
Five years ago they would have said practically the same words, but just talked about
utilizing the cloud instead of AI.
I'm also a little skeptical of any study published by a company looking to sell you what the
study has just claimed to be great. That doesn't mean its a complete sham, but how hard did they
look for other explanations why some companies are more successful than others?
I notice the targets are all set from the company's point of view... including customer satisfaction.
However it's quite easy to meet any goal, as long as you set it low enough.
Companies like Comcast or Qwest objectively have abysmal customer satisfaction ratings; but
they likely meet their internal goal for that metric. I notice, in their public communications,
they always use phrasing along the lines of "giving you an even better customer service experience"
- again, the trick is to set the target low and
This might be good for developers, but it's a nightmare for the poor, bloody, customers.
Any professional outfit will test a new release (in-house or commercial product) thoroughly
before letting it get anywhere close to an environment where their business is at stake.
This process can take anywhere from a day or two to several months, depending on the complexity
of the operation, the scope of the changes, HOW MANY (developers note: not if any ) bugs
are found and whether any alterations to working practices have to be introduced.
So to have developers lob a new "release" over the wall at frequent intervals is not useful,
it isn't clever, nor does it save (the users) any money or speed up their acceptance. It just
costs more in integration testing, floods the change control process with "issues" and means that
when you report (again, developers: not if ) problems, it is virtually impossible to describe
exactly which release you are referring to and even more impossible for whoever fixes the bugs
to produce the same version to fix and then incorporate those fixes into whatever happens to be
the latest version - that hour. Even more so when dozens of major corporate customers are ALL
reporting bugs with each new version they test.
Any professional outfit will test a new release (in-house or commercial product) thoroughly
before letting it get anywhere close to an environment where their business is at stake. This
process can take anywhere from a day or two to several months, depending on the complexity of
the operation, the scope of the changes, HOW MANY (developers note: not if any) bugs are found
and whether any alterations to working practices have to be introduced.
I wanted to chime in with a tangible anecdote to support your
I can sympathize with that few, of it appearing to have too many developers focused upon deployment/testing
then actual development.
However, I come at it from the other side, the developers just push new development out
and production support is responsible for addressing the mess, it is horrible, there is too much
disconnect between developers and their resulting output creating consistent outages. The most
successful teams follow the mantra "Eat your own dog food" , developers who support the crap they
push
But do you know who likes Continuous Delivery? Not the users. The users hate stuff changing
for the sake of change, but trying to convince management seems an impossible task.
Why should users not like it? If you shop on amazon you don't know if a specific feature you
notice today came there via continuous delivery or a more traditional process.
The crux of the problem is that we (in these discussions and the analysts) describe *all* manner
of 'software development' as the same thing. Whether it's a desktop application, an embedded microcontroller
in industrial equipment, a web application for people to get work done, or a webapp to let people
see the latest funny cat video.
Then we start talking past each other, some of us terrified what 'continious delivery'
means in the context of software in the microcontroller of a health care device, others t
Well, 'continuous delievery' is a term with a defined meaning. And releasing phone apps with
unwanted UI/functionality in rapid succession is not part of that definition. Continuous delievery
basically only is the next logical step after continuous integration. You deploy the new functionallity
automatically (or with a click of a button) when certain test criteria are met. Usually on a subset
of your nodes so only a subset of your customers sees it. If you have crashes on those nodes or
customer complaints you
You deploy the new functionallity automatically (or with a click of a button) when certain
test criteria are met. Usually on a subset of your nodes so only a subset of your customers
sees it. If you have crashes on those nodes or customer complaints you roll back.
Why do you consider this to be a good thing? It's certainly not for those poor customers who
were chosen to be involuntary beta testers, and it's also not for the rest of the customers who
have to deal with software that is constantly changing underneath them.
'continuous delievery' is a term with a defined meaning. And releasing phone apps with unwanted
UI/functionality in rapid succession is not part of that definition.
It is a natural consequence of a continuous delivery, emphasis on always evolving and changing
and that the developer is king and no one can question developer opinion. Developer decides it
should move, it moves. No pesky human testers to stand up and say 'you confused the piss out of
us' to make them rethink it. No automatic test is going to capture 'confuse the piss out of the
poor non-developer users'.
If you have crashes on those nodes or customer complaints you roll back.
Note that a customer with a choice is likely to just go somewhere else rather than use your
software.
IT in my company does network, Windows, Office and Virus etc. type of work. Is this what they
talk about? Anyway, it's been long outsourced to IT (as in "Indian" technology)...
I recently interviewed at a couple of the new fangled big data marketing startups that correlate
piles of stuff to help target ads better, and they were continuously deploying up the wazoo. In
fact, they had something like zero people doing traditional QA.
It was not totally insane at all. But they did have a blaze attitude about deployments -- if
stuff don't work in production they just roll back, and not worry about customer input data being
dropped on the floor. Heck, they did not worry much about da
But they did have a blaze attitude about deployments -- if stuff don't work in production
they just roll back, and not worry about customer input data being dropped on the floor.
It's amazing how common this attitude has become. It's aggressively anti-customer, and
a big part of the reason for the acceleration of the decline of software quality over the past
several years.
You want your deployment system to be predictable, and as my old AI professor used to say,
intelligent means hard to predict. You don't want AI for systems that just have to do the exact
same thing reliably over and over again.
All I know is that, as a user, rapid-release or continuous delivery has been nothing but
an enormous pain in the ass to me and I wish it would die the horrible death it deserves already.
...At Duke University's Office of Information Technology (OIT), we began looking at containers as
a way to achieve higher density from the virtualized infrastructure used to host websites. Virtual
machine (VM) sprawl had started to become a problem. We favored separating each client's website
onto its own VM for both segregation and organization, but steady growth meant we were managing more
servers than we could handle. As we looked for ways to lower management overhead and make better
use of resources, Docker hit the news, and we began to experiment with
containerization
for our web applications.
For us, the initial investigation of containers mirrors a shift toward a DevOps culture.
Where we started
When we first looked into container technology, OIT was highly process driven and composed of
monolithic applications and a monolithic organizational structure. Some early forays into automation
were beginning to lead the shift toward a new cultural organization inside the department, but even
so, the vast majority of our infrastructure consisted of "pet" servers (to use the
pets
vs. cattle analogy). Developers created their applications on staging servers designed to match
production hosting environments and deployed by migrating code from the former to the latter. Operations
still approached hosting as it always had: creating dedicated VMs for individual services and filing
manual tickets for monitoring and backups. A service's lifecycle was marked by change requests, review
boards, standard maintenance windows, and lots of personal attention.
A shift in culture
As we began to embrace containers, some of these longstanding attitudes toward development and
hosting began to shift a bit. Two of the larger container success stories came from our investigation
into cloud infrastructure. The first project was created to host hundreds of R-Studio containers
for student classes on Microsoft Azure hosts, breaking from our existing model of individually managed
servers and moving toward "cattle"-style infrastructure designed for hosting containerized applications.
The other was a rapid containerization and deployment of the Duke website to Amazon Web Services
while in the midst of a denial-of-service attack, dynamically creating infrastructure and rapidly
deploying services.
The success of these two wildly nonstandard projects helped to legitimize containers within the
department, and more time and effort was put into looking further into their benefits and those of
on-demand and disposable cloud infrastructure, both on-premises and through public cloud providers.
It became apparent early on that containers lived within a different timescale from traditional
infrastructure. We started to notice cases where short-lived, single-purpose services were created,
deployed, lived their entire lifecycle, and were decommissioned before we completed the tickets created
to enter them into inventory, monitoring, or backups. Our policies and procedures were not able to
keep up with the timescales that accompanied container development and deployment.
In addition, humans couldn't keep up with the automation that went into creating and managing
the containers on our hosts. In response, we began to develop more automation to accomplish usually
human-gated processes. For example, the dynamic migration of containers from one host to another
required a change in our approach to monitoring. It is no longer enough to tie host and service monitoring
together or to submit a ticket manually, as containers are automatically destroyed and recreated
on other hosts in response to events.
Some of this was in the works for us already-automation and container adoption seem to parallel
one another. At some point, they become inextricably intertwined.
As containers continued to grow in popularity and OIT began to develop tools for container orchestration,
we tried to further reinforce the "cattle not pets" approach to infrastructure. We limited login
of the hosts to operations staff only (breaking with tradition) and gave all hosts destined for container
hosting a generic name. Similar to being coached to avoid naming a stray animal in an effort to prevent
attachment, servers with generic names became literally forgettable. Management of the infrastructure
itself became the responsibility of automation, not humans, and humans focused their efforts on the
services inside the containers.
Containers also helped to usher continuous integration into our everyday workflows. OIT's Identity
Management team members were early adopters and began to build Kerberos key distribution centers
(KDCs) inside containers using Jenkins, building regularly to incorporate patches and test the resulting
images. This allowed the team to catch breaking builds before they were pushed out onto production
servers. Prior to that, the complexity of the environment and the widespread impact of an outage
made patching the systems a difficult task.
Embracing continuous deployment
Since that initial use case, we've also embraced continuous deployment. There is a solid pattern
for every project that gets involved with our continuous integration/continuous deployment (CI/CD)
system. Many teams initially have a lot of hesitation about automatically deploying when tests pass,
and they tend to build checkpoints requiring human intervention. However, as they become more comfortable
with the system and learn how to write good tests, they almost always remove these checkpoints.
Within our container orchestration automation, we use Jenkins to patch base images on a regular
basis and rebuild all the child images when the parent changes. We made the decision early that the
images could be rebuilt and redeployed at any time by automated processes. This meant that any code
included in the branch of the git repository used in the build job would be included in the image
and potentially deployed without any humans involved. While some developers initially were uncomfortable
with this, it ultimately led to better development practices: Developers merge into the production
branch only code that is truly ready to be deployed.
This practice facilitated rebuilding container images immediately when code is merged into the
production branch and allows us to automatically deploy the new image once it's built. At this point,
almost every project using the automatic rebuild has also enabled automated deployment.
Looking ahead
Today the adoption of both containers and DevOps is still a work in progress for OIT.
Internally we still have to fight the entropy of history even as we adopt new tools and culture.
Our biggest challenge will be convincing people to break away from the repetitive break-fix
mentality that currently dominates their jobs and to focus more on automation. While time is
always short, and the first step always daunting, in the long run adopting automation for day-to-day
tasks will free them to work on more interesting and complex projects.
Thankfully, people within the organization are starting to embrace working in organized or ad
hoc groups of cross-discipline members and developing automation together. This will definitely become
necessary as we embrace automated orchestration and complex systems. A group of talented individuals
who possess complementary skills will be required to fully manage the new environments.
"... All of the hype around software and developers, which tends to significantly skew even the
DevOps discussion, runs the risk of creating the perception that IT ops is just a necessary evil. Indeed,
some go so far as to make the case for a 'NoOps' world in which the public cloud magically takes care
of everything downstream once developers have 'innovated' and 'created'. ..."
"... This kind of view comes about from people looking through the wrong end of the telescope. Turn
the thing around and look up close at what goes on in the world of ops, and you get a much better sense
of perspective. Teams operating in this space are not just there to deploy the next custom software
release and make sure it runs quickly and robustly - in fact that's often a relatively small part of
what they do. ..."
"... And coming back to operations, you are sadly mistaken if you think that the public cloud makes
all challenges and requirements go away. If anything, the piecemeal adoption of cloud services has made
things more complex and unpredictable from an integration and management perspective. ..."
"... There are all kinds of valid reasons to keep an application sitting on your own infrastructure
anyway - regulation, performance, proximity to dependent solutions and data, etc. Then let's not forget
the simple fact that running things in the cloud is often more expensive over the longer term. ..."
Get real – it's not all about developers and DevOps
Listen to some DevOps evangelists talk, and you would get the impression that IT operations teams
exist only to serve the needs of developers. Don't get me wrong, software development is a good competence
to have in-house if your organisation depends on custom applications and services to differentiate
its business.
As an ex-developer, I appreciate the value of being able to deliver something tailored to a specific
need, even if it does pain me to see the shortcuts too often taken nowadays due to ignorance of some
of the old disciplines, or an obsession with time-to-market above all else.
But before this degenerates into an 'old guy' rant about 'youngsters today', let's get back to
the point that I really want to make.
All of the hype around software and developers, which tends to significantly skew even the
DevOps discussion, runs the risk of creating the perception that IT ops is just a necessary evil.
Indeed, some go so far as to make the case for a 'NoOps' world in which the public cloud magically
takes care of everything downstream once developers have 'innovated' and 'created'.
This kind of view comes about from people looking through the wrong end of the telescope.
Turn the thing around and look up close at what goes on in the world of ops, and you get a much better
sense of perspective. Teams operating in this space are not just there to deploy the next custom
software release and make sure it runs quickly and robustly - in fact that's often a relatively small
part of what they do.
This becomes obvious when you recognize how much stuff runs in an Enterprise IT landscape - software
packages enabling core business processes, messaging, collaboration and workflow platforms keeping
information flowing, analytics environments generating critical business insights, and desktop and
mobile estates serving end user access needs - to name but a few.
Vital operations
There's then everything required to deal with security, data protection, compliance and other
aspects of risk. Apart from the odd bit of integration and tailoring work - the need for which is
diminishing with modern 'soft-coded', connector-driven solutions - very little of all this has anything
to do with development and developers.
A big part of the rationale for modernising your application landscape and migrating to the latest
flexible and open software packages and platforms is to eradicate the need for coding wherever you
can. Code is expensive to build and maintain, and the same can often be achieved today through software
switches, policy-driven workflow, drag-and-drop interface design, and so on. Sensible IT teams only
code when they absolutely have to.
And coming back to operations, you are sadly mistaken if you think that the public cloud makes
all challenges and requirements go away. If anything, the piecemeal adoption of cloud services has
made things more complex and unpredictable from an integration and management perspective.
There are all kinds of valid reasons to keep an application sitting on your own infrastructure
anyway - regulation, performance, proximity to dependent solutions and data, etc. Then let's not
forget the simple fact that running things in the cloud is often more expensive over the longer term.
Against this background, an 'appropriate' level of custom development and the selective use of
cloud services will be the way forward for most organisations, all underpinned by a well-run data
centre environment acting as the hub for hybrid delivery. This is the approach that tends to be taken
by the most successful enterprise IT teams, and the element that makes particularly high achievers
stand out is agile and effective IT operations.
This isn't just to support any DevOps agenda you might have; it is demonstrably a key enabler
across the board. Of course if you work in operations, you will know already intuitively know all
this. But if you want some ammunition to spell it out to others who need enlightenment, take a look
at our research report entitled
IT Ops and a Digital
Business Enabler; more than just keeping the lights on . This is based on input from 400 Senior
European IT professionals. ®
I think this is one fad that has run its course. If nothing else, the one thing that cloud has
brought to the software world is the separation of software from the environment it runs in, and
since the the Ops side of DevOps is all about the integration of the platform and software, what
you end up with in a cloudy world is a lot of people looking for a new job.
For decades developers have been ignored by infrastructure vendors because the decision makers
buying infrastructure sit in the infrastructure teams. Now with the cloud etc vendors realize
they will lose supporters within these teams.
So instead - infrastructure vendors target developers to become their next fanboys.
E.g. Dear developer, you won't need to speak to your infrastructure admins anymore to setup
a development environment. Now you can automate, orchestrate the provisioning of your containerized
development environment at the push of a button. Blah blah blah, but you have to buy our storage.
I remember the days when every DBA wanted RAID10 just because thats what the whitepaper recommended.
By that time storage technology had long moved on, but the DBA still talked about Full Stripe
Writes.
Now with DevOps you'll have Developers influencing infrastructure decisions, because they just
learned about snapshots. And yes - it has to be all flash - and designed from the ground up by
millenials that eat avocado.
Re: DevOps was never supposed to replace Operations
Yes, DevOps isn't about replacing Ops. But try telling that to the powers that be. It is sold
and seen as a cost cutting measure.
As for devs learning Ops and vice versa, there are very few on both sides who really understand
what it takes to do the others job. I have a very high regard for Devs, but when it comes to infra,
they are, as a whole, very incompetent. Just like I'm incompetent in Dev. can't have one without
the other. I feel that in time, the pendulum will swing away from cloud as execs and accountants
realize how it isn't really saving any money.
The real question is: Will there be any qualified operations engineers available or will they
all have retired out or have found work elsewhere. It isn't easy to be an ops engineer, takes
a lot of experience to get there, and qualified candidates are hard to come by. Let's face it,
in today's world, its a dying breed.
Nice of you to point out what us in Ops have known all along. I'm afraid it will fall on deaf
ears, though. Until the executives who constantly fall for the new shiny are made to actually
examine business needs and processes and make business decisions based on said.
Our laughable move to cloud here involved migrating off of on prem Exchange to O365. The idea
was to free up our operations team to allow us to do more in house projects. Funny thing is, it
takes more management of the service than we ever did on premises. True, we aren't maintaining
the Exchange infra, but now we have SQL servers, DCs, ADFS, etc, to maintain in the MS cloud to
allow authentication just to use the product. And because mail and messaging is business critical,
we have to have geographically disparate instances of both. And the cost isn't pretty. Yay cloud.
Linus Torvalds believes the technology industry's celebration of innovation is smug, self-congratulatory,
and self-serving. The term of art he used was more blunt: "The innovation the industry talks about
so much is bullshit," he said. "Anybody can innovate. Don't do this big 'think different'... screw
that. It's meaningless.
In a deferential interview at the
Open Source Leadership Summit in California on Wednesday, conducted by Jim Zemlin, executive
director of the Linux Foundation, Torvalds discussed how he has managed the development of the Linux
kernel and his attitude toward work.
"All that hype is not where the real work is," said Torvalds. "The real work is in the details."
Torvalds said he subscribes to the view that successful projects are 99 per cent perspiration,
and one per cent innovation.
As the creator and benevolent dictator of the
open-source Linux kernel , not to mention the
inventor of the Git distributed version control system, Torvalds has demonstrated that his approach
produces results. It's difficult to overstate the impact that Linux has had on the technology industry.
Linux is the dominant operating system for servers. Almost all high-performance computing runs on
Linux. And the majority of mobile devices and embedded devices rely on Linux under the hood.
The Linux kernel is perhaps the most successful collaborative technology project of the PC era.
Kernel contributors, totaling more than 13,500 since 2005, are adding about 10,000 lines of code,
removing 8,000, and modifying between 1,500 and 1,800 daily, according to Zemlin. And this has been
going on – though not at the current pace – for more than two and a half decades.
"We've been doing this for 25 years and one of the constant issues we've had is people stepping
on each other's toes," said Torvalds. "So for all of that history what we've done is organize the
code, organize the flow of code, [and] organize our maintainership so the pain point – which is people
disagreeing about a piece of code – basically goes away."
The project is structured so people can work independently, Torvalds explained. "We've been able
to really modularize the code and development model so we can do a lot in parallel," he said.
Technology plays an obvious role but process is at least as important, according to Torvalds.
"It's a social project," said Torvalds. "It's about technology and the technology is what makes
people able to agree on issues, because ... there's usually a fairly clear right and wrong."
But now that Torvalds isn't personally reviewing every change as he did 20 years ago, he relies
on a social network of contributors. "It's the social network and the trust," he said. "...and we
have a very strong network. That's why we can have a thousand people involved in every release."
The emphasis on trust explains the difficulty of becoming involved in kernel development, because
people can't sign on, submit code, and disappear. "You shoot off a lot of small patches until the
point where the maintainers trust you, and at that point you become more than just a guy who sends
patches, you become part of the network of trust," said Torvalds.
Ten years ago, Torvalds said he told other kernel contributors that he wanted to have an eight-week
release schedule, instead of a release cycle that could drag on for years. The kernel developers
managed to reduce their release cycle to around two and half months. And since then, development
has continued without much fuss.
"It's almost boring how well our process works," Torvalds said. "All the really stressful times
for me have been about process. They haven't been about code. When code doesn't work, that can actually
be exciting ... Process problems are a pain in the ass. You never, ever want to have process problems
... That's when people start getting really angry at each other." ®
"... Most of us use some form of desired state solution already. Desired state solutions basically
involve an OS agent that gets a config from a centralized location and applies the relevant configuration
to the operating system and/or applications. ..."
12 May 2017 at 14:56, Trevor
Pott Infrastructure as code is a buzzword frequently thrown out alongside DevOps and continuous
integration as being the modern way of doing things. Proponents cite benefits ranging from an amorphous
"agility" to reducing the time to deploy new workloads. I have an argument for infrastructure as
code that boils down to "cover your ass", and have discovered it's not quite so difficult as we might
think.
... ... ...
None of this is particularly surprising. When you have an environment where each workload is
a
pet , change is slow, difficult, and requires a lot of testing. Reverting changes is equally
tedious, and so a lot of planning goes into making sure than any given change won't cascade and cause
knock-on effects elsewhere.
In the real world this is really the result of two unfortunate aspects of human nature. First:
everyone hates doing documentation, so it's highly unlikely that in an unstructured environment every
change from the last refresh was documented. The second driver of chaos and problems is that there
are few things more permanent than a temporary fix.
When you don't have the budget for the right hardware, software or services you make do. When
something doesn't work you "innovate" a solution. When that breaks something, you patch it. You move
from one problem to the next, and if you're not careful, you end up with something so fragile that
if you breathe on it, it falls over. At this point, you burn it all down and restart from scratch.
This approach to IT is fine - if you have 5, 10 or even 50 workloads. A single techie can reasonably
be expected to keep that all in their head, know their network and solve any problems they encounter.
Unfortunately, 50 workloads is today restricted to only the smallest of shops. Everyone else is juggling
too many workloads to be playing the pets game any more.
Most of us use some form of desired state solution already. Desired state solutions basically
involve an OS agent that gets a config from a centralized location and applies the relevant configuration
to the operating system and/or applications. Microsoft's group policy can be considered a really
primitive version of this, with System Center being a more powerful but miserable to use example.
The modern friendly tools being Puppet, Chef, Saltstack, Ansible and the like.
Once you have desired state configs in place we're no longer beating individual workloads into
shape, or checking them manually for deviation from design. If all does what it says on the tin,
configurations are applied and errors thrown if they can't be. Usually there is some form of analysis
software to determine how many of what is out of compliance. This is a big step forward.
The Technocult, also known as the Machine cult is the semi-offical name given by
The Church
of the Crossed Heart to followers of the Mechanicum faith who supply and maintain virtually all
of the church's technology, engineering and industry.
Although they serve with the Church of the Crossed Heart they have their own version of worship
that differs substantially in theology and ritualistic forms from that of
The Twelve Angels .
Instead the Technocult worships a deity they call the Machine god or Omnissiah. The Technocult believes
that knowledge is divine and comes only form the Omnissiah thus making any objects that demonstrate
the application of knowledge , i.e machinery, or contain it (books) holy in the eyes/optical implants
of the Techcult. The Technocult regard organic flesh as weak and imperfect, with
the Rot being veiwed as a divine
message from the Omnissah demonstrating its weakness, thus making its removal and replacement by
mechanical, bionic parts a sacred process that brings them closer to their god with many of its older
members having very little of their original bodies remaining.
The date of the cults formation is unknown, or a closely guarded secret...
1. Saying you're doing Agile just cos you're doing daily stand-ups. You're not doing agile. There
is so much more to agile practices than this! Yet I'm surprised how often I've heard that story.
It really is remarkable.
... ... ....
3. Thinking that agile is a silver bullet and will solve all your problems. That's so naiive,
of course it won't! Humans and software are a complex mix with any methodology, let alone with an
added dose of organisational complexity. Agile development will probably help with many things, but
it still requires a great deal of skill and there is no magic button.
... ... ...
8. People who use agile as an excuse for having no process or producing no documentation. If documents
are required or useful, there's no reason why an agile development team shouldn't produce them. Just
not all up-front; do it as required to support each feature or iteration. JFDI (Just F'ing Do It)
is not agile!
David, 23 February 2010 at 1:21 am
So agree on number 1. Following "Certified" Scrum Master training (prior to the exam requirement),
a manager I know now calls every regular status meeting a "scrum", regardless of project or methodology.
Somehow the team is more agile as a result.
Ironically he pulled up another staff member for "incorrectly" using the term retrospective.
Andy Till, 23 February 2010 at 9:28 am
I can think of far worse, how about pairing with the guy in the office who is incapable of
compromise?
Steve Watson, 13 May 2010 at 10:06 am
Kelly
Good list!
I like number 9 as I find with testing people think that they no longer need to write proper
test cases and scripts – a list of confirmations on a user story will do. Well, if its a simple
change I guess you can dispense with test scripts, but if its something more complex then there
is no reason NOT to write scripts. If you have a reasonably large team of people who could execute
the tests, they can follow the test steps and validate against the expected results. It also means
that you can sensibly lump together test cases and cover them with one test.
If you dont think about how you will execute them and just tackle them one by one off the confirmations
list, you miss the opportunity to run one test and cover many separate cases, saving time.
I always find test scripts useful if someone different re-runs a test, as they then follow
the same process as before. This is why we automate regression so the tests are executed the same
each time.
John Quincy, 24 October 2011 at 12:02 am
I am not a fan of agile. Unless you have a small group of developers who are in perfect sync
with each other at all times, this "one size fits all" methodology is destructive and downright
dangerous. I have personally witnessed a very good company go out of business this year because
they transformed their development shop from a home-grown iterative methodology to SCRUM. The
team was required to abide by the SCRUM rules 100%. They could not keep up with customer requirements
and produced bug filled releases that were always late. These developers went from fun, friendly,
happy people (pre-SCRUM) [who NEVER missed a date] to bitter, sarcastic, hard to be around 'employees'.
When the writing was on the wall a couple of months back, the good ones got the hell out of there,
and the company could not recover.
Some day, I'm convinced that Beck through Thomas will proclaim that the Agile Manifesto was
all a big practical joke that got out of control.
This video pretty much lays out the one and only reason why management wants to implement Agile:
It's a cycle of violence when a project claims to be Agile just because of standups and iterations
and don't think about resolving the core challenges they've had to begin with. People are left
still battling said challenges and then say that Agile sucks.
"... while DevOps is appealing to startups, there are important stumbling blocks in the path of DevOps adoption within the enterprise. ..."
"... The tools needed to implement a DevOps culture are lacking. While some of the tools can be provided by vendors and others can be created within the enterprise, a process which takes a long period of time, "there is a marathon of organizational change and restructuring that must occur before such tools could ever be bought or built." ..."
Rachel Shannon-Solomon suggests that most enterprises are not ready for DevOps, while Gene Kim says
that they must make themselves ready if they want to survive.
While acknowledging that large companies such as Google and Facebook benefit from implementing
DevOps, and that "there is no lack of appetite to experiment with DevOps practices" within "Fortune
500s and specifically financial services firms", Shannon-Solomon remarks that "there are few true
change agents within enterprise IT willing to affect DevOps implementations."
Shehas come to this conclusion basedon "conversations with startup founders, technology incumbents
offering DevOps solutions, and technologists within large enterprises."
Shannon-Solomon brings four arguments to support her position:
Siloed structures and organizational change . According to Shannon-Solomon, enterprises
create siloed structures between teams because that's how large organizations maximize value results,
and one of DevOps'main purpose is to bring thosesilos down. Sometimes a large company relies on
another vendor for operational support, and such vendors "tend to bring their own degree of siloing."
Also, it is expensive for an enterprise to invest into "holistic solutions to facilitate DevOps"
after they "invested heavily on integration for existing solutions."
Buy vs. build.The tools needed to implement a DevOps culture are lacking. While
some of the tools can be provided by vendors and others can be created within the enterprise,
a process which takes a long period of time, "there is a marathon of organizational change and
restructuring that must occur before such tools could ever be bought or built."
Vendors of DevOps solutions acknowledge that when selling to the enterprise, they are trying
to sell a cultural revolution. According to Shannon-Solomon, it is hard to introduce DevOps
to development teams because Vendors need to first win over individual developers with the efficiency
of their solution, for example, by initially eschewing procurement and offering a sandbox environment
directly to developers to test out the environment.
Selling DevOps toolkits to the enterprise means facing the well-documented challenges of navigating
procurement and a mindset currently more primed to build than buy DevOps tools.
Return on investment . Shannon-Solomon cites a senior IT professional working at an
investment bank saying that his company "has not been very successful at tracking DevOps projects
occurring within individual business units using homegrown tools, and any evaluation of a project's
success has been limited to anecdotal assessments and perceived results."
Shannon-Solomonends her post wondering "how long will it be until enterprises are forced to accept
that they must accelerate their experiments with DevOps" and hoping that "more individual change
agents within large organizations may emerge" in the future.
No methodology can substitute good engineers who actually talk to and work with each other. Good
engineers can benefit from a better software development methodology, but even the best software development
methodology is powerless to convert mediocre developers into stars.
Notable quotes:
"... disorganized and never-ending ..."
"... Agile is to proper software engineering what Red Bull Flugtag is to proper aeronautic engineering.... ..."
"... As TFA points out, that always works fine when your requirements are *all* known an are completely static. That rarely happens in most fields. ..."
"... The problem with Agile is that it gives too much freedom to the customer to change their mind late in the project and make the developers do it all over again. ..."
"... If you are delivering to customer requests you will always be a follower and never succeed. You need to anticipate what the customers need. ..."
"... It frequently is. It doesn't matter what methodology you use -- if you change major features/priorities at the last minute it will cost multiple times as much. Yet frequently customers expect it to be cheap because "we're agile". And by accepting that change will happen you don't push the customers to make important decisions early, ensuring that major changes will happen, instead of just being possible. ..."
"... The problem with all methodologies, or processes, or whatever today's buzzword is, is that too many people want to practice them in their purest form. Excessive zeal in using any one approach is the enemy of getting things done. ..."
"... On a sufficiently large project, some kind of upfront design is necessary. ..."
"... If you insist on spinning back every little change to a monstrously detailed Master Design Document, you'll move at a snail's pace. As much as I hate the buzzword "design patterns", some pattern is highly desirable. ..."
"... there is no substitute for good engineers who actually talk to and work with each other. ..."
"... If you don't trust those people to make intelligent decisions (including about when things do have to be passed up) then you've either got the wrong people or a micromanagement fetish ..."
"... The problem the article refers to about an upfront design being ironclad promises is tough. Some customers will work with you, and others will get their lawyers and "systems" people to waste your time complaining about every discrepancy, ..."
"... In defense everything has to meet spec, but it doesn't have to work. ..."
"... There is absolutely no willingness to make tradeoffs as the design progresses and you find out what's practical and necessary and what's not. ..."
"... I'll also admit that there is a tendency to get sloppy in software specs because it is easier to make changes. Hardware, with the need to order materials, have things fabbed, tape out a chip, whatever, imposes a certain discipline that's lacking when you know you can change the source code at anytime. Being both, I'm not saying this is because hardware engineers are virtuous and software engineers are sloppy, but because engineers are human (at least some of them). ..."
"... Impressive stuff, and not unique to the space shuttle. Fly-by-wire systems are the same way. You're talking DO-178B [wikipedia.org] Level A stuff. It works, and it's very very expensive. If it was only 10x the cost of normal software development I'd be amazed. I agree that way too much software is poorly planned and implemented crap, and part of the reason is that nobody wants realistic cost estimates or to make the difficult decisions about what it's supposed to do up-front. But what you're talking about is aerospace quality. You couldn't afford a car or even a dishwasher made to those standards. ..."
This article discusses how some experienced developers have changed that perception. '... She's
been frustrated by her Agile experiences - and so have her clients.
"There is no process. Things fly all directions, and despite SVN [version control] developers
overwrite each other and then have to have meetings to discuss why things were changed. Too many
people are involved, and, again, I repeat, there is no process.' The premise here is not that Agile
sucks - quite to the contrary - but that developers have to understand how Agile processes can make
users anxious, and learn to respond to those fears. Not all those answers are foolproof.
For example: 'Detailed designs and planning done prior to a project seems to provide a "safety
net" to business sponsors, says Semeniuk. "By providing a Big Design Up Front you are pacifying this
request by giving them a best guess based on what you know at that time - which is at best partial
or incorrect in the first place." The danger, he cautions, is when Big Design becomes Big Commitment
- as sometimes business sponsors see this plan as something that needs to be tracked against.
"The big concern with doing a Big Design up front is when it sets a rigid expectation that
must be met, regardless of the changes and knowledge discovered along the way," says Semeniuk.' How
do you respond to user anxiety from Agile processes?"
Shinobi
Agile summed up (Score:5, Funny)
Agile is to proper software engineering what Red Bull Flugtag is to proper aeronautic engineering....
Nerdfest
Re: doesn't work
As TFA points out, that always works fine when your requirements are *all* known an are
completely static. That rarely happens in most fields.
Even in the ones where it does it's usually just management having the balls to say "No, you
can give us the next bunch of additions and changes when this is delivered, we agreed on that".
It frequently ends up delivering something less than useful.
MichaelSmith
Re: doesn't work (Score:5, Insightful)
The problem with Agile is that it gives too much freedom to the customer to change their
mind late in the project and make the developers do it all over again.
ArsonSmith
Re: doesn't work (Score:4, Insightful)
...but they can be trusted to say what is most important to them at the time.
No they can't. If you are delivering to customer requests you will always be a follower
and never succeed. You need to anticipate what the customers need. As with the I guess made
up quote attributed to Henry Ford, "If I listened to my customers I'd have been trying to make
faster horses." Whether he said it or not, the statement is true. Customers know what they have
and just want it to be faster/better/etc you need to find out what they really need.
AuMatar
Re: doesn't work (Score:5, Insightful)
It frequently is. It doesn't matter what methodology you use -- if you change major features/priorities
at the last minute it will cost multiple times as much. Yet frequently customers expect it to
be cheap because "we're agile". And by accepting that change will happen you don't push the customers
to make important decisions early, ensuring that major changes will happen, instead of just being
possible.
ebno-10db
Re: doesn't work (Score:5, Interesting)
"Proper software engineering" doesn't work.
You're right, but you're going to the other extreme. The problem with all methodologies,
or processes, or whatever today's buzzword is, is that too many people want to practice them in
their purest form. Excessive zeal in using any one approach is the enemy of getting things done.
On a sufficiently large project, some kind of upfront design is necessary. Spending too much
time on it or going into too much detail is a waste though. Once you start to implement things,
you'll see what was overlooked or why some things won't work as planned. If you insist on
spinning back every little change to a monstrously detailed Master Design Document, you'll move
at a snail's pace. As much as I hate the buzzword "design patterns", some pattern is highly desirable.
Don't get bent out of shape though when someone has a good reason for occasionally breaking that
pattern or, as you say, you'll wind up with 500 SLOC's to add 2+2 in the approved manner.
Lastly, I agree that there is no substitute for good engineers who actually talk to and
work with each other. Also don't require that every 2 bit decision they make amongst themselves
has to be cleared, or even communicated, to the highest levels. If you don't trust those people
to make intelligent decisions (including about when things do have to be passed up) then you've
either got the wrong people or a micromanagement fetish. Without good people you'll never
get anything decent done, but with good people you still need some kind of organization.
The problem the article refers to about an upfront design being ironclad promises is tough.
Some customers will work with you, and others will get their lawyers and "systems" people to waste
your time complaining about every discrepancy, without regard to how important it is. Admittedly
bad vendors will try and screw their customers with "that doesn't matter" to excuse every screw-up
and bit of laziness. For that reason I much prefer working on in-house projects, where "sure we
could do exactly what we planned" gets balanced with the cost and other tradeoffs.
The worst example of those problems is defense projects. As someone I used to work with said:
In defense everything has to meet spec, but it doesn't have to work. In the commercial
world specs are flexible, but it has to work.
If you've ever worked in that atmosphere you'll understand why every defense project costs
a trillion dollars. There is absolutely no willingness to make tradeoffs as the design progresses
and you find out what's practical and necessary and what's not. I'm not talking about meeting
difficult requirements if they serve a purpose (that's what you're paid for) but being unwilling
to compromise on any spec that somebody at the beginning of the project pulled out of their posterior
and obviously doesn't need to be so stringent. An elephant is a mouse built to government specifications.
Ok, you can get such things changed, but it requires 10 hours from program managers for every
hour of engineering. Conversely, don't even think about offering a feature or capability that
will be useful and easy to implement, but is not in the spec. They'll just start writing additional
specs to define it and screw you by insisting you meet those.
As you might imagine, I'm very happy to be back in the commercial world.
Anonymous Coward
Re: doesn't work (Score:2, Interesting)
You've fallen into the trap of using their terminology. As soon as 'the problem' is defined
in terms of 'upfront design', you've already lost half the ideological battle.
'The problem' (with methodology) is that people want to avoid the difficult work of thinking
hard about the business/customer's problem and coming up with solutions that meet all their needs.
But there isn't a substitute for thinking hard about the problem and almost certainly never will
be.
The earlier you do that hard thinking about the customer's problems that you are trying to
solve the cheaper, faster and better quality the result will be. Cheaper? Yes, because bugfixing
that is done later in the project is a lot more expensive (as numerous software engineering studies
have shown) Faster? Yes, because there's less rework. (Also, since there is usually a time = money
equivalency, you can't have it done cheap unless it is also done fast. Higher quality? Yes, because
you don't just randomly stumble across quality. Good design trumps bad design every single time.
... ... ...
ebno-10db
Re: doesn't work (Score:4, Interesting)
Until the thing is built or the software is shipped there are many options and care should
be taken that artificial administrative constraints don't remove too many of them.
Exactly, and as someone who does both hardware and software I can tell you that that's better
understood by Whoever Controls The Great Spec in hardware than in software. Hardware is understood
to have physical constraints, so not every change is seen as the result of a screw-up. It's a
mentality.
I'll also admit that there is a tendency to get sloppy in software specs because it is easier
to make changes. Hardware, with the need to order materials, have things fabbed, tape out a chip,
whatever, imposes a certain discipline that's lacking when you know you can change the source
code at anytime. Being both, I'm not saying this is because hardware engineers are virtuous and
software engineers are sloppy, but because engineers are human (at least some of them).
This is my evidence that "proper software engineering" *can* work. The fact that most businesses
(and their customers) are willing to save money by accepting less from their software is not the
fault of software engineering. We could and did build buildings much faster than we do today,
if you are willing to make more mistakes and pay more in human lives. If established industries
and their customers began demanding software at that higher standard and were willing to pay for
it like it was real engineering, then maybe it would happen more often.
Impressive stuff, and not unique to the space shuttle. Fly-by-wire systems are the same way.
You're talking DO-178B [wikipedia.org] Level A stuff. It works, and it's very very expensive.
If it was only 10x the cost of normal software development I'd be amazed. I agree that way too
much software is poorly planned and implemented crap, and part of the reason is that nobody wants
realistic cost estimates or to make the difficult decisions about what it's supposed to do up-front.
But what you're talking about is aerospace quality. You couldn't afford a car or even a dishwasher
made to those standards.
donscarletti
Re: doesn't work (Score:3)
260 people maintaining 420,000 lines of code, written to precise externally provided specifications
that change once every few years.
This is fine for NASA, but if you want something that does roughly what you need before your competitors
come up with something better, you'd better find some better programmers.
In light of all the hype, we have created a DevOps parody Series – DevOps: Fact or Fiction .
For
those of you who did not see, in October we created an entirely separateblog(inspired by
this ) – however decided that
it is relevant enough to transform into a series on the
AppDynamics Blog . The series will point
out the good, the bad, and the funny about IT and DevOps. Don't take anything too seriously – it's
nearly 100% stereotypes : ).
Stay tuned for more DevOps: Fact or Fiction to come. Here we go
"... Start-ups taught us this. Good developers can be passable DBAs, if need be. They make decent testers, "deployment engineers", and whatever other ridiculous term you'd like to use. Their job requires them to know much of the domain of "lower" roles. There's one big problem with this, and hopefully by now you see it: It doesn't work in the opposite direction. ..."
"... An example will make this more clear. My dad is a dentist running his own practice. He employs a secretary, hygienist, and dental assistant. Under some sort of "DentOps" movement, my dad would be making appointments and cleaning people's teeth while trying to find time to drill cavities, perform root canals, etc. My dad can do all of the other jobs in his office, because he has all the specialized knowledge required to do so. But no one, not even all of his employees combined, can do his job. ..."
"... Such a movement does a disservice to everyone involved, except (of course) employers. What began as an experiment aimed at increasing software quality has become a farce, where the most talented employees are overworked (while doing less, less useful work) and lower-level positions simply don't exist. ..."
"... you're right. Pure DevOps is no more efficient or sensible than pure Agile (or the pure Extreme Programming or the pure Structured Programming that preceeded it). The problem is purists and ideological zealotry not the particular brand of religion in question. Insistence on adherence to dogma is the problem as it prevents adoption of flexible, 'fit for purpose' solutions. Exposure to all the alternatives is good. Insisting that one hammer is ideal for every sort of nail we have and ever will have is not. ..."
"... There are developers who have a decent set of skills outside of development in QA, Operations, DB Admin, Networking, etc. Equally so, there are operations engineers who have a decent set of skills outside of operations in QA, Development, DB Admin, networking, etc. Extend this to QA and other disciplines. What I have never seen is one person who can perform all those jobs outside of their main discipline with the same level of professionalism, experience and acumen that each of those roles require to do it well at an Enterprise/World Class level. ..."
"... I prefer to think of DevOps as more of a full-stack team concept. Applying the full-stack principle at the individual levels is not sustainable, as you point out. ..."
"... DevOps roles are strictly automation focused, at least according to all job specifications I see on the internet. They don't need any development skills at all. To me it looks like a new term for what we used to call IT Operations, but more scripting/automation focused. DevOps engineer will need to know Puppet, Chef, Ansible, OS management, public cloud management, know how to set up monitoring, logging and all that stuff usual sysadmin used to do but in the modern world. In fact I used to apply for DevOps roles but quickly changed my mind as it turned out no companies need a person wearing many hats, it has absolutely nothing to do with creating software. Am I wrong? ..."
There are two recent trends I really hate: DevOps and the notion of the "full-stack" developer.
The DevOps movement is so popular that I may as well say I hate the x86 architecture or monolithic
kernels. But it's true: I can't stand it. The underlying cause of my pain? This fact: not every
company is a start-up, though it appears that every company must act as though they were.
DevOps
"DevOps" is meant to denote a close collaboration and cross-pollination between what were previously
purely development roles, purely operations roles, and purely QA roles. Because software needs to
be released at an ever-increasing rate, the old "waterfall" develop-test-release cycle is seen as
broken. Developers must also take responsibility for the quality of the testing and release environments.
The increasing scope of responsibility of the "developer" (whether or not that term is even appropriate
anymore is debatable) has given rise to a chimera-like job candidate: the "full-stack" developer.
Such a developer is capable of doing the job of developer, QA team member, operations analyst, sysadmin,
and DBA. Before you accuse me of hyperbole, go back and read that list again. Is there any role in
the list whose duties you wouldn't expect a "full-stack" developer to be well versed in?
Where did these concepts come from? Start-ups, of course (and the Agile methodology). Start-ups
are a peculiar beast and need to function in a very lean way to survive their first few years. I
don't deny this . Unfortunately, we've taken the multiple technical roles that engineers at start-ups
were forced to play due to lack of resources into a set of minimum qualifications for the
role of "developer".
Many Hats
Imagine you're at a start-up with a development team of seven. You're one year into development
of a web applications that X's all the Y's and things are going well, though it's always a frantic
scramble to keep everything going. If there's a particularly nasty issue that seems to require deep
database knowledge, you don't have the liberty of saying "that's not my specialty," and handing it
off to a DBA team to investigate. Due to constrained resources, you're forced to take on the role
of DBA and fix the issue yourself.
Now expand that scenario across all the roles listed earlier. At any one time, a developer at
a start-up may be acting as a developer, QA tester, deployment/operations analyst, sysadmin, or DBA.
That's just the nature of the business, and some people thrive in that type of environment. Somewhere
along the way, however, we tricked ourselves into thinking that because, at any one time, a start-up
developer had to take on different roles he or she should actually be all those things
at once.
If such people even existed , "full-stack" developers still wouldn't be used as they should.
Rather than temporarily taking on a single role for a short period of time, then transitioning
into the next role, they are meant to be performing all the roles, all the time . And here's what
really sucks: most good developers can almost pull this off.
The Totem Pole
Good developers are smart people. I know I'm going to get a ton of hate mail, but there is
a hierarchy of usefulness of technology roles in an organization. Developer is at the top, followed
by sysadmin and DBA. QA teams, "operations" people, release coordinators and the like are at the
bottom of the totem pole. Why is it arranged like this?
Because each role can do the job of all roles below it if necessary.
Start-ups taught us this. Good developers can be passable DBAs, if need be. They make decent
testers, "deployment engineers", and whatever other ridiculous term you'd like to use. Their job
requires them to know much of the domain of "lower" roles. There's one big problem with this, and
hopefully by now you see it: It doesn't work in the opposite direction.
A QA person can't just do the job of a developer in a pinch, nor can a build-engineer do the job
of a DBA. They never acquired the specialized knowledge required to perform the role. And
that's fine. Like it or not, there are hierarchies in every organization, and people have different
skill sets and levels of ability. However, when you make developers take on other roles, you don't
have anyone to take on the role of development!
An example will make this more clear. My dad is a dentist running his own practice. He employs
a secretary, hygienist, and dental assistant. Under some sort of "DentOps" movement, my dad would
be making appointments and cleaning people's teeth while trying to find time to drill cavities, perform
root canals, etc. My dad can do all of the other jobs in his office, because he has all the specialized
knowledge required to do so. But no one, not even all of his employees combined, can do his job.
Such a movement does a disservice to everyone involved, except (of course) employers. What
began as an experiment aimed at increasing software quality has become a farce, where the most talented
employees are overworked (while doing less, less useful work) and lower-level positions simply don't
exist.
And this is the crux of the issue. All of the positions previously held by people of various levels
of ability are made redundant by the "full-stack" engineer. Large companies love this, as it means
they can hire far fewer people to do the same amount of work. In the process, though, actual development
becomes a vanishingly small part of a developer's job . This is why we see so many developers
that can't pass FizzBuzz: they never really had to write any code. All too common a question now,
can you imagine interviewing a chef and asking him what portion of the day he actually devotes to
cooking?
Jack of All Trades, Master of None
If you are a developer of moderately sized software, you need a deployment system in place. Quick,
what are the benefits and drawbacks of the following such systems: Puppet, Chef, Salt, Ansible, Vagrant,
Docker. Now implement your deployment solution! Did you even realize which systems had no business
being in that list?
We specialize for a reason: human beings are only capable of retaining so much knowledge. Task-switching
is cognitively expensive. Forcing developers to take on additional roles traditionally performed
by specialists means that they:
aren't spending their time developing
need to keep up with an enormous domain of knowledge
are going to burn out
What's more, by forcing developers to take on "full-stack" responsibilities, they are paying their
employees far more than the market average for most of those tasks. If a developer makes 100K
a year, you can pay four developers 100K per year to do 50% development and 50% release management
on a single, two-person task. Or, simply hire a release manager at, say, 75K and two developers
who develop full-time. And notice the time wasted by developers who are part time release-managers
but don't always have releases to manage.
Don't Kill the Developer
The effect of all of this is to destroy the role of "developer" and replace it with a sort of
"technology utility-player". Every developer I know got into programming because they actually enjoyed
doing it (at one point). You do a disservice to everyone involved when you force your brightest people
to take on additional roles.
Not every company is a start-up. Start-ups don't make developers wear multiple hats by choice,
they do so out of necessity. Your company likely has enough resource constraints without you inventing
some. Please, don't confuse "being lean" with "running with the fewest possible employees". And for
God's sake, let developers write code!
Some background... I started life as a dev (30years ago), have mostly been doing sysadmin and
project tech lead sorts of work for the last 15. I've always assumed the DevOps movement was resulting
in sub-par development and sub-par sysadmin/ops precisely because people were timesharing their
concerns.
But what it does bring to the party is a greater level of awareness of the other guys problems.
There's nothing quite like being rung out of bed at 3am to motivate a developer to improve his
products logging to make supporting it easier. Similarly the admin exposed to the vagaries of
promoting things into production in a supportable, repeatable, deterministic manner quickly learns
to appreciate the issues there. So DevOps has served a purpose and has offered benefits to the
organisations that signed on for it.
But, you're right. Pure DevOps is no more efficient or sensible than pure Agile (or the
pure Extreme Programming or the pure Structured Programming that preceeded it). The problem is
purists and ideological zealotry not the particular brand of religion in question. Insistence
on adherence to dogma is the problem as it prevents adoption of flexible, 'fit for purpose' solutions.
Exposure to all the alternatives is good. Insisting that one hammer is ideal for every sort of
nail we have and ever will have is not.
I'm very disappointed to see this kind of rubbish. It's this type of egocentric thinking and generalization
that the developer is an omniscient deity requiring worshiping and pampering that prevents DevOps
from being successful. Based on the tone and your perspective it sounds like you've been doing
DevOps wrong.
A developer role alone is not the linchpin that keeps DevOps humming - instead it's the respect
that each team member holds for each discipline and each team member's area of expertise, the
willingness of the entire team to own the product, feature delivery and operational stability
end to end, to leverage each others skills and abilities, to not blame Dev or Ops or QA for failure,
and to share knowledge.
There are developers who have a decent set of skills outside of development in QA, Operations,
DB Admin, Networking, etc. Equally so, there are operations engineers who have a decent set of
skills outside of operations in QA, Development, DB Admin, networking, etc. Extend this to QA
and other disciplines. What I have never seen is one person who can perform all those jobs outside
of their main discipline with the same level of professionalism, experience and acumen that each
of those roles require to do it well at an Enterprise/World Class level.
If you're a developer doing QA and operations, you're doing it because you have to, but there
should be no illusion that you're as good in alternate roles as someone trained and experienced
in those disciplines. To do so is a disservice to yourself and your organization that signs your
paycheck. If you're in this situation and you'd prefer making a difference rather than spewing
complains, I would recommend talking to your manager and above about changing their skewed vision
of DevOps. If they aren't open to communication, collaboration, experimentation and continual
improvement, then their DevOps vision is dysfunctional and they're not supporting DevOps from
the top down. Saying your DevOps and not doing it is *almost* more egregious than saying the developer
is the top of a Totem Pole of existence.
he prefaced it with 'crybabies please ignore' It's his opinion. That everyone but the lower totem
pole people agree with so.. agree to disagree. I also don't think being at the bottom of the totem
pole is a big f'in deal. If you're getting paid.. embrace it! So many other ways to enjoy life!
The top dog people have all the pressure and die young! 99% of the people on earth dont know the
difference between one nerd and another. And other nerds are always going to be egomaniacs who
will find some way to justify their own superiority no matter what your achievements. So this
kind of posturing is a waste of time.
I think there's a problem with your definition of DevOps. It doesn't mean developers have to be
"full-stack" or do ops stuff. And it doesn't mean "act like a startup." It simply means, at its
basis, that Developers and Operations work well together and do not have any communication barriers.
This is why I hate DevOps as a title or department, because DevOps is a culture.
Let's take your DentOps example. The dentist has 3 support staff. What if they rarely spoke
to the dentist? What if they were on different floors of the building? What if the dentist wrote
an email about how teeth should be cleaned and wasn't available to answer questions or willing
to consider feedback? What if once in a while the dentist needed to understand enough about the
basics of appointment scheduling to point out problems with the system? Maybe appointments are
being scheduled too close together. Would the patients get backed up throughout the day because
that's the secretary's problem? Of course not. Now we'd be getting into a more accurate analogy
to DevOps. If anything a dentist's office is ALREADY "DentOps" and the whole point of DevOps is
to make the dev/ops interaction work in a logical culture that other industries (like dentists)
already use!
I would tend to agree with some of that. Being able to trouble shoot network issues using monitoring
tools like Fiddler is a good thing to be aware of. I can also see a lot of companies using it
as a way to make one person do everything. Moreover, there are probably folks out there that perpetuate
that behavior by taking on the machismo argument.
By saying that if I can do it that you should be able to do it too or else you're not as good
of a developer as I am. I have never heard anyone outright claim this, but I've seen this attitude
time and time again from ambitious analysts looking to get a leg up, a pay raise, and a way to
template their values on the rest of the team. One of the first things that you're taught as a
dev is that you can't hope to know it all.
Your responsibility first and foremost as a developer is the stability and reliability of your
code and the services that you provide. In some industries this is literally a matter of life
and death(computers in your car, mission critical medical systems). It doesn't work for everyplace.
I wouldn't want to pay a receptionist 200k a year like a dentist though. Learn to hire better
receptionists. Even a moderately charming woman can create more customer loyalty, and cheaper,
than the best dentist in the world. I want my dentist to keep quiet and have a steady hand. I
want my receptionist to engage me and acknolwedge my existence.
I want my secretary to be a multitasking master. I want my dentist not to multitask at all
- OUCH!
Good points, I tend to agree. I prefer to think of DevOps as more of a full-stack team concept.
Applying the full-stack principle at the individual levels is not sustainable, as you point out.
The full-stack DevOps team will have team members with primary skills in either of the traditional
specialties, and will, over time, develop decent secondary skills. But the value is not in people
constantly content switching - that actually kills efficiency. The value is in developers understanding
and developing an open relationship with testing and operations - and vice versa. And this cooperation
is inhibited by putting people in separate teams with conflicting goals. DevOps in practice is
not a despecialization. It's bringing the specialists together.
The more isolated or silo'd developers become, the less they realize what constitutes delivering
software, and the more problems are contributed to the IT process of test/build/release/scale/monitor,
etc. Writing code is a small fraction of that delivery process. I've written about the success
of devops and microservices that touches on this stuff because they're highly related. The future
success of devops/microservices/cloud/etc isn't related to technology insofar as it is culture:
http://blog.christianposta....
Great article and you're definitely describing one form of dysfunctional organisation where DevOps,
Agile, Full Stack, and every other $2 word has been corrupted to become a cost cutting justification;
cramming more work onto people who aren't skilled for it, and eho end up not having any time to
do what they were hired as experts for!
But I'd also agree with other posters that it's a little developer centric. I'm a terrible
programmer and a great DBA. I can tell you most programmers who try to be DBAs are equally terrible.
It's definitely not "doing the job of the receptionist" 😄
And we shouldn't forget what DevOps is meant to be about; teams making sure nobody gets called
at night to fix each other's messes. That means neither developers with shitty deployments straight
to production nor operations letting the disks silently fill because "if it ain't C: it ain't
our problem."
I know of 0 developers that can manage a network of any appreciable scale.
In cloud and large enterprise networks, if there were a totem (which there isn't) using your
methodology would place the dev under the network engineer. Their software implements the protocol
and configuration intent of the NE. Good thing the whole concept is a pile of rubbish. I think
you fell into the trap you called out which is thinking at limited scale.
It's true. We can all create LAN's at home but I wouldn't dare f with a corporate network and
risk shutting down amazon for a day. Which seems to happen quite a bit.... maybe they're DEVOPPING
a bit too much.
Jeff Knupp is to one side of the spectrum. DevOps Reaper is to the other side.
Enno is more
attune to what is really going on. So I won't repeat any of those arguments.
However I will ask you to put me in a box. What am I?
I graduated as a Computer Engineer (hybrid between Electrical Engineering and Computer Science).
I don't say that anymore as companies have no idea as to what that means. So I called myself a
Digital Electronics and Software Engineer for a while. The repeated question was all too often:
"So what are you, software or hardware?"
I spent my first few years working down from board design, writing VHDL and Verilog, to embedded
software in C and C++, then algorithms in optimization with the CUDA framework in C, with C++
wrappers and C# for the logic tier. Then worked another few year in particle physics with C++
compute engines with x86 assembly declarations for speed and C# for WPF UIs.
After that I went to work for a wind turbines company as system architect where is was mostly
embedded and programming ARM Cortex microprocessors, high power electronics controls, custom service
and diagnostics tools in C#. Real-time web based dashboards with Angular, Bootrap, and the likes
for a good looking web app.
Nowadays I'm working with mobile first web applications that have a massive backend to power them.
It is mostly a .NET stack form Entity Framework, to .NET WebAPI, to Angular power font ends. This
company is not a start up but it is a small company. Therefore I wear the many hats. I introduced
the new software life cycle with includes continuous integration and continuous deployment. Yes,
I manage build servers, build tools, I develop, I'm QA, I'm a tester, I'm a DBA., I'm the deployment
and configuration manager.
If you are wondering I have resorted to start calling a full stack developer. It has that edgy
sound that companies like to hear. I'm still a young developer. I've only been developing for
10 years.
In my team we are all "Jack of all Trades" and "Masters of Many". We switch tasks and hats
because it is fun and keep everyone from getting bored/stuck. Our process is called "Best practices
that work for this team".
So, I think of myself as a software engineer. I think I'm a developer. I think I'm DevOps,
I think I'm QA.
Lets start with that DevOps didn't come from startups. It came from Boeing mainly, and a few other
major blue chip IT shops, investing heavily in systems management technology around the turn of
the century. The goal at the time was simply to change the ratio of servers to IT support personnel,
and the re-thinking and re-organizing of development and operations into one organization with
one set of common goals. The 'wearing many hats' thing you discuss is a feature of startups, but
that feature is independent of siloed or integrated organizations.
I prefer the 'sportzing' analogy of basketball and football. Football has specialist teams
that are largely functionally independent because they focus on distinct goals. Basketball has
specialist positions, but the whole team is focused on the same goals. I'm not saying one sport
is better than the other. I am saying the basketball mentality works better in the IT environment.
Delivering the product or service to the customer is the common goal that everyone should be thinking
about, and how the details of their job fits into that overall picture. It sounds like to me you
are really saying "Hey, its my job and only my job to think about how it all fits together and
works"
Secondly, while it is pretty clear that the phrase 'full stack engineer' is about as useful
as "Cloud Computing", your perspective that somehow developers are the 'top' of the tree able
to do any job is very mistaken. There are key contributors from every specialty who have that
ability, and more useful names for them are things like "10x", or "T-shaped". Again, you are describing
a real situation, but correlating it with unrelated associations. It is just as likely, and just
as valuable, to find an information architect who can also code, or a systems admin that can also
diagnose database performance, or an electrician that can also hang sheetrock. Those people do
fit your analogy of 'being on top', because they are not siloed and stovepiped into just their
speciality.
The DevOps mindset fosters this way of thinking, instead of the old and outdated specialist
way of thinking you are defending. Is it possible your emotional reaction is fear based against
the possibility that your relative value will decrease if others start thinking outside their
boxes?
Interesting to note that Agile also started at Boeing, but 10 years earlier. I live in the
startup world of Seattle, but know my history and realize that much of what appears new is actually
just 'new to you'(or me), and that most of cutting edge technology and thinking is just combining
ideas from other industries in new ways.
The problem is that developers are trained to crank out code and hope that QA teams will find
problems, many times not even sure how to catch holes. DevOps trains people to think critically
and do both. It isn't killing developers, it is making them look like noobs while phasing them
out.
Yeah, good luck with that attitude. Your company's gonna have a good'ole time looking for and
keeping new developer talent. Because as we all know, smart people love working with dummies.
I'd love to see 'your QA' team work on our 'spatial collision algorithm' and make our devs "look
like noob". You sound like most middle management schmucks.
Funniest article so far on full stack. It's a harsh reality for devs, because were asked to do
everything, know everything, so how can you really believe QA or DBA can do the job of someone
like that? There is a crazy amount of hours a fullstack dev invests in aquiring that kind of knowledge,
not to mention some people are also talented at their job. Imagine trying to tell the QA to do
that? Maybe for a few hours someone can be a backup just in case something happens, but really
it's like replacing the head surgeon.
The best skill you can learn in your coding career is your next career. Noone wants a 45 year
old coder.
I see so much time wasted learning every new thing when you should just be plugging away to
get the job done, bank the $$, and move on. All your accumulated skills will be worthless in a
decade or so, and your entire knowledge useless in 2 decades. My ability to turn a wrench is what's
keeping me from the poor house. And I have a engineering degree from UIUC! I also don't mind.
Think about a 100 week as a plumber with OT in a reasonably priced neighborhood, vs a coder. Who
do you think is making more? Now I'm not saying you cant survive into your 50's programming, but
typically they get retired forcefully, and permanently.. by a heart attack!
But rambling aside.. the author makes a good point and i think is the future of big companies
in tech. The current model is driven by temporary factors. Ideally you'd have a specialized workforce.
But I think that as a programmer you are in constant fear of being obsolete so you don't want
to be pigeon-holed. It's just not mathematically possible to have that 10,000 hour mastery in
50 different areas.. unless you are Bill Murray in Groundhog Day.
A developer who sees himself at the top of a pyramid. Not surprising, your myopic and egotistical
view. I laugh at people who code a few SELECT statements and think they can fill the DBA role.
HA HA HA. God, the arrogance. "Well it worked on my machine." - How many sys admins have heard
this out of a developers mouth. Unfortunately, projects get stuck with supporting such issues
because that very ego has led the developer too far down the road to turn back. They had no common
sense or modesty to call on the knowledge of their Sys Ops team to help design the application.
I interview jobs candidates all the time calling themselves full stack simply because they compliment
their programming language of choice with a mere smattering of knowledge in client-side technologies
and can write a few SQL queries. Most developers have NO PERCEPTION of the myriad intricacies
it takes to get an application from their unabated desktop with its FULL ADMIN perms and "unlimited
resources", through a staging/QA environment, and eventually to the securely locked down production
system with limited and perhaps shared or hosted limited resources. Respect of your support teams,
communication and coordination, and the knowledge that you do not know it all. THAT'S being Full
Stack and DevOps sir.
there's always that one query that noone can do in a way that takes less than 2 hours until u
pass it off to a real DBA.. its the 80/20 rule basically. I truly dont believe 'full stack' exists.
It's an illusion. There's always something that suffers.
The real problem is smart people are in such demand we're forced to adapt to this tribal pre-civilization
hodgepodge. Once the industry matures, it'll disappear. Until then they will think they re-invented
the wheel.
'm confused here. DevOps roles are strictly automation focused, at least
according to all job specifications I see on the internet. They don't need any development skills
at all. To me it looks like a new term for what we used to call IT Operations, but more scripting/automation
focused. DevOps engineer will need to know Puppet, Chef, Ansible, OS management, public cloud management,
know how to set up monitoring, logging and all that stuff usual sysadmin used to do but in the modern
world. In fact I used to apply for DevOps roles but quickly changed my mind as it turned out no companies
need a person wearing many hats, it has absolutely nothing to do with creating software. Am I wrong?
It depends on what you mean with development skills. Have you ever tried to automate the deployment
of a large web application? In fact the scripts that automate the deployment of large scalable
web applications are pretty complex softwares which require in-depth thinking and should follow
all the important principles a good developers should know: components isolation, scalability,
maintainability, extensibility, etc..
Successful DevOps doesn't mean a full stack developer does it all, that's only true for a broken
company that succeeded despite bad organization. For example, Twitter's Dev only culture is downright
sick, and ONLY works because they are in the tech field. Mind you, I still believe personally
that it works for them DESPITE its unbalanced structure. In other words, bad DevOps means the
Dev has no extra resources and just more requirements, yea that sucks!....
BUT, on the flip,
Infrastructure works with QA/Build to define supportable deployment standards, they gotta learn
all the automatic bits and practice using them. Now Devs have to package all their applications
properly, in the formats supported by QABuild's CI and Repositories (that 'working just fine'
install script definitly doesn't count). BUT the Dev's get pre-made CI-ready examples, and if
needed, code-migration assistance from the QA/Build team. Pretty soon they learn how to package
that type of app, like a J2EE Maven EAR, or a Webdeploy to IIS.... and the rest should be hadled
for them, as automaticlly as possible by the proactive operations teams.
Make sense? This is how its supposed to work, It sounds like your left alone in a terrible
Dev Only/heavy world. The key to DevOps that is great, and everybody likes, vs. more work... is
having a very balanced work flow between the teams, and making sure the pass-off points are VERY
well defined. Essentially it requires management that cut the responsibility properly, so they
have a shared interest in collaborating. In a Dev heavy organization, the Devs can just throw
garbage over the wall, and operations has to react to constant problems... they start to hate
each other and ..... Dev managers get the idea that they can cut out ops if they do "DevOps",
so then they throw it all at you like right now.
I see in this post so much rubbish and narrow mindedness, so much of the exact stuff that is
killing any type of companies. In the last 10 years I had many roles that required of me, as a
system engineer, to come in and straighten out all kind of really bad compromises developers did
just to make stuff work.
The role never shows the level of intelligence or capabilities. I've seen so many situations
in the last 10 years when smart people with the wrong attitude and awareness are too smart for
anyone's good and limited people still providing more value than a very smart ones acting as if
he is too smart to even have a conversation about anything.
This post is embarrassing for you Jeff, I am sorry for you man.... you just don't get it!
A developer do not have to do full stack, the developer can continue with development, but has
to adopt some things for packaging,testing, and how it is operated.
Operations can continue with operations, but has to know how things are built and packaged.
Developers and operations needs to share things like use the same application server for example.
Developer needs to understand how it is operated to make sure that the code is written in a proper
way. Operations needs to adopt in the need for fast delivery and be able to support a controlled
way of deploying daily into production.
Here is a complementing post I have around the topic
http://bit.ly/1r3iVff
I will share my experience , I started off my career teaching Programming which included database
programming (Oracle-pl/sql, SQL Server-transact sql) which gave good insights into database internals
which landed me into DBA world for last 10 years . During these 10 years where i have worked in
technology companies regarded as top-notch , I have seen very smart Developers writing excellent
application codes but missing out on writing optimized piece to interact with the database. Hence,
I think each Job has a scale and professionals of any group can not do what the top professionals
of other group can do. I have seen Developers with fairly good database internals knowledge and
I have dbas writing code for their automation which can compares well with features of some commercial
database products like TOAD. So , generalization like this does not hold.
The idea that there is a hierachy of usefulness is bunk. Most developers are horrible at operations
because they dislike it. Most sysadmins and DBAs are horrible at coding because they dislike it.
People gravitate to what interests them and a disinterested person does a much poorer job than
an interested one. DevOps aims to combine roles by removing barriers, but there are costs to quality
that no one likes to talk about. Using your hierarchy example most doctors could obtain their
RN but they would not make good nurses.
This is an excellent article on the general concepts of DevOps and the DevOps movement. It helps
to identify the cultural shifts required to facilitate proper DevOps implementations. I also write
about DevOps.. I authored a book on implementing CI, CD and DevOps related functions within an
organization and it was recently published. The book is aptly titled Mastering Jenkins (
http://www.masteringjenkins... ) and aims to codify not only the architectural implementations
and requirements of DevOps but the cultural shift needed to propery advocate for the proper adoption
of DevOps practices. Let me know what you think.
I agree. Although I'm not in the business (yet), I will be soon. What I've noticed just playing
around with Vagrant and Chef, Puppet, Ansible is the great amount of time to try and master just
one of these provisioners. I can't imagine being responsible for all these roles you spoke of
in the article. How can one possibly master all of them, and be good at any of them?
hmmm.... users & business see as one application.... for them how it was developed, deployed does
not matter.... IT is an enabler by definition... so DevOps is mostly about that... giving one
view to the customer; quick changes, stable changes, stable application....
Frankly, DevOps is not about developers or testers... it is about the right architecture, right
framework... developers/testers anyways do what is there in the script... DevOps is just a new
scripts to them.
For right DevOps, you need right framework. architecture for the whole of the program; you
need architecture which is built end to end and not in silos...
Software Developer write code that business/customer use
Test Developer write test code to test SUT
Release Developer write code to automate release process
Infrastructure developer write code to create infrastructure automatically.
Performance Developer writes code to performance test the SUT
Security Developer writes code to scan the SUT for security
Database Developer write code for DB
So which developer are you thinking DevOps going to kill?
Today's TDD world, a developer (it could be anyone above) needs to get out of their comfort
zone to makes sure they write a testable, releasable, deployable, performable, security complaint
and maintainable code.
DevOps brings all this roles together to collaborate and deliver.
Why wouldn't they be? What are the basic responsibilities that make for a passable DBA and which
of those responsibilities cannot be done by a good developer? Say a good developer has just average
experience writing stored procs, analyzing query performance, creating (or choosing not to create,
for performance reasons) indexes, constraints and triggers, configuring database access rights,
setting up regular backups, regular maintenance (ex. rebuilding indexes to avoid fragmentation)...
just to name a few.
I'm sure there's several responsibilities that DBA's have that developers
would have very little to no experience in, but we're talking about making for a passable DBA.
Developers may not be as good at the job as someone who specializes in it for a living, but the
author's wording seems to have been chosen very carefully.
Yup, I see lots of people trying to defend the DBA as a thing, just like people keep
trying to defend the traditional sysadmin as a thing. I started my career as a sysadmin in the 90s,
but times have changed and I don't call myself a sysadmin anymore, because that's not what I do.
Now I'm a Systems Engineer/SRE. My mode of working isn't slamming software together, but engineering
automation to do it for me.
But I also do QA, Data storage performance analysis, networking, and [have a] deep knowledge of the applications
I support.
"... The Emergence of the "DevOps' DevOp", a pseudo intellectual loudly spewing theories about distantly unrelated fields that are entirely irrelevant and are designed to make them feel more intelligent and myself more inadequate ..."
"... "The Copenhagen interpretation certainly applies to DevOps" ..."
"... "I'm modeling the relationship between Dev and Ops using quantum entanglement, with a focus on relative quantum superposition - it's the only way to look at it. Why aren't you?" ..."
"... Enterprise Architects. They used to talk about the "Enterprise Continuum". Now they talk about "The Delivery Continuum" or the "DevOps Continuum". ..."
DevOps and I sort of have a love/hate relationship. DevOps is near and dear to our heart here at
UpGuard and there are plenty
of things that I love about it . Love it or hate it, there is little doubt that it is here to
stay. I've enjoyed a great deal of success thanks to agile software development and DevOps methods,
but here are 10 things I hate about DevOps!
#1 Everyone thinks it's about Automation.
#2 "True" DevOps apparently have no processes - because DevOps takes care of that.
#3 The Emergence of the "DevOps' DevOp", a pseudo intellectual loudly spewing theories about
distantly unrelated fields that are entirely irrelevant and are designed to make them feel more intelligent
and myself more inadequate:
"The Copenhagen interpretation certainly applies to DevOps"
"I'm modeling the relationship between Dev and Ops using quantum entanglement, with a focus
on relative quantum superposition - it's the only way to look at it. Why aren't you?"
#4 Enterprise Architects. They used to talk about the "Enterprise Continuum". Now they talk
about "The Delivery Continuum" or the "DevOps Continuum". How about talking about the business
guys?
#5 Heroes abound with tragic statements like "It took 3 days to automate everything.. it's great
now!" - Clearly these people have never worked in a serious enterprise.
#6 No-one talks about Automation failure...it's everywhere. i.e Listen for the words "Pockets
of Automation". Adoption of technology, education and adaptation of process is rarely mentioned (or
measured).
#7 People constantly pointing to Etsy, Facebook & Netflix as DevOps. Let's promote the stories
of companies that better represent the market at large.
#8 Tech hipsters discounting, or underestimating, Windows sysadmins. There are a lot of them and
they better represent the Enterprise than many of the higher profile blowhards.
#9 The same hipsters saying their threads have filled up with DevOps tweets where there were none
before.
#10 I've never heard of a Project Manager taking on DevOps. I intend on finding one.
What do you think - did I miss anything? Rants encouraged ;-) Please add your comments.
"... DevOps. The latest software development fad. ..."
"... Continuous Delivery (CD), the act of small, frequent, releases was defined in detail by Jez Humble and Dave Farley in their book – Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. The approach makes a lot of sense and encourages a number of healthy behaviors in a team. ..."
"... The problem is we now have teams saying they're doing DevOps. By that they mean is they make small, frequent, releases to production AND the developers are working closely with the Ops team to get things out to production and to keep them running. ..."
"... Well, the problem is the name. We now have a term "DevOps" to describe the entire build, test, release approach. The problem is when you call something DevOps anyone who doesn't identify themselves as a dev or as Ops automatically assumes they're not part of the process. ..."
DevOps. The latest software development fad. Now you can be Agile, use Continuous Delivery, and believe
in DevOps.
Continuous Delivery (CD), the act of small, frequent, releases was defined in detail by Jez Humble
and Dave Farley in their book – Continuous Delivery: Reliable Software Releases Through Build, Test,
and Deployment Automation. The approach makes a lot of sense and encourages a number of healthy
behaviors
in a team. For example, frequent releases more or less have to be small. Small releases are easier
to understand, which in turn increases our chances of building good features, but also our chances
of testing for the right risks. If you do run into problems during testing then it's pretty easy
to work out the change that caused them, reducing the time to debug and fix issues.
Unfortunately, along with all the good parts of CD we have a slight problem. The book focused
on the areas which were considered to be the most broken, and unfortunately that led to the original
CD description implying "Done" meant the code was shipped to production. As anyone who has ever worked
on software will know, running code in production also requires a fair bit of work.
So, teams started adopting CD but no one was talking about how the Ops team fitted into the release
cycle. Everything from knowing when production systems were in trouble, to reliable release systems
was just assumed to be fully functional, and unnecessary for explanation.
To try to plug the gap DevOps rose up.
Now, just to make things even more confusing. Dave Farley later said that not talking about Ops
was an omission and CD does include the entire development and release cycle, including running in
production. So DevOps and CD have some overlap there.
DevOps does take a slightly different angle on the approach than CD. The emphasis for DevOps is
on the collaboration rather than the process. Silos should be actively broken down to help developers
understand systems well enough to be able to write good, robust and scalable code.
So far so good.
The problem is we now have teams saying they're doing DevOps. By that they mean is they make small,
frequent, releases to production AND the developers are working closely with the Ops team to get
things out to production and to keep them running.
Sounds good. So what's the problem?
Well, the problem is the name. We now have a term "DevOps" to describe the entire build, test,
release approach. The problem is when you call something DevOps anyone who doesn't identify themselves
as a dev or as Ops automatically assumes they're not part of the process.
Seriously, go and ask your designers what they think of DevOps. Or how about your testers. Or
Product Managers. Or Customer Support.
And that's a problem.
We've managed to take something that is completely dependant on collaboration, and trust, and
name it in a way that excludes a significant number of people. All of the name suggestions that arise
when you mention this are just ridiculous. DevTestOps? BusinessDevTestOps? DesignDevOps? Aside from
just being stupid names these continue to exclude anyone who doesn't have these words in their title.
So do I hate DevOps? Well no, not the practice. I think we should always be thinking about how
things will actually work in production. We need an Ops team to help us do that so it makes total
sense to have them involved in the process. Just take care with that name.
Is there a solution? Well, in my mindwe're still talking about collaboration above all else. Thinking
about CD as "Delivery on demand" also makes more sense to me. We, the whole team, should be ready
to deliver working software to the customer when they want it. By being aware of the confusion, and
exclusion that some of these names create we can hopefully bring everyone into the project before
it's too late.
DevOps initiatives include a range of technologies and methodologies spanning the software delivery
process. IT leaders and DevOps practitioners should proactively understand the readiness and capabilities
of technology to identify the most appropriate choices for their specific DevOps initiative.
Table of Contents
Analysis
What You Need to Know
The Hype Cycle
The Priority Matrix
On the Rise
DevOps Toolchain Orchestration
DevOps Toolchain
Mobile DevOps
Web-Scale Operations
Continuous Delivery
Lean IT
User and Entity Behavior Analytics
Continuous Experience
Application Release Automation
Mediated APIs
At the Peak
Web-Scale Development
Container Management
Web-Scale Application Architecture
Microservices
Crowdtesting
Enterprise-Class Agile Development
Software-Defined Data Center
Continuous Configuration Automation
Behavior-Driven Development
Sliding Into the Trough
Citizen Developers
Configuration Auditing
Software-Defined Networking
Application Testing Services
Climbing the Slope
Application Performance Monitoring Suites
Test Data Management
Appendixes
Hype Cycle Phases, Benefit Ratings and Maturity Levels
The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win (2013) is the third
book by Gene Kim. The business novel tells the story of an IT manager who has ninety days to rescue
an over-budget and late IT initiative, code-named The Phoenix Project. The book was co-authored by
Kevin Behr and George Spafford and published by IT Revolution Press in January 2013.[1][2]
Background
The novel is thought of as the modern day version of The Goal by Eliyahu M. Goldratt.[3] The novel
describes the problems that almost every IT organization faces, and then shows the practices of how
to solve the problems, improve the lives of those who work in IT and be recognized for helping the
business win.[1] The goal of the book is to show that a truly collaborative approach between IT and
business is possible.[4]
Synopsis
The novel tells the story of Bill, the IT manager at Parts Unlimited.[4][5][6] The company's new
IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the
project is massively over budget and very late. The CEO wants Bill to report directly to him and
fix the mess in ninety days or else Bill's entire department will be outsourced. With the help of
a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that
IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking,
Bill must organize work flow, streamline interdepartmental communications, and effectively serve
the other business functions at Parts Unlimited.[7][8]
Reception
The book has been called a "must read" for IT professionals and quickly reached #1 in its Amazon.com
categories.[9][10] The Phoenix Project was featured on 800 CEO Reads Top 25: What Corporate America
Is Reading for June, 2013.[11] InfoQ stated, "This book will resonate at one point or another with
anyone who's ever worked in IT."[4] Jeremiah Shirk, Integration & Infrastructure Manager at Kansas
State University, said of the book: "Some books you give to friends, for the joy of sharing a great
novel. Some books you recommend to your colleagues and employees, to create common ground. Some books
you share with your boss, to plant the seeds of a big idea. The Phoenix Project is all three."[4]
Other reviewers were more skeptical, including the IT Skeptic "Fictionalising allows you to paint
an idealised picture, and yet make it seem real, plausible... Sorry but it is all too good to be
true... none of the answers are about people or culture or behaviour. They're about tools and techniques
and processes." [12] Jez Humble (author of Continuous Delivery) said "unlike real life, there aren't
many experiments in the book that end up making things worse..."
In a recent webinar, XebiaLabs VP of DevOps Strategy Andrew Phillips sat down with Atos Global
Thought Leader in DevOps Dick van der Sar to separate the facts from the fiction. Their findings:
most myths come attached with a small piece of fact and vice versa.
1. DevOps Is Developers Doing Operations: Myth
An integral part of DevOps' automation component involves a significant amount of code. This causes
people to believe Developers do most of the heavy lifting in the equation. In reality, what ends
up happening is due to the amount of infrastructure as Code, Ops begin to look a lot like Dev.
2. Projects Are Dead: Myth
Projects are an ongoing process of evolving systems and failures. To think they can just be handed
off to maintenance forever after completion is simply incorrect. This is only true for tightly scoped
software needs, including systems built for specific events. When you adopt DevOps and Agile, you
are replacing traditional project-based approaches with a focus on product lifecycles.
3. DevOps Doesn't Work in Complex Environments: Myth
DevOps is actually made to thrive in complex environments. The only instance in which it doesn't
work is when unrealistic and/or inappropriate goals are set for the enterprise. Complex environments
typically suffer due to lack of communication about the state of, and changes to, the interconnected
systems. DevOps, on the other hand, encourages communication and collaboration that prevent these
issues from arising.
4. It's Hard to Sell DevOps to the Business: Myth
The benefits to DevOps are closely tied benefiting the business. However, that's hard to believe
when you pitch adopting DevOps as a plan to "stop working on features and sink a lot of your money
into playing with shiny new IT tech." Truth is, DevOps is going to impact the entire enterprise.
This may be the source of resistance, but as long as you find the balance between adoption and disruption,
you will experience a successful transition.
5. Agile Is for Lazy Engineers: Myth
DevOps prides itself on eliminating unnecessary overhead. Through automation, your enterprise
can see a reduction in documentation, meetings, and even manual tasks, giving team members more time
to focus on more important priorities. You know your team is running successfully if their productivity
increases.
Nonetheless, DevOps does not come without its own form of "boring" processes, including test plans
or code audits. Agile may eliminate waste but that doesn't include the tedious yet necessary aspects.
6. If You Can't Code, You Have No Chance in DevOps: Fact
This is only afact because the automation side of DevOps is all Infrastructure as Code (IaC).
This typically requires some sort of software development skill such as modularization, automated
testing, and Continuous Integration (CI) as IaC. Regardless of scale, automating anything will require,
at the very least, software development skills.
7. Managers Disappear: Myth
Rather than disappear, managers take a different role with DevOps. In fact, they are still a necessity
to the team. Managers are tasked with the responsibility of keeping the entire DevOps team on track.
Classic management tasks may seem to disappear but only because the role is changing to be more focused
on empowerment.
8. DevOps or Die: Fact!
Many of today's market leaders already have some sort of advanced DevOps structure in place. As
industries incorporate IT further into their business, we will begin to see DevOps as a basic necessity
to the modern business and those that can't adapt will simply fall behind.
That being said, you shouldn't think of DevOps as the magic invincibility potion that will keep
your enterprise failure free. Rather, DevOps can prevent many types of failure, but there will always
be environment specific threats unique to every organization that DevOps can't rescue you from.
Discover how to optimize your DevOps workflows with our cloud-based automated testing infrastructure,
brought to you in partnership with
Sauce Labs .
Out of these misunderstandings several common myths
have been created. Acceptance of these myths misleads
business further.
Here are some of the most common myths and the facts
that debunk them.
Myth 1: DevOps needs agile.
Although DevOps and agile are terms frequently used
together, they are a long way away from being
synonymous with one another. Agile development refers
to a method of software delivery that builds software
incrementally, whereas DevOps refers not only to a
method of delivery but to a culture, which when
adopted, results in
many business benefits
, including faster software
delivery.
DevOps processes can help to compliment agile
development, but it is not reliant on agile and can
support a range of operation models such as
Waterfall – where build processes can be optimised and accelerated, and automation can be
implemented.
Agile – where heightened communication between
development and operations increases end-product
quality.
Hybrid approach – where speed, quality and
compliance are all increased.
For optimum results, full adoption of the DevOps
philosophy is necessary.
Myth 2: DevOps can't work with legacy.
DevOps is often regarded as a modern concept that
helps forward-thinking businesses innovate. Although
this is true, it can also help those organisations with
long-established, standard IT practices. In fact, with
legacy applications there are usually big advantages to
DevOps adoption.
Managing legacy care and bringing new software to
market quickly; blending stability and agility, is a
frequently encountered problem in this new era of
digital transformation. Bi-modal IT is an approach
where Mode 1 refers to legacy systems focussed on
stability, and Mode 2 refers to agile IT focussed on
rapid application delivery. DevOps principles are often
included exclusively within Mode 2, but automation and
collaboration can also be used with success within Mode
1 to increase delivery speed whilst ensuring stability.
Myth 3: DevOps is only for continuous delivery.
DevOps doesn't (necessarily) imply continuous
delivery. The aim of a DevOps culture is to increase
the delivery frequency of an organisation, often from
quarterly/monthly to daily releases or more, and
improve their ability to respond to changes in the
market.
While continuous delivery relies heavily on
automation and is aimed at
agile and lean thinking
organisations, unlike
DevOps it is not reliant on a shared culture which
enhances collaboration. Gartner summed up the
distinction with a report that stated that: "DevOps is
not a market, but a tool-centric philosophy that
supports a continuous delivery value chain."
Myth 4: DevOps requires new tools.
As with the implementation of any new concept or
idea, a common misconception about
DevOps
adoption
is that new toolsets, and skills are
required. Though the provision of appropriate and
relevant tools can aid adoption, organisations are by
no means required to replace tools and processes they
use to produce software.
DevOps enables organisations to deliver new
capabilities more easily, and bring new software into
production more rapidly in order to respond to market
changes. It is not strictly reliant on new tools to get
this job done.
Myth 5: DevOps is a skill.
The rapid growth of the DevOps movement has resulted
in huge demand for professionals who are skilled within
the methodology. However, this fact is often
misconstrued to suggest that DevOps is itself a skill –
this is not the case.
DevOps is a culture – one that needs to be fully
adopted throughout an entire organisation for optimum
results, and one that is best supported with
appropriate and relevant tools.
Myth 6: DevOps is software.
Understanding that DevOps adoption can be better
facilitated with software is important, however, maybe
more so is understanding that they are not one and the
same. Although it is true that there is a significant
amount of DevOps software available on the market
today, purchasing a specific ad-hoc DevOps product, or
even suite of products, will not make your business
'DevOps'.
The DevOps methodology is the communication,
collaboration and automation of your development and
operations functions, and as described above, is
required to be adopted by an entire organisation to
achieve optimum results. The software and tools
available will undoubtedly reduce the strain of
adoption on your business but conscious adoption is
required for your business to fully reach the potential
that DevOps offers.
Conclusion
Like any new and popular term, people have somewhat
confused and sometimes contradictory or partial
impressions of what DevOps is and how it works.
DevOps is a philosophy which enables businesses to
automate their processes and work more collaboratively
to achieve a common goal and deliver software more
rapidly.
At
VASSIT
we
help organisations to successfully implement DevOps,
click here
to learn how we made DevOps a reality at
TSB bank Sabadell
These are just a few things that jumped out at me (and annoyed me)
However, there are teams at Netflix that do traditional Operations, and teams that do DevOps as well.
Ops is ops is ops. No matter what you call it, Operations is operations.
Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime
There's no such thing as a "DevOps tool". People were using CFengine, Puppet and Chef long before DevOps was even a term. These
are configuration management tools. In fact Adrian has even said they use Puppet in their legacy datacenter:
yet he seems to make the distinction between the ops guys there and the "devops" guys (whatever those are).
There is no ops organization involved in running our cloud…
Just because you outsourced it, doesn't mean it doesn't exist. Oh and it's not your cloud. It's Amazon's.
Reading between the lines
Actually this doesn't take much reading between the lines. It's out there in plain sight:
In reality we had the usual complaints about how long it took to get new capacity, the lack of consistency across supposedly
identical systems, and failures in Oracle, in the SAN and the networks, that took the site down too often for too long.
We tried bringing in new ops managers, and new engineers, but they were always overwhelmed by the fire fighting needed to keep
the current systems running.
This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration
bugs in systems that were supposed to be identical.
The developers used to spend hours a week in meetings with Ops discussing what they needed, figuring out capacity forecasts
and writing tickets to request changes for the datacenter.
There is no ops organization involved in running our cloud, no need for the developers to interact with ops people to get things
done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done to someone else.
I'm glad to see this spelled out in such detail. This is what I've been telling people semi-privately for a while now. Because
Netflix had such a terrible experience with its operations team, they went to the opposite extreme and disintermediated them.
Imagine you were scared as a kid by a clown. Now imagine you have kids of your own. You hate clowns. You had a bad experience
with clowns. But it's your kid's birthday party so here you are making baloon animals, telling jokes and doing silly things to entertain
the kids.Just because you aren't wearing makeup doesn't make you any less of a clown. You're doing clown shit. Through the eyes of
the kids, you're a clown. Deal with it. Netflix is still doing operations. What should be telling and frightening to operations teams
everywhere is this:
The Netflix response to poorly run operations that can't service the business is going to become the norm and not the exception.
Evolve or die.
Please note that I don't lay all the blame on the Netflix operations team. I would love to hear the flipside of this story from
someone who was there originally when the streaming initiative started. It would probably be full of stories we've heard before -
no resources, misalignment of incentives and a whole host of others.
Adrian, thank you for writing the blog post. I hope it serves as a warning to those who come. Hopefully someday you'll be able
to see a clown again and not get scared ;)
Adrian Cockcroft's article about
NoOps at Netflix ignited a
controversy that has been smouldering for some months. John Allspaw's detailed
response to Adrian's article makes a key point: What Adrian described as "NoOps" isn't really. Operations doesn't go away. Responsibilities
can, and do, shift over time, and as they shift, so do job descriptions. But no matter how you slice it, the same jobs need to be
done, and one of those jobs is operations. What Adrian is calling NoOps at Netflix isn't all that different from Operations at Etsy.
But that just begs the question: What do we mean by "operations" in the 21st century? If NoOps is a movement for replacing operations
with something that looks suspiciously like operations, there's clearly confusion. Now that some of the passion has died down, it's
time to get to a better understanding of what we mean by operations and how it's changed over the years.
At a recent lunch, John noted that back in the dawn of the computer age, there was no distinction between dev and ops. If you
developed, you operated. You mounted the tapes, you flipped the switches on the front panel, you rebooted when things crashed, and
possibly even replaced the burned out vacuum tubes. And you got to wear a geeky white lab coat. Dev and ops started to separate in
the '60s, when programmer/analysts dumped boxes of punch cards into readers, and "computer operators" behind a glass wall scurried
around mounting tapes in response to IBM JCL. The operators also pulled printouts from line printers and shoved them in labeled cubbyholes,
where you got your output filed under your last name.
The arrival of minicomputers in the 1970s and PCs in the '80s broke down the wall between mainframe operators and users, leading
to the system and network administrators of the 1980s and '90s. That was the birth of modern "IT operations" culture. Minicomputer
users tended to be computing professionals with just enough knowledge to be dangerous. (I remember when a new director was given
the root password and told to "create an account for yourself" … and promptly crashed the VAX, which was shared by about 30 users).
PC users required networks; they required support; they required shared resources, such as file servers and mail servers. And yes,
BOFH ("Bastard Operator from Hell") serves as a reminder of those days. I
remember being told that "no one" else is having the problem you're having - and not getting beyond it until at a company meeting
we found that everyone was having the exact same problem, in slightly different ways. No wonder we want ops to disappear. No wonder
we wanted a wall between the developers and the sysadmins, particularly since, in theory, the advent of the personal computer and
desktop workstation meant that we could all be responsible for our own machines.
But somebody has to keep the infrastructure running, including the increasingly important websites. As companies and computing
facilities grew larger, the fire-fighting mentality of many system administrators didn't scale. When the whole company runs on one
386 box (like O'Reilly in 1990), mumbling obscure command-line incantations is an appropriate way to fix problems. But that doesn't
work when you're talking hundreds or thousands of nodes at Rackspace or Amazon. From an operations standpoint, the big story of the
web isn't the evolution toward full-fledged applications that run in the browser; it's the growth from single servers to tens of
servers to hundreds, to thousands, to (in the case of Google or Facebook) millions. When you're running at that scale, fixing problems
on the command line just isn't an option. You can't afford letting machines get out of sync through ad-hoc fixes and patches. Being
told "We need 125 servers online ASAP, and there's no time to automate it" (as
Sascha Bates encountered) is a recipe for disaster.
The response of the operations community to the problem of scale isn't surprising. One of the themes of O'Reilly's
Velocity Conference is "Infrastructure as Code." If you're going to do operations
reliably, you need to make it reproducible and programmatic. Hence virtual machines to shield software from configuration issues.
Hence Puppet and Chef
to automate configuration, so you know every machine has an identical software configuration and is running the right services. Hence
Vagrant to ensure that all your virtual machines are constructed identically
from the start. Hence automated monitoring tools to ensure that your clusters are running properly. It doesn't matter whether the
nodes are in your own data center, in a hosting facility, or in a public cloud. If you're not writing software to manage them, you're
not surviving.
Furthermore, as we move further and further away from traditional hardware servers and networks, and into a world that's virtualized
on every level, old-style system administration ceases to work. Physical machines in a physical machine room won't disappear, but
they're no longer the only thing a system administrator has to worry about. Where's the root disk drive on a virtual instance running
at some colocation facility? Where's a network port on a virtual switch? Sure, system administrators of the '90s managed these resources
with software; no sysadmin worth his salt came without a portfolio of Perl scripts. The difference is that now the resources themselves
may be physical, or they may just be software; a network port, a disk drive, or a CPU has nothing to do with a physical entity you
can point at or unplug. The only effective way to manage this layered reality is through software.
So infrastructure had to become code. All those Perl scripts show that it was already becoming code as early as the late '80s;
indeed, Perl was designed as a programming language for automating system administration. It didn't take long for leading-edge sysadmins
to realize that handcrafted configurations and non-reproducible incantations were a bad way to run their shops. It's possible that
this trend means the end of traditional system administrators, whose jobs are reduced to racking up systems for Amazon or Rackspace.
But that's only likely to be the fate of those sysadmins who refuse to grow and adapt as the computing industry evolves. (And I suspect
that sysadmins who refuse to adapt swell the ranks of the BOFH fraternity, and most of us would be happy to see them leave.) Good
sysadmins have always realized that automation was a significant component of their job and will adapt as automation becomes even
more important. The new sysadmin won't power down a machine, replace a failing disk drive, reboot, and restore from backup; he'll
write software to detect a misbehaving EC2 instance automatically, destroy the bad instance, spin up a new one, and configure it,
all without interrupting service. With automation at this level, the new "ops guy" won't care if he's responsible for a dozen systems
or 10,000. And the modern BOFH is, more often than not, an old-school sysadmin who has chosen not to adapt.
James Urquhart nails it when he describes how
modern applications, running in the cloud, still need to be resilient and fault tolerant, still need monitoring, still need to adapt
to huge swings in load, etc. But he notes that those features, formerly provided by the IT/operations infrastructures, now need to
be part of the application, particularly in "platform as a service" environments. Operations doesn't go away, it becomes part of
the development. And rather than envision some sort of uber developer, who understands big data, web performance optimization, application
middleware, and fault tolerance in a massively distributed environment, we need operations specialists on the development teams.
The infrastructure doesn't go away - it moves into the code; and the people responsible for the infrastructure, the system administrators
and corporate IT groups, evolve so that they can write the code that maintains the infrastructure. Rather than being isolated, they
need to cooperate and collaborate with the developers who create the applications. This is the movement informally known as "DevOps."
Amazon's EBS outage last year demonstrates how the nature of
"operations" has changed. There was a marked distinction between companies that suffered and lost money, and companies that rode
through the outage just fine. What was the difference? The companies that didn't suffer, including Netflix, knew how to design for
reliability; they understood resilience, spreading data across zones, and a whole lot of reliability engineering. Furthermore, they
understood that resilience was a property of the application, and they worked with the development teams to ensure that the applications
could survive when parts of the network went down. More important than the flames about Amazon's services are the testimonials of
how intelligent and careful design kept applications running while EBS was down.
Netflix's ChaosMonkey is
an excellent, if extreme, example of a tool to ensure that a complex distributed application can survive outages; ChaosMonkey randomly
kills instances and services within the application. The development and operations teams collaborate to ensure that the application
is sufficiently robust to withstand constant random (and self-inflicted!) outages without degrading.
On the other hand, during the EBS outage, nobody who wasn't an Amazon employee touched a single piece of hardware. At the time,
JD Long tweeted that the best thing about the EBS outage was that his
guys weren't running around like crazy trying to fix things. That's how it should be. It's important, though, to notice how this
differs from operations practices 20, even 10 years ago. It was all over before the outage even occurred: The sites that dealt with
it successfully had written software that was robust, and carefully managed their data so that it wasn't reliant on a single zone.
And similarly, the sites that scrambled to recover from the outage were those that hadn't built resilience into their applications
and hadn't replicated their data across different zones.
In addition to this redistribution of responsibility, from the lower layers of the stack to the application itself, we're also
seeing a redistribution of costs. It's a mistake to think that the cost of operations goes away. Capital expense for new servers
may be replaced by monthly bills from Amazon, but it's still cost. There may be fewer traditional IT staff, and there will certainly
be a higher ratio of servers to staff, but that's because some IT functions have disappeared into the development groups. The bonding
is fluid, but that's precisely the point. The task - providing a solid, stable application for customers - is the same. The locations
of the servers on which that application runs, and how they're managed, are all that changes.
One important task of operations is understanding the cost trade-offs between public clouds like Amazon's, private clouds, traditional
colocation, and building their own infrastructure. It's hard to beat Amazon if you're a startup trying to conserve cash and need
to allocate or deallocate hardware to respond to fluctuations in load. You don't want to own a huge cluster to handle your peak capacity
but leave it idle most of the time. But Amazon isn't inexpensive, and a larger company can probably get a better deal taking its
infrastructure to a colocation facility. A few of the largest companies will build their own datacenters. Cost versus flexibility
is an important trade-off; scaling is inherently slow when you own physical hardware, and when you build your data centers to handle
peak loads, your facility is underutilized most of the time. Smaller companies will develop hybrid strategies, with parts of the
infrastructure hosted on public clouds like AWS or Rackspace, part running on private hosting services, and part running in-house.
Optimizing how tasks are distributed between these facilities isn't simple; that is the province of operations groups. Developing
applications that can run effectively in a hybrid environment: that's the responsibility of developers, with healthy cooperation
with an operations team.
The use of metrics to monitor system performance is another respect in which system administration has evolved. In the early '80s
or early '90s, you knew when a machine crashed because you started getting phone calls. Early system monitoring tools like HP's OpenView
provided limited visibility into system and network behavior but didn't give much more information than simple heartbeats or reachability
tests. Modern tools like DTrace provide insight into almost every aspect of system behavior; one of the biggest challenges facing
modern operations groups is developing analytic tools and metrics that can take advantage of the data that's available to predict
problems before they become outages. We now have access to the data we need, we just don't know how to use it. And the more we rely
on distributed systems, the more important monitoring becomes. As with so much else, monitoring needs to become part of the application
itself. Operations is crucial to success, but operations can only succeed to the extent that it collaborates with developers and
participates in the development of applications that can monitor and heal themselves.
Success isn't based entirely on integrating operations into development. It's naive to think that even the best development groups,
aware of the challenges of high-performance, distributed applications, can write software that won't fail. On this two-way street,
do developers wear the beepers, or IT staff? As Allspaw points out, it's important not to divorce developers from the consequences
of their work since the fires are frequently set by their code. So, both developers and operations carry the beepers. Sharing responsibilities
has another benefit. Rather than finger-pointing post-mortems that try to figure out whether an outage was caused by bad code or
operational errors, when operations and development teams work together to solve outages,
a post-mortem can focus less on assigning
blame than on making systems more resilient in the future. Although we used to practice "root cause analysis" after failures, we're
recognizing that finding out the single cause is unhelpful. Almost every outage is the result of a "perfect storm" of normal, everyday
mishaps. Instead of figuring out what went wrong and building procedures to ensure that something bad can never happen again (a process
that almost always introduces inefficiencies and unanticipated vulnerabilities), modern operations designs systems that are resilient
in the face of everyday errors, even when they occur in unpredictable combinations.
In the past decade, we've seen major changes in software development practice. We've moved from various versions of the "waterfall"
method, with interminable up-front planning, to "minimum viable product," continuous integration, and continuous deployment. It's
important to understand that the waterfall and methodology of the '80s aren't "bad ideas" or mistakes. They were perfectly adapted
to an age of shrink-wrapped software. When you produce a "gold disk" and manufacture thousands (or millions) of copies, the penalties
for getting something wrong are huge. If there's a bug, you can't fix it until the next release. In this environment, a software
release is a huge event. But in this age of web and mobile applications, deployment isn't such a big thing. We can release early,
and release often; we've moved from continuous integration to continuous deployment. We've developed techniques for quick resolution
in case a new release has serious problems; we've mastered A/B testing to test releases on a small subset of the user base.
All of these changes require cooperation and collaboration between developers and operations staff. Operations groups are adopting,
and in many cases, leading in the effort to implement these changes. They're the specialists in resilience, in monitoring, in deploying
changes and rolling them back. And the many attendees, hallway discussions, talks, and keynotes at O'Reilly's Velocity conference
show us that they are adapting. They're learning about adopting approaches to resilience that are completely new to software engineering;
they're learning about monitoring and diagnosing distributed systems, doing large-scale automation, and debugging under pressure.
At a recent meeting, Jesse Robbins described scheduling EMT training sessions for operations staff so that they understood how to
handle themselves and communicate with each other in an emergency. It's an interesting and provocative idea, and one of many things
that modern operations staff bring to the mix when they work with developers.
What does the future hold for operations? System and network monitoring used to be exotic and bleeding-edge; now, it's expected.
But we haven't taken it far enough. We're still learning how to monitor systems, how to analyze the data generated by modern monitoring
tools, and how to build dashboards that let us see and use the results effectively. I've joked about "using a Hadoop cluster to monitor
the Hadoop cluster," but that may not be far from reality. The amount of information we can capture is tremendous, and far beyond
what humans can analyze without techniques like machine learning.
Likewise, operations groups are playing a huge role in the deployment of new, more efficient protocols for the web, like
SPDY. Operations is involved, more than ever, in tuning the performance
of operating systems and servers (even ones that aren't under our physical control); a lot of our "best practices" for TCP tuning
were developed in the days of ISDN and 56 Kbps analog modems, and haven't been adapted to the reality of Gigabit Ethernet, OC48*
fiber, and their descendants. Operations groups are responsible for figuring out how to use these technologies (and their successors)
effectively. We're only beginning to digest IPv6 and the changes it implies for network infrastructure. And, while I've written a
lot about building resilience
into applications, so far we've only taken baby steps. There's a lot there that we still don't know. Operations groups have been
leaders in taking best practices from older disciplines (control systems theory, manufacturing, medicine) and integrating them into
software development.
And what about NoOps? Ultimately, it's a bad name, but the name doesn't really matter. A group practicing "NoOps" successfully
hasn't banished operations. It's just moved operations elsewhere and called it something else. Whether a poorly chosen name helps
or hinders progress remains to be seen, but operations won't go away; it will evolve to meet the challenges of delivering effective,
reliable software to customers. Old-style system administrators may indeed be disappearing. But if so, they are being replaced by
more sophisticated operations experts who work closely with development teams to get continuous deployment right; to build highly
distributed systems that are resilient; and yes, to answer the pagers in the middle of the night when EBS goes down. DevOps.
There has been a sometimes heated discussion on twitter about the term NoOps recently, and I've been quoted extensively as
saying that NoOps is the way developers work at Netflix. However, there are teams at Netflix that do traditional Operations, and
teams that do DevOps as well. To try and clarify things I need to explain the history and current practices at Netflix in chunks
of more than 140 characters at a time.
When I joined Netflix about five years ago, I managed a development team, building parts of the web site. We also had an operations
team who ran the systems in the single datacenter that we deployed our code to. The systems were high end IBM P-series virtualized
machines with storage on a virtualized Storage Area Network. The idea was that this was reliable hardware with great operational
flexibility so that developers could assume low failure rates and concentrate on building features. In reality we had the usual
complaints about how long it took to get new capacity, the lack of consistency across supposedly identical systems, and failures
in Oracle, in the SAN and the networks, that took the site down too often for too long.
At that time we had just launched the streaming service, and it was still an experiment, with little content and no TV device
support. As we grew streaming over the next few years, we saw that we needed higher availability and more capacity, so we added
a second datacenter. This project took far longer than initial estimates, and it was clear that deploying capacity at the scale
and rates we were going to need as streaming took off was a skill set that we didn't have in-house. We tried bringing in new ops
managers, and new engineers, but they were always overwhelmed by the fire fighting needed to keep the current systems running.
Netflix is a developer oriented culture, from the top down. I sometimes have to remind people that our CEO Reed Hastings was
the founder and initial developer of Purify, which anyone developing serious C++ code in the 1990's would have used to find memory
leaks and optimize their code. Pure Software merged with Atria and Rational before being swallowed up by IBM. Reed left IBM and
formed Netflix. Reed hired a team of very strong software engineers who are now the VPs who run developer engineering for our
products. When we were deciding what to do next Reed was directly involved in deciding that we should move to cloud, and even
pushing us to build an aggressively cloud optimized architecture based on NoSQL. Part of that decision was to outsource the problems
of running large scale infrastructure and building new datacenters to AWS. AWS has far more resources to commit to getting cloud
to work and scale, and to building huge datacenters. We could leverage this rather than try to duplicate it at a far smaller scale,
with greater certainty of success. So the budget and responsibility for managing AWS and figuring out cloud was given directly
to the developer organization, and the ITops organization was left to run its datacenters. In addition, the goal was to keep datacenter
capacity flat, while growing the business rapidly by leveraging additional capacity on AWS.
Over the next three years, most of the ITops staff have left and been replaced by a smaller team. Netflix has never had a CIO,
but we now have an excellent VP of ITops Mike Kail (@mdkail), who now runs the datacenters. These still support the DVD shipping
functions of Netflix USA, and he also runs corporate IT, which is increasingly moving to SaaS applications like Workday. Mike
runs a fairly conventional ops team and is usually hiring, so there are sysadmin, database,, storage and network admin positions.
The datacenter footprint hasn't increased since 2009, although there have been technology updates, and the over-all size is order-of-magnitude
a thousand systems.
As the developer organization started to figure out cloud technologies and build a platform to support running Netflix on AWS,
we transferred a few ITops staff into a developer team that formed the core of our DevOps function. They build the Linux based
base AMI (Amazon Machine Image) and after a long discussion we decided to leverage developer oriented tools such as Perforce for
version control, Ivy for dependencies, Jenkins to automate the build process, Artifactory as the binary repository and to construct
a "bakery" that produces complete AMIs that contain all the code for a service. Along with AWS Autoscale Groups this ensured that
every instance of a service would be totally identical. Notice that we didn't use the typical DevOps tools Puppet or Chef to create
builds at runtime. This is largely because the people making decisions are development managers, who have been burned repeatedly
by configuration bugs in systems that were supposed to be identical.
By 2012 the cloud capacity has grown to be order-of-magnitude 10,000 instances, ten times the capacity of the datacenter, running
in nine AWS Availability zones (effectively separate datacenters) on the US East and West coast, and in Europe. A handful of DevOps
engineers working for Carl Quinn (@cquinn - well known from the Java Posse podcast) are coding and running the build tools and
bakery, and updating the base AMI from time to time. Several hundred development engineers use these tools to build code, run
it in a test account in AWS, then deploy it to production themselves. They never have to have a meeting with ITops, or file a
ticket asking someone from ITops to make a change to a production system, or request extra capacity in advance. They use a web
based portal to deploy hundreds of new instances running their new code alongside the old code, put one "canary" instance into
traffic, if it looks good the developer flips all the traffic to the new code. If there are any problems they flip the traffic
back to the previous version (in seconds) and if it's all running fine, some time later the old instances are automatically removed.
This is part of what we call NoOps. The developers used to spend hours a week in meetings with Ops discussing what they needed,
figuring out capacity forecasts and writing tickets to request changes for the datacenter. Now they spend seconds doing it themselves
in the cloud. Code pushes to the datacenter are rigidly scheduled every two weeks, with emergency pushes in between to fix bugs.
Pushes to the cloud are as frequent as each team of developers needs them to be, incremental agile updates several times a week
is common, and some teams are working towards several updates a day. Other teams and more mature services update every few weeks
or months. There is no central control, the teams are responsible for figuring out their own dependencies and managing AWS security
groups that restrict who can talk to who.
Automated deployment is part of the normal process of running in the cloud. The other big issue is what happens if something
breaks. Netflix ITops always ran a Network Operations Center (NOC) which was staffed 24x7 with system administrators. They were
familiar with the datacenter systems, but had no experience with cloud. If there was a problem, they would start and run a conference
call, and get the right people on the call to diagnose and fix the issue. As the Netflix web site and streaming functionality
moved to the cloud it became clear that we needed a cloud operations reliability engineering (CORE) team, and that it would be
part of the development organization. The CORE team was lucky enough to get Jeremy Edberg (@jedberg - well know from running Reddit)
as its initial lead engineer, and also picked up some of the 24x7 shift sysadmins from the original NOC. The CORE team is still
staffing up, looking for Site Reliability Engineer skill set, and is the second group of DevOps engineers within Netflix. There
is a strong emphasis on building tools too make as much of their processes go away as possible, for example they have no run-books,
they develop code instead,
To get themselves out of the loop, the CORE team has built an alert processing gateway. It collects alerts from several different
systems, does filtering, has quenching and routing controls (that developers can configure), and automatically routes alerts either
to the PagerDuty system (a SaaS application service that manages on call calendars, escalation and alert life cycles) or to a
developer team email address. Every developer is responsible for running what they wrote, and the team members take turns to be
on call in the PagerDuty rota. Some teams never seem to get calls, and others are more often on the critical path. During a major
production outage con call, the CORE team never make changes to production applications, they always call a developer to make
the change. The alerts mostly refer to business transaction flows (rather than typical operations oriented Linux level issues)
and contain deep links to dashboards and developer oriented Application Performance Management tools like AppDynamics which let
developers quickly see where the problem is at the Java method level and what to fix,
The transition from datacenter to cloud also invoked a transition from Oracle, initially to SimpleDB (which AWS runs) and now
to Apache Cassandra, which has its own dedicated team. We moved a few Oracle DBAs over from the ITops team and they have become
experts in helping developers figure out how to translate their previous experience in relational schemas into Cassandra key spaces
and column families. We have a few key development engineers who are working on the Cassandra code itself (an open source Java
distributed systems toolkit), adding features that we need, tuning performance and testing new versions. We have three key open
source projects from this team available on github.com/Netflix. Astyanax is a client library for Java applications to talk to
Cassandra, CassJmeter is a Jmeter plugin for automated benchmarking and regression testing of Cassandra, and Priam provides automated
operation of Cassandra including creating, growing and shrinking Cassandra clusters, and performing full and incremental backups
and restores. Priam is also written in Java. Finally we have three DevOps engineers maintaining about 55 Cassandra clusters (including
many that span the US and Europe), a total of 600 or so instances. They have developed automation for rolling upgrades to new
versions, and sequencing compaction and repair operations. We are still developing our Cassandra tools and skill sets, and are
looking for a manager to lead this critical technology, as well as additional engineers. Individual Cassandra clusters are automatically
created by Priam, and it's trivial for a developer to create their own cluster of any size without assistance (NoOps again). We
have found that the first attempts to produce schemas for Cassandra use cases tend to cause problems for engineers who are new
to the technology, but with some familiarity and assistance from the Cloud Database Engineering team, we are starting to develop
better common patterns to work to, and are extending the Astyanax client to avoid common problems.
In summary, Netflix stil does Ops to run its datacenter DVD business. we have a small number of DevOps engineers embedded in
the development organization who are building and extending automation for our PaaS, and we have hundreds of developers using
NoOps to get their code and datastores deployed in our PaaS and to get notified directly when something goes wrong. We have built
tooling that removes many of the operations tasks completely from the developer, and which makes the remaining tasks quick and
self service. There is no ops organization involved in running our cloud, no need for the developers to interact with ops people
to get things done, and less time spent actually doing ops tasks than developers would spend explaining what needed to be done
to someone else. I think that's different to the way most DevOps places run, but its similar to other PaaS enviroments, so it
needs it's own name, NoOps. [Update: the DevOps community argues that although it's different, it's really just a more advanced
end state for DevOps, so lets just call it PaaS for now, and work on a better definition of DevOps].
Thanks John. I agree with some of what you point out. Netflix in effect over-reacted to a dysfunctional ops organization. I think
there are several other organizations who would recognize our situation, and would also find a need to over-react to make the
solution stick.
Your definition of DevOps seems far broader than the descriptions and definitions I can find by googling or
looking on Wikipedia. I don't recognize what we do in those definitions - since they are so focused on the relationship between
a Dev org and an Ops org, so someone should post an updated definition to Wikipedia or devops.com.
Until then maybe I'll just call it NetflOps or botchagalops.
I have a loaded NoOps question for you :) I am very interesting in understanding how a decentralized environments he said-she
said issues get solved. For example, I know netflix uses horizontally scalable rest layers as integration points.
Suppose one team/application is having an intermittent problem/bug with another team/application. Team 1 opens a issue. Team
2 reads investigates closes the issue as not a problem. Team 1 double checks and reasserts the issue is Team 2.
In a decentralized environment how is this road block cleared?
As an ops person I spend a good deal of time chasing down problems very external to me. I accept this as an ops person. Since
developers are pressed into ops how much time will a developer spend on another teams reported problems. Will team 1 forgo there
own scrum deadlines this week because team 2 sucks up all their time reporting bogus problems?
We aren't decentralized. So in the scenario you mention everyone gets in a room and figures it out, or we just end up with a long
email thread if it's less serious. APM tools help pinpoint what is going on at the request level down to Java code. Once we have
root cause someone files a Jira to fix the issue. There is a manager rotation for centrally prioritizing and coordinating response
to major outages. (I'm on duty this week, it comes up every few months.)
We have a few people who have "mad wireshark skills" to debug network layer problems, but that's infrequent and I'm hopeful
that boundary.com will come up with better tools in this space.
We don't follow a rigid methodology or fixed release deadlines, we ship code frequently enough that delays aren't usually a
big issue, and we have a culture of responsible adults so we communicate and respect each others needs across teams. The infrequent
large coordinating events like a new country launch are dealt with by picking one manager to own the overall big picture and lead
the coordination.
Adrian - Great article! I'm always fascinated to read & hear how folks have solved problems that plague a lot of IT organizations.
I've got a question about something in your reply above: "we have a culture of responsible adults so we communicate and respect
each others needs across teams".
I've found that, time and time again, the most difficult thing about organizational change is the people. How does one go about
hiring "responsible adults"? I know that it might sound like a silly or a flippant question, but seriously -- I've lost count
of how many times grown up folks act like childish, selfish, spoiled brats.
My views on culture may not be much help - read http://perfcap.blogspot.com/2011/12/how-netflix-gets-out-of-way-of.html to see
my explanation of how Netflix does things differently.
Culture is very hard to create or modify but easy to destroy. This is
because everyone has to buy into it for it to be effective, and then every manager has to hire only people who are compatible
with the culture, and also get rid of people who turn out not to fit in, even if they are doing good work.
So the short answer is start a new company from scratch with the culture you want, and pay a lot of attention to who you hire.
I don't think it is possible to do a culture shift if there are more than a roomful of people involved.
Part of getting a "culture of responsible adults" together is partly down to "culture" - although it helps to have mature sensible
individuals, fostering that also means avoiding finger-pointing and blame.
The more defensive people are made to feel, the more likely they are to start throwing tantrums when under pressure. A culture
where you can put your hand up and say: "I got that wrong, how can we put it right?" gets better results in the long term than
one where you might be fired or disciplined for a genuine mistake.
I always wondered how evil geniuses like Ernst Blofeld recruit when getting it wrong means you might end up in the shark tank...
Yes, incident reviews that don't involve blame and finger-pointing are also key. Making the same mistake several times, trying
to hide your mistakes, or clear lapses of judgement can't be tolerated though.
Great article Adrian. I have a question. Is a consecutive IP space important? Since AWS EIP doesn't guarantee consecutive addresses,
I've wondered if this mattered to app developers.
Anything that could have been done by subnet is out the window. For example
if you wanted to do port sweeps of your network blocks for an audit, perform penetration testing, or parsing logs by IPs. I suppose
this could be done pragmatically but was curious about your experiences. Does it matter?
If the network topology matters you can use VPC to manage it. Also if you are a big enough customer to have an enterprise level
support contract with AWS and use a lot of EIPs it is possible to get them allocated in contiguous blocks.
I'm curious about your statement "Notice that we didn't use the typical DevOps tools Puppet or Chef to create builds at runtime.
This is largely because the people making decisions are development managers, who have been burned repeatedly by configuration
bugs in systems that were supposed to be identical."
If systems are built from the same puppet manifests, what kind of configuration
bugs can occur? Also, how is the alternative method you choose any less likely to cause the same problems?
Puppet is overkill for what we end up doing. We have a base AMI that is always identical, we install an rpm on it or untar a file
once, rebake the AMI and we are done. That AMI is then turned into running systems using the AWS autoscaler. It's more reliable
because there are fewer things to go wrong at boot time, no dependencies on repo's or orchestration tools, we know that the install
had completely succeeded the same way on every instance before the instance booted.
Adrian, Last time we talked you mentioned that you were not 100% transitioned to AWS, but were - looks like this was achieved,
with the exception for the core DVD Business.
As many companies are talking about Hyrbrid cloud as a target state, and Netflix went through the transition from Pvt/Managed
Ops to Public-AWS/NoOps, can you talk about the interim state you were in and what development and operations use cases you optimized
around during transition – ala, what were the Hybrid models and principles you all followed – to move from Ops to NoOps. Did you
attempt to keep these teams and tools separate or did you all try to create a transition strategy that allowed you to hedge the
bets with AWS and the 'all-in strategy" to possibly come back if needed etc..
What Ops Governance and Dev Tooling approach did you take in the state? Specifically around cloud abstraction layers to ease
the Access management, support, tooling, elasticity needs. Can you shed some light on the thinking and approach you took while
you were in mid-state of transition?
Also can you comment on the how much do you govern and drive the development and deployment approach so you can unify the continuous
Integration and Continuous deployment tools so that you can reduce the chaos in this space?
Tim Fraser
If you look at the presentations on slideshare.net/adrianco I have discussed in some detail what the transition strategy and
tools looked like. We continued to run the old code in the datacenter as we gradually moved functionality away from it. The old
DC practices were left intact as we moved developers to the cloud one group at a time.
It seems to me that rather than NoOps, this is an outsourcing of ops. I assume that the issues you had with Ops came from
the need to control cost, scale etc? If not, please would you clarify?
If it was, how has moving to AWS solved your problems? Is it that AWS can scale more readily based on experience to date? Was
cost control an issue before, and is it now, in terms of Ops
costs?
Given that NoOps = outsourcing, I can see that managing the relationship with the service provider becomes vital for you?
Dev and admin teams struggle these days with keeping up with agile development. DevOps helps by breaking down some of the walls.
But one of the biggest challenges is getting the entire development team involved and not just 1 or 2 people who help do deployments.
The entire team needs visibility to the production server environments to help support and troubleshoot applications.
The Last but not LeastTechnology is dominated by
two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt.
Ph.D
FAIR USE NOTICEThis site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
You can use PayPal to to buy a cup of coffee for authors
of this site
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society.We do not warrant the correctness
of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.
