Classification of System Administrators

Unix system administrators can administer systems in three different environments

• Small sites have, say, less then 20 machines, all running the same flavor of Unix OS (for example Suse 10 or CentOS). Usually the administrator of a small site has number of users only slightly bigger then the number of the servers. Also typically he/she is a single administrator for all the servers or have just one colleague administrator who often works different shift. The server maintained are usually small servers often from a low cost provider (Dell). This is a typical environment for startups.
• Medium sites have less then a hundred servers, and may be running two-three different flavors of Unix. Typically they are installed in a single location. Number of users can exceed the number of servers by factor 2 or 3. There are multiple administrators and there is some level specialization (for example, backup vs. monitoring). Most servers are small or medium but there can be large servers in the environment as well. The latter requires more specialized hardware knowledge.
• Large sites often called enterprise sites typically have several hundreds servers and  several datacenters. Remote datacenters are common in this environment, so you might never see actual servers you administer. More often then not all major flavors of Unix are present (due to acquisitions), the situation which create serious additional difficulties for administrators. The team on administrators consists of at least half-dozen sysadmins with different administrators specializing in different flavors of Unix and different applications. At enterprise level there is also a formal hierarchy of administrators:
  • Entry level Unix admins. They are just one step above operators and know basic administration of Unix boxes and not much more. Paradoxically a lot of enterprise sysadmins remain at this level for years, especially if they need to administer both Windows and Unix servers.
    
  • Junior Unix admins. Typically have BS degree and usually do not have in-depth knowledge of scripting, although some knowledge of shell and the ability to write simple scripts is a must. A+ certification level of understanding of networking is a must. RHEL technician certification is the typical certification required. They need to know how to work with just three enterprise applications:
    • backup, for example Data Protector
    • monitoring, for example HP Operations Manager
    • scheduling (minimal)
      
  • Senior Unix admins. Typically have MS degree (or considerable work experience along with natural talent ) and/or a couple of Unix certifications (for example Solaris administration and networking and Red Hat engineer certification) as well as in-depth knowledge two or more scripting languages (shell and Perl, or shell and TCL) and one compiled language (C, C++) in the level enough to develop and port small utilities, recompile kernel, compile PHP, apache, bind and such. In addition they should have a solid understanding of TCP/IP networking including detailed knowledge of SMTP, DNS and HTTP protocols. In addition to applications listed in prev section knowledge of Web servers, and database is a must. They also should know substantial set of enterprise applications and be able to extend them, for example by writing monitoring probes for HP Operations Manager of Tivoli TEC.
    
  • Gugus. Those are people who in addition to deep technical knowledge have talent to see architectural issues behind technical issues along with in-depth (often) on the kernel level knowledge of Unix. Typically gurus have written some useful for administration application which is their "claim to fame". The key feature of a guru is the ability to work on several drastically different technical levels (say architectural and low level coding) equally well and extract tremendous synergy from the ability to switch from one level to another and see their interaction. This is a pretty rare quality and people with such ability are equally rare.

The level of specialization among administrators of the same hierarchical level in large organizations is also higher with typically specialized admin responsible for backup (say DataProtector-based) and monitoring (say Tivoli based). Often all flavors of Unix are present in the environment so there are categories of admin specializing in each flavor or a couple of flavors (it is difficult to become proficient in more then two). Say one administrator is the primary specialist in Solaris but also knows Linux well, the other is primary specialist in HP-UX and AIX so on.

Also at enterprise level there is considerably more politics and red tape involved in the sysadmin job in comparison with smaller sites.

Here are example of qualifications for a senior enterprise Unix administrator from a Monster job advertisement:

• MS degree in computer science or a related technical field.
• Solaris certification required; 10+ years of Solaris-10 (SPARC/X86) operating system administration
• Basic networking skills, solid knowledge of DNS (bind), Sendmail, firewalls. Solaris network certification preferred.
• Minimum 3 years experience with Sun Web Portal
• Experience with Veritas Volume Manager and Veritas Filesystem
• Proven understanding of EMC SAN technology
• Experience with Source Control, CVS or equivalent
• Experience with relational database and SQL.
• Knowledge of system monitoring (such as Tivoli ITM 6.1), availability monitoring and SLA reporting
• Experience overseeing maintenance of mid range Unix servers including but not limited to Linux, Solaris, and AIX
• Ability to coordinate special efforts such as emergency upgrades and *deep-dive" technical analysis
• Experience in administrating of mid-range servers from Sun, Dell and IBM.
• History of establishing and insuring 24x7 support services, operational standards, processes, and procedures
• Ability to be flexible with work schedule, including weekends and holidays
• Ability to prepare, present, and translate technical requirements into successful business cases
• Must be highly skilled and proficient in problem solving and able to follow a systematic troubleshooting approach
• Require minimum support or supervision and is able to solve most issues and/or challenges independently
• Leadership potential

Enterprise environment has many requirements that "admin-in-large" should met addition to usual "sysadmins-in-small" staff typical for small companies. Nearly half of U.S. IT jobs involve the maintenance of computers and/or software -- approximately 270,000 people.

NEWS CONTENTS

• 20110603 : What does a Unix System Administrator do ( What does a Unix System Administrator do, Jun 03, 2011 )
• 20110603 : Required skills: ( )
• 20110603 : Required background: ( )
• 20110603 : Appropriate responsibilities: ( )
• 20110603 : Required skills: ( )
• 20110603 : Required background: ( )
• 20110603 : Appropriate Responsibilities: ( )
• 20110603 : Required Skills ( )
• 20110603 : Required Background: ( )
• 20110603 : Appropriate Responsibilities: ( )
• 20110603 : Required Skills ( )
• 20110603 : Required Background: ( )
• 20110603 : Appropriate Responsibilities: ( )
• 20100409 : Business Has Killed IT With Overspecialization - by Charlie Schluting ( April 7, 2010 , Enterprise Networking Planet )

Old News ;-)

[Jun 03, 2011] What does a Unix System Administrator do

System administration skills can be classified in four general levels. The links below discuss the required skills, desired skills, and responsibilities of each of those levels. Following the levels are some general thoughts on system administration in general.

Novice Junior Intermediate/advanced Senior Some thoughts on System Administration.

Novice System Administrator:

• Required skills:

  • Has strong inter-personal and communication skills: is capable of explaining simple procedures in writing or verbally; has good phone skills.
  • Is familiar with Unix and its commands/utilities at the user level. Can edit files using more than one editor. Uses at least two shells one of them being the Bourne shell.
  • Can perform standard file processing tasks; find, move, remove, redirection.
  
  

• Required background:

  • Two years of college or equivalent post-high school education or experience.
  
  

• Desirable:

  • A degree or certificat in computer science or related field.
  • Previous experience in customer support, computer operations, system administration, or another related area.
  • Motivated to advance in the profession.
  
  

• Appropriate responsibilities:

  • Perform routine tasks under the direct supervision of a more experienced administrator.
  • Be the front-line interface for users; accepting problem reports and passing them to the appropriate system administrators.
  • Performs some security functions, especially monitoring the system

Junior System Administrator:

• Required skills:

  • Has strong inter-personal and communication skills: capable of training users in applications and Unix fundamentals. Able to write basic system and user documentation.
  • High skill level with most Unix commands and utilities.
  • Familiar with most basic system administration tools and tasks. For example, can cleanly boot and shutdown the system, add and remove user accounts, use backup programs, perform fsck and maintain system database files (groups, hosts, aliases, etc.)
  • Fundamental understanding of the functioning of the Unix operating system: for example understands job control, hard and soft linking, the difference between shell programs and kernel programs.
  • Basic understanding of Unix security procedures
  
  

• Required background:

  • One to three years of system administration experience.
  
  

• Desirable:

  • Degree in CS or a related field.
  • Familiarity with networked/ distributed computing environments. For example: can use the route command, add a workstation to a network, or mount a remote filesystem.
  • Ability to write functional scripts in an administrative language (shell, Perl, Tk).
  • Some programming experience in an applicable language like C.
  
  

• Appropriate Responsibilities:

  • Administer a small site alone, or assist in the administration of a larger site.
  • Work under the general supervision of a more senior system administrator or computer systems manager.
  • Perform normal security procedures, able to advise users on standard security protocol.

Intermediate/Advanced System Administrator

• Required Skills

  • Has strong inter-personal and communication skills: capable of training users in complex topics, making presentations to internal groups. Able to write intricate system and user documentation. Capable of writing and explaining purchase justifications.
  • Independent problem solving; self-directed, self-starting.
  • Very comfortable with most aspects of the Unix operating system: paging/swapping, inter-process communication, devices and device driver fundamentals, file system concepts like inode and superblock.
  • Familiar with fundamental networking/distributed computing environments and concepts. Can configure NFS and NIS, use nslookup or research to check information in the DNS.
  • Ability to write detailed scripts in at least one, preferably two administrative lnaguages, (shell scripts, Perl, Tk).
  • Ability to perform at least minimal debugging and modification of C programs.
  • Ability to perform most security audits, and protect the system against intrusion.
  
  

• Required Background:

  • Three to five years of system administration experience.
  
  

• Desirable:

  • At least a BS in Computer Science or a related field.
  • Significant programming background in any applicable language.
  
  

• Appropriate Responsibilities:

  • Receive general instructions for new duties from supervisor.
  • Administers a mid-size site alone, or assists in administration of a larger site.
  • Initiates some new responsibilities and helps plan for the future of the site and network.
  • Manages novice system administrators or operators.
  • Evaluates and/or recommends purchases; has strong influence on the purchasing process.
  • Serves as the first line of defense against intrusion and inadvertent system damage.

Senior System Administrator:

• Required Skills

  • Strong inter-personal and communication skills; capable of writing proposals and papers, acting as a vendor liaison, making presentations to customer/client audiences or making professional presentations, work closely with upper management.
  • Ability to solve problems quickly and completely.
  • Ability to identify tasks which should be automated and then write tools to automate them.
  • Solid understanding of the Unix based operations system: understands paging and swapping, interprocess communication, devices and device drivers, can perform system analysis and tuning.
  • Ability to program in at least one, preferably two administrative languages, (shell, Perl, Tk) and port C programs from one platform to another, write small C programs.
  • Solid understanding of networking/distributed computing environments, understanding the principals of routing, client/server programming, and the design of consistent network-wide filesystems.

• Required Background:

  • More than 5 years of previous system administration experience.
  
  

• Desirable:

  • A degree in CS or a related field. Advanced degree preferred.
  • Extensive programming experience in an applicable language.
  • Publications within the field of system administration.
  
  

• Appropriate Responsibilities:

  • Design/implement complex local and wide-area networks of machines.
  • Manages a large site or network.
  • Works under general direction of senior management.
  • Establishes/recommends policies and procedures for system use and services.
  • Provides the technical lead and/or supervision for system administrators, system programmers, or others.
  • Has purchasing authority and responsibility for purchase justification.

Finally, some important thoughts for system Administrators:

1. Never do something you can't undo.
2. Always check the backups, never assume they are working. Make sure you can restore from them, too.
3. Write down what you did, even if you know you will never forget it, you will.
4. If you do it more than once, write a script.
5. Get to know your users before there is a problem, then when there is, they will know who you are and maybe have a little understanding.
6. Remember you are performing a service for your users, you don't own the system, you just get to play with it.
7. Check your backups.
8. Never stop learning, there is always something you should know to make your job easier and your system more stable and secure.
9. Check your backups, again.

[Apr 09, 2010] Business Has Killed IT With Overspecialization - By Charlie Schluting

April 7, 2010 | Enterprise Networking Planet

What happened to the old "sysadmin" of just a few years ago? We've split what used to be the sysadmin into application teams, server teams, storage teams, and network teams. There were often at least a few people, the holders of knowledge, who knew how everything worked, and I mean everything. Every application, every piece of network gear, and how every server was configured -- these people could save a business in times of disaster.

Now look at what we've done. Knowledge is so decentralized we must invent new roles to act as liaisons between all the IT groups. Architects now hold much of the high-level "how it works" knowledge, but without knowing how any one piece actually does work. In organizations with more than a few hundred IT staff and developers, it becomes nearly impossible for one person to do and know everything. This movement toward specializing in individual areas seems almost natural. That, however, does not provide a free ticket for people to turn a blind eye.

Specialization

You know the story: Company installs new application, nobody understands it yet, so an expert is hired. Often, the person with a certification in using the new application only really knows how to run that application. Perhaps they aren't interested in learning anything else, because their skill is in high demand right now. And besides, everything else in the infrastructure is run by people who specialize in those elements. Everything is taken care of.

Except, how do these teams communicate when changes need to take place? Are the storage administrators teaching the Windows administrators about storage multi-pathing; or worse logging in and setting it up because it's faster for the storage gurus to do it themselves? A fundamental level of knowledge is often lacking, which makes it very difficult for teams to brainstorm about new ways evolve IT services. The business environment has made it OK for IT staffers to specialize and only learn one thing.

If you hire someone certified in the application, operating system, or network vendor you use, that is precisely what you get. Certifications may be a nice filter to quickly identify who has direct knowledge in the area you're hiring for, but often they indicate specialization or compensation for lack of experience.

Resource Competition

Does your IT department function as a unit? Even 20-person IT shops have turf wars, so the answer is very likely, "no." As teams are split into more and more distinct operating units, grouping occurs. One IT budget gets split between all these groups. Often each group will have a manager who pitches his needs to upper management in hopes they will realize how important the team is.

The "us vs. them" mentality manifests itself at all levels, and it's reinforced by management having to define each team's worth in the form of a budget. One strategy is to illustrate a doomsday scenario. If you paint a bleak enough picture, you may get more funding. Only if you are careful enough to illustrate the failings are due to lack of capital resources, not management or people. A manager of another group may explain that they are not receiving the correct level of service, so they need to duplicate the efforts of another group and just implement something themselves. On and on, the arguments continue.

Most often, I've seen competition between server groups result in horribly inefficient uses of hardware. For example, what happens in your organization when one team needs more server hardware? Assume that another team has five unused servers sitting in a blade chassis. Does the answer change? No, it does not. Even in test environments, sharing doesn't often happen between IT groups.

With virtualization, some aspects of resource competition get better and some remain the same. When first implemented, most groups will be running their own type of virtualization for their platform. The next step, I've most often seen, is for test servers to get virtualized. If a new group is formed to manage the virtualization infrastructure, virtual machines can be allocated to various application and server teams from a central pool and everyone is now sharing. Or, they begin sharing and then demand their own physical hardware to be isolated from others' resource hungry utilization. This is nonetheless a step in the right direction. Auto migration and guaranteed resource policies can go a long way toward making shared infrastructure, even between competing groups, a viable option.

Blamestorming

The most damaging side effect of splitting into too many distinct IT groups is the reinforcement of an "us versus them" mentality. Aside from the notion that specialization creates a lack of knowledge, blamestorming is what this article is really about. When a project is delayed, it is all too easy to blame another group. The SAN people didn't allocate storage on time, so another team was delayed. That is the timeline of the project, so all work halted until that hiccup was restored. Having someone else to blame when things get delayed makes it all too easy to simply stop working for a while.

More related to the initial points at the beginning of this article, perhaps, is the blamestorm that happens after a system outage.

Say an ERP system becomes unresponsive a few times throughout the day. The application team says it's just slowing down, and they don't know why. The network team says everything is fine. The server team says the application is "blocking on IO," which means it's a SAN issue. The SAN team say there is nothing wrong, and other applications on the same devices are fine. You've ran through nearly every team, but without an answer still. The SAN people don't have access to the application servers to help diagnose the problem. The server team doesn't even know how the application runs.

See the problem? Specialized teams are distinct and by nature adversarial. Specialized staffers often relegate themselves into a niche knowing that as long as they continue working at large enough companies, "someone else" will take care of all the other pieces.

I unfortunately don't have an answer to this problem. Maybe rotating employees between departments will help. They gain knowledge and also get to know other people, which should lessen the propensity to view them as outsiders

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019