Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells

Slightly Skeptical View on Enterprise Unix Administration

News Webliography of problems with "pure" cloud environment Recommended Books Recommended Links Recommended Tools to Enhance Command Line Usage in Windows Programmable Keyboards Microsoft IntelliType Macros
Open source politics: IBM acquires Red Hat Over 50 and unemployed Shadow IT Is DevOps a yet another "for profit" technocult? Dealing with multiple flavors of Unix Classification of System Administrators Red Hat Certification Program
Unix Configuration Management Tools Job schedulers Unix System Monitoring Red Hat Enterprise Linux Life Cycle Corporate bullshit as a communication method Diplomatic Communication Bosos or Empty Suits (Aggressive Incompetent Managers)
ILO command line interface Using HP ILO virtual CDROM iDRAC7 goes unresponsive - can't connect to iDRAC7 Resetting frozen iDRAC without unplugging the server Troubleshooting HPOM agents Webliography of problems with "pure" cloud environment The tar pit of Red Hat overcomplexity
Bare metal recovery of Linux systems Number of Servers per Sysadmin Is DevOps a yet another "for profit" technocult Carpal tunnel syndrome Sysadmin Horror Stories Humor Etc


The KISS rule can be expanded as: Keep It Simple, Sysadmin ;-)

This page is written as a pretest against overcomplexity and bizarre data center atmosphere dominant in "semi-outsourced" or fully outsourced datacenters ;-). Unix/Linux sysadmins are being killed with overcomplexity of the environment. As Charlie Schluting noted in 2010: (Enterprise Networking Plane, April 7, 2010)

What happened to the old "sysadmin" of just a few years ago? We've split what used to be the sysadmin into application teams, server teams, storage teams, and network teams. There were often at least a few people, the holders of knowledge, who knew how everything worked, and I mean everything. Every application, every piece of network gear, and how every server was configured -- these people could save a business in times of disaster.

Now look at what we've done. Knowledge is so decentralized we must invent new roles to act as liaisons between all the IT groups.

Architects now hold much of the high-level "how it works" knowledge, but without knowing how any one piece actually does work.

In organizations with more than a few hundred IT staff and developers, it becomes nearly impossible for one person to do and know everything. This movement toward specializing in individual areas seems almost natural. That, however, does not provide a free ticket for people to turn a blind eye.

Specialization

You know the story: Company installs new application, nobody understands it yet, so an expert is hired. Often, the person with a certification in using the new application only really knows how to run that application. Perhaps they aren't interested in learning anything else, because their skill is in high demand right now. And besides, everything else in the infrastructure is run by people who specialize in those elements. Everything is taken care of.

Except, how do these teams communicate when changes need to take place? Are the storage administrators teaching the Windows administrators about storage multipathing; or worse logging in and setting it up because it's faster for the storage gurus to do it themselves? A fundamental level of knowledge is often lacking, which makes it very difficult for teams to brainstorm about new ways evolve IT services. The business environment has made it OK for IT staffers to specialize and only learn one thing.

If you hire someone certified in the application, operating system, or network vendor you use, that is precisely what you get. Certifications may be a nice filter to quickly identify who has direct knowledge in the area you're hiring for, but often they indicate specialization or compensation for lack of experience.

Resource Competition

Does your IT department function as a unit? Even 20-person IT shops have turf wars, so the answer is very likely, "no." As teams are split into more and more distinct operating units, grouping occurs. One IT budget gets split between all these groups. Often each group will have a manager who pitches his needs to upper management in hopes they will realize how important the team is.

The "us vs. them" mentality manifests itself at all levels, and it's reinforced by management having to define each team's worth in the form of a budget. One strategy is to illustrate a doomsday scenario. If you paint a bleak enough picture, you may get more funding. Only if you are careful enough to illustrate the failings are due to lack of capital resources, not management or people. A manager of another group may explain that they are not receiving the correct level of service, so they need to duplicate the efforts of another group and just implement something themselves. On and on, the arguments continue.

Most often, I've seen competition between server groups result in horribly inefficient uses of hardware. For example, what happens in your organization when one team needs more server hardware? Assume that another team has five unused servers sitting in a blade chassis. Does the answer change? No, it does not. Even in test environments, sharing doesn't often happen between IT groups.

With virtualization, some aspects of resource competition get better and some remain the same. When first implemented, most groups will be running their own type of virtualization for their platform. The next step, I've most often seen, is for test servers to get virtualized. If a new group is formed to manage the virtualization infrastructure, virtual machines can be allocated to various application and server teams from a central pool and everyone is now sharing. Or, they begin sharing and then demand their own physical hardware to be isolated from others' resource hungry utilization. This is nonetheless a step in the right direction. Auto migration and guaranteed resource policies can go a long way toward making shared infrastructure, even between competing groups, a viable option.

Blamestorming

The most damaging side effect of splitting into too many distinct IT groups is the reinforcement of an "us versus them" mentality. Aside from the notion that specialization creates a lack of knowledge, blamestorming is what this article is really about. When a project is delayed, it is all too easy to blame another group. The SAN people didn't allocate storage on time, so another team was delayed. That is the timeline of the project, so all work halted until that hiccup was restored. Having someone else to blame when things get delayed makes it all too easy to simply stop working for a while.

More related to the initial points at the beginning of this article, perhaps, is the blamestorm that happens after a system outage.

Say an ERP system becomes unresponsive a few times throughout the day. The application team says it's just slowing down, and they don't know why. The network team says everything is fine. The server team says the application is "blocking on IO," which means it's a SAN issue. The SAN team say there is nothing wrong, and other applications on the same devices are fine. You've ran through nearly every team, but without an answer still. The SAN people don't have access to the application servers to help diagnose the problem. The server team doesn't even know how the application runs.

See the problem? Specialized teams are distinct and by nature adversarial. Specialized staffers often relegate themselves into a niche knowing that as long as they continue working at large enough companies, "someone else" will take care of all the other pieces.

I unfortunately don't have an answer to this problem. Maybe rotating employees between departments will help. They gain knowledge and also get to know other people, which should lessen the propensity to view them as outsiders

Additional useful material on the topic can also be found in my older article Solaris vs Linux:

Abstract

Introduction

Nine factors framework for comparison of two flavors of Unix in a large enterprise environment

Four major areas of Linux and Solaris deployment

Comparison of internal architecture and key subsystems

Security

Hardware: SPARC vs. X86

Development environment

Solaris as a cultural phenomenon

Using Solaris-Linux enterprise mix as the least toxic Unix mix available

Conclusions

Acknowledgements

Webliography

Here are my notes/reflection of sysadmin problem in strange (and typically pretty toxic) IT departments of large corporations:


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

2018 2017 2016 2015 2014 2013 2012 2011 2010 2009
2008 2007 2006 2005 2004 2003 2002 2001 2000 1999

"I appreciate Woody Allen's humor because one of my safety valves is an appreciation for life's absurdities. His message is that life isn't a funeral march to the grave. It's a polka."

-- Dennis Kusinich

[Nov 13, 2018] GridFTP : User s Guide

Notable quotes:
"... file:///path/to/my/file ..."
"... gsiftp://hostname/path/to/remote/file ..."
"... third party transfer ..."
toolkit.globus.org

Table of Contents

1. Introduction
2. Usage scenarios
2.1. Basic procedure for using GridFTP (globus-url-copy)
2.2. Accessing data in...
3. Command line tools
4. Graphical user interfaces
4.1. Globus GridFTP GUI
4.2. UberFTP
5. Security Considerations
5.1. Two ways to configure your server
5.2. New authentication options
5.3. Firewall requirements
6. Troubleshooting
6.1. Establish control channel connection
6.2. Try running globus-url-copy
6.3. If your server starts...
7. Usage statistics collection by the Globus Alliance
1. Introduction The GridFTP User's Guide provides general end user-oriented information. 2. Usage scenarios 2.1. Basic procedure for using GridFTP (globus-url-copy) If you just want the "rules of thumb" on getting started (without all the details), the following options using globus-url-copy will normally give acceptable performance:
globus-url-copy -vb -tcp-bs 2097152 -p 4 source_url destination_url
The source/destination URLs will normally be one of the following: 2.1.1. Putting files One of the most basic tasks in GridFTP is to "put" files, i.e., moving a file from your file system to the server. So for example, if you want to move the file /tmp/foo from a file system accessible to the host on which you are running your client to a file name /tmp/bar on a host named remote.machine.my.edu running a GridFTP server, you would use this command:
globus-url-copy -vb -tcp-bs 2097152 -p 4 file:///tmp/foo gsiftp://remote.machine.my.edu/tmp/bar
[Note] Note
In theory, remote.machine.my.edu could be the same host as the one on which you are running your client, but that is normally only done in testing situations.
2.1.2. Getting files A get, i.e, moving a file from a server to your file system, would just reverse the source and destination URLs:
[Tip] Tip
Remember file: always refers to your file system.
globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://remote.machine.my.edu/tmp/bar file:///tmp/foo
2.1.3. Third party transfers Finally, if you want to move a file between two GridFTP servers (a third party transfer ), both URLs would use gsiftp: as the protocol:
globus-url-copy -vb -tcp-bs 2097152 -p 4 gsiftp://other.machine.my.edu/tmp/foo gsiftp://remote.machine.my.edu/tmp/bar
2.1.4. For more information If you want more information and details on URLs and the command line options , the Key Concepts Guide gives basic definitions and an overview of the GridFTP protocol as well as our implementation of it. 2.2. Accessing data in... 2.2.1. Accessing data in a non-POSIX file data source that has a POSIX interface If you want to access data in a non-POSIX file data source that has a POSIX interface, the standard server will do just fine. Just make sure it is really POSIX-like (out of order writes, contiguous byte writes, etc). 2.2.2. Accessing data in HPSS The following information is helpful if you want to use GridFTP to access data in HPSS. Architecturally, the Globus GridFTP server can be divided into 3 modules: In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
[Note] Note
This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).
2.2.2.1. GridFTP Protocol Module
The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
2.2.2.2. Data Transform Functionality
The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
2.2.2.3. Data Storage Interface (DSI) / Data Transform module
The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.
2.2.2.4. HPSS info
Last Update: August 2005 Working with Los Alamos National Laboratory and the High Performance Storage System (HPSS) collaboration ( http://www.hpss-collaboration.org ), we have written a Data Storage Interface (DSI) for read/write access to HPSS. This DSI would allow an existing application that uses a GridFTP compliant client to utilize an HPSS data resources. This DSI is currently in testing. Due to changes in the HPSS security mechanisms, it requires HPSS 6.2 or later, which is due to be released in Q4 2005. Distribution for the DSI has not been worked out yet, but it will *probably* be available from both Globus and the HPSS collaboration. While this code will be open source, it requires underlying HPSS libraries which are NOT open source (proprietary).
[Note] Note
This is a purely server side change, the client does not know what DSI is running, so only a site that is already running HPSS and wants to allow GridFTP access needs to worry about access to these proprietary libraries.
2.2.3. Accessing data in SRB The following information is helpful if you want to use GridFTP to access data in SRB. Architecturally, the Globus GridFTP server can be divided into 3 modules: In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
[Note] Note
This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).
2.2.3.1. GridFTP Protocol Module
The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
2.2.3.2. Data Transform Functionality
The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
2.2.3.3. Data Storage Interface (DSI) / Data Transform module
The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN). The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc.. Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly.
2.2.3.4. SRB info
Last Update: August 2005 Working with the SRB team at the San Diego Supercomputing Center, we have written a Data Storage Interface (DSI) for read/write access to data in the Storage Resource Broker (SRB) (http://www.npaci.edu/DICE/SRB). This DSI will enable GridFTP compliant clients to read and write data to an SRB server, similar in functionality to the sput/sget commands. This DSI is currently in testing and is not yet publicly available, but will be available from both the SRB web site (here) and the Globus web site (here). It will also be included in the next stable release of the toolkit. We are working on performance tests, but early results indicate that for wide area network (WAN) transfers, the performance is comparable. When might you want to use this functionality: 2.2.4. Accessing data in some other non-POSIX data source The following information is helpful If you want to use GridFTP to access data in a non-POSIX data source. Architecturally, the Globus GridFTP server can be divided into 3 modules: In the GT4.0.x implementation, the data transform module and the DSI have been merged, although we plan to have separate, chainable, data transform modules in the future.
[Note] Note
This architecture does NOT apply to the WU-FTPD implementation (GT3.2.1 and lower).
2.2.4.1. GridFTP Protocol Module
The GridFTP protocol module is the module that reads and writes to the network and implements the GridFTP protocol. This module should not need to be modified since to do so would make the server non-protocol compliant, and unable to communicate with other servers.
2.2.4.2. Data Transform Functionality
The data transform functionality is invoked by using the ERET (extended retrieve) and ESTO (extended store) commands. It is seldom used and bears careful consideration before it is implemented, but in the right circumstances can be very useful. In theory, any computation could be invoked this way, but it was primarily intended for cases where some simple pre-processing (such as a partial get or sub-sampling) can greatly reduce the network load. The disadvantage to this is that you remove any real option for planning, brokering, etc., and any significant computation could adversely affect the data transfer performance. Note that the client must also support the ESTO/ERET functionality as well.
2.2.4.3. Data Storage Interface (DSI) / Data Transform module
Nov 13, 2018 | toolkit.globus.org

The Data Storage Interface (DSI) / Data Transform module knows how to read and write to the "local" storage system and can optionally transform the data. We put local in quotes because in a complicated storage system, the storage may not be directly attached, but for performance reasons, it should be relatively close (for instance on the same LAN).

The interface consists of functions to be implemented such as send (get), receive (put), command (simple commands that simply succeed or fail like mkdir), etc..

Once these functions have been implemented for a specific storage system, a client should not need to know or care what is actually providing the data. The server can either be configured specifically with a specific DSI, i.e., it knows how to interact with a single class of storage system, or one particularly useful function for the ESTO/ERET functionality mentioned above is to load and configure a DSI on the fly. 3. Command line tools

Please see the GridFTP Command Reference .

[Nov 13, 2018] Resuming rsync partial (-P/--partial) on a interrupted transfer

Notable quotes:
"... should ..."
May 15, 2013 | stackoverflow.com

Glitches , May 15, 2013 at 18:06

I am trying to backup my file server to a remove file server using rsync. Rsync is not successfully resuming when a transfer is interrupted. I used the partial option but rsync doesn't find the file it already started because it renames it to a temporary file and when resumed it creates a new file and starts from beginning.

Here is my command:

rsync -avztP -e "ssh -p 2222" /volume1/ myaccont@backup-server-1:/home/myaccount/backup/ --exclude "@spool" --exclude "@tmp"

When this command is ran, a backup file named OldDisk.dmg from my local machine get created on the remote machine as something like .OldDisk.dmg.SjDndj23 .

Now when the internet connection gets interrupted and I have to resume the transfer, I have to find where rsync left off by finding the temp file like .OldDisk.dmg.SjDndj23 and rename it to OldDisk.dmg so that it sees there already exists a file that it can resume.

How do I fix this so I don't have to manually intervene each time?

Richard Michael , Nov 6, 2013 at 4:26

TL;DR : Use --timeout=X (X in seconds) to change the default rsync server timeout, not --inplace .

The issue is the rsync server processes (of which there are two, see rsync --server ... in ps output on the receiver) continue running, to wait for the rsync client to send data.

If the rsync server processes do not receive data for a sufficient time, they will indeed timeout, self-terminate and cleanup by moving the temporary file to it's "proper" name (e.g., no temporary suffix). You'll then be able to resume.

If you don't want to wait for the long default timeout to cause the rsync server to self-terminate, then when your internet connection returns, log into the server and clean up the rsync server processes manually. However, you must politely terminate rsync -- otherwise, it will not move the partial file into place; but rather, delete it (and thus there is no file to resume). To politely ask rsync to terminate, do not SIGKILL (e.g., -9 ), but SIGTERM (e.g., pkill -TERM -x rsync - only an example, you should take care to match only the rsync processes concerned with your client).

Fortunately there is an easier way: use the --timeout=X (X in seconds) option; it is passed to the rsync server processes as well.

For example, if you specify rsync ... --timeout=15 ... , both the client and server rsync processes will cleanly exit if they do not send/receive data in 15 seconds. On the server, this means moving the temporary file into position, ready for resuming.

I'm not sure of the default timeout value of the various rsync processes will try to send/receive data before they die (it might vary with operating system). In my testing, the server rsync processes remain running longer than the local client. On a "dead" network connection, the client terminates with a broken pipe (e.g., no network socket) after about 30 seconds; you could experiment or review the source code. Meaning, you could try to "ride out" the bad internet connection for 15-20 seconds.

If you do not clean up the server rsync processes (or wait for them to die), but instead immediately launch another rsync client process, two additional server processes will launch (for the other end of your new client process). Specifically, the new rsync client will not re-use/reconnect to the existing rsync server processes. Thus, you'll have two temporary files (and four rsync server processes) -- though, only the newer, second temporary file has new data being written (received from your new rsync client process).

Interestingly, if you then clean up all rsync server processes (for example, stop your client which will stop the new rsync servers, then SIGTERM the older rsync servers, it appears to merge (assemble) all the partial files into the new proper named file. So, imagine a long running partial copy which dies (and you think you've "lost" all the copied data), and a short running re-launched rsync (oops!).. you can stop the second client, SIGTERM the first servers, it will merge the data, and you can resume.

Finally, a few short remarks:

JamesTheAwesomeDude , Dec 29, 2013 at 16:50

Just curious: wouldn't SIGINT (aka ^C ) be 'politer' than SIGTERM ? – JamesTheAwesomeDude Dec 29 '13 at 16:50

Richard Michael , Dec 29, 2013 at 22:34

I didn't test how the server-side rsync handles SIGINT, so I'm not sure it will keep the partial file - you could check. Note that this doesn't have much to do with Ctrl-c ; it happens that your terminal sends SIGINT to the foreground process when you press Ctrl-c , but the server-side rsync has no controlling terminal. You must log in to the server and use kill . The client-side rsync will not send a message to the server (for example, after the client receives SIGINT via your terminal Ctrl-c ) - might be interesting though. As for anthropomorphizing, not sure what's "politer". :-) – Richard Michael Dec 29 '13 at 22:34

d-b , Feb 3, 2015 at 8:48

I just tried this timeout argument rsync -av --delete --progress --stats --human-readable --checksum --timeout=60 --partial-dir /tmp/rsync/ rsync://$remote:/ /src/ but then it timed out during the "receiving file list" phase (which in this case takes around 30 minutes). Setting the timeout to half an hour so kind of defers the purpose. Any workaround for this? – d-b Feb 3 '15 at 8:48

Cees Timmerman , Sep 15, 2015 at 17:10

@user23122 --checksum reads all data when preparing the file list, which is great for many small files that change often, but should be done on-demand for large files. – Cees Timmerman Sep 15 '15 at 17:10

[Nov 12, 2018] Linux Find Out Which Process Is Listening Upon a Port

Jun 25, 2012 | www.cyberciti.biz

How do I find out running processes were associated with each open port? How do I find out what process has open tcp port 111 or udp port 7000 under Linux?

You can the following programs to find out about port numbers and its associated process:

  1. netstat – a command-line tool that displays network connections, routing tables, and a number of network interface statistics.
  2. fuser – a command line tool to identify processes using files or sockets.
  3. lsof – a command line tool to list open files under Linux / UNIX to report a list of all open files and the processes that opened them.
  4. /proc/$pid/ file system – Under Linux /proc includes a directory for each running process (including kernel processes) at /proc/PID, containing information about that process, notably including the processes name that opened port.

You must run above command(s) as the root user.

netstat example

Type the following command:
# netstat -tulpn
Sample outputs:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1138/mysqld     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      850/portmap     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1607/apache2    
tcp        0      0 0.0.0.0:55091           0.0.0.0:*               LISTEN      910/rpc.statd   
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1467/dnsmasq    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      992/sshd        
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1565/cupsd      
tcp        0      0 0.0.0.0:7000            0.0.0.0:*               LISTEN      3813/transmission
tcp6       0      0 :::22                   :::*                    LISTEN      992/sshd        
tcp6       0      0 ::1:631                 :::*                    LISTEN      1565/cupsd      
tcp6       0      0 :::7000                 :::*                    LISTEN      3813/transmission
udp        0      0 0.0.0.0:111             0.0.0.0:*                           850/portmap     
udp        0      0 0.0.0.0:662             0.0.0.0:*                           910/rpc.statd   
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1467/dnsmasq    
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1467/dnsmasq    
udp        0      0 0.0.0.0:68              0.0.0.0:*                           3697/dhclient   
udp        0      0 0.0.0.0:7000            0.0.0.0:*                           3813/transmission
udp        0      0 0.0.0.0:54746           0.0.0.0:*                           910/rpc.statd

TCP port 3306 was opened by mysqld process having PID # 1138. You can verify this using /proc, enter:
# ls -l /proc/1138/exe
Sample outputs:

lrwxrwxrwx 1 root root 0 2010-10-29 10:20 /proc/1138/exe -> /usr/sbin/mysqld

You can use grep command to filter out information:
# netstat -tulpn | grep :80
Sample outputs:

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1607/apache2
Video demo

https://www.youtube.com/embed/h3fJlmuGyos

fuser command

Find out the processes PID that opened tcp port 7000, enter:
# fuser 7000/tcp
Sample outputs:

7000/tcp:             3813

Finally, find out process name associated with PID # 3813, enter:
# ls -l /proc/3813/exe
Sample outputs:

lrwxrwxrwx 1 vivek vivek 0 2010-10-29 11:00 /proc/3813/exe -> /usr/bin/transmission

/usr/bin/transmission is a bittorrent client, enter:
# man transmission
OR
# whatis transmission
Sample outputs:

transmission (1)     - a bittorrent client
Task: Find Out Current Working Directory Of a Process

To find out current working directory of a process called bittorrent or pid 3813, enter:
# ls -l /proc/3813/cwd
Sample outputs:

lrwxrwxrwx 1 vivek vivek 0 2010-10-29 12:04 /proc/3813/cwd -> /home/vivek

OR use pwdx command, enter:
# pwdx 3813
Sample outputs:

3813: /home/vivek
Task: Find Out Owner Of a Process

Use the following command to find out the owner of a process PID called 3813:
# ps aux | grep 3813
OR
# ps aux | grep '[3]813'
Sample outputs:

vivek     3813  1.9  0.3 188372 26628 ?        Sl   10:58   2:27 transmission

OR try the following ps command:
# ps -eo pid,user,group,args,etime,lstart | grep '[3]813'
Sample outputs:

3813 vivek    vivek    transmission                   02:44:05 Fri Oct 29 10:58:40 2010

Another option is /proc/$PID/environ, enter:
# cat /proc/3813/environ
OR
# grep --color -w -a USER /proc/3813/environ
Sample outputs (note –colour option):

Fig.01: grep output
Fig.01: grep output

lsof Command Example

Type the command as follows:

lsof -i :portNumber 
lsof -i tcp:portNumber 
lsof -i udp:portNumber 
lsof -i :80
lsof -i :80 | grep LISTEN

lsof -i :portNumber lsof -i tcp:portNumber lsof -i udp:portNumber lsof -i :80 lsof -i :80 | grep LISTEN

Sample outputs:

apache2   1607     root    3u  IPv4   6472      0t0  TCP *:www (LISTEN)
apache2   1616 www-data    3u  IPv4   6472      0t0  TCP *:www (LISTEN)
apache2   1617 www-data    3u  IPv4   6472      0t0  TCP *:www (LISTEN)
apache2   1618 www-data    3u  IPv4   6472      0t0  TCP *:www (LISTEN)
apache2   1619 www-data    3u  IPv4   6472      0t0  TCP *:www (LISTEN)
apache2   1620 www-data    3u  IPv4   6472      0t0  TCP *:www (LISTEN)

Now, you get more information about pid # 1607 or 1616 and so on:
# ps aux | grep '[1]616'
Sample outputs:
www-data 1616 0.0 0.0 35816 3880 ? S 10:20 0:00 /usr/sbin/apache2 -k start
I recommend the following command to grab info about pid # 1616:
# ps -eo pid,user,group,args,etime,lstart | grep '[1]616'
Sample outputs:

1616 www-data www-data /usr/sbin/apache2 -k start     03:16:22 Fri Oct 29 10:20:17 2010

Where,

Help: I Discover an Open Port Which I Don't Recognize At All

The file /etc/services is used to map port numbers and protocols to service names. Try matching port numbers:
$ grep port /etc/services
$ grep 443 /etc/services

Sample outputs:

https		443/tcp				# http protocol over TLS/SSL
https		443/udp
Check For rootkit

I strongly recommend that you find out which processes are really running, especially servers connected to the high speed Internet access. You can look for rootkit which is a program designed to take fundamental control (in Linux / UNIX terms "root" access, in Windows terms "Administrator" access) of a computer system, without authorization by the system's owners and legitimate managers. See how to detecting / checking rootkits under Linux .

Keep an Eye On Your Bandwidth Graphs

Usually, rooted servers are used to send a large number of spam or malware or DoS style attacks on other computers.

See also:

See the following man pages for more information:
$ man ps
$ man grep
$ man lsof
$ man netstat
$ man fuser

Posted by: Vivek Gite

The author is the creator of nixCraft and a seasoned sysadmin, DevOps engineer, and a trainer for the Linux operating system/Unix shell scripting. Get the latest tutorials on SysAdmin, Linux/Unix and open source topics via RSS/XML feed or weekly email newsletter . GOT FEEDBACK? CLICK HERE TO JOIN THE DISCUSSION

[Nov 12, 2018] Shell Games Linux Magazine

Nov 12, 2018 | www.linux-magazine.com

First pdsh Commands

To begin, I'll try to get the kernel version of a node by using its IP address:

$ pdsh -w 192.168.1.250 uname -r
192.168.1.250: 2.6.32-431.11.2.el6.x86_64

The -w option means I am specifying the node(s) that will run the command. In this case, I specified the IP address of the node (192.168.1.250). After the list of nodes, I add the command I want to run, which is uname -r in this case. Notice that pdsh starts the output line by identifying the node name.

If you need to mix rcmd modules in a single command, you can specify which module to use in the command line,

$ pdsh -w ssh:laytonjb@192.168.1.250 uname -r
192.168.1.250: 2.6.32-431.11.2.el6.x86_64

by putting the rcmd module before the node name. In this case, I used ssh and typical ssh syntax.

A very common way of using pdsh is to set the environment variable WCOLL to point to the file that contains the list of hosts you want to use in the pdsh command. For example, I created a subdirectory PDSH where I create a file hosts that lists the hosts I want to use:

[laytonjb@home4 ~]$ mkdir PDSH
[laytonjb@home4 ~]$ cd PDSH
[laytonjb@home4 PDSH]$ vi hosts
[laytonjb@home4 PDSH]$ more hosts
192.168.1.4
192.168.1.250

I'm only using two nodes: 192.168.1.4 and 192.168.1.250. The first is my test system (like a cluster head node), and the second is my test compute node. You can put hosts in the file as you would on the command line separated by commas. Be sure not to put a blank line at the end of the file because pdsh will try to connect to it. You can put the environment variable WCOLL in your .bashrc file:

export WCOLL=/home/laytonjb/PDSH/hosts

As before, you can source your .bashrc file, or you can log out and log back in. Specifying Hosts

I won't list all the several other ways to specify a list of nodes, because the pdsh website [9] discusses virtually all of them; however, some of the methods are pretty handy. The simplest way is to specify the nodes on the command line is to use the -w option:

$ pdsh -w 192.168.1.4,192.168.1.250 uname -r
192.168.1.4: 2.6.32-431.17.1.el6.x86_64
192.168.1.250: 2.6.32-431.11.2.el6.x86_64

In this case, I specified the node names separated by commas. You can also use a range of hosts as follows:

pdsh -w host[1-11]
pdsh -w host[1-4,8-11]

In the first case, pdsh expands the host range to host1, host2, host3, , host11. In the second case, it expands the hosts similarly (host1, host2, host3, host4, host8, host9, host10, host11). You can go to the pdsh website for more information on hostlist expressions [10] .

Another option is to have pdsh read the hosts from a file other than the one to which WCOLL points. The command shown in Listing 2 tells pdsh to take the hostnames from the file /tmp/hosts , which is listed after -w ^ (with no space between the "^" and the filename). You can also use several host files,

Listing 2 Read Hosts from File
$ more /tmp/hosts
192.168.1.4
$ more /tmp/hosts2
192.168.1.250
$ pdsh -w ^/tmp/hosts,^/tmp/hosts2 uname -r
192.168.1.4: 2.6.32-431.17.1.el6.x86_64
192.168.1.250: 2.6.32-431.11.2.el6.x86_64

or you can exclude hosts from a list:

$ pdsh -w -192.168.1.250 uname -r
192.168.1.4: 2.6.32-431.17.1.el6.x86_64

The option -w -192.168.1.250 excluded node 192.168.1.250 from the list and only output the information for 192.168.1.4. You can also exclude nodes using a node file:

$ pdsh -w -^/tmp/hosts2 uname -r
192.168.1.4: 2.6.32-431.17.1.el6.x86_64

In this case, /tmp/hosts2 contains 192.168.1.250, which isn't included in the output. Using the -x option with a hostname,

$ pdsh -x 192.168.1.4 uname -r
192.168.1.250: 2.6.32-431.11.2.el6.x86_64
$ pdsh -x ^/tmp/hosts uname -r
192.168.1.250: 2.6.32-431.11.2.el6.x86_64
$ more /tmp/hosts
192.168.1.4

or a list of hostnames to be excluded from the command to run also works.

More Useful pdsh Commands

Now I can shift into second gear and try some fancier pdsh tricks. First, I want to run a more complicated command on all of the nodes ( Listing 3 ). Notice that I put the entire command in quotes. This means the entire command is run on each node, including the first ( cat /proc/cpuinfo ) and second ( grep bogomips ) parts.

Listing 3 Quotation Marks 1

In the output, the node precedes the command results, so you can tell what output is associated with which node. Notice that the BogoMips values are different on the two nodes, which is perfectly understandable because the systems are different. The first node has eight cores (four cores and four Hyper-Thread cores), and the second node has four cores.

You can use this command across a homogeneous cluster to make sure all the nodes are reporting back the same BogoMips value. If the cluster is truly homogeneous, this value should be the same. If it's not, then I would take the offending node out of production and check it.

A slightly different command shown in Listing 4 runs the first part contained in quotes, cat /proc/cpuinfo , on each node and the second part of the command, grep bogomips , on the node on which you issue the pdsh command.

Listing 4 Quotation Marks 2

The point here is that you need to be careful on the command line. In this example, the differences are trivial, but other commands could have differences that might be difficult to notice.

One very important thing to note is that pdsh does not guarantee a return of output in any particular order. If you have a list of 20 nodes, the output does not necessarily start with node 1 and increase incrementally to node 20. For example, in Listing 5 , I run vmstat on each node and get three lines of output from each node.

[Nov 12, 2018] Edge Computing vs. Cloud Computing What's the Difference by Andy Patrizio ,

"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."
Notable quotes:
"... Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business ..."
"... Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work. ..."
"... Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet." ..."
Jan 23, 2018 | www.datamation.com
Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business

The term cloud computing is now as firmly lodged in our technical lexicon as email and Internet, and the concept has taken firm hold in business as well. By 2020, Gartner estimates that a "no cloud" policy will be as prevalent in business as a "no Internet" policy. Which is to say no one who wants to stay in business will be without one.

You are likely hearing a new term now, edge computing . One of the problems with technology is terms tend to come before the definition. Technologists (and the press, let's be honest) tend to throw a word around before it is well-defined, and in that vacuum come a variety of guessed definitions, of varying accuracy.

Edge computing is a term you are going to hear more of in the coming years because it precedes another term you will be hearing a lot, the Internet of Things (IoT). You see, the formally adopted definition of edge computing is a form of technology that is necessary to make the IoT work.

Tech research firm IDC defines edge computing is a "mesh network of micro data centers that process or store critical data locally and push all received data to a central data center or cloud storage repository, in a footprint of less than 100 square feet."

It is typically used in IoT use cases, where edge devices collect data from IoT devices and do the processing there, or send it back to a data center or the cloud for processing. Edge computing takes some of the load off the central data center, reducing or even eliminating the processing work at the central location.

IoT Explosion in the Cloud Era

To understand the need for edge computing you must understand the explosive growth in IoT in the coming years, and it is coming on big. There have been a number of estimates of the growth in devices, and while they all vary, they are all in the billions of devices.

This is taking place in a number of areas, most notably cars and industrial equipment. Cars are becoming increasingly more computerized and more intelligent. Gone are the days when the "Check engine" warning light came on and you had to guess what was wrong. Now it tells you which component is failing.

The industrial sector is a broad one and includes sensors, RFID, industrial robotics, 3D printing, condition monitoring, smart meters, guidance, and more. This sector is sometimes called the Industrial Internet of Things (IIoT) and the overall market is expected to grow from $93.9 billion in 2014 to $151.01 billion by 2020.

All of these sensors are taking in data but they are not processing it. Your car does some of the processing of sensor data but much of it has to be sent in to a data center for computation, monitoring and logging.

The problem is that this would overload networks and data centers. Imaging the millions of cars on the road sending in data to data centers around the country. The 4G network would be overwhelmed, as would the data centers. And if you are in California and the car maker's data center is in Texas, that's a long round trip.

[Nov 09, 2018] Cloud-hosted date must be accessed by users over existing WAN which creates performance issues due to bandwidth and latency constraints

Notable quotes:
"... Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps ..."
Nov 09, 2018 | www.eiseverywhere.com

However, cloud-hosted information assets must still be accessed by users over existing WAN infrastructures, where there are performance issues due to bandwidth and latency constraints.

THE EXTREMELY UNFUNNY PART - UP TO 20x SLOWER

Public/Private

Cloud

Thousands of companies
Millions of users
Varied bandwidth

♦ Per-unit provisioning costs does not decrease much with size after, say, 100 units.

> Cloud data centers are potentially "far away"

♦ Cloud infrastructure supports many enterprises

♦ Large scale drives lower per-unit cost for data center
services

> All employees will be "remote" from their data

♦ Even single-location companies will be remote from their data

♦ HQ employees previously local to servers, but not with Cloud model

> Lots of data needs to be sent over limited WAN bandwidth

Congestion problems lead to miserable performance. We have one WAN pipe, typically 1.5 Mbps to 10 MBps

> Disk-based deduplication technology

♦ Identify redundant data at the byte level, not application (e.g., file) level

♦ Use disks to store vast dictionaries of byte sequences for long periods of time

♦ Use symbols to transfer repetitive sequences of byte-level raw data

♦ Only deduplicated data stored on disk

[Nov 09, 2018] Troubleshoot WAN Performance Issues SD Wan Experts by Steve Garson

Feb 08, 2013 | www.sd-wan-experts.com

Troubleshooting MPLS Networks

How should you troubleshoot WAN performance issues. Your MPLS or VPLS network and your clients in field offices are complaining about slow WAN performance. Your network should be performing better and you can't figure out what the problem is. You can contact SD-WAN-Experts to have their engineers solve your problem, but you want to try to solve the problems yourself.

  1. The first thing to check, seems trivial, but you need to confirm that the ports on your router and switch ports are configured for the same speed and duplex. Log into your switches and check the logs for mismatches of speed or duplex. Auto-negotiation sometimes does not work properly, so a 10M port connected to a 100M port is mismatched. Or you might have a half-duplex port connected to a full-duplex port. Don't assume that a 10/100/1000 port is auto-negotiating correctly!
  2. Is your WAN performance problem consistent? Does it occur at roughly the same time of day? Or is it completely random? If you don't have the monitoring tools to measure this, you are at a big disadvantage in resolving the issues on your own.
  3. Do you have Class of Service configured on your WAN? Do you have DSCP configured on your LAN? What is the mapping of your DSCP values to CoS?
  4. What kind of applications are traversing your WAN? Are there specific apps that work better than others?
  5. Have your reviewed bandwidth utilization on your carrier's web portal to determine if you are saturating the MPLS port of any locations? Even brief peaks will be enough to generate complaints. Large files, such as CAD drawings, can completely saturate a WAN link.
  6. Are you backing up or synchronizing data over the WAN? Have you confirmed 100% that this work is completed before the work day begins.
  7. Might your routing be taking multiple paths and not the most direct path? Look at your routing tables.
  8. Next, you want to see long term trend statistics. This means monitoring the SNMP streams from all your routers, using tools such as MRTG, NTOP or Cacti. A two week sampling should provide a very good picture of what is happening on your network to help troubleshoot your WAN.

NTOP allows you to

MRTG (Multi-Router Traffic Grapher) provides easy to understand graphs of your network bandwidth utilization.

MRTG Picture

Cacti requires a MySQL database. It is a complete network graphing solution designed to harness the power of RRDTool 's data storage and graphing functionality. Cacti provides a fast poller, advanced graph templating, multiple data acquisition methods, and user management features out of the box. All of this is wrapped in an intuitive, easy to use interface that makes sense for LAN-sized installations up to complex networks with hundreds of devices.

Both NTOP and MRTG are freeware applications to help troubleshoot your WAN that will run on the freeware versions of Linux. As a result, they can be installed on almost any desktop computer that has out-lived its value as a Windows desktop machine. If you are skilled with Linux and networking, and you have the time, you can install this monitoring system on your own. You will need to get your carrier to provide read-only access to your router SNMP traffic.

But you might find it more cost effective to have the engineers at SD-WAN-Experts do the work for you. All you need to do is provide an available machine with a Linux install (Ubuntu, CentOS, RedHat, etc) with remote access via a VPN. Our engineers will then download all the software remotely, install and configure the machine. When we are done with the monitoring, beside understanding how to solve your problem (and solving it!) you will have your own network monitoring system installed for your use on a daily basis. We'll teach you how to use it, which is quite simple using the web based tools, so you can view it from any machine on your network.

If you need assistance in troubleshooting your wide area network, contact SD-WAN-Experts today !

You might also find these troubleshooting tips of interest;

Troubleshooting MPLS Network Performance Issues

Packet Loss and How It Affects Performance

Troubleshooting VPLS and Ethernet Tunnels over MPLS

[Nov 09, 2018] Storage in private clouds

Nov 09, 2018 | www.redhat.com

Storage in private clouds

Storage is one of the most popular uses of cloud computing, particularly for consumers. The user-friendly design of service-based companies have helped make "cloud" a pretty normal term -- even reaching meme status in 2016.

However, cloud storage means something very different to businesses. Big data and the Internet of Things (IoT) have made it difficult to appraise the value of data until long after it's originally stored -- when finding that piece of data becomes the key to revealing valuable business insights or unlocking an application's new feature. Even after enterprises decide where to store their data in the cloud (on-premise, off-premise, public, or private), they still have to decide how they're going to store it. What good is data that can't be found?

It's common to store data in the cloud using software-defined storage . Software-defined storage decouples storage software from hardware so you can abstract and consolidate storage capacity in a cloud. It allows you to scale beyond whatever individual hardware components your cloud is built on.

Two of the more common software-defined storage solutions include Ceph for structured data and Gluster for unstructured data. Ceph is a massively scalable, programmable storage system that works well with clouds -- particularly those deployed using OpenStack ® -- because of its ability to unify object, block, and file storage into 1 pool of resources. Gluster is designed to handle the requirements of traditional file storage and is particularly adept at provisioning and managing elastic storage for container-based applications.

[Nov 09, 2018] Cloud Computing vs Edge Computing Which Will Prevail

Notable quotes:
"... The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing. ..."
"... For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles ..."
"... the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud. ..."
Nov 09, 2018 | www.lannerinc.com

The recent widespread of edge computing in some 5G showcases, like the major sports events, has generated the ongoing discussion about the possibility of edge computing to replace cloud computing.

In fact, there have been announcements from global tech leaders like Nokia and Huawei demonstrating increased efforts and resources in developing edge computing.

For instance, Satya Nadella, the CEO of Microsoft, announced in Microsoft Build 2017 that the company will focus its strategy on edge computing. Indeed, edge computing will be the key for the success of smart home and driverless vehicles.

... ... ...

Cloud or edge, which will lead the future?

The answer to that question is "Cloud – Edge Mixing". The cloud and the edge will complement each other to offer the real IoT experience. For instance, while the cloud coordinates all the technology and offers SaaS to users, the edge will be the first to process and store the data generated by user devices. This will reduce the latency for the data to travel to the cloud. In other words, the edge optimizes the efficiency for the cloud.

It is strongly suggested to implement open architecture white-box servers for both cloud and edge, to minimize the latency for cloud-edge synchronization and optimize the compatibility between the two. For example, Lanner Electronics offers a wide range of Intel x86 white box appliances for data centers and edge uCPE/vCPE.

http://www.lannerinc.com/telecom-datacenter-appliances/vcpe/ucpe-platforms/

[Nov 09, 2018] OpenStack is overkill for Docker

Notable quotes:
"... OpenStack's core value is to gather a pool of hypervisor-enabled computers and enable the delivery of virtual machines (VMs) on demand to users. ..."
Nov 09, 2018 | www.techrepublic.com

javascript:void(0)

Both OpenStack and Docker were conceived to make IT more agile. OpenStack has strived to do this by turning hitherto static IT resources into elastic infrastructure, whereas Docker has reached for this goal by harmonizing development, test, and production resources, as Red Hat's Neil Levine suggests .

But while Docker adoption has soared, OpenStack is still largely stuck in neutral. OpenStack is kept relevant by so many wanting to believe its promise, but never hitting its stride due to a host of factors , including complexity.

And yet Docker could be just the thing to turn OpenStack's popularity into productivity. Whether a Docker-plus-OpenStack pairing is right for your enterprise largely depends on the kind of capacity your enterprise hopes to deliver. If simply Docker, OpenStack is probably overkill.

An open source approach to delivering virtual machines

OpenStack is an operational model for delivering virtualized compute capacity.

Sure, some give it a more grandiose definition ("OpenStack is a set of software tools for building and managing cloud computing platforms for public and private clouds"), but if we ignore secondary services like Cinder, Heat, and Magnum, for example, OpenStack's core value is to gather a pool of hypervisor-enabled computers and enable the delivery of virtual machines (VMs) on demand to users.

That's it.

Not that this is a small thing. After all, without OpenStack, the hypervisor sits idle, lonesome on a single computer, with no way to expose that capacity programmatically (or otherwise) to users.

Before cloudy systems like OpenStack or Amazon's EC2, users would typically file a help ticket with IT. An IT admin, in turn, would use a GUI or command line to create a VM, and then share the credentials with the user.

Systems like OpenStack significantly streamline this process, enabling IT to programmatically deliver capacity to users. That's a big deal.

Docker peanut butter, meet OpenStack jelly

Docker, the darling of the containers world, is similar to the VM in the IaaS picture painted above.

A Docker host is really the unit of compute capacity that users need, and not the container itself. Docker addresses what you do with a host once you've got it, but it doesn't really help you get the host in the first place.

A Docker machine provides a client-side tool that lets you request Docker hosts from an IaaS provider (like EC2 or OpenStack or vSphere), but it's far from a complete solution. In part, this stems from the fact that Docker doesn't have a tenancy model.

With a hypervisor, each VM is a tenant. But in Docker, the Docker host is a tenant. You typically don't want multiple users sharing a Docker host because then they see each others' containers. So typically an enterprise will layer a cloud system underneath Docker to add tenancy. This yields a stack that looks like: hardware > hypervisor > Docker host > container.

A common approach today would be to take OpenStack and use it as the enterprise platform to deliver capacity on demand to users. In other words, users rely on OpenStack to request a Docker host, and then they use Docker to run containers in their Docker host.

So far, so good.

If all you need is Docker...

Things get more complicated when we start parsing what capacity needs delivering.

When an enterprise wants to use Docker, they need to get Docker hosts from a data center. OpenStack can do that, and it can do it alongside delivering all sorts of other capacity to the various teams within the enterprise.

But if all an enterprise IT team needs is Docker containers delivered, then OpenStack -- or a similar orchestration tool -- may be overkill, as VMware executive Jared Rosoff told me.

For this sort of use case, we really need a new platform. This platform could take the form of a piece of software that an enterprise installs on all of its computers in the data center. It would expose an interface to developers that lets them programmatically create Docker hosts when they need them, and then use Docker to create containers in those hosts.

Google has a vision for something like this with its Google Container Engine . Amazon has something similar in its EC2 Container Service . These are both API's that developers can use to provision some Docker-compatible capacity from their data center.

As for Docker, the company behind Docker, the technology, it seems to have punted on this problem. focusing instead on what happens on the host itself.

While we probably don't need to build up a big OpenStack cloud simply to manage Docker instances, it's worth asking what OpenStack should look like if what we wanted to deliver was only Docker hosts, and not VMs.

Again, we see Google and Amazon tackling the problem, but when will OpenStack, or one of its supporters, do the same? The obvious candidate would be VMware, given its longstanding dominance of tooling around virtualization. But the company that solves this problem first, and in a way that comforts traditional IT with familiar interfaces yet pulls them into a cloudy future, will win, and win big.

[Nov 09, 2018] What is Hybrid Cloud Computing

Nov 09, 2018 | www.dummies.com

The hybrid cloud

A hybrid cloud is a combination of a private cloud combined with the use of public cloud services where one or several touch points exist between the environments. The goal is to combine services and data from a variety of cloud models to create a unified, automated, and well-managed computing environment.

Combining public services with private clouds and the data center as a hybrid is the new definition of corporate computing. Not all companies that use some public and some private cloud services have a hybrid cloud. Rather, a hybrid cloud is an environment where the private and public services are used together to create value.

A cloud is hybrid

A cloud is not hybrid

[Nov 09, 2018] Why Micro Data Centers Deliver Good Things in Small Packages by Calvin Hennick

Notable quotes:
"... "There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture." ..."
"... In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing . ..."
Nov 09, 2018 | solutions.cdw.com

Enterprises are deploying self-contained micro data centers to power computing at the network edge.

The location for data processing has changed significantly throughout the history of computing. During the mainframe era, data was processed centrally, but client/server architectures later decentralized computing. In recent years, cloud computing centralized many processing workloads, but digital transformation and the Internet of Things are poised to move computing to new places, such as the network edge .

"There's a big transformation happening," says Thomas Humphrey, segment director for edge computing at APC . "Technologies like IoT have started to require that some local computing and storage happen out in that distributed IT architecture."

For example, some IoT systems require processing of data at remote locations rather than a centralized data center , such as at a retail store instead of a corporate headquarters.

To meet regulatory requirements and business needs, IoT solutions often need low latency, high bandwidth, robust security and superior reliability . To meet these demands, many organizations are deploying micro data centers: self-contained solutions that provide not only essential infrastructure, but also physical security, power and cooling and remote management capabilities.

"Digital transformation happens at the network edge, and edge computing will happen inside micro data centers ," says Bruce A. Taylor, executive vice president at Datacenter Dynamics . "This will probably be one of the fastest growing segments -- if not the fastest growing segment -- in data centers for the foreseeable future."

What Is a Micro Data Center?

Delivering the IT capabilities needed for edge computing represents a significant challenge for many organizations, which need manageable and secure solutions that can be deployed easily, consistently and close to the source of computing . Vendors such as APC have begun to create comprehensive solutions that provide these necessary capabilities in a single, standardized package.

"From our perspective at APC, the micro data center was a response to what was happening in the market," says Humphrey. "We were seeing that enterprises needed more robust solutions at the edge."

Most micro data center solutions rely on hyperconverged infrastructure to integrate computing, networking and storage technologies within a compact footprint . A typical micro data center also incorporates physical infrastructure (including racks), fire suppression, power, cooling and remote management capabilities. In effect, the micro data center represents a sweet spot between traditional IT closets and larger modular data centers -- giving organizations the ability to deploy professional, powerful IT resources practically anywhere .

Standardized Deployments Across the Country

Having robust IT resources at the network edge helps to improve reliability and reduce latency, both of which are becoming more and more important as analytics programs require that data from IoT deployments be processed in real time .

"There's always been edge computing," says Taylor. "What's new is the need to process hundreds of thousands of data points for analytics at once."

Standardization, redundant deployment and remote management are also attractive features, especially for large organizations that may need to deploy tens, hundreds or even thousands of micro data centers. "We spoke to customers who said, 'I've got to roll out and install 3,500 of these around the country,'" says Humphrey. "And many of these companies don't have IT staff at all of these sites." To address this scenario, APC designed standardized, plug-and-play micro data centers that can be rolled out seamlessly. Additionally, remote management capabilities allow central IT departments to monitor and troubleshoot the edge infrastructure without costly and time-intensive site visits.

In part because micro data centers operate in far-flung environments, security is of paramount concern. The self-contained nature of micro data centers ensures that only authorized personnel will have access to infrastructure equipment , and security tools such as video surveillance provide organizations with forensic evidence in the event that someone attempts to infiltrate the infrastructure.

How Micro Data Centers Can Help in Retail, Healthcare

Micro data centers make business sense for any organization that needs secure IT infrastructure at the network edge. But the solution is particularly appealing to organizations in fields such as retail, healthcare and finance , where IT environments are widely distributed and processing speeds are often a priority.

In retail, for example, edge computing will become more important as stores find success with IoT technologies such as mobile beacons, interactive mirrors and real-time tools for customer experience, behavior monitoring and marketing .

"It will be leading-edge companies driving micro data center adoption, but that doesn't necessarily mean they'll be technology companies," says Taylor. "A micro data center can power real-time analytics for inventory control and dynamic pricing in a supermarket."

In healthcare, digital transformation is beginning to touch processes and systems ranging from medication carts to patient records, and data often needs to be available locally; for example, in case of a data center outage during surgery. In finance, the real-time transmission of data can have immediate and significant financial consequences. And in both of these fields, regulations governing data privacy make the monitoring and security features of micro data centers even more important.

Micro data centers also have enormous potential to power smart city initiatives and to give energy companies a cost-effective way of deploying resources in remote locations , among other use cases.

"The proliferation of edge computing will be greater than anything we've seen in the past," Taylor says. "I almost can't think of a field where this won't matter."

Learn more about how solutions and services from CDW and APC can help your organization overcome its data center challenges.

Micro Data Centers Versus IT Closets

Think the micro data center is just a glorified update on the traditional IT closet? Think again.

"There are demonstrable differences," says Bruce A. Taylor, executive vice president at Datacenter Dynamics. "With micro data centers, there's a tremendous amount of computing capacity in a very small, contained space, and we just didn't have that capability previously ."

APC identifies three key differences between IT closets and micro data centers:

  1. Difference #1: Uptime Expectations. APC notes that, of the nearly 3 million IT closets in the U.S., over 70 percent report outages directly related to human error. In an unprotected IT closet, problems can result from something as preventable as cleaning staff unwittingly disconnecting a cable. Micro data centers, by contrast, utilize remote monitoring, video surveillance and sensors to reduce downtime related to human error.
  2. Difference #2: Cooling Configurations. The cooling of IT wiring closets is often approached both reactively and haphazardly, resulting in premature equipment failure. Micro data centers are specifically designed to assure cooling compatibility with anticipated loads.
  3. Difference #3: Power Infrastructure. Unlike many IT closets, micro data centers incorporate uninterruptible power supplies, ensuring that infrastructure equipment has the power it needs to help avoid downtime.

Calvin Hennick is a freelance journalist who specializes in business and technology writing. He is a contributor to the CDW family of technology magazines.

[Nov 09, 2018] Solving Office 365 and SaaS Performance Issues with SD-WAN

Notable quotes:
"... most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. ..."
"... Why enterprises overlook the importance of strategically placing cloud gateways ..."
Nov 09, 2018 | www.brighttalk.com

About this webinar Major research highlights that most of the Office365 deployments face network related problems - typically manifesting as screen freezes. Limited WAN optimization capability further complicates the problems for most SaaS applications. To compound the issue, different SaaS applications issue different guidelines for solving performance issues. We will investigate the major reasons for these problems.

SD-WAN provides an essential set of features that solves these networking issues related to Office 365 and SaaS applications. This session will cover the following major topics:

[Nov 09, 2018] Make sense of edge computing vs. cloud computing

Notable quotes:
"... We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure. ..."
"... The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening. ..."
Nov 09, 2018 | www.infoworld.com

The internet of things is real, and it's a real part of the cloud. A key challenge is how you can get data processed from so many devices. Cisco Systems predicts that cloud traffic is likely to rise nearly fourfold by 2020, increasing 3.9 zettabytes (ZB) per year in 2015 (the latest full year for which data is available) to 14.1ZB per year by 2020.

As a result, we could have the cloud computing perfect storm from the growth of IoT. After all, IoT is about processing device-generated data that is meaningful, and cloud computing is about using data from centralized computing and storage. Growth rates of both can easily become unmanageable.

So what do we do? The answer is something called "edge computing." We already know that computing at the edge pushes most of the data processing out to the edge of the network, close to the source of the data. Then it's a matter of dividing the processing between the edge and the centralized system, meaning a public cloud such as Amazon Web Services, Google Cloud, or Microsoft Azure.

That may sound a like a client/server architecture, which also involved figuring out what to do at the client versus at the server. For IoT and any highly distributed applications, you've essentially got a client/network edge/server architecture going on, or -- if your devices can't do any processing themselves, a network edge/server architecture.

The goal is to process near the device the data that it needs quickly, such as to act on. There are hundreds of use cases where reaction time is the key value of the IoT system, and consistently sending the data back to a centralized cloud prevents that value from happening.

You would still use the cloud for processing that is either not as time-sensitive or is not needed by the device, such as for big data analytics on data from all your devices.

There's another dimension to this: edge computing and cloud computing are two very different things. One does not replace the other. But too many articles confuse IT pros by suggesting that edge computing will displace cloud computing. It's no more true than saying PCs would displace the datacenter.

It makes perfect sense to create purpose-built edge computing-based applications, such as an app that places data processing in a sensor to quickly process reactions to alarms. But you're not going to place your inventory-control data and applications at the edge -- moving all compute to the edge would result in a distributed, unsecured, and unmanageable mess.

All the public cloud providers have IoT strategies and technology stacks that include, or will include, edge computing. Edge and cloud computing can and do work well together, but edge computing is for purpose-built systems with special needs. Cloud computing is a more general-purpose platform that also can work with purpose-built systems in that old client/server model.

Related:

David S. Linthicum is a chief cloud strategy officer at Deloitte Consulting, and an internationally recognized industry expert and thought leader. His views are his own.

[Nov 08, 2018] GT 6.0 GridFTP

Notable quotes:
"... GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks ..."
Nov 08, 2018 | toolkit.globus.org

The open source Globus® Toolkit is a fundamental enabling technology for the "Grid," letting people share computing power, databases, and other tools securely online across corporate, institutional, and geographic boundaries without sacrificing local autonomy. The toolkit includes software services and libraries for resource monitoring, discovery, and management, plus security and file management. In addition to being a central part of science and engineering projects that total nearly a half-billion dollars internationally, the Globus Toolkit is a substrate on which leading IT companies are building significant commercial Grid products.

The toolkit includes software for security, information infrastructure, resource management, data management, communication, fault detection, and portability. It is packaged as a set of components that can be used either independently or together to develop applications. Every organization has unique modes of operation, and collaboration between multiple organizations is hindered by incompatibility of resources such as data archives, computers, and networks. The Globus Toolkit was conceived to remove obstacles that prevent seamless collaboration. Its core services, interfaces and protocols allow users to access remote resources as if they were located within their own machine room while simultaneously preserving local control over who can use resources and when.

The Globus Toolkit has grown through an open-source strategy similar to the Linux operating system's, and distinct from proprietary attempts at resource-sharing software. This encourages broader, more rapid adoption and leads to greater technical innovation, as the open-source community provides continual enhancements to the product.

Essential background is contained in the papers " Anatomy of the Grid " by Foster, Kesselman and Tuecke and " Physiology of the Grid " by Foster, Kesselman, Nick and Tuecke.

Acclaim for the Globus Toolkit

From version 1.0 in 1998 to the 2.0 release in 2002 and now the latest 4.0 version based on new open-standard Grid services, the Globus Toolkit has evolved rapidly into what The New York Times called "the de facto standard" for Grid computing. In 2002 the project earned a prestigious R&D 100 award, given by R&D Magazine in a ceremony where the Globus Toolkit was named "Most Promising New Technology" among the year's top 100 innovations. Other honors include project leaders Ian Foster of Argonne National Laboratory and the University of Chicago, Carl Kesselman of the University of Southern California's Information Sciences Institute (ISI), and Steve Tuecke of Argonne being named among 2003's top ten innovators by InfoWorld magazine, and a similar honor from MIT Technology Review, which named Globus Toolkit-based Grid computing one of "Ten Technologies That Will Change the World." The Globus Toolkit also GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks . The GridFTP protocol is based on FTP, the highly-popular Internet file transfer protocol. We have selected a set of protocol features and extensions defined already in IETF RFCs and added a few additional features to meet requirements from current data grid projects.

The following guides are available for this component:

Data Management Key Concepts For important general concepts [ pdf ].
Admin Guide For system administrators and those installing, building and deploying GT. You should already have read the Installation Guide and Quickstart [ pdf ]
User's Guide Describes how end-users typically interact with this component. [ pdf ].
Developer's Guide Reference and usage scenarios for developers. [ pdf ].
Other information available for this component are:
Release Notes What's new with the 6.0 release for this component. [ pdf ]
Public Interface Guide Information for all public interfaces (including APIs, commands, etc). Please note this is a subset of information in the Developer's Guide [ pdf ].
Quality Profile Information about test coverage reports, etc. [ pdf ].
Migrating Guide Information for migrating to this version if you were using a previous version of GT. [ pdf ]
All GridFTP Guides (PDF only) Includes all GridFTP guides except Public Interfaces (which is a subset of the Developer's Guide)

[Nov 08, 2018] globus-gridftp-server-control-6.2-1.el7.x86_64.rpm

Nov 08, 2018 | centos.pkgs.org
6.2 x86_64 EPEL Testing
globus-gridftp-server-control - - -
Requires
Name Value
/sbin/ldconfig -
globus-xio-gsi-driver(x86-64) >= 2
globus-xio-pipe-driver(x86-64) >= 2
libc.so.6(GLIBC_2.14)(64bit) -
libglobus_common.so.0()(64bit) -
libglobus_common.so.0(GLOBUS_COMMON_14)(64bit) -
libglobus_gss_assist.so.3()(64bit) -
libglobus_gssapi_error.so.2()(64bit) -
libglobus_gssapi_gsi.so.4()(64bit) -
libglobus_gssapi_gsi.so.4(globus_gssapi_gsi)(64bit) -
libglobus_openssl_error.so.0()(64bit) -
libglobus_xio.so.0()(64bit) -
rtld(GNU_HASH) -
See Also
Package Description
globus-gridftp-server-control-devel-6.1-1.el7.x86_64.rpm Globus Toolkit - Globus GridFTP Server Library Development Files
globus-gridftp-server-devel-12.5-1.el7.x86_64.rpm Globus Toolkit - Globus GridFTP Server Development Files
globus-gridftp-server-progs-12.5-1.el7.x86_64.rpm Globus Toolkit - Globus GridFTP Server Programs
globus-gridmap-callout-error-2.5-1.el7.x86_64.rpm Globus Toolkit - Globus Gridmap Callout Errors
globus-gridmap-callout-error-devel-2.5-1.el7.x86_64.rpm Globus Toolkit - Globus Gridmap Callout Errors Development Files
globus-gridmap-callout-error-doc-2.5-1.el7.noarch.rpm Globus Toolkit - Globus Gridmap Callout Errors Documentation Files
globus-gridmap-eppn-callout-1.13-1.el7.x86_64.rpm Globus Toolkit - Globus gridmap ePPN callout
globus-gridmap-verify-myproxy-callout-2.9-1.el7.x86_64.rpm Globus Toolkit - Globus gridmap myproxy callout
globus-gsi-callback-5.13-1.el7.x86_64.rpm Globus Toolkit - Globus GSI Callback Library
globus-gsi-callback-devel-5.13-1.el7.x86_64.rpm Globus Toolkit - Globus GSI Callback Library Development Files
globus-gsi-callback-doc-5.13-1.el7.noarch.rpm Globus Toolkit - Globus GSI Callback Library Documentation Files
globus-gsi-cert-utils-9.16-1.el7.x86_64.rpm Globus Toolkit - Globus GSI Cert Utils Library
globus-gsi-cert-utils-devel-9.16-1.el7.x86_64.rpm Globus Toolkit - Globus GSI Cert Utils Library Development Files
globus-gsi-cert-utils-doc-9.16-1.el7.noarch.rpm Globus Toolkit - Globus GSI Cert Utils Library Documentation Files
globus-gsi-cert-utils-progs-9.16-1.el7.noarch.rpm Globus Toolkit - Globus GSI Cert Utils Library Programs
Provides
Name Value
globus-gridftp-server-control = 6.1-1.el7
globus-gridftp-server-control(x86-64) = 6.1-1.el7
libglobus_gridftp_server_control.so.0()(64bit) -
Required By Download
Type URL
Binary Package globus-gridftp-server-control-6.1-1.el7.x86_64.rpm
Source Package globus-gridftp-server-control-6.1-1.el7.src.rpm
Install Howto
  1. Download the latest epel-release rpm from
    http://dl.fedoraproject.org/pub/epel/7/x86_64/
    
  2. Install epel-release rpm:
    # rpm -Uvh epel-release*rpm
    
  3. Install globus-gridftp-server-control rpm package:
    # yum install globus-gridftp-server-control
    
Files
Path
/usr/lib64/libglobus_gridftp_server_control.so.0
/usr/lib64/libglobus_gridftp_server_control.so.0.6.1
/usr/share/doc/globus-gridftp-server-control-6.1/README
/usr/share/licenses/globus-gridftp-server-control-6.1/GLOBUS_LICENSE
Changelog
2018-04-07 - Mattias Ellert <mattias.ellert@physics.uu.se> - 6.1-1
- GT6 update: Don't error if acquire_cred fails when vhost env is set

[Nov 08, 2018] 9 Aspera Sync Alternatives Top Best Alternatives

Nov 08, 2018 | www.topbestalternatives.com

Aspera Sync is an elite, versatile, multi-directional no concurrent record replication and synchronization. It is intended to conquer the execution and versatility inadequacies of conventional synchronization instruments like Rsync. Aspera Sync can scale up and out for most extreme rate replication and synchronization over WANs. Prominent capacities are The FASP advantage, superior, smart trade for Rsync, underpins complex synchronization arrangements, propelled record taking care of, and so on. Aspera Sync is reason worked by Aspera for elite, versatile, multi-directional offbeat record replication and synchronization. Intended to beat the execution and adaptability deficiencies of conventional synchronization instruments like Rsync, Aspera Sync can scale up and out for greatest pace replication and synchronization over WANs, for now,'s biggest vast information record stores -- from a great many individual documents to the most significant document sizes. Hearty reinforcement and recuperation strategies secure business necessary information and frameworks so undertakings can rapidly recoup necessary documents, structures or a whole site in the occasion if a calamity. Be that as it may, these strategies can be undermined by average exchange speeds amongst essential and reinforcement locales, bringing about fragmented reinforcements and augmented recuperation times. With FASP – controlled transactions, replication fits inside the little operational window so you can meet your recuperation point objective (RPO) and recovery time objective (RTO).

1. Syncthing Syncthing replaces exclusive synchronize and cloud administrations with something open, reliable and decentralized. Your information is your information alone, and you should pick where it is put away if it is imparted to some outsider and how it's transmitted over the Internet. Syncthing is a record sharing application that permits you to share reports between various gadgets in an advantageous way. Its online Graphical User Interface (GUI) makes it conceivable Website Syncthing Alternatives

[Nov 08, 2018] Can rsync resume after being interrupted?

Sep 15, 2012 | unix.stackexchange.com

Tim , Sep 15, 2012 at 23:36

I used rsync to copy a large number of files, but my OS (Ubuntu) restarted unexpectedly.

After reboot, I ran rsync again, but from the output on the terminal, I found that rsync still copied those already copied before. But I heard that rsync is able to find differences between source and destination, and therefore to just copy the differences. So I wonder in my case if rsync can resume what was left last time?

Gilles , Sep 16, 2012 at 1:56

Yes, rsync won't copy again files that it's already copied. There are a few edge cases where its detection can fail. Did it copy all the already-copied files? What options did you use? What were the source and target filesystems? If you run rsync again after it's copied everything, does it copy again? – Gilles Sep 16 '12 at 1:56

Tim , Sep 16, 2012 at 2:30

@Gilles: Thanks! (1) I think I saw rsync copied the same files again from its output on the terminal. (2) Options are same as in my other post, i.e. sudo rsync -azvv /home/path/folder1/ /home/path/folder2 . (3) Source and target are both NTFS, buy source is an external HDD, and target is an internal HDD. (3) It is now running and hasn't finished yet. – Tim Sep 16 '12 at 2:30

jwbensley , Sep 16, 2012 at 16:15

There is also the --partial flag to resume partially transferred files (useful for large files) – jwbensley Sep 16 '12 at 16:15

Tim , Sep 19, 2012 at 5:20

@Gilles: What are some "edge cases where its detection can fail"? – Tim Sep 19 '12 at 5:20

Gilles , Sep 19, 2012 at 9:25

@Tim Off the top of my head, there's at least clock skew, and differences in time resolution (a common issue with FAT filesystems which store times in 2-second increments, the --modify-window option helps with that). – Gilles Sep 19 '12 at 9:25

DanielSmedegaardBuus , Nov 1, 2014 at 12:32

First of all, regarding the "resume" part of your question, --partial just tells the receiving end to keep partially transferred files if the sending end disappears as though they were completely transferred.

While transferring files, they are temporarily saved as hidden files in their target folders (e.g. .TheFileYouAreSending.lRWzDC ), or a specifically chosen folder if you set the --partial-dir switch. When a transfer fails and --partial is not set, this hidden file will remain in the target folder under this cryptic name, but if --partial is set, the file will be renamed to the actual target file name (in this case, TheFileYouAreSending ), even though the file isn't complete. The point is that you can later complete the transfer by running rsync again with either --append or --append-verify .

So, --partial doesn't itself resume a failed or cancelled transfer. To resume it, you'll have to use one of the aforementioned flags on the next run. So, if you need to make sure that the target won't ever contain files that appear to be fine but are actually incomplete, you shouldn't use --partial . Conversely, if you want to make sure you never leave behind stray failed files that are hidden in the target directory, and you know you'll be able to complete the transfer later, --partial is there to help you.

With regards to the --append switch mentioned above, this is the actual "resume" switch, and you can use it whether or not you're also using --partial . Actually, when you're using --append , no temporary files are ever created. Files are written directly to their targets. In this respect, --append gives the same result as --partial on a failed transfer, but without creating those hidden temporary files.

So, to sum up, if you're moving large files and you want the option to resume a cancelled or failed rsync operation from the exact point that rsync stopped, you need to use the --append or --append-verify switch on the next attempt.

As @Alex points out below, since version 3.0.0 rsync now has a new option, --append-verify , which behaves like --append did before that switch existed. You probably always want the behaviour of --append-verify , so check your version with rsync --version . If you're on a Mac and not using rsync from homebrew , you'll (at least up to and including El Capitan) have an older version and need to use --append rather than --append-verify . Why they didn't keep the behaviour on --append and instead named the newcomer --append-no-verify is a bit puzzling. Either way, --append on rsync before version 3 is the same as --append-verify on the newer versions.

--append-verify isn't dangerous: It will always read and compare the data on both ends and not just assume they're equal. It does this using checksums, so it's easy on the network, but it does require reading the shared amount of data on both ends of the wire before it can actually resume the transfer by appending to the target.

Second of all, you said that you "heard that rsync is able to find differences between source and destination, and therefore to just copy the differences."

That's correct, and it's called delta transfer, but it's a different thing. To enable this, you add the -c , or --checksum switch. Once this switch is used, rsync will examine files that exist on both ends of the wire. It does this in chunks, compares the checksums on both ends, and if they differ, it transfers just the differing parts of the file. But, as @Jonathan points out below, the comparison is only done when files are of the same size on both ends -- different sizes will cause rsync to upload the entire file, overwriting the target with the same name.

This requires a bit of computation on both ends initially, but can be extremely efficient at reducing network load if for example you're frequently backing up very large files fixed-size files that often contain minor changes. Examples that come to mind are virtual hard drive image files used in virtual machines or iSCSI targets.

It is notable that if you use --checksum to transfer a batch of files that are completely new to the target system, rsync will still calculate their checksums on the source system before transferring them. Why I do not know :)

So, in short:

If you're often using rsync to just "move stuff from A to B" and want the option to cancel that operation and later resume it, don't use --checksum , but do use --append-verify .

If you're using rsync to back up stuff often, using --append-verify probably won't do much for you, unless you're in the habit of sending large files that continuously grow in size but are rarely modified once written. As a bonus tip, if you're backing up to storage that supports snapshotting such as btrfs or zfs , adding the --inplace switch will help you reduce snapshot sizes since changed files aren't recreated but rather the changed blocks are written directly over the old ones. This switch is also useful if you want to avoid rsync creating copies of files on the target when only minor changes have occurred.

When using --append-verify , rsync will behave just like it always does on all files that are the same size. If they differ in modification or other timestamps, it will overwrite the target with the source without scrutinizing those files further. --checksum will compare the contents (checksums) of every file pair of identical name and size.

UPDATED 2015-09-01 Changed to reflect points made by @Alex (thanks!)

UPDATED 2017-07-14 Changed to reflect points made by @Jonathan (thanks!)

Alex , Aug 28, 2015 at 3:49

According to the documentation --append does not check the data, but --append-verify does. Also, as @gaoithe points out in a comment below, the documentation claims --partial does resume from previous files. – Alex Aug 28 '15 at 3:49

DanielSmedegaardBuus , Sep 1, 2015 at 13:29

Thank you @Alex for the updates. Indeed, since 3.0.0, --append no longer compares the source to the target file before appending. Quite important, really! --partial does not itself resume a failed file transfer, but rather leaves it there for a subsequent --append(-verify) to append to it. My answer was clearly misrepresenting this fact; I'll update it to include these points! Thanks a lot :) – DanielSmedegaardBuus Sep 1 '15 at 13:29

Cees Timmerman , Sep 15, 2015 at 17:21

This says --partial is enough. – Cees Timmerman Sep 15 '15 at 17:21

DanielSmedegaardBuus , May 10, 2016 at 19:31

@CMCDragonkai Actually, check out Alexander's answer below about --partial-dir -- looks like it's the perfect bullet for this. I may have missed something entirely ;) – DanielSmedegaardBuus May 10 '16 at 19:31

Jonathan Y. , Jun 14, 2017 at 5:48

What's your level of confidence in the described behavior of --checksum ? According to the man it has more to do with deciding which files to flag for transfer than with delta-transfer (which, presumably, is rsync 's default behavior). – Jonathan Y. Jun 14 '17 at 5:48

[Nov 08, 2018] How to remove all installed dependent packages while removing a package in centos 7?

Aug 16, 2016 | unix.stackexchange.com

ukll , Aug 16, 2016 at 15:26

I am kinda new to Linux so this may be a dumb question. I searched both in stackoverflow and google but could not find any answer.

I am using CentOS 7. I installed okular, which is a PDF viewer, with the command:

sudo yum install okular

As you can see in the picture below, it installed 37 dependent packages to install okular.

But I wasn't satisfied with the features of the application and I decided to remove it. The problem is that if I remove it with the command:

sudo yum autoremove okular

It only removes four dependent packages.

And if I remove it with the command:

sudo yum remove okular

It removes only one package which is okular.x86_64.

Now, my question is that is there a way to remove all 37 installed packages with a command or do I have to remove all of them one by one?

Thank you in advance.

Jason Powell , Aug 16, 2016 at 17:25

Personally, I don't like yum plugins because they don't work a lot of the time, in my experience.

You can use the yum history command to view your yum history.

[root@testbox ~]# yum history
Loaded plugins: product-id, rhnplugin, search-disabled-repos, subscription-manager, verify, versionlock
ID     | Login user               | Date and time    | Action(s)      | Altered
----------------------------------------------------------------------------------
19 | Jason <jason>  | 2016-06-28 09:16 | Install        |   10

You can find info about the transaction by doing yum history info <transaction id> . So:

yum history info 19 would tell you all the packages that were installed with transaction 19 and the command line that was used to install the packages. If you want to undo transaction 19, you would run yum history undo 19 .

Alternatively, if you just wanted to undo the last transaction you did (you installed a software package and didn't like it), you could just do yum history undo last

Hope this helps!

ukll , Aug 16, 2016 at 18:34

Firstly, thank you for your excellent answer. And secondly, when I did sudo yum history , it showed only actions with id 30 through 49. Is there a way to view all actions history (including with id 1-29)? – ukll Aug 16 '16 at 18:34

Jason Powell , Aug 16, 2016 at 19:00

You're welcome! Yes, there is a way to show all of your history. Just do yum history list all . – Jason Powell Aug 16 '16 at 19:00

,

yum remove package_name will remove only that package and all their dependencies.

yum autoremove will remove the unused dependencies

To remove a package with it's dependencies , you need to install yum plugin called: remove-with-leaves

To install it type:

yum install yum-plugin-remove-with-leaves

To remove package_name type:

yum remove package_name --remove-leaves

[Nov 08, 2018] collectl

Nov 08, 2018 | collectl.sourceforge.net
Collectl Get collectl at SourceForge.net. Fast, secure and Free Open Source software downloads
Latest Version: 4.2.0 June 12, 2017
To use it download the tarball, unpack it and run ./INSTALL
Collectl now supports OpenStack Clouds
Colmux now part of collectl package
Looking for colplot ? It's now here!

Remember, to get lustre support contact Peter Piela to get his custom plugin.

Home | Architecture | Features | Documentation | Releases | FAQ | Support | News | Acknowledgements

There are a number of times in which you find yourself needing performance data. These can include benchmarking, monitoring a system's general heath or trying to determine what your system was doing at some time in the past. Sometimes you just want to know what the system is doing right now. Depending on what you're doing, you often end up using different tools, each designed to for that specific situation.

Unlike most monitoring tools that either focus on a small set of statistics, format their output in only one way, run either interatively or as a daemon but not both, collectl tries to do it all. You can choose to monitor any of a broad set of subsystems which currently include buddyinfo, cpu, disk, inodes, infiniband, lustre, memory, network, nfs, processes, quadrics, slabs, sockets and tcp.

The following is an example taken while writing a large file and running the collectl command with no arguments. By default it shows cpu, network and disk stats in brief format . The key point of this format is all output appears on a single line making it much easier to spot spikes or other anomalies in the output:

collectl

#<--------CPU--------><-----------Disks-----------><-----------Network---------->
#cpu sys inter  ctxsw KBRead  Reads  KBWrit Writes netKBi pkt-in  netKBo pkt-out
  37  37   382    188      0      0   27144    254     45     68       3      21
  25  25   366    180     20      4   31280    296      0      1       0       0
  25  25   368    183      0      0   31720    275      2     20       0       1
In this example, taken while writing to an NFS mounted filesystem, collectl displays interrupts, memory usage and nfs activity with timestamps. Keep in mind that you can mix and match any data and in the case of brief format you simply need to have a window wide enough to accommodate your output.
collectl -sjmf -oT

#         <-------Int--------><-----------Memory-----------><------NFS Totals------>
#Time     Cpu0 Cpu1 Cpu2 Cpu3 Free Buff Cach Inac Slab  Map  Reads Writes Meta Comm
08:36:52  1001   66    0    0   2G 201M 609M 363M 219M 106M      0      0    5    0
08:36:53   999 1657    0    0   2G 201M   1G 918M 252M 106M      0  12622    0    2
08:36:54  1001 7488    0    0   1G 201M   1G   1G 286M 106M      0  20147    0    2
You can also display the same information in verbose format , in which case you get a single line for each type of data at the expense of more screen real estate, as can be seen in this example of network data during NFS writes. Note how you can actually see the network traffic stall while waiting for the server to physically write the data.
collectl -sn --verbose -oT

# NETWORK SUMMARY (/sec)
#          KBIn  PktIn SizeIn  MultI   CmpI  ErrIn  KBOut PktOut  SizeO   CmpO ErrOut
08:46:35   3255  41000     81      0      0      0 112015  78837   1454      0      0
08:46:36      0      9     70      0      0      0     29     25   1174      0      0
08:46:37      0      2     70      0      0      0      0      2    134      0      0
In this last example we see what detail format looks like where we see multiple lines of output for a partitular type of data, which in this case is interrupts. We've also elected to show the time in msecs as well.
collectl -sJ -oTm

#              Int    Cpu0   Cpu1   Cpu2   Cpu3   Type            Device(s)
08:52:32.002   225       0      4      0      0   IO-APIC-level   ioc0
08:52:32.002   000    1000      0      0      0   IO-APIC-edge    timer
08:52:32.002   014       0      0     18      0   IO-APIC-edge    ide0
08:52:32.002   090       0      0      0  15461   IO-APIC-level   eth1
Collectl output can also be saved in a rolling set of logs for later playback or displayed interactively in a variety of formats. If all that isn't enough there are plugins that allow you to report data in alternate formats or even send them over a socket to remote tools such as ganglia or graphite. You can even create files in space-separated format for plotting with external packages like gnuplot. The one below was created with colplot, part of the collectl utilities project, which provides a web-based interface to gnuplot.

Are you a big user of the top command? Have you ever wanted to look across a cluster to see what the top processes are? Better yet, how about using iostat across a cluster? Or maybe vmstat or even looking at top network interfaces across a cluster? Look no more because if collectl reports it for one node, colmux can do it across a cluster AND you can sort by any column of your choice by simply using the right/left arrow keys.

Collectl and Colmux run on all linux distros and are available in redhat and debian respositories and so getting it may be as simple as running yum or apt-get. Note that since colmux has just been merged into the collectl V4.0.0 package it may not yet be available in the repository of your choice and you should install collectl-utils V4.8.2 or earlier to get it for the time being.

Collectl requires perl which is usually installed by default on all major Linux distros and optionally uses Time::Hires which is also usually installed and allows collectl to use fractional intervals and display timestamps in msec. The Compress::Zlib module is usually installed as well and if present the recorded data will be compressed and therefore use on average 90% less storage when recording to a file.

If you're still not sure if collectl is right for you, take a couple of minutes to look at the Collectl Tutorial to get a better feel for what collectl can do. Also be sure to check back and see what's new on the website, sign up for a Mailing List or watch the Forums .

"I absolutely love it and have been using it extensively for months."
Kevin Closson: Performance Architect, EMC
"Collectl is indispensable to any system admin."
Matt Heaton: President, Bluehost.com

[Nov 08, 2018] How to find which process is regularly writing to disk?

Notable quotes:
"... tick...tick...tick...trrrrrr ..."
"... /var/log/syslog ..."
Nov 08, 2018 | unix.stackexchange.com

Cedric Martin , Jul 27, 2012 at 4:31

How can I find which process is constantly writing to disk?

I like my workstation to be close to silent and I just build a new system (P8B75-M + Core i5 3450s -- the 's' because it has a lower max TDP) with quiet fans etc. and installed Debian Wheezy 64-bit on it.

And something is getting on my nerve: I can hear some kind of pattern like if the hard disk was writing or seeking someting ( tick...tick...tick...trrrrrr rinse and repeat every second or so).

In the past I had a similar issue in the past (many, many years ago) and it turned out it was some CUPS log or something and I simply redirected that one (not important) logging to a (real) RAM disk.

But here I'm not sure.

I tried the following:

ls -lR /var/log > /tmp/a.tmp && sleep 5 && ls -lR /var/log > /tmp/b.tmp && diff /tmp/?.tmp

but nothing is changing there.

Now the strange thing is that I also hear the pattern when the prompt asking me to enter my LVM decryption passphrase is showing.

Could it be something in the kernel/system I just installed or do I have a faulty harddisk?

hdparm -tT /dev/sda report a correct HD speed (130 GB/s non-cached, sata 6GB) and I've already installed and compiled from big sources (Emacs) without issue so I don't think the system is bad.

(HD is a Seagate Barracude 500GB)

Mat , Jul 27, 2012 at 6:03

Are you sure it's a hard drive making that noise, and not something else? (Check the fans, including PSU fan. Had very strange clicking noises once when a very thin cable was too close to a fan and would sometimes very slightly touch the blades and bounce for a few "clicks"...) – Mat Jul 27 '12 at 6:03

Cedric Martin , Jul 27, 2012 at 7:02

@Mat: I'll take the hard drive outside of the case (the connectors should be long enough) to be sure and I'll report back ; ) – Cedric Martin Jul 27 '12 at 7:02

camh , Jul 27, 2012 at 9:48

Make sure your disk filesystems are mounted relatime or noatime. File reads can be causing writes to inodes to record the access time. – camh Jul 27 '12 at 9:48

mnmnc , Jul 27, 2012 at 8:27

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]

Cedric Martin , Aug 2, 2012 at 15:56

thanks for that tip. I didn't know about iotop . On Debian I did an apt-cache search iotop to find out that I had to apt-get iotop . Very cool command! – Cedric Martin Aug 2 '12 at 15:56

ndemou , Jun 20, 2016 at 15:32

I use iotop -o -b -d 10 which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used. – ndemou Jun 20 '16 at 15:32

scai , Jul 27, 2012 at 10:48

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog . This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

dan3 , Jul 15, 2013 at 8:32

It is absolutely crazy to leave sysloging enabled when block_dump is active. Logging causes disk activity, which causes logging, which causes disk activity etc. Better stop syslog before enabling this (and use dmesg to read the messages) – dan3 Jul 15 '13 at 8:32

scai , Jul 16, 2013 at 6:32

You are absolutely right, although the effect isn't as dramatic as you describe it. If you just want to have a short peek at the disk activity there is no need to stop the syslog daemon. – scai Jul 16 '13 at 6:32

dan3 , Jul 16, 2013 at 7:22

I've tried it about 2 years ago and it brought my machine to a halt. One of these days when I have nothing important running I'll try it again :) – dan3 Jul 16 '13 at 7:22

scai , Jul 16, 2013 at 10:50

I tried it, nothing really happened. Especially because of file system buffering. A write to syslog doesn't immediately trigger a write to disk. – scai Jul 16 '13 at 10:50

Volker Siegel , Apr 16, 2014 at 22:57

I would assume there is rate general rate limiting in place for the log messages, which handles this case too(?) – Volker Siegel Apr 16 '14 at 22:57

Gilles , Jul 28, 2012 at 1:34

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem , you can use the audit subsystem (install the auditd package ). Put a watch on the sync calls and its friends:
auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log . Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none .

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf : if a log file name is not be preceded by - , then that log is flushed to disk after each write.

Gilles , Mar 23 at 18:24

@StephenKitt Huh. No. The asker mentioned Debian so I've changed it to a link to the Debian package. – Gilles Mar 23 at 18:24

cas , Jul 27, 2012 at 9:40

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do
  /sbin/hdparm -q -S 0 "/dev/$disk"
done

Cedric Martin , Aug 2, 2012 at 16:03

that you can't query SMART readings without spinning up the drive leaves me speechless :-/ Now obviously the "spinning down" issue can become quite complicated. Regarding disabling the spinning down: wouldn't that in itself cause the HD to wear out faster? I mean: it's never ever "resting" as long as the system is on then? – Cedric Martin Aug 2 '12 at 16:03

cas , Aug 2, 2012 at 21:42

IIRC you can query some SMART values without causing the drive to spin up, but temperature isn't one of them on any of the drives i've tested (incl models from WD, Seagate, Samsung, Hitachi). Which is, of course, crazy because concern over temperature is one of the reasons for idling a drive. re: wear: AIUI 1. constant velocity is less wearing than changing speed. 2. the drives have to park the heads in a safe area and a drive is only rated to do that so many times (IIRC up to a few hundred thousand - easily exceeded if the drive is idling and spinning up every few seconds) – cas Aug 2 '12 at 21:42

Micheal Johnson , Mar 12, 2016 at 20:48

It's a long debate regarding whether it's better to leave drives running or to spin them down. Personally I believe it's best to leave them running - I turn my computer off at night and when I go out but other than that I never spin my drives down. Some people prefer to spin them down, say, at night if they're leaving the computer on or if the computer's idle for a long time, and in such cases the advantage of spinning them down for a few hours versus leaving them running is debatable. What's never good though is when the hard drive repeatedly spins down and up again in a short period of time. – Micheal Johnson Mar 12 '16 at 20:48

Micheal Johnson , Mar 12, 2016 at 20:51

Note also that spinning the drive down after it's been idle for a few hours is a bit silly, because if it's been idle for a few hours then it's likely to be used again within an hour. In that case, it would seem better to spin the drive down promptly if it's idle (like, within 10 minutes), but it's also possible for the drive to be idle for a few minutes when someone is using the computer and is likely to need the drive again soon. – Micheal Johnson Mar 12 '16 at 20:51

,

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

[Nov 08, 2018] Determining what process is bound to a port

Mar 14, 2011 | unix.stackexchange.com
I know that using the command:
lsof -i TCP

(or some variant of parameters with lsof) I can determine which process is bound to a particular port. This is useful say if I'm trying to start something that wants to bind to 8080 and some else is already using that port, but I don't know what.

Is there an easy way to do this without using lsof? I spend time working on many systems and lsof is often not installed.

Cakemox , Mar 14, 2011 at 20:48

netstat -lnp will list the pid and process name next to each listening port. This will work under Linux, but not all others (like AIX.) Add -t if you want TCP only.
# netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:24800           0.0.0.0:*               LISTEN      27899/synergys
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      3361/python
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      2264/mysqld
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      22964/apache2
tcp        0      0 192.168.99.1:53         0.0.0.0:*               LISTEN      3389/named
tcp        0      0 192.168.88.1:53         0.0.0.0:*               LISTEN      3389/named

etc.

xxx , Mar 14, 2011 at 21:01

Cool, thanks. Looks like that that works under RHEL, but not under Solaris (as you indicated). Anybody know if there's something similar for Solaris? – user5721 Mar 14 '11 at 21:01

Rich Homolka , Mar 15, 2011 at 19:56

netstat -p above is my vote. also look at lsof . – Rich Homolka Mar 15 '11 at 19:56

Jonathan , Aug 26, 2014 at 18:50

As an aside, for windows it's similar: netstat -aon | more – Jonathan Aug 26 '14 at 18:50

sudo , May 25, 2017 at 2:24

What about for SCTP? – sudo May 25 '17 at 2:24

frielp , Mar 15, 2011 at 13:33

On AIX, netstat & rmsock can be used to determine process binding:
[root@aix] netstat -Ana|grep LISTEN|grep 80
f100070000280bb0 tcp4       0      0  *.37               *.*        LISTEN
f1000700025de3b0 tcp        0      0  *.80               *.*        LISTEN
f1000700002803b0 tcp4       0      0  *.111              *.*        LISTEN
f1000700021b33b0 tcp4       0      0  127.0.0.1.32780    *.*        LISTEN

# Port 80 maps to f1000700025de3b0 above, so we type:
[root@aix] rmsock f1000700025de3b0 tcpcb
The socket 0x25de008 is being held by process 499790 (java).

Olivier Dulac , Sep 18, 2013 at 4:05

Thanks for this! Is there a way, however, to just display what process listen on the socket (instead of using rmsock which attempt to remove it) ? – Olivier Dulac Sep 18 '13 at 4:05

Vitor Py , Sep 26, 2013 at 14:18

@OlivierDulac: "Unlike what its name implies, rmsock does not remove the socket, if it is being used by a process. It just reports the process holding the socket." ( ibm.com/developerworks/community/blogs/cgaix/entry/ ) – Vitor Py Sep 26 '13 at 14:18

Olivier Dulac , Sep 26, 2013 at 16:00

@vitor-braga: Ah thx! I thought it was trying but just said which process holds in when it couldn't remove it. Apparently it doesn't even try to remove it when a process holds it. That's cool! Thx! – Olivier Dulac Sep 26 '13 at 16:00

frielp , Mar 15, 2011 at 13:27

Another tool available on Linux is ss . From the ss man page on Fedora:
NAME
       ss - another utility to investigate sockets
SYNOPSIS
       ss [options] [ FILTER ]
DESCRIPTION
       ss is used to dump socket statistics. It allows showing information 
       similar to netstat. It can display more TCP and state informations  
       than other tools.

Example output below - the final column shows the process binding:

[root@box] ss -ap
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
LISTEN     0      128                    :::http                    :::*        users:(("httpd",20891,4),("httpd",20894,4),("httpd",20895,4),("httpd",20896,4)
LISTEN     0      128             127.0.0.1:munin                    *:*        users:(("munin-node",1278,5))
LISTEN     0      128                    :::ssh                     :::*        users:(("sshd",1175,4))
LISTEN     0      128                     *:ssh                      *:*        users:(("sshd",1175,3))
LISTEN     0      10              127.0.0.1:smtp                     *:*        users:(("sendmail",1199,4))
LISTEN     0      128             127.0.0.1:x11-ssh-offset                  *:*        users:(("sshd",25734,8))
LISTEN     0      128                   ::1:x11-ssh-offset                 :::*        users:(("sshd",25734,7))

Eugen Constantin Dinca , Mar 14, 2011 at 23:47

For Solaris you can use pfiles and then grep by sockname: or port: .

A sample (from here ):

pfiles `ptree | awk '{print $1}'` | egrep '^[0-9]|port:'

rickumali , May 8, 2011 at 14:40

I was once faced with trying to determine what process was behind a particular port (this time it was 8000). I tried a variety of lsof and netstat, but then took a chance and tried hitting the port via a browser (i.e. http://hostname:8000/ ). Lo and behold, a splash screen greeted me, and it became obvious what the process was (for the record, it was Splunk ).

One more thought: "ps -e -o pid,args" (YMMV) may sometimes show the port number in the arguments list. Grep is your friend!

Gilles , Oct 8, 2015 at 21:04

In the same vein, you could telnet hostname 8000 and see if the server prints a banner. However, that's mostly useful when the server is running on a machine where you don't have shell access, and then finding the process ID isn't relevant. – Gilles May 8 '11 at 14:45

[Nov 08, 2018] How to find which process is regularly writing to disk?

Notable quotes:
"... tick...tick...tick...trrrrrr ..."
"... /var/log/syslog ..."
Jul 27, 2012 | unix.stackexchange.com

Cedric Martin , Jul 27, 2012 at 4:31

How can I find which process is constantly writing to disk?

I like my workstation to be close to silent and I just build a new system (P8B75-M + Core i5 3450s -- the 's' because it has a lower max TDP) with quiet fans etc. and installed Debian Wheezy 64-bit on it.

And something is getting on my nerve: I can hear some kind of pattern like if the hard disk was writing or seeking someting ( tick...tick...tick...trrrrrr rinse and repeat every second or so).

In the past I had a similar issue in the past (many, many years ago) and it turned out it was some CUPS log or something and I simply redirected that one (not important) logging to a (real) RAM disk.

But here I'm not sure.

I tried the following:

ls -lR /var/log > /tmp/a.tmp && sleep 5 && ls -lR /var/log > /tmp/b.tmp && diff /tmp/?.tmp

but nothing is changing there.

Now the strange thing is that I also hear the pattern when the prompt asking me to enter my LVM decryption passphrase is showing.

Could it be something in the kernel/system I just installed or do I have a faulty harddisk?

hdparm -tT /dev/sda report a correct HD speed (130 GB/s non-cached, sata 6GB) and I've already installed and compiled from big sources (Emacs) without issue so I don't think the system is bad.

(HD is a Seagate Barracude 500GB)

Mat , Jul 27, 2012 at 6:03

Are you sure it's a hard drive making that noise, and not something else? (Check the fans, including PSU fan. Had very strange clicking noises once when a very thin cable was too close to a fan and would sometimes very slightly touch the blades and bounce for a few "clicks"...) – Mat Jul 27 '12 at 6:03

Cedric Martin , Jul 27, 2012 at 7:02

@Mat: I'll take the hard drive outside of the case (the connectors should be long enough) to be sure and I'll report back ; ) – Cedric Martin Jul 27 '12 at 7:02

camh , Jul 27, 2012 at 9:48

Make sure your disk filesystems are mounted relatime or noatime. File reads can be causing writes to inodes to record the access time. – camh Jul 27 '12 at 9:48

mnmnc , Jul 27, 2012 at 8:27

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]
    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]
    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]
 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]
   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]

Cedric Martin , Aug 2, 2012 at 15:56

thanks for that tip. I didn't know about iotop . On Debian I did an apt-cache search iotop to find out that I had to apt-get iotop . Very cool command! – Cedric Martin Aug 2 '12 at 15:56

ndemou , Jun 20, 2016 at 15:32

I use iotop -o -b -d 10 which every 10secs prints a list of processes that read/wrote to disk and the amount of IO bandwidth used. – ndemou Jun 20 '16 at 15:32

scai , Jul 27, 2012 at 10:48

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog . This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

dan3 , Jul 15, 2013 at 8:32

It is absolutely crazy to leave sysloging enabled when block_dump is active. Logging causes disk activity, which causes logging, which causes disk activity etc. Better stop syslog before enabling this (and use dmesg to read the messages) – dan3 Jul 15 '13 at 8:32

scai , Jul 16, 2013 at 6:32

You are absolutely right, although the effect isn't as dramatic as you describe it. If you just want to have a short peek at the disk activity there is no need to stop the syslog daemon. – scai Jul 16 '13 at 6:32

dan3 , Jul 16, 2013 at 7:22

I've tried it about 2 years ago and it brought my machine to a halt. One of these days when I have nothing important running I'll try it again :) – dan3 Jul 16 '13 at 7:22

scai , Jul 16, 2013 at 10:50

I tried it, nothing really happened. Especially because of file system buffering. A write to syslog doesn't immediately trigger a write to disk. – scai Jul 16 '13 at 10:50

Volker Siegel , Apr 16, 2014 at 22:57

I would assume there is rate general rate limiting in place for the log messages, which handles this case too(?) – Volker Siegel Apr 16 '14 at 22:57

Gilles , Jul 28, 2012 at 1:34

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem , you can use the audit subsystem (install the auditd package ). Put a watch on the sync calls and its friends:
auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log . Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none .

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf : if a log file name is not be preceded by - , then that log is flushed to disk after each write.

Gilles , Mar 23 at 18:24

@StephenKitt Huh. No. The asker mentioned Debian so I've changed it to a link to the Debian package. – Gilles Mar 23 at 18:24

cas , Jul 27, 2012 at 9:40

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do
  /sbin/hdparm -q -S 0 "/dev/$disk"
done

Cedric Martin , Aug 2, 2012 at 16:03

that you can't query SMART readings without spinning up the drive leaves me speechless :-/ Now obviously the "spinning down" issue can become quite complicated. Regarding disabling the spinning down: wouldn't that in itself cause the HD to wear out faster? I mean: it's never ever "resting" as long as the system is on then? – Cedric Martin Aug 2 '12 at 16:03

cas , Aug 2, 2012 at 21:42

IIRC you can query some SMART values without causing the drive to spin up, but temperature isn't one of them on any of the drives i've tested (incl models from WD, Seagate, Samsung, Hitachi). Which is, of course, crazy because concern over temperature is one of the reasons for idling a drive. re: wear: AIUI 1. constant velocity is less wearing than changing speed. 2. the drives have to park the heads in a safe area and a drive is only rated to do that so many times (IIRC up to a few hundred thousand - easily exceeded if the drive is idling and spinning up every few seconds) – cas Aug 2 '12 at 21:42

Micheal Johnson , Mar 12, 2016 at 20:48

It's a long debate regarding whether it's better to leave drives running or to spin them down. Personally I believe it's best to leave them running - I turn my computer off at night and when I go out but other than that I never spin my drives down. Some people prefer to spin them down, say, at night if they're leaving the computer on or if the computer's idle for a long time, and in such cases the advantage of spinning them down for a few hours versus leaving them running is debatable. What's never good though is when the hard drive repeatedly spins down and up again in a short period of time. – Micheal Johnson Mar 12 '16 at 20:48

Micheal Johnson , Mar 12, 2016 at 20:51

Note also that spinning the drive down after it's been idle for a few hours is a bit silly, because if it's been idle for a few hours then it's likely to be used again within an hour. In that case, it would seem better to spin the drive down promptly if it's idle (like, within 10 minutes), but it's also possible for the drive to be idle for a few minutes when someone is using the computer and is likely to need the drive again soon. – Micheal Johnson Mar 12 '16 at 20:51

,

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

[Nov 08, 2018] How to use parallel ssh (PSSH) for executing ssh in parallel on a number of Linux-Unix-BSD servers

Looks like -h option is slightly more convenient then -w option.
Notable quotes:
"... Each line in the host file are of the form [user@]host[:port] and can include blank lines and comments lines beginning with "#". ..."
Nov 08, 2018 | www.cyberciti.biz

First you need to create a text file called hosts file from which pssh read hosts names. The syntax is pretty simple.

Each line in the host file are of the form [user@]host[:port] and can include blank lines and comments lines beginning with "#".

Here is my sample file named ~/.pssh_hosts_files:
$ cat ~/.pssh_hosts_files
vivek@dellm6700
root@192.168.2.30
root@192.168.2.45
root@192.168.2.46

Run the date command all hosts:
$ pssh -i -h ~/.pssh_hosts_files date
Sample outputs:

[1] 18:10:10 [SUCCESS] root@192.168.2.46
Sun Feb 26 18:10:10 IST 2017
[2] 18:10:10 [SUCCESS] vivek@dellm6700
Sun Feb 26 18:10:10 IST 2017
[3] 18:10:10 [SUCCESS] root@192.168.2.45
Sun Feb 26 18:10:10 IST 2017
[4] 18:10:10 [SUCCESS] root@192.168.2.30
Sun Feb 26 18:10:10 IST 2017

Run the uptime command on each host:
$ pssh -i -h ~/.pssh_hosts_files uptime
Sample outputs:

[1] 18:11:15 [SUCCESS] root@192.168.2.45
 18:11:15 up  2:29,  0 users,  load average: 0.00, 0.00, 0.00
[2] 18:11:15 [SUCCESS] vivek@dellm6700
 18:11:15 up 19:06,  0 users,  load average: 0.13, 0.25, 0.27
[3] 18:11:15 [SUCCESS] root@192.168.2.46
 18:11:15 up  1:55,  0 users,  load average: 0.00, 0.00, 0.00
[4] 18:11:15 [SUCCESS] root@192.168.2.30
 6:11PM  up 1 day, 21:38, 0 users, load averages: 0.12, 0.14, 0.09

You can now automate common sysadmin tasks such as patching all servers:
$ pssh -h ~/.pssh_hosts_files -- sudo yum -y update
OR
$ pssh -h ~/.pssh_hosts_files -- sudo apt-get -y update
$ pssh -h ~/.pssh_hosts_files -- sudo apt-get -y upgrade

How do I use pssh to copy file to all servers?

The syntax is:
pscp -h ~/.pssh_hosts_files src dest
To copy $HOME/demo.txt to /tmp/ on all servers, enter:
$ pscp -h ~/.pssh_hosts_files $HOME/demo.txt /tmp/
Sample outputs:

[1] 18:17:35 [SUCCESS] vivek@dellm6700
[2] 18:17:35 [SUCCESS] root@192.168.2.45
[3] 18:17:35 [SUCCESS] root@192.168.2.46
[4] 18:17:35 [SUCCESS] root@192.168.2.30

Or use the prsync command for efficient copying of files:
$ prsync -h ~/.pssh_hosts_files /etc/passwd /tmp/
$ prsync -h ~/.pssh_hosts_files *.html /var/www/html/

How do I kill processes in parallel on a number of hosts?

Use the pnuke command for killing processes in parallel on a number of hosts. The syntax is:
$ pnuke -h .pssh_hosts_files process_name
### kill nginx and firefox on hosts:
$ pnuke -h ~/.pssh_hosts_files firefox
$ pnuke -h ~/.pssh_hosts_files nginx

See pssh/pscp command man pages for more information.

[Nov 08, 2018] Parallel command execution with PDSH

Notable quotes:
"... (did I mention that Rittman Mead laptops are Macs, so I can do all of this straight from my work machine... :-) ) ..."
"... open an editor and paste the following lines into it and save the file as /foo/bar ..."
Nov 08, 2018 | www.rittmanmead.com

In this series of blog posts I'm taking a look at a few very useful tools that can make your life as the sysadmin of a cluster of Linux machines easier. This may be a Hadoop cluster, or just a plain simple set of 'normal' machines on which you want to run the same commands and monitoring.

Previously we looked at using SSH keys for intra-machine authorisation , which is a pre-requisite what we'll look at here -- executing the same command across multiple machines using PDSH. In the next post of the series we'll see how we can monitor OS metrics across a cluster with colmux.

PDSH is a very smart little tool that enables you to issue the same command on multiple hosts at once, and see the output. You need to have set up ssh key authentication from the client to host on all of them, so if you followed the steps in the first section of this article you'll be good to go.

The syntax for using it is nice and simple:

For example run against a small cluster of four machines that I have:

robin@RNMMBP $ pdsh -w root@rnmcluster02-node0[1-4] date

rnmcluster02-node01: Fri Nov 28 17:26:17 GMT 2014  
rnmcluster02-node02: Fri Nov 28 17:26:18 GMT 2014  
rnmcluster02-node03: Fri Nov 28 17:26:18 GMT 2014  
rnmcluster02-node04: Fri Nov 28 17:26:18 GMT 2014

PDSH can be installed on the Mac under Homebrew (did I mention that Rittman Mead laptops are Macs, so I can do all of this straight from my work machine... :-) )

brew install pdsh

And if you want to run it on Linux from the EPEL yum repository (RHEL-compatible, but packages for other distros are available):

yum install pdsh

You can run it from a cluster node, or from your client machine (assuming your client machine is mac/linux).

Example - install and start collectl on all nodes

I started looking into pdsh when it came to setting up a cluster of machines from scratch. One of the must-have tools I like to have on any machine that I work with is the excellent collectl . This is an OS resource monitoring tool that I initially learnt of through Kevin Closson and Greg Rahn , and provides the kind of information you'd get from top etc – and then some! It can run interactively, log to disk, run as a service – and it also happens to integrate very nicely with graphite , making it a no-brainer choice for any server.

So, instead of logging into each box individually I could instead run this:

pdsh -w root@rnmcluster02-node0[1-4] yum install -y collectl  
pdsh -w root@rnmcluster02-node0[1-4] service collectl start  
pdsh -w root@rnmcluster02-node0[1-4] chkconfig collectl on

Yes, I know there are tools out there like puppet and chef that are designed for doing this kind of templated build of multiple servers, but the point I want to illustrate here is that pdsh enables you to do ad-hoc changes to a set of servers at once. Sure, once I have my cluster built and want to create an image/template for future builds, then it would be daft if I were building the whole lot through pdsh-distributed yum commands.

Example - setting up the date/timezone/NTPD

Often the accuracy of the clock on each server in a cluster is crucial, and we can easily do this with pdsh:

Install packages

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] yum install -y ntp ntpdate

Set the timezone:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime

Force a time refresh:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] ntpdate pool.ntp.org  
rnmcluster02-node03: 30 Nov 20:46:22 ntpdate[27610]: step time server 176.58.109.199 offset -2.928585 sec  
rnmcluster02-node02: 30 Nov 20:46:22 ntpdate[28527]: step time server 176.58.109.199 offset -2.946021 sec  
rnmcluster02-node04: 30 Nov 20:46:22 ntpdate[27615]: step time server 129.250.35.250 offset -2.915713 sec  
rnmcluster02-node01: 30 Nov 20:46:25 ntpdate[29316]: 178.79.160.57 rate limit response from server.  
rnmcluster02-node01: 30 Nov 20:46:22 ntpdate[29316]: step time server 176.58.109.199 offset -2.925016 sec

Set NTPD to start automatically at boot:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] chkconfig ntpd on

Start NTPD:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] service ntpd start
Example - using a HEREDOC (here-document) and sending quotation marks in a command with PDSH

Here documents (heredocs) are a nice way to embed multi-line content in a single command, enabling the scripting of a file creation rather than the clumsy instruction to " open an editor and paste the following lines into it and save the file as /foo/bar ".

Fortunately heredocs work just fine with pdsh, so long as you remember to enclose the whole command in quotation marks. And speaking of which, if you need to include quotation marks in your actual command, you need to escape them with a backslash. Here's an example of both, setting up the configuration file for my ever-favourite gnu screen on all the nodes of the cluster:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] "cat > ~/.screenrc <<EOF  
hardstatus alwayslastline \"%{= RY}%H %{kG}%{G} Screen(s): %{c}%w %=%{kG}%c  %D, %M %d %Y  LD:%l\"  
startup_message off  
msgwait 1  
defscrollback 100000  
nethack on  
EOF  
"

Now when I login to each individual node and run screen, I get a nice toolbar at the bottom:

Combining commands

To combine commands together that you send to each host you can use the standard bash operator semicolon ;

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] "date;sleep 5;date"  
rnmcluster02-node01: Sun Nov 30 20:57:06 GMT 2014  
rnmcluster02-node03: Sun Nov 30 20:57:06 GMT 2014  
rnmcluster02-node04: Sun Nov 30 20:57:06 GMT 2014  
rnmcluster02-node02: Sun Nov 30 20:57:06 GMT 2014  
rnmcluster02-node01: Sun Nov 30 20:57:11 GMT 2014  
rnmcluster02-node03: Sun Nov 30 20:57:11 GMT 2014  
rnmcluster02-node04: Sun Nov 30 20:57:11 GMT 2014  
rnmcluster02-node02: Sun Nov 30 20:57:11 GMT 2014

Note the use of the quotation marks to enclose the entire command string. Without them the bash interpretor will take the ; as the delineator of the local commands, and try to run the subsequent commands locally:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node0[1-4] date;sleep 5;date  
rnmcluster02-node03: Sun Nov 30 20:57:53 GMT 2014  
rnmcluster02-node04: Sun Nov 30 20:57:53 GMT 2014  
rnmcluster02-node02: Sun Nov 30 20:57:53 GMT 2014  
rnmcluster02-node01: Sun Nov 30 20:57:53 GMT 2014  
Sun 30 Nov 2014 20:58:00 GMT

You can also use && and || to run subsequent commands conditionally if the previous one succeeds or fails respectively:

robin@RNMMBP $ pdsh -w root@rnmcluster02-node[01-4] "chkconfig collectl on && service collectl start"

rnmcluster02-node03: Starting collectl: [  OK  ]  
rnmcluster02-node02: Starting collectl: [  OK  ]  
rnmcluster02-node04: Starting collectl: [  OK  ]  
rnmcluster02-node01: Starting collectl: [  OK  ]
Piping and file redirects

Similar to combining commands above, you can pipe the output of commands, and you need to use quotation marks to enclose the whole command string.

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] "chkconfig|grep collectl"  
rnmcluster02-node03: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node01: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node04: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node02: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off

However, you can pipe the output from pdsh to a local process if you want:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] chkconfig|grep collectl  
rnmcluster02-node02: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node04: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node03: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off  
rnmcluster02-node01: collectl           0:off   1:off   2:on    3:on    4:on    5:on    6:off

The difference is that you'll be shifting the whole of the pipe across the network in order to process it locally, so if you're just grepping etc this doesn't make any sense. For use of utilities held locally and not on the remote server though, this might make sense.

File redirects work the same way – within quotation marks and the redirect will be to a file on the remote server, outside of them it'll be local:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] "chkconfig>/tmp/pdsh.out"  
robin@RNMMBP ~ $ ls -l /tmp/pdsh.out  
ls: /tmp/pdsh.out: No such file or directory

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] chkconfig>/tmp/pdsh.out  
robin@RNMMBP ~ $ ls -l /tmp/pdsh.out  
-rw-r--r--  1 robin  wheel  7608 30 Nov 19:23 /tmp/pdsh.out
Cancelling PDSH operations

As you can see from above, the precise syntax of pdsh calls can be hugely important. If you run a command and it appears 'stuck', or if you have that heartstopping realisation that the shutdown -h now you meant to run locally you ran across the cluster, you can press Ctrl-C once to see the status of your commands:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] sleep 30  
^Cpdsh@RNMMBP: interrupt (one more within 1 sec to abort)
pdsh@RNMMBP:  (^Z within 1 sec to cancel pending threads)  
pdsh@RNMMBP: rnmcluster02-node01: command in progress  
pdsh@RNMMBP: rnmcluster02-node02: command in progress  
pdsh@RNMMBP: rnmcluster02-node03: command in progress  
pdsh@RNMMBP: rnmcluster02-node04: command in progress

and press it twice (or within a second of the first) to cancel:

robin@RNMMBP ~ $ pdsh -w root@rnmcluster02-node[01-4] sleep 30  
^Cpdsh@RNMMBP: interrupt (one more within 1 sec to abort)
pdsh@RNMMBP:  (^Z within 1 sec to cancel pending threads)  
pdsh@RNMMBP: rnmcluster02-node01: command in progress  
pdsh@RNMMBP: rnmcluster02-node02: command in progress  
pdsh@RNMMBP: rnmcluster02-node03: command in progress  
pdsh@RNMMBP: rnmcluster02-node04: command in progress  
^Csending SIGTERM to ssh rnmcluster02-node01
sending signal 15 to rnmcluster02-node01 [ssh] pid 26534  
sending SIGTERM to ssh rnmcluster02-node02  
sending signal 15 to rnmcluster02-node02 [ssh] pid 26535  
sending SIGTERM to ssh rnmcluster02-node03  
sending signal 15 to rnmcluster02-node03 [ssh] pid 26533  
sending SIGTERM to ssh rnmcluster02-node04  
sending signal 15 to rnmcluster02-node04 [ssh] pid 26532  
pdsh@RNMMBP: interrupt, aborting.

If you've got threads yet to run on the remote hosts, but want to keep running whatever has already started, you can use Ctrl-C, Ctrl-Z:

robin@RNMMBP ~ $ pdsh -f 2 -w root@rnmcluster02-node[01-4] "sleep 5;date"  
^Cpdsh@RNMMBP: interrupt (one more within 1 sec to abort)
pdsh@RNMMBP:  (^Z within 1 sec to cancel pending threads)  
pdsh@RNMMBP: rnmcluster02-node01: command in progress  
pdsh@RNMMBP: rnmcluster02-node02: command in progress  
^Zpdsh@RNMMBP: Canceled 2 pending threads.
rnmcluster02-node01: Mon Dec  1 21:46:35 GMT 2014  
rnmcluster02-node02: Mon Dec  1 21:46:35 GMT 2014

NB the above example illustrates the use of the -f argument to limit how many threads are run against remote hosts at once. We can see the command is left running on the first two nodes and returns the date, whilst the Ctrl-C - Ctrl-Z stops it from being executed on the remaining nodes.

PDSH_SSH_ARGS_APPEND

By default, when you ssh to new host for the first time you'll be prompted to validate the remote host's SSH key fingerprint.

The authenticity of host 'rnmcluster02-node02 (172.28.128.9)' can't be established.  
RSA key fingerprint is 00:c0:75:a8:bc:30:cb:8e:b3:8e:e4:29:42:6a:27:1c.  
Are you sure you want to continue connecting (yes/no)?

This is one of those prompts that the majority of us just hit enter at and ignore; if that includes you then you will want to make sure that your PDSH call doesn't fall in a heap because you're connecting to a bunch of new servers all at once. PDSH is not an interactive tool, so if it requires input from the hosts it's connecting to it'll just fail. To avoid this SSH prompt, you can set up the environment variable PDSH SSH ARGS_APPEND as follows:

export PDSH_SSH_ARGS_APPEND="-q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"

The -q makes failures less verbose, and the -o passes in a couple of options, StrictHostKeyChecking to disable the above check, and UserKnownHostsFile to stop SSH keeping a list of host IP/hostnames and corresponding SSH fingerprints (by pointing it at /dev/null ). You'll want this if you're working with VMs that are sharing a pool of IPs and get re-used, otherwise you get this scary failure:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!  
Someone could be eavesdropping on you right now (man-in-the-middle attack)!  
It is also possible that a host key has just been changed.  
The fingerprint for the RSA key sent by the remote host is  
00:c0:75:a8:bc:30:cb:8e:b3:8e:e4:29:42:6a:27:1c.  
Please contact your system administrator.

For both of these above options, make sure you're aware of the security implications that you're opening yourself up to. For a sandbox environment I just ignore them; for anything where security is of importance make sure you are aware of quite which server you are connecting to by SSH, and protecting yourself from MitM attacks.

PDSH Reference

You can find out more about PDSH at https://code.google.com/p/pdsh/wiki/UsingPDSH

Summary

When working with multiple Linux machines I would first and foremost make sure SSH keys are set up in order to ease management through password-less logins.

After SSH keys, I would recommend pdsh for parallel execution of the same SSH command across the cluster. It's a big time saver particularly when initially setting up the cluster given the installation and configuration changes that are inevitably needed.

In the next article of this series we'll see how the tool colmux is a powerful way to monitor OS metrics across a cluster.

So now your turn – what particular tools or tips do you have for working with a cluster of Linux machines? Leave your answers in the comments below, or tweet them to me at @rmoff .

[Nov 08, 2018] pexec utility is similar to parallel

Nov 08, 2018 | www.gnu.org

Welcome to the web page of the pexec program!

The main purpose of the program pexec is to execute the given command or shell script (e.g. parsed by /bin/sh ) in parallel on the local host or on remote hosts, while some of the execution parameters, namely the redirected standard input, output or error and environmental variables can be varied. This program is therefore capable to replace the classic shell loop iterators (e.g. for ~ in ~ done , in bash ) by executing the body of the loop in parallel. Thus, the program pexec implements shell level data parallelism in a barely simple form. The capabilities of the program is extended with additional features, such as allowing to define mutual exclusions, do atomic command executions and implement higher level resource and job control. See the complete manual for more details. See a brief Hungarian description of the program here .

The actual version of the program package is 1.0rc8 .

You may browse the package directory here (for FTP access, see this directory ). See the GNU summary page of this project here . The latest version of the program source package is pexec-1.0rc8.tar.gz . Here is another mirror of the package directory.

Please consider making donations to the author (via PayPal ) in order to help further development of the program or support the GNU project via the FSF .

[Nov 08, 2018] How to split one string into multiple variables in bash shell? [duplicate]

Nov 08, 2018 | stackoverflow.com
This question already has an answer here:

Rob I , May 9, 2012 at 19:22

For your second question, see @mkb's comment to my answer below - that's definitely the way to go! – Rob I May 9 '12 at 19:22

Dennis Williamson , Jul 4, 2012 at 16:14

See my edited answer for one way to read individual characters into an array. – Dennis Williamson Jul 4 '12 at 16:14

Nick Weedon , Dec 31, 2015 at 11:04

Here is the same thing in a more concise form: var1=$(cut -f1 -d- <<<$STR) – Nick Weedon Dec 31 '15 at 11:04

Rob I , May 9, 2012 at 17:00

If your solution doesn't have to be general, i.e. only needs to work for strings like your example, you could do:
var1=$(echo $STR | cut -f1 -d-)
var2=$(echo $STR | cut -f2 -d-)

I chose cut here because you could simply extend the code for a few more variables...

crunchybutternut , May 9, 2012 at 17:40

Can you look at my post again and see if you have a solution for the followup question? thanks! – crunchybutternut May 9 '12 at 17:40

mkb , May 9, 2012 at 17:59

You can use cut to cut characters too! cut -c1 for example. – mkb May 9 '12 at 17:59

FSp , Nov 27, 2012 at 10:26

Although this is very simple to read and write, is a very slow solution because forces you to read twice the same data ($STR) ... if you care of your script performace, the @anubhava solution is much better – FSp Nov 27 '12 at 10:26

tripleee , Jan 25, 2016 at 6:47

Apart from being an ugly last-resort solution, this has a bug: You should absolutely use double quotes in echo "$STR" unless you specifically want the shell to expand any wildcards in the string as a side effect. See also stackoverflow.com/questions/10067266/tripleee Jan 25 '16 at 6:47

Rob I , Feb 10, 2016 at 13:57

You're right about double quotes of course, though I did point out this solution wasn't general. However I think your assessment is a bit unfair - for some people this solution may be more readable (and hence extensible etc) than some others, and doesn't completely rely on arcane bash feature that wouldn't translate to other shells. I suspect that's why my solution, though less elegant, continues to get votes periodically... – Rob I Feb 10 '16 at 13:57

Dennis Williamson , May 10, 2012 at 3:14

read with IFS are perfect for this:
$ IFS=- read var1 var2 <<< ABCDE-123456
$ echo "$var1"
ABCDE
$ echo "$var2"
123456

Edit:

Here is how you can read each individual character into array elements:

$ read -a foo <<<"$(echo "ABCDE-123456" | sed 's/./& /g')"

Dump the array:

$ declare -p foo
declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]="-" [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'

If there are spaces in the string:

$ IFS=$'\v' read -a foo <<<"$(echo "ABCDE 123456" | sed 's/./&\v/g')"
$ declare -p foo
declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]=" " [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'

insecure , Apr 30, 2014 at 7:51

Great, the elegant bash-only way, without unnecessary forks. – insecure Apr 30 '14 at 7:51

Martin Serrano , Jan 11 at 4:34

this solution also has the benefit that if delimiter is not present, the var2 will be empty – Martin Serrano Jan 11 at 4:34

mkb , May 9, 2012 at 17:02

If you know it's going to be just two fields, you can skip the extra subprocesses like this:
var1=${STR%-*}
var2=${STR#*-}

What does this do? ${STR%-*} deletes the shortest substring of $STR that matches the pattern -* starting from the end of the string. ${STR#*-} does the same, but with the *- pattern and starting from the beginning of the string. They each have counterparts %% and ## which find the longest anchored pattern match. If anyone has a helpful mnemonic to remember which does which, let me know! I always have to try both to remember.

Jens , Jan 30, 2015 at 15:17

Plus 1 For knowing your POSIX shell features, avoiding expensive forks and pipes, and the absence of bashisms. – Jens Jan 30 '15 at 15:17

Steven Lu , May 1, 2015 at 20:19

Dunno about "absence of bashisms" considering that this is already moderately cryptic .... if your delimiter is a newline instead of a hyphen, then it becomes even more cryptic. On the other hand, it works with newlines , so there's that. – Steven Lu May 1 '15 at 20:19

mkb , Mar 9, 2016 at 17:30

@KErlandsson: done – mkb Mar 9 '16 at 17:30

mombip , Aug 9, 2016 at 15:58

I've finally found documentation for it: Shell-Parameter-Expansionmombip Aug 9 '16 at 15:58

DS. , Jan 13, 2017 at 19:56

Mnemonic: "#" is to the left of "%" on a standard keyboard, so "#" removes a prefix (on the left), and "%" removes a suffix (on the right). – DS. Jan 13 '17 at 19:56

tripleee , May 9, 2012 at 17:57

Sounds like a job for set with a custom IFS .
IFS=-
set $STR
var1=$1
var2=$2

(You will want to do this in a function with a local IFS so you don't mess up other parts of your script where you require IFS to be what you expect.)

Rob I , May 9, 2012 at 19:20

Nice - I knew about $IFS but hadn't seen how it could be used. – Rob I May 9 '12 at 19:20

Sigg3.net , Jun 19, 2013 at 8:08

I used triplee's example and it worked exactly as advertised! Just change last two lines to <pre> myvar1= echo $1 && myvar2= echo $2 </pre> if you need to store them throughout a script with several "thrown" variables. – Sigg3.net Jun 19 '13 at 8:08

tripleee , Jun 19, 2013 at 13:25

No, don't use a useless echo in backticks . – tripleee Jun 19 '13 at 13:25

Daniel Andersson , Mar 27, 2015 at 6:46

This is a really sweet solution if we need to write something that is not Bash specific. To handle IFS troubles, one can add OLDIFS=$IFS at the beginning before overwriting it, and then add IFS=$OLDIFS just after the set line. – Daniel Andersson Mar 27 '15 at 6:46

tripleee , Mar 27, 2015 at 6:58

FWIW the link above is broken. I was lazy and careless. The canonical location still works; iki.fi/era/unix/award.html#echotripleee Mar 27 '15 at 6:58

anubhava , May 9, 2012 at 17:09

Using bash regex capabilities:
re="^([^-]+)-(.*)$"
[[ "ABCDE-123456" =~ $re ]] && var1="${BASH_REMATCH[1]}" && var2="${BASH_REMATCH[2]}"
echo $var1
echo $var2

OUTPUT

ABCDE
123456

Cometsong , Oct 21, 2016 at 13:29

Love pre-defining the re for later use(s)! – Cometsong Oct 21 '16 at 13:29

Archibald , Nov 12, 2012 at 11:03

string="ABCDE-123456"
IFS=- # use "local IFS=-" inside the function
set $string
echo $1 # >>> ABCDE
echo $2 # >>> 123456

tripleee , Mar 27, 2015 at 7:02

Hmmm, isn't this just a restatement of my answer ? – tripleee Mar 27 '15 at 7:02

Archibald , Sep 18, 2015 at 12:36

Actually yes. I just clarified it a bit. – Archibald Sep 18 '15 at 12:36

[Nov 08, 2018] How to split a string in shell and get the last field

Nov 08, 2018 | stackoverflow.com

cd1 , Jul 1, 2010 at 23:29

Suppose I have the string 1:2:3:4:5 and I want to get its last field ( 5 in this case). How do I do that using Bash? I tried cut , but I don't know how to specify the last field with -f .

Stephen , Jul 2, 2010 at 0:05

You can use string operators :
$ foo=1:2:3:4:5
$ echo ${foo##*:}
5

This trims everything from the front until a ':', greedily.

${foo  <-- from variable foo
  ##   <-- greedy front trim
  *    <-- matches anything
  :    <-- until the last ':'
 }

eckes , Jan 23, 2013 at 15:23

While this is working for the given problem, the answer of William below ( stackoverflow.com/a/3163857/520162 ) also returns 5 if the string is 1:2:3:4:5: (while using the string operators yields an empty result). This is especially handy when parsing paths that could contain (or not) a finishing / character. – eckes Jan 23 '13 at 15:23

Dobz , Jun 25, 2014 at 11:44

How would you then do the opposite of this? to echo out '1:2:3:4:'? – Dobz Jun 25 '14 at 11:44

Mihai Danila , Jul 9, 2014 at 14:07

And how does one keep the part before the last separator? Apparently by using ${foo%:*} . # - from beginning; % - from end. # , % - shortest match; ## , %% - longest match. – Mihai Danila Jul 9 '14 at 14:07

Putnik , Feb 11, 2016 at 22:33

If i want to get the last element from path, how should I use it? echo ${pwd##*/} does not work. – Putnik Feb 11 '16 at 22:33

Stan Strum , Dec 17, 2017 at 4:22

@Putnik that command sees pwd as a variable. Try dir=$(pwd); echo ${dir##*/} . Works for me! – Stan Strum Dec 17 '17 at 4:22

a3nm , Feb 3, 2012 at 8:39

Another way is to reverse before and after cut :
$ echo ab:cd:ef | rev | cut -d: -f1 | rev
ef

This makes it very easy to get the last but one field, or any range of fields numbered from the end.

Dannid , Jan 14, 2013 at 20:50

This answer is nice because it uses 'cut', which the author is (presumably) already familiar. Plus, I like this answer because I am using 'cut' and had this exact question, hence finding this thread via search. – Dannid Jan 14 '13 at 20:50

funroll , Aug 12, 2013 at 19:51

Some cut-and-paste fodder for people using spaces as delimiters: echo "1 2 3 4" | rev | cut -d " " -f1 | revfunroll Aug 12 '13 at 19:51

EdgeCaseBerg , Sep 8, 2013 at 5:01

the rev | cut -d -f1 | rev is so clever! Thanks! Helped me a bunch (my use case was rev | -d ' ' -f 2- | rev – EdgeCaseBerg Sep 8 '13 at 5:01

Anarcho-Chossid , Sep 16, 2015 at 15:54

Wow. Beautiful and dark magic. – Anarcho-Chossid Sep 16 '15 at 15:54

shearn89 , Aug 17, 2017 at 9:27

I always forget about rev , was just what I needed! cut -b20- | rev | cut -b10- | revshearn89 Aug 17 '17 at 9:27

William Pursell , Jul 2, 2010 at 7:09

It's difficult to get the last field using cut, but here's (one set of) solutions in awk and perl
$ echo 1:2:3:4:5 | awk -F: '{print $NF}'
5
$ echo 1:2:3:4:5 | perl -F: -wane 'print $F[-1]'
5

eckes , Jan 23, 2013 at 15:20

great advantage of this solution over the accepted answer: it also matches paths that contain or do not contain a finishing / character: /a/b/c/d and /a/b/c/d/ yield the same result ( d ) when processing pwd | awk -F/ '{print $NF}' . The accepted answer results in an empty result in the case of /a/b/c/d/eckes Jan 23 '13 at 15:20

stamster , May 21 at 11:52

@eckes In case of AWK solution, on GNU bash, version 4.3.48(1)-release that's not true, as it matters whenever you have trailing slash or not. Simply put AWK will use / as delimiter, and if your path is /my/path/dir/ it will use value after last delimiter, which is simply an empty string. So it's best to avoid trailing slash if you need to do such a thing like I do. – stamster May 21 at 11:52

Nicholas M T Elliott , Jul 1, 2010 at 23:39

Assuming fairly simple usage (no escaping of the delimiter, for example), you can use grep:
$ echo "1:2:3:4:5" | grep -oE "[^:]+$"
5

Breakdown - find all the characters not the delimiter ([^:]) at the end of the line ($). -o only prints the matching part.

Dennis Williamson , Jul 2, 2010 at 0:05

One way:
var1="1:2:3:4:5"
var2=${var1##*:}

Another, using an array:

var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
var2=${var2[@]: -1}

Yet another with an array:

var1="1:2:3:4:5"
saveIFS=$IFS
IFS=":"
var2=($var1)
IFS=$saveIFS
count=${#var2[@]}
var2=${var2[$count-1]}

Using Bash (version >= 3.2) regular expressions:

var1="1:2:3:4:5"
[[ $var1 =~ :([^:]*)$ ]]
var2=${BASH_REMATCH[1]}

liuyang1 , Mar 24, 2015 at 6:02

Thanks so much for array style, as I need this feature, but not have cut, awk these utils. – liuyang1 Mar 24 '15 at 6:02

user3133260 , Dec 24, 2013 at 19:04

$ echo "a b c d e" | tr ' ' '\n' | tail -1
e

Simply translate the delimiter into a newline and choose the last entry with tail -1 .

Yajo , Jul 30, 2014 at 10:13

It will fail if the last item contains a \n , but for most cases is the most readable solution. – Yajo Jul 30 '14 at 10:13

Rafael , Nov 10, 2016 at 10:09

Using sed :
$ echo '1:2:3:4:5' | sed 's/.*://' # => 5

$ echo '' | sed 's/.*://' # => (empty)

$ echo ':' | sed 's/.*://' # => (empty)
$ echo ':b' | sed 's/.*://' # => b
$ echo '::c' | sed 's/.*://' # => c

$ echo 'a' | sed 's/.*://' # => a
$ echo 'a:' | sed 's/.*://' # => (empty)
$ echo 'a:b' | sed 's/.*://' # => b
$ echo 'a::c' | sed 's/.*://' # => c

Ab Irato , Nov 13, 2013 at 16:10

If your last field is a single character, you could do this:
a="1:2:3:4:5"

echo ${a: -1}
echo ${a:(-1)}

Check string manipulation in bash .

gniourf_gniourf , Nov 13, 2013 at 16:15

This doesn't work: it gives the last character of a , not the last field . – gniourf_gniourf Nov 13 '13 at 16:15

Ab Irato , Nov 25, 2013 at 13:25

True, that's the idea, if you know the length of the last field it's good. If not you have to use something else... – Ab Irato Nov 25 '13 at 13:25

sphakka , Jan 25, 2016 at 16:24

Interesting, I didn't know of these particular Bash string manipulations. It also resembles to Python's string/array slicing . – sphakka Jan 25 '16 at 16:24

ghostdog74 , Jul 2, 2010 at 1:16

Using Bash.
$ var1="1:2:3:4:0"
$ IFS=":"
$ set -- $var1
$ eval echo  \$${#}
0

Sopalajo de Arrierez , Dec 24, 2014 at 5:04

I would buy some details about this method, please :-) . – Sopalajo de Arrierez Dec 24 '14 at 5:04

Rafa , Apr 27, 2017 at 22:10

Could have used echo ${!#} instead of eval echo \$${#} . – Rafa Apr 27 '17 at 22:10

Crytis , Dec 7, 2016 at 6:51

echo "a:b:c:d:e"|xargs -d : -n1|tail -1

First use xargs split it using ":",-n1 means every line only have one part.Then,pring the last part.

BDL , Dec 7, 2016 at 13:47

Although this might solve the problem, one should always add an explanation to it. – BDL Dec 7 '16 at 13:47

Crytis , Jun 7, 2017 at 9:13

already added.. – Crytis Jun 7 '17 at 9:13

021 , Apr 26, 2016 at 11:33

There are many good answers here, but still I want to share this one using basename :
 basename $(echo "a:b:c:d:e" | tr ':' '/')

However it will fail if there are already some '/' in your string . If slash / is your delimiter then you just have to (and should) use basename.

It's not the best answer but it just shows how you can be creative using bash commands.

Nahid Akbar , Jun 22, 2012 at 2:55

for x in `echo $str | tr ";" "\n"`; do echo $x; done

chepner , Jun 22, 2012 at 12:58

This runs into problems if there is whitespace in any of the fields. Also, it does not directly address the question of retrieving the last field. – chepner Jun 22 '12 at 12:58

Christoph Böddeker , Feb 19 at 15:50

For those that comfortable with Python, https://github.com/Russell91/pythonpy is a nice choice to solve this problem.
$ echo "a:b:c:d:e" | py -x 'x.split(":")[-1]'

From the pythonpy help: -x treat each row of stdin as x .

With that tool, it is easy to write python code that gets applied to the input.

baz , Nov 24, 2017 at 19:27

a solution using the read builtin
IFS=':' read -a field <<< "1:2:3:4:5"
echo ${field[4]}

[Nov 08, 2018] How do I split a string on a delimiter in Bash?

Notable quotes:
"... Bash shell script split array ..."
"... associative array ..."
"... pattern substitution ..."
"... Debian GNU/Linux ..."
Nov 08, 2018 | stackoverflow.com

stefanB , May 28, 2009 at 2:03

I have this string stored in a variable:
IN="bla@some.com;john@home.com"

Now I would like to split the strings by ; delimiter so that I have:

ADDR1="bla@some.com"
ADDR2="john@home.com"

I don't necessarily need the ADDR1 and ADDR2 variables. If they are elements of an array that's even better.


After suggestions from the answers below, I ended up with the following which is what I was after:

#!/usr/bin/env bash

IN="bla@some.com;john@home.com"

mails=$(echo $IN | tr ";" "\n")

for addr in $mails
do
    echo "> [$addr]"
done

Output:

> [bla@some.com]
> [john@home.com]

There was a solution involving setting Internal_field_separator (IFS) to ; . I am not sure what happened with that answer, how do you reset IFS back to default?

RE: IFS solution, I tried this and it works, I keep the old IFS and then restore it:

IN="bla@some.com;john@home.com"

OIFS=$IFS
IFS=';'
mails2=$IN
for x in $mails2
do
    echo "> [$x]"
done

IFS=$OIFS

BTW, when I tried

mails2=($IN)

I only got the first string when printing it in loop, without brackets around $IN it works.

Brooks Moses , May 1, 2012 at 1:26

With regards to your "Edit2": You can simply "unset IFS" and it will return to the default state. There's no need to save and restore it explicitly unless you have some reason to expect that it's already been set to a non-default value. Moreover, if you're doing this inside a function (and, if you aren't, why not?), you can set IFS as a local variable and it will return to its previous value once you exit the function. – Brooks Moses May 1 '12 at 1:26

dubiousjim , May 31, 2012 at 5:21

@BrooksMoses: (a) +1 for using local IFS=... where possible; (b) -1 for unset IFS , this doesn't exactly reset IFS to its default value, though I believe an unset IFS behaves the same as the default value of IFS ($' \t\n'), however it seems bad practice to be assuming blindly that your code will never be invoked with IFS set to a custom value; (c) another idea is to invoke a subshell: (IFS=$custom; ...) when the subshell exits IFS will return to whatever it was originally. – dubiousjim May 31 '12 at 5:21

nicooga , Mar 7, 2016 at 15:32

I just want to have a quick look at the paths to decide where to throw an executable, so I resorted to run ruby -e "puts ENV.fetch('PATH').split(':')" . If you want to stay pure bash won't help but using any scripting language that has a built-in split is easier. – nicooga Mar 7 '16 at 15:32

Jeff , Apr 22 at 17:51

This is kind of a drive-by comment, but since the OP used email addresses as the example, has anyone bothered to answer it in a way that is fully RFC 5322 compliant, namely that any quoted string can appear before the @ which means you're going to need regular expressions or some other kind of parser instead of naive use of IFS or other simplistic splitter functions. – Jeff Apr 22 at 17:51

user2037659 , Apr 26 at 20:15

for x in $(IFS=';';echo $IN); do echo "> [$x]"; doneuser2037659 Apr 26 at 20:15

Johannes Schaub - litb , May 28, 2009 at 2:23

You can set the internal field separator (IFS) variable, and then let it parse into an array. When this happens in a command, then the assignment to IFS only takes place to that single command's environment (to read ). It then parses the input according to the IFS variable value into an array, which we can then iterate over.
IFS=';' read -ra ADDR <<< "$IN"
for i in "${ADDR[@]}"; do
    # process "$i"
done

It will parse one line of items separated by ; , pushing it into an array. Stuff for processing whole of $IN , each time one line of input separated by ; :

 while IFS=';' read -ra ADDR; do
      for i in "${ADDR[@]}"; do
          # process "$i"
      done
 done <<< "$IN"

Chris Lutz , May 28, 2009 at 2:25

This is probably the best way. How long will IFS persist in it's current value, can it mess up my code by being set when it shouldn't be, and how can I reset it when I'm done with it? – Chris Lutz May 28 '09 at 2:25

Johannes Schaub - litb , May 28, 2009 at 3:04

now after the fix applied, only within the duration of the read command :) – Johannes Schaub - litb May 28 '09 at 3:04

lhunath , May 28, 2009 at 6:14

You can read everything at once without using a while loop: read -r -d '' -a addr <<< "$in" # The -d '' is key here, it tells read not to stop at the first newline (which is the default -d) but to continue until EOF or a NULL byte (which only occur in binary data). – lhunath May 28 '09 at 6:14

Charles Duffy , Jul 6, 2013 at 14:39

@LucaBorrione Setting IFS on the same line as the read with no semicolon or other separator, as opposed to in a separate command, scopes it to that command -- so it's always "restored"; you don't need to do anything manually. – Charles Duffy Jul 6 '13 at 14:39

chepner , Oct 2, 2014 at 3:50

@imagineerThis There is a bug involving herestrings and local changes to IFS that requires $IN to be quoted. The bug is fixed in bash 4.3. – chepner Oct 2 '14 at 3:50

palindrom , Mar 10, 2011 at 9:00

Taken from Bash shell script split array :
IN="bla@some.com;john@home.com"
arrIN=(${IN//;/ })

Explanation:

This construction replaces all occurrences of ';' (the initial // means global replace) in the string IN with ' ' (a single space), then interprets the space-delimited string as an array (that's what the surrounding parentheses do).

The syntax used inside of the curly braces to replace each ';' character with a ' ' character is called Parameter Expansion .

There are some common gotchas:

  1. If the original string has spaces, you will need to use IFS :
    • IFS=':'; arrIN=($IN); unset IFS;
  2. If the original string has spaces and the delimiter is a new line, you can set IFS with:
    • IFS=$'\n'; arrIN=($IN); unset IFS;

Oz123 , Mar 21, 2011 at 18:50

I just want to add: this is the simplest of all, you can access array elements with ${arrIN[1]} (starting from zeros of course) – Oz123 Mar 21 '11 at 18:50

KomodoDave , Jan 5, 2012 at 15:13

Found it: the technique of modifying a variable within a ${} is known as 'parameter expansion'. – KomodoDave Jan 5 '12 at 15:13

qbolec , Feb 25, 2013 at 9:12

Does it work when the original string contains spaces? – qbolec Feb 25 '13 at 9:12

Ethan , Apr 12, 2013 at 22:47

No, I don't think this works when there are also spaces present... it's converting the ',' to ' ' and then building a space-separated array. – Ethan Apr 12 '13 at 22:47

Charles Duffy , Jul 6, 2013 at 14:39

This is a bad approach for other reasons: For instance, if your string contains ;*; , then the * will be expanded to a list of filenames in the current directory. -1 – Charles Duffy Jul 6 '13 at 14:39

Chris Lutz , May 28, 2009 at 2:09

If you don't mind processing them immediately, I like to do this:
for i in $(echo $IN | tr ";" "\n")
do
  # process
done

You could use this kind of loop to initialize an array, but there's probably an easier way to do it. Hope this helps, though.

Chris Lutz , May 28, 2009 at 2:42

You should have kept the IFS answer. It taught me something I didn't know, and it definitely made an array, whereas this just makes a cheap substitute. – Chris Lutz May 28 '09 at 2:42

Johannes Schaub - litb , May 28, 2009 at 2:59

I see. Yeah i find doing these silly experiments, i'm going to learn new things each time i'm trying to answer things. I've edited stuff based on #bash IRC feedback and undeleted :) – Johannes Schaub - litb May 28 '09 at 2:59

lhunath , May 28, 2009 at 6:12

-1, you're obviously not aware of wordsplitting, because it's introducing two bugs in your code. one is when you don't quote $IN and the other is when you pretend a newline is the only delimiter used in wordsplitting. You are iterating over every WORD in IN, not every line, and DEFINATELY not every element delimited by a semicolon, though it may appear to have the side-effect of looking like it works. – lhunath May 28 '09 at 6:12

Johannes Schaub - litb , May 28, 2009 at 17:00

You could change it to echo "$IN" | tr ';' '\n' | while read -r ADDY; do # process "$ADDY"; done to make him lucky, i think :) Note that this will fork, and you can't change outer variables from within the loop (that's why i used the <<< "$IN" syntax) then – Johannes Schaub - litb May 28 '09 at 17:00

mklement0 , Apr 24, 2013 at 14:13

To summarize the debate in the comments: Caveats for general use : the shell applies word splitting and expansions to the string, which may be undesired; just try it with. IN="bla@some.com;john@home.com;*;broken apart" . In short: this approach will break, if your tokens contain embedded spaces and/or chars. such as * that happen to make a token match filenames in the current folder. – mklement0 Apr 24 '13 at 14:13

F. Hauri , Apr 13, 2013 at 14:20

Compatible answer

To this SO question, there is already a lot of different way to do this in bash . But bash has many special features, so called bashism that work well, but that won't work in any other shell .

In particular, arrays , associative array , and pattern substitution are pure bashisms and may not work under other shells .

On my Debian GNU/Linux , there is a standard shell called dash , but I know many people who like to use ksh .

Finally, in very small situation, there is a special tool called busybox with his own shell interpreter ( ash ).

Requested string

The string sample in SO question is:

IN="bla@some.com;john@home.com"

As this could be useful with whitespaces and as whitespaces could modify the result of the routine, I prefer to use this sample string:

 IN="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
Split string based on delimiter in bash (version >=4.2)

Under pure bash, we may use arrays and IFS :

var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
oIFS="$IFS"
IFS=";"
declare -a fields=($var)
IFS="$oIFS"
unset oIFS

IFS=\; read -a fields <<<"$var"

Using this syntax under recent bash don't change $IFS for current session, but only for the current command:

set | grep ^IFS=
IFS=$' \t\n'

Now the string var is split and stored into an array (named fields ):

set | grep ^fields=\\\|^var=
fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>")
var='bla@some.com;john@home.com;Full Name <fulnam@other.org>'

We could request for variable content with declare -p :

declare -p var fields
declare -- var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
declare -a fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>")

read is the quickiest way to do the split, because there is no forks and no external resources called.

From there, you could use the syntax you already know for processing each field:

for x in "${fields[@]}";do
    echo "> [$x]"
    done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

or drop each field after processing (I like this shifting approach):

while [ "$fields" ] ;do
    echo "> [$fields]"
    fields=("${fields[@]:1}")
    done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

or even for simple printout (shorter syntax):

printf "> [%s]\n" "${fields[@]}"
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]
Split string based on delimiter in shell

But if you would write something usable under many shells, you have to not use bashisms .

There is a syntax, used in many shells, for splitting a string across first or last occurrence of a substring:

${var#*SubStr}  # will drop begin of string up to first occur of `SubStr`
${var##*SubStr} # will drop begin of string up to last occur of `SubStr`
${var%SubStr*}  # will drop part of string from last occur of `SubStr` to the end
${var%%SubStr*} # will drop part of string from first occur of `SubStr` to the end

(The missing of this is the main reason of my answer publication ;)

As pointed out by Score_Under :

# and % delete the shortest possible matching string, and

## and %% delete the longest possible.

This little sample script work well under bash , dash , ksh , busybox and was tested under Mac-OS's bash too:

var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"
while [ "$var" ] ;do
    iter=${var%%;*}
    echo "> [$iter]"
    [ "$var" = "$iter" ] && \
        var='' || \
        var="${var#*;}"
  done
> [bla@some.com]
> [john@home.com]
> [Full Name <fulnam@other.org>]

Have fun!

Score_Under , Apr 28, 2015 at 16:58

The # , ## , % , and %% substitutions have what is IMO an easier explanation to remember (for how much they delete): # and % delete the shortest possible matching string, and ## and %% delete the longest possible. – Score_Under Apr 28 '15 at 16:58

sorontar , Oct 26, 2016 at 4:36

The IFS=\; read -a fields <<<"$var" fails on newlines and add a trailing newline. The other solution removes a trailing empty field. – sorontar Oct 26 '16 at 4:36

Eric Chen , Aug 30, 2017 at 17:50

The shell delimiter is the most elegant answer, period. – Eric Chen Aug 30 '17 at 17:50

sancho.s , Oct 4 at 3:42

Could the last alternative be used with a list of field separators set somewhere else? For instance, I mean to use this as a shell script, and pass a list of field separators as a positional parameter. – sancho.s Oct 4 at 3:42

F. Hauri , Oct 4 at 7:47

Yes, in a loop: for sep in "#" "ł" "@" ; do ... var="${var#*$sep}" ...F. Hauri Oct 4 at 7:47

DougW , Apr 27, 2015 at 18:20

I've seen a couple of answers referencing the cut command, but they've all been deleted. It's a little odd that nobody has elaborated on that, because I think it's one of the more useful commands for doing this type of thing, especially for parsing delimited log files.

In the case of splitting this specific example into a bash script array, tr is probably more efficient, but cut can be used, and is more effective if you want to pull specific fields from the middle.

Example:

$ echo "bla@some.com;john@home.com" | cut -d ";" -f 1
bla@some.com
$ echo "bla@some.com;john@home.com" | cut -d ";" -f 2
john@home.com

You can obviously put that into a loop, and iterate the -f parameter to pull each field independently.

This gets more useful when you have a delimited log file with rows like this:

2015-04-27|12345|some action|an attribute|meta data

cut is very handy to be able to cat this file and select a particular field for further processing.

MisterMiyagi , Nov 2, 2016 at 8:42

Kudos for using cut , it's the right tool for the job! Much cleared than any of those shell hacks. – MisterMiyagi Nov 2 '16 at 8:42

uli42 , Sep 14, 2017 at 8:30

This approach will only work if you know the number of elements in advance; you'd need to program some more logic around it. It also runs an external tool for every element. – uli42 Sep 14 '17 at 8:30

Louis Loudog Trottier , May 10 at 4:20

Excatly waht i was looking for trying to avoid empty string in a csv. Now i can point the exact 'column' value as well. Work with IFS already used in a loop. Better than expected for my situation. – Louis Loudog Trottier May 10 at 4:20

, May 28, 2009 at 10:31

How about this approach:
IN="bla@some.com;john@home.com" 
set -- "$IN" 
IFS=";"; declare -a Array=($*) 
echo "${Array[@]}" 
echo "${Array[0]}" 
echo "${Array[1]}"

Source

Yzmir Ramirez , Sep 5, 2011 at 1:06

+1 ... but I wouldn't name the variable "Array" ... pet peev I guess. Good solution. – Yzmir Ramirez Sep 5 '11 at 1:06

ata , Nov 3, 2011 at 22:33

+1 ... but the "set" and declare -a are unnecessary. You could as well have used just IFS";" && Array=($IN)ata Nov 3 '11 at 22:33

Luca Borrione , Sep 3, 2012 at 9:26

+1 Only a side note: shouldn't it be recommendable to keep the old IFS and then restore it? (as shown by stefanB in his edit3) people landing here (sometimes just copying and pasting a solution) might not think about this – Luca Borrione Sep 3 '12 at 9:26

Charles Duffy , Jul 6, 2013 at 14:44

-1: First, @ata is right that most of the commands in this do nothing. Second, it uses word-splitting to form the array, and doesn't do anything to inhibit glob-expansion when doing so (so if you have glob characters in any of the array elements, those elements are replaced with matching filenames). – Charles Duffy Jul 6 '13 at 14:44

John_West , Jan 8, 2016 at 12:29

Suggest to use $'...' : IN=$'bla@some.com;john@home.com;bet <d@\ns* kl.com>' . Then echo "${Array[2]}" will print a string with newline. set -- "$IN" is also neccessary in this case. Yes, to prevent glob expansion, the solution should include set -f . – John_West Jan 8 '16 at 12:29

Steven Lizarazo , Aug 11, 2016 at 20:45

This worked for me:
string="1;2"
echo $string | cut -d';' -f1 # output is 1
echo $string | cut -d';' -f2 # output is 2

Pardeep Sharma , Oct 10, 2017 at 7:29

this is sort and sweet :) – Pardeep Sharma Oct 10 '17 at 7:29

space earth , Oct 17, 2017 at 7:23

Thanks...Helped a lot – space earth Oct 17 '17 at 7:23

mojjj , Jan 8 at 8:57

cut works only with a single char as delimiter. – mojjj Jan 8 at 8:57

lothar , May 28, 2009 at 2:12

echo "bla@some.com;john@home.com" | sed -e 's/;/\n/g'
bla@some.com
john@home.com

Luca Borrione , Sep 3, 2012 at 10:08

-1 what if the string contains spaces? for example IN="this is first line; this is second line" arrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) ) will produce an array of 8 elements in this case (an element for each word space separated), rather than 2 (an element for each line semi colon separated) – Luca Borrione Sep 3 '12 at 10:08

lothar , Sep 3, 2012 at 17:33

@Luca No the sed script creates exactly two lines. What creates the multiple entries for you is when you put it into a bash array (which splits on white space by default) – lothar Sep 3 '12 at 17:33

Luca Borrione , Sep 4, 2012 at 7:09

That's exactly the point: the OP needs to store entries into an array to loop over it, as you can see in his edits. I think your (good) answer missed to mention to use arrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) ) to achieve that, and to advice to change IFS to IFS=$'\n' for those who land here in the future and needs to split a string containing spaces. (and to restore it back afterwards). :) – Luca Borrione Sep 4 '12 at 7:09

lothar , Sep 4, 2012 at 16:55

@Luca Good point. However the array assignment was not in the initial question when I wrote up that answer. – lothar Sep 4 '12 at 16:55

Ashok , Sep 8, 2012 at 5:01

This also works:
IN="bla@some.com;john@home.com"
echo ADD1=`echo $IN | cut -d \; -f 1`
echo ADD2=`echo $IN | cut -d \; -f 2`

Be careful, this solution is not always correct. In case you pass "bla@some.com" only, it will assign it to both ADD1 and ADD2.

fersarr , Mar 3, 2016 at 17:17

You can use -s to avoid the mentioned problem: superuser.com/questions/896800/ "-f, --fields=LIST select only these fields; also print any line that contains no delimiter character, unless the -s option is specified" – fersarr Mar 3 '16 at 17:17

Tony , Jan 14, 2013 at 6:33

I think AWK is the best and efficient command to resolve your problem. AWK is included in Bash by default in almost every Linux distribution.
echo "bla@some.com;john@home.com" | awk -F';' '{print $1,$2}'

will give

bla@some.com john@home.com

Of course your can store each email address by redefining the awk print field.

Jaro , Jan 7, 2014 at 21:30

Or even simpler: echo "bla@some.com;john@home.com" | awk 'BEGIN{RS=";"} {print}' – Jaro Jan 7 '14 at 21:30

Aquarelle , May 6, 2014 at 21:58

@Jaro This worked perfectly for me when I had a string with commas and needed to reformat it into lines. Thanks. – Aquarelle May 6 '14 at 21:58

Eduardo Lucio , Aug 5, 2015 at 12:59

It worked in this scenario -> "echo "$SPLIT_0" | awk -F' inode=' '{print $1}'"! I had problems when trying to use atrings (" inode=") instead of characters (";"). $ 1, $ 2, $ 3, $ 4 are set as positions in an array! If there is a way of setting an array... better! Thanks! – Eduardo Lucio Aug 5 '15 at 12:59

Tony , Aug 6, 2015 at 2:42

@EduardoLucio, what I'm thinking about is maybe you can first replace your delimiter inode= into ; for example by sed -i 's/inode\=/\;/g' your_file_to_process , then define -F';' when apply awk , hope that can help you. – Tony Aug 6 '15 at 2:42

nickjb , Jul 5, 2011 at 13:41

A different take on Darron's answer , this is how I do it:
IN="bla@some.com;john@home.com"
read ADDR1 ADDR2 <<<$(IFS=";"; echo $IN)

ColinM , Sep 10, 2011 at 0:31

This doesn't work. – ColinM Sep 10 '11 at 0:31

nickjb , Oct 6, 2011 at 15:33

I think it does! Run the commands above and then "echo $ADDR1 ... $ADDR2" and i get "bla@some.com ... john@home.com" output – nickjb Oct 6 '11 at 15:33

Nick , Oct 28, 2011 at 14:36

This worked REALLY well for me... I used it to itterate over an array of strings which contained comma separated DB,SERVER,PORT data to use mysqldump. – Nick Oct 28 '11 at 14:36

dubiousjim , May 31, 2012 at 5:28

Diagnosis: the IFS=";" assignment exists only in the $(...; echo $IN) subshell; this is why some readers (including me) initially think it won't work. I assumed that all of $IN was getting slurped up by ADDR1. But nickjb is correct; it does work. The reason is that echo $IN command parses its arguments using the current value of $IFS, but then echoes them to stdout using a space delimiter, regardless of the setting of $IFS. So the net effect is as though one had called read ADDR1 ADDR2 <<< "bla@some.com john@home.com" (note the input is space-separated not ;-separated). – dubiousjim May 31 '12 at 5:28

sorontar , Oct 26, 2016 at 4:43

This fails on spaces and newlines, and also expand wildcards * in the echo $IN with an unquoted variable expansion. – sorontar Oct 26 '16 at 4:43

gniourf_gniourf , Jun 26, 2014 at 9:11

In Bash, a bullet proof way, that will work even if your variable contains newlines:
IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")

Look:

$ in=$'one;two three;*;there is\na newline\nin this field'
$ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")
$ declare -p array
declare -a array='([0]="one" [1]="two three" [2]="*" [3]="there is
a newline
in this field")'

The trick for this to work is to use the -d option of read (delimiter) with an empty delimiter, so that read is forced to read everything it's fed. And we feed read with exactly the content of the variable in , with no trailing newline thanks to printf . Note that's we're also putting the delimiter in printf to ensure that the string passed to read has a trailing delimiter. Without it, read would trim potential trailing empty fields:

$ in='one;two;three;'    # there's an empty field
$ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")
$ declare -p array
declare -a array='([0]="one" [1]="two" [2]="three" [3]="")'

the trailing empty field is preserved.


Update for Bash≥4.4

Since Bash 4.4, the builtin mapfile (aka readarray ) supports the -d option to specify a delimiter. Hence another canonical way is:

mapfile -d ';' -t array < <(printf '%s;' "$in")

John_West , Jan 8, 2016 at 12:10

I found it as the rare solution on that list that works correctly with \n , spaces and * simultaneously. Also, no loops; array variable is accessible in the shell after execution (contrary to the highest upvoted answer). Note, in=$'...' , it does not work with double quotes. I think, it needs more upvotes. – John_West Jan 8 '16 at 12:10

Darron , Sep 13, 2010 at 20:10

How about this one liner, if you're not using arrays:
IFS=';' read ADDR1 ADDR2 <<<$IN

dubiousjim , May 31, 2012 at 5:36

Consider using read -r ... to ensure that, for example, the two characters "\t" in the input end up as the same two characters in your variables (instead of a single tab char). – dubiousjim May 31 '12 at 5:36

Luca Borrione , Sep 3, 2012 at 10:07

-1 This is not working here (ubuntu 12.04). Adding echo "ADDR1 $ADDR1"\n echo "ADDR2 $ADDR2" to your snippet will output ADDR1 bla@some.com john@home.com\nADDR2 (\n is newline) – Luca Borrione Sep 3 '12 at 10:07

chepner , Sep 19, 2015 at 13:59

This is probably due to a bug involving IFS and here strings that was fixed in bash 4.3. Quoting $IN should fix it. (In theory, $IN is not subject to word splitting or globbing after it expands, meaning the quotes should be unnecessary. Even in 4.3, though, there's at least one bug remaining--reported and scheduled to be fixed--so quoting remains a good idea.) – chepner Sep 19 '15 at 13:59

sorontar , Oct 26, 2016 at 4:55

This breaks if $in contain newlines even if $IN is quoted. And adds a trailing newline. – sorontar Oct 26 '16 at 4:55

kenorb , Sep 11, 2015 at 20:54

Here is a clean 3-liner:
in="foo@bar;bizz@buzz;fizz@buzz;buzz@woof"
IFS=';' list=($in)
for item in "${list[@]}"; do echo $item; done

where IFS delimit words based on the separator and () is used to create an array . Then [@] is used to return each item as a separate word.

If you've any code after that, you also need to restore $IFS , e.g. unset IFS .

sorontar , Oct 26, 2016 at 5:03

The use of $in unquoted allows wildcards to be expanded. – sorontar Oct 26 '16 at 5:03

user2720864 , Sep 24 at 13:46

+ for the unset command – user2720864 Sep 24 at 13:46

Emilien Brigand , Aug 1, 2016 at 13:15

Without setting the IFS

If you just have one colon you can do that:

a="foo:bar"
b=${a%:*}
c=${a##*:}

you will get:

b = foo
c = bar

Victor Choy , Sep 16, 2015 at 3:34

There is a simple and smart way like this:
echo "add:sfff" | xargs -d: -i  echo {}

But you must use gnu xargs, BSD xargs cant support -d delim. If you use apple mac like me. You can install gnu xargs :

brew install findutils

then

echo "add:sfff" | gxargs -d: -i  echo {}

Halle Knast , May 24, 2017 at 8:42

The following Bash/zsh function splits its first argument on the delimiter given by the second argument:
split() {
    local string="$1"
    local delimiter="$2"
    if [ -n "$string" ]; then
        local part
        while read -d "$delimiter" part; do
            echo $part
        done <<< "$string"
        echo $part
    fi
}

For instance, the command

$ split 'a;b;c' ';'

yields

a
b
c

This output may, for instance, be piped to other commands. Example:

$ split 'a;b;c' ';' | cat -n
1   a
2   b
3   c

Compared to the other solutions given, this one has the following advantages:

If desired, the function may be put into a script as follows:

#!/usr/bin/env bash

split() {
    # ...
}

split "$@"

sandeepkunkunuru , Oct 23, 2017 at 16:10

works and neatly modularized. – sandeepkunkunuru Oct 23 '17 at 16:10

Prospero , Sep 25, 2011 at 1:09

This is the simplest way to do it.
spo='one;two;three'
OIFS=$IFS
IFS=';'
spo_array=($spo)
IFS=$OIFS
echo ${spo_array[*]}

rashok , Oct 25, 2016 at 12:41

IN="bla@some.com;john@home.com"
IFS=';'
read -a IN_arr <<< "${IN}"
for entry in "${IN_arr[@]}"
do
    echo $entry
done

Output

bla@some.com
john@home.com

System : Ubuntu 12.04.1

codeforester , Jan 2, 2017 at 5:37

IFS is not getting set in the specific context of read here and hence it can upset rest of the code, if any. – codeforester Jan 2 '17 at 5:37

shuaihanhungry , Jan 20 at 15:54

you can apply awk to many situations
echo "bla@some.com;john@home.com"|awk -F';' '{printf "%s\n%s\n", $1, $2}'

also you can use this

echo "bla@some.com;john@home.com"|awk -F';' '{print $1,$2}' OFS="\n"

ghost , Apr 24, 2013 at 13:13

If no space, Why not this?
IN="bla@some.com;john@home.com"
arr=(`echo $IN | tr ';' ' '`)

echo ${arr[0]}
echo ${arr[1]}

eukras , Oct 22, 2012 at 7:10

There are some cool answers here (errator esp.), but for something analogous to split in other languages -- which is what I took the original question to mean -- I settled on this:
IN="bla@some.com;john@home.com"
declare -a a="(${IN/;/ })";

Now ${a[0]} , ${a[1]} , etc, are as you would expect. Use ${#a[*]} for number of terms. Or to iterate, of course:

for i in ${a[*]}; do echo $i; done

IMPORTANT NOTE:

This works in cases where there are no spaces to worry about, which solved my problem, but may not solve yours. Go with the $IFS solution(s) in that case.

olibre , Oct 7, 2013 at 13:33

Does not work when IN contains more than two e-mail addresses. Please refer to same idea (but fixed) at palindrom's answerolibre Oct 7 '13 at 13:33

sorontar , Oct 26, 2016 at 5:14

Better use ${IN//;/ } (double slash) to make it also work with more than two values. Beware that any wildcard ( *?[ ) will be expanded. And a trailing empty field will be discarded. – sorontar Oct 26 '16 at 5:14

jeberle , Apr 30, 2013 at 3:10

Use the set built-in to load up the $@ array:
IN="bla@some.com;john@home.com"
IFS=';'; set $IN; IFS=$' \t\n'

Then, let the party begin:

echo $#
for a; do echo $a; done
ADDR1=$1 ADDR2=$2

sorontar , Oct 26, 2016 at 5:17

Better use set -- $IN to avoid some issues with "$IN" starting with dash. Still, the unquoted expansion of $IN will expand wildcards ( *?[ ). – sorontar Oct 26 '16 at 5:17

NevilleDNZ , Sep 2, 2013 at 6:30

Two bourne-ish alternatives where neither require bash arrays:

Case 1 : Keep it nice and simple: Use a NewLine as the Record-Separator... eg.

IN="bla@some.com
john@home.com"

while read i; do
  # process "$i" ... eg.
    echo "[email:$i]"
done <<< "$IN"

Note: in this first case no sub-process is forked to assist with list manipulation.

Idea: Maybe it is worth using NL extensively internally , and only converting to a different RS when generating the final result externally .

Case 2 : Using a ";" as a record separator... eg.

NL="
" IRS=";" ORS=";"

conv_IRS() {
  exec tr "$1" "$NL"
}

conv_ORS() {
  exec tr "$NL" "$1"
}

IN="bla@some.com;john@home.com"
IN="$(conv_IRS ";" <<< "$IN")"

while read i; do
  # process "$i" ... eg.
    echo -n "[email:$i]$ORS"
done <<< "$IN"

In both cases a sub-list can be composed within the loop is persistent after the loop has completed. This is useful when manipulating lists in memory, instead storing lists in files. {p.s. keep calm and carry on B-) }

fedorqui , Jan 8, 2015 at 10:21

Apart from the fantastic answers that were already provided, if it is just a matter of printing out the data you may consider using awk :
awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN"

This sets the field separator to ; , so that it can loop through the fields with a for loop and print accordingly.

Test
$ IN="bla@some.com;john@home.com"
$ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN"
> [bla@some.com]
> [john@home.com]

With another input:

$ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "a;b;c   d;e_;f"
> [a]
> [b]
> [c   d]
> [e_]
> [f]

18446744073709551615 , Feb 20, 2015 at 10:49

In Android shell, most of the proposed methods just do not work:
$ IFS=':' read -ra ADDR <<<"$PATH"                             
/system/bin/sh: can't create temporary file /sqlite_stmt_journals/mksh.EbNoR10629: No such file or directory

What does work is:

$ for i in ${PATH//:/ }; do echo $i; done
/sbin
/vendor/bin
/system/sbin
/system/bin
/system/xbin

where // means global replacement.

sorontar , Oct 26, 2016 at 5:08

Fails if any part of $PATH contains spaces (or newlines). Also expands wildcards (asterisk *, question mark ? and braces [ ]). – sorontar Oct 26 '16 at 5:08

Eduardo Lucio , Apr 4, 2016 at 19:54

Okay guys!

Here's my answer!

DELIMITER_VAL='='

read -d '' F_ABOUT_DISTRO_R <<"EOF"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"
NAME="Ubuntu"
VERSION="14.04.4 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.4 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
EOF

SPLIT_NOW=$(awk -F$DELIMITER_VAL '{for(i=1;i<=NF;i++){printf "%s\n", $i}}' <<<"${F_ABOUT_DISTRO_R}")
while read -r line; do
   SPLIT+=("$line")
done <<< "$SPLIT_NOW"
for i in "${SPLIT[@]}"; do
    echo "$i"
done

Why this approach is "the best" for me?

Because of two reasons:

  1. You do not need to escape the delimiter;
  2. You will not have problem with blank spaces . The value will be properly separated in the array!

[]'s

gniourf_gniourf , Jan 30, 2017 at 8:26

FYI, /etc/os-release and /etc/lsb-release are meant to be sourced, and not parsed. So your method is really wrong. Moreover, you're not quite answering the question about spiltting a string on a delimiter.gniourf_gniourf Jan 30 '17 at 8:26

Michael Hale , Jun 14, 2012 at 17:38

A one-liner to split a string separated by ';' into an array is:
IN="bla@some.com;john@home.com"
ADDRS=( $(IFS=";" echo "$IN") )
echo ${ADDRS[0]}
echo ${ADDRS[1]}

This only sets IFS in a subshell, so you don't have to worry about saving and restoring its value.

Luca Borrione , Sep 3, 2012 at 10:04

-1 this doesn't work here (ubuntu 12.04). it prints only the first echo with all $IN value in it, while the second is empty. you can see it if you put echo "0: "${ADDRS[0]}\n echo "1: "${ADDRS[1]} the output is 0: bla@some.com;john@home.com\n 1: (\n is new line) – Luca Borrione Sep 3 '12 at 10:04

Luca Borrione , Sep 3, 2012 at 10:05

please refer to nickjb's answer at for a working alternative to this idea stackoverflow.com/a/6583589/1032370 – Luca Borrione Sep 3 '12 at 10:05

Score_Under , Apr 28, 2015 at 17:09

-1, 1. IFS isn't being set in that subshell (it's being passed to the environment of "echo", which is a builtin, so nothing is happening anyway). 2. $IN is quoted so it isn't subject to IFS splitting. 3. The process substitution is split by whitespace, but this may corrupt the original data. – Score_Under Apr 28 '15 at 17:09

ajaaskel , Oct 10, 2014 at 11:33

IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)'
set -f
oldifs="$IFS"
IFS=';'; arrayIN=($IN)
IFS="$oldifs"
for i in "${arrayIN[@]}"; do
echo "$i"
done
set +f

Output:

bla@some.com
john@home.com
Charlie Brown <cbrown@acme.com
!"#$%&/()[]{}*? are no problem
simple is beautiful :-)

Explanation: Simple assignment using parenthesis () converts semicolon separated list into an array provided you have correct IFS while doing that. Standard FOR loop handles individual items in that array as usual. Notice that the list given for IN variable must be "hard" quoted, that is, with single ticks.

IFS must be saved and restored since Bash does not treat an assignment the same way as a command. An alternate workaround is to wrap the assignment inside a function and call that function with a modified IFS. In that case separate saving/restoring of IFS is not needed. Thanks for "Bize" for pointing that out.

gniourf_gniourf , Feb 20, 2015 at 16:45

!"#$%&/()[]{}*? are no problem well... not quite: []*? are glob characters. So what about creating this directory and file: `mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem' and running your command? simple may be beautiful, but when it's broken, it's broken. – gniourf_gniourf Feb 20 '15 at 16:45

ajaaskel , Feb 25, 2015 at 7:20

@gniourf_gniourf The string is stored in a variable. Please see the original question. – ajaaskel Feb 25 '15 at 7:20

gniourf_gniourf , Feb 25, 2015 at 7:26

@ajaaskel you didn't fully understand my comment. Go in a scratch directory and issue these commands: mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem' . They will only create a directory and a file, with weird looking names, I must admit. Then run your commands with the exact IN you gave: IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)' . You'll see that you won't get the output you expect. Because you're using a method subject to pathname expansions to split your string. – gniourf_gniourf Feb 25 '15 at 7:26

gniourf_gniourf , Feb 25, 2015 at 7:29

This is to demonstrate that the characters * , ? , [...] and even, if extglob is set, !(...) , @(...) , ?(...) , +(...) are problems with this method! – gniourf_gniourf Feb 25 '15 at 7:29

ajaaskel , Feb 26, 2015 at 15:26

@gniourf_gniourf Thanks for detailed comments on globbing. I adjusted the code to have globbing off. My point was however just to show that rather simple assignment can do the splitting job. – ajaaskel Feb 26 '15 at 15:26

> , Dec 19, 2013 at 21:39

Maybe not the most elegant solution, but works with * and spaces:
IN="bla@so me.com;*;john@home.com"
for i in `delims=${IN//[^;]}; seq 1 $((${#delims} + 1))`
do
   echo "> [`echo $IN | cut -d';' -f$i`]"
done

Outputs

> [bla@so me.com]
> [*]
> [john@home.com]

Other example (delimiters at beginning and end):

IN=";bla@so me.com;*;john@home.com;"
> []
> [bla@so me.com]
> [*]
> [john@home.com]
> []

Basically it removes every character other than ; making delims eg. ;;; . Then it does for loop from 1 to number-of-delimiters as counted by ${#delims} . The final step is to safely get the $i th part using cut .

[Nov 08, 2018] 15 Linux Split and Join Command Examples to Manage Large Files

Nov 08, 2018 | www.thegeekstuff.com

by Himanshu Arora on October 16, 2012

https://apis.google.com/se/0/_/+1/fastbutton?usegapi=1&size=medium&origin=https%3A%2F%2Fwww.thegeekstuff.com&url=https%3A%2F%2Fwww.thegeekstuff.com%2F2012%2F10%2F15-linux-split-and-join-command-examples-to-manage-large-files%2F&gsrc=3p&jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.en_US.f5JujS1eFMY.O%2Fm%3D__features__%2Fam%3DQQE%2Frt%3Dj%2Fd%3D1%2Frs%3DAGLTcCNDI1_ftdVIpg6jNiygedEKTreQ2A#_methods=onPlusOne%2C_ready%2C_close%2C_open%2C_resizeMe%2C_renderstart%2Concircled%2Cdrefresh%2Cerefresh&id=I0_1529064634502&_gfid=I0_1529064634502&parent=https%3A%2F%2Fwww.thegeekstuff.com&pfname=&rpctoken=68750732

https://www.facebook.com/plugins/like.php?href=https%3A%2F%2Fwww.thegeekstuff.com%2F2012%2F10%2F15-linux-split-and-join-command-examples-to-manage-large-files%2F&send=false&layout=button_count&width=450&show_faces=false&action=like&colorscheme=light&font&height=21

https://platform.twitter.com/widgets/tweet_button.c5b006ac082bc92aa829181b9ce63af1.en.html#dnt=false&id=twitter-widget-0&lang=en&original_referer=https%3A%2F%2Fwww.thegeekstuff.com%2F2012%2F10%2F15-linux-split-and-join-command-examples-to-manage-large-files%2F&size=m&text=15%20Linux%20Split%20and%20Join%20Command%20Examples%20to%20Manage%20Large%20Files&time=1529064635577&type=share&url=https%3A%2F%2Fwww.thegeekstuff.com%2F2012%2F10%2F15-linux-split-and-join-command-examples-to-manage-large-files%2F

Linux split and join commands are very helpful when you are manipulating large files. This article explains how to use Linux split and join command with descriptive examples.

Join and split command syntax:

join [OPTION] FILE1 FILE2
split [OPTION] [INPUT [PREFIX]]

Linux Split Command Examples 1. Basic Split Example

Here is a basic example of split command.

$ split split.zip 

$ ls
split.zip  xab  xad  xaf  xah  xaj  xal  xan  xap  xar  xat  xav  xax  xaz  xbb  xbd  xbf  xbh  xbj  xbl  xbn
xaa        xac  xae  xag  xai  xak  xam  xao  xaq  xas  xau  xaw  xay  xba  xbc  xbe  xbg  xbi  xbk  xbm  xbo

So we see that the file split.zip was split into smaller files with x** as file names. Where ** is the two character suffix that is added by default. Also, by default each x** file would contain 1000 lines.

$ wc -l *
   40947 split.zip
    1000 xaa
    1000 xab
    1000 xac
    1000 xad
    1000 xae
    1000 xaf
    1000 xag
    1000 xah
    1000 xai
...
...
...

So the output above confirms that by default each x** file contains 1000 lines.

2.Change the Suffix Length using -a option

As discussed in example 1 above, the default suffix length is 2. But this can be changed by using -a option.

me name=

As you see in the following example, it is using suffix of length 5 on the split files.

$ split -a5 split.zip
$ ls
split.zip  xaaaac  xaaaaf  xaaaai  xaaaal  xaaaao  xaaaar  xaaaau  xaaaax  xaaaba  xaaabd  xaaabg  xaaabj  xaaabm
xaaaaa     xaaaad  xaaaag  xaaaaj  xaaaam  xaaaap  xaaaas  xaaaav  xaaaay  xaaabb  xaaabe  xaaabh  xaaabk  xaaabn
xaaaab     xaaaae  xaaaah  xaaaak  xaaaan  xaaaaq  xaaaat  xaaaaw  xaaaaz  xaaabc  xaaabf  xaaabi  xaaabl  xaaabo

Note: Earlier we also discussed about other file manipulation utilities – tac, rev, paste .

3.Customize Split File Size using -b option

Size of each output split file can be controlled using -b option.

In this example, the split files were created with a size of 200000 bytes.

$ split -b200000 split.zip 

$ ls -lart
total 21084
drwxrwxr-x 3 himanshu himanshu     4096 Sep 26 21:20 ..
-rw-rw-r-- 1 himanshu himanshu 10767315 Sep 26 21:21 split.zip
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xad
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xac
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xab
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xaa
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xah
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xag
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xaf
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xae
-rw-rw-r-- 1 himanshu himanshu   200000 Sep 26 21:35 xar
...
...
...
4. Create Split Files with Numeric Suffix using -d option

As seen in examples above, the output has the format of x** where ** are alphabets. You can change this to number using -d option.

Here is an example. This has numeric suffix on the split files.

$ split -d split.zip
$ ls
split.zip  x01  x03  x05  x07  x09  x11  x13  x15  x17  x19  x21  x23  x25  x27  x29  x31  x33  x35  x37  x39
x00        x02  x04  x06  x08  x10  x12  x14  x16  x18  x20  x22  x24  x26  x28  x30  x32  x34  x36  x38  x40
5. Customize the Number of Split Chunks using -C option

To get control over the number of chunks, use the -C option.

This example will create 50 chunks of split files.

$ split -n50 split.zip
$ ls
split.zip  xac  xaf  xai  xal  xao  xar  xau  xax  xba  xbd  xbg  xbj  xbm  xbp  xbs  xbv
xaa        xad  xag  xaj  xam  xap  xas  xav  xay  xbb  xbe  xbh  xbk  xbn  xbq  xbt  xbw
xab        xae  xah  xak  xan  xaq  xat  xaw  xaz  xbc  xbf  xbi  xbl  xbo  xbr  xbu  xbx
6. Avoid Zero Sized Chunks using -e option

While splitting a relatively small file in large number of chunks, its good to avoid zero sized chunks as they do not add any value. This can be done using -e option.

Here is an example:

$ split -n50 testfile

$ ls -lart x*
-rw-rw-r-- 1 himanshu himanshu 0 Sep 26 21:55 xag
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xaf
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xae
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xad
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xac
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xab
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:55 xaa
-rw-rw-r-- 1 himanshu himanshu 0 Sep 26 21:55 xbx
-rw-rw-r-- 1 himanshu himanshu 0 Sep 26 21:55 xbw
-rw-rw-r-- 1 himanshu himanshu 0 Sep 26 21:55 xbv
...
...
...

So we see that lots of zero size chunks were produced in the above output. Now, lets use -e option and see the results:

$ split -n50 -e testfile
$ ls
split.zip  testfile  xaa  xab  xac  xad  xae  xaf

$ ls -lart x*
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xaf
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xae
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xad
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xac
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xab
-rw-rw-r-- 1 himanshu himanshu 1 Sep 26 21:57 xaa

So we see that no zero sized chunk was produced in the above output.

7. Customize Number of Lines using -l option

Number of lines per output split file can be customized using the -l option.

As seen in the example below, split files are created with 20000 lines.

$ split -l20000 split.zip

$ ls
split.zip  testfile  xaa  xab  xac

$ wc -l x*
   20000 xaa
   20000 xab
     947 xac
   40947 total
Get Detailed Information using –verbose option

To get a diagnostic message each time a new split file is opened, use –verbose option as shown below.

$ split -l20000 --verbose split.zip
creating file `xaa'
creating file `xab'
creating file `xac'

[Nov 08, 2018] Utilizing multi core for tar+gzip-bzip compression-decompression

Nov 08, 2018 | stackoverflow.com

Ask Question up vote 163 down vote favorite 67


user1118764 , Sep 7, 2012 at 6:58

I normally compress using tar zcvf and decompress using tar zxvf (using gzip due to habit).

I've recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression.

Is there any way I can utilize the unused cores to make it faster?

Warren Severin , Nov 13, 2017 at 4:37

The solution proposed by Xiong Chiamiov above works beautifully. I had just backed up my laptop with .tar.bz2 and it took 132 minutes using only one cpu thread. Then I compiled and installed tar from source: gnu.org/software/tar I included the options mentioned in the configure step: ./configure --with-gzip=pigz --with-bzip2=lbzip2 --with-lzip=plzip I ran the backup again and it took only 32 minutes. That's better than 4X improvement! I watched the system monitor and it kept all 4 cpus (8 threads) flatlined at 100% the whole time. THAT is the best solution. – Warren Severin Nov 13 '17 at 4:37

Mark Adler , Sep 7, 2012 at 14:48

You can use pigz instead of gzip, which does gzip compression on multiple cores. Instead of using the -z option, you would pipe it through pigz:
tar cf - paths-to-archive | pigz > archive.tar.gz

By default, pigz uses the number of available cores, or eight if it could not query that. You can ask for more with -p n, e.g. -p 32. pigz has the same options as gzip, so you can request better compression with -9. E.g.

tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz

user788171 , Feb 20, 2013 at 12:43

How do you use pigz to decompress in the same fashion? Or does it only work for compression? – user788171 Feb 20 '13 at 12:43

Mark Adler , Feb 20, 2013 at 16:18

pigz does use multiple cores for decompression, but only with limited improvement over a single core. The deflate format does not lend itself to parallel decompression. The decompression portion must be done serially. The other cores for pigz decompression are used for reading, writing, and calculating the CRC. When compressing on the other hand, pigz gets close to a factor of n improvement with n cores. – Mark Adler Feb 20 '13 at 16:18

Garrett , Mar 1, 2014 at 7:26

The hyphen here is stdout (see this page ). – Garrett Mar 1 '14 at 7:26

Mark Adler , Jul 2, 2014 at 21:29

Yes. 100% compatible in both directions. – Mark Adler Jul 2 '14 at 21:29

Mark Adler , Apr 23, 2015 at 5:23

There is effectively no CPU time spent tarring, so it wouldn't help much. The tar format is just a copy of the input file with header blocks in between files. – Mark Adler Apr 23 '15 at 5:23

Jen , Jun 14, 2013 at 14:34

You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use.

For example use:

tar -c --use-compress-program=pigz -f tar.file dir_to_zip

ranman , Nov 13, 2013 at 10:01

This is an awesome little nugget of knowledge and deserves more upvotes. I had no idea this option even existed and I've read the man page a few times over the years. – ranman Nov 13 '13 at 10:01

Valerio Schiavoni , Aug 5, 2014 at 22:38

Unfortunately by doing so the concurrent feature of pigz is lost. You can see for yourself by executing that command and monitoring the load on each of the cores. – Valerio Schiavoni Aug 5 '14 at 22:38

bovender , Sep 18, 2015 at 10:14

@ValerioSchiavoni: Not here, I get full load on all 4 cores (Ubuntu 15.04 'Vivid'). – bovender Sep 18 '15 at 10:14

Valerio Schiavoni , Sep 28, 2015 at 23:41

On compress or on decompress ? – Valerio Schiavoni Sep 28 '15 at 23:41

Offenso , Jan 11, 2017 at 17:26

I prefer tar - dir_to_zip | pv | pigz > tar.file pv helps me estimate, you can skip it. But still it easier to write and remember. – Offenso Jan 11 '17 at 17:26

Maxim Suslov , Dec 18, 2014 at 7:31

Common approach

There is option for tar program:

-I, --use-compress-program PROG
      filter through PROG (must accept -d)

You can use multithread version of archiver or compressor utility.

Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:

$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):

$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz

Input and output of singlethread and multithread are compatible. You can compress using multithread version and decompress using singlethread version and vice versa.

p7zip

For p7zip for compression you need a small shell script like the following:

#!/bin/sh
case $1 in
  -d) 7za -txz -si -so e;;
   *) 7za -txz -si -so a .;;
esac 2>/dev/null

Save it as 7zhelper.sh. Here the example of usage:

$ tar -I 7zhelper.sh -cf OUTPUT_FILE.tar.7z paths_to_archive
$ tar -I 7zhelper.sh -xf OUTPUT_FILE.tar.7z
xz

Regarding multithreaded XZ support. If you are running version 5.2.0 or above of XZ Utils, you can utilize multiple cores for compression by setting -T or --threads to an appropriate value via the environmental variable XZ_DEFAULTS (e.g. XZ_DEFAULTS="-T 0" ).

This is a fragment of man for 5.1.0alpha version:

Multithreaded compression and decompression are not implemented yet, so this option has no effect for now.

However this will not work for decompression of files that haven't also been compressed with threading enabled. From man for version 5.2.2:

Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't even if --block-size=size is used.

Recompiling with replacement

If you build tar from sources, then you can recompile with parameters

--with-gzip=pigz
--with-bzip2=lbzip2
--with-lzip=plzip

After recompiling tar with these options you can check the output of tar's help:

$ tar --help | grep "lbzip2\|plzip\|pigz"
  -j, --bzip2                filter the archive through lbzip2
      --lzip                 filter the archive through plzip
  -z, --gzip, --gunzip, --ungzip   filter the archive through pigz

> , Apr 28, 2015 at 20:41

This is indeed the best answer. I'll definitely rebuild my tar! – user1985657 Apr 28 '15 at 20:41

mpibzip2 , Apr 28, 2015 at 20:57

I just found pbzip2 and mpibzip2 . mpibzip2 looks very promising for clusters or if you have a laptop and a multicore desktop computer for instance. – user1985657 Apr 28 '15 at 20:57

oᴉɹǝɥɔ , Jun 10, 2015 at 17:39

This is a great and elaborate answer. It may be good to mention that multithreaded compression (e.g. with pigz ) is only enabled when it reads from the file. Processing STDIN may in fact be slower. – oᴉɹǝɥɔ Jun 10 '15 at 17:39

selurvedu , May 26, 2016 at 22:13

Plus 1 for xz option. It the simplest, yet effective approach. – selurvedu May 26 '16 at 22:13

panticz.de , Sep 1, 2014 at 15:02

You can use the shortcut -I for tar's --use-compress-program switch, and invoke pbzip2 for bzip2 compression on multiple cores:
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 DIRECTORY_TO_COMPRESS/

einpoklum , Feb 11, 2017 at 15:59

A nice TL;DR for @MaximSuslov's answer . – einpoklum Feb 11 '17 at 15:59

,

If you want to have more flexibility with filenames and compression options, you can use:
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec \
tar -P --transform='s@/my/path/@@g' -cf - {} + | \
pigz -9 -p 4 > myarchive.tar.gz
Step 1: find

find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec

This command will look for the files you want to archive, in this case /my/path/*.sql and /my/path/*.log . Add as many -o -name "pattern" as you want.

-exec will execute the next command using the results of find : tar

Step 2: tar

tar -P --transform='s@/my/path/@@g' -cf - {} +

--transform is a simple string replacement parameter. It will strip the path of the files from the archive so the tarball's root becomes the current directory when extracting. Note that you can't use -C option to change directory as you'll lose benefits of find : all files of the directory would be included.

-P tells tar to use absolute paths, so it doesn't trigger the warning "Removing leading `/' from member names". Leading '/' with be removed by --transform anyway.

-cf - tells tar to use the tarball name we'll specify later

{} + uses everyfiles that find found previously

Step 3: pigz

pigz -9 -p 4

Use as many parameters as you want. In this case -9 is the compression level and -p 4 is the number of cores dedicated to compression. If you run this on a heavy loaded webserver, you probably don't want to use all available cores.

Step 4: archive name

> myarchive.tar.gz

Finally.

[Nov 08, 2018] Technology Detox The Health Benefits of Unplugging Unwinding by Sara Tipton

Notable quotes:
"... Another great tip is to buy one of those old-school alarm clocks so the smartphone isn't ever in your bedroom. ..."
Nov 07, 2018 | www.zerohedge.com

Authored by Sara Tipton via ReadyNutrition.com,

Recent studies have shown that 90% of Americans use digital devices for two or more hours each day and the average American spends more time a day on high-tech devices than they do sleeping: 8 hours and 21 minutes to be exact. If you've ever considered attempting a "digital detox", there are some health benefits to making that change and a few tips to make things a little easier on yourself.

Many Americans are on their phones rather than playing with their children or spending quality family time together. Some people give up technology, or certain aspects of it, such as social media for varying reasons, and there are some shockingly terrific health benefits that come along with that type of a detox from technology. In fact, more and more health experts and medical professionals are suggesting a periodic digital detox; an extended period without those technology gadgets. Studies continue to show that a digital detox, has proven to be beneficial for relationships, productivity, physical health, and mental health. If you find yourself overly stressed or unproductive or generally disengaged from those closest to you, it might be time to unplug.

DIGITAL ADDICTION RESOLUTION

It may go unnoticed but there are many who are actually addicted to their smartphones or tablet. It could be social media or YouTube videos, but these are the people who never step away. They are the ones with their face in their phone while out to dinner with their family. They can't have a quiet dinner without their phone on the table. We've seen them at the grocery store aimlessly pushing around a cart while ignoring their children and scrolling on their phone. A whopping 83% of American teenagers claim to play video games while other people are in the same room and 92% of teens report to going online daily . 24% of those users access the internet via laptops, tablets, and mobile devices.

Addiction therapists who treat gadget-obsessed people say their patients aren't that different from other kinds of addicts. Whereas alcohol, tobacco, and drugs involve a substance that a user's body gets addicted to, in behavioral addiction, it's the mind's craving to turn to the smartphone or the Internet. Taking a break teaches us that we can live without constant stimulation, and lessens our dependence on electronics. Trust us: that Facebook message with a funny meme attached or juicy tidbit of gossip can wait.

IMPROVE RELATIONSHIPS AND BE MORE PERSONABLE

Another benefit to keeping all your electronics off is that it will allow you to establish good mannerisms and people skills and build your relationships to a strong level of connection. If you have ever sat across someone at the dinner table who made more phone contact than eye contact, you know it feels to take a backseat to a screen. Cell phones and other gadgets force people to look down and away from their surroundings, giving them a closed off and inaccessible (and often rude) demeanor. A digital detox has the potential of forcing you out of that unhealthy comfort zone. It could be a start toward rebuilding a struggling relationship too. In a Forbes study , 3 out of 5 people claimed that they spend more time on their digital devices than they do with their partners. This can pose a real threat to building and maintaining real-life relationships. The next time you find yourself going out on a dinner date, try leaving your cell phone and other devices at home and actually have a conversation. Your significant other will thank you.

BETTER SLEEP AND HEALTHIER EATING HABITS

The sleep interference caused by these high-tech gadgets is another mental health concern. The stimulation caused by artificial light can make you feel more awake than you really are, which can potentially interfere with your sleep quality. It is recommended that you give yourself at least two hours of technology-free time before bedtime. The "blue light" has been shown to interfere with sleeping patterns by inhibiting melatonin (the hormone which controls our sleep/wake cycle known as circadian rhythm) production. Try shutting off your phone after dinner and leaving it in a room other than your bedroom. Another great tip is to buy one of those old-school alarm clocks so the smartphone isn't ever in your bedroom. This will help your body readjust to a normal and healthy sleep schedule.

Your eating habits can also suffer if you spend too much time checking your newsfeed. The Rochester Institute of Technology released a study that revealed students are more likely to eat while staring into digital media than they are to eat at a dinner table. This means that eating has now become a multi-tasking activity, rather than a social and loving experience in which healthy foods meant to sustain the body are consumed. This can prevent students from eating consciously, which promotes unhealthy eating habits such as overeating and easy choices, such as a bag of chips as opposed to washing and peeling some carrots. Whether you're an overworked college student checking your Facebook, or a single bachelor watching reruns of The Office , a digital detox is a great way to promote healthy and conscious eating.

IMPROVE OVERALL MENTAL HEALTH

Social media addicts experience a wide array of emotions when looking at the photos of Instagram models and the exercise regimes of others who live in exotic locations. These emotions can be mentally draining and psychologically unhealthy and lead to depression. Smartphone use has been linked to loneliness, shyness, and less engagement at work. In other words, one may have many "social media friends" while being lonely and unsatisfied because those friends are only accessible through their screen. Start by limiting your time on social media. Log out of all social media accounts. That way, you've actually got to log back in if you want to see what that Parisian Instagram vegan model is up to.

If you feel like a detox is in order but don't know how to go about it, start off small. Try shutting off your phone after dinner and don't turn it back on until after breakfast. Keep your phone in another room besides your bedroom overnight. If you use your phone as an alarm clock, buy a cheap alarm clock to use instead to lessen your dependence on your phone. Boredom is often the biggest factor in the beginning stages of a detox, but try playing an undistracted board game with your children, leaving your phone at home during a nice dinner out, or playing with a pet. All of these things are not only good for you but good for your family and beloved furry critter as well!

[Nov 07, 2018] Stuxnet 2.0? Iran claims Israel launched new cyber attacks

Nov 07, 2018 | arstechnica.com

President Rouhani's phone "bugged," attacks against network infrastructure claimed.

Sean Gallagher - 11/5/2018, 5:10 PM

reader comments

Last week, Iran's chief of civil defense claimed that the Iranian government had fought off Israeli attempts to infect computer systems with what he described as a new version of Stuxnet -- the malware reportedly developed jointly by the US and Israel that targeted Iran's uranium-enrichment program. Gholamreza Jalali, chief of the National Passive Defense Organization (NPDO), told Iran's IRNA news service, "Recently, we discovered a new generation of Stuxnet which consisted of several parts... and was trying to enter our systems."

On November 5, Iran Telecommunications Minister Mohammad-Javad Azari Jahromi accused Israel of being behind the attack, and he said that the malware was intended to "harm the country's communication infrastructures." Jahromi praised "technical teams" for shutting down the attack, saying that the attackers "returned empty-handed." A report from Iran's Tasnim news agency quoted Deputy Telecommunications Minister Hamid Fattahi as stating that more details of the cyber attacks would be made public soon.

Jahromi said that Iran would sue Israel over the attack through the International Court of Justice. The Iranian government has also said it would sue the US in the ICJ over the reinstatement of sanctions. Israel has remained silent regarding the accusations .

The claims come a week after the NPDO's Jalali announced that President Hassan Rouhani's cell phone had been "tapped" and was being replaced with a new, more secure device. This led to a statement by Iranian Supreme Leader Ayatollah Ali Khamenei, exhorting Iran's security apparatus to "confront infiltration through scientific, accurate, and up-to-date action."

While Iran protests the alleged attacks -- about which the Israeli government has been silent -- Iranian hackers have continued to conduct their own cyber attacks. A recent report from security tools company Carbon Black based on data from the company's incident-response partners found that Iran had been a significant source of attacks in the third quarter of this year, with one incident-response professional noting, "We've seen a lot of destructive actions from Iran and North Korea lately, where they've effectively wiped machines they suspect of being forensically analyzed."


SymmetricChaos </> , 2018-11-05T17:16:46-05:00 I feel like governments still think of cyber warfare as something that doesn't really count and are willing to be dangerously provocative in their use of it. ihatewinter , 2018-11-05T17:27:06-05:00 Another day in international politics. Beats lobbing bombs at each other. +13 ( +16 / -3 ) fahrenheit_ak </> , 2018-11-05T17:46:44-05:00

corey_1967 wrote:
The twin pillars of Iran's foreign policy - America is evil and Wipe Israel off the map - do not appear to be serving the country very well.

They serve Iran very well, America is an easy target to gather support against, and Israel is more than willing to play the bad guy (for a bunch of reasons including Israels' policy of nuclear hegemony in the region and historical antagonism against Arab states).
revision0 , 2018-11-05T17:48:22-05:00 Israeli hackers?

Go on!

Quote:

Israeli hackers offered Cambridge Analytica, the data collection firm that worked on U.S. President Donald Trump's election campaign, material on two politicians who are heads of state, the Guardian reported Wednesday, citing witnesses.

https://www.haaretz.com/israel-news/isr ... -1.5933977

Quote:

For $20M, These Israeli Hackers Will Spy On Any Phone On The Planet

https://www.forbes.com/sites/thomasbrew ... -ulin-ss7/

Quote:

While Israelis are not necessarily number one in technical skills -- that award goes to Russian hackers -- Israelis are probably the best at thinking on their feet and adjusting to changing situations on the fly, a trait essential for success in a wide range of areas, including cyber-security, said Forzieri. "In modern attacks, the human factor -- for example, getting someone to click on a link that will install malware -- constitutes as much as 85% of a successful attack," he said.

http://www.timesofisrael.com/israeli-ha ... ty-expert/

+5 ( +9 / -4 )
ihatewinter </> , 2018-11-05T17:52:15-05:00
dramamoose wrote:
thorpe wrote:
The pro-Israel trolls out in front of this comment section...

You don't have to be pro-Israel to be anti-Iran. Far from it. I think many of Israel's actions in Palestine are reprehensible, but I also know to (rightly) fear an Islamic dictatorship who is actively funding terrorism groups and is likely a few years away from having a working nuclear bomb, should they resume research (which the US actions seem likely to cause).

The US created the Islamic Republic of Iran by holding a cruel dictator in power rather than risking a slide into communism. We should be engaging diplomatically, rather than trying sanctions which clearly don't work. But I don't think that the original Stuxnet was a bad idea, nor do I think that intense surveillance of what could be a potentially very dangerous country is a bad one either.

If the Israelis (slash US) did in fact target civilian infrastructure, that's a problem. Unless, of course, they were bugging them for espionage purposes.

Agree. While Israel is not about to win Humanitarian Nation of the year Award any time soon, I don't see it going to Iran in a close vote tally either.

[Nov 05, 2018] Frequently is no way to judge whether individual is competent or incompetent to hold a given position. Stated another way: there is no adequate competence criterion for technical managers.

Nov 05, 2018 | www.rako.com

However, there is another anomaly with more interesting consequences; namely, there frequently is no way to judge whether individual is competent or incompetent to hold a given position. Stated another way: there is no adequate competence criterion for technical managers.

Consider. for example. the manager of a small group of chemists. He asked his group to develop a nonfading system of dyes using complex organic compounds that they had been studying for some time. Eighteen months later they reported little success with dyes but had discovered a new substance that was rather effective as an insect repellent.

Should the manager be chastised for failing to accomplish anything toward his original objective, or should he be praised for resourcefulness
in finding something useful in the new chemical system? Was 18 months a long time or a short time for this accomplishment?

[Nov 05, 2018] Management theories for CIOs The Peter Principle and Parkinson's Law

Notable quotes:
"... Josι Ortega y Gasset. ..."
"... "Works expands so as to fill the time available for its completion." ..."
"... "The time spent on any item of the agenda will be in inverse proportion to the sum of money involved." ..."
"... Gφdel, Escher, Bach: An Eternal Golden Braid, ..."
"... "It always takes longer than you expect, even when you take into account Hofstadter's Law." ..."
"... "Anything that can go wrong, will go wrong." ..."
"... "Anything that can go wrong, will go wrong - at the worst possible moment." ..."
Nov 05, 2018 | cio.co.uk

From the semi-serious to the confusingly ironic, the business world is not short of pseudo-scientific principles, laws and management theories concerning how organisations and their leaders should and should not behave. CIO UK takes a look at some sincere, irreverent and leftfield management concepts that are relevant to CIOs and all business leaders.

The Peter Principle

A concept formulated by Laurence J Peter in 1969, the Peter Principle runs that in a hierarchical structure, employees are promoted to their highest level of incompetence at which point they are no longer able to fulfil an effective role for their organisation.

In the Peter Principle people are promoted when they excel, but this process falls down when they are unlikely to gain further promotion or be demoted with the logical end point, according to Peter, where "every post tends to be occupied by an employee who is incompetent to carry out its duties" and that "work is accomplished by those employees who have not yet reached their level of incompetence".

To counter the Peter Principle leaders could seek the advice of Spanish liberal philosopher Josι Ortega y Gasset. While he died 14 years before the Peter Principle was published, Ortega had been in exile in Argentina during the Spanish Civil War and prompted by his observations in South America had quipped: "All public employees should be demoted to their immediately lower level, as they have been promoted until turning incompetent."

Parkinson's Law

Cyril Northcote Parkinson's eponymous law, derived from his extensive experience in the British Civil Service, states that: "Works expands so as to fill the time available for its completion."

The first sentence of a humorous essay published in The Economist in 1955, Parkinson's Law is familiar with CIOs, IT teams, journalists, students, and every other occupation that can learn from Parkinson's mocking of pubic administration in the UK. The corollary law most applicable to CIOs runs that "data expands to fill the space available for storage", while Parkinson's broader work about the self-satisfying uncontrolled growth of bureaucratic apparatus is as relevant for the scaling startup as it is to the large corporate.

Related Parkinson's Law of Triviality

Flirting with the ground between flippancy and seriousness, Parkinson argued that boards and members of an organisation give disproportional weight to trivial issues and those that are easiest to grasp for non-experts. In his words: "The time spent on any item of the agenda will be in inverse proportion to the sum of money involved."

Parkinson's anecdote is of a fictional finance committee's three-item agenda to cover a £10 million contract discussing the components of a new nuclear reactor, a proposal to build a new £350 bicycle shed, and finally which coffee and biscuits should be supplied at future committee meetings. While the first item on the agenda is far too complex and ironed out in two and a half minutes, 45 minutes is spent discussing bike sheds, and debates about the £21 refreshment provisions are so drawn out that the committee runs over its two-hour time allocation with a note to provide further information about coffee and biscuits to be continued at the next meeting.

The Dilbert Principle

Referring to a 1990s theory by popular Dilbert cartoonist Scott Adams, the Dilbert Principle runs that companies tend to promote their least competent employees to management roles to curb the amount of damage they are capable of doing to the organisation.

Unlike the Peter Principle , which is positive in its aims by rewarding competence, the Dilbert Principle assumes people are moved to quasi-senior supervisory positions in a structure where they are less likely to have an effect on productive output of the company which is performed by those lower down the ladder.

Hofstadter's Law

Coined by Douglas Hofstadter in his 1979 book Gφdel, Escher, Bach: An Eternal Golden Braid, Hofstadter's Law states: "It always takes longer than you expect, even when you take into account Hofstadter's Law."

Particularly relevant to CIOs and business leaders overseeing large projects and transformation programmes, Hofstadter's Law suggests that even appreciating your own subjective pessimism in your projected timelines, they are still worth re-evaluating.

Related Murphy's Law

"Anything that can go wrong, will go wrong."

An old adage and without basis in any scientific laws or management principles, Murphy's Law is always worth bearing in mind for CIOs or when undertaking thorough scenario planning for adverse situations. It's also perhaps worth bearing in mind the corollary principle Finagle's Law , which states: "Anything that can go wrong, will go wrong - at the worst possible moment."

Lindy Effect

Concerning the life expectancy of non-perishable things, the Lindy Effect is as relevant to CIOs procuring new technologies or maintaining legacy infrastructure as it is to the those buying homes, used cars, a fountain pen or mobile phone.

Harder to define than other principles and laws, the Lindy Effect suggests that mortality rate decreases with time, unlike in nature and in human beings where - after childhood - mortality rate increases with time. Ergo, every day of server uptime implies a longer remaining life expectancy.

A corollary effect related to the Lindy Effect which is a good explanation is the Copernican Principle , which states that the future life expectancy is equal to the current age, i.e. that barring any addition evidence on the contrary, something must be halfway through its life span.

The Lindy Effect and the idea that older things are more robust has specific relevance to CIOs beyond servers and IT infrastructure with its association with source code, where newer code will in general have lower probability of remaining within a year and an increased likelihood of causing problems compared to code written a long time ago, and in project management where the lifecycle of a project grows and its scope changes, an Agile methodology can be used to mitigate project risks and fix mistakes.

The Jevons Paradox

Wikipedia offers the best economic description of the Jevons Paradox or Jevons effect, in which a technological progress increases efficiency with which a resource is used, but the rate of consumption of that resource subsequently rises because of increasing demand.

Think email, think Slack, instant messaging, printing, how easy it is to create Excel reports, coffee-making, conference calls, network and internet speeds, the list is endless. If you suspect demand in these has increased along with technological advancement negating the positive impact of said efficiency gains in the first instance, sounds like the paradox first described by William Stanley Jevons in 1865 when observing coal consumption following the introduction of the Watt steam engine.

Ninety-Ninety Rule

A light-hearted quip bespoke to computer programming and software development, the Ninety-Ninety Rule states that: "The first 90% of the code accounts for the first 90% of the development time. The remaining 10% of the code accounts for the other 90% of the development time." See also, Hofstadter's Law .

Related to this is the Pareto Principle , or the 80-20 Rule, and how it relates to software, with supporting anecdotes that "20% of the code has 80% of the errors" or in load testing that it is common practice to estimate that 80% of the traffic occurs during 20% of the time.

Pygmalion Effect and Golem Effect

Named after the Greek myth of Pygmalion, a sculptor who fell in love with a statue he carved, and relevant to managers across industry and seniority, the Pygmalion Effect runs that higher expectations lead to an increased performance.

Counter to the Pygmalion Effect is the Golem effect , whereby low expectations result in a decrease in performance.

Dunning-Kruger Effect

The Dunning-Kruger Effect , named after two psychologists from Cornell University, states that incompetent people are significantly less able to recognise their own lack of skill, the extent of their inadequacy, and even to gauge the skill of others. Furthermore, they are only able to acknowledge their own incompetence after they have been exposed to training in that skill.

At a loss to find a better visual representation of the Dunning-Kruger Effect , here is Simon Wardley's graph with Knowledge and Expertise axes - a warning as to why self-professed experts are the worst people to listen to on a given subject.

me title=

See also this picture of AOL "Digital Prophet" David Shing and web developer Sir Tim Berners-Lee.

[Nov 05, 2018] Putt's Law

Nov 05, 2018 | davewentzel.com

... ... ...

Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand. --Putt's Law

If you are in IT and are not familiar with Archibald Putt, I suggest you stop reading this blog post, RIGHT NOW, and go buy the book Putt's Law and the Successful Technocrat. How to Win in the Information Age . Putt's Law , for short, is a combination of Dilbert and The Mythical Man-Month . It shows you exactly how managers of technologists think, how they got to where they are, and how they stay there. Just like Dilbert, you'll initially laugh, then you'll cry, because you'll realize just how true Putt's Law really is. But, unlike Dilbert, whose technologist-fans tend to have a revulsion for management, Putt tries to show the technologist how to become one of the despised. Now granted, not all of us technologists have a desire to be management, it is still useful to "know one's enemy."

Two amazing facts:

  1. Archibald Putt is a pseudonym and his true identity has yet to be revealed. A true "Deep Throat" for us IT guys.
  2. Putt's Law was written back in 1981. It amazes me how the Old IT Classics (Putt's Law, Mythical Man-Month, anything by Knuth) are even more relevant today than ever.

Every technical hierarchy, in time, develops a competence inversion. --Putt's Corollary

Putt's Corollary says that in a corporate technocracy, the more technically competent people will remain in charge of the technology, whereas the less competent will be promoted to management. That sounds a lot like The Peter Principle (another timeless classic written in 1969).

People rise to their level of incompetence. --Dave's Summary of the Peter Principle

I can tell you that managers have the least information about technical issues and they should be the last people making technical decisions. Period. I've often heard that managers are used as the arbiters of technical debates. Bad idea. Arbiters should always be the [[benevolent dictators]] (the most admired/revered technologist you have). The exception is when your manager is also your benevolent dictator, which is rare. Few humans have the capability, or time, for both.

I see more and more hit-and-run managers where I work. They feel as though they are the technical decision-makers. They attend technical meetings they were not invited to. Then they ask pointless, irrelevant questions that suck the energy out of the team. Then they want status updates hourly. Eventually after they have totally derailed the process they move along to some other, sexier problem with more management visibility.

I really admire managers who follow the MBWA ( management by walking around ) principle. This management philosophy is very simple...the best managers are those who leave their offices and observe. By observing they learn what the challenges are for their teams and how to help them better.

So, what I am looking for in a manager

  1. He knows he is the least qualified person to make a technical decision.
  2. He is a facilitator. He knows how to help his technologists succeed.
  3. MBWA

[Nov 05, 2018] Why the Peter Principle Works

Notable quotes:
"... The Corner Office ..."
Aug 15, 2011 | www.cbsnews.com
Why The Peter Principle Works Everyone's heard of the Peter Principle - that employees tend to rise to their level of incompetence - a concept that walks that all-too-fine line between humor and reality.

We've all seen it in action more times than we'd like. Ironically, some percentage of you will almost certainly be promoted to a position where you're no longer effective. For some of you, that's already happened. Sobering thought.

Well, here's the thing. Not only is the Peter Principle alive and well in corporate America, but contrary to popular wisdom, it's actually necessary for a healthy capitalist system. That's right, you heard it here, folks, incompetence is a good thing. Here's why.

Robert Browning once said, "A man's reach should exceed his grasp." It's a powerful statement that means you should seek to improve your situation, strive to go above and beyond. Not only is that an embodiment of capitalism, but it also leads directly to the Peter Principle because, well, how do you know when to quit?

Now, most of us don't perpetually reach for the stars, but until there's clear evidence that we're not doing ourselves or anyone else any good, we're bound to keep right on reaching. After all, objectivity is notoriously difficult when opportunities for a better life are staring you right in the face.

I mean, who turns down promotions? Who doesn't strive to reach that next rung on the ladder? When you get an email from an executive recruiter about a VP or CEO job, are you likely to respond, "Sorry, I think that may be beyond my competency" when you've got to send two kids to college and you may actually want to retire someday?

Wasn't America founded by people who wanted a better life for themselves and their children? God knows, there were plenty of indications that they shouldn't take the plunge and, if they did, wouldn't succeed. That's called a challenge and, well, do you ever really know if you've reached too far until after the fact?

Perhaps the most interesting embodiment of all this is the way people feel about CEOs. Some think pretty much anyone can do a CEO's job for a fraction of the compensation. Seriously, you hear that sort of thing a lot, especially these days with class warfare being the rage and all.

One The Corner Office reader asked straight out in an email: "Would you agree that, in most cases, the company could fire the CEO and hire someone young, smart, and hungry at 1/10 the salary/perks/bonuses who would achieve the same performance?"

Sure, it's easy: you just set the direction, hire a bunch of really smart executives, then get out of the way and let them do their jobs. Once in a blue moon you swoop in, deal with a problem, then return to your ivory tower. Simple.

Well, not exactly.

You see, I sort of grew up at Texas Instruments in the 80s when the company was nearly run into the ground by Mark Shepherd and J. Fred Bucy - two CEOs who never should have gotten that far in their careers.

But the company's board, in its wisdom, promoted Jerry Junkins and, after his untimely death, Tom Engibous , to the CEO post. Not only were those guys competent, they revived the company and transformed it into what it is today.

I've seen what a strong CEO can do for a company, its customers, its shareholders, and its employees. I've also seen the destruction the Peter Principle can bring to those same stakeholders. But, even now, after 30 years of corporate and consulting experience, the one thing I've never seen is a CEO or executive with an easy job.

That's because there's no such thing. And to think you can eliminate incompetency from the executive ranks when it exists at every organizational level is, to be blunt, childlike or Utopian thinking. It's silly and trite. It doesn't even make sense.

It's not as if TI's board knew ahead of time that Shepherd and Bucy weren't the right guys for the job. They'd both had long, successful careers at the company. But the board did right the ship in time. And that's the mark of a healthy system at work.

The other day I read a truly fantastic story in Fortune about the rise and fall of Jeffrey Kindler as CEO of troubled pharmaceutical giant Pfizer . I remember when he suddenly stepped down amidst all sorts of rumor and conjecture about the underlying causes of the shocking news.

What really happened is the guy had a fabulous career as a litigator, climbed the corporate ladder to general ounsel of McDonald's and then Pfizer, had some limited success in operations, and once he was promoted to CEO, flamed out. Not because he was incompetent - he wasn't. And certainly not because he was a dysfunctional, antagonistic, micromanaging control freak - he was.

He failed because it was a really tough job and he was in over his head. It happens. It happens a lot. After all, this wasn't just some everyday company that's simple to run. This was Pfizer - a pharmaceutical giant with its top products going generic and a dried-up drug pipeline in need of a major overhaul.

The guy couldn't handle it. And when executives with issues get in over their heads, their issues become their undoing. It comes as no surprise that folks at McDonald's were surprised at the way he flamed out at Pfizer. That was a whole different ballgame.

Now, I bet those same people who think a CEO's job is a piece of cake will have a similar response to the Kindler situation at Pfizer. Why take the job if he knew he couldn't handle it? The board should have canned him before it got to that point. Why didn't the guy's executives speak up sooner?

Because, just like at TI, nobody knows ahead of time if people are going to be effective on the next rung of the ladder. Every situation is unique and there are no questions or test that will foretell the future. I mean, it's not as if King Solomon comes along and writes who the right guy for the job is on the wall.

The Peter Principle works because, in a capitalist system, there are top performers, abysmal failures, and everything in between. Expecting anything different when people must reach for the stars to achieve growth and success so our children have a better life than ours isn't how it works in the real world.

The Peter Principle works because it's the yin to Browning's yang, the natural outcome of striving to better our lives. Want to know how to bring down a free market capitalist system? Don't take the promotion because you're afraid to fail.

[Nov 05, 2018] Putt's Law, Peter Principle, Dilbert Principle of Incompetence Parkinson's Law

Nov 05, 2018 | asmilingassasin.blogspot.com

Putt's Law, Peter Principle, Dilbert Principle of Incompetence & Parkinson's Law

June 10, 2015 Putt's Law, Peter Principle, Dilbert Principle of Incompetence & Parkinson's Law I am a big fan of Scott Adams & Dilbert Comic Series. I realize that these laws and principles - the Putt's law, Peter Principle, the Dilbert Principle, and Parkinson's Law - aren't necessarily founded in reality. It's easy to look at a manager's closed doors and wonder he or she does all day, if anything. But having said that and having come to realize the difficulty and scope of what management entails. It's hard work and requires a certain skill-set that I'm only beginning to develop. One should therefore look at these principles and laws with an acknowledgment that they most likely developed from the employee's perspective, not the manager's. Take with a pinch of salt!
Source: Google Images
The Putt's law: · Putt's Law: " Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand. " · Putt's Corollary: " Every technical hierarchy, in time, develops a competence inversion. " with incompetence being "flushed out of the lower levels" of a technocratic hierarchy, ensuring that technically competent people remain directly in charge of the actual technology while those without technical competence move into management. The Peter Principle: The Peter Principle states that " in a hierarchy every employee tends to rise to his level of incompetence." In other words, employees who perform their roles with competence are promoted into successively higher levels until they reach a level at which they are no longer competent. There they remain. For example, let's say you are a brilliant programmer. You spend your days coding with amazing efficiency and prowess. After a couple of years, you're promoted to lead programmer, and then promoted to team manager. You may have no interest in managing other programmers, but it's the reward for your competence. There you sit -- you have risen to a level of incompetence. Your technical skills lie dormant while you fill your day with one-on-one meetings, department strategy meetings, planning meetings, budgets, and reports. The Dilbert Principle The principle states that companies tend to promote the most incompetent employees to management as a form of damage control . The principle argues that leaders, specifically those in middle management, are in reality the ones that have little effect on productivity. In order to limit the harm caused by incompetent employees who are actually not doing the work, companies make them leaders. The Dilbert Principle assumes that "the majority of real, productive work in a company is done by people lower in the power ladder." Those in management don't actually do anything to move forward the work. How it happens? The Incompetent Leader Stereotype often hits new leaders, specifically those who have no prior experience in a particular field. Often times, leaders who have been transferred from other departments are viewed as mere figureheads, rather than actual leaders who have knowledge of the work situation. Failure to prove technical capability can also lead to a leader being branded incompetent. Why it's bad? Being a victim of the incompetent leader stereotype is bad. Firstly, no one takes you seriously. Your ability to insert input into projects is hampered when your followers actively disregard anything you say as fluff. This is especially true if you are in middle management, where your power as a leader is limited. Secondly, your chances of rising ranks are curtailed. If viewed as an incompetent leader by your followers, your superiors are unlikely to entrust you with further projects which have more impact. How to get over it Know when to concede. As a leader, no one expects you to be competent in every area; though basic knowledge of every section you are leading is necessary. Readily admitting incompetency in certain areas will take out the impact out of it when others paint you as incompetent. Prove competency somewhere. Quickly establish yourself as having some purpose in the workplace, rather than being a mere picture of tokenism. This can be done by personally involving yourself in certain projects. Parkinson's Law Parkinson's Law states that " work expands so as to fill the time available for its completion ." Although this law has application with procrastination, storage capacity, and resource usage, Parkinson focuses his law on Corporate lethargy. Parkinson says that lethargy swell for two reasons: (1) "A manager wants to multiply subordinates, not rivals" and (2) "Managers make work for each other." In other words, a team size may swell not because the workload increases, but because they have the capacity and resources that allow for an increased workload even if the workload does not in fact increase. People without any work find ways to increase the amount of "work" and therefore add to the size of their lethargy. My Analysis I know none of these principles or laws gives much credit to management. The wrong person fills the wrong role, the role exists only to minimize damage control, or the role swells unnecessarily simply because it can. I find the whole topic of management somewhat fascinating, not because I think these theories apply to my own managers. These management theories are however relevant. Software coders looking to leverage coding talent for their projects often find themselves in management roles, without a strong understanding of how to manage people. Most of the time, these coders fail to engage. The project leaders are usually brilliant at their technical job but don't excel at management.
However the key principle to follow should be this: put individuals to work in their core competencies . It makes little sense to take your most brilliant engineer and have him or her manage people and budgets. Likewise, it makes no sense to take a shrewd consultant, one who can negotiate projects and requirements down to the minutest detail, and put that individual into a role involving creative design and content generation. However, to implement this model, you have to allow for reward without a dramatic change in job responsibilities or skills.

[Nov 04, 2018] Archibald Putt The Unknown Technocrat Returns - IEEE Spectrum

Nov 04, 2018 | spectrum.ieee.org

While similar things can, and do, occur in large technical hierarchies, incompetent technical people experience a social pressure from their more competent colleagues that causes them to seek security within the ranks of management. In technical hierarchies, there is always the possibility that incompetence will be rewarded by promotion.

Other Putt laws we love include the law of failure: "Innovative organizations abhor little failures but reward big ones." And the first law of invention: "An innovated success is as good as a successful innovation."

Now Putt has revised and updated his short, smart book, to be released in a new edition by Wiley-IEEE Press ( http://www.wiley.com/ieee ) at the end of this month. There have been murmurings that Putt's identity, the subject of much rumormongering, will be revealed after the book comes out, but we think that's unlikely. How much more interesting it is to have an anonymous chronicler wandering the halls of the tech industry, codifying its unstated, sometimes bizarre, and yet remarkably consistent rules of behavior.

This is management writing the way it ought to be. Think Dilbert , but with a very big brain. Read it and weep. Or laugh, depending on your current job situation.

[Nov 04, 2018] Two Minutes on Hiring by Eric Samuelson

Notable quotes:
"... Eric Samuelson is the creator of the Confident Hiring System™. Working with Dave Anderson of Learn to Lead, he provides the Anderson Profiles and related services to clients in the automotive retail industry as well as a variety of other businesses. ..."
Nov 04, 2018 | www.andersonprofiles.com

In 1981, an author in the Research and Development field, writing under the pseudonym Archibald Putt, penned this famous quote, now known as Putt's Law:

"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand."

Have you ever hired someone without knowing for sure if they can do the job? Have you promoted a good salesperson to management only to realize you made a dire mistake? The qualities needed to succeed in a technical field are quite different than for a leader.

The legendary immigrant engineer Charles Steinmetz worked at General Electric in the early 1900s. He made phenomenal advancements in the field of electric motors. His work was instrumental to the growth of the electric power industry. With a goal of rewarding him, GE promoted him to a management position, but he failed miserably. Realizing their error, and not wanting to offend this genius, GE's leadership retitled him as a Chief Engineer, with no supervisory duties, and let him go back to his research.

Avoid the double disaster of losing a good worker by promoting him to management failure. By using the unique Anderson Position Overlay system, you can avoid future regret by comparing your candidate's qualities to the requirements of the position before saying "Welcome Aboard".

Eric Samuelson is the creator of the Confident Hiring System™. Working with Dave Anderson of Learn to Lead, he provides the Anderson Profiles and related services to clients in the automotive retail industry as well as a variety of other businesses.

[Nov 04, 2018] Putt's Law and the Successful Technocrat

Nov 04, 2018 | en.wikipedia.org

From Wikipedia, the free encyclopedia Jump to navigation Jump to search

Question book-new.svg This article relies too much on references to primary sources . Please improve this by adding secondary or tertiary sources . (January 2015) ( Learn how and when to remove this template message )
Putt's Law and the Successful Technocrat
Putt's Law and the Successful Technocrat cover.jpg
Author Archibald Putt (pseudonym)
Illustrator Dennis Driscoll
Country United States
Language English
Genre Industrial Management
Publisher Wiley-IEEE Press
Publication date 28 April 2006
Media type Print ( hardcover )
Pages 171 pages
ISBN 0-471-71422-4
OCLC 68710099
Dewey Decimal 658.22
LC Class HD31 .P855 2006

Putt's Law and the Successful Technocrat is a book, credited to the pseudonym Archibald Putt, published in 1981. An updated edition, subtitled How to Win in the Information Age , was published by Wiley-IEEE Press in 2006. The book is based upon a series of articles published in Research/Development Magazine in 1976 and 1977.

It proposes Putt's Law and Putt's Corollary [1] which are principles of negative selection similar to The Dilbert principle by Scott Adams proposed in the 1990s. Putt's law is sometimes grouped together with the Peter principle , Parkinson's Law and Stephen Potter 's Gamesmanship series as "P-literature". [2]

Contents Putt's Law [ edit ]

The book proposes Putt's Law and Putt's Corollary

See also [ edit ] References [ edit ]
  1. Jump up ^ Archibald Putt. Putt's Law and the Successful Technocrat: How to Win in the Information Age , Wiley-IEEE Press (2006), ISBN 0-471-71422-4 . Preface.
  2. Jump up ^ John Walker (October 1981). "Review of Putt's Law and the Successful Technocrat " . New Scientist : 52.
  3. ^ Jump up to: a b Archibald Putt. Putt's Law and the Successful Technocrat: How to Win in the Information Age , Wiley-IEEE Press (2006), ISBN 0-471-71422-4 . page 7.
External links [ edit ]

[Nov 03, 2018] David Both

Jun 22, 2017 | opensource.com
...

The long listing of the /lib64 directory above shows that the first character in the filemode is the letter "l," which means that each is a soft or symbolic link.

Hard links

In An introduction to Linux's EXT4 filesystem , I discussed the fact that each file has one inode that contains information about that file, including the location of the data belonging to that file. Figure 2 in that article shows a single directory entry that points to the inode. Every file must have at least one directory entry that points to the inode that describes the file. The directory entry is a hard link, thus every file has at least one hard link.

In Figure 1 below, multiple directory entries point to a single inode. These are all hard links. I have abbreviated the locations of three of the directory entries using the tilde ( ~ ) convention for the home directory, so that ~ is equivalent to /home/user in this example. Note that the fourth directory entry is in a completely different directory, /home/shared , which might be a location for sharing files between users of the computer.

fig1directory_entries.png Figure 1

Hard links are limited to files contained within a single filesystem. "Filesystem" is used here in the sense of a partition or logical volume (LV) that is mounted on a specified mount point, in this case /home . This is because inode numbers are unique only within each filesystem, and a different filesystem, for example, /var or /opt , will have inodes with the same number as the inode for our file.

Because all the hard links point to the single inode that contains the metadata about the file, all of these attributes are part of the file, such as ownerships, permissions, and the total number of hard links to the inode, and cannot be different for each hard link. It is one file with one set of attributes. The only attribute that can be different is the file name, which is not contained in the inode. Hard links to a single file/inode located in the same directory must have different names, due to the fact that there can be no duplicate file names within a single directory.

The number of hard links for a file is displayed with the ls -l command. If you want to display the actual inode numbers, the command ls -li does that.

Symbolic (soft) links

The difference between a hard link and a soft link, also known as a symbolic link (or symlink), is that, while hard links point directly to the inode belonging to the file, soft links point to a directory entry, i.e., one of the hard links. Because soft links point to a hard link for the file and not the inode, they are not dependent upon the inode number and can work across filesystems, spanning partitions and LVs.

The downside to this is: If the hard link to which the symlink points is deleted or renamed, the symlink is broken. The symlink is still there, but it points to a hard link that no longer exists. Fortunately, the ls command highlights broken links with flashing white text on a red background in a long listing.

Lab project: experimenting with links

I think the easiest way to understand the use of and differences between hard and soft links is with a lab project that you can do. This project should be done in an empty directory as a non-root user . I created the ~/temp directory for this project, and you should, too. It creates a safe place to do the project and provides a new, empty directory to work in so that only files associated with this project will be located there.

Initial setup

First, create the temporary directory in which you will perform the tasks needed for this project. Ensure that the present working directory (PWD) is your home directory, then enter the following command.

mkdir temp

Change into ~/temp to make it the PWD with this command.

cd temp

To get started, we need to create a file we can link to. The following command does that and provides some content as well.

du -h > main.file.txt

Use the ls -l long list to verify that the file was created correctly. It should look similar to my results. Note that the file size is only 7 bytes, but yours may vary by a byte or two.

[ dboth @ david temp ] $ ls -l
total 4
-rw-rw-r-- 1 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice the number "1" following the file mode in the listing. That number represents the number of hard links that exist for the file. For now, it should be 1 because we have not created any additional links to our test file.

Experimenting with hard links

Hard links create a new directory entry pointing to the same inode, so when hard links are added to a file, you will see the number of links increase. Ensure that the PWD is still ~/temp . Create a hard link to the file main.file.txt , then do another long list of the directory.

[ dboth @ david temp ] $ ln main.file.txt link1.file.txt
[ dboth @ david temp ] $ ls -l
total 8
-rw-rw-r-- 2 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 2 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice that both files have two links and are exactly the same size. The date stamp is also the same. This is really one file with one inode and two links, i.e., directory entries to it. Create a second hard link to this file and list the directory contents. You can create the link to either of the existing ones: link1.file.txt or main.file.txt .

[ dboth @ david temp ] $ ln link1.file.txt link2.file.txt ; ls -l
total 16
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 link2.file.txt
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice that each new hard link in this directory must have a different name because two files -- really directory entries -- cannot have the same name within the same directory. Try to create another link with a target name the same as one of the existing ones.

[ dboth @ david temp ] $ ln main.file.txt link2.file.txt
ln: failed to create hard link 'link2.file.txt' : File exists

Clearly that does not work, because link2.file.txt already exists. So far, we have created only hard links in the same directory. So, create a link in your home directory, the parent of the temp directory in which we have been working so far.

[ dboth @ david temp ] $ ln main.file.txt .. / main.file.txt ; ls -l .. / main *
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt

The ls command in the above listing shows that the main.file.txt file does exist in the home directory with the same name as the file in the temp directory. Of course, these are not different files; they are the same file with multiple links -- directory entries -- to the same inode. To help illustrate the next point, add a file that is not a link.

[ dboth @ david temp ] $ touch unlinked.file ; ls -l
total 12
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link2.file.txt
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt
-rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Look at the inode number of the hard links and that of the new file using the -i option to the ls command.

[ dboth @ david temp ] $ ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Notice the number 657024 to the left of the file mode in the example above. That is the inode number, and all three file links point to the same inode. You can use the -i option to view the inode number for the link we created in the home directory as well, and that will also show the same value. The inode number of the file that has only one link is different from the others. Note that the inode numbers will be different on your system.

Let's change the size of one of the hard-linked files.

[ dboth @ david temp ] $ df -h > link2.file.txt ; ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The file size of all the hard-linked files is now larger than before. That is because there is really only one file that is linked to by multiple directory entries.

I know this next experiment will work on my computer because my /tmp directory is on a separate LV. If you have a separate LV or a filesystem on a different partition (if you're not using LVs), determine whether or not you have access to that LV or partition. If you don't, you can try to insert a USB memory stick and mount it. If one of those options works for you, you can do this experiment.

Try to create a link to one of the files in your ~/temp directory in /tmp (or wherever your different filesystem directory is located).

[ dboth @ david temp ] $ ln link2.file.txt / tmp / link3.file.txt
ln: failed to create hard link '/tmp/link3.file.txt' = > 'link2.file.txt' :
Invalid cross-device link

Why does this error occur? The reason is each separate mountable filesystem has its own set of inode numbers. Simply referring to a file by an inode number across the entire Linux directory structure can result in confusion because the same inode number can exist in each mounted filesystem.

There may be a time when you will want to locate all the hard links that belong to a single inode. You can find the inode number using the ls -li command. Then you can use the find command to locate all links with that inode number.

[ dboth @ david temp ] $ find . -inum 657024
. / main.file.txt
. / link1.file.txt
. / link2.file.txt

Note that the find command did not find all four of the hard links to this inode because we started at the current directory of ~/temp . The find command only finds files in the PWD and its subdirectories. To find all the links, we can use the following command, which specifies your home directory as the starting place for the search.

[ dboth @ david temp ] $ find ~ -samefile main.file.txt
/ home / dboth / temp / main.file.txt
/ home / dboth / temp / link1.file.txt
/ home / dboth / temp / link2.file.txt
/ home / dboth / main.file.txt

You may see error messages if you do not have permissions as a non-root user. This command also uses the -samefile option instead of specifying the inode number. This works the same as using the inode number and can be easier if you know the name of one of the hard links.

Experimenting with soft links

As you have just seen, creating hard links is not possible across filesystem boundaries; that is, from a filesystem on one LV or partition to a filesystem on another. Soft links are a means to answer that problem with hard links. Although they can accomplish the same end, they are very different, and knowing these differences is important.

Let's start by creating a symlink in our ~/temp directory to start our exploration.

[ dboth @ david temp ] $ ln -s link2.file.txt link3.file.txt ; ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The hard links, those that have the inode number 657024 , are unchanged, and the number of hard links shown for each has not changed. The newly created symlink has a different inode, number 658270 . The soft link named link3.file.txt points to link2.file.txt . Use the cat command to display the contents of link3.file.txt . The file mode information for the symlink starts with the letter " l " which indicates that this file is actually a symbolic link.

The size of the symlink link3.file.txt is only 14 bytes in the example above. That is the size of the text link3.file.txt -> link2.file.txt , which is the actual content of the directory entry. The directory entry link3.file.txt does not point to an inode; it points to another directory entry, which makes it useful for creating links that span file system boundaries. So, let's create that link we tried before from the /tmp directory.

[ dboth @ david temp ] $ ln -s / home / dboth / temp / link2.file.txt
/ tmp / link3.file.txt ; ls -l / tmp / link *
lrwxrwxrwx 1 dboth dboth 31 Jun 14 21 : 53 / tmp / link3.file.txt - >
/ home / dboth / temp / link2.file.txt Deleting links

There are some other things that you should consider when you need to delete links or the files to which they point.

First, let's delete the link main.file.txt . Remember that every directory entry that points to an inode is simply a hard link.

[ dboth @ david temp ] $ rm main.file.txt ; ls -li
total 8
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The link main.file.txt was the first link created when the file was created. Deleting it now still leaves the original file and its data on the hard drive along with all the remaining hard links. To delete the file and its data, you would have to delete all the remaining hard links.

Now delete the link2.file.txt hard link.

[ dboth @ david temp ] $ rm link2.file.txt ; ls -li
total 8
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Notice what happens to the soft link. Deleting the hard link to which the soft link points leaves a broken link. On my system, the broken link is highlighted in colors and the target hard link is flashing. If the broken link needs to be fixed, you can create another hard link in the same directory with the same name as the old one, so long as not all the hard links have been deleted. You could also recreate the link itself, with the link maintaining the same name but pointing to one of the remaining hard links. Of course, if the soft link is no longer needed, it can be deleted with the rm command.

The unlink command can also be used to delete files and links. It is very simple and has no options, as the rm command does. It does, however, more accurately reflect the underlying process of deletion, in that it removes the link -- the directory entry -- to the file being deleted.

Final thoughts

I worked with both types of links for a long time before I began to understand their capabilities and idiosyncrasies. It took writing a lab project for a Linux class I taught to fully appreciate how links work. This article is a simplification of what I taught in that class, and I hope it speeds your learning curve. David Both - David Both is a Linux and Open Source advocate who resides in Raleigh, North Carolina. He has been in the IT industry for over forty years and taught OS/2 for IBM where he worked for over 20 years. While at IBM, he wrote the first training course for the original IBM PC in 1981. He has taught RHCE classes for Red Hat and has worked at MCI Worldcom, Cisco, and the State of North Carolina. He has been working with Linux and Open Source Software for almost 20 years. dgrb on 23 Jun 2017 Permalink There is a hard link "gotcha" which IMHO is worth mentioning.

If you use an editor which makes automatic backups - emacs certainly is one such - then you may end up with a new version of the edited file, while the backup is the linked copy, because the editor simply renames the file to the backup name (with emacs, test.c would be renamed test.c~) and the new version when saved under the old name is no longer linked.

Symbolic links avoid this problem, so I tend to use them for source code where required.

[Nov 03, 2018] Neoliberal Measurement Mania

Highly recommended!
Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand. -- Archibald Putt
Neoliberal PHBs like talk about KJLOCs, error counts, tickets closed and other types of numerical measurements designed so that they can be used by lower-level PHBs to report fake results to higher level PHBs. These attempts to quantify 'the quality' and volume of work performed by software developers and sysadmins completely miss the point. For software is can lead to code bloat.
The number of tickets taken and resolved in a specified time period probably the most ignorant way to measure performance of sysadmins. For sysadmin you can invent creative creating way of generating and resolving tickets. And spend time accomplishing fake task, instead of thinking about real problem that datacenter face. Using Primitive measurement strategies devalue the work being performed by Sysadmins and programmers. They focus on the wrong things. They create the boundaries that are supposed to contain us in a manner that is comprehensible to the PHB who knows nothing about real problems we face.
Notable quotes:
"... Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand. ..."
Nov 03, 2018 | www.rako.com

In an advanced research or development project, success or failure is largely determined when the goals or objectives are set and before a manager is chosen. While a hard-working and diligent manager can increase the chances of success, the outcome of the project is most strongly affected by preexisting but unknown technological factors over which the project manager has no control. The success or failure of the project should not, therefore, be used as the sole measure or even the primary measure of the manager's competence.

Putt's Law Is promulgated

Without an adequate competence criterion for technical managers, there is no way to determine when a person has reached his level of incompetence. Thus a clever and ambitious individual may be promoted from one level of incompetence to another. He will ultimately perform incompetently in the highest level of the hierarchy just as he did in numerous lower levels. The lack of an adequate competence criterion combined with the frequent practice of creative incompetence in technical hierarchies results in a competence inversion, with the most competent people remaining near the bottom while persons of lesser talent rise to the top. It also provides the basis for Putt's Law, which can be stated in an intuitive and nonmathematical form as follows:

Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand.

As in any other hierarchy, the majority of persons in technology neither understand nor manage much of anything. This, however, does not create an exception to Putt's Law, because such persons clearly do not dominate the hierarchy. While this was not previously stated as a basic law, it is clear that the success of every technocrat depends on his ability to deal with and benefit from the consequences of Putt's Law.

[Nov 03, 2018] Archibald Putt The Unknown Technocrat Returns - IEEE Spectrum

Notable quotes:
"... Who is Putt? Well, for those of you under 40, the pseudonymous Archibald Putt, Ph.D., penned a series of articles for Research/Development magazine in the 1970s that eventually became the 1981 cult classic Putt's Law and the Successful Technocrat , an unorthodox and archly funny how-to book for achieving tech career success. ..."
"... His first law, "Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand," along with its corollary, "Every technical hierarchy, in time, develops a competence inversion," have been immortalized on Web sites around the world. ..."
"... what's a competence inversion? It means that the best and the brightest in a technology company tend to settle on the lowest rungs of the corporate ladder -- where things like inventing and developing new products get done -- while those who manage what they cannot hope to make or understand float to the top (see Putt's first law, above, and a fine example of Putt's law in action in the editorial, " Is Bad Design a Nuisance? "). ..."
"... Other Putt laws we love include the law of failure: "Innovative organizations abhor little failures but reward big ones." And the first law of invention: "An innovated success is as good as a successful innovation." ..."
"... This is management writing the way it ought to be. Think Dilbert , but with a very big brain. Read it and weep. Or laugh, depending on your current job situation. ..."
"... n.hantman@ieee.org ..."
Nov 03, 2018 | spectrum.ieee.org

If you want to jump-start your technology career, put aside your Peter Drucker, your Tom Peters, and your Marcus Buckingham management tomes. Archibald Putt is back.

Who is Putt? Well, for those of you under 40, the pseudonymous Archibald Putt, Ph.D., penned a series of articles for Research/Development magazine in the 1970s that eventually became the 1981 cult classic Putt's Law and the Successful Technocrat , an unorthodox and archly funny how-to book for achieving tech career success.

In the book, Putt put forth a series of laws and axioms for surviving and succeeding in the unique corporate cultures of big technology companies, where being the builder of the best technology and becoming the top dog on the block almost never mix. His first law, "Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand," along with its corollary, "Every technical hierarchy, in time, develops a competence inversion," have been immortalized on Web sites around the world.

The first law is obvious, but what's a competence inversion? It means that the best and the brightest in a technology company tend to settle on the lowest rungs of the corporate ladder -- where things like inventing and developing new products get done -- while those who manage what they cannot hope to make or understand float to the top (see Putt's first law, above, and a fine example of Putt's law in action in the editorial, " Is Bad Design a Nuisance? ").

Other Putt laws we love include the law of failure: "Innovative organizations abhor little failures but reward big ones." And the first law of invention: "An innovated success is as good as a successful innovation."

Now Putt has revised and updated his short, smart book, to be released in a new edition by Wiley-IEEE Press ( http://www.wiley.com/ieee ) at the end of this month. There have been murmurings that Putt's identity, the subject of much rumormongering, will be revealed after the book comes out, but we think that's unlikely. How much more interesting it is to have an anonymous chronicler wandering the halls of the tech industry, codifying its unstated, sometimes bizarre, and yet remarkably consistent rules of behavior.

This is management writing the way it ought to be. Think Dilbert , but with a very big brain. Read it and weep. Or laugh, depending on your current job situation.

The editorial content of IEEE Spectrum does not represent official positions of the IEEE or its organizational units. Please address comments to Forum at n.hantman@ieee.org .

[Nov 03, 2018] Technology is dominated by two types of people; those who understand what they don t manage; and those who manage what they don t understand – ARCHIBALD PUTT ( PUTTS LAW )

Notable quotes:
"... These C level guys see cloud services – applications, data, backup, service desk – as a great way to free up a blockage in how IT service is being delivered on premise. ..."
"... IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. ..."
Nov 03, 2018 | brummieruss.wordpress.com

...Cloud introduces a whole new ball game and will no doubt perpetuate Putts Law for ever more. Why?

Well unless 100% of IT infrastructure goes up into the clouds ( unlikely for any organization with a history ; likely for a new organization ( probably micro small ) that starts up in the next few years ) the 'art of IT management' will demand even more focus and understanding.

I always think a great acid test of Putts Law is to look at one of the two aspects of IT management

  1. Show me a simple process that you follow each day that delivers an aspect of IT service i.e. how to buy a piece of IT stuff, or a way to report a fault
  2. Show me how you manage a single entity on the network i.e. a file server, a PC, a network switch

Usually the answers ( which will be different from people on the same team, in the same room and from the same person on different days !) will give you an insight to Putts Law.

Childs play for most of course who are challenged with some real complex management situations such as data center virtualization projects, storage explosion control, edge device management, backend application upgrades, global messaging migrations and B2C identity integration. But of course if its evidenced that they seem to be managing (simple things ) without true understanding one could argue 'how the hell can they be expected to manage what they understand with the complex things?' Fair point?

Of course many C level people have an answer to Putts Law. Move the problem to people who do understand what they manage. Professionals who provide cloud versions of what the C level person struggles to get a professional service from. These C level guys see cloud services – applications, data, backup, service desk – as a great way to free up a blockage in how IT service is being delivered on premise. And they are right ( and wrong ).

... ... ...

( Quote attributed to Archibald Putt author of Putt's Law and the Successful Technocrat: How to Win in the Information Age )

rowan says: March 9, 2012 at 9:03 am

IMHO there is a big difference between management of IT and management of IT service. Rarely do you get people who can do both. Understanding inventory, disk space, security etc is one thing; but understanding the performance of apps and user impact is another ball game. Putts Law is alive and well in my organisation. TGIF.

Rowan in Belfast.

stephen777 says: March 31, 2012 at 7:32 am

Rowan is right I used to be an IT Manager but now my title is Service Delivery Manager. Why? Because we had a new CTO who changed how people saw what we did. I ve been doing this new role for 5 years and I really do understand what i don't manage. LOL

Stephen777

[Nov 03, 2018] David Both

Nov 03, 2018 | opensource.com

Feed 161 up 4 comments Links Image by : Paul Lewin . Modified by Opensource.com. CC BY-SA 2.0 x Get the newsletter

Join the 85,000 open source advocates who receive our giveaway alerts and article roundups.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0

An introduction to Linux's EXT4 filesystem ; Managing devices in Linux ; An introduction to Linux filesystems ; and A Linux user's guide to Logical Volume Management , I have briefly mentioned an interesting feature of Linux filesystems that can make some tasks easier by providing access to files from multiple locations in the filesystem directory tree.

There are two types of Linux filesystem links: hard and soft. The difference between the two types of links is significant, but both types are used to solve similar problems. They both provide multiple directory entries (or references) to a single file, but they do it quite differently. Links are powerful and add flexibility to Linux filesystems because everything is a file .

More Linux resources

I have found, for instance, that some programs required a particular version of a library. When a library upgrade replaced the old version, the program would crash with an error specifying the name of the old, now-missing library. Usually, the only change in the library name was the version number. Acting on a hunch, I simply added a link to the new library but named the link after the old library name. I tried the program again and it worked perfectly. And, okay, the program was a game, and everyone knows the lengths that gamers will go to in order to keep their games running.

In fact, almost all applications are linked to libraries using a generic name with only a major version number in the link name, while the link points to the actual library file that also has a minor version number. In other instances, required files have been moved from one directory to another to comply with the Linux file specification, and there are links in the old directories for backwards compatibility with those programs that have not yet caught up with the new locations. If you do a long listing of the /lib64 directory, you can find many examples of both.

lrwxrwxrwx. 1 root root 36 Dec 8 2016 cracklib_dict.hwm -> ../../usr/share/cracklib/pw_dict.hwm
lrwxrwxrwx. 1 root root 36 Dec 8 2016 cracklib_dict.pwd -> ../../usr/share/cracklib/pw_dict.pwd
lrwxrwxrwx. 1 root root 36 Dec 8 2016 cracklib_dict.pwi -> ../../usr/share/cracklib/pw_dict.pwi
lrwxrwxrwx. 1 root root 27 Jun 9 2016 libaccountsservice.so.0 -> libaccountsservice.so.0.0.0
-rwxr-xr-x. 1 root root 288456 Jun 9 2016 libaccountsservice.so.0.0.0
lrwxrwxrwx 1 root root 15 May 17 11:47 libacl.so.1 -> libacl.so.1.1.0
-rwxr-xr-x 1 root root 36472 May 17 11:47 libacl.so.1.1.0
lrwxrwxrwx. 1 root root 15 Feb 4 2016 libaio.so.1 -> libaio.so.1.0.1
-rwxr-xr-x. 1 root root 6224 Feb 4 2016 libaio.so.1.0.0
-rwxr-xr-x. 1 root root 6224 Feb 4 2016 libaio.so.1.0.1
lrwxrwxrwx. 1 root root 30 Jan 16 16:39 libakonadi-calendar.so.4 -> libakonadi-calendar.so.4.14.26
-rwxr-xr-x. 1 root root 816160 Jan 16 16:39 libakonadi-calendar.so.4.14.26
lrwxrwxrwx. 1 root root 29 Jan 16 16:39 libakonadi-contact.so.4 -> libakonadi-contact.so.4.14.26

A few of the links in the /lib64 directory

The long listing of the /lib64 directory above shows that the first character in the filemode is the letter "l," which means that each is a soft or symbolic link.

Hard links

In An introduction to Linux's EXT4 filesystem , I discussed the fact that each file has one inode that contains information about that file, including the location of the data belonging to that file. Figure 2 in that article shows a single directory entry that points to the inode. Every file must have at least one directory entry that points to the inode that describes the file. The directory entry is a hard link, thus every file has at least one hard link.

In Figure 1 below, multiple directory entries point to a single inode. These are all hard links. I have abbreviated the locations of three of the directory entries using the tilde ( ~ ) convention for the home directory, so that ~ is equivalent to /home/user in this example. Note that the fourth directory entry is in a completely different directory, /home/shared , which might be a location for sharing files between users of the computer.

fig1directory_entries.png Figure 1

Hard links are limited to files contained within a single filesystem. "Filesystem" is used here in the sense of a partition or logical volume (LV) that is mounted on a specified mount point, in this case /home . This is because inode numbers are unique only within each filesystem, and a different filesystem, for example, /var or /opt , will have inodes with the same number as the inode for our file.

Because all the hard links point to the single inode that contains the metadata about the file, all of these attributes are part of the file, such as ownerships, permissions, and the total number of hard links to the inode, and cannot be different for each hard link. It is one file with one set of attributes. The only attribute that can be different is the file name, which is not contained in the inode. Hard links to a single file/inode located in the same directory must have different names, due to the fact that there can be no duplicate file names within a single directory.

The number of hard links for a file is displayed with the ls -l command. If you want to display the actual inode numbers, the command ls -li does that.

Symbolic (soft) links

The difference between a hard link and a soft link, also known as a symbolic link (or symlink), is that, while hard links point directly to the inode belonging to the file, soft links point to a directory entry, i.e., one of the hard links. Because soft links point to a hard link for the file and not the inode, they are not dependent upon the inode number and can work across filesystems, spanning partitions and LVs.

The downside to this is: If the hard link to which the symlink points is deleted or renamed, the symlink is broken. The symlink is still there, but it points to a hard link that no longer exists. Fortunately, the ls command highlights broken links with flashing white text on a red background in a long listing.

Lab project: experimenting with links

I think the easiest way to understand the use of and differences between hard and soft links is with a lab project that you can do. This project should be done in an empty directory as a non-root user . I created the ~/temp directory for this project, and you should, too. It creates a safe place to do the project and provides a new, empty directory to work in so that only files associated with this project will be located there.

Initial setup

First, create the temporary directory in which you will perform the tasks needed for this project. Ensure that the present working directory (PWD) is your home directory, then enter the following command.

mkdir temp

Change into ~/temp to make it the PWD with this command.

cd temp

To get started, we need to create a file we can link to. The following command does that and provides some content as well.

du -h > main.file.txt

Use the ls -l long list to verify that the file was created correctly. It should look similar to my results. Note that the file size is only 7 bytes, but yours may vary by a byte or two.

[ dboth @ david temp ] $ ls -l
total 4
-rw-rw-r-- 1 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice the number "1" following the file mode in the listing. That number represents the number of hard links that exist for the file. For now, it should be 1 because we have not created any additional links to our test file.

Experimenting with hard links

Hard links create a new directory entry pointing to the same inode, so when hard links are added to a file, you will see the number of links increase. Ensure that the PWD is still ~/temp . Create a hard link to the file main.file.txt , then do another long list of the directory.

[ dboth @ david temp ] $ ln main.file.txt link1.file.txt
[ dboth @ david temp ] $ ls -l
total 8
-rw-rw-r-- 2 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 2 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice that both files have two links and are exactly the same size. The date stamp is also the same. This is really one file with one inode and two links, i.e., directory entries to it. Create a second hard link to this file and list the directory contents. You can create the link to either of the existing ones: link1.file.txt or main.file.txt .

[ dboth @ david temp ] $ ln link1.file.txt link2.file.txt ; ls -l
total 16
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 link2.file.txt
-rw-rw-r-- 3 dboth dboth 7 Jun 13 07: 34 main.file.txt

Notice that each new hard link in this directory must have a different name because two files -- really directory entries -- cannot have the same name within the same directory. Try to create another link with a target name the same as one of the existing ones.

[ dboth @ david temp ] $ ln main.file.txt link2.file.txt
ln: failed to create hard link 'link2.file.txt' : File exists

Clearly that does not work, because link2.file.txt already exists. So far, we have created only hard links in the same directory. So, create a link in your home directory, the parent of the temp directory in which we have been working so far.

[ dboth @ david temp ] $ ln main.file.txt .. / main.file.txt ; ls -l .. / main *
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt

The ls command in the above listing shows that the main.file.txt file does exist in the home directory with the same name as the file in the temp directory. Of course, these are not different files; they are the same file with multiple links -- directory entries -- to the same inode. To help illustrate the next point, add a file that is not a link.

[ dboth @ david temp ] $ touch unlinked.file ; ls -l
total 12
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link1.file.txt
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link2.file.txt
-rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt
-rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Look at the inode number of the hard links and that of the new file using the -i option to the ls command.

[ dboth @ david temp ] $ ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 7 Jun 13 07: 34 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Notice the number 657024 to the left of the file mode in the example above. That is the inode number, and all three file links point to the same inode. You can use the -i option to view the inode number for the link we created in the home directory as well, and that will also show the same value. The inode number of the file that has only one link is different from the others. Note that the inode numbers will be different on your system.

Let's change the size of one of the hard-linked files.

[ dboth @ david temp ] $ df -h > link2.file.txt ; ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The file size of all the hard-linked files is now larger than before. That is because there is really only one file that is linked to by multiple directory entries.

I know this next experiment will work on my computer because my /tmp directory is on a separate LV. If you have a separate LV or a filesystem on a different partition (if you're not using LVs), determine whether or not you have access to that LV or partition. If you don't, you can try to insert a USB memory stick and mount it. If one of those options works for you, you can do this experiment.

Try to create a link to one of the files in your ~/temp directory in /tmp (or wherever your different filesystem directory is located).

[ dboth @ david temp ] $ ln link2.file.txt / tmp / link3.file.txt
ln: failed to create hard link '/tmp/link3.file.txt' = > 'link2.file.txt' :
Invalid cross-device link

Why does this error occur? The reason is each separate mountable filesystem has its own set of inode numbers. Simply referring to a file by an inode number across the entire Linux directory structure can result in confusion because the same inode number can exist in each mounted filesystem.

There may be a time when you will want to locate all the hard links that belong to a single inode. You can find the inode number using the ls -li command. Then you can use the find command to locate all links with that inode number.

[ dboth @ david temp ] $ find . -inum 657024
. / main.file.txt
. / link1.file.txt
. / link2.file.txt

Note that the find command did not find all four of the hard links to this inode because we started at the current directory of ~/temp . The find command only finds files in the PWD and its subdirectories. To find all the links, we can use the following command, which specifies your home directory as the starting place for the search.

[ dboth @ david temp ] $ find ~ -samefile main.file.txt
/ home / dboth / temp / main.file.txt
/ home / dboth / temp / link1.file.txt
/ home / dboth / temp / link2.file.txt
/ home / dboth / main.file.txt

You may see error messages if you do not have permissions as a non-root user. This command also uses the -samefile option instead of specifying the inode number. This works the same as using the inode number and can be easier if you know the name of one of the hard links.

Experimenting with soft links

As you have just seen, creating hard links is not possible across filesystem boundaries; that is, from a filesystem on one LV or partition to a filesystem on another. Soft links are a means to answer that problem with hard links. Although they can accomplish the same end, they are very different, and knowing these differences is important.

Let's start by creating a symlink in our ~/temp directory to start our exploration.

[ dboth @ david temp ] $ ln -s link2.file.txt link3.file.txt ; ls -li
total 12
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657024 -rw-rw-r-- 4 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The hard links, those that have the inode number 657024 , are unchanged, and the number of hard links shown for each has not changed. The newly created symlink has a different inode, number 658270 . The soft link named link3.file.txt points to link2.file.txt . Use the cat command to display the contents of link3.file.txt . The file mode information for the symlink starts with the letter " l " which indicates that this file is actually a symbolic link.

The size of the symlink link3.file.txt is only 14 bytes in the example above. That is the size of the text link3.file.txt -> link2.file.txt , which is the actual content of the directory entry. The directory entry link3.file.txt does not point to an inode; it points to another directory entry, which makes it useful for creating links that span file system boundaries. So, let's create that link we tried before from the /tmp directory.

[ dboth @ david temp ] $ ln -s / home / dboth / temp / link2.file.txt
/ tmp / link3.file.txt ; ls -l / tmp / link *
lrwxrwxrwx 1 dboth dboth 31 Jun 14 21 : 53 / tmp / link3.file.txt - >
/ home / dboth / temp / link2.file.txt Deleting links

There are some other things that you should consider when you need to delete links or the files to which they point.

First, let's delete the link main.file.txt . Remember that every directory entry that points to an inode is simply a hard link.

[ dboth @ david temp ] $ rm main.file.txt ; ls -li
total 8
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link2.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

The link main.file.txt was the first link created when the file was created. Deleting it now still leaves the original file and its data on the hard drive along with all the remaining hard links. To delete the file and its data, you would have to delete all the remaining hard links.

Now delete the link2.file.txt hard link.

[ dboth @ david temp ] $ rm link2.file.txt ; ls -li
total 8
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 link1.file.txt
658270 lrwxrwxrwx 1 dboth dboth 14 Jun 14 15 : 21 link3.file.txt - >
link2.file.txt
657024 -rw-rw-r-- 3 dboth dboth 1157 Jun 14 14 : 14 main.file.txt
657863 -rw-rw-r-- 1 dboth dboth 0 Jun 14 08: 18 unlinked.file

Notice what happens to the soft link. Deleting the hard link to which the soft link points leaves a broken link. On my system, the broken link is highlighted in colors and the target hard link is flashing. If the broken link needs to be fixed, you can create another hard link in the same directory with the same name as the old one, so long as not all the hard links have been deleted. You could also recreate the link itself, with the link maintaining the same name but pointing to one of the remaining hard links. Of course, if the soft link is no longer needed, it can be deleted with the rm command.

The unlink command can also be used to delete files and links. It is very simple and has no options, as the rm command does. It does, however, more accurately reflect the underlying process of deletion, in that it removes the link -- the directory entry -- to the file being deleted.

Final thoughts

I worked with both types of links for a long time before I began to understand their capabilities and idiosyncrasies. It took writing a lab project for a Linux class I taught to fully appreciate how links work. This article is a simplification of what I taught in that class, and I hope it speeds your learning curve. Topics Linux About the author David Both - David Both is a Linux and Open Source advocate who resides in Raleigh, North Carolina. He has been in the IT industry for over forty years and taught OS/2 for IBM where he worked for over 20 years. While at IBM, he wrote the first training course for the original IBM PC in 1981. He has taught RHCE classes for Red Hat and has worked at MCI Worldcom, Cisco, and the State of North Carolina. He has been working with Linux and Open Source Software for almost 20 years. David has written articles for...

[Nov 03, 2018] Is Red Hat IBM's 'Hail Mary' pass

Notable quotes:
"... if those employees become unhappy, they can effectively go anywhere they want. ..."
"... IBM's partner/reseller ecosystem is nowhere near what it was since it owned the PC and Server businesses that Lenovo now owns. And IBM's Softlayer/BlueMix cloud is largely tied to its legacy software business, which, again, is slowing. ..."
"... I came to IBM from their SoftLayer acquisition. Their ability to stomp all over the things SoftLayer was almost doing right were astounding. I stood and listened to Ginni say things like, "We purchased SoftLayer because we need to learn from you," and, "We want you to teach us how to do Cloud the right way, since we spent all these years doing things the wrong way," and, "If you find yourself in a meeting with one of our old teams, you guys are gonna be the ones in charge. You are the ones who know how this is supposed to work - our culture has failed at it." Promises which were nothing more than hollow words. ..."
"... Next, it's a little worrisome that the author, now over the whole IBM thing is recommending firing "older people," you know, the ones who helped the company retain its performance in years' past. The smartest article I've read about IBM worried about its cheap style of "acquiring" non-best-of-breed companies and firing oodles of its qualified R&D guys. THAT author was right. ..."
"... Four years in GTS ... joined via being outsourced to IBM by my previous employer. Left GTS after 4 years. ..."
"... The IBM way of life was throughout the Oughts and the Teens an utter and complete failure from the perspective of getting work done right and using people to their appropriate and full potential. ..."
"... As a GTS employee, professional technical training was deemed unnecessary, hence I had no access to any unless I paid for it myself and used my personal time ... the only training available was cheesy presentations or other web based garbage from the intranet, or casual / OJT style meetings with other staff who were NOT professional or expert trainers. ..."
"... As a GTS employee, I had NO access to the expert and professional tools that IBM fricking made and sold to the same damn customers I was supposed to be supporting. Did we have expert and professional workflow / document management / ITIL aligned incident and problem management tools? NO, we had fricking Lotus Notes and email. Instead of upgrading to the newest and best software solutions for data center / IT management & support, we degraded everything down the simplest and least complex single function tools that no "best practices" organization on Earth would ever consider using. ..."
"... And the people management paradigm ... employees ranked annually not against a static or shared goal or metric, but in relation to each other, and there was ALWAYS a "top 10 percent" and a "bottom ten percent" required by upper management ... a system that was sociopathic in it's nature because it encourages employees to NOT work together ... by screwing over one's coworkers, perhaps by not giving necessary information, timely support, assistance as needed or requested, one could potentially hurt their performance and make oneself look relatively better. That's a self-defeating system and it was encouraged by the way IBM ran things. ..."
Nov 03, 2018 | www.zdnet.com
Brain drain is a real risk

IBM has not had a particularly great track record when it comes to integrating the cultures of other companies into its own, and brain drain with a company like Red Hat is a real risk because if those employees become unhappy, they can effectively go anywhere they want. They have the skills to command very high salaries at any of the top companies in the industry.

The other issue is that IBM hasn't figured out how to capture revenue from SMBs -- and that has always been elusive for them. Unless a deal is worth at least $1 million, and realistically $10 million, sales guys at IBM don't tend to get motivated.

Also: Red Hat changes its open-source licensing rules

The 5,000-seat and below market segment has traditionally been partner territory, and when it comes to reseller partners for its cloud, IBM is way, way behind AWS, Microsoft, Google, or even (gasp) Oracle, which is now offering serious margins to partners that land workloads on the Oracle cloud.

IBM's partner/reseller ecosystem is nowhere near what it was since it owned the PC and Server businesses that Lenovo now owns. And IBM's Softlayer/BlueMix cloud is largely tied to its legacy software business, which, again, is slowing.

... ... ...

But I think that it is very unlikely the IBM Cloud, even when juiced on Red Hat steroids, will become anything more ambitious than a boutique business for hybrid workloads when compared with AWS or Azure. Realistically, it has to be the kind of cloud platform that interoperates well with the others or nobody will want it.


geek49203_z , Wednesday, April 26, 2017 10:27 AM

Ex-IBM contractor here...

1. IBM used to value long-term employees. Now they "value" short-term contractors -- but they still pull them out of production for lots of training that, quite frankly, isn't exactly needed for what they are doing. Personally, I think that IBM would do well to return to valuing employees instead of looking at them as expendable commodities, but either way, they need to get past the legacies of when they had long-term employees all watching a single main frame.

2. As IBM moved to an army of contractors, they killed off the informal (but important!) web of tribal knowledge. You know, a friend of a friend who new the answer to some issue, or knew something about this customer? What has happened is that the transaction costs (as economists call it) have escalated until IBM can scarcely order IBM hardware for its own projects, or have SDM's work together.

M Wagner geek49203_z , Wednesday, April 26, 2017 10:35 AM
geek49203_z Number 2 is a problem everywhere. As long-time employees (mostly baby-boomers) retire, their replacements are usually straight out of college with various non-technical degrees. They come in with little history and few older-employees to which they can turn for "the tricks of the trade".
Shmeg , Wednesday, April 26, 2017 10:41 AM
I came to IBM from their SoftLayer acquisition. Their ability to stomp all over the things SoftLayer was almost doing right were astounding. I stood and listened to Ginni say things like, "We purchased SoftLayer because we need to learn from you," and, "We want you to teach us how to do Cloud the right way, since we spent all these years doing things the wrong way," and, "If you find yourself in a meeting with one of our old teams, you guys are gonna be the ones in charge. You are the ones who know how this is supposed to work - our culture has failed at it." Promises which were nothing more than hollow words.
geek49203_z , Wednesday, April 26, 2017 10:27 AM
Ex-IBM contractor here...

1. IBM used to value long-term employees. Now they "value" short-term contractors -- but they still pull them out of production for lots of training that, quite frankly, isn't exactly needed for what they are doing. Personally, I think that IBM would do well to return to valuing employees instead of looking at them as expendable commodities, but either way, they need to get past the legacies of when they had long-term employees all watching a single main frame.

2. As IBM moved to an army of contractors, they killed off the informal (but important!) web of tribal knowledge. You know, a friend of a friend who new the answer to some issue, or knew something about this customer? What has happened is that the transaction costs (as economists call it) have escalated until IBM can scarcely order IBM hardware for its own projects, or have SDM's work together.

M Wagner geek49203_z , Wednesday, April 26, 2017 10:35 AM
geek49203_z Number 2 is a problem everywhere. As long-time employees (mostly baby-boomers) retire, their replacements are usually straight out of college with various non-technical degrees. They come in with little history and few older-employees to which they can turn for "the tricks of the trade".
Shmeg , Wednesday, April 26, 2017 10:41 AM
I came to IBM from their SoftLayer acquisition. Their ability to stomp all over the things SoftLayer was almost doing right were astounding. I stood and listened to Ginni say things like, "We purchased SoftLayer because we need to learn from you," and, "We want you to teach us how to do Cloud the right way, since we spent all these years doing things the wrong way," and, "If you find yourself in a meeting with one of our old teams, you guys are gonna be the ones in charge. You are the ones who know how this is supposed to work - our culture has failed at it." Promises which were nothing more than hollow words.
cavman , Wednesday, April 26, 2017 3:58 PM
In the 1970's 80's and 90's I was working in tech support for a company called ROLM. We were doing communications , voice and data and did many systems for Fortune 500 companies along with 911 systems and the secure system at the White House. My job was to fly all over North America to solve problems with customers and integration of our equipment into their business model. I also did BETA trials and documented systems so others would understand what it took to make it run fine under all conditions.

In 84 IBM bought a percentage of the company and the next year they bought out the company. When someone said to me "IBM just bought you out , you must thing you died and went to heaven." My response was "Think of them as being like the Federal Government but making a profit". They were so heavily structured and hide bound that it was a constant battle working with them. Their response to any comments was "We are IBM"

I was working on an equipment project in Colorado Springs and IBM took control. I was immediately advised that I could only talk to the people in my assigned group and if I had a question outside of my group I had to put it in writing and give it to my manager and if he thought it was relevant it would be forwarded up the ladder of management until it reached a level of a manager that had control of both groups and at that time if he thought it was relevant it would be sent to that group who would send the answer back up the ladder.

I'm a Vietnam Veteran and I used my military training to get things done just like I did out in the field. I went looking for the person I could get an answer from.

At first others were nervous about doing that but within a month I had connections all over the facility and started introducing people at the cafeteria. Things moved quickly as people started working together as a unit. I finished my part of the work which was figuring all the spares technicians would need plus the costs for packaging and service contract estimates. I submitted it to all the people that needed it. I was then hauled into a meeting room by the IBM management and advised that I was a disruptive influence and would be removed. Just then the final contracts that vendors had to sign showed up and it used all my info. The IBM people were livid that they were not involved.

By the way a couple months later the IBM THINK magazine came out with a new story about a radical concept they had tried. A cover would not fit on a component and under the old system both the component and the cover would be thrown out and they would start from scratch doing it over. They decided to have the two groups sit together and figure out why it would not fit and correct it on the spot.

Another great example of IBM people is we had a sales contract to install a multi node voice mail system at WANG computers but we lost it because the IBM people insisted on bundling in AS0400 systems into the sale to WANG computer. Instead we lost a multi million dollar contract.

Eventually Siemens bought 50% of the company and eventually full control. Now all we heard was "That is how we do it in Germany" Our response was "How did that WW II thing work out".

Stockholder , Wednesday, April 26, 2017 7:20 PM
The author may have more loyalty to Microsoft than he confides, is the first thing noticeable about this article. The second thing is that in terms of getting rid of those aged IBM workers, I think he may have completely missed the mark, in fairness, that may be the product of his IBM experience, The sheer hubris of tech-talking from the middle of the story and missing the global misstep that is today's IBM is noticeable. As a stockholder, the first question is, "Where is the investigation to the breach of fiduciary duty by a board that owes its loyalty to stockholders who are scratching their heads at the 'positive' spin the likes of Ginni Rometty is putting on 20 quarters of dead losses?" Got that, 20 quarters of losses.

Next, it's a little worrisome that the author, now over the whole IBM thing is recommending firing "older people," you know, the ones who helped the company retain its performance in years' past. The smartest article I've read about IBM worried about its cheap style of "acquiring" non-best-of-breed companies and firing oodles of its qualified R&D guys. THAT author was right.

IBM's been run into the ground by Ginni, I'll use her first name, since apparently my money is now used to prop up this sham of a leader, who from her uncomfortable public announcement with Tim Cook of Apple, which HAS gone up, by the way, has embraced every political trend, not cause but trend from hiring more women to marginalizing all those old-time white males...You know the ones who produced for the company based on merit, sweat, expertise, all those non-feeling based skills that ultimately are what a shareholder is interested in and replaced them with young, and apparently "social" experts who are pasting some phony "modernity" on a company that under Ginni's leadership has become more of a pet cause than a company.

Finally, regarding ageism and the author's advocacy for the same, IBM's been there, done that as they lost an age discrimination lawsuit decades ago. IBM gave up on doing what it had the ability to do as an enormous business and instead under Rometty's leadership has tried to compete with the scrappy startups where any halfwit knows IBM cannot compete.

The company has rendered itself ridiculous under Rometty, a board that collects paychecks and breaches any notion of fiduciary duty to shareholders, an attempt at partnering with a "mod" company like Apple that simply bolstered Apple and left IBM languishing and a rejection of what has a track record of working, excellence, rewarding effort of employees and the steady plod of performance. Dump the board and dump Rometty.

jperlow Stockholder , Wednesday, April 26, 2017 8:36 PM
Stockholder Your comments regarding any inclination towards age discrimination are duly noted, so I added a qualifier in the piece.
Gravyboat McGee , Wednesday, April 26, 2017 9:00 PM
Four years in GTS ... joined via being outsourced to IBM by my previous employer. Left GTS after 4 years.

The IBM way of life was throughout the Oughts and the Teens an utter and complete failure from the perspective of getting work done right and using people to their appropriate and full potential. I went from a multi-disciplinary team of engineers working across technologies to support corporate needs in the IT environment to being siloed into a single-function organization.

My first year of on-boarding with IBM was spent deconstructing application integration and cross-organizational structures of support and interwork that I had spent 6 years building and maintaining. Handing off different chunks of work (again, before the outsourcing, an Enterprise solution supported by one multi-disciplinary team) to different IBM GTS work silos that had no physical spacial relationship and no interworking history or habits. What we're talking about here is the notion of "left hand not knowing what the right hand is doing" ...

THAT was the IBM way of doing things, and nothing I've read about them over the past decade or so tells me it has changed.

As a GTS employee, professional technical training was deemed unnecessary, hence I had no access to any unless I paid for it myself and used my personal time ... the only training available was cheesy presentations or other web based garbage from the intranet, or casual / OJT style meetings with other staff who were NOT professional or expert trainers.

As a GTS employee, I had NO access to the expert and professional tools that IBM fricking made and sold to the same damn customers I was supposed to be supporting. Did we have expert and professional workflow / document management / ITIL aligned incident and problem management tools? NO, we had fricking Lotus Notes and email. Instead of upgrading to the newest and best software solutions for data center / IT management & support, we degraded everything down the simplest and least complex single function tools that no "best practices" organization on Earth would ever consider using.

And the people management paradigm ... employees ranked annually not against a static or shared goal or metric, but in relation to each other, and there was ALWAYS a "top 10 percent" and a "bottom ten percent" required by upper management ... a system that was sociopathic in it's nature because it encourages employees to NOT work together ... by screwing over one's coworkers, perhaps by not giving necessary information, timely support, assistance as needed or requested, one could potentially hurt their performance and make oneself look relatively better. That's a self-defeating system and it was encouraged by the way IBM ran things.

The "not invented here" ideology was embedded deeply in the souls of all senior IBMers I ever met or worked with ... if you come on board with any outside knowledge or experience, you must not dare to say "this way works better" because you'd be shut down before you could blink. The phrase "best practices" to them means "the way we've always done it".

IBM gave up on innovation long ago. Since the 90's the vast majority of their software has been bought, not built. Buy a small company, strip out the innovation, slap an IBM label on it, sell it as the next coming of Jesus even though they refuse to expend any R&D to push the product to the next level ... damn near everything IBM sold was gentrified, never cutting edge.

And don't get me started on sales practices ... tell the customer how product XYZ is a guaranteed moonshot, they'll be living on lunar real estate in no time at all, and after all the contracts are signed hand the customer a box of nuts & bolts and a letter telling them where they can look up instructions on how to build their own moon rocket. Or for XX dollars more a year, hire a Professional Services IBMer to build it for them.

I have no sympathy for IBM. They need a clean sweep throughout upper management, especially any of the old True Blue hard-core IBMers.

billa201 , Thursday, April 27, 2017 11:24 AM
You obviously have been gone from IBM as they do not treat their employees well anymore and get rid of good talent not keep it a sad state.
ClearCreek , Tuesday, May 9, 2017 7:04 PM
We tried our best to be SMB partners with IBM & Arrow in the early 2000s ... but could never get any traction. I personally needed a mentor, but never found one. I still have/wear some of their swag, and I write this right now on a re-purposed IBM 1U server that is 10 years old, but ... I can't see any way our small company can make $ with them.

Watson is impressive, but you can't build a company on just Watson. This author has some great ideas, yet the phrase that keeps coming to me is internal politics. That corrosive reality has & will kill companies, and it will kill IBM unless it is dealt with.

Turn-arounds are possible (look at MS), but they are hard and dangerous. Hope IBM can figure it out...

[Nov 03, 2018] The evaluation system in which there was ALWAYS a "top 10 percent" and a "bottom ten percent" is sociopathic in it's nature

Notable quotes:
"... Four years in GTS ... joined via being outsourced to IBM by my previous employer. Left GTS after 4 years. ..."
"... The IBM way of life was throughout the Oughts and the Teens an utter and complete failure from the perspective of getting work done right and using people to their appropriate and full potential. ..."
"... As a GTS employee, professional technical training was deemed unnecessary, hence I had no access to any unless I paid for it myself and used my personal time ... the only training available was cheesy presentations or other web based garbage from the intranet, or casual / OJT style meetings with other staff who were NOT professional or expert trainers. ..."
"... As a GTS employee, I had NO access to the expert and professional tools that IBM fricking made and sold to the same damn customers I was supposed to be supporting. Did we have expert and professional workflow / document management / ITIL aligned incident and problem management tools? NO, we had fricking Lotus Notes and email. Instead of upgrading to the newest and best software solutions for data center / IT management & support, we degraded everything down the simplest and least complex single function tools that no "best practices" organization on Earth would ever consider using. ..."
"... And the people management paradigm ... employees ranked annually not against a static or shared goal or metric, but in relation to each other, and there was ALWAYS a "top 10 percent" and a "bottom ten percent" required by upper management ... a system that was sociopathic in it's nature because it encourages employees to NOT work together ... by screwing over one's coworkers, perhaps by not giving necessary information, timely support, assistance as needed or requested, one could potentially hurt their performance and make oneself look relatively better. That's a self-defeating system and it was encouraged by the way IBM ran things. ..."
Nov 03, 2018 | www.zdnet.com

Gravyboat McGee , Wednesday, April 26, 2017 9:00 PM

Four years in GTS ... joined via being outsourced to IBM by my previous employer. Left GTS after 4 years.

The IBM way of life was throughout the Oughts and the Teens an utter and complete failure from the perspective of getting work done right and using people to their appropriate and full potential. I went from a multi-disciplinary team of engineers working across technologies to support corporate needs in the IT environment to being siloed into a single-function organization.

My first year of on-boarding with IBM was spent deconstructing application integration and cross-organizational structures of support and interwork that I had spent 6 years building and maintaining. Handing off different chunks of work (again, before the outsourcing, an Enterprise solution supported by one multi-disciplinary team) to different IBM GTS work silos that had no physical special relationship and no interworking history or habits. What we're talking about here is the notion of "left hand not knowing what the right hand is doing" ...

THAT was the IBM way of doing things, and nothing I've read about them over the past decade or so tells me it has changed.

As a GTS employee, professional technical training was deemed unnecessary, hence I had no access to any unless I paid for it myself and used my personal time ... the only training available was cheesy presentations or other web based garbage from the intranet, or casual / OJT style meetings with other staff who were NOT professional or expert trainers.

As a GTS employee, I had NO access to the expert and professional tools that IBM fricking made and sold to the same damn customers I was supposed to be supporting. Did we have expert and professional workflow / document management / ITIL aligned incident and problem management tools? NO, we had fricking Lotus Notes and email. Instead of upgrading to the newest and best software solutions for data center / IT management & support, we degraded everything down the simplest and least complex single function tools that no "best practices" organization on Earth would ever consider using.

And the people management paradigm ... employees ranked annually not against a static or shared goal or metric, but in relation to each other, and there was ALWAYS a "top 10 percent" and a "bottom ten percent" required by upper management ... a system that was sociopathic in it's nature because it encourages employees to NOT work together ... by screwing over one's coworkers, perhaps by not giving necessary information, timely support, assistance as needed or requested, one could potentially hurt their performance and make oneself look relatively better. That's a self-defeating system and it was encouraged by the way IBM ran things.

The "not invented here" ideology was embedded deeply in the souls of all senior IBMers I ever met or worked with ... if you come on board with any outside knowledge or experience, you must not dare to say "this way works better" because you'd be shut down before you could blink. The phrase "best practices" to them means "the way we've always done it".

IBM gave up on innovation long ago. Since the 90's the vast majority of their software has been bought, not built. Buy a small company, strip out the innovation, slap an IBM label on it, sell it as the next coming of Jesus even though they refuse to expend any R&D to push the product to the next level ... damn near everything IBM sold was gentrified, never cutting edge.

And don't get me started on sales practices ... tell the customer how product XYZ is a guaranteed moonshot, they'll be living on lunar real estate in no time at all, and after all the contracts are signed hand the customer a box of nuts & bolts and a letter telling them where they can look up instructions on how to build their own moon rocket. Or for XX dollars more a year, hire a Professional Services IBMer to build it for them.

I have no sympathy for IBM. They need a clean sweep throughout upper management, especially any of the old True Blue hard-core IBMers.

[Nov 02, 2018] How to Recover from an Accidental SSH Disconnection on Linux RoseHosting

Nov 02, 2018 | www.rosehosting.com

... I can get a list of all previous screens using the command:

screen -ls

And this gives me the output as shown here:

Previous Screen Session is Preserved

As you can see, there is a screen session here with the name:

pts-0.test-centos-server

To reconnect to it, just type:

screen -r

And this will take you back to where you were before the SSH connection was terminated! It's an amazing tool that you need to use for all important operations as insurance against accidental terminations.

Manually Detaching Screens

When you break an SSH session, what actually happens is that the screen is automatically detached from it and exists independently. While this is great, you can also detach screens manually and have multiple screens existing at the same time.

For example, to detach a screen just type:

screen -d

And the current screen will be detached and preserved. However, all the processes inside it are still running, and all the states are preserved:

Manually Detaching Screens

You can re-attach to a screen at any time using the "screen -r" command. To connect to a specific screen instead of the most recent, use:

screen -r [screenname]
Changing the Screen Names to Make Them More Relevant

By default, the screen names don't mean much. And when you have a bunch of them present, you won't know which screens contain which processes. Fortunately, renaming a screen is easy when inside one. Just type:

ctrl+a :

We saw in the previous article that "ctrl+a" is the trigger condition for screen commands. The colon (:) will take you to the bottom of the screen where you can type commands. To rename, use:

sessionname [newscreenname]

As shown here:

Change the Session Name

And now when you detach the screen, it will show with the new name like this:

New Session Name Created

Now you can have as many screens as you want without getting confused about which one is which!


If you are one of our Managed VPS hosting clients, we can do all of this for you. Simply contact our system administrators and they will respond to your request as soon as possible.

If you liked this blog post on how to recover from an accidental SSH disconnection on Linux, please share it with your friends on social media networks, or if you have any question regarding this blog post, simply leave a comment below and we will answer it. Thanks!

[Nov 02, 2018] The D in Systemd stands for 'Dammmmit!' A nasty DHCPv6 packet can pwn a vulnerable Linux box by Shaun Nichols

Notable quotes:
"... Hole opens up remote-code execution to miscreants – or a crash, if you're lucky ..."
"... You can use NAT with IPv6. ..."
Oct 26, 2018 | theregister.co.uk

Hole opens up remote-code execution to miscreants – or a crash, if you're lucky A security bug in Systemd can be exploited over the network to, at best, potentially crash a vulnerable Linux machine, or, at worst, execute malicious code on the box.

The flaw therefore puts Systemd-powered Linux computers – specifically those using systemd-networkd – at risk of remote hijacking: maliciously crafted DHCPv6 packets can try to exploit the programming cockup and arbitrarily change parts of memory in vulnerable systems, leading to potential code execution. This code could install malware, spyware, and other nasties, if successful.

The vulnerability – which was made public this week – sits within the written-from-scratch DHCPv6 client of the open-source Systemd management suite, which is built into various flavors of Linux.

This client is activated automatically if IPv6 support is enabled, and relevant packets arrive for processing. Thus, a rogue DHCPv6 server on a network, or in an ISP, could emit specially crafted router advertisement messages that wake up these clients, exploit the bug, and possibly hijack or crash vulnerable Systemd-powered Linux machines.

Here's the Red Hat Linux summary :

systemd-networkd is vulnerable to an out-of-bounds heap write in the DHCPv6 client when handling options sent by network adjacent DHCP servers. A attacker could exploit this via malicious DHCP server to corrupt heap memory on client machines, resulting in a denial of service or potential code execution.

Felix Wilhelm, of the Google Security team, was credited with discovering the flaw, designated CVE-2018-15688 . Wilhelm found that a specially crafted DHCPv6 network packet could trigger "a very powerful and largely controlled out-of-bounds heap write," which could be used by a remote hacker to inject and execute code.

"The overflow can be triggered relatively easy by advertising a DHCPv6 server with a server-id >= 493 characters long," Wilhelm noted.

In addition to Ubuntu and Red Hat Enterprise Linux, Systemd has been adopted as a service manager for Debian, Fedora, CoreOS, Mint, and SUSE Linux Enterprise Server. We're told RHEL 7, at least, does not use the vulnerable component by default.

Systemd creator Lennart Poettering has already published a security fix for the vulnerable component – this should be weaving its way into distros as we type.

If you run a Systemd-based Linux system, and rely on systemd-networkd, update your operating system as soon as you can to pick up the fix when available and as necessary.

The bug will come as another argument against Systemd as the Linux management tool continues to fight for the hearts and minds of admins and developers alike. Though a number of major admins have in recent years adopted and championed it as the replacement for the old Init era, others within the Linux world seem to still be less than impressed with Systemd and Poettering's occasionally controversial management of the tool. ® Page:

2 3 Next →

Oh Homer , 6 days

Meh

As anyone who bothers to read my comments (BTW "hi" to both of you) already knows, I despise systemd with a passion, but this one is more an IPv6 problem in general.

Yes this is an actual bug in networkd, but IPv6 seems to be far more bug prone than v4, and problems are rife in all implementations. Whether that's because the spec itself is flawed, or because nobody understands v6 well enough to implement it correctly, or possibly because there's just zero interest in making any real effort, I don't know, but it's a fact nonetheless, and my primary reason for disabling it wherever I find it. Which of course contributes to the "zero interest" problem that perpetuates v6's bug prone condition, ad nauseam.

IPv6 is just one of those tech pariahs that everyone loves to hate, much like systemd, albeit fully deserved IMO.

Oh yeah, and here's the obligatory "systemd sucks". Personally I always assumed the "d" stood for "destroyer". I believe the "IP" in "IPv6" stands for "Idiot Protocol".

Anonymous Coward , 6 days
Re: Meh

"nonetheless, and my primary reason for disabling it wherever I find it. "

The very first guide I read to hardening a system recommended disabling services you didn't need and emphasized IPV6 for the reasons you just stated.

Wasn't there a bux in Xorg reported recently as well?

https://www.theregister.co.uk/2018/10/25/x_org_server_vulnerability/

"FreeDesktop.org Might Formally Join Forces With The X.Org Foundation"

https://www.phoronix.com/scan.php?page=news_item&px=FreeDesktop-org-Xorg-Forces

Also, does this mean that Facebook was vulnerable to attack, again?

"Simply put, you could say Facebook loves systemd."

https://www.phoronix.com/scan.php?page=news_item&px=Facebook-systemd-2018

Jay Lenovo , 6 days
Re: Meh

IPv6 and SystemD: Forced industry standard diseases that requires most of us to bite our lips and bear it.

Fortunately, IPv6 by lack of adopted use, limits the scope of this bug.

vtcodger , 6 days
Re: Meh
Fortunately, IPv6 by lack of adopted use, limits the scope of this bug.

Yeah, fortunately IPv6 is only used by a few fringe organizations like Google and Microsoft.

Seriously, I personally want nothing to do with either systemd or IPv6. Both seem to me to fall into the bin labeled "If it ain't broke, let's break it" But still it's troubling that things that some folks regard as major system components continue to ship with significant security flaws. How can one trust anything connected to the Internet that is more sophisticated and complex than a TV streaming box?

DougS , 6 days
Re: Meh

Was going to say the same thing, and I disable IPv6 for the exact same reason. IPv6 code isn't as well tested, as well audited, or as well targeted looking for exploits as IPv4. Stuff like this only proves that it was smart to wait, and I should wait some more.

Nate Amsden , 6 days
Re: Meh

Count me in the camp of who hates systemd(hates it being "forced" on just about every distro, otherwise wouldn't care about it - and yes I am moving my personal servers to Devuan, thought I could go Debian 7->Devuan but turns out that may not work, so I upgraded to Debian 8 a few weeks ago, and will go to Devuan from there in a few weeks, upgraded one Debian 8 to Devuan already 3 more to go -- Debian user since 1998), when reading this article it reminded me of

https://www.theregister.co.uk/2017/06/29/systemd_pwned_by_dns_query/

bombastic bob , 6 days
The gift that keeps on giving (systemd) !!!

This makes me glad I'm using FreeBSD. The Xorg version in FreeBSD's ports is currently *slightly* older than the Xorg version that had that vulnerability in it. AND, FreeBSD will *NEVER* have systemd in it!

(and, for Linux, when I need it, I've been using Devuan)

That being said, the whole idea of "let's do a re-write and do a 'systemd' instead of 'system V init' because WE CAN and it's OUR TURN NOW, 'modern' 'change for the sake of change' etc." kinda reminds me of recent "update" problems with Win-10-nic...

Oh, and an obligatory Schadenfreude laugh: HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA HA!!!!!!!!!!!!!!!!!!!

Long John Brass , 6 days
Re: The gift that keeps on giving (systemd) !!!

Finally got all my machines cut over from Debian to Devuan.

Might spin a FreeBSD system up in a VM and have a play.

I suspect that the infestation of stupid into the Linux space won't stop with or be limited to SystemD. I will wait and watch to see what damage the re-education gulag has done to Sweary McSwearFace (Mr Torvalds)

Dan 55 , 6 days
Re: Meh

I despise systemd with a passion, but this one is more an IPv6 problem in general.

Not really, systemd has its tentacles everywhere and runs as root. Exploits which affect systemd therefore give you the keys to the kingdom.

Orv , 3 days
Re: Meh
Not really, systemd has its tentacles everywhere and runs as root.

Yes, but not really the problem in this case. Any DHCP client is going to have to run at least part of the time as root. There's not enough nuance in the Linux privilege model to allow it to manipulate network interfaces, otherwise.

4 1
Long John Brass , 3 days
Re: Meh
Yes, but not really the problem in this case. Any DHCP client is going to have to run at least part of the time as root. There's not enough nuance in the Linux privilege model to allow it to manipulate network interfaces, otherwise.

Sorry but utter bullshit. You can if you are so inclined you can use the Linux Capabilities framework for this kind of thing. See https://wiki.archlinux.org/index.php/capabilities

3 0
JohnFen , 6 days
Yay for me

"If you run a Systemd-based Linux system"

I remain very happy that I don't use systemd on any of my machines anymore. :)

"others within the Linux world seem to still be less than impressed with Systemd"

Yep, I'm in that camp. I gave it a good, honest go, but it increased the amount of hassle and pain of system management without providing any noticeable benefit, so I ditched it.

ElReg!comments!Pierre , 2 days
Re: Time to troll

> Just like it's entirely possible to have a Linux system without any GNU in it

Just like it's possible to have a GNU system without Linux on it - ho well as soon as GNU MACH is finally up to the task ;-)

On the systemd angle, I, too, am in the process of switching all my machines from Debian to Devuan but on my personnal(*) network a few systemd-infected machines remain, thanks to a combination of laziness from my part and stubborn "systemd is quite OK" attitude from the raspy foundation. That vuln may be the last straw : one on the aforementionned machines sits on my DMZ, chatting freely with the outside world. Nothing really crucial on it, but i'd hate it if it became a foothold for nasties on my network.

(*) policy at work is RHEL, and that's negociated far above my influence level, but I don't really care as all my important stuff runs on Z/OS anyway ;-) . Ok we have to reboot a few VMs occasionnally when systemd throws a hissy fit -which is surprisingly often for an "enterprise" OS -, but meh.

Destroy All Monsters , 5 days
Re: Not possible

This code is actually pretty bad and should raise all kinds of red flags in a code review.

Anonymous Coward , 5 days
Re: Not possible

ITYM Lennart

Christian Berger , 5 days
Re: Not possible

"This code is actually pretty bad and should raise all kinds of red flags in a code review."

Yeah, but for that you need people who can do code reviews, and also people who can accept criticism. That also means saying "no" to people who are bad at coding, and saying that repeatedly if they don't learn.

SystemD seems to be the area where people gather who want to get code in for their resumes, not for people who actually want to make the world a better place.

26 1
jake , 6 days
There is a reason ...

... that an init, traditionally, is a small bit of code that does one thing very well. Like most of the rest of the *nix core utilities. All an init should do is start PID1, set run level, spawn a tty (or several), handle a graceful shutdown, and log all the above in plaintext to make troubleshooting as simplistic as possible. Anything else is a vanity project that is best placed elsewhere, in it's own stand-alone code base.

Inventing a clusterfuck init variation that's so big and bulky that it needs to be called a "suite" is just asking for trouble.

IMO, systemd is a cancer that is growing out of control, and needs to be cut out of Linux before it infects enough of the system to kill it permanently.

AdamWill , 6 days
Re: There is a reason ...

That's why systemd-networkd is a separate, optional component, and not actually part of the init daemon at all. Most systemd distros do not use it by default and thus are not vulnerable to this unless the user actively disables the default network manager and chooses to use networkd instead.

Anonymous Coward , 4 days
Re: There is a reason ...

"Just go install a default Fedora or Ubuntu system and check for yourself: you'll have systemd, but you *won't* have systemd-networkd running."

Funny that I installed ubuntu 18.04 a few weeks ago and the fucking thing installed itself then! ( and was a fucking pain to remove).

LP is a fucking arsehole.

Orv , 3 days
Re: There is a reason ...
Pardon my ignorance (I don't use a distro with systemd) why bother with networkd in the first place if you don't have to use it.

Mostly because the old-style init system doesn't cope all that well with systems that move from network to network. It works for systems with a static IP, or that do a DHCP request at boot, but it falls down on anything more dynamic.

In order to avoid restarting the whole network system every time they switch WiFi access points, people have kludged on solutions like NetworkManager. But it's hard to argue it's more stable or secure than networkd. And this is always going to be a point of vulnerability because anything that manipulates network interfaces will have to be running as root.

These days networking is essential to the basic functionality of most computers; I think there's a good argument that it doesn't make much sense to treat it as a second-class citizen.

AdamWill , 2 days
Re: There is a reason ...

"Funny that I installed ubuntu 18.04 a few weeks ago and the fucking thing installed itself then! ( and was a fucking pain to remove)."

So I looked into it a bit more, and from a few references at least, it seems like Ubuntu has a sort of network configuration abstraction thingy that can use both NM and systemd-networkd as backends; on Ubuntu desktop flavors NM is usually the default, but apparently for recent Ubuntu Server, networkd might indeed be the default. I didn't notice that as, whenever I want to check what's going on in Ubuntu land, I tend to install the default desktop spin...

"LP is a fucking arsehole."

systemd's a lot bigger than Lennart, you know. If my grep fu is correct, out of 1543 commits to networkd, only 298 are from Lennart...

1 0
alain williams , 6 days
Old is good

in many respects when it comes to software because, over time, the bugs will have been found and squashed. Systemd brings in a lot of new code which will, naturally, have lots of bugs that will take time to find & remove. This is why we get problems like this DHCP one.

Much as I like the venerable init: it did need replacing. Systemd is one way to go, more flexible, etc, etc. Something event driven is a good approach.

One of the main problems with systemd is that it has become too big, slurped up lots of functionality which has removed choice, increased fragility. They should have concentrated on adding ways of talking to existing daemons, eg dhcpd, through an API/something. This would have reused old code (good) and allowed other implementations to use the API - this letting people choose what they wanted to run.

But no: Poettering seems to want to build a Cathedral rather than a Bazzar.

He appears to want to make it his way or no way. This is bad, one reason that *nix is good is because different solutions to a problem have been able to be chosen, one removed and another slotted in. This encourages competition and the 'best of breed' comes out on top. Poettering is endangering that process.

Also: he refusal to accept patches to let it work on non-Linux Unix is just plain nasty.

oiseau , 4 days
Re: Old is good

Hello:

One of the main problems with systemd is that it has become too big, slurped up lots of functionality which has removed choice, increased fragility.

IMO, there is a striking paralell between systemd and the registry in Windows OSs.

After many years of dealing with the registry (W98 to XPSP3) I ended up seeing the registry as a sort of developer sanctioned virus running inside the OS, constantly changing and going deeper and deeper into the OS with every iteration and as a result, progressively putting an end to the possibility of knowing/controlling what was going on inside your box/the OS.

Years later, when I learned about the existence of systemd (I was already running Ubuntu) and read up on what it did and how it did it, it dawned on me that systemd was nothing more than a registry class virus and it was infecting Linux_land at the behest of the developers involved.

So I moved from Ubuntu to PCLinuxOS and then on to Devuan.

Call me paranoid but I am convinced that there are people both inside and outside IT that actually want this and are quite willing to pay shitloads of money for it to happen.

I don't see this MS cozying up to Linux in various ways lately as a coincidence: these things do not happen just because or on a senior manager's whim.

What I do see (YMMV) is systemd being a sort of convergence of Linux with Windows, which will not be good for Linux and may well be its undoing.

Cheers,

O.

Rich 2 , 4 days
Re: Old is good

"Also: he refusal to accept patches to let it work on non-Linux Unix is just plain nasty"

Thank goodness this crap is unlikely to escape from Linux!

By the way, for a systemd-free Linux, try void - it's rather good.

Michael Wojcik , 3 days
Re: Old is good

Much as I like the venerable init: it did need replacing.

For some use cases, perhaps. Not for any of mine. SysV init, or even BSD init, does everything I need a Linux or UNIX init system to do. And I don't need any of the other crap that's been built into or hung off systemd, either.

Orv , 3 days
Re: Old is good

BSD init and SysV init work pretty darn well for their original purpose -- servers with static IP addresses that are rebooted no more than once in a fortnight. Anything more dynamic starts to give it trouble.

Chairman of the Bored , 6 days
Too bad Linus swore off swearing

Situations like this go beyond a little "golly gee, I screwed up some C"...

jake , 6 days
Re: Too bad Linus swore off swearing

Linus doesn't care. systemd has nothing to do with the kernel ... other than the fact that the lead devs for systemd have been banned from working on the kernel because they don't play nice with others.

JLV , 6 days
how did it get to this?

I've been using runit, because I am too lazy and clueless to write init scripts reliably. It's very lightweight, runs on a bunch of systems and really does one thing - keep daemons up.

I am not saying it's the best - but it looks like it has a very small codebase, it doesn't do much and generally has not bugged me after I configured each service correctly. I believe other systems also exist to avoid using init scripts directly. Not Monit, as it relies on you configuring the daemon start/stop commands elsewhere.

On the other hand, systemd is a massive sprawl, does a lot of things - some of them useful, like dependencies and generally has needed more looking after. Twice I've had errors on a Django server that, after a lot of looking around ended up because something had changed in the, Chef-related, code that's exposed to systemd and esoteric (not emitted by systemd) errors resulted when systemd could not make sense of the incorrect configuration.

I don't hate it - init scripts look a bit antiquated to me and they seem unforgiving to beginners - but I don't much like it. What I certainly do hate is how, in an OS that is supposed to be all about choice, sometime excessively so as in the window manager menagerie, we somehow ended up with one mandatory daemon scheduler on almost all distributions. Via, of all types of dependencies, the GUI layer. For a window manager that you may not even have installed.

Talk about the antithesis of the Unix philosophy of do one thing, do it well.

Oh, then there are also the security bugs and the project owner is an arrogant twat. That too.

Doctor Syntax , 6 days
Re: how did it get to this?

"init scripts look a bit antiquated to me and they seem unforgiving to beginners"

Init scripts are shell scripts. Shell scripts are as old as Unix. If you think that makes them antiquated then maybe Unix-like systems are not for you. In practice any sub-system generally gets its own scripts installed with the rest of the S/W so if being unforgiving puts beginners off tinkering with them so much the better. If an experienced Unix user really needs to modify one of the system-provided scripts their existing shell knowledge will let them do exactly what's needed. In the extreme, if you need to develop a new init script then you can do so in the same way as you'd develop any other script - edit and test from the command line.

33 4
onefang , 6 days
Re: how did it get to this?

"Init scripts are shell scripts."

While generally true, some sysv init style inits can handle init "scripts" written in any language.

sed gawk , 6 days
Re: how did it get to this?

I personally like openrc as an init system, but systemd is a symptom of the tooling problem.

It's for me a retrograde step but again, it's linux, one can, as you and I do, just remove systemd.

There are a lot of people in the industry now who don't seem able to cope with shell scripts nor are minded to research the arguments for or against shell as part of a unix style of system design.

In conclusion, we are outnumbered, but it will eventually collapse under its own weight and a worthy successor shall rise, perhaps called SystemV, might have to shorten that name a bit.

AdamWill , 6 days
Just about nothing actually uses networkd

"In addition to Ubuntu and Red Hat Enterprise Linux, Systemd has been adopted as a service manager for Debian, Fedora, CoreOS, Mint, and SUSE Linux Enterprise Server. We're told RHEL 7, at least, does not use the vulnerable component by default."

I can tell you for sure that no version of Fedora does, either, and I'm fairly sure that neither does Debian, SLES or Mint. I don't know anything much about CoreOS, but https://coreos.com/os/docs/latest/network-config-with-networkd.html suggests it actually *might* use systemd-networkd.

systemd-networkd is not part of the core systemd init daemon. It's an optional component, and most distros use some other network manager (like NetworkManager or wicd) by default.

Christian Berger , 5 days
The important word here is "still"

I mean commercial distributions seem to be particularly interested in trying out new things that can increase their number of support calls. It's probably just that networkd is either to new and therefore not yet in the release, or still works so badly even the most rudimentary tests fail.

There is no reason to use that NTP daemon of systemd, yet more and more distros ship with it enabled, instead of some sane NTP-server.

NLCSGRV , 6 days
The Curse of Poettering strikes again.
_LC_ , 6 days
Now hang on, please!

Ser iss no neet to worry, systemd will becum stable soon after PulseAudio does.

Ken Hagan , 6 days
Re: Now hang on, please!

I won't hold my breath, then. I have a laptop at the moment that refuses to boot because (as I've discovered from looking at the journal offline) pulseaudio is in an infinite loop waiting for the successful detection of some hardware that, presumably, I don't have.

I imagine I can fix it by hacking the file-system (offline) so that fuckingpulse is no longer part of the boot configuration, but I shouldn't have to. A decent init system would be able to kick of everything else in parallel and if one particular service doesn't come up properly then it just logs the error. I *thought* that was one of the claimed advantages of systemd, but apparently that's just a load of horseshit.

26 0
Obesrver1 , 5 days
Reason for disabling IVP6

That it punches thru NAT routers enabling all your little goodies behind them as directly accessible.

MS even supplies tunneling (Ivp4 to Ivp6) so if using Linux in a VM on a MS system you may still have it anyway.

NAT was always recommended to be used in hardening your system, I prefer to keep all my idIoT devices behind one.

As they are just Idiot devices.

In future I will need a NAT that acts as a DNS and offers some sort of solution to keeping Ivp4.

Orv , 3 days
Re: Reason for disabling IVP6

My NAT router statefully firewalls incoming IPv6 by default, which I consider equivalently secure. NAT adds security mostly by accident, because it de-facto adds a firewall that blocks incoming packets. It's not the address translation itself that makes things more secure, it's the inability to route in from the outside.

dajames , 3 days
Re: Reason for disabling IVP6

You can use NAT with IPv6.

You can, but why would you want to.

NAT is schtick for connecting a whole LAN to a WAN using a single IPv4 address (useful with IPv4 because most ISPs don't give you a /24 when you sign up). If you have a native IPv6 address you'll have something like 2^64 addresses, so machines on your LAN can have an actual WAN-visible address of their own without needing a trick like NAT.

Using NAT with IPv6 is just missing the point.

JohnFen , 3 days
Re: Reason for disabling IVP6

"so machines on your LAN can have an actual WAN-visible address of their own without needing a trick like NAT."

Avoiding that configuration is exactly the use case for using NAT with IPv6. As others have pointed out, you can accomplish the same thing with IPv6 router configuration, but NAT is easier in terms of configuration and maintenance. Given that, and assuming that you don't want to be able to have arbitrary machines open ports that are visible to the internet, then why not use NAT?

Also, if your goal is to make people more likely to move to IPv6, pointing out IPv4 methods that will work with IPv6 (even if you don't consider them optimal) seems like a really, really good idea. It eases the transition.

Destroy All Monsters , 5 days
Please El Reg these stories make ma rage at breakfast, what's this?

The bug will come as another argument against Systemd as the Linux management tool continues to fight for the hearts and minds of admins and developers alike.

Less against systemd (which should get attacked on the design & implementation level) or against IPv6 than against the use of buffer-overflowable languages in 2018 in code that processes input from the Internet (it's not the middle ages anymore) or at least very hard linting of the same.

But in the end, what did it was a violation of the Don't Repeat Yourself principle and lack of sufficently high-level datastructures. Pointer into buffer, and the remaining buffer length are two discrete variables that need to be updated simultaneously to keep the invariant and this happens in several places. This is just a catastrophe waiting to happen. You forget to update it once, you are out! Use structs and functions updating the structs correctly.

And use assertions in the code , this stuff all seems disturbingly assertion-free.

Excellent explanation by Felix Wilhelm:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1795921

The function receives a pointer to the option buffer buf, it's remaining size buflen and the IA to be added to the buffer. While the check at (A) tries to ensure that the buffer has enough space left to store the IA option, it does not take the additional 4 bytes from the DHCP6Option header into account (B). Due to this the memcpy at (C) can go out-of-bound and *buflen can underflow [i.e. you suddenly have a gazillion byte buffer, Ed.] in (D) giving an attacker a very powerful and largely controlled OOB heap write starting at (E).

TheSkunkyMonk , 5 days
Init is 1026 lines of code in one file and it works great.
Anonymous Coward , 5 days
"...and Poettering's occasionally controversial management of the tool."

Shouldn't that be "...Potterings controversial management as a tool."?

clocKwize , 4 days
Re: Contractor rights

why don't we stop writing code in languages that make it easy to screw up so easily like this?

There are plenty about nowadays, I'd rather my DHCP client be a little bit slower at processing packets if I had more confidence it would not process then incorrectly and execute code hidden in said packets...

Anonymous Coward , 4 days
Switch, as easy as that

The circus that is called "Linux" have forced me to Devuan and the likes however the circus is getting worse and worse by the day, thus I have switched to the BSD world, I will learn that rather than sit back and watch this unfold As many of us have been saying, the sudden switch to SystemD was rather quick, perhaps you guys need to go investigate why it really happened, don't assume you know, go dig and you will find the answers, it's rather scary, thus I bid the Linux world a farewell after 10 years of support, I will watch the grass dry out from the other side of the fence, It was destined to fail by means of infiltration and screw it up motive(s) on those we do not mention here.

oiseau , 3 days
Re: Switch, as easy as that

Hello:

As many of us have been saying, the sudden switch to SystemD was rather quick, perhaps you guys need to go investigate why it really happened, don't assume you know, go dig and you will find the answers, it's rather scary ...

Indeed, it was rather quick and is very scary.

But there's really no need to dig much, just reason it out.

It's like a follow the money situation of sorts.

I'll try to sum it up in three short questions:

Q1: Hasn't the Linux philosophy (programs that do one thing and do it well) been a success?

A1: Indeed, in spite of the many init systems out there, it has been a success in stability and OS management. And it can easily be tested and debugged, which is an essential requirement.

Q2: So what would Linux need to have the practical equivalent of the registry in Windows for?

A2: So that whatever the registry does in/to Windows can also be done in/to Linux.

Q3: I see. And just who would want that to happen? Makes no sense, it is a huge step backwards.

A3: ....

Cheers,

O.

Dave Bell , 4 days
Reporting weakness

OK, so I was able to check through the link you provided, which says "up to and including 239", but I had just installed a systemd update and when you said there was already a fix written, working it's way through the distro update systems, all I had to do was check my log.

Linux Mint makes it easy.

But why didn't you say something such as "reported to affect systemd versions up to and including 239" and then give the link to the CVE? That failure looks like rather careless journalism.

W.O.Frobozz , 3 days
Hmm.

/sbin/init never had these problems. But then again /sbin/init didn't pretend to be the entire operating system.

[Nov 01, 2018] IBM was doing their best to demoralize their staff (and contractors) and annoy their customers as much as possible!

Notable quotes:
"... I worked as a contractor for IBM's IGS division in the late '90s and early 2000s at their third biggest customer, and even then, IBM was doing their best to demoralize their staff (and contractors) and annoy their customers as much as possible! ..."
"... IBM charged themselves 3x the actual price to customers for their ThinkPads at the time! This meant that I never had a laptop or desktop PC from IBM in the 8 years I worked there. If it wasn't for the project work I did I would not have had a PC to work on! ..."
"... What was strange is that every single time I got a pay cut, IBM would then announce that they had bought a new company! I would have quit long before I did, but I was tied to them while waiting for my Green Card to be approved. I know that raises are few in the current IBM for normal employees and that IBM always pleads poverty for any employee request. Yet, they somehow manage to pay billions of dollars for a new company. Strange that, isn't it? ..."
"... I moved to the company that had won the contract and regret not having the chance to tell that IBM manager what I thought about him and where he could stick the new laptop. ..."
"... After that experience I decided to never work for them in any capacity ever again. I feel pity for the current Red Hat employees and my only advice to them is to get out while they can. "DON'T FOLLOW THE RED HAT TO HELL" ..."
Nov 01, 2018 | theregister.co.uk

Edwin Tumblebunny Ashes to ashes, dust to dust - Red Hat is dead.

Red Hat will be a distant memory in a few years as it gets absorbed by the abhorrent IBM culture and its bones picked clean by the IBM beancounters. Nothing good ever happens to a company bought by IBM.

I worked as a contractor for IBM's IGS division in the late '90s and early 2000s at their third biggest customer, and even then, IBM was doing their best to demoralize their staff (and contractors) and annoy their customers as much as possible!

Some examples:

The on-site IBM employees (and contractors) had to use Lotus Notes for email. That was probably the worst piece of software I have ever used - I think baboons on drugs could have done a better design job. IBM set up a T1 (1.54 Mbps) link between the customer and the local IBM hub for email, etc. It sounds great until you realize there were over 150 people involved and due to the settings of Notes replication, it could often take over an hour to actually download email to read.

To do my job I needed to install some IBM software. My PC did not have enough disk space for this software as well as the other software I needed. Rather than buy me a bigger hard disk I had to spend 8 hours a week installing and reinstalling software to do my job.

I waited three months for a $50 stick of memory to be approved. When it finally arrived my machine had been changed out (due to a new customer project) and the memory was not compatible! Since I worked on a lot of projects I often had machines supplied by the customer on my desk. So, I would use one of these as my personal PC and would get an upgrade when the next project started!

I was told I could not be supplied with a laptop or desktop from IBM as they were too expensive (my IBM division did not want to spend money on anything). IBM charged themselves 3x the actual price to customers for their ThinkPads at the time! This meant that I never had a laptop or desktop PC from IBM in the 8 years I worked there. If it wasn't for the project work I did I would not have had a PC to work on!

IBM has many strange and weird processes that allow them to circumvent the contract they have with their preferred contractor companies. This meant that for a number of years I ended up getting a pay cut. What was strange is that every single time I got a pay cut, IBM would then announce that they had bought a new company! I would have quit long before I did, but I was tied to them while waiting for my Green Card to be approved. I know that raises are few in the current IBM for normal employees and that IBM always pleads poverty for any employee request. Yet, they somehow manage to pay billions of dollars for a new company. Strange that, isn't it?

Eventually I was approved to get a laptop and excitedly watched it move slowly through the delivery system. I got confused when it was reported as delivered to Ohio rather than my work (not in Ohio). After some careful searching I discovered that my manager and his wife both worked for IBM from their home in, yes you can guess, Ohio. It looked like he had redirected my new laptop for his own use and most likely was going to send me his old one and claim it was a new one. I never got the chance to confront him about it, though, as IBM lost the contract with the customer that month and before the laptop should have arrived IBM was out! I moved to the company that had won the contract and regret not having the chance to tell that IBM manager what I thought about him and where he could stick the new laptop.

After that experience I decided to never work for them in any capacity ever again. I feel pity for the current Red Hat employees and my only advice to them is to get out while they can. "DON'T FOLLOW THE RED HAT TO HELL"

Certain Hollywood stars seem to be psychic types : https://twitter.com/JimCarrey/status/1057328878769721344

rmstock , 2 days

Re: "DON'T FOLLOW THE RED HAT TO HELL"

I sense that a global effort is ongoing to shutdown open source software by brute force. First, the enforcement of the EU General Data Protection Regulation (GDPR) by ICANN.org to enable untraceable takeovers of domains. Microsoft buying github. Linus Torvalds forced out of his own Linux kernel project because of the Code of Conduct and now IBM buying RedHat. I wrote the following at https://lulz.com/linux-devs-threaten-killswitch-coc-controversy-1252/ "Torvalds should lawyer up. The problems are the large IT Tech firms who platinum donated all over the place in Open Source land. When IBM donated with 1 billion USD to Linux in 2000 https://itsfoss.com/ibm-invest-1-billion-linux/ a friend who vehemently was against the GPL and what Torvalds was doing, told me that in due time OSS would simply just go away.

These Community Organizers, not Coders per se, are on a mission to overtake and control the Linux Foundation, and if they can't, will search and destroy all of it, even if it destroys themselves. Coraline is merely a expendable pion here. Torvalds is now facing unjust confrontations and charges resembling the nomination of Judge Brett Kavanaugh. Looking at the CoC document it even might have been written by a Google executive, who themselves currently are facing serious charges and lawsuits from their own Code of Conduct. See theintercept.com, their leaked video the day after the election of 2016. They will do anything to pursue this. However to pursue a personal bias or agenda furnishing enactments or acts such as to, omit contradicting facts (code), commit perjury, attend riots and harassments, cleanse Internet archives and search engines of exculpatory evidence and ultimately hire hit-men to exterminate witnesses of truth (developers), in an attempt to elevate bias as fabricated fact (code) are crimes and should be prosecuted accordingly."

[Nov 01, 2018] Will Red Hat Survive IBM Ownership

It does not matter if somebody "stresses independence word are cheap. The mere fact that this is not IBM changes relationships. also IBM executives need to show "leadership" and that entails some "general direction" for Red Hat from now on. At least with Microsoft and HP relationship will be much cooler, then before.
Also IBM does not like "charity" projects like CentOS and that will be affected too, no matter what executives tell you right now. Paradoxically this greatly strengthen the position of Oracle Linux.
The status of IBM software in corporate world 9outside finance companies) is low and their games with licenses (licensing products per core, etc) are viewed by most of their customers as despicable. This was one of the reasons IBM lost it share in enterprise software. For example, greed in selling Tivoli more or less killed the software. All credits to Lou Gerstner who initially defined this culture of relentless outsource and cult of the "bottom line" (which was easy for his because he does not understood technology at all). His successor was even more active in driving company into the ground. R ampant cost-arbitrage driven offshoring has left a legacy of dissatisfied customers. Most projects are over priced. and most of those which were priced more or less on industry level had bad quality results and costs overruns.
IBM cut severance pay to 1 month, is firing people left and right and is insensitive to the fired employees and the result is enormous negativity for them. Good people are scared to work for them and people are asking tough questions.
It's strategy since Gerstner. Palmisano (this guys with history diploma witched into Cybersecurity after retirement in 2012 and led Obama cybersecurity commission) & Ginni Rometty was completely neoliberal mantra "share holder value", grow earnings per share at all costs. They all manage IBM for financial appearance rather than the quaility products. There was no focus on breakthrough-innovation, market leadership in mega-trends (like cloud computing) or even giving excellent service to customers.
Ginni Rometty accepted bonuses during times of layoffs and outsourcing.
When company have lost it architectural talent the brand will eventually die. IBM still has very high number of patents every year. It is still a very large company in terms of revenue and number of employees. However, there are strong signs that its position in the technology industry might deteriorate further.
Nov 01, 2018 | www.itprotoday.com

Cormier also stressed Red Hat independence when we asked how the acquisition would affect ongoing partnerships in place with Amazon Web Services, Microsoft Azure, and other public clouds.

"One of the things you keep seeing in this is that we're going to run Red Hat as a separate entity within IBM," he said. "One of the reasons is business. We need to, and will, remain Switzerland in terms of how we interact with our partners. We're going to continue to prioritize what we do for our partners within our products on a business case perspective, including IBM as a partner. We're not going to do unnatural things. We're going to do the right thing for the business, and most importantly, the customer, in terms of where we steer our products."

Red Hat promises that independence will extend to Red Hat's community projects, such as its freely available Fedora Linux distribution which is widely used by developers. When asked what impact the sale would have on Red Hat maintained projects like Fedora and CentoOS, Cormier replied, "None. We just came from an all-hands meeting from the company around the world for Red Hat, and we got asked this question and my answer was that the day after we close, I don't intend to do anything different or in any other way than we do our every day today. Arvind, Ginnie, Jim, and I have talked about this extensively. For us, it's business as usual. Whatever we were going to do for our road maps as a stand alone will continue. We have to do what's right for the community, upstream, our associates, and our business."

This all sounds good, but as they say, talk is cheap. Six months from now we'll have a better idea of how well IBM can walk the walk and leave Red Hat alone. If it can't and Red Hat doesn't remain clearly distinct from IBM ownership control, then Big Blue will have wasted $33 billion it can't afford to lose and put its future, as well as the future of Red Hat, in jeopardy.

[Nov 01, 2018] 3 scary sysadmin stories

Notable quotes:
"... "It was, it was " ..."
Nov 01, 2018 | opensource.com

The ghost of the failed restore

In a well-known data center (whose name I do not want to remember), one cold October night we had a production outage in which thousands of web servers stopped responding due to downtime in the main database. The database administrator asked me, the rookie sysadmin, to recover the database's last full backup and restore it to bring the service back online.

But, at the end of the process, the database was still broken. I didn't worry, because there were other full backup files in stock. However, even after doing the process several times, the result didn't change.

With great fear, I asked the senior sysadmin what to do to fix this behavior.

"You remember when I showed you, a few days ago, how the full backup script was running? Something about how important it was to validate the backup?" responded the sysadmin.

"Of course! You told me that I had to stay a couple of extra hours to perform that task," I answered.

"Exactly! But you preferred to leave early without finishing that task," he said.

"Oh my! I thought it was optional!" I exclaimed.

"It was, it was "

Moral of the story: Even with the best solution that promises to make the most thorough backups, the ghost of the failed restoration can appear, darkening our job skills, if we don't make a habit of validating the backup every time.

[Oct 30, 2018] Red Hat hired the CentOS developers 4.5-years ago

Oct 30, 2018 | linux.slashdot.org

quantaman ( 517394 ) , Sunday October 28, 2018 @04:22PM ( #57550805 )

Re:Well at least we'll still have Cent ( Score: 4 , Informative)
Fedora is fully owned by Red Hat and CentOS requires the availability of the Red Hat repositories which they aren't obliged to make public to non-customers..

Fedora is fully under Red Hat's control. It's used as a bleeding edge distro for hobbyists and as a testing ground for code before it goes into RHEL. I doubt its going away since it does a great job of establishing mindshare but no business in their right mind is going to run Fedora in production.

But CentOS started as a separate organization with a fairly adversarial relationship to Red Hat since it really is free RHEL which cuts into their actual customer base. They didn't need Red Hat repos back then, just the code which they rebuilt from scratch (which is why they were often a few months behind).

If IBM kills CentOS a new one will pop up in a week, that's the beauty of the GPL.

Luthair ( 847766 ) , Sunday October 28, 2018 @04:22PM ( #57550799 )
Re:Well at least we'll still have Cent ( Score: 3 )

Red Hat hired the CentOS developers 4.5-years ago.

[Oct 30, 2018] We run just about everything on CentOS around here, downstream of RHEL. Should we be worried?

Oct 30, 2018 | arstechnica.com

Muon , Ars Scholae Palatinae 6 hours ago Popular

We run just about everything on CentOS around here, downstream of RHEL. Should we be worried? 649 posts | registered 1/26/2009
brandnewmath , Smack-Fu Master, in training et Subscriptor 6 hours ago Popular
We'll see. Companies in an acquisition always rush to explain how nothing will change to reassure their customers. But we'll see.
Kilroy420 , Ars Tribunus Militum 6 hours ago Popular
Perhaps someone can explain this... Red Hat's revenue and assets barely total about $5B. Even factoring in market share and capitalization, how the hey did IBM come up with $34B cash being a justifiable purchase price??

Honestly, why would Red Hat have said no? 1648 posts | registered 4/3/2012

dorkbert , Ars Tribunus Militum 6 hours ago Popular
My personal observation of IBM over the past 30 years or so is that everything it acquires dies horribly.
barackorama , Smack-Fu Master, in training 6 hours ago
...IBM's own employees see it as a company in free fall. This is not good news.

In other news, property values in Raleigh will rise even more...

Moodyz , Ars Centurion 6 hours ago Popular
Quote:
This is fine

Looking back at what's happened with many of IBM's past acquisitions, I'd say no, not quite fine.
I am not your friend , Wise, Aged Ars Veteran et Subscriptor 6 hours ago Popular
I just can't comprehend that price. Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth.
jandrese , Ars Tribunus Angusticlavius et Subscriptor 6 hours ago Popular
50me12 wrote:
Will IBM even know what to do with them?

IBM has been fumbling around for a while. They didn't know how to sell Watson as they sold it like a weird magical drop in service and it failed repeatedly, where really it should be a long term project that you bring customer's along for the ride...

I had a buddy using their cloud service and they went to spin up servers and IBM was all "no man we have to set them up first".... like that's not cloud IBM...

If IBM can't figure out how to sell its own services I'm not sure the powers that be are capable of getting the job done ever. IBM's own leadership seems incompatible with the state of the world.

IBM basically bought a ton of service contracts for companies all over the world. This is exactly what the suits want: reliable cash streams without a lot of that pesky development stuff.

IMHO this is perilous for RHEL. It would be very easy for IBM to fire most of the developers and just latch on to the enterprise services stuff to milk it till its dry.

skizzerz , Wise, Aged Ars Veteran et Subscriptor 6 hours ago
toturi wrote:
I can only see this as a net positive - the ability to scale legacy mainframes onto "Linux" and push for even more security auditing.

I would imagine the RHEL team will get better funding but I would be worried if you're a centos or fedora user.

I'm nearly certain that IBM's management ineptitude will kill off Fedora and CentOS (or at least severely gimp them compared to how they currently are), not realizing how massively important both of these projects are to the core RHEL product. We'll see RHEL itself suffer as a result.

I normally try to understand things with an open mindset, but in this case, IBM has had too long of a history of doing things wrong for me to trust them. I'll be watching this carefully and am already prepping to move off of my own RHEL servers once the support contract expires in a couple years just in case it's needed.

Iphtashu Fitz , Ars Scholae Palatinae 6 hours ago Popular
50me12 wrote:
Will IBM even know what to do with them?

My previous job (6+ years ago now) was at a university that was rather heavily invested in IBM for a high performance research cluster. It was something around 100 or so of their X-series blade servers, all of which were running Red Hat Linux. It wouldn't surprise me if they decided to acquire Red Hat in large part because of all these sorts of IBM systems that run Red Hat on them.

TomXP411 , Ars Tribunus Angusticlavius 6 hours ago Popular
Iphtashu Fitz wrote:
50me12 wrote:
blockquote> Will IBM even know what to do with them?

My previous job (6+ years ago now) was at a university that was rather heavily invested in IBM for a high performance research cluster. It was something around 100 or so of their X-series blade servers, all of which were running Red Hat Linux. It wouldn't surprise me if they decided to acquire Red Hat in large part because of all these sorts of IBM systems that run Red Hat on them.

That was my thought. IBM wants to own an operating system again. With AIX being relegated to obscurity, buying Red Hat is simpler than creating their own Linux fork.

anon_lawyer , Wise, Aged Ars Veteran 6 hours ago Popular
Valuing Red Hat at $34 billion means valuing it at more than 1/4 of IBMs current market cap. From my perspective this tells me IBM is in even worse shape than has been reported.
dmoan , Ars Centurion 6 hours ago
I am not your friend wrote:
I just can't comprehend that price. Cloud has a rich future, but I didn't even know Red Hat had any presence there, let alone $35 billion worth.

Redhat made 258 million in income last year so they paid over 100 times its net income that's crazy valuation here...

[Oct 30, 2018] I have worked at IBM 17 years and have worried about being layed off for about 11 of them. Moral is in the toilet. Bonuses for the rank and file are in the under 1% range while the CEO gets millions

Notable quotes:
"... Adjusting for inflation, I make $6K less than I did my first day. My group is a handful of people as at least 1/2 have quit or retired. To support our customers, we used to have several people, now we have one or two and if someone is sick or on vacation, our support structure is to hope nothing breaks. ..."
Oct 30, 2018 | features.propublica.org

Buzz , Friday, March 23, 2018 12:00 PM

I've worked there 17 years and have worried about being layed off for about 11 of them. Moral is in the toilet. Bonuses for the rank and file are in the under 1% range while the CEO gets millions. Pay raises have been non existent or well under inflation for years.

Adjusting for inflation, I make $6K less than I did my first day. My group is a handful of people as at least 1/2 have quit or retired. To support our customers, we used to have several people, now we have one or two and if someone is sick or on vacation, our support structure is to hope nothing breaks.

We can't keep millennials because of pay, benefits and the expectation of being available 24/7 because we're shorthanded. As the unemployment rate drops, more leave to find a different job, leaving the old people as they are less willing to start over with pay, vacation, moving, selling a house, pulling kids from school, etc.

The younger people are generally less likely to be willing to work as needed on off hours or to pull work from a busier colleague.

I honestly have no idea what the plan is when the people who know what they are doing start to retire, we are way top heavy with 30-40 year guys who are on their way out, very few of the 10-20 year guys due to hiring freezes and we can't keep new people past 2-3 years. It's like our support business model is designed to fail.

[Oct 30, 2018] Will systemd become standard on mainframes as well?

It will be interesting to see what happens in any case.
Oct 30, 2018 | theregister.co.uk

Doctor Syntax , 1 day

So now it becomes Blue Hat. Will systemd become standard on mainframes as well?
DCFusor , 15 hrs
@Doctor

Maybe we get really lucky and they RIF Lennart Poettering or he quits? I hear IBM doesn't tolerate prima donnas and cults of personality quite as much as RH?

Anonymous Coward , 15 hrs
I hear IBM doesn't tolerate prima donnas and cults of personality quite as much as RH?

Quite the contrary. IBM is run and managed by prima donnas and personality cults.

Waseem Alkurdi , 18 hrs
Re: Poettering

OS/2 and Poettering? Best joke I've ever heard!

(It'd be interesting if somebody locked them both up in an office and see what happens!)

Glen Turner 666 , 16 hrs
Re: Patents

IBM already had access to Red Hat's patents, including for patent defence purposes. Look up "open innovation network".

This acquisition is about: (1) IBM needing growth, or at least a plausible scenario for growth. (2) Red Hat wanting an easy expansion of its sales channels, again for growth. (3) Red Hat stockholders being given an offer they can't refuse.

This acquisition is not about: cultural change at IBM. Which is why the acquisition will 'fail'. The bottom line is that engineering matters at the moment (see: Google, Amazon), and IBM sacked their engineering culture across the past two decades. To be successful IBM need to get that culture back, and acquiring Red Hat gives IBM the opportunity to create a product-building, client-service culture within IBM. Except that IBM aren't taking the opportunity, so there's a large risk the reverse will happen -- the acquisition will destroy Red Hat's engineering- and service-oriented culture.

Anonymous Coward , 1 day
The kraken versus the container ship

This could be interesting: will the systemd kraken manage to wrap its tentacles around the big blue container ship and bring it to a halt, or will the container ship turn out to be well armed and fatally harpoon the kraken (causing much rejoicing in the rest of the Linux world)?

Sitaram Chamarty , 1 day
disappointed...

Honestly, this is a time for optimism: if they manage to get rid of Lennart Poettering, everything else will be tolerable!

dbtx , 1 day
*if*

you can change the past so that a "proper replacement" isn't automatically expected to do lots of things that systemd does. That damage is done. We got "better is worse" and enough people liked it-- good luck trying to go back to "worse is better"

tfb , 13 hrs
I presume they were waiting to see what happened to Solaris. When Oracle bought Sun (presumably the only other company who might have bought them was IBM) there were really three enterprise unixoid platforms: Solaris, AIX and RHEL (there were some smaller ones and some which were clearly dying like HPUX). It seemed likely at the time, but not yet certain, that Solaris was going to die (I worked for Sun at the time this happened and that was my opinion anyway). If Solaris did die, then if one company owned both AIX and RHEL then that company would own the enterprise unixoid market. If Solaris didn't die on the other hand then RHEL would be a lot less valuable to IBM as there would be meaningful competition. So, obviously, they waited to see what would happen.

Well, Solaris is perhaps not technically quite dead yet but certainly is moribund, and IBM now owns both AIX and RHEL & hence the enterprise unixoid market. As an interesting side-note, unless Oracle can keep Solaris on life-support this means that IBM own all-but own Oracle's OS as well ('Oracle Linux' is RHEL with, optionally, some of their own additions to the kernel).

[Oct 30, 2018] IBM's Red Hat acquisition is a 'desperate deal,' says analyst

Notable quotes:
"... "It's a desperate deal by a company that missed the boat for the last five years," the managing director at BTIG said on " Closing Bell ." "I'm not surprised that they bought Red Hat , I'm surprised that it took them so long. They've been behind the cloud eight ball." ..."
Oct 30, 2018 | www.cnbc.com
IBM's $34 billion acquisition of Red Hat is a last-ditch effort by IBM to play catch-up in the cloud industry, analyst Joel Fishbein told CNBC on Monday.

"It's a desperate deal by a company that missed the boat for the last five years," the managing director at BTIG said on " Closing Bell ." "I'm not surprised that they bought Red Hat , I'm surprised that it took them so long. They've been behind the cloud eight ball."

This is IBM's largest deal ever and the third-biggest tech deal in the history of the United States. IBM is paying more than a 60 percent premium for the software maker, but CEO Ginni Rometty told CNBC earlier in the day it was a "fair price."

[Oct 30, 2018] Sam Palmisano now infamous Roadmap 2015 ran the company into the ground through its maniacal focus on increasing EPS at any and all costs. Literally.

Oct 30, 2018 | features.propublica.org

GoingGone , Friday, April 13, 2018 6:06 PM

As a 25yr+ vet of IBM, I can confirm that this article is spot-on true. IBM used to be a proud and transparent company that clearly demonstrated that it valued its employees as much as it did its stock performance or dividend rate or EPS, simply because it is good for business. Those principles helped make and keep IBM atop the business world as the most trusted international brand and business icon of success for so many years. In 2000, all that changed when Sam Palmisano became the CEO. Palmisano's now infamous "Roadmap 2015" ran the company into the ground through its maniacal focus on increasing EPS at any and all costs. Literally.

Like, its employees, employee compensation, benefits, skills, and education opportunities. Like, its products, product innovation, quality, and customer service.

All of which resulted in the devastation of its technical capability and competitiveness, employee engagement, and customer loyalty. Executives seemed happy enough as their compensation grew nicely with greater financial efficiencies, and Palisano got a sweet $270M+ exit package in 2012 for a job well done.

The new CEO, Ginni Rometty has since undergone a lot of scrutiny for her lack of business results, but she was screwed from day one. Of course, that doesn't leave her off the hook for the business practices outlined in the article, but what do you expect: she was hand picked by Palmisano and approved by the same board that thought Palmisano was golden.

People (and companies) who have nothing to hide, hide nothing. People (and companies) who are proud of their actions, share it proudly. IBM believes it is being clever and outsmarting employment discrimination laws and saving the company money while retooling its workforce. That may end up being so (but probably won't), but it's irrelevant. Through its practices, IBM has lost the trust of its employees, customers, and ironically, stockholders (just ask Warren Buffett), who are the very(/only) audience IBM was trying to impress. It's just a huge shame.

HiJinks , Sunday, March 25, 2018 3:07 AM
I agree with many who state the report is well done. However, this crap started in the early 1990s. In the late 1980s, IBM offered decent packages to retirement eligible employees. For those close to retirement age, it was a great deal - 2 weeks pay for every year of service (capped at 26 years) plus being kept on to perform their old job for 6 months (while collecting retirement, until the government stepped in an put a halt to it). Nobody eligible was forced to take the package (at least not to general knowledge). The last decent package was in 1991 - similar, but not able to come back for 6 months. However, in 1991, those offered the package were basically told take it or else. Anyone with 30 years of service or 15 years and 55 was eligible and anyone within 5 years of eligibility could "bridge" the difference. They also had to sign a form stating they would not sue IBM in order to get up to a years pay - not taxable per IRS documents back then (but IBM took out the taxes anyway and the IRS refused to return - an employee group had hired lawyers to get the taxes back, a failed attempt which only enriched the lawyers). After that, things went downhill and accelerated when Gerstner took over. After 1991, there were still a some workers who could get 30 years or more, but that was more the exception. I suspect the way the company has been run the past 25 years or so has the Watsons spinning in their graves. Gone are the 3 core beliefs - "Respect for the individual", "Service to the customer" and "Excellence must be a way of life".
ArnieTracey , Saturday, March 24, 2018 7:15 PM
IBM's policy reminds me of the "If a citizen = 30 y.o., then mass execute such, else if they run then hunt and kill them one by one" social policy in the Michael York movie "Logan's Run."

From Wiki, in case you don't know: "It depicts a utopian future society on the surface, revealed as a dystopia where the population and the consumption of resources are maintained in equilibrium by killing everyone who reaches the age of 30. The story follows the actions of Logan 5, a "Sandman" who has terminated others who have attempted to escape death, and is now faced with termination himself."

Jr Jr , Saturday, March 24, 2018 4:37 PM
Corporate loyalty has been gone for 25 years. This isnt surprising. But this age discrimination is blatantly illegal.

[Oct 30, 2018] This might just be the deal that kills IBM because there's no way that they don't do a writedown of 90% of the value of this acquisition within 5 years.

Oct 30, 2018 | arstechnica.com

afidel, 2018-10-29T13:17:22-04:00

tipoo wrote:
Kilroy420 wrote:
Perhaps someone can explain this... Red Hat's revenue and assets barely total about $5B. Even factoring in market share and capitalization, how the hey did IBM come up with $34B cash being a justifiable purchase price??

Honestly, why would Red Hat have said no?

You don't trade at your earnings, you trade at your share price, which for Red Hat and many other tech companies can be quite high on Price/Earnings. They were trading at 52 P/E. Investors factor in a bunch of things involving future growth, and particularly for any companies in the cloud can quite highly overvalue things.

A 25 year old company trading at a P/E of 52 was already overpriced, buying at more than 2x that is insane. This might just be the deal that kills IBM because there's no way that they don't do a writedown of 90% of the value of this acquisition within 5 years.

[Oct 30, 2018] The insttuinaliuzed stupidity of IBM brass is connected with the desire to get bonuses

Oct 30, 2018 | arstechnica.com

3 hours ago afidel wrote: show nested quotes Kilroy420 wrote: Perhaps someone can explain this... Red Hat's revenue and assets barely total about $5B. Even factoring in market share and capitalization, how the hey did IBM come up with $34B cash being a justifiable purchase price??

Honestly, why would Red Hat have said no?

You don't trade at your earnings, you trade at your share price, which for Red Hat and many other tech companies can be quite high on Price/Earnings. They were trading at 52 P/E. Investors factor in a bunch of things involving future growth, and particularly for any companies in the cloud can quite highly overvalue things.
A 25 year old company trading at a P/E of 52 was already overpriced, buying at more than 2x that is insane. This might just be the deal that kills IBM because there's no way that they don't do a writedown of 90% of the value of this acquisition within 5 years.

OK. I did 10 years at IBM Boulder..

The problem isn't the purchase price or the probable write-down later.

The problem is going to be with the executives above it. One thing I noticed at IBM is that the executives needed to put their own stamp on operations to justify their bonuses. We were on a 2 year cycle of execs coming in and saying "Whoa.. things are too centralized, we need to decentralize", then the next exec coming in and saying "things are too decentralized, we need to centralize".

No IBM exec will get a bonus if they are over RedHat and exercise no authority over it. "We left it alone" generates nothing for the PBC. If they are in the middle of a re-org, then the specific metrics used to calculate their bonus can get waived. (Well, we took an unexpected hit this year on sales because we are re-orging to better optimize our resources). With that P/E, no IBM exec is going to get a bonus based on metrics. IBM execs do *not* care about what is good for IBM's business. They are all about gaming the bonuses. Customers aren't even on the list of things they care about.

I am reminded of a coworker who quit in frustration back in the early 2000's due to just plain bad management. At the time, IBM was working on Project Monterey. This was supposed to be a Unix system across multiple architectures. My coworker sent his resignation out to all hands basically saying "This is stupid. we should just be porting Linux". He even broke down the relative costs. Billions for Project Monterey vs thousands for a Linux port. Six months later, we get an email from on-high announcing this great new idea that upper management had come up with. It would be far cheaper to just support Linux than write a new OS.. you'd think that would be a great thing, but the reality is that all it did was create the AIX 5L family, which was AIX 5 with an additional CD called Linux ToolBox, which was loaded with a few Linux programs ported to a specific version of AIX, but never kept current. IBM can make even great decisions into bad decisions.

In May 2007, IBM announced the transition to LEAN. Sounds great, but this LEAN was not on the manufacturing side of the equation. It was in e-Business under Global Services. The new procedures were basically call center operations. Now, prior to this, IBM would have specific engineers for specific accounts. So, Major Bank would have that AIX admin, that Sun admin, that windows admin, etc. They knew who to call and those engineers would have docs and institutional knowledge of that account. During the LEAN announcement, Bob Moffat described the process. Accounts would now call an 800 number and the person calling would open a ticket. This would apply to *any* work request as all the engineers would be pooled and whoever had time would get the ticket. So, reset a password - ticket. So, load a tape - ticket. Install 20 servers - ticket.

Now, the kicker to this was that the change was announced at 8AM and went live at noon. IBM gave their customers who represented over $12 Billion in contracts 4 *hours* notice that they were going to strip their support teams and treat them like a call center. (I will leave it as an exercise to the reader to determine if they would accept that kind of support after spending hundreds of millions on a support contract).

(The pilot program for the LEAN process had its call center outsourced overseas, if that helps you try to figure out why IBM wanted to get rid of dedicated engineers and move to a call-center operation).

[Oct 30, 2018] Presumably the acquisition will have to jump various regulatory hurdles before it is set in stone. If it is successful, Red Hat will be absorbed into IBM's Hybrid Cloud unit

IBM will have to work hard to overcome RH customers' natural (and IMHO largely justified) suspicion of Big Blue.
Notable quotes:
"... focused its workforce on large "hub" cities where graduate engineers prefer to live – New York City, San Francisco, and Austin, in the US for instance – which allowed it to drive out older, settled staff who refused to move closer to the office. ..."
"... The acquisition of Sun Microsystems by Oracle comes to mind there. ..."
"... When Microsoft bought out GitHub, they made a promise to let it run independently and now IBM's given a similar pledge in respect of RedHat. They ought to abide by that promise because the alternatives are already out there in the form of Ubuntu and SUSE Linux Enterprise Server. ..."
Oct 30, 2018 | theregister.co.uk

...That transformation has led to accusations of Big Blue ditching its older staff for newer workers to somehow spark some new energy within it. It also cracked down on remote employees , and focused its workforce on large "hub" cities where graduate engineers prefer to live – New York City, San Francisco, and Austin, in the US for instance – which allowed it to drive out older, settled staff who refused to move closer to the office.

Ledswinger, 1 day

Easy, the same way they deal with their existing employees. It'll be the IBM way or the highway. We'll see the usual repetitive and doomed IBM strategy of brutal downsizings accompanied by the earnest IBM belief that a few offshore wage slaves can do as good a job as anybody else.

The product and service will deteriorate, pricing will have to go up significantly to recover the tens of billions of dollars of "goodwill" that IBM have just splurged, and in five years time we'll all be saying "remember how the clueless twats at IBM bought Red Hat and screwed it up?"

One of IBM's main problems is lack of revenue, and yet Red Hat only adds about $3bn to their revenue. As with most M&A the motivators here are a surplus of cash and hopeless optimism, accompanied by the suppression of all common sense.

Well done Gini, it's another winner.

TVU, 12 hrs
Re: "they will buy a lot of talent."

"What happens over the next 12 -- 24 months will be ... interesting. Usually the acquisition of a relatively young, limber outfit with modern product and service by one of the slow-witted traditional brontosaurs does not end well"

The acquisition of Sun Microsystems by Oracle comes to mind there.

When Microsoft bought out GitHub, they made a promise to let it run independently and now IBM's given a similar pledge in respect of RedHat. They ought to abide by that promise because the alternatives are already out there in the form of Ubuntu and SUSE Linux Enterprise Server.

[Oct 30, 2018] About time: Red Hat support was bad alread

Oct 30, 2018 | theregister.co.uk
Anonymous Coward, 15 hrs

and the support goes bad already...

Someone (RH staff, or one of the Gods) is unhappy with the deal: Red Hat's support site has been down all day.

https://status.redhat.com

[Oct 30, 2018] Purple rain will fall from that blue cloud

Notable quotes:
"... IBM was already "working on Linux." For decades. With multiple hundreds of full-time Linux developers--more than any other corporate contributor--around the world. And not just on including IBM-centric function into Linux, but on mainstream community projects. There have been lots of Linux people in IBM since the early-90's. ..."
"... From a customer standpoint the main thing RedHat adds is formal support. There are still a lot of companies who are uncomfortable deploying an OS that has product support only from StackExchange and web forums ..."
"... You would do better to look at the execution on the numbers - RedHat is not hitting it's targets and there are signs of trouble. These two businesses were both looking for a prop and the RedHat shareholders are getting out while the business is near it's peak. ..."
Oct 30, 2018 | theregister.co.uk

Anonymous Coward , 6 hrs

Re: So exactly how is IBM going to tame employees that are used to going to work in shorts...

Purple rain will fall from that cloud...

asdf , 5 hrs
Wow bravo brutal and accurate. +1
I can't believe its not butter , 1 day
Redhat employees - get out now

You're utterly fucked. Run away now as you have zero future in IBM.

Anonymous Coward , 1 day
Re: Redhat employees - get out now

You're utterly fucked. Run away now as you have zero future in IBM.

That was my immediate thought upon hearing this. I've already worked for IBM, swore never to do it again. Time to dust off & update the resume.

Jove , 19 hrs
Re: Redhat employees - get out now

Another major corporate splashes out a fortune on a star business only to find the clash of cultures destroys substantial value.

W@ldo , 1 day
Re: At least is isnt oracle or M$

Sort of the lesser of evils---do you want to be shot or hung by the neck? No good choice for this acquisition.

Anonymous Coward , 1 day
Re: At least is isnt oracle or M$

honestly, MS would be fine. They're big into Linux and open source and still a heavily pro-engineering company.

Companies like Oracle and IBM are about nothing but making money. Which is why they're both going down the tubes. No-one who doesn't already have them goes near them.

Uncle Ron , 14 hrs
Re: At least is isnt oracle or M$

IBM was already "working on Linux." For decades. With multiple hundreds of full-time Linux developers--more than any other corporate contributor--around the world. And not just on including IBM-centric function into Linux, but on mainstream community projects. There have been lots of Linux people in IBM since the early-90's.

Orv , 9 hrs
Re: At least it is not Oracle or M$

The OS is from Linus and chums, Redhat adds a few storage bits and some Redhat logos and erm.......

From a customer standpoint the main thing RedHat adds is formal support. There are still a lot of companies who are uncomfortable deploying an OS that has product support only from StackExchange and web forums. This market is fairly insensitive to price, which is good for a company like RedHat. (Although there has been an exodus of higher education customers as the price has gone up; like Sun did back in the day, they've been squeezing out that market. Two campuses I've worked for have switched wholesale to CentOS.)

Jove , 11 hrs
RedHat take-over IBM - @HmmmYes

"Just compare the share price of RH v IBM"

You would do better to look at the execution on the numbers - RedHat is not hitting it's targets and there are signs of trouble. These two businesses were both looking for a prop and the RedHat shareholders are getting out while the business is near it's peak.

[Oct 30, 2018] Pay your licensing fee

Oct 30, 2018 | linux.slashdot.org

red crab ( 1044734 ) , Monday October 29, 2018 @12:10AM ( #57552797 )

Re:Pay your licensing fee ( Score: 4 , Interesting)

Footnote: $699 License Fee applies to your systemP server running RHEL 7 with 4 cores activated for one year.

To activate additional processor cores on the systemP server, a fee of $199 per core applies. systemP offers a new Semi-Activation Mode now. In systemP Semi-Activation Mode, you will be only charged for all processor calls exceeding 258 MIPS, which will be processed by additional semi-activated cores on a pro-rata basis.

RHEL on systemP servers also offers a Partial Activation Mode, where additional cores can be activated in Inhibited Efficiency Mode.

To know more about Semi-Activation Mode, Partial Activation Mode and Inhibited Efficiency Mode, visit http://www.ibm.com/systemp [ibm.com] or contact your IBM systemP Sales Engineer.

[Oct 30, 2018] $34B? I was going to say this is the biggest tech acquisition ever, but it's second after Dell buying EMC

Notable quotes:
"... I'm not too sure what IBM is going to do with that, but congrats to whoever is getting the money... ..."
Oct 30, 2018 | theregister.co.uk

ratfox , 1 day

$34B? I was going to say this is the biggest tech acquisition ever, but it's second after Dell buying EMC. I'm not too sure what IBM is going to do with that, but congrats to whoever is getting the money...

[Oct 30, 2018] "OMG" comments rests on three assumptions: Red Hat is 100% brilliant and speckless, IBM is beyond hope and unchangeable, this is a hostile takeover

Notable quotes:
"... But I do beg to differ about the optimism, because, as my boss likes to quote, "culture eats strategy for breakfast". ..."
"... So the problem is that IBM have bought a business whose competencies and success factors differ from the IBM core. Its culture is radically different, and incompatible with IBM. ..."
"... Many of its best employees will be hostile to IBM ..."
"... And just like a gas giant, IBM can be considered a failed star ..."
Oct 30, 2018 | theregister.co.uk

LeoP , 1 day

Less pessimistic here

Quite a lot of the "OMG" moments rests on three assumptions:

I beg to differ on all counts. Call me beyond hope myself because of my optimism, but I do think what IBM bought most is a way to run a business. RH is just too big to be borged into a failing giant without leaving quite a substantial mark.

Ledswinger , 1 day
Re: Less pessimistic here

I beg to differ on all counts.

Note: I didn't downvote you, its a valid argument. I can understand why you think that, because I'm in the minority that think the IBM "bear" case is overdone. They've been cleaning their stables for some years now, and that means dropping quite a lot of low margin business, and seeing the topline shrink. That attracts a lot of criticism, although it is good business sense.

But I do beg to differ about the optimism, because, as my boss likes to quote, "culture eats strategy for breakfast".

And (speaking as a strategist) that's 100% true, and 150% true when doing M&A.

So the problem is that IBM have bought a business whose competencies and success factors differ from the IBM core. Its culture is radically different, and incompatible with IBM.

Many of its best employees will be hostile to IBM . RedHat will be borged, and it will leave quite a mark. A bit like Shoemaker-Levi did on Jupiter. Likewise there will be lots of turbulence, but it won't endure, and at the end of it all the gas giant will be unchanged (just a bit poorer). And just like a gas giant, IBM can be considered a failed star .

[Oct 30, 2018] If IBM buys Redhat then what will happen to CentOS?

Notable quotes:
"... As long as IBM doesn't close-source RH stuff -- most of which they couldn't if they wanted to -- CentOS will still be able to do builds of it. The only thing RH can really enforce control over is the branding and documentation. ..."
"... Might be a REALLY good time to fork CentOS before IBM pulls an OpenSolaris on it. Same thing with Fedora. ..."
"... I used to be a Solaris admin. Now I am a linux admin in a red hat shop. Sun was bought by Oracle and more or less died a death. Will the same happen now? I know that Sun and RH are _very_ differeht beasts but I am thinking that now is the time to stop playing on the merry go round called systems administration. ..."
Oct 30, 2018 | theregister.co.uk

Orv , 9 hrs

Re: If IBM buys Redhat then what will happen to CentOS?

If IBM buys Redhat then what will happen to CentOS?

As long as IBM doesn't close-source RH stuff -- most of which they couldn't if they wanted to -- CentOS will still be able to do builds of it. The only thing RH can really enforce control over is the branding and documentation.

Anonymous Coward , 9 hrs
Re: "they will buy a lot of talent."

"I found that those with real talent that matched IBM needs are well looked after."

The problem is that when you have served your need you tend to get downsized as the expectation is that the cheaper offshore bodies can simply take over support etc after picking up the skills they need over a few months !!!

This 'Blue meets Red and assimilates' will be very interesting to watch and will need lots and lots of popcorn on hand !!!

:)

FrankAlphaXII , 1 day
Might be a REALLY good time to fork CentOS before IBM pulls an OpenSolaris on it. Same thing with Fedora.

Kind of sad really, when I used Linux CentOS and Fedora were my go-to distros.

Missing Semicolon , 1 day
Goodbye Centos

Centos is owned by RedHat now. So why on earth would IBM bother keeping it?

Plus, of course, all the support and coding will be done in India now.....

Doctor Syntax , 17 hrs
Re: Goodbye Centos

"Centos is owned by RedHat now."

RedHat is The Upstream Vendor of Scientific Linux. What happens to them if IBM turn nasty?

Anonymous Coward , 16 hrs
Re: Scientific Linux

Scientific Linux is a good idea in theory, dreadful in practice.

The idea of a research/academic-software-focused distro is a good one: unfortunately (I say unfortunately, but it's certainly what I would do myself), increasing numbers of researchers are now developing their pet projects on Debian or Ubuntu, and so therefore often only make .deb packages available.

Anyone who has had any involvement in research software knows that if you find yourself in the position of needing to compile someone else's pet project from source, you are often in for an even more bumpy ride than usual.

And the lack of compatible RPM packages just encourages more and more researchers to go where the packages (and the free-ness) are, namely Debian and friends, which continue to gather momentum, while Red Hat continues to stagnate.

Red Hat may be very stable for running servers (as long as you don't need anything reasonably new (not bleeding edge, but at least newer than three years old)), but I have never really seen the attraction in it myself (especially as there isn't much of a "community" feeling around it, as its commercial focus gets in the way).

Roger Kynaston , 1 day
another buy out of my bread and butter

I used to be a Solaris admin. Now I am a linux admin in a red hat shop. Sun was bought by Oracle and more or less died a death. Will the same happen now? I know that Sun and RH are _very_ differeht beasts but I am thinking that now is the time to stop playing on the merry go round called systems administration.

Anonymous Coward , 1 day
Don't become a Mad Hatter

Dear Red Hatters, welcome to the world of battery hens, all clucking and clicking away to produce the elusive golden egg while the axe looms over their heads. Even if your mental health survives, you will become chicken feed by the time you are middle aged. IBM doesn't have a heart; get out while you can. Follow the example of IBMers in Australia who are jumping ship in their droves, leaving behind a crippled, demoralised workforce. Don't become a Mad Hatter.

[Oct 30, 2018] Kind of surprising since IBM was aligned with SUSE for so many years.

Oct 30, 2018 | arstechnica.com

tgx Ars Centurion reply 5 hours ago

Kind of surprising since IBM was aligned with SUSE for so many years.

IBM is an awful software company. OS/2, Lotus Notes, AIX all withered on the vine.

Doesn't bode well for RedHat.

[Oct 30, 2018] Hello, we are mandating some new policies, git can no longer be used, we must use IBM synergy software with rational rose

Oct 30, 2018 | arstechnica.com

lordofshadows , Smack-Fu Master, in training 6 hours ago

Hello, we are mandating some new policies, git can no longer be used, we must use IBM synergy software with rational rose.

All open source software is subject to corporate approval first, we know we know, to help streamline this process we have approved GNU CC and are looking into this Mak file program.

We are very pleased with systemd, we wish to further expand it's dhcp capabilities and also integrate IBM analytics -- We are also going through a rebranding operation as we feel the color red is too jarring for our customers, we will now be known as IBM Rational Hat and will only distribute through our retail channels to boost sales -- Look for us at walmart, circuit city, and staples

[Oct 30, 2018] And RH customers will want to check their contracts...

Oct 30, 2018 | arstechnica.com

CousinSven , Smack-Fu Master, in training et Subscriptor 4 hours ago New Poster

IBM are paying around 12x annual revenue for Red Hat which is a significant multiple so they will have to squeeze more money out of the business somehow. Either they grow customers or they increase margins or both.

IBM had little choice but to do something like this. They are in a terminal spiral thanks to years of bad leadership. The confused billing of the purchase smacks of rush, so far I have seen Red Hat described as a cloud company, an info sec company, an open source company...

So IBM are buying Red Hat as a last chance bid to avoid being put through the PE threshing machine. Red Hat get a ludicrous premium so will take the money.

And RH customers will want to check their contracts...

[Oct 30, 2018] IBM To Buy Red Hat, the Top Linux Distributor, For $34 Billion

Notable quotes:
"... IBM license fees are predatory. Plus they require you to install agents on your servers for the sole purpose of calculating use and licenses. ..."
"... IBM exploits workers by offshoring and are slow to fix bugs and critical CVEs ..."
Oct 30, 2018 | linux.slashdot.org

Anonymous Coward , Sunday October 28, 2018 @03:34PM ( #57550555 )

Re: Damn. ( Score: 5 , Insightful)

IBM license fees are predatory. Plus they require you to install agents on your servers for the sole purpose of calculating use and licenses.

IBM exploits workers by offshoring and are slow to fix bugs and critical CVEs (WAS and DB2 especially)

The Evil Atheist ( 2484676 ) , Sunday October 28, 2018 @04:13PM ( #57550755 ) Homepage
Re:Damn. ( Score: 4 , Insightful)

IBM buys a company, fires all the transferred employees and hopes they can keep selling their acquired software without further development. If they were serious, they'd have improved their own Linux contribution efforts.

But they literally think they can somehow keep selling software without anyone with knowledge of the software, or for transferring skills to their own employees.

They literally have no interest in actual software development. It's all about sales targets.

Anonymous Coward , Monday October 29, 2018 @01:00AM ( #57552963 )
Re:Damn. ( Score: 3 , Informative)

My advice to Red Hat engineers is to get out now. I was an engineer at a company that was acquired by IBM. I was fairly senior so I stayed on and ended up retiring from the IBM, even though I hated my last few years working there. I worked for several companies during my career, from startups to fortune 100 companies. IBM was the worst place I worked by far. Consider every bad thing you've ever heard about IBM. I've heard those things too, and the reality was much worse.

IBM hasn't improved their Linux contribution efforts because it wouldn't know how. It's not for lack of talented engineers. The management culture is simply pathological. No dissent is allowed. Everyone lives in fear of a low stack ranking and getting laid off. In the end it doesn't matter anyway. Eventually the product you work on that they originally purchased becomes unprofitable and they lay you off anyway. They've long forgotten how to develop software on their own. Don't believe me? Try to think of an IBM branded software product that they built from the ground up in the last 25 years that has significant market share. Development managers chase one development fad after another hoping to find the silver bullet that will allow them to continue the relentless cost cutting regime made necessary in order to make up revenue that has been falling consistently for over a decade now.

As far as I could tell, IBM is good at two things:

  1. Financial manipulation to disguise there shrinking revenue
  2. Buying software companies and mining them for value

Yes, there are still some brilliant people that work there. But IBM is just not good at turning ideas into revenue producing products. They are nearly always unsuccessful when they try and then the go out and buy a company the succeeded in bring to market the kind of product that they tried and failed to build themselves.

They used to be good at customer support, but that is mainly lip service now. Just before I left the company I was tapped to deliver a presentation at a customer seminar. The audience did not care much about my presentation. The only thing they wanted to talk about was how much they had invested millions in re-engineering their business to use our software and now IBM appeared to be wavering in their long term commitment to supporting the product. It was all very embarrassing because I knew what they didn't, that the amount of development and support resources currently allocated to the product line were a small fraction of what they once were. After having worked there I don't know why anyone would ever want to buy a license for any of their products.

gtall ( 79522 ) , Sunday October 28, 2018 @03:59PM ( #57550691 )
Re:A Cloudy argument. ( Score: 5 , Insightful)

So you are saying that IBM has been asleep at the wheel for the last 8 years. Buying Red Hat won't save them, IBM is IBM's enemy.

Aighearach ( 97333 ) writes:
Re: ( Score: 3 )

They're already one of the large cloud providers, but you don't know that because they only focus on big customers.

The Evil Atheist ( 2484676 ) , Sunday October 28, 2018 @04:29PM ( #57550829 ) Homepage
Re:A Cloudy argument. ( Score: 5 , Insightful)

IBM engineers aren't actually crappy. It's the fucking MBAs in management who have no clue about how to run a software development company. Their engineers will want to do good work, but management will worry more about headcount and sales.

The Evil Atheist ( 2484676 ) , Sunday October 28, 2018 @03:57PM ( #57550679 ) Homepage
Goodbye Redhat. ( Score: 5 , Insightful)

IBM acquisitions never go well. All companies acquired by IBM go through a process of "Blue washing", in which the heart and soul of the acquired company is ripped out, the body burnt, and the remaining ashes to be devoured and defecated by its army of clueless salesmen and consultants. It's a sad, and infuriating, repeated pattern. They no longer develop internal talent. They drive away the remaining people left over from the time when they still did develop things. They think they can just buy their way into a market or technology, somehow completely oblivious to the fact that their strategy of firing all their acquired employees/knowledge and hoping to sell software they have no interest in developing would somehow still retain customers. They literally could have just reshuffled and/or hired more developers to work on the kernel, but the fact they didn't shows they have no intention of actually contributing.

Nkwe ( 604125 ) , Sunday October 28, 2018 @04:26PM ( #57550819 )
Cha-Ching ( Score: 3 )

Red Hat closed Friday at $116.68 per share, looks like the buy out is for $190. Not everyone will be unhappy with this. I hope the Red Hat employees that won't like the upcoming cultural changes have stock and options, it may soften the blow a bit.

DougDot ( 966387 ) writes: < dougr@parrot-farm.net > on Sunday October 28, 2018 @05:43PM ( #57551189 ) Homepage
AIX Redux ( Score: 5 , Interesting)

Oh, good. Now IBM can turn RH into AIX while simultaneously suffocating whatever will be left of Redhat's staff with IBM's crushing, indifferent, incompetent bureaucracy.

This is what we call a lose - lose situation. Well, except for the president of Redhat, of course. Jim Whitehurst just got rich.

Tough Love ( 215404 ) writes: on Sunday October 28, 2018 @10:55PM ( #57552583 )
Re:AIX Redux ( Score: 2 )

Worse than Redhat's crushing, indifferent, incompetent bureaucracy? Maybe, but it's close.

ArchieBunker ( 132337 ) , Sunday October 28, 2018 @06:01PM ( #57551279 ) Homepage
Re:AIX Redux ( Score: 5 , Insightful)

Redhat is damn near AIX already. AIX had binary log files long before systemd.

Antique Geekmeister ( 740220 ) , Monday October 29, 2018 @05:51AM ( #57553587 )
Re:Please God No ( Score: 4 , Informative)

The core CentOS leadership are now Red Hat employees. They're not clear of nor uninvolved in this purchase.

alvinrod ( 889928 ) , Sunday October 28, 2018 @03:46PM ( #57550623 )
Re:Please God No ( Score: 5 , Insightful)

Depends on the state. Non-compete clauses are unenforceable in some jurisdictions. IBM would want some of the people to stick around. You can't just take over a complex system from someone else and expect everything to run smoothly or know how to fix or extend it. Also, not everyone who works at Red Hat gets anything from the buyout unless they were regularly giving employees stock. A lot of people are going to want the stable paycheck of working for IBM instead of trying to start a new company.

However, some will inevitably get sick of working at IBM or end up being laid off at some point. If these people want to keep doing what they're doing, they can start a new company. If they're good at what they do, they probably won't have much trouble attracting some venture capital either.

wyattstorch516 ( 2624273 ) , Sunday October 28, 2018 @04:41PM ( #57550887 )
Re:Please God No ( Score: 2 )

Red Hat went public in 1999, they are far from being a start-up. They have acquired several companies themselves so they are just as corporate as IBM although significantly smaller.

Anonymous Coward , Sunday October 28, 2018 @05:10PM ( #57551035 )
Re:Please God No ( Score: 5 , Funny)

Look on the bright side: Poettering works for Red Hat. (Reposting because apparently Poettering has mod points.)

Anonymous Coward , Monday October 29, 2018 @09:24AM ( #57554491 )
Re: It all ( Score: 5 , Interesting)

My feelings exactly. As a former employee for both places, I see this as the death knell for Red Hat. Not immediately, not quickly, but eventually Red Hat's going to go the same way as every other company IBM has acquired.

Red Hat's doom (again, all IMO) started about 10 years ago or so when Matt Szulik left and Jim Whitehurst came on board. Nothing against Jim, but he NEVER seemed to grasp what F/OSS was about. Hell, when he came onboard he wouldn't (and never did) use Linux at all: instead he used a Mac, and so did the rest of the EMT (executive management team) over time. What company is run by people who refuse to use its own product except for one that doesn't have faith. The person on top of the BRAND AND PEOPLE team "needed" an iPad, she said, to do her work (quoting a friend in the IT dept who was asked to get it and set it up for her).

Then when they (the EMTs) wanted to move away from using F/OSS internally to outsourcing huge aspects of our infrastructure (like no longer using F/OSS for email and instead contracting with GOOGLE to do our email, calendaring and document sharing) is when, again for me, the plane started to spiral. How can we sell to OUR CUSTOMERS the idea that "Red Hat and F/OSS will suit all of your corporate needs" when, again, the people running the ship didn't think it would work for OURS? We had no special email or calendar needs, and if we did WE WERE THE LEADERS OF OPEN SOURCE, couldn't we make it do what we want? Hell, I was on an internal (but on our own time) team whose goal was to take needs like this and incubate them with an open source solution to meet that need.

But the EMTs just didn't want to do that. They were too interested in what was "the big thing" (at the time Open Shift was where all of our hiring and resources were being poured) to pay attention to the foundations that were crumbling.

And now, here we are. Red Hat is being subsumed by the largest closed-source company on the planet, one who does their job sub-optimally (to be nice). This is the end of Red Hat as we know it. Without 5-7 years Red Hat will go the way of Tivoli and Lotus: it will be a brand name that lacks any of what made the original company what it was when it was acquired.

[Oct 30, 2018] Why IBM did do the same as Oracle?

Notable quotes:
"... Just fork it, call it Blue Hat Linux, and rake in those sweet support dollars. ..."
Oct 30, 2018 | arstechnica.com

Twilight Sparkle , Ars Scholae Palatinae 4 hours ago

Why would you pay even 34 dollars for software worth $0?

Just fork it, call it Blue Hat Linux, and rake in those sweet support dollars.

[Oct 30, 2018] Arbitrators overwhelmingly favor employers

Oct 30, 2018 | features.propublica.org

When it comes to employment claims, studies have found that arbitrators overwhelmingly favor employers. Research by Cornell University law and labor relations specialist Alexander Colvin found that workers win only 19 percent of the time when their cases are arbitrated. By contrast, they win 36 percent of the time when they go to federal court, and 57 percent in state courts. Average payouts when an employee wins follow a similar pattern.

Given those odds, and having signed away their rights to go to court, some laid-off IBM workers have chosen the one independent forum companies can't deny them: the U.S. Equal Employment Opportunity Commission. That's where Moos, the Long Beach systems security specialist, and several of her colleagues, turned for help when they were laid off. In their complaints to the agency, they said they'd suffered age discrimination because of the company's effort to "drastically change the IBM employee age mix to be seen as a startup."

In its formal reply to the EEOC, IBM said that age couldn't have been a factor in their dismissals. Among the reasons it cited: The managers who decided on the layoffs were in their 40s and therefore older too.

[Oct 30, 2018] I see the Performance Improvement Plan (PIP) problem as its nearly impossible to take the fact that we know PIP is a scam to court. IBM will say its an issue with you, your performance nose dived and your manager tried to fix that. You have to not only fight those simple statements, but prove that PIP is actually systematic worker abuse.

Notable quotes:
"... It is in fact a modern corporate horror story; it's also life at a modern corporation, period. ..."
Oct 30, 2018 | features.propublica.org

Cindy Gallop , Thursday, March 22, 2018 10:24 AM

This makes for absolutely horrifying, chills-down-your-spine reading. A modern corporate horror story - worthy of a 'Black Mirror' episode. Phenomenal reporting by Ariana Tobin and Peter Gosselin. Thank you for exposing this. I hope this puts an end to this at IBM and makes every other company and industry doing this in covert and illegal ways think twice about continuing.
Daisy S Cindy Gallop , in reply to" aria-label="in reply to"> •
Agree..a well written expose'. I've been a victim of IBM's "PIP" (Performance Improvement Plan) strategy, not because of my real performance mind you, but rather, I wasn't billing hours between projects and it was hurting my unit's bottom line. The way IBM instructs management to structure the PIP, it's almost impossible to dig your way out, and it's intentional. If you have a PIP on your record, nobody in IBM wants to touch you, so in effect you're already gone.
Paul Brinker Daisy S , in reply to" aria-label="in reply to"> •
I see the PIP problem as its nearly impossible to take the fact that we know PIP is a scam to court. IBM will say its an issue with you, your performance nose dived and your manager tried to fix that. You have to not only fight those simple statements, but prove that PIP is actually systematic worker abuse.
dragonflap Cindy Gallop , in reply to" aria-label="in reply to"> •
Cindy, they've been doing this for at least 15-20 years, or even longer according to some of the previous comments. It is in fact a modern corporate horror story; it's also life at a modern corporation, period.
Maria Stone dragonflap , in reply to" aria-label="in reply to"> •
This started happening in the 1990's when they added 5 years to your age and years of service and ASKED you to retire.
Matt_Z , Thursday, March 22, 2018 6:01 PM
After over 35 years working there, 19 of them as a manager sending out more of those notification letters than I care to remember, I can vouch for the accuracy of this investigative work. It's an incredibly toxic and hostile environment and has been for the last 5 or so years. One of the items I was appraised on annually was how many US jobs I moved offshore. It was a relief when I received my notification letter after a two minute phone call telling me it was on the way. Sleeping at night and looking myself in the mirror aren't as hard as they were when I worked there.
IBM will never regain any semblance of their former glory (or profit) until they begin to treat employees well again.
With all the offshoring and resource actions with no backfill over the last 10 years, so much is broken. Customers suffer almost as much as the employees.
I don't know how in the world they ended up on that LinkedIn list. Based on my fairly recent experience there are a half dozen happy employees in the US, and most of them are C level.
Jennifer , Thursday, March 22, 2018 9:42 AM
Well done. It squares well with my 18 years at IBM, watching resource action after resource action and hearing what my (unusually honest) manager told me. Things got progressively worse from 2012 onward. I never realized how stressful it was to live under the shadow of impending layoffs until I finally found the courage to leave in 2015. Best decision I've made.

IBM answers to its shareholders, period. Employees are an afterthought - simply a means to an end. It's shameful. (That's not to say that individual people managers feel that way. I'm speaking about IBM executives.)

Herb Jennifer , in reply to" aria-label="in reply to"> •
Well, they almost answer to their shareholders, but that's after the IBM executives take their share. Ginni's compensation is tied to stock price (apparently not earnings) and buy backs maintain the stock price.
Ribit , Thursday, March 22, 2018 8:17 AM
If the criteria for layoff is being allegedly overpaid and allegedly a poor performer, then it follows that Grinnin' Jenny should have been let go long ago.
Mr. Hand Ribit , in reply to" aria-label="in reply to"> •
Yes! After the 4th of those 22 consecutive quarters of declining revenues. And she's no spring chicken either. ;-)
DDRLSGC Ribit ,
Especially these CEOs who have ran their companies into the ground for the last 38 years.
owswitch , Thursday, March 22, 2018 8:58 AM
Just another fine example of how people become disposable.
And, when it comes to cost containment and profit maximization, there is no place for ethics in American business.
Businesses can lie just as well as politicians.

Millennials are smart to avoid this kind of problem by remaining loyal only to themselves. Companies certainly define anyone as replaceable - even their over-paid CEO's.

DDRLSGC owswitch

The millennials saw what happen to their parents and grandparents getting screwed over after a life time of work and loyalty. You can't blame them for not caring about so called traditional American work ethics and then they are attacked for not having them when the business leaders threw away all those value decades ago.

Some of these IBM people have themselves to blame for cutting their own economic throats for fighting against unions, putting in politicians who are pro-business and thinking that their education and high paying white collar STEM jobs will give them economic immunity.

If America was more of a free market and free enterprise instead of being more of a close market of oligarchies and monopolies, and strong government regulations, companies would think twice about treating their workforce badly because they know their workforce would leave for other companies or start up their own companies without too much of a hassle.

HiJinks DDRLSGC

Under the old IBM you could not get a union as workers were treated with dignity and respect - see the 3 core beliefs. Back then a union would not have accomplished anything.

DDRLSGC HiJinks
Doesn't matter if it was the old IBM or new IBM, you wonder how many still actually voted against their economic interests in the political elections that in the long run undermine labor rights in this country.
HiJinks DDRLSGC
So one shouldn't vote? Neither party cares about the average voter except at election time. Both sell out to Big Business - after all, that's where the big campaign donations come from. If you believe only one party favors Big Business, then you have been watching to much "fake news". Even the unions know they have been sold out by both and are wising up. How many of those jobs were shipped overseas the past 25 years.
DDRLSGC HiJinks ,
No, they should have been more active in voting for politicians who would look after the workers' rights in this country for the last 38 years plus ensuring that Congressional people and the president would not be packing the court system with pro-business judges. Sorry, but it is the Big Business that have been favoring the Republican Party for a long, long time and the jobs have been shipped out for the last 38 years.

[Oct 30, 2018] The women who run large US companies are as shallow and ruthless as the sociopathic men.

Oct 30, 2018 | features.propublica.org

Bob Gort , Saturday, March 31, 2018 9:49 PM

Age discrimination has been standard operating procedure in IT for at least 30 years. And there are no significant consequences, if any consequences at all, for doing it in a blatant fashion. The companies just need to make sure the quota of H1B visas is increased when they are doing this on an IBM scale!
900DeadWomen Bob Gort , in reply to" aria-label="in reply to">
Age discrimination and a myriad other forms of discrimination have been standard operating procedure in the US. Period. Full stop. No need to equivocate.
Anon , Friday, March 30, 2018 12:49 PM
Wait for a few years and we can see the same happening to "millenials".

And the women who run these companies are as shallow and ruthless as the sociopathic men.

[Oct 30, 2018] If I (hypothetically) worked for a company acquired by Big Blue

Notable quotes:
"... If I (hypothetically) worked for a company acquired by Big Blue, I would offer the following: Say hello to good salaries. Say goodbye to perks, bonuses, and your company culture...oh, and you can't work from home anymore. ...but this is all hypothetical. ..."
Oct 30, 2018 | arstechnica.com

MagicDot / Ars Praetorian reply 6 hours ago

If I (hypothetically) worked for a company acquired by Big Blue, I would offer the following: Say hello to good salaries. Say goodbye to perks, bonuses, and your company culture...oh, and you can't work from home anymore. ...but this is all hypothetical.

Belisarius , Ars Tribunus Angusticlavius et Subscriptor 5 hours ago

sviola wrote:
show nested quotes
I can see what each company will get out of the deal and how they might potentially benefit. However, Red Hat's culture is integral to their success. Both company CEOs were asked today at an all-hands meeting about how they intend to keep the promise of remaining distinct and maintaining the RH culture without IBM suffocating it. Nothing is supposed to change (for now), but IBM has a track record of driving successful companies and open dynamic cultures into the ground. Many, many eyes will be watching this.

Hopefully IBM current top Brass will be smart and give some autonomy to Red Hat and leave it to its own management style. Of course, that will only happen if they deliver IBM goals (and that will probably mean high double digit y2y growth) on regular basis...

One thing is sure, they'll probably kill any overlapsing product in the medium term (who will survive between JBOSS and Websphere is an open bet).

(On a dream side note, maybe, just maybe they'll move some of their software development to Red Hat)

Good luck. Every CEO thinks they're the latest incarnation of Adam Smith, and they're all dying to be seen as "doing something." Doing nothing, while sometimes a really smart thing and oftentimes the right thing to do, isn't looked upon favorably these days in American business. IBM will definitely do something with Red Hat; it's just a matter of what.

[Oct 30, 2018] No good will come from this. IBM's corporate environment and financial near-sightedness will kill Red Hat. Time to start looking for a new standard bearer in Linux for business

Oct 30, 2018 | arstechnica.com

Dilbert Ars Legatus Legionis reply 6 hours ago Popular motytrah wrote: bolomkxxviii wrote: No good will come from this. IBM's corporate environment and financial near-sightedness will kill Red Hat. Time to start looking for a new standard bearer in Linux for business.

I agree. Redhat has dev offices all over. A lot of them in higher cost areas of the US and Europe. There's no way IBM doesn't consolidate and offshore a bunch of that work.
This. To a bean counter a developer in a RH office in North America or Europe who's been coding for RH for 10 years is valued same as a developer in Calcutta who just graduated from college. For various definitions of word 'graduated'.

[Oct 30, 2018] Soon after I started, the company fired hundreds of 50-something employees and put we "kids" in their jobs. Seeing that employee loyalty was a one way street at that place, I left after a couple of years. Best career move I ever made.

Oct 30, 2018 | features.propublica.org

Al Romig , Wednesday, April 18, 2018 5:20 AM

As a new engineering graduate, I joined a similar-sized multinational US-based company in the early '70s. Their recruiting pitch was, "Come to work here, kid. Do your job, keep your nose clean, and you will enjoy great, secure work until you retire on easy street".

Soon after I started, the company fired hundreds of 50-something employees and put we "kids" in their jobs. Seeing that employee loyalty was a one way street at that place, I left after a couple of years. Best career move I ever made.

GoingGone , Friday, April 13, 2018 6:06 PM
As a 25yr+ vet of IBM, I can confirm that this article is spot-on true. IBM used to be a proud and transparent company that clearly demonstrated that it valued its employees as much as it did its stock performance or dividend rate or EPS, simply because it is good for business. Those principles helped make and keep IBM atop the business world as the most trusted international brand and business icon of success for so many years. In 2000, all that changed when Sam Palmisano became the CEO. Palmisano's now infamous "Roadmap 2015" ran the company into the ground through its maniacal focus on increasing EPS at any and all costs. Literally. Like, its employees, employee compensation, benefits, skills, and education opportunities. Like, its products, product innovation, quality, and customer service. All of which resulted in the devastation of its technical capability and competitiveness, employee engagement, and customer loyalty. Executives seemed happy enough as their compensation grew nicely with greater financial efficiencies, and Palisano got a sweet $270M+ exit package in 2012 for a job well done. The new CEO, Ginni Rometty has since undergone a lot of scrutiny for her lack of business results, but she was screwed from day one. Of course, that doesn't leave her off the hook for th