Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

IFS variable and Field splitting in the Korn shell and Bash

News

See also

Best Shell Books

Recommended Links Papers, ebooks tutorials

Pipes

Reference
Pipes in Loops Process Substitution in Shell Tee Sysadmin Horror Stories Tips Humor Etc

System variable IFS specifies internal field separators (normally space, tab, and new line) used to separate command words that result from command or parameter substitution and for separating words that are read using built-in command read. The first character of the IFS parameter is used to separate arguments for the $* substitution.

After performing command substitution, the Korn shell scans the results of substitutions for those field separator characters found in the IFS (Internal Field Separator) variable. Where such characters are found, the shell splits the substitutions into distinct arguments.

The shell retains explicit null arguments ("" or '') and removes implicit null arguments (those resulting from parameters that have no values).
IFS=','
print $*

Changing IFS in a script is fairly risky, but it's probably OK as long as nothing is called from the the script that  depends on the value of IFS. If this script were called arglist, then the command arglist bob dave ed would produce the output bob,dave,ed.


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jun 25, 2007] Useful Shell Scripting Variables - Part III - IFS (Internal Field Separator)

October 13, 2003

... The shell uses the value stored in IFS, which is the space, tab, and newline characters by default, to delimit words for the read and set commands, when parsing output from command substitution, and when performing variable substitution.

IFS can be redefined to parse one or more lines of data whose fields are not delimited by the default white-space characters. Consider this sequence of variable assignments and for loops:

line=learn:unix:at:livefire:labs
... ... ... OIFS=$IFS $ IFS=':' for i in $line; do; echo $i; done
learn
unix
		at
livefire
labs

The first command assigns the string "learn:unix:at:livefire:labs" to the variable named line. You can see from the first for loop that the shell treats the entire string as a single field. This is because the string does not contain a space, tab, or newline character.

After redefining IFS, the second for loop treats the string as four separated fields, each delimited by a colon. Using a colon for IFS would be appropriate when parsing the fields in a record from /etc/passwd, the user account information file:

livefire:x:100:1::/export/home/livefire:/bin/ksh		
Notice that the original value of IFS was stored in OIFS ("O" for original) prior to changing its value. After you are finished using the new definition, it would be wise to return it to its original value to avoid unexpected side effects that may surface later on in your script.

TIP – The current value of IFS may be viewed using the following pipeline:

echo "$IFS" | od -b
0000000 040 011 012 012
	0000004
$

The output of the echo command is piped into the octal dump command, giving you its octal equivalent. You can then use an ASCII table to determine what characters are stored in the variable. Hint: Ignore the first set of zeros and the second newline character (012), which was generated by echo.

Korn Shell Unix and Linux Programming Manual, Third Edition

Substitution

The first step the shell takes in executing a simple-command is to perform substitutions on the words of the command. There are three kinds of substitution: parameter, command and arithmetic. Parameter substitutions, which are described in detail in the next section, take the form $name or ${...}; command substitutions take the form $(command) or 'command'; and arithmetic substitutions take the form $((expression)). If a substitution appears outside of double quotes, the results of the substitution are generally subject to word or field splitting according to the current value of the IFS parameter. The IFS parameter specifies a list of characters which are used to break a string up into several words; any characters from the set space, tab and newline that appear in the IFS characters are called IFS white space. Sequences of one or more IFS white space characters, in combination with zero or one non-IFS white space characters delimit a field. As a special case, leading and trailing IFS white space is stripped (i.e., no leading or trailing empty field is created by it); leading or trailing non-IFS white space does create an empty field. Example: if IFS is set to ':', the sequence of characters 'A:B::D' contains four fields: 'A', 'B', '' and 'D'. Note that if the IFS parameter is set to the null string, no field splitting is done; if the parameter is unset, the default value of space, tab and newline is used.

[Chapter 10] Korn Shell Administration

Here is a script that looks for ^ in shell scripts in every directory in your PATH:

[2] The exact message varies from system to system; make sure that yours prints this message when given the name of a shell script. If not, just substitute the message your file command prints for "shell script" in the code below.

IFS=:
for d in $PATH; do
    print checking $d:
    cd $d
    scripts=$(file * | grep 'shell script' | cut -d: -f1)
    for f in $scripts; do
        grep '' $f /dev/null
    done
done

The first line of this script make it possible to use $PATH as an item list in the for loop. For each directory, it cds there and finds all shell scripts by piping the file command into grep and then, to extract the filename only, into cut. Then for each shell script, it searches for the ^ character. [3]

[3] The inclusion of /dev/null in the grep command is a kludge that forces grep to print the names of files that contain a match, even if there is only one such file in a given directory.

Embedding Newlines in Shell and Environment Values

The Answer Guy

Extra tidbit: I recently found a quirky difference between Korn shell ('93) and bash. Consider the following:

echo foo | read bar; echo $bar 
... whenever you see a "|" operator in a shell command sequence you should understand that there is implicitly a subshell (new process) that is created (forked) on one side of it or the other.

Of course other processes (including subshells) cannot affect the values of your shell variables. So the sequence above consists of three commands (echo the string "foo", read something and assign it to a shell variable named "bar", and echo the value of (read the $ dereferencing operator as "the value of") the shell named "bar"). It consists of two processes. One on one side of the pipe, and the other on the other side of the pipe. At the semicolon the shell waits for the completion of any programs and commands that precede it, and then continues with a new command sequence in the current shell.

The question becomes whether the subshell was created on the left or the right of the | in this command. In bash it is clearly created on the right. The 'read' command executes in a subshell. That then exits (thus "forgetting" its variable and environment heaps). Thus $bar is unaffected after the semicolon.

In ksh '93 and in zsh the subshell seems to be created to the left of the pipe. The 'read' command is executed in the current shell and thus the local value of "bar" is affected. Then the subsequent access to that shell variable does reflect the new value.

As far as I know the POSIX spec is silent on this point. It may even be that ksh '93 and zsh are in violation of the spec. If so, the spec is wrong!

It is very useful to be able to parse a set of command outputs into a local list of shell variables. Note that for a single variable this is easy:

bar=$(echo foo)
or: bar=`echo foo` ... are equivalent expressions and they work just fine.

However, when we want to read the outputs into several values, and especially when we want to do so using the IFS environment value to parse these values then we have to resort of inordinate amounts of fussing in bash while ksh '93 and newer versions of zsh allow us to do something like:

grep ^joe /etc/passwd | IFS=":" read login pw uid gid gecos home sh 
(Note the form: 'VAR=val cmd' as shown here is also a bit obscure but handy. The value of VAR is only affected for the duration of the following command --- thus saving us the trouble of saving the old IFS value, executing our 'read' command and restoring the IFS).

BTW: If you do need to save/restore something like IFS you must using proper quoting. For example:

OLDIFS="$IFS"
# MUST have double/soft quotes here!
IFS=:,
# do stuff parsing words on colons and commas
IFS="$OLDIFS"
# MUST also have double/soft quotes here!

Anyway, I would like to do some more teaching in the field of shell scripting. I also plan to get as good with C and Python as I currently am with 'sh'. That'll take at least another year or so, and a lot more practice!

sh vs. ksh

Others have showed you:

IFS='
'   # actual newline in quotes

My usual idiom for what you're doing is:

ps | while read nxt; do
    ...
done

But there's a gotcha here. The Bourne shell forks a separate shell for
a set of shell commands that are being piped to. It does a decent job
of hiding this fact, but your:

eval jbid_$dex=$e

Parameter Substitution in the Korn Shell or POSIX Shell

IFS Specifies internal field separators (normally space, tab, and new line) used to separate command words that result from command or parameter substitution and for separating words with the regular built-in command read. The first character of the IFS parameter is used to separate arguments for the $* substitution.

Korn shell exec, read and miscellaneous commands

The prompt can be specified in the read statement:

$ read var?prompt
If var is not defined, input is assigned to variable REPLY. Field separator can be assigned with the IFS ( Internal Field Separator) variable.
Example:
> cat kpwd
#!/bin/ksh
#-----------kpwd: read example in Korn shell
echo Proc $0: type pwd info with Korn shell
echo
read ok?"Type pwd info? (y/n)"                  #read with prompt
[[ $ok = @([Nn])* ]] && exit 1                  #test read variable
echo pwd data are:
echo ""
IFS=:                                           #set IFS to :
exec 0</etc/passwd                              #redirect stdin to /etc/passwd
# list users
#
while read -r NAME PAS UID GID COM HOME SHELL
do
   print "acct= $NAME - home= $HOME - shell= $SHELL:"
done
#----------end script------------------
> kpwd

Type pwd info? (y/n)y
pwd data are:

acct= john - home= /home/john - shell= /bin/tcsh:
acct= mary - home= /home/mary - shell= /bin/tcsh:
acct= tester - home= /d4/check - shell= /bin/sh:
>
If you need to run a script in the current shell either use the dot [.] or the source command:
$ . .profile
$ source .profile
$ rehash
In this way any variable, alias or function setting stay in effect. Note the rehash command that recreates the in-memory shell tables and grants that the system acknowledges new .profile definitions.



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July, 28, 2019