Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Bash expressions

News

Bash Recommended  Links BASH Debugging Bash Control Structures Single square bracket conditional

if statements in shell

Arithmetic Expressions in BASH String Operations in Shell Process Substitution in Shell Brace Expansion Command history reuse Advanced filesystem navigation Command completion
 IFS Dotfiles Shell Prompts Bash tips Annotated List of Bash Enhancements Humor Etc

This page is based on a lecture by Prof Nikolai Bezroukov, which in turn was partially based on Chapter 6 of Linux Shell Scripting with Bash by Ken O. Burtch

An expression in bash can be understood as formula that calculates some value. There are several types of expressions in bash. We will mention only three:

  1. Arithmetic Expressions
  2. String Expressions
  3. File expressions

bash simultaneously is a scripting language and a macro processor. So before expression is calculated its content is subjected to several passes of analysis on each of which specific language  element are processed (which is called macro expressions). In other languages there is a separate "preprocessor" phase, performed by a separate program. But in bash they are integrated into interpreter. During this process of macroexpansion macrovariables are replaced by their values. Each such action is usually called  macrosubstitution or simply substitution.  In case of double quoted literals the process of substituting of values of Bash variables into their  values is also called interpolation (the term more widely used in Perl then in bash).  

Bash performs certain types of macrosubstitution only outside literals(single or double quoted strings and backticked string are called literals) . For example Bash expands  tilde  to the user home directory only outside any literals

Outside literals after the values of variables and backticked strings were substituted Bash performs two additional operations:

  1. Word splitting of the resulting string ( arguments separation by whitespace, or the contents of the IFS variable). Outside double quoted literals there is  a real danger, that the string resulting from substitution of the  value of a variable contains spaces.  Such string  will be treated as several "tokens" by syntax analyzer, which often leads to syntax error. Or is the variable expanded to zero length string.  In this case the  systex analyser will report missing token. That's why you see "paranoid" style of in olde shells (and sometimes even in bash) where when you compare two variables in if statement, noth variable are enclosed in double quotes. This is done to prevent errors mentioned above. But in modern bash in most case this is unneeded precousion,  if you use double square or round bracket expressions, so this paranoid style is to a certain extent an archaism: just use [[...]] or ((...)) expression and you will be fine.
  2. Pathname expansion ( aka pathname pattern matching using basic regex)

The result is passed to syntax phase of Bash interpreter, where the syntax analyzer try to split script text into statements. If errors are found the process is aborted. After that script is executed one statement at a time.

The order is important and can cause subtle problems in scripts. Suppose you assign a path with a tilde in it in a variable (tile means home directory in Bash):

ls -d ~/bin
/home/joeuser/bin

You can see the tilde here work as expected. Now let's enclose the expression ~/bin in double quotes:

$ TEMP="~/bin"
$ ls $TEMP
~/bin not found

You can see that this does not work -- tilde substitution double  quoted literals not performed as explained above. In double quotes literals tilde symbol is not treated as marcosymbol, represnting home directory, but as a regular symbol. that means that within any literal including double quote literals the tilde will remain  "as is". In such cases you should use the $HOME variable instead.

Conditional expressions were not part of the original Borne shell (which was a blunder on the part of its designer even in the situation when memory was tight -- ancient computers that run Unix often have one megabyte (megabyte, not gigabyte) of memory or even less.

In any case in original Borne shell conditional expressions were implemented via external command called test which accepts sequence of arguments which are interpreted by this program as expression.  As later version of shell (Korn shell) introduced double square brackets and double round brackets expressions which are much better deal, this archaic way  should better be forgotten (double square brackets or double round brackets expressions should be used exclusively in you new scrips), but it is preserved for compatibility so we  need to know  about its existence.

the test command evaluates expression and return the corresponding "return  code" -- 0 if expression is true, and non-zero it is false. 

Now let's see hot it issued within bash if command, which allow to structure control stream of your scripts into two branches: then branch which is executed if the condition is true and (optional) else branch which is executed, if the conditions false.

File expressions in bash

Bash allows for a wide variety of tests for files. It can test the type of file, the accessibility of the file, compare the ages of files, or other file attributes. The following is a complete list of Bash file tests:

File testing is commonly used in the sanity checks at the beginning of a script. They can be used to check that all files are present and readable (and writable, if necessary). All commands must be executable

#!/bin/bash
#
# Nikolai Bezroukov

shopt -s -o nounset

# get machine from SGE $PE_HOSTFILE  variable
cat /dev/null > machines # make ./machine a zero length file and add lines to it using while loop
cat $PE_HOSTFILE | while read line; do
   host=`echo $line | cut -d" " -f1`
   cores=`echo $line | cut -d" " -f2`
      while (($cores > 0 )) ; do
         echo $host >> machines
         let cores--
      done
done
## done with $PE_HOSTFILE
# Sanity check 
if [[ -f ./machines ]] ; then
  echo "The ./machines file exists. We can proceed"
else 
  echo "The machines file does not exits. Exiting ..."
  exit
fi 

This script fragment ensures that the who command is executable and that the temp file named in TMP either doesn't exist or exists and can be overwritten.

Arithmetic Expressions using let and ((...))

The built-in let command performs math calculations on variables without declaring then to be integer. let expects a string containing a variable, an equals sign, and an expression to calculate. the result is assigned to the variable.

let "large_run=$cores*2"
printf "%d" $large_run

One good thing about ((...)) is that they are impressive: if you forget to put dollar sign in front of the variable name to extract the value of this this variable, it still will interpret the statement correctly. Which is a big plus for users of other languages, other then Perl:

let "large_run=cores*2"
printf "%d" $large_run

Double quote are required with let because you need to prevent bash from interpreting special characters such as "*" and "/" that are used in bash for other purposes too. This way you can avoid subtle ambiguity due to double meaning of * and / in bash. You do not need to do this is you use ((..)), so this is preferable form for complex arithmetic expressions and the only one that is suitable for use in if , while and other conditional statement. The same can better be written as

(( large_run=$cores*3 )) 
printf "%d" $large_run

Another example

$ (( double_cores = 2*$cores ))
$ echo $double_cores
A couple more examples
(( RC = 5 / 2 )) 
printf "5 divided by 2 is %d\n" "$RC"
5 divided by 2 is 2

(( RC = 5 % 2 ))
printf "remainder of 5 divided by 2 is %d\n" "$RC"
remainder of 5 divided by 2 is 1

Here are the arithmetic operators in order of precedence. Many of these operators will be familiar to C programmers:

The parentheses must be escaped to prevent the shell from treating them as a reference to a subshell.

Relational Operations

In if statement true is represented by the value of 1, and false by 0. Any value other than 1 or 0 is treated as true, but the logical operators themselves only return 1 or 0.Unlike string comparisons, let provides a full complement of numeric comparisons. These are of limited value because most of comparisons are tested with the test command in the if command, resulting in two sets of tests. In logical expressions, 0 is a failure.

declare -i RC=(( 1> 0 )
If (( $RC )) ; then  
   printf "1 greater than 0 is TRUE: %d\n" 
else
   printf "something is really wrong %d\n" $RC
   exit
fi 

Let's discuss C-style self-referential operations again

As we already know, self-referential operators are shorthand notations that combine assignment with one of the other basic operations.

This is the only case where let is superior of ((..)) expressions.

declare NODE="HP_blade"
declare -i cores=20
if [[ $NODE == 'HP_blade" ]] ; 
   let cores+=8
}
printf "We will use %d cores" cores

The other self-referential operators are multiplication (*=), division (/=), remainder (%=), subtraction (-=), right shift (<<=), left shift (>>=), bitwise AND (&=), bitwise exclusive OR (^=) , and bitwise OR (|=).

Notice that certain kinds of self-referential operations are impossible to express with the shorthand operators. count=count-10 can be written as count-=10 but count=10-count must be written out in full.

The increment (++) operator adds one to a variable. The decrement operator (--) subtracts one from a variable.

$ RUNNO=`cat ./runno"
$ let RUNNO++"
printf "================= RUN %d ========================\n" "$RUNNO"

The self-referential operators cannot be used with integer variables without the let statement.

$ COST+=5
bash: COST+=5: command not found

Other let and ((.. )) Features

Parentheses are allowed in the expressions.

if (( $cores=(20+8)*3 )) then 
   printf "WARNING ! You are probably too greedy and this amount of cores will porbably slow down your computation %d" cores
fi

Assignment in the let command is an operator that returns the value being assigned. As a result, multiple variables can be assigned at once.

(( threads=cores=16 )) 
printf "wr will be using are %d threads and %d cores\n" $threads $cores

let can evaluate more than one expression at a time. Several, small assignments can be combined on one line.

let "SUM=5+5" "SUM2=10+5"				

NOTE: Excessive numbers of assignments in a single let command will lead to readability problems in a script. When each let is on a single line, it's easier to look through the script for a specific assignment.

The conditional expression operator (?) is shorthand for an if statement when one of two different expression are evaluated based on the left condition. Because this is a feature of the let command, it works only with numeric expressions, not strings. The following example constrains a truth value to a 1 or a 0.

cores=16
let "NODE= $cores > 20 ? 'HP_blade' : 'Dell_blade'
printf "%d\n" $cores

Often you are better off using regular if.

input_file_count=`ls -l *.in | wc -l`
if (( $input_file_count == 0 )) ; then 
   echo No input file provides. Exiting..."
   exit
fi

Again, in most case it is better to use double parentheses then let. Two exceptions are simple assignment of a type a=b+c and self referential oprators. In both cases you do not need to use double quotes with let unless you use * or /.

Double parenthesis are also natural in conditional statement such as if and various loops (which we will study later). For example while in while loop:

declare -i count=0;
while (( count < 10 )) ; do
  printf "%d\n" "$count"
  let count++
done

To review let arithmetic, the script temperature.bash, shown in Listing 6.2, converts a temperature from Fahrenheit to Celsius.

#!/bin/bash
#
# temperature.bash: Convert Fahrenheit to Celsius
#
# Ken O. Burtch
# CVS $Header$

shopt -s -o nounset

declare -i FTEMP  # Fahrenheit temperature
declare -i CTEMP  # Celsius temperature

# Title

printf "%s\n" "Fahrenheit-Celsius Conversion"
printf "\n"

# Get the value to convert

read -p "Enter a Fahrenheit temperature: " FTEMP

# Do the conversion

(( CTEMP=(5*($FTEMP-32) )/9 ))
printf "The Celsius temperature is %d\n" "$CTEMP"

exit 0

Pattern Recognition

Bash pattern recognition is called globbing. Globbing is used to match filenames given to a command, and it is also used by the Korn shell test command to match strings.

$ ls *.txt
notes.txt  project_notes.txt

The pattern-recognition feature works by supplying wildcard symbols that Bash will attempt to match to a string or a filename.

The asterisk (*) character represents zero or more characters. The Korn shell test can be used to match the value of a variable to a string with an asterisk.

# Future of the companies predictor
COMPANY="Abracadabra"
if [[ $COMPANY = A* ]] ; then
   printf "The company name begins with a letter A. This company will probably prosper... \n"
fi
if [[ $COMPANY = Z* ]] ; then
   printf "The company name begins with a letter Z. this company probably will go down...\n"
fi

This behavior doesn't work when quotation marks are used. Quotation marks tell the Korn shell test command not to interpret special characters. A test for "A*" would indicate a file named "A*".

COMPANY="Behemot"
if [[ $COMPANY = B* ]] ; then
   printf "The company name is starting with letter B\n"
fi

The question mark (?) character is a wildcard representing any single character.

COMPANY="Behemot"
if [[ $COMPANY = ???? ]] ; then
   printf "The company name is 4 characters long\n"
fi

You can specify a set of characters using square brackets. You can list individual characters or ranges.

if [[ $COMPANY = [ABC]* ]] ; then
   printf "The company name begins with a A, B or C\n"
fi
if [[ $COMPANY = [A-Z]* ]] ; then
   printf "The company name begins with a letter an uppercase letter\n"
fi
if [[ $COMPANY = [A-Z0-9]* ]] ; then
   printf "The company name begins with a letter an uppercase "\
"letter or number\n"
fi

Partial ranges are allowed. If the start of the range is omitted, the range is all ASCII values up to that character. Likewise, the end of the range can be omitted to specify any ASCII characters after the range start character.

If the first character in a square bracket set is a exclamation point (!) or caret (^), characters not in the set count as a match.

Bash defines short forms for common ranges called character classes:

For example, [:lower:] is equivalent to the range [a-z].

COMPANY="2nd Rate Solutions"
if [[ $COMPANY = [[:digit:]]*]] ; then
   printf "Company name starts with a digit\n"
fi

Basic regular expressions

There are a number of shell options that affect pattern matching. Filename pattern matching can be turned off completely by setting the noglob shell option (shopt -s -o noglob). If the nullglob option is set (with shopt -s nullglob), the pattern is discarded if not matched. In the previous example, ls would receive no argument and would list all the files in the current directory. You can disable case sensitivity by setting the nocaseglob option (shopt -s nocaseglob). Likewise, you can turn off the special treatment of leading periods in filenames by setting the dotglob option (shopt -s dotglob).

These shell options have a wide-ranging effect; use them with care. Change the setting for as few lines of a script as possible and clearly comment the effect of the change.

The GLOBIGNORE variable also affects how filenames are matched. This variable resembles the PATH variable; it's a colon-separated list of filenames that Bash will not attempt to match. If you are aware of files with pattern symbols in their names, adding them to the GLOBIGNORE list will ensure that Bash will treat them as files and not as globbing expressions.

If you need more sophisticated pattern recognition, or if you need to apply a pattern to an entire file, you can use the grep family of commands.

Filename Brace Expansion ( {..} )

One filename can be expanded into multiple filenames using curly braces. Inside the curly braces, use commas to list each substring in order to build a new argument. Brace expansion is typically used to specify a root filename with different endings.

printf "%s %s %s\n" "Files should be named:" orders{.txt,.out}
Files should be named: orders.txt orders.out

Because Bash expands the first line to this:

printf "%s %s %s\n" "Files should be named:" orders.txt orders.out
				

three %s codes are necessary, one for each parameter to printf.

The braces can contain file-matching characters. The actual file matching occurs after the braces are processed by Bash.

Escape sequences

If a dollar sign is followed by a string in single quotes, the string is searched for ANSI C escape sequences and the sequences are replaced by the corresponding characters. The acceptable escape sequences are similar to the special format escape codes used by the printf command:

Variable Name Matching {!*}

If the contents of curly braces start with an exclamation point (!) and end with an asterisk (*), a list of all variables starting with the specified letters is returned.

$ COMPANY="Nightlight Inc."
printf "%s\n" "${!COMP*}"
COMPANY

Variable Length (#)

A dollar sign with curly braces and a leading number sign (#) returns the length of the variable's value.

printf "%s\n" "${#COMPANY}"
15

An asterisk(*) or at sign (@) returns the number of parameters to the shell script.

printf "%s\n" "${#*}"
0

This is identical to the value of $* built in variable

Variable Existence Check (:?)

If a trailing colon question mark appears after the variable name, the message following the question mark is returned and the Bash script exits. This provides a crude form of a sanity check. The message is optional.

printf "Company is %s\n" ${COMPANY:? Error: Not defined.}

Substrings (:n)

If a trailing colon followed by a number appears after the variable name, a substring is returned. The number indicates the first position of the substring, minus one. The first character in the string is character 0. Optionally, a colon and a second number can follow, which indicates the length of the substring.

printf "%s\n" "${COMPANY:5}"
light Inc.
printf "%s\n" "${COMPANY:5:5}"
light

Substring Removal by Pattern (%, #, %%, and ##)

If a trailing number sign appears after the variable name, the substring returned has the matching pattern removed. One number sign matches the smallest possible substring and two number signs matches the largest possible substring. The expression returns the characters to the right of the pattern.

printf "%s\n" "${COMPANY#Ni*}"
ghtlight Inc.
printf "%s\n" "${COMPANY##Ni*}"
printf "%s\n" "${COMPANY##*t}"
 Inc.
printf "%s\n" "${COMPANY#*t}"
light Inc.

Using percent signs (%), the expression returns the characters to the left of the pattern.

printf "%s\n" "${COMPANY%t*}"
Nightligh
printf "%s\n" "${COMPANY%%t*}"
Nigh

Here is one useful idiom: obtaining script name from the built-in variable $0

declare -r SCRIPT_NAME=${0##*/}            #  the name of this script

Substring Replacement by Pattern (//)

If the variable is followed by a slash (/), the first occurrence of the pattern following the slash is replaced. Following the pattern, there is a second slash and the replacement string. If the variable is followed by two slashes, all occurrences of the pattern are replaced.

printf "%s\n" "${COMPANY/Inc./Incorporated}"
Nightlight Incorporated
printf "You are the I in %s" "${COMPANY//i/I}"
You are the I in NIghtlIght Inc.

If the pattern begins with a number sign (#), the pattern matches at the beginning of the variable's value. If the pattern ends with a percent sign (%), the pattern matches at the end of the variable's value. Other occurrences are ignored.

$ COMPANY="NightLight Night Lighting Inc."
printf "%s\n" "$COMPANY"
NightLight Night Lighting Inc.
printf "%s" "${COMPANY//Night/NIGHT}"
NIGHTLight NIGHT Lighting Inc.
printf "%s" "${COMPANY//#Night/NIGHT}"
NIGHTLight Night Lighting Inc.

If no new value is indicated, the matching substring is deleted.

$ COMPANY="Nightlight Inc."
printf "%s\n" "${COMPANY/light}"
Night Inc.

Ranges can also be used. For example, to delete all the punctuation in a string, use the range [:punct:]:

printf "%s" "${COMPANY//[[:punct:]]}"
Nightlight Inc

Using an asterisk or at sign instead of a variable applies the substitutions to all the parameters to the shell script. Likewise, an array with an asterisk or at sign applies the substitutions to all elements in the array.

Command result substitution $( (..) )

When parentheses are used adder the dollar sign , the command inside them is executed. This has the same effect as enclosing the commands in backquotes.

printf "There are %d files in this directory\n" "$(ls -1 | wc -l)"
There are 28 files in this directory
printf "There are %d files in this directory\n" `ls -1 | wc -l`
There are 28 files in this directory

Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Sites

Top articles

Sites

...



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 12, 2019