See Also
Referenced By
The man page hstr(1) is an alias of hh(1).
Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells
| Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers | |
|
May the source be with you, but remember the KISS principle ;-) Bigger doesn't imply better. Bigger often is a sign of obesity, of lost control, of overcomplexity, of cancerous cells |
|||||||
|
|
Bash is a standard shell on Linux and with version 3.2 or later available on all enterprise platform and installed on Solaris and AIX it make sense to make it standard interactive shell. The current version as of June 2017 is 4.2. Only HP-UX does not include bash by-default, but HP does provide a depot package for bash 4.00. Bash 3.2 and later has important enhancements that make it a better shell (see Annotated List of Bash Enhancements) although scripts should be generally limited to small to medium size as Perl represents much better scripting environment then bash and is installed by default on all platforms including HP-UX and AIX.
|
|
Bash can be called an accidental scripting language. If started very modestly more then 40 years ago with Borne shell (1977) and C shell (1978). And then gradually had grown into a scripting language. the first major enhancement were done in ksh88 and then ksh93, Then bash 30, and 4,0 introduced additional one (but generally in the framework of ksh93).
The key problem with bash is that it is "rarely used language" for most users. That means that while users might use bash as a command interpreter daily, they program more or less complex scripts rarely (say once a quarter on average). And from one time to another manage to forget most of the important staff related to this language. That makes programming in bash a real challenge.
Also your programming skills evaporate in several months or so, unless you carefully document your "achievements". You will forget them and go down again to the most basic level of usage of this language when you need to write another bash script.
And you might repeat the same mistakes or even blunders again and again. Logbooks is better then nothing but generally is not enough. You need a personal wiki. Frontage can serve as a surrogate Wiki on windows very well. One of the advantages is that you do not need to learn special wiki script. You can use HTML. The author uses it in this capacity for 20 year or so.
Bash is so called "glue" language and set of classic Unix utilities represent important part of bash functionality. So in addition to knowledge of bash you need to know at least several dozens of Unix utilities (and you better know them well ;-). That requirement alone often lead to 'stack overflow". So mental crutches in the form of your custom references are needed and creating them is a the necessary step because of complexity of the environment. Which now is beyond human understanding (the situation called Blind men and an elephant - Wikipedia).
At the current level of complexity the fact that bash has source code available changes absolutely nothing for 99% of Unix/linux sysadmins. It is a classic example of self-closing open source ;-).
At the beginning a good reference book might suffice too (see Best Shell Books for recommendations) but gradually you should create you own website or wiki and work on enhancing it. Even people with 20 year Unix experience often can't remember of the vital option of the most common utilities such as ls, grep and find.
Also blunders in bash can be very costly and lead to the lostt of data, crash of the OS or both. They are often connected iether with misunderstanding of bash behaviour or misunderstanding of behaviour or one of Unix utilities you use, or Unix itself, or make accidentally or out of exhaustion or under time pressure some nasty error yourself. See Sysadmin Horror Stories
| The key problem with bash is that it is "rarely used language" for most users. That means that while users might use bash as a command interpreter daily, they program more or less complex scripts rarely (say once a quarter on average). And from one time to another manage to forget most of the important staff related to this language. That makes programming in bash a real challenge. |
While bash has a humble start and much weaker designer then ksh93 and was at times extremely buggy, the gap narrowed in version 3.2 to the extent that better command line user friendliness of bash make it more attractive, especially as an interactive shell then supplied with OS ksh and C-shell. Most ksh93 innovations are now present in some form in bash 3.2. As bash is standard shell on Linux and is installed by default in Solaris it status as an enterprise shell is almost as strong as ksh93 (which is mostly present in old, weaker versions. Current bash has the best debugger and from this point of view represents the best shell. But portable script still are probably better to be written for ksh88 or POSIX shell which is lowest common denominator available on all Unixes. To write scripts in Borne shell now is extremely stupid and wasteful.
Bash 3.2 and later is the one of the most portable advanced shell around (ksh93 and zsh are still a strong competition; ksh93 is definitely more reliable for scripts and does not contain such design blunders as the last stage of the pipe belonging to subshell instead of the invoking shell (actually fixed in lager version of bash).
Bash-related books dominates shell-related publications and as such the level of known-how for bash is higher then for other shells (see Best Shell Books). The advantage of bash for large enterprise environment is that it comes by default with linux, Solaris and AIX (unfortunately in pretty different versions). Only HP-UX does not have bash installed by default. Also it is the best portable interactive shell, much closer to tcsh then any competition.
Still you need some efforts to make the default shell. Unfortunately the default shell for Solaris is the "Bourne shell" or /usr/bin/sh
Bourne shell is a pretty weak outdated shell and attempt to base shell scripting portability on this outdated shell is a serious strategic error (bash probably should be used instead). It is still used as root shell in Solaris ,but that's due to Solaris legacy not because it gives anything but disadvantages; attempts to claim that this somehow increases the security of root due to the fact that it is static linked are weak and the argumentation is open for discussion. All-in-all usage of Bourne shell as a default root shell in Solaris might be considered to be a blunder: system administrators definitely need a better shell.
Usage of Borne shall as a default shell might slightly increase the chances of recovery in case /usr partition is damaged, but this is a pretty serious case and usually means serious troubles with other partitions on the disk anyway (unless this is the case when in Solaris link /bin -> usr/bin is destroyed, but such cases are simple to fight by refereeing shell as /usr/bin/ksh in /etc/passwd). If this is a serious trouble than booting from a CD a mounting the damaged volume is always a good idea and in this case it does not matter what shell root is using; you can change it anyway.
Bash 3.2 is reasonably easy to build from source. Get and unpack in your home directory the archive. This will create a directory called bash-3.2 in your home directory. If you do not have the gunzip utility, you can obtain it in the same way you obtained bash or simply use gzip -d instead. The archive contains all of the source code needed to compile bash and a large amount of documentation and examples. the latter have their own value.The bash archive contains a main directory and a set of files and subdirectories. Among the first files you should examine are:
CHANGES A comprehensive list of bug fixes and new features since the last version
COPYING The GNU Copyleft for bash
MANIFEST A list of all the files and directories in the archive
NEWS A list of new features since the last version
README A short introduction and instructions for compiling bash
doc Directory with information related to bash in various formats (please note that many syadmin never read those docs)
examples Directory with examples of startup files, scripts, and functions
The doc directory contains a few articles that are worth reading. You can print man page for reference with command nroff-man bash.1 | more It is convenient to have a hardcopy so you can write notes all over it. Also valuable, FAQ is a Frequently Asked Questions document with answers, readline.3 is the manual entry for the readline facility, and article.ms is an article about the shell that appeared in Linux Journal, and was written by the current bash maintainer Chet Ramey.
An examples directory is especially important and is well worth exploring (after you've finished reading this book, of course). It includes sample code, scripts, functions, and startup files. See Examples shipped with bash 3.2 and newer
| An examples directory directory is especially important and is well worth exploring (after you've finished reading this book, of course). It includes sample code, scripts, functions, and startup files. See Examples shipped with bash 3.2 and newer |
Some interesting features of bash include:
<(list) To substitute a command pipeline for
an input file >(list) To substitute a command pipeline for
an output file In the case of the < and > forms, the shell will
run process list asynchronously, connected to a named pipe (FIFO).
The name of this pipe will become the argument to the command.
|
Please visit Heiner Steven SHELLdorado the best shell scripting site on the Internet |
< is selected then result of execution of the
process list will serve as an input file. This, for allows you to use the output
of one or several commands as parameters to the utilities that accepts file.
For instance you can compare the contents of two directories by typing:diff <( ls dir1 ) <( ls dir2 )
That can be used for concatenating input in pipes:
cat <(echo hello) <(echo world) | echo
If < is used, then the file passed as an argument will be a
named pipe connected to the output of the list process.
Another, more complex, example,
paste <(cut -f1 file1) < (cut -f3 file2) | tee >(process1) >(process2) >/dev/null
cuts fields 1 and 3 from the files file1 and file2 respectively, pastes the results together, and sends it to the processes process1 and process2. Note that the file, which is passed as an argument to the command, is a system pipe so programs that expect to lseek(2) on the file will not work. Also note that the previous example can be more compactly and efficiently written as:
paste <(cut -f1 file1) <(cut -f3 file2) >>(process1)
>>(process2)
The shell uses pipes instead of FIFOs to implement the latter two process substitutions in the above example.
See also the article in Linux Journal:
Bash
Process Substitution
Due to this bash shell is gradually gaining grounds as the preferred interactive shell for Solaris and other enterprise class Unixes.
Older version of bash (2.x series) are obsolete and should not be used. The recommended version is 3.2 patch level 3 or above. 4.x is recommended on RHEL 6.x and up.
Some pages that form kind of bash tuorial:
Dr. Nikolai Bezroukov
 |
 |
 |
Feb 21, 2019 | alvinalexander.com
By Alvin Alexander. Last updated: June 22 2017 Unix/Linux bash shell script FAQ: How do I prompt a user for input from a shell script (Bash shell script), and then read the input the user provides?
Answer: I usually use the shell script
readfunction to read input from a shell script. Here are two slightly different versions of the same shell script. This first version prompts the user for input only once, and then dies if the user doesn't give a correct Y/N answer:# (1) prompt user, and read command line argument read -p "Run the cron script now? " answer # (2) handle the command line argument we were given while true do case $answer in [yY]* ) /usr/bin/wget -O - -q -t 1 http://www.example.com/cron.php echo "Okay, just ran the cron script." break;; [nN]* ) exit;; * ) echo "Dude, just enter Y or N, please."; break ;; esac doneThis second version stays in a loop until the user supplies a Y/N answer:
while true do # (1) prompt user, and read command line argument read -p "Run the cron script now? " answer # (2) handle the input we were given case $answer in [yY]* ) /usr/bin/wget -O - -q -t 1 http://www.example.com/cron.php echo "Okay, just ran the cron script." break;; [nN]* ) exit;; * ) echo "Dude, just enter Y or N, please.";; esac doneI prefer the second approach, but I thought I'd share both of them here. They are subtly different, so not the extra break in the first script.
This Linux Bash 'read' function is nice, because it does both things, prompting the user for input, and then reading the input. The other nice thing it does is leave the cursor at the end of your prompt, as shown here:
Run the cron script now? _(This is so much nicer than what I had to do years ago.)
|
|
|
Nov 17, 2018 | dvorka.github.io
hstr -- Bash and zsh shell history suggest box - easily view, navigate, search and manage your command history.
Configuration
Get most of HSTR by configuring it with:
hstr --show-configuration >> ~/.bashrcRun
hstr --show-configurationto determine what will be appended to your Bash profile. Don't forget to source~/.bashrcto apply changes.
For more configuration options details please refer to:
- bind HSTR to a keyboard shortcut
- Bash Emacs keymap (default)
- Bash Vim keymap
- zsh Emacs keymap (default)
- create
hhalias forhstr- het more colors
- choose default history view
- set filtering preferences
- configure commands blacklist
- disable confirm on delete
- tune verbosity
- history settings:
Check also configuration examples .
Binding HSTR to Keyboard ShortcutBash uses Emacs style keyboard shortcuts by default. There is also Vi mode. Find out how to bind HSTR to a keyboard shortcut based on the style you prefer below.
Check your active Bash keymap with:
bind -v | grep editing-mode bind -v | grep keymapTo determine character sequence emitted by a pressed key in terminal, type Ctrlv and then press the key. Check your current bindings using:
bind -SBash Emacs Keymap (default)Bind HSTR to a Bash key e.g. to Ctrlr :
bind '"\C-r": "\C-ahstr -- \C-j"'or CtrlAltr :
bind '"\e\C-r":"\C-ahstr -- \C-j"'or CtrlF12 :
bind '"\e[24;5~":"\C-ahstr -- \C-j"'Bind HSTR to Ctrlr only if it is interactive shell:
if [[ $- =~ .*i.* ]]; then bind '"\C-r": "\C-a hstr -- \C-j"'; fiYou can bind also other HSTR commands like
--kill-last-command:if [[ $- =~ .*i.* ]]; then bind '"\C-xk": "\C-a hstr -k \C-j"'; fiBash Vim KeymapBind HSTR to a Bash key e.g. to Ctrlr :
bind '"\C-r": "\e0ihstr -- \C-j"'Zsh Emacs KeymapBind HSTR to a
zshkey e.g. to Ctrlr :bindkey -s "\C-r" "\eqhstr --\n"AliasIf you want to make running of
hstrfrom command line even easier, then define alias in your~/.bashrc:alias hh=hstrDon't forget to source
Colors~/.bashrcto be able to to usehhcommand.Let HSTR to use colors:
export HSTR_CONFIG=hicoloror ensure black and white mode:
export HSTR_CONFIG=monochromaticDefault History ViewTo show normal history by default (instead of metrics-based view, which is default) use:
export HSTR_CONFIG=raw-history-viewTo show favorite commands as default view use:
export HSTR_CONFIG=favorites-viewFilteringTo use regular expressions based matching:
export HSTR_CONFIG=regexp-matchingTo use substring based matching:
export HSTR_CONFIG=substring-matchingTo use keywords (substrings whose order doesn't matter) search matching (default):
export HSTR_CONFIG=keywords-matching
Make search case sensitive (insensitive by default):
export HSTR_CONFIG=case-sensitiveKeep duplicates in
raw-history-view(duplicate commands are discarded by default):export HSTR_CONFIG=duplicatesStatic favoritesLast selected favorite command is put the head of favorite commands list by default. If you want to disable this behavior and make favorite commands list static, then use the following configuration:
export HSTR_CONFIG=static-favoritesSkip favorites commentsIf you don't want to show lines starting with
#(comments) among favorites, then use the following configuration:export HSTR_CONFIG=skip-favorites-commentsBlacklistSkip commands when processing history i.e. make sure that these commands will not be shown in any view:
export HSTR_CONFIG=blacklistCommands to be stored in
~/.hstr_blacklistfile with trailing empty line. For instance:cd my-private-command ls llConfirm on DeleteDo not prompt for confirmation when deleting history items:
export HSTR_CONFIG=no-confirmVerbosityShow a message when deleting the last command from history:
export HSTR_CONFIG=verbose-killShow warnings:
export HSTR_CONFIG=warningShow debug messages:
export HSTR_CONFIG=debugBash History SettingsUse the following Bash settings to get most out of HSTR.
Increase the size of history maintained by BASH - variables defined below increase the number of history items and history file size (default value is 500):
export HISTFILESIZE=10000 export HISTSIZE=${HISTFILESIZE}Ensure syncing (flushing and reloading) of
.bash_historywith in-memory history:export PROMPT_COMMAND="history -a; history -n; ${PROMPT_COMMAND}"Force appending of in-memory history to
.bash_history(instead of overwriting):shopt -s histappendUse leading space to hide commands from history:
export HISTCONTROL=ignorespaceSuitable for a sensitive information like passwords.
zsh History SettingsIf you use
zsh, setHISTFILEenvironment variable in~/.zshrc:export HISTFILE=~/.zsh_historyExamplesMore colors with case sensitive search of history:
export HSTR_CONFIG=hicolor,case-sensitiveFavorite commands view in black and white with prompt at the bottom of the screen:
export HSTR_CONFIG=favorites-view,prompt-bottomKeywords based search in colors with debug mode verbosity:
export HSTR_CONFIG=keywords-matching,hicolor,debug
|
|
|
May 14, 2012 | stackoverflow.com
Lgn ,May 14, 2012 at 15:15
In a Bash script I would like to split a line into pieces and store them in an array.The line:
Paris, France, EuropeI would like to have them in an array like this:
array[0] = Paris array[1] = France array[2] = EuropeI would like to use simple code, the command's speed doesn't matter. How can I do it?
antak ,Jun 18, 2018 at 9:22
This is #1 Google hit but there's controversy in the answer because the question unfortunately asks about delimiting on,(comma-space) and not a single character such as comma. If you're only interested in the latter, answers here are easier to follow: stackoverflow.com/questions/918886/ – antak Jun 18 '18 at 9:22Dennis Williamson ,May 14, 2012 at 15:16
IFS=', ' read -r -a array <<< "$string"Note that the characters in
$IFSare treated individually as separators so that in this case fields may be separated by either a comma or a space rather than the sequence of the two characters. Interestingly though, empty fields aren't created when comma-space appears in the input because the space is treated specially.To access an individual element:
echo "${array[0]}"To iterate over the elements:
for element in "${array[@]}" do echo "$element" doneTo get both the index and the value:
for index in "${!array[@]}" do echo "$index ${array[index]}" doneThe last example is useful because Bash arrays are sparse. In other words, you can delete an element or add an element and then the indices are not contiguous.
unset "array[1]" array[42]=EarthTo get the number of elements in an array:
echo "${#array[@]}"As mentioned above, arrays can be sparse so you shouldn't use the length to get the last element. Here's how you can in Bash 4.2 and later:
echo "${array[-1]}"in any version of Bash (from somewhere after 2.05b):
echo "${array[@]: -1:1}"Larger negative offsets select farther from the end of the array. Note the space before the minus sign in the older form. It is required.
l0b0 ,May 14, 2012 at 15:24
Just useIFS=', ', then you don't have to remove the spaces separately. Test:IFS=', ' read -a array <<< "Paris, France, Europe"; echo "${array[@]}"– l0b0 May 14 '12 at 15:24Dennis Williamson ,May 14, 2012 at 16:33
@l0b0: Thanks. I don't know what I was thinking. I like to usedeclare -p arrayfor test output, by the way. – Dennis Williamson May 14 '12 at 16:33Nathan Hyde ,Mar 16, 2013 at 21:09
@Dennis Williamson - Awesome, thorough answer. – Nathan Hyde Mar 16 '13 at 21:09dsummersl ,Aug 9, 2013 at 14:06
MUCH better than multiplecut -fcalls! – dsummersl Aug 9 '13 at 14:06caesarsol ,Oct 29, 2015 at 14:45
Warning: the IFS variable means split by one of these characters , so it's not a sequence of chars to split by.IFS=', ' read -a array <<< "a,d r s,w"=>${array[*]} == a d r s w– caesarsol Oct 29 '15 at 14:45Jim Ho ,Mar 14, 2013 at 2:20
Here is a way without setting IFS:string="1:2:3:4:5" set -f # avoid globbing (expansion of *). array=(${string//:/ }) for i in "${!array[@]}" do echo "$i=>${array[i]}" doneThe idea is using string replacement:
${string//substring/replacement}to replace all matches of $substring with white space and then using the substituted string to initialize a array:
(element1 element2 ... elementN)Note: this answer makes use of the split+glob operator . Thus, to prevent expansion of some characters (such as
*) it is a good idea to pause globbing for this script.Werner Lehmann ,May 4, 2013 at 22:32
Used this approach... until I came across a long string to split. 100% CPU for more than a minute (then I killed it). It's a pity because this method allows to split by a string, not some character in IFS. – Werner Lehmann May 4 '13 at 22:32Dieter Gribnitz ,Sep 2, 2014 at 15:46
WARNING: Just ran into a problem with this approach. If you have an element named * you will get all the elements of your cwd as well. thus string="1:2:3:4:*" will give some unexpected and possibly dangerous results depending on your implementation. Did not get the same error with (IFS=', ' read -a array <<< "$string") and this one seems safe to use. – Dieter Gribnitz Sep 2 '14 at 15:46akostadinov ,Nov 6, 2014 at 14:31
not reliable for many kinds of values, use with care – akostadinov Nov 6 '14 at 14:31Andrew White ,Jun 1, 2016 at 11:44
quoting${string//:/ }prevents shell expansion – Andrew White Jun 1 '16 at 11:44Mark Thomson ,Jun 5, 2016 at 20:44
I had to use the following on OSX:array=(${string//:/ })– Mark Thomson Jun 5 '16 at 20:44bgoldst ,Jul 19, 2017 at 21:20
All of the answers to this question are wrong in one way or another.
IFS=', ' read -r -a array <<< "$string"1: This is a misuse of
$IFS. The value of the$IFSvariable is not taken as a single variable-length string separator, rather it is taken as a set of single-character string separators, where each field thatreadsplits off from the input line can be terminated by any character in the set (comma or space, in this example).Actually, for the real sticklers out there, the full meaning of
$IFSis slightly more involved. From the bash manual :The shell treats each character of IFS as a delimiter, and splits the results of the other expansions into words using these characters as field terminators. If IFS is unset, or its value is exactly <space><tab><newline> , the default, then sequences of <space> , <tab> , and <newline> at the beginning and end of the results of the previous expansions are ignored, and any sequence of IFS characters not at the beginning or end serves to delimit words. If IFS has a value other than the default, then sequences of the whitespace characters <space> , <tab> , and <newline> are ignored at the beginning and end of the word, as long as the whitespace character is in the value of IFS (an IFS whitespace character). Any character in IFS that is not IFS whitespace, along with any adjacent IFS whitespace characters, delimits a field. A sequence of IFS whitespace characters is also treated as a delimiter. If the value of IFS is null, no word splitting occurs.
Basically, for non-default non-null values of
$IFS, fields can be separated with either (1) a sequence of one or more characters that are all from the set of "IFS whitespace characters" (that is, whichever of <space> , <tab> , and <newline> ("newline" meaning line feed (LF) ) are present anywhere in$IFS), or (2) any non-"IFS whitespace character" that's present in$IFSalong with whatever "IFS whitespace characters" surround it in the input line.For the OP, it's possible that the second separation mode I described in the previous paragraph is exactly what he wants for his input string, but we can be pretty confident that the first separation mode I described is not correct at all. For example, what if his input string was
'Los Angeles, United States, North America'?IFS=', ' read -ra a <<<'Los Angeles, United States, North America'; declare -p a; ## declare -a a=([0]="Los" [1]="Angeles" [2]="United" [3]="States" [4]="North" [5]="America")2: Even if you were to use this solution with a single-character separator (such as a comma by itself, that is, with no following space or other baggage), if the value of the
$stringvariable happens to contain any LFs, thenreadwill stop processing once it encounters the first LF. Thereadbuiltin only processes one line per invocation. This is true even if you are piping or redirecting input only to thereadstatement, as we are doing in this example with the here-string mechanism, and thus unprocessed input is guaranteed to be lost. The code that powers thereadbuiltin has no knowledge of the data flow within its containing command structure.You could argue that this is unlikely to cause a problem, but still, it's a subtle hazard that should be avoided if possible. It is caused by the fact that the
readbuiltin actually does two levels of input splitting: first into lines, then into fields. Since the OP only wants one level of splitting, this usage of thereadbuiltin is not appropriate, and we should avoid it.3: A non-obvious potential issue with this solution is that
readalways drops the trailing field if it is empty, although it preserves empty fields otherwise. Here's a demo:string=', , a, , b, c, , , '; IFS=', ' read -ra a <<<"$string"; declare -p a; ## declare -a a=([0]="" [1]="" [2]="a" [3]="" [4]="b" [5]="c" [6]="" [7]="")Maybe the OP wouldn't care about this, but it's still a limitation worth knowing about. It reduces the robustness and generality of the solution.
This problem can be solved by appending a dummy trailing delimiter to the input string just prior to feeding it to
read, as I will demonstrate later.
string="1:2:3:4:5" set -f # avoid globbing (expansion of *). array=(${string//:/ })t="one,two,three" a=($(echo $t | tr ',' "\n"))(Note: I added the missing parentheses around the command substitution which the answerer seems to have omitted.)
string="1,2,3,4" array=(`echo $string | sed 's/,/\n/g'`)These solutions leverage word splitting in an array assignment to split the string into fields. Funnily enough, just like
read, general word splitting also uses the$IFSspecial variable, although in this case it is implied that it is set to its default value of <space><tab><newline> , and therefore any sequence of one or more IFS characters (which are all whitespace characters now) is considered to be a field delimiter.This solves the problem of two levels of splitting committed by
read, since word splitting by itself constitutes only one level of splitting. But just as before, the problem here is that the individual fields in the input string can already contain$IFScharacters, and thus they would be improperly split during the word splitting operation. This happens to not be the case for any of the sample input strings provided by these answerers (how convenient...), but of course that doesn't change the fact that any code base that used this idiom would then run the risk of blowing up if this assumption were ever violated at some point down the line. Once again, consider my counterexample of'Los Angeles, United States, North America'(or'Los Angeles:United States:North America').Also, word splitting is normally followed by filename expansion ( aka pathname expansion aka globbing), which, if done, would potentially corrupt words containing the characters
*,?, or[followed by](and, ifextglobis set, parenthesized fragments preceded by?,*,+,@, or!) by matching them against file system objects and expanding the words ("globs") accordingly. The first of these three answerers has cleverly undercut this problem by runningset -fbeforehand to disable globbing. Technically this works (although you should probably addset +fafterward to reenable globbing for subsequent code which may depend on it), but it's undesirable to have to mess with global shell settings in order to hack a basic string-to-array parsing operation in local code.Another issue with this answer is that all empty fields will be lost. This may or may not be a problem, depending on the application.
Note: If you're going to use this solution, it's better to use the
${string//:/ }"pattern substitution" form of parameter expansion , rather than going to the trouble of invoking a command substitution (which forks the shell), starting up a pipeline, and running an external executable (trorsed), since parameter expansion is purely a shell-internal operation. (Also, for thetrandsedsolutions, the input variable should be double-quoted inside the command substitution; otherwise word splitting would take effect in theechocommand and potentially mess with the field values. Also, the$(...)form of command substitution is preferable to the old`...`form since it simplifies nesting of command substitutions and allows for better syntax highlighting by text editors.)
str="a, b, c, d" # assuming there is a space after ',' as in Q arr=(${str//,/}) # delete all occurrences of ','This answer is almost the same as #2 . The difference is that the answerer has made the assumption that the fields are delimited by two characters, one of which being represented in the default
$IFS, and the other not. He has solved this rather specific case by removing the non-IFS-represented character using a pattern substitution expansion and then using word splitting to split the fields on the surviving IFS-represented delimiter character.This is not a very generic solution. Furthermore, it can be argued that the comma is really the "primary" delimiter character here, and that stripping it and then depending on the space character for field splitting is simply wrong. Once again, consider my counterexample:
'Los Angeles, United States, North America'.Also, again, filename expansion could corrupt the expanded words, but this can be prevented by temporarily disabling globbing for the assignment with
set -fand thenset +f.Also, again, all empty fields will be lost, which may or may not be a problem depending on the application.
string='first line second line third line' oldIFS="$IFS" IFS=' ' IFS=${IFS:0:1} # this is useful to format your code with tabs lines=( $string ) IFS="$oldIFS"This is similar to #2 and #3 in that it uses word splitting to get the job done, only now the code explicitly sets
$IFSto contain only the single-character field delimiter present in the input string. It should be repeated that this cannot work for multicharacter field delimiters such as the OP's comma-space delimiter. But for a single-character delimiter like the LF used in this example, it actually comes close to being perfect. The fields cannot be unintentionally split in the middle as we saw with previous wrong answers, and there is only one level of splitting, as required.One problem is that filename expansion will corrupt affected words as described earlier, although once again this can be solved by wrapping the critical statement in
set -fandset +f.Another potential problem is that, since LF qualifies as an "IFS whitespace character" as defined earlier, all empty fields will be lost, just as in #2 and #3 . This would of course not be a problem if the delimiter happens to be a non-"IFS whitespace character", and depending on the application it may not matter anyway, but it does vitiate the generality of the solution.
So, to sum up, assuming you have a one-character delimiter, and it is either a non-"IFS whitespace character" or you don't care about empty fields, and you wrap the critical statement in
set -fandset +f, then this solution works, but otherwise not.(Also, for information's sake, assigning a LF to a variable in bash can be done more easily with the
$'...'syntax, e.g.IFS=$'\n';.)
countries='Paris, France, Europe' OIFS="$IFS" IFS=', ' array=($countries) IFS="$OIFS"IFS=', ' eval 'array=($string)'This solution is effectively a cross between #1 (in that it sets
$IFSto comma-space) and #2-4 (in that it uses word splitting to split the string into fields). Because of this, it suffers from most of the problems that afflict all of the above wrong answers, sort of like the worst of all worlds.Also, regarding the second variant, it may seem like the
evalcall is completely unnecessary, since its argument is a single-quoted string literal, and therefore is statically known. But there's actually a very non-obvious benefit to usingevalin this way. Normally, when you run a simple command which consists of a variable assignment only , meaning without an actual command word following it, the assignment takes effect in the shell environment:IFS=', '; ## changes $IFS in the shell environmentThis is true even if the simple command involves multiple variable assignments; again, as long as there's no command word, all variable assignments affect the shell environment:
IFS=', ' array=($countries); ## changes both $IFS and $array in the shell environmentBut, if the variable assignment is attached to a command name (I like to call this a "prefix assignment") then it does not affect the shell environment, and instead only affects the environment of the executed command, regardless whether it is a builtin or external:
IFS=', ' :; ## : is a builtin command, the $IFS assignment does not outlive it IFS=', ' env; ## env is an external command, the $IFS assignment does not outlive itRelevant quote from the bash manual :
If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.
It is possible to exploit this feature of variable assignment to change
$IFSonly temporarily, which allows us to avoid the whole save-and-restore gambit like that which is being done with the$OIFSvariable in the first variant. But the challenge we face here is that the command we need to run is itself a mere variable assignment, and hence it would not involve a command word to make the$IFSassignment temporary. You might think to yourself, well why not just add a no-op command word to the statement like the: builtinto make the$IFSassignment temporary? This does not work because it would then make the$arrayassignment temporary as well:IFS=', ' array=($countries) :; ## fails; new $array value never escapes the : commandSo, we're effectively at an impasse, a bit of a catch-22. But, when
evalruns its code, it runs it in the shell environment, as if it was normal, static source code, and therefore we can run the$arrayassignment inside theevalargument to have it take effect in the shell environment, while the$IFSprefix assignment that is prefixed to theevalcommand will not outlive theevalcommand. This is exactly the trick that is being used in the second variant of this solution:IFS=', ' eval 'array=($string)'; ## $IFS does not outlive the eval command, but $array doesSo, as you can see, it's actually quite a clever trick, and accomplishes exactly what is required (at least with respect to assignment effectation) in a rather non-obvious way. I'm actually not against this trick in general, despite the involvement of
eval; just be careful to single-quote the argument string to guard against security threats.But again, because of the "worst of all worlds" agglomeration of problems, this is still a wrong answer to the OP's requirement.
IFS=', '; array=(Paris, France, Europe) IFS=' ';declare -a array=(Paris France Europe)Um... what? The OP has a string variable that needs to be parsed into an array. This "answer" starts with the verbatim contents of the input string pasted into an array literal. I guess that's one way to do it.
It looks like the answerer may have assumed that the
$IFSvariable affects all bash parsing in all contexts, which is not true. From the bash manual:IFS The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline> .
So the
$IFSspecial variable is actually only used in two contexts: (1) word splitting that is performed after expansion (meaning not when parsing bash source code) and (2) for splitting input lines into words by thereadbuiltin.Let me try to make this clearer. I think it might be good to draw a distinction between parsing and execution . Bash must first parse the source code, which obviously is a parsing event, and then later it executes the code, which is when expansion comes into the picture. Expansion is really an execution event. Furthermore, I take issue with the description of the
$IFSvariable that I just quoted above; rather than saying that word splitting is performed after expansion , I would say that word splitting is performed during expansion, or, perhaps even more precisely, word splitting is part of the expansion process. The phrase "word splitting" refers only to this step of expansion; it should never be used to refer to the parsing of bash source code, although unfortunately the docs do seem to throw around the words "split" and "words" a lot. Here's a relevant excerpt from the linux.die.net version of the bash manual:Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion , tilde expansion , parameter and variable expansion , command substitution , arithmetic expansion , word splitting , and pathname expansion .
The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.
You could argue the GNU version of the manual does slightly better, since it opts for the word "tokens" instead of "words" in the first sentence of the Expansion section:
Expansion is performed on the command line after it has been split into tokens.
The important point is,
$IFSdoes not change the way bash parses source code. Parsing of bash source code is actually a very complex process that involves recognition of the various elements of shell grammar, such as command sequences, command lists, pipelines, parameter expansions, arithmetic substitutions, and command substitutions. For the most part, the bash parsing process cannot be altered by user-level actions like variable assignments (actually, there are some minor exceptions to this rule; for example, see the variouscompatxxshell settings , which can change certain aspects of parsing behavior on-the-fly). The upstream "words"/"tokens" that result from this complex parsing process are then expanded according to the general process of "expansion" as broken down in the above documentation excerpts, where word splitting of the expanded (expanding?) text into downstream words is simply one step of that process. Word splitting only touches text that has been spit out of a preceding expansion step; it does not affect literal text that was parsed right off the source bytestream.
string='first line second line third line' while read -r line; do lines+=("$line"); done <<<"$string"This is one of the best solutions. Notice that we're back to using
read. Didn't I say earlier thatreadis inappropriate because it performs two levels of splitting, when we only need one? The trick here is that you can callreadin such a way that it effectively only does one level of splitting, specifically by splitting off only one field per invocation, which necessitates the cost of having to call it repeatedly in a loop. It's a bit of a sleight of hand, but it works.But there are problems. First: When you provide at least one NAME argument to
read, it automatically ignores leading and trailing whitespace in each field that is split off from the input string. This occurs whether$IFSis set to its default value or not, as described earlier in this post. Now, the OP may not care about this for his specific use-case, and in fact, it may be a desirable feature of the parsing behavior. But not everyone who wants to parse a string into fields will want this. There is a solution, however: A somewhat non-obvious usage ofreadis to pass zero NAME arguments. In this case,readwill store the entire input line that it gets from the input stream in a variable named$REPLY, and, as a bonus, it does not strip leading and trailing whitespace from the value. This is a very robust usage ofreadwhich I've exploited frequently in my shell programming career. Here's a demonstration of the difference in behavior:string=$' a b \n c d \n e f '; ## input string a=(); while read -r line; do a+=("$line"); done <<<"$string"; declare -p a; ## declare -a a=([0]="a b" [1]="c d" [2]="e f") ## read trimmed surrounding whitespace a=(); while read -r; do a+=("$REPLY"); done <<<"$string"; declare -p a; ## declare -a a=([0]=" a b " [1]=" c d " [2]=" e f ") ## no trimmingThe second issue with this solution is that it does not actually address the case of a custom field separator, such as the OP's comma-space. As before, multicharacter separators are not supported, which is an unfortunate limitation of this solution. We could try to at least split on comma by specifying the separator to the
-doption, but look what happens:string='Paris, France, Europe'; a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string"; declare -p a; ## declare -a a=([0]="Paris" [1]=" France")Predictably, the unaccounted surrounding whitespace got pulled into the field values, and hence this would have to be corrected subsequently through trimming operations (this could also be done directly in the while-loop). But there's another obvious error: Europe is missing! What happened to it? The answer is that
readreturns a failing return code if it hits end-of-file (in this case we can call it end-of-string) without encountering a final field terminator on the final field. This causes the while-loop to break prematurely and we lose the final field.Technically this same error afflicted the previous examples as well; the difference there is that the field separator was taken to be LF, which is the default when you don't specify the
-doption, and the<<<("here-string") mechanism automatically appends a LF to the string just before it feeds it as input to the command. Hence, in those cases, we sort of accidentally solved the problem of a dropped final field by unwittingly appending an additional dummy terminator to the input. Let's call this solution the "dummy-terminator" solution. We can apply the dummy-terminator solution manually for any custom delimiter by concatenating it against the input string ourselves when instantiating it in the here-string:a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string,"; declare -p a; declare -a a=([0]="Paris" [1]=" France" [2]=" Europe")There, problem solved. Another solution is to only break the while-loop if both (1)
readreturned failure and (2)$REPLYis empty, meaningreadwas not able to read any characters prior to hitting end-of-file. Demo:a=(); while read -rd,|| [[ -n "$REPLY" ]]; do a+=("$REPLY"); done <<<"$string"; declare -p a; ## declare -a a=([0]="Paris" [1]=" France" [2]=$' Europe\n')This approach also reveals the secretive LF that automatically gets appended to the here-string by the
<<<redirection operator. It could of course be stripped off separately through an explicit trimming operation as described a moment ago, but obviously the manual dummy-terminator approach solves it directly, so we could just go with that. The manual dummy-terminator solution is actually quite convenient in that it solves both of these two problems (the dropped-final-field problem and the appended-LF problem) in one go.So, overall, this is quite a powerful solution. It's only remaining weakness is a lack of support for multicharacter delimiters, which I will address later.
string='first line second line third line' readarray -t lines <<<"$string"(This is actually from the same post as #7 ; the answerer provided two solutions in the same post.)
The
readarraybuiltin, which is a synonym formapfile, is ideal. It's a builtin command which parses a bytestream into an array variable in one shot; no messing with loops, conditionals, substitutions, or anything else. And it doesn't surreptitiously strip any whitespace from the input string. And (if-Ois not given) it conveniently clears the target array before assigning to it. But it's still not perfect, hence my criticism of it as a "wrong answer".First, just to get this out of the way, note that, just like the behavior of
readwhen doing field-parsing,readarraydrops the trailing field if it is empty. Again, this is probably not a concern for the OP, but it could be for some use-cases. I'll come back to this in a moment.Second, as before, it does not support multicharacter delimiters. I'll give a fix for this in a moment as well.
Third, the solution as written does not parse the OP's input string, and in fact, it cannot be used as-is to parse it. I'll expand on this momentarily as well.
For the above reasons, I still consider this to be a "wrong answer" to the OP's question. Below I'll give what I consider to be the right answer.
Right answer
Here's a naïve attempt to make #8 work by just specifying the
-doption:string='Paris, France, Europe'; readarray -td, a <<<"$string"; declare -p a; ## declare -a a=([0]="Paris" [1]=" France" [2]=$' Europe\n')We see the result is identical to the result we got from the double-conditional approach of the looping
readsolution discussed in #7 . We can almost solve this with the manual dummy-terminator trick:readarray -td, a <<<"$string,"; declare -p a; ## declare -a a=([0]="Paris" [1]=" France" [2]=" Europe" [3]=$'\n')The problem here is that
readarraypreserved the trailing field, since the<<<redirection operator appended the LF to the input string, and therefore the trailing field was not empty (otherwise it would've been dropped). We can take care of this by explicitly unsetting the final array element after-the-fact:readarray -td, a <<<"$string,"; unset 'a[-1]'; declare -p a; ## declare -a a=([0]="Paris" [1]=" France" [2]=" Europe")The only two problems that remain, which are actually related, are (1) the extraneous whitespace that needs to be trimmed, and (2) the lack of support for multicharacter delimiters.
The whitespace could of course be trimmed afterward (for example, see How to trim whitespace from a Bash variable? ). But if we can hack a multicharacter delimiter, then that would solve both problems in one shot.
Unfortunately, there's no direct way to get a multicharacter delimiter to work. The best solution I've thought of is to preprocess the input string to replace the multicharacter delimiter with a single-character delimiter that will be guaranteed not to collide with the contents of the input string. The only character that has this guarantee is the NUL byte . This is because, in bash (though not in zsh, incidentally), variables cannot contain the NUL byte. This preprocessing step can be done inline in a process substitution. Here's how to do it using awk :
readarray -td '' a < <(awk '{ gsub(/, /,"\0"); print; }' <<<"$string, "); unset 'a[-1]'; declare -p a; ## declare -a a=([0]="Paris" [1]="France" [2]="Europe")There, finally! This solution will not erroneously split fields in the middle, will not cut out prematurely, will not drop empty fields, will not corrupt itself on filename expansions, will not automatically strip leading and trailing whitespace, will not leave a stowaway LF on the end, does not require loops, and does not settle for a single-character delimiter.
Trimming solution
Lastly, I wanted to demonstrate my own fairly intricate trimming solution using the obscure
-C callbackoption ofreadarray. Unfortunately, I've run out of room against Stack Overflow's draconian 30,000 character post limit, so I won't be able to explain it. I'll leave that as an exercise for the reader.function mfcb { local val="$4"; "$1"; eval "$2[$3]=\$val;"; }; function val_ltrim { if [[ "$val" =~ ^[[:space:]]+ ]]; then val="${val:${#BASH_REMATCH[0]}}"; fi; }; function val_rtrim { if [[ "$val" =~ [[:space:]]+$ ]]; then val="${val:0:${#val}-${#BASH_REMATCH[0]}}"; fi; }; function val_trim { val_ltrim; val_rtrim; }; readarray -c1 -C 'mfcb val_trim a' -td, <<<"$string,"; unset 'a[-1]'; declare -p a; ## declare -a a=([0]="Paris" [1]="France" [2]="Europe")fbicknel ,Aug 18, 2017 at 15:57
It may also be helpful to note (though understandably you had no room to do so) that the-doption toreadarrayfirst appears in Bash 4.4. – fbicknel Aug 18 '17 at 15:57Cyril Duchon-Doris ,Nov 3, 2017 at 9:16
You should add a "TL;DR : scroll 3 pages to see the right solution at the end of my answer" – Cyril Duchon-Doris Nov 3 '17 at 9:16dawg ,Nov 26, 2017 at 22:28
Great answer (+1). If you change your awk toawk '{ gsub(/,[ ]+|$/,"\0"); print }'and eliminate that concatenation of the final", "then you don't have to go through the gymnastics on eliminating the final record. So:readarray -td '' a < <(awk '{ gsub(/,[ ]+/,"\0"); print; }' <<<"$string")on Bash that supportsreadarray. Note your method is Bash 4.4+ I think because of the-dinreadarray– dawg Nov 26 '17 at 22:28datUser ,Feb 22, 2018 at 14:54
Looks likereadarrayis not an available builtin on OSX. – datUser Feb 22 '18 at 14:54bgoldst ,Feb 23, 2018 at 3:37
@datUser That's unfortunate. Your version of bash must be too old forreadarray. In this case, you can use the second-best solution built onread. I'm referring to this:a=(); while read -rd,; do a+=("$REPLY"); done <<<"$string,";(with theawksubstitution if you need multicharacter delimiter support). Let me know if you run into any problems; I'm pretty sure this solution should work on fairly old versions of bash, back to version 2-something, released like two decades ago. – bgoldst Feb 23 '18 at 3:37Jmoney38 ,Jul 14, 2015 at 11:54
t="one,two,three" a=($(echo "$t" | tr ',' '\n')) echo "${a[2]}"Prints three
shrimpwagon ,Oct 16, 2015 at 20:04
I actually prefer this approach. Simple. – shrimpwagon Oct 16 '15 at 20:04Ben ,Oct 31, 2015 at 3:11
I copied and pasted this and it did did not work with echo, but did work when I used it in a for loop. – Ben Oct 31 '15 at 3:11Pinaki Mukherjee ,Nov 9, 2015 at 20:22
This is the simplest approach. thanks – Pinaki Mukherjee Nov 9 '15 at 20:22abalter ,Aug 30, 2016 at 5:13
This does not work as stated. @Jmoney38 or shrimpwagon if you can paste this in a terminal and get the desired output, please paste the result here. – abalter Aug 30 '16 at 5:13leaf ,Jul 17, 2017 at 16:28
@abalter Works for me witha=($(echo $t | tr ',' "\n")). Same result witha=($(echo $t | tr ',' ' ')). – leaf Jul 17 '17 at 16:28Luca Borrione ,Nov 2, 2012 at 13:44
Sometimes it happened to me that the method described in the accepted answer didn't work, especially if the separator is a carriage return.
In those cases I solved in this way:string='first line second line third line' oldIFS="$IFS" IFS=' ' IFS=${IFS:0:1} # this is useful to format your code with tabs lines=( $string ) IFS="$oldIFS" for line in "${lines[@]}" do echo "--> $line" doneStefan van den Akker ,Feb 9, 2015 at 16:52
+1 This completely worked for me. I needed to put multiple strings, divided by a newline, into an array, andread -a arr <<< "$strings"did not work withIFS=$'\n'. – Stefan van den Akker Feb 9 '15 at 16:52Stefan van den Akker ,Feb 10, 2015 at 13:49
Here is the answer to make the accepted answer work when the delimiter is a newline . – Stefan van den Akker Feb 10 '15 at 13:49,Jul 24, 2015 at 21:24
The accepted answer works for values in one line.
If the variable has several lines:string='first line second line third line'We need a very different command to get all lines:
while read -r line; do lines+=("$line"); done <<<"$string"Or the much simpler bash readarray :
readarray -t lines <<<"$string"Printing all lines is very easy taking advantage of a printf feature:
printf ">[%s]\n" "${lines[@]}" >[first line] >[ second line] >[ third line]Mayhem ,Dec 31, 2015 at 3:13
While not every solution works for every situation, your mention of readarray... replaced my last two hours with 5 minutes... you got my vote – Mayhem Dec 31 '15 at 3:13Derek 朕會功夫 ,Mar 23, 2018 at 19:14
readarrayis the right answer. – Derek 朕會功夫 Mar 23 '18 at 19:14ssanch ,Jun 3, 2016 at 15:24
This is similar to the approach by Jmoney38, but using sed:string="1,2,3,4" array=(`echo $string | sed 's/,/\n/g'`) echo ${array[0]}Prints 1
dawg ,Nov 26, 2017 at 19:59
The key to splitting your string into an array is the multi character delimiter of", ". Any solution usingIFSfor multi character delimiters is inherently wrong since IFS is a set of those characters, not a string.If you assign
IFS=", "then the string will break on EITHER","OR" "or any combination of them which is not an accurate representation of the two character delimiter of", ".You can use
awkorsedto split the string, with process substitution:#!/bin/bash str="Paris, France, Europe" array=() while read -r -d $'\0' each; do # use a NUL terminated field separator array+=("$each") done < <(printf "%s" "$str" | awk '{ gsub(/,[ ]+|$/,"\0"); print }') declare -p array # declare -a array=([0]="Paris" [1]="France" [2]="Europe") outputIt is more efficient to use a regex you directly in Bash:
#!/bin/bash str="Paris, France, Europe" array=() while [[ $str =~ ([^,]+)(,[ ]+|$) ]]; do array+=("${BASH_REMATCH[1]}") # capture the field i=${#BASH_REMATCH} # length of field + delimiter str=${str:i} # advance the string by that length done # the loop deletes $str, so make a copy if needed declare -p array # declare -a array=([0]="Paris" [1]="France" [2]="Europe") output...With the second form, there is no sub shell and it will be inherently faster.
Edit by bgoldst: Here are some benchmarks comparing my
readarraysolution to dawg's regex solution, and I also included thereadsolution for the heck of it (note: I slightly modified the regex solution for greater harmony with my solution) (also see my comments below the post):## competitors function c_readarray { readarray -td '' a < <(awk '{ gsub(/, /,"\0"); print; };' <<<"$1, "); unset 'a[-1]'; }; function c_read { a=(); local REPLY=''; while read -r -d ''; do a+=("$REPLY"); done < <(awk '{ gsub(/, /,"\0"); print; };' <<<"$1, "); }; function c_regex { a=(); local s="$1, "; while [[ $s =~ ([^,]+),\ ]]; do a+=("${BASH_REMATCH[1]}"); s=${s:${#BASH_REMATCH}}; done; }; ## helper functions function rep { local -i i=-1; for ((i = 0; i<$1; ++i)); do printf %s "$2"; done; }; ## end rep() function testAll { local funcs=(); local args=(); local func=''; local -i rc=-1; while [[ "$1" != ':' ]]; do func="$1"; if [[ ! "$func" =~ ^[_a-zA-Z][_a-zA-Z0-9]*$ ]]; then echo "bad function name: $func" >&2; return 2; fi; funcs+=("$func"); shift; done; shift; args=("$@"); for func in "${funcs[@]}"; do echo -n "$func "; { time $func "${args[@]}" >/dev/null 2>&1; } 2>&1| tr '\n' '/'; rc=${PIPESTATUS[0]}; if [[ $rc -ne 0 ]]; then echo "[$rc]"; else echo; fi; done| column -ts/; }; ## end testAll() function makeStringToSplit { local -i n=$1; ## number of fields if [[ $n -lt 0 ]]; then echo "bad field count: $n" >&2; return 2; fi; if [[ $n -eq 0 ]]; then echo; elif [[ $n -eq 1 ]]; then echo 'first field'; elif [[ "$n" -eq 2 ]]; then echo 'first field, last field'; else echo "first field, $(rep $[$1-2] 'mid field, ')last field"; fi; }; ## end makeStringToSplit() function testAll_splitIntoArray { local -i n=$1; ## number of fields in input string local s=''; echo "===== $n field$(if [[ $n -ne 1 ]]; then echo 's'; fi;) ====="; s="$(makeStringToSplit "$n")"; testAll c_readarray c_read c_regex : "$s"; }; ## end testAll_splitIntoArray() ## results testAll_splitIntoArray 1; ## ===== 1 field ===== ## c_readarray real 0m0.067s user 0m0.000s sys 0m0.000s ## c_read real 0m0.064s user 0m0.000s sys 0m0.000s ## c_regex real 0m0.000s user 0m0.000s sys 0m0.000s ## testAll_splitIntoArray 10; ## ===== 10 fields ===== ## c_readarray real 0m0.067s user 0m0.000s sys 0m0.000s ## c_read real 0m0.064s user 0m0.000s sys 0m0.000s ## c_regex real 0m0.001s user 0m0.000s sys 0m0.000s ## testAll_splitIntoArray 100; ## ===== 100 fields ===== ## c_readarray real 0m0.069s user 0m0.000s sys 0m0.062s ## c_read real 0m0.065s user 0m0.000s sys 0m0.046s ## c_regex real 0m0.005s user 0m0.000s sys 0m0.000s ## testAll_splitIntoArray 1000; ## ===== 1000 fields ===== ## c_readarray real 0m0.084s user 0m0.031s sys 0m0.077s ## c_read real 0m0.092s user 0m0.031s sys 0m0.046s ## c_regex real 0m0.125s user 0m0.125s sys 0m0.000s ## testAll_splitIntoArray 10000; ## ===== 10000 fields ===== ## c_readarray real 0m0.209s user 0m0.093s sys 0m0.108s ## c_read real 0m0.333s user 0m0.234s sys 0m0.109s ## c_regex real 0m9.095s user 0m9.078s sys 0m0.000s ## testAll_splitIntoArray 100000; ## ===== 100000 fields ===== ## c_readarray real 0m1.460s user 0m0.326s sys 0m1.124s ## c_read real 0m2.780s user 0m1.686s sys 0m1.092s ## c_regex real 17m38.208s user 15m16.359s sys 2m19.375s ##bgoldst ,Nov 27, 2017 at 4:28
Very cool solution! I never thought of using a loop on a regex match, nifty use of$BASH_REMATCH. It works, and does indeed avoid spawning subshells. +1 from me. However, by way of criticism, the regex itself is a little non-ideal, in that it appears you were forced to duplicate part of the delimiter token (specifically the comma) so as to work around the lack of support for non-greedy multipliers (also lookarounds) in ERE ("extended" regex flavor built into bash). This makes it a little less generic and robust. – bgoldst Nov 27 '17 at 4:28bgoldst ,Nov 27, 2017 at 4:28
Secondly, I did some benchmarking, and although the performance is better than the other solutions for smallish strings, it worsens exponentially due to the repeated string-rebuilding, becoming catastrophic for very large strings. See my edit to your answer. – bgoldst Nov 27 '17 at 4:28dawg ,Nov 27, 2017 at 4:46
@bgoldst: What a cool benchmark! In defense of the regex, for 10's or 100's of thousands of fields (what the regex is splitting) there would probably be some form of record (like\ndelimited text lines) comprising those fields so the catastrophic slow-down would likely not occur. If you have a string with 100,000 fields -- maybe Bash is not ideal ;-) Thanks for the benchmark. I learned a thing or two. – dawg Nov 27 '17 at 4:46Geoff Lee ,Mar 4, 2016 at 6:02
Try thisIFS=', '; array=(Paris, France, Europe) for item in ${array[@]}; do echo $item; doneIt's simple. If you want, you can also add a declare (and also remove the commas):
IFS=' ';declare -a array=(Paris France Europe)The IFS is added to undo the above but it works without it in a fresh bash instance
MrPotatoHead ,Nov 13, 2018 at 13:19
Pure bash multi-character delimiter solution.As others have pointed out in this thread, the OP's question gave an example of a comma delimited string to be parsed into an array, but did not indicate if he/she was only interested in comma delimiters, single character delimiters, or multi-character delimiters.
Since Google tends to rank this answer at or near the top of search results, I wanted to provide readers with a strong answer to the question of multiple character delimiters, since that is also mentioned in at least one response.
If you're in search of a solution to a multi-character delimiter problem, I suggest reviewing Mallikarjun M 's post, in particular the response from gniourf_gniourf who provides this elegant pure BASH solution using parameter expansion:
#!/bin/bash str="LearnABCtoABCSplitABCaABCString" delimiter=ABC s=$str$delimiter array=(); while [[ $s ]]; do array+=( "${s%%"$delimiter"*}" ); s=${s#*"$delimiter"}; done; declare -p arrayLink to cited comment/referenced post
Link to cited question: Howto split a string on a multi-character delimiter in bash?
Eduardo Cuomo ,Dec 19, 2016 at 15:27
Use this:countries='Paris, France, Europe' OIFS="$IFS" IFS=', ' array=($countries) IFS="$OIFS" #${array[1]} == Paris #${array[2]} == France #${array[3]} == Europegniourf_gniourf ,Dec 19, 2016 at 17:22
Bad: subject to word splitting and pathname expansion. Please don't revive old questions with good answers to give bad answers. – gniourf_gniourf Dec 19 '16 at 17:22Scott Weldon ,Dec 19, 2016 at 18:12
This may be a bad answer, but it is still a valid answer. Flaggers / reviewers: For incorrect answers such as this one, downvote, don't delete! – Scott Weldon Dec 19 '16 at 18:12George Sovetov ,Dec 26, 2016 at 17:31
@gniourf_gniourf Could you please explain why it is a bad answer? I really don't understand when it fails. – George Sovetov Dec 26 '16 at 17:31gniourf_gniourf ,Dec 26, 2016 at 18:07
@GeorgeSovetov: As I said, it's subject to word splitting and pathname expansion. More generally, splitting a string into an array asarray=( $string )is a (sadly very common) antipattern: word splitting occurs:string='Prague, Czech Republic, Europe'; Pathname expansion occurs:string='foo[abcd],bar[efgh]'will fail if you have a file named, e.g.,foodorbarfin your directory. The only valid usage of such a construct is whenstringis a glob. – gniourf_gniourf Dec 26 '16 at 18:07user1009908 ,Jun 9, 2015 at 23:28
UPDATE: Don't do this, due to problems with eval.With slightly less ceremony:
IFS=', ' eval 'array=($string)'e.g.
string="foo, bar,baz" IFS=', ' eval 'array=($string)' echo ${array[1]} # -> barcaesarsol ,Oct 29, 2015 at 14:42
eval is evil! don't do this. – caesarsol Oct 29 '15 at 14:42user1009908 ,Oct 30, 2015 at 4:05
Pfft. No. If you're writing scripts large enough for this to matter, you're doing it wrong. In application code, eval is evil. In shell scripting, it's common, necessary, and inconsequential. – user1009908 Oct 30 '15 at 4:05caesarsol ,Nov 2, 2015 at 18:19
put a$in your variable and you'll see... I write many scripts and I never ever had to use a singleeval– caesarsol Nov 2 '15 at 18:19Dennis Williamson ,Dec 2, 2015 at 17:00
Eval command and security issues – Dennis Williamson Dec 2 '15 at 17:00user1009908 ,Dec 22, 2015 at 23:04
You're right, this is only usable when the input is known to be clean. Not a robust solution. – user1009908 Dec 22 '15 at 23:04Eduardo Lucio ,Jan 31, 2018 at 20:45
Here's my hack!Splitting strings by strings is a pretty boring thing to do using bash. What happens is that we have limited approaches that only work in a few cases (split by ";", "/", "." and so on) or we have a variety of side effects in the outputs.
The approach below has required a number of maneuvers, but I believe it will work for most of our needs!
#!/bin/bash # -------------------------------------- # SPLIT FUNCTION # ---------------- F_SPLIT_R=() f_split() { : 'It does a "split" into a given string and returns an array. Args: TARGET_P (str): Target string to "split". DELIMITER_P (Optional[str]): Delimiter used to "split". If not informed the split will be done by spaces. Returns: F_SPLIT_R (array): Array with the provided string separated by the informed delimiter. ' F_SPLIT_R=() TARGET_P=$1 DELIMITER_P=$2 if [ -z "$DELIMITER_P" ] ; then DELIMITER_P=" " fi REMOVE_N=1 if [ "$DELIMITER_P" == "\n" ] ; then REMOVE_N=0 fi # NOTE: This was the only parameter that has been a problem so far! # By Questor # [Ref.: https://unix.stackexchange.com/a/390732/61742] if [ "$DELIMITER_P" == "./" ] ; then DELIMITER_P="[.]/" fi if [ ${REMOVE_N} -eq 1 ] ; then # NOTE: Due to bash limitations we have some problems getting the # output of a split by awk inside an array and so we need to use # "line break" (\n) to succeed. Seen this, we remove the line breaks # momentarily afterwards we reintegrate them. The problem is that if # there is a line break in the "string" informed, this line break will # be lost, that is, it is erroneously removed in the output! # By Questor TARGET_P=$(awk 'BEGIN {RS="dn"} {gsub("\n", "3F2C417D448C46918289218B7337FCAF"); printf $0}' <<< "${TARGET_P}") fi # NOTE: The replace of "\n" by "3F2C417D448C46918289218B7337FCAF" results # in more occurrences of "3F2C417D448C46918289218B7337FCAF" than the # amount of "\n" that there was originally in the string (one more # occurrence at the end of the string)! We can not explain the reason for # this side effect. The line below corrects this problem! By Questor TARGET_P=${TARGET_P%????????????????????????????????} SPLIT_NOW=$(awk -F"$DELIMITER_P" '{for(i=1; i<=NF; i++){printf "%s\n", $i}}' <<< "${TARGET_P}") while IFS= read -r LINE_NOW ; do if [ ${REMOVE_N} -eq 1 ] ; then # NOTE: We use "'" to prevent blank lines with no other characters # in the sequence being erroneously removed! We do not know the # reason for this side effect! By Questor LN_NOW_WITH_N=$(awk 'BEGIN {RS="dn"} {gsub("3F2C417D448C46918289218B7337FCAF", "\n"); printf $0}' <<< "'${LINE_NOW}'") # NOTE: We use the commands below to revert the intervention made # immediately above! By Questor LN_NOW_WITH_N=${LN_NOW_WITH_N%?} LN_NOW_WITH_N=${LN_NOW_WITH_N#?} F_SPLIT_R+=("$LN_NOW_WITH_N") else F_SPLIT_R+=("$LINE_NOW") fi done <<< "$SPLIT_NOW" } # -------------------------------------- # HOW TO USE # ---------------- STRING_TO_SPLIT=" * How do I list all databases and tables using psql? \" sudo -u postgres /usr/pgsql-9.4/bin/psql -c \"\l\" sudo -u postgres /usr/pgsql-9.4/bin/psql <DB_NAME> -c \"\dt\" \" \" \list or \l: list all databases \dt: list all tables in the current database \" [Ref.: https://dba.stackexchange.com/questions/1285/how-do-i-list-all-databases-and-tables-using-psql] " f_split "$STRING_TO_SPLIT" "bin/psql -c" # -------------------------------------- # OUTPUT AND TEST # ---------------- ARR_LENGTH=${#F_SPLIT_R[*]} for (( i=0; i<=$(( $ARR_LENGTH -1 )); i++ )) ; do echo " > -----------------------------------------" echo "${F_SPLIT_R[$i]}" echo " < -----------------------------------------" done if [ "$STRING_TO_SPLIT" == "${F_SPLIT_R[0]}bin/psql -c${F_SPLIT_R[1]}" ] ; then echo " > -----------------------------------------" echo "The strings are the same!" echo " < -----------------------------------------" fisel-en-ium ,May 31, 2018 at 5:56
Another way to do it without modifying IFS:read -r -a myarray <<< "${string//, /$IFS}"Rather than changing IFS to match our desired delimiter, we can replace all occurrences of our desired delimiter
", "with contents of$IFSvia"${string//, /$IFS}".Maybe this will be slow for very large strings though?
This is based on Dennis Williamson's answer.
rsjethani ,Sep 13, 2016 at 16:21
Another approach can be:str="a, b, c, d" # assuming there is a space after ',' as in Q arr=(${str//,/}) # delete all occurrences of ','After this 'arr' is an array with four strings. This doesn't require dealing IFS or read or any other special stuff hence much simpler and direct.
gniourf_gniourf ,Dec 26, 2016 at 18:12
Same (sadly common) antipattern as other answers: subject to word splitting and filename expansion. – gniourf_gniourf Dec 26 '16 at 18:12Safter Arslan ,Aug 9, 2017 at 3:21
Another way would be:string="Paris, France, Europe" IFS=', ' arr=(${string})Now your elements are stored in "arr" array. To iterate through the elements:
for i in ${arr[@]}; do echo $i; donebgoldst ,Aug 13, 2017 at 22:38
I cover this idea in my answer ; see Wrong answer #5 (you might be especially interested in my discussion of theevaltrick). Your solution leaves$IFSset to the comma-space value after-the-fact. – bgoldst Aug 13 '17 at 22:38
|
|
|
flowblok's blog
that diagram shows what happens according to the man page, and not what happens when you actually try it out in real life. This second diagram more accurately captures the insanity of bash:See how remote interactive login shells read /etc/bash.bashrc, but normal interactive login shells don't? Sigh.
Finally, here's a repository containing my implementation and the graphviz files for the above diagram. If your POSIX-compliant shell isn't listed here, or if I've made a horrible mistake (or just a tiny one), please send me a pull request or make a comment below, and I'll update this post accordingly.
[1]and since I'm writing this, I can make you say whatever I want for the purposes of narrative.
|
|
|
Jan 26, 2019 | flowblok.id.au
Adrian • a month ago ,
6 years late, but...
In my experience, if your bash sources /etc/bash.bashrc, odds are good it also sources /etc/bash.bash_logout or something similar on logout (after ~/.bash_logout, of course).
From bash-4.4/config-top.h:
/* System-wide .bashrc file for interactive shells. */ /* #define SYS_BASHRC "/etc/bash.bashrc" */ /* System-wide .bash_logout for login shells. */ /* #define SYS_BASH_LOGOUT "/etc/bash.bash_logout" */(Yes, they're disabled by default.)
Check the FILES section of your system's bash man page for details.
|
|
|
Jan 06, 2018 | zwischenzugs.com
IntroRecently I wanted to deepen my understanding of bash by researching as much of it as possible. Because I felt bash is an often-used (and under-understood) technology, I ended up writing a book on it .
A preview is available here .
You don't have to look hard on the internet to find plenty of useful one-liners in bash, or scripts. And there are guides to bash that seem somewhat intimidating through either their thoroughness or their focus on esoteric detail.
Here I've focussed on the things that either confused me or increased my power and productivity in bash significantly, and tried to communicate them (as in my book) in a way that emphasises getting the understanding right.
Enjoy!
1)
``vs$()These two operators do the same thing. Compare these two lines:
$ echo `ls` $ echo $(ls)Why these two forms existed confused me for a long time.
If you don't know, both forms substitute the output of the command contained within it into the command.
The principal difference is that nesting is simpler.
Which of these is easier to read (and write)?
$ echo `echo \`echo \\\`echo inside\\\`\``or:
$ echo $(echo $(echo $(echo inside)))If you're interested in going deeper, see here or here .
2) globbing vs regexpsAnother one that can confuse if never thought about or researched.
While globs and regexps can look similar, they are not the same.
Consider this command:
$ rename -n 's/(.*)/new$1/' *The two asterisks are interpreted in different ways.
The first is ignored by the shell (because it is in quotes), and is interpreted as '0 or more characters' by the rename application. So it's interpreted as a regular expression.
The second is interpreted by the shell (because it is not in quotes), and gets replaced by a list of all the files in the current working folder. It is interpreted as a glob.
So by looking at
man bashcan you figure out why these two commands produce different output?$ ls * $ ls .*The second looks even more like a regular expression. But it isn't!
3) Exit CodesNot everyone knows that every time you run a shell command in bash, an 'exit code' is returned to bash.
Generally, if a command 'succeeds' you get an error code of
0. If it doesn't succeed, you get a non-zero code.1is a 'general error', and others can give you more information (eg which signal killed it, for example).But these rules don't always hold:
$ grep not_there /dev/null $ echo $?
$?is a special bash variable that's set to the exit code of each command after it runs.Grep uses exit codes to indicate whether it matched or not. I have to look up every time which way round it goes: does finding a match or not return
0?Grok this and a lot will click into place in what follows.
4)ifstatements,[and[[Here's another 'spot the difference' similar to the backticks one above.
What will this output?
if grep not_there /dev/null then echo hi else echo lo figrep's return code makes code like this work more intuitively as a side effect of its use of exit codes.
Now what will this output?
a)
hihi
b)lolo
c) something elseif [ $(grep not_there /dev/null) = '' ] then echo -n hi else echo -n lo fi if [[ $(grep not_there /dev/null) = '' ]] then echo -n hi else echo -n lo fiThe difference between
[and[[was another thing I never really understood.[is the original form for tests, and then[[was introduced, which is more flexible and intuitive. In the firstifblock above, the if statement barfs because the$(grep not_there /dev/null)is evaluated to nothing, resulting in this comparison:
[ = '' ]which makes no sense. The double bracket form handles this for you.
This is why you occasionally see comparisons like this in bash scripts:
if [ x$(grep not_there /dev/null) = 'x' ]so that if the command returns nothing it still runs. There's no need for it, but that's why it exists.
5)setsBash has configurable options which can be set on the fly. I use two of these all the time:
set -eexits from a script if any command returned a non-zero exit code (see above).
This outputs the commands that get run as they run:
set -xSo a script might start like this:
#!/bin/bash set -e set -x grep not_there /dev/null echo $?What would that script output?
6) <()This is my favourite. It's so under-used, perhaps because it can be initially baffling, but I use it all the time.
It's similar to
$()in that the output of the command inside is re-used.In this case, though, the output is treated as a file. This file can be used as an argument to commands that take files as an argument.
Confused? Here's an example.
Have you ever done something like this?
$ grep somestring file1 > /tmp/a $ grep somestring file2 > /tmp/b $ diff /tmp/a /tmp/bThat works, but instead you can write:
diff <(grep somestring file1) <(grep somestring file2)Isn't that neater?
7) QuotingQuoting's a knotty subject in bash, as it is in many software contexts.
Firstly, variables in quotes:
A='123' echo "$A" echo '$A'Pretty simple – double quotes dereference variables, while single quotes go literal.
So what will this output?
mkdir -p tmp cd tmp touch a echo "*" echo '*'Surprised? I was.
8) Top three shortcutsThere are plenty of shortcuts listed in
man bash, and it's not hard to find comprehensive lists. This list consists of the ones I use most often, in order of how often I use them.Rather than trying to memorize them all, I recommend picking one, and trying to remember to use it until it becomes unconscious. Then take the next one. I'll skip over the most obvious ones (eg
!!– repeat last command, and~– your home directory).
!$I use this dozens of times a day. It repeats the last argument of the last command. If you're working on a file, and can't be bothered to re-type it command after command it can save a lot of work:
grep somestring /long/path/to/some/file/or/other.txt vi !$
!:1-$This bit of magic takes this further. It takes all the arguments to the previous command and drops them in. So:
grep isthere /long/path/to/some/file/or/other.txt egrep !:1-$ fgrep !:1-$The
!means 'look at the previous command', the:is a separator, and the1means 'take the first word', the-means 'until' and the$means 'the last word'.Note: you can achieve the same thing with
!*. Knowing the above gives you the control to limit to a specific contiguous subset of arguments, eg with!:2-3.
:hI use this one a lot too. If you put it after a filename, it will change that filename to remove everything up to the folder. Like this:
grep isthere /long/path/to/some/file/or/other.txt cd !$:hwhich can save a lot of work in the course of the day.
9) startup orderThe order in which bash runs startup scripts can cause a lot of head-scratching. I keep this diagram handy (from this great page):
It shows which scripts bash decides to run from the top, based on decisions made about the context bash is running in (which decides the colour to follow).
So if you are in a local (non-remote), non-login, interactive shell (eg when you run bash itself from the command line), you are on the 'green' line, and these are the order of files read:
/etc/bash.bashrc ~/.bashrc [bash runs, then terminates] ~/.bash_logoutThis can save you a hell of a lot of time debugging.
10) getopts (cheapci)If you go deep with bash, you might end up writing chunky utilities in it. If you do, then getting to grips with
getoptscan pay large dividends.For fun, I once wrote a script called
cheapciwhich I used to work like a Jenkins job.The code here implements the reading of the two required, and 14 non-required arguments . Better to learn this than to build up a bunch of bespoke code that can get very messy pretty quickly as your utility grows.
This is based on some of the contents of my book Learn Bash the Hard Way , available at $7 :
|
|
|
Dec 20, 2018 | forums.debian.net
pawRoot " 2018-10-15 17:13
Just spent some time editing .bashrc to make my life easier, and wondering if anyone has some cool "tricks" for bash as well.Here is mine:
- Code: Select all
# changing shell appearance
PS1='\[\033[0;32m\]\[\033[0m\033[0;32m\]\u\[\033[0;36m\] @ \[\033[0;36m\]\h \w\[\033[0;32m\]$(__git_ps1)\n\[\033[0;32m\]└─\[\033[0m\033[0;32m\] \$\[\033[0m\033[0;32m\] ▶\[\033[0m\] '
# aliases
alias la="ls -la --group-directories-first --color"
# clear terminal
alias cls="clear"
#
alias sup="sudo apt update && sudo apt upgrade"
# search for package
alias apts='apt-cache search'
# start x session
alias x="startx"
# download mp3 in best quality from YouTube
# usage: ytmp3 https://www.youtube.com/watch?v=LINK
alias ytmp3="youtube-dl -f bestaudio --extract-audio --audio-format mp3 --audio-quality 0"
# perform 'la' after 'cd'
alias cd="listDir"
listDir() {
builtin cd "$*"
RESULT=$?
if [ "$RESULT" -eq 0 ]; then
la
fi
}
# type "extract filename" to extract the file
extract () {
if [ -f $1 ] ; then
case $1 in
*.tar.bz2) tar xvjf $1 ;;
*.tar.gz) tar xvzf $1 ;;
*.bz2) bunzip2 $1 ;;
*.rar) unrar x $1 ;;
*.gz) gunzip $1 ;;
*.tar) tar xvf $1 ;;
*.tbz2) tar xvjf $1 ;;
*.tgz) tar xvzf $1 ;;
*.zip) unzip $1 ;;
*.Z) uncompress $1 ;;
*.7z) 7z x $1 ;;
*) echo "don't know how to extract '$1'..." ;;
esac
else
echo "'$1' is not a valid file!"
fi
}
# obvious one
alias ..="cd .."
alias ...="cd ../.."
alias ....="cd ../../.."
alias .....="cd ../../../.."
# tail all logs in /var/log
alias logs="find /var/log -type f -exec file {} \; | grep 'text' | cut -d' ' -f1 | sed -e's/:$//g' | grep -v '[0-9]$' | xargs tail -f"
Head_on_a_Stick " 2018-10-15 18:11
pawRoot wrote:Erm, did you know that `tar` autoextracts these days? This will work for pretty much anything:
- Code: Select all
extract () {
if [ -f $1 ] ; then
case $1 in
*.tar.bz2) tar xvjf $1 ;;
*.tar.gz) tar xvzf $1 ;;
*.bz2) bunzip2 $1 ;;
*.rar) unrar x $1 ;;
*.gz) gunzip $1 ;;
*.tar) tar xvf $1 ;;
*.tbz2) tar xvjf $1 ;;
*.tgz) tar xvzf $1 ;;
*.zip) unzip $1 ;;
*.Z) uncompress $1 ;;
*.7z) 7z x $1 ;;
*) echo "don't know how to extract '$1'..." ;;
esac
else
echo "'$1' is not a valid file!"
fi
}
I have these functions in my .mkshrc (bash is bloat!):
- Code: Select all
tar xf whatever.tar.whateverThe mnt function acts like a poor person's arch-chroot and will bind mount /proc /sys & /dev before chrooting then tear it down afterwards.
- Code: Select all
function mnt {
for i in proc sys dev dev/pts; do sudo mount --bind /$i "$1"$i; done &
sudo chroot "$1" /bin/bash
sudo umount -R "$1"{proc,sys,dev}
}
function mkiso {
xorriso -as mkisofs \
-iso-level 3 \
-full-iso9660-filenames \
-volid SharpBang-stretch \
-eltorito-boot isolinux/isolinux.bin \
-eltorito-catalog isolinux/boot.cat \
-no-emul-boot -boot-load-size 4 -boot-info-table \
-isohybrid-mbr isolinux/isohdpfx.bin \
-eltorito-alt-boot \
-e boot/grub/efi.img \
-no-emul-boot -isohybrid-gpt-basdat \
-output ../"$1" ./
}The mkiso function builds a UEFI-capable Debian live system (with the name of the image given as the first argument).
The only other stuff I have are aliases, not really worth posting.
dbruce wrote: Ubuntu forums try to be like a coffee shop in Seattle. Debian forums strive for the charm and ambience of a skinhead bar in Bacau. We intend to keep it that way.pawRoot " 2018-10-15 18:23
Head_on_a_Stick wrote: Erm, did you know that `tar` autoextracts these days? This will work for pretty much anything:But it won't work for zip or rar right ?
None1975 " 2018-10-16 13:02
Here is compilation of cool "tricks" for bash. This is similar to oh-my-zsh. OS: Debian Stretch / WM : Fluxbox
Debian Wiki | DontBreakDebian , My config files in github
debiman " 2018-10-21 14:38
i have a LOT of stuff in my /etc/bash.bashrc, because i want it to be available for the root user too.
i won't post everything, but here's a "best of" from both /etc/bash.bashrc and ~/.bashrc:
- Code: Select all
case ${TERM} in
xterm*|rxvt*|Eterm|aterm|kterm|gnome*)
PROMPT_COMMAND=${PROMPT_COMMAND:+$PROMPT_COMMAND; }'printf "\033]0;%s: %s\007" "${SHELL##*/}" "${PWD/#$HOME/\~}"'
;;
screen)
PROMPT_COMMAND=${PROMPT_COMMAND:+$PROMPT_COMMAND; }'printf "\033_%s@%s:%s\033\\" "${USER}" "${HOSTNAME%%.*}" "${PWD/#$HOME/\~}"'
;;
linux)
setterm --blength 0
setterm --blank 4
setterm --powerdown 8
;;
esac
PS2='cont> '
PS3='Choice: '
PS4='DEBUG: '
# Bash won't get SIGWINCH if another process is in the foreground.
# Enable checkwinsize so that bash will check the terminal size when
# it regains control.
# http://cnswww.cns.cwru.edu/~chet/bash/FAQ (E11)
shopt -s checkwinsize
# forums.bunsenlabs.org/viewtopic.php?pid=27494#p27494
# also see aliases '...' and '....'
shopt -s autocd
# opensource.com/article/18/5/bash-tricks
shopt -s cdspell
# as big as possible!!!
HISTSIZE=500000
HISTFILESIZE=2000000
# unix.stackexchange.com/a/18443
# history: erase duplicates...
HISTCONTROL=ignoredups:erasedups
shopt -s histappend
# next: enables usage of CTRL-S (backward search) with CTRL-R (forward search)
# digitalocean.com/community/tutorials/how-to-use-bash-history-commands-and-expansions-on-a-linux-vps#searching-through-bash-history
stty -ixon
if [[ ${EUID} == 0 ]] ; then
# root = color=1 # red
if [ "$TERM" != "linux" ]; then
PS1="\[$(tput setaf 1)\]\[$(tput rev)\] \[$(tput sgr0)\]\[$(tput setaf 5)\]\${?#0}\[$(tput setaf 1)\] \u@\h \w\[$(tput sgr0)\]\n\[$(tput rev)\] \[$(tput sgr0)\] "
else
# adding \t = time to tty prompt
PS1="\[$(tput setaf 1)\]\[$(tput rev)\] \[$(tput sgr0)\]\[$(tput setaf 5)\]\${?#0}\[$(tput setaf 1)\] \t \u@\h \w\[$(tput sgr0)\]\n\[$(tput rev)\] \[$(tput sgr0)\] "
fi
else
if [ "$TERM" != "linux" ]; then
PS1="\[$(tput setaf 2)\]\[$(tput rev)\] \[$(tput sgr0)\]\[$(tput setaf 5)\]\${?#0}\[$(tput setaf 2)\] \u@\h \w\[$(tput sgr0)\]\n\[$(tput rev)\] \[$(tput sgr0)\] "
else
# adding \t = time to tty prompt
PS1="\[$(tput setaf 2)\]\[$(tput rev)\] \[$(tput sgr0)\]\[$(tput setaf 5)\]\${?#0}\[$(tput setaf 2)\] \t \u@\h \w\[$(tput sgr0)\]\n\[$(tput rev)\] \[$(tput sgr0)\] "
fi
fi
[ -r /usr/share/bash-completion/bash_completion ] && . /usr/share/bash-completion/bash_completion || true
export EDITOR="nano"
man() {
env LESS_TERMCAP_mb=$(printf "\e[1;31m") \
LESS_TERMCAP_md=$(printf "\e[1;31m") \
LESS_TERMCAP_me=$(printf "\e[0m") \
LESS_TERMCAP_se=$(printf "\e[0m") \
LESS_TERMCAP_so=$(printf "\e[7m") \
LESS_TERMCAP_ue=$(printf "\e[0m") \
LESS_TERMCAP_us=$(printf "\e[1;32m") \
man "$@"
}
#LESS_TERMCAP_so=$(printf "\e[1;44;33m")
# that used to be in the man function for less's annoyingly over-colorful status line.
# changed it to simple reverse video (tput rev)
alias ls='ls --group-directories-first -hF --color=auto'
alias ll='ls --group-directories-first -hF --color=auto -la'
alias mpf='/usr/bin/ls -1 | mpv --playlist=-'
alias ruler='slop -o -c 1,0.3,0'
alias xmeasure='slop -o -c 1,0.3,0'
alias obxprop='obxprop | grep -v _NET_WM_ICON'
alias sx='exec startx > ~/.local/share/xorg/xlog 2>&1'
alias pngq='pngquant --nofs --speed 1 --skip-if-larger --strip '
alias screencap='ffmpeg -r 15 -s 1680x1050 -f x11grab -i :0.0 -vcodec msmpeg4v2 -qscale 2'
alias su='su -'
alias fblc='fluxbox -list-commands | column'
alias torrench='torrench -t -k -s -x -r -l -i -b --sorted'
alias F5='while sleep 60; do notify-send -u low "Pressed F5 on:" "$(xdotool getwindowname $(xdotool getwindowfocus))"; xdotool key F5; done'
alias aurs='aurman --sort_by_name -Ss'
alias cal3='cal -3 -m -w --color'
alias mkdir='mkdir -p -v'
alias ping='ping -c 5'
alias cd..='cd ..'
alias off='systemctl poweroff'
alias xg='xgamma -gamma'
alias find='find 2>/dev/null'
alias stressme='stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --timeout'
alias hf='history|grep'
alias du1='du -m --max-depth=1|sort -g|sed "s/\t./M\t/g ; s/\///g"'
alias zipcat='gunzip -c'
mkcd() {
mkdir -p "$1"
echo cd "$1"
cd "$1"
}
|
|
|
Nov 17, 2018 | www.mankier.com
hh -- easily view, navigate, sort and use your command history with shell history suggest box.
SynopsisDescription
hh [option] [arg1] [arg2]...
hstr [option] [arg1] [arg2]...hh uses shell history to provide suggest box like functionality for commands used in the past. By default it parses .bash-history file that is filtered as you type a command substring. Commands are not just filtered, but also ordered by a ranking algorithm that considers number of occurrences, length and timestamp. Favorite and frequently used commands can be bookmarked . In addition hh allows removal of commands from history - for instance with a typo or with a sensitive content.
OptionsKeys
- -h --help
- Show help
- -n --non-interactive
- Print filtered history on standard output and exit
- -f --favorites
- Show favorites view immediately
- -s --show-configuration
- Show configuration that can be added to ~/.bashrc
- -b --show-blacklist
- Show blacklist of commands to be filtered out before history processing
- -V --version
- Show version information
Environment Variables
- pattern
- Type to filter shell history.
- Ctrl-e
- Toggle regular expression and substring search.
- Ctrl-t
- Toggle case sensitive search.
- Ctrl-/ , Ctrl-7
- Rotate view of history as provided by Bash, ranked history ordered by the number of occurences/length/timestamp and favorites.
- Ctrl-f
- Add currently selected command to favorites.
- Ctrl-l
- Make search pattern lowercase or uppercase.
- Ctrl-r , UP arrow, DOWN arrow, Ctrl-n , Ctrl-p
- Navigate in the history list.
- TAB , RIGHT arrow
- Choose currently selected item for completion and let user to edit it on the command prompt.
- LEFT arrow
- Choose currently selected item for completion and let user to edit it in editor (fix command).
- ENTER
- Choose currently selected item for completion and execute it.
- DEL
- Remove currently selected item from the shell history.
- BACSKSPACE , Ctrl-h
- Delete last pattern character.
- Ctrl-u , Ctrl-w
- Delete pattern and search again.
- Ctrl-x
- Write changes to shell history and exit.
- Ctrl-g
- Exit with empty prompt.
hh defines the following environment variables:
Files
- HH_CONFIG
- Configuration options:
hicolor
Get more colors with this option (default is monochromatic).monochromatic
Ensure black and white view.prompt-bottom
Show prompt at the bottom of the screen (default is prompt at the top).regexp
Filter command history using regular expressions (substring match is default)substring
Filter command history using substring.keywords
Filter command history using keywords - item matches if contains all keywords in pattern in any order.casesensitive
Make history filtering case sensitive (it's case insensitive by default).rawhistory
Show normal history as a default view (metric-based view is shown otherwise).favorites
Show favorites as a default view (metric-based view is shown otherwise).duplicates
Show duplicates in rawhistory (duplicates are discarded by default).blacklist
Load list of commands to skip when processing history from ~/.hh_blacklist (built-in blacklist used otherwise).big-keys-skip
Skip big history entries i.e. very long lines (default).big-keys-floor
Use different sorting slot for big keys when building metrics-based view (big keys are skipped by default).big-keys-exit
Exit (fail) on presence of a big key in history (big keys are skipped by default).warning
Show warning.debug
Show debug information.Example:
export HH_CONFIG=hicolor,regexp,rawhistory- HH_PROMPT
- Change prompt string which is user@host$ by default.
Example:
export HH_PROMPT="$ "Bash Configuration
- ~/.hh_favorites
- Bookmarked favorite commands.
- ~/.hh_blacklist
- Command blacklist.
Optionally add the following lines to ~/.bashrc:
export HH_CONFIG=hicolor # get more colors shopt -s histappend # append new history items to .bash_history export HISTCONTROL=ignorespace # leading space hides commands from history export HISTFILESIZE=10000 # increase history file size (default is 500) export HISTSIZE=${HISTFILESIZE} # increase history size (default is 500) export PROMPT_COMMAND="history -a; history -n; ${PROMPT_COMMAND}" # if this is interactive shell, then bind hh to Ctrl-r (for Vi mode check doc) if [[ $- =~ .*i.* ]]; then bind '"\C-r": "\C-a hh -- \C-j"'; fiThe prompt command ensures synchronization of the history between BASH memory and history file.
ZSH ConfigurationOptionally add the following lines to ~/.zshrc:
export HISTFILE=~/.zsh_history # ensure history file visibility export HH_CONFIG=hicolor # get more colors bindkey -s "\C-r" "\eqhh\n" # bind hh to Ctrl-r (for Vi mode check doc, experiment with --)ExamplesAuthor
- hh git
- Start `hh` and show only history items containing 'git'.
- hh --non-interactive git
- Print history items containing 'git' to standard output and exit.
- hh --show-configuration >> ~/.bashrc
- Append default hh configuration to your Bash profile.
- hh --show-blacklist
- Show blacklist configured for history processing.
Written by Martin Dvorak <martin.dvorak@mindforger.com>
BugsReport bugs to https://github.com/dvorka/hstr/issues
See Also
Referenced By
The man page hstr(1) is an alias of hh(1).
|
|
|
Nov 08, 2018 | stackoverflow.com
This question already has an answer here:Rob I , May 9, 2012 at 19:22
For your second question, see @mkb's comment to my answer below - that's definitely the way to go! – Rob I May 9 '12 at 19:22Dennis Williamson , Jul 4, 2012 at 16:14
See my edited answer for one way to read individual characters into an array. – Dennis Williamson Jul 4 '12 at 16:14Nick Weedon , Dec 31, 2015 at 11:04
Here is the same thing in a more concise form: var1=$(cut -f1 -d- <<<$STR) – Nick Weedon Dec 31 '15 at 11:04Rob I , May 9, 2012 at 17:00
If your solution doesn't have to be general, i.e. only needs to work for strings like your example, you could do:var1=$(echo $STR | cut -f1 -d-) var2=$(echo $STR | cut -f2 -d-)I chose
cuthere because you could simply extend the code for a few more variables...crunchybutternut , May 9, 2012 at 17:40
Can you look at my post again and see if you have a solution for the followup question? thanks! – crunchybutternut May 9 '12 at 17:40mkb , May 9, 2012 at 17:59
You can usecutto cut characters too!cut -c1for example. – mkb May 9 '12 at 17:59FSp , Nov 27, 2012 at 10:26
Although this is very simple to read and write, is a very slow solution because forces you to read twice the same data ($STR) ... if you care of your script performace, the @anubhava solution is much better – FSp Nov 27 '12 at 10:26tripleee , Jan 25, 2016 at 6:47
Apart from being an ugly last-resort solution, this has a bug: You should absolutely use double quotes inecho "$STR"unless you specifically want the shell to expand any wildcards in the string as a side effect. See also stackoverflow.com/questions/10067266/ – tripleee Jan 25 '16 at 6:47Rob I , Feb 10, 2016 at 13:57
You're right about double quotes of course, though I did point out this solution wasn't general. However I think your assessment is a bit unfair - for some people this solution may be more readable (and hence extensible etc) than some others, and doesn't completely rely on arcane bash feature that wouldn't translate to other shells. I suspect that's why my solution, though less elegant, continues to get votes periodically... – Rob I Feb 10 '16 at 13:57Dennis Williamson , May 10, 2012 at 3:14
readwithIFSare perfect for this:$ IFS=- read var1 var2 <<< ABCDE-123456 $ echo "$var1" ABCDE $ echo "$var2" 123456Edit:
Here is how you can read each individual character into array elements:
$ read -a foo <<<"$(echo "ABCDE-123456" | sed 's/./& /g')"Dump the array:
$ declare -p foo declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]="-" [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'If there are spaces in the string:
$ IFS=$'\v' read -a foo <<<"$(echo "ABCDE 123456" | sed 's/./&\v/g')" $ declare -p foo declare -a foo='([0]="A" [1]="B" [2]="C" [3]="D" [4]="E" [5]=" " [6]="1" [7]="2" [8]="3" [9]="4" [10]="5" [11]="6")'insecure , Apr 30, 2014 at 7:51
Great, the elegant bash-only way, without unnecessary forks. – insecure Apr 30 '14 at 7:51Martin Serrano , Jan 11 at 4:34
this solution also has the benefit that if delimiter is not present, thevar2will be empty – Martin Serrano Jan 11 at 4:34mkb , May 9, 2012 at 17:02
If you know it's going to be just two fields, you can skip the extra subprocesses like this:var1=${STR%-*} var2=${STR#*-}What does this do?
${STR%-*}deletes the shortest substring of$STRthat matches the pattern-*starting from the end of the string.${STR#*-}does the same, but with the*-pattern and starting from the beginning of the string. They each have counterparts%%and##which find the longest anchored pattern match. If anyone has a helpful mnemonic to remember which does which, let me know! I always have to try both to remember.Jens , Jan 30, 2015 at 15:17
Plus 1 For knowing your POSIX shell features, avoiding expensive forks and pipes, and the absence of bashisms. – Jens Jan 30 '15 at 15:17Steven Lu , May 1, 2015 at 20:19
Dunno about "absence of bashisms" considering that this is already moderately cryptic .... if your delimiter is a newline instead of a hyphen, then it becomes even more cryptic. On the other hand, it works with newlines , so there's that. – Steven Lu May 1 '15 at 20:19mkb , Mar 9, 2016 at 17:30
@KErlandsson: done – mkb Mar 9 '16 at 17:30mombip , Aug 9, 2016 at 15:58
I've finally found documentation for it: Shell-Parameter-Expansion – mombip Aug 9 '16 at 15:58DS. , Jan 13, 2017 at 19:56
Mnemonic: "#" is to the left of "%" on a standard keyboard, so "#" removes a prefix (on the left), and "%" removes a suffix (on the right). – DS. Jan 13 '17 at 19:56tripleee , May 9, 2012 at 17:57
Sounds like a job forsetwith a customIFS.IFS=- set $STR var1=$1 var2=$2(You will want to do this in a function with a
local IFSso you don't mess up other parts of your script where you requireIFSto be what you expect.)Rob I , May 9, 2012 at 19:20
Nice - I knew about$IFSbut hadn't seen how it could be used. – Rob I May 9 '12 at 19:20Sigg3.net , Jun 19, 2013 at 8:08
I used triplee's example and it worked exactly as advertised! Just change last two lines to <pre> myvar1=echo $1&& myvar2=echo $2</pre> if you need to store them throughout a script with several "thrown" variables. – Sigg3.net Jun 19 '13 at 8:08tripleee , Jun 19, 2013 at 13:25
No, don't use a uselessechoin backticks . – tripleee Jun 19 '13 at 13:25Daniel Andersson , Mar 27, 2015 at 6:46
This is a really sweet solution if we need to write something that is not Bash specific. To handleIFStroubles, one can addOLDIFS=$IFSat the beginning before overwriting it, and then addIFS=$OLDIFSjust after thesetline. – Daniel Andersson Mar 27 '15 at 6:46tripleee , Mar 27, 2015 at 6:58
FWIW the link above is broken. I was lazy and careless. The canonical location still works; iki.fi/era/unix/award.html#echo – tripleee Mar 27 '15 at 6:58anubhava , May 9, 2012 at 17:09
Using bash regex capabilities:re="^([^-]+)-(.*)$" [[ "ABCDE-123456" =~ $re ]] && var1="${BASH_REMATCH[1]}" && var2="${BASH_REMATCH[2]}" echo $var1 echo $var2OUTPUT
ABCDE 123456Cometsong , Oct 21, 2016 at 13:29
Love pre-defining therefor later use(s)! – Cometsong Oct 21 '16 at 13:29Archibald , Nov 12, 2012 at 11:03
string="ABCDE-123456" IFS=- # use "local IFS=-" inside the function set $string echo $1 # >>> ABCDE echo $2 # >>> 123456tripleee , Mar 27, 2015 at 7:02
Hmmm, isn't this just a restatement of my answer ? – tripleee Mar 27 '15 at 7:02Archibald , Sep 18, 2015 at 12:36
Actually yes. I just clarified it a bit. – Archibald Sep 18 '15 at 12:36
|
|
|
Nov 08, 2018 | stackoverflow.com
cd1 , Jul 1, 2010 at 23:29
Suppose I have the string1:2:3:4:5and I want to get its last field (5in this case). How do I do that using Bash? I triedcut, but I don't know how to specify the last field with-f.Stephen , Jul 2, 2010 at 0:05
You can use string operators :$ foo=1:2:3:4:5 $ echo ${foo##*:} 5This trims everything from the front until a ':', greedily.
${foo <-- from variable foo ## <-- greedy front trim * <-- matches anything : <-- until the last ':' }eckes , Jan 23, 2013 at 15:23
While this is working for the given problem, the answer of William below ( stackoverflow.com/a/3163857/520162 ) also returns5if the string is1:2:3:4:5:(while using the string operators yields an empty result). This is especially handy when parsing paths that could contain (or not) a finishing/character. – eckes Jan 23 '13 at 15:23Dobz , Jun 25, 2014 at 11:44
How would you then do the opposite of this? to echo out '1:2:3:4:'? – Dobz Jun 25 '14 at 11:44Mihai Danila , Jul 9, 2014 at 14:07
And how does one keep the part before the last separator? Apparently by using${foo%:*}.#- from beginning;%- from end.#,%- shortest match;##,%%- longest match. – Mihai Danila Jul 9 '14 at 14:07Putnik , Feb 11, 2016 at 22:33
If i want to get the last element from path, how should I use it?echo ${pwd##*/}does not work. – Putnik Feb 11 '16 at 22:33Stan Strum , Dec 17, 2017 at 4:22
@Putnik that command seespwdas a variable. Trydir=$(pwd); echo ${dir##*/}. Works for me! – Stan Strum Dec 17 '17 at 4:22a3nm , Feb 3, 2012 at 8:39
Another way is to reverse before and aftercut:$ echo ab:cd:ef | rev | cut -d: -f1 | rev efThis makes it very easy to get the last but one field, or any range of fields numbered from the end.
Dannid , Jan 14, 2013 at 20:50
This answer is nice because it uses 'cut', which the author is (presumably) already familiar. Plus, I like this answer because I am using 'cut' and had this exact question, hence finding this thread via search. – Dannid Jan 14 '13 at 20:50funroll , Aug 12, 2013 at 19:51
Some cut-and-paste fodder for people using spaces as delimiters:echo "1 2 3 4" | rev | cut -d " " -f1 | rev– funroll Aug 12 '13 at 19:51EdgeCaseBerg , Sep 8, 2013 at 5:01
the rev | cut -d -f1 | rev is so clever! Thanks! Helped me a bunch (my use case was rev | -d ' ' -f 2- | rev – EdgeCaseBerg Sep 8 '13 at 5:01Anarcho-Chossid , Sep 16, 2015 at 15:54
Wow. Beautiful and dark magic. – Anarcho-Chossid Sep 16 '15 at 15:54shearn89 , Aug 17, 2017 at 9:27
I always forget aboutrev, was just what I needed!cut -b20- | rev | cut -b10- | rev– shearn89 Aug 17 '17 at 9:27William Pursell , Jul 2, 2010 at 7:09
It's difficult to get the last field using cut, but here's (one set of) solutions in awk and perl$ echo 1:2:3:4:5 | awk -F: '{print $NF}' 5 $ echo 1:2:3:4:5 | perl -F: -wane 'print $F[-1]' 5eckes , Jan 23, 2013 at 15:20
great advantage of this solution over the accepted answer: it also matches paths that contain or do not contain a finishing/character:/a/b/c/dand/a/b/c/d/yield the same result (d) when processingpwd | awk -F/ '{print $NF}'. The accepted answer results in an empty result in the case of/a/b/c/d/– eckes Jan 23 '13 at 15:20stamster , May 21 at 11:52
@eckes In case of AWK solution, on GNU bash, version 4.3.48(1)-release that's not true, as it matters whenever you have trailing slash or not. Simply put AWK will use/as delimiter, and if your path is/my/path/dir/it will use value after last delimiter, which is simply an empty string. So it's best to avoid trailing slash if you need to do such a thing like I do. – stamster May 21 at 11:52Nicholas M T Elliott , Jul 1, 2010 at 23:39
Assuming fairly simple usage (no escaping of the delimiter, for example), you can use grep:$ echo "1:2:3:4:5" | grep -oE "[^:]+$" 5Breakdown - find all the characters not the delimiter ([^:]) at the end of the line ($). -o only prints the matching part.
Dennis Williamson , Jul 2, 2010 at 0:05
One way:var1="1:2:3:4:5" var2=${var1##*:}Another, using an array:
var1="1:2:3:4:5" saveIFS=$IFS IFS=":" var2=($var1) IFS=$saveIFS var2=${var2[@]: -1}Yet another with an array:
var1="1:2:3:4:5" saveIFS=$IFS IFS=":" var2=($var1) IFS=$saveIFS count=${#var2[@]} var2=${var2[$count-1]}Using Bash (version >= 3.2) regular expressions:
var1="1:2:3:4:5" [[ $var1 =~ :([^:]*)$ ]] var2=${BASH_REMATCH[1]}liuyang1 , Mar 24, 2015 at 6:02
Thanks so much for array style, as I need this feature, but not have cut, awk these utils. – liuyang1 Mar 24 '15 at 6:02user3133260 , Dec 24, 2013 at 19:04
$ echo "a b c d e" | tr ' ' '\n' | tail -1 eSimply translate the delimiter into a newline and choose the last entry with
tail -1.Yajo , Jul 30, 2014 at 10:13
It will fail if the last item contains a\n, but for most cases is the most readable solution. – Yajo Jul 30 '14 at 10:13Rafael , Nov 10, 2016 at 10:09
Usingsed:$ echo '1:2:3:4:5' | sed 's/.*://' # => 5 $ echo '' | sed 's/.*://' # => (empty) $ echo ':' | sed 's/.*://' # => (empty) $ echo ':b' | sed 's/.*://' # => b $ echo '::c' | sed 's/.*://' # => c $ echo 'a' | sed 's/.*://' # => a $ echo 'a:' | sed 's/.*://' # => (empty) $ echo 'a:b' | sed 's/.*://' # => b $ echo 'a::c' | sed 's/.*://' # => cAb Irato , Nov 13, 2013 at 16:10
If your last field is a single character, you could do this:a="1:2:3:4:5" echo ${a: -1} echo ${a:(-1)}Check string manipulation in bash .
gniourf_gniourf , Nov 13, 2013 at 16:15
This doesn't work: it gives the last character ofa, not the last field . – gniourf_gniourf Nov 13 '13 at 16:15Ab Irato , Nov 25, 2013 at 13:25
True, that's the idea, if you know the length of the last field it's good. If not you have to use something else... – Ab Irato Nov 25 '13 at 13:25sphakka , Jan 25, 2016 at 16:24
Interesting, I didn't know of these particular Bash string manipulations. It also resembles to Python's string/array slicing . – sphakka Jan 25 '16 at 16:24ghostdog74 , Jul 2, 2010 at 1:16
Using Bash.$ var1="1:2:3:4:0" $ IFS=":" $ set -- $var1 $ eval echo \$${#} 0Sopalajo de Arrierez , Dec 24, 2014 at 5:04
I would buy some details about this method, please :-) . – Sopalajo de Arrierez Dec 24 '14 at 5:04Rafa , Apr 27, 2017 at 22:10
Could have usedecho ${!#}instead ofeval echo \$${#}. – Rafa Apr 27 '17 at 22:10Crytis , Dec 7, 2016 at 6:51
echo "a:b:c:d:e"|xargs -d : -n1|tail -1First use xargs split it using ":",-n1 means every line only have one part.Then,pring the last part.
BDL , Dec 7, 2016 at 13:47
Although this might solve the problem, one should always add an explanation to it. – BDL Dec 7 '16 at 13:47Crytis , Jun 7, 2017 at 9:13
already added.. – Crytis Jun 7 '17 at 9:13021 , Apr 26, 2016 at 11:33
There are many good answers here, but still I want to share this one using basename :basename $(echo "a:b:c:d:e" | tr ':' '/')However it will fail if there are already some '/' in your string . If slash / is your delimiter then you just have to (and should) use basename.
It's not the best answer but it just shows how you can be creative using bash commands.
Nahid Akbar , Jun 22, 2012 at 2:55
for x in `echo $str | tr ";" "\n"`; do echo $x; donechepner , Jun 22, 2012 at 12:58
This runs into problems if there is whitespace in any of the fields. Also, it does not directly address the question of retrieving the last field. – chepner Jun 22 '12 at 12:58Christoph Böddeker , Feb 19 at 15:50
For those that comfortable with Python, https://github.com/Russell91/pythonpy is a nice choice to solve this problem.$ echo "a:b:c:d:e" | py -x 'x.split(":")[-1]'From the pythonpy help:
-x treat each row of stdin as x.With that tool, it is easy to write python code that gets applied to the input.
baz , Nov 24, 2017 at 19:27
a solution using the read builtinIFS=':' read -a field <<< "1:2:3:4:5" echo ${field[4]}
|
|
|
Nov 08, 2018 | stackoverflow.com
stefanB , May 28, 2009 at 2:03
I have this string stored in a variable:IN="bla@some.com;john@home.com"Now I would like to split the strings by
;delimiter so that I have:ADDR1="bla@some.com" ADDR2="john@home.com"I don't necessarily need the
ADDR1andADDR2variables. If they are elements of an array that's even better.
After suggestions from the answers below, I ended up with the following which is what I was after:
#!/usr/bin/env bash IN="bla@some.com;john@home.com" mails=$(echo $IN | tr ";" "\n") for addr in $mails do echo "> [$addr]" doneOutput:
> [bla@some.com] > [john@home.com]There was a solution involving setting Internal_field_separator (IFS) to
;. I am not sure what happened with that answer, how do you resetIFSback to default?RE:
IFSsolution, I tried this and it works, I keep the oldIFSand then restore it:IN="bla@some.com;john@home.com" OIFS=$IFS IFS=';' mails2=$IN for x in $mails2 do echo "> [$x]" done IFS=$OIFSBTW, when I tried
mails2=($IN)I only got the first string when printing it in loop, without brackets around
$INit works.Brooks Moses , May 1, 2012 at 1:26
With regards to your "Edit2": You can simply "unset IFS" and it will return to the default state. There's no need to save and restore it explicitly unless you have some reason to expect that it's already been set to a non-default value. Moreover, if you're doing this inside a function (and, if you aren't, why not?), you can set IFS as a local variable and it will return to its previous value once you exit the function. – Brooks Moses May 1 '12 at 1:26dubiousjim , May 31, 2012 at 5:21
@BrooksMoses: (a) +1 for usinglocal IFS=...where possible; (b) -1 forunset IFS, this doesn't exactly reset IFS to its default value, though I believe an unset IFS behaves the same as the default value of IFS ($' \t\n'), however it seems bad practice to be assuming blindly that your code will never be invoked with IFS set to a custom value; (c) another idea is to invoke a subshell:(IFS=$custom; ...)when the subshell exits IFS will return to whatever it was originally. – dubiousjim May 31 '12 at 5:21nicooga , Mar 7, 2016 at 15:32
I just want to have a quick look at the paths to decide where to throw an executable, so I resorted to runruby -e "puts ENV.fetch('PATH').split(':')". If you want to stay pure bash won't help but using any scripting language that has a built-in split is easier. – nicooga Mar 7 '16 at 15:32Jeff , Apr 22 at 17:51
This is kind of a drive-by comment, but since the OP used email addresses as the example, has anyone bothered to answer it in a way that is fully RFC 5322 compliant, namely that any quoted string can appear before the @ which means you're going to need regular expressions or some other kind of parser instead of naive use of IFS or other simplistic splitter functions. – Jeff Apr 22 at 17:51user2037659 , Apr 26 at 20:15
for x in $(IFS=';';echo $IN); do echo "> [$x]"; done– user2037659 Apr 26 at 20:15Johannes Schaub - litb , May 28, 2009 at 2:23
You can set the internal field separator (IFS) variable, and then let it parse into an array. When this happens in a command, then the assignment toIFSonly takes place to that single command's environment (toread). It then parses the input according to theIFSvariable value into an array, which we can then iterate over.IFS=';' read -ra ADDR <<< "$IN" for i in "${ADDR[@]}"; do # process "$i" doneIt will parse one line of items separated by
;, pushing it into an array. Stuff for processing whole of$IN, each time one line of input separated by;:while IFS=';' read -ra ADDR; do for i in "${ADDR[@]}"; do # process "$i" done done <<< "$IN"Chris Lutz , May 28, 2009 at 2:25
This is probably the best way. How long will IFS persist in it's current value, can it mess up my code by being set when it shouldn't be, and how can I reset it when I'm done with it? – Chris Lutz May 28 '09 at 2:25Johannes Schaub - litb , May 28, 2009 at 3:04
now after the fix applied, only within the duration of the read command :) – Johannes Schaub - litb May 28 '09 at 3:04lhunath , May 28, 2009 at 6:14
You can read everything at once without using a while loop: read -r -d '' -a addr <<< "$in" # The -d '' is key here, it tells read not to stop at the first newline (which is the default -d) but to continue until EOF or a NULL byte (which only occur in binary data). – lhunath May 28 '09 at 6:14Charles Duffy , Jul 6, 2013 at 14:39
@LucaBorrione SettingIFSon the same line as thereadwith no semicolon or other separator, as opposed to in a separate command, scopes it to that command -- so it's always "restored"; you don't need to do anything manually. – Charles Duffy Jul 6 '13 at 14:39chepner , Oct 2, 2014 at 3:50
@imagineerThis There is a bug involving herestrings and local changes to IFS that requires$INto be quoted. The bug is fixed inbash4.3. – chepner Oct 2 '14 at 3:50palindrom , Mar 10, 2011 at 9:00
Taken from Bash shell script split array :IN="bla@some.com;john@home.com" arrIN=(${IN//;/ })Explanation:
This construction replaces all occurrences of
';'(the initial//means global replace) in the stringINwith' '(a single space), then interprets the space-delimited string as an array (that's what the surrounding parentheses do).The syntax used inside of the curly braces to replace each
';'character with a' 'character is called Parameter Expansion .There are some common gotchas:
Oz123 , Mar 21, 2011 at 18:50
I just want to add: this is the simplest of all, you can access array elements with ${arrIN[1]} (starting from zeros of course) – Oz123 Mar 21 '11 at 18:50KomodoDave , Jan 5, 2012 at 15:13
Found it: the technique of modifying a variable within a ${} is known as 'parameter expansion'. – KomodoDave Jan 5 '12 at 15:13qbolec , Feb 25, 2013 at 9:12
Does it work when the original string contains spaces? – qbolec Feb 25 '13 at 9:12Ethan , Apr 12, 2013 at 22:47
No, I don't think this works when there are also spaces present... it's converting the ',' to ' ' and then building a space-separated array. – Ethan Apr 12 '13 at 22:47Charles Duffy , Jul 6, 2013 at 14:39
This is a bad approach for other reasons: For instance, if your string contains;*;, then the*will be expanded to a list of filenames in the current directory. -1 – Charles Duffy Jul 6 '13 at 14:39Chris Lutz , May 28, 2009 at 2:09
If you don't mind processing them immediately, I like to do this:for i in $(echo $IN | tr ";" "\n") do # process doneYou could use this kind of loop to initialize an array, but there's probably an easier way to do it. Hope this helps, though.
Chris Lutz , May 28, 2009 at 2:42
You should have kept the IFS answer. It taught me something I didn't know, and it definitely made an array, whereas this just makes a cheap substitute. – Chris Lutz May 28 '09 at 2:42Johannes Schaub - litb , May 28, 2009 at 2:59
I see. Yeah i find doing these silly experiments, i'm going to learn new things each time i'm trying to answer things. I've edited stuff based on #bash IRC feedback and undeleted :) – Johannes Schaub - litb May 28 '09 at 2:59lhunath , May 28, 2009 at 6:12
-1, you're obviously not aware of wordsplitting, because it's introducing two bugs in your code. one is when you don't quote $IN and the other is when you pretend a newline is the only delimiter used in wordsplitting. You are iterating over every WORD in IN, not every line, and DEFINATELY not every element delimited by a semicolon, though it may appear to have the side-effect of looking like it works. – lhunath May 28 '09 at 6:12Johannes Schaub - litb , May 28, 2009 at 17:00
You could change it to echo "$IN" | tr ';' '\n' | while read -r ADDY; do # process "$ADDY"; done to make him lucky, i think :) Note that this will fork, and you can't change outer variables from within the loop (that's why i used the <<< "$IN" syntax) then – Johannes Schaub - litb May 28 '09 at 17:00mklement0 , Apr 24, 2013 at 14:13
To summarize the debate in the comments: Caveats for general use : the shell applies word splitting and expansions to the string, which may be undesired; just try it with.IN="bla@some.com;john@home.com;*;broken apart". In short: this approach will break, if your tokens contain embedded spaces and/or chars. such as*that happen to make a token match filenames in the current folder. – mklement0 Apr 24 '13 at 14:13F. Hauri , Apr 13, 2013 at 14:20
Compatible answerTo this SO question, there is already a lot of different way to do this in bash . But bash has many special features, so called bashism that work well, but that won't work in any other shell .
In particular, arrays , associative array , and pattern substitution are pure bashisms and may not work under other shells .
On my Debian GNU/Linux , there is a standard shell called dash , but I know many people who like to use ksh .
Finally, in very small situation, there is a special tool called busybox with his own shell interpreter ( ash ).
Requested stringThe string sample in SO question is:
IN="bla@some.com;john@home.com"As this could be useful with whitespaces and as whitespaces could modify the result of the routine, I prefer to use this sample string:
IN="bla@some.com;john@home.com;Full Name <fulnam@other.org>"Split string based on delimiter in bash (version >=4.2)Under pure bash, we may use arrays and IFS :
var="bla@some.com;john@home.com;Full Name <fulnam@other.org>"oIFS="$IFS" IFS=";" declare -a fields=($var) IFS="$oIFS" unset oIFSIFS=\; read -a fields <<<"$var"Using this syntax under recent bash don't change
$IFSfor current session, but only for the current command:set | grep ^IFS= IFS=$' \t\n'Now the string
varis split and stored into an array (namedfields):set | grep ^fields=\\\|^var= fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>") var='bla@some.com;john@home.com;Full Name <fulnam@other.org>'We could request for variable content with
declare -p:declare -p var fields declare -- var="bla@some.com;john@home.com;Full Name <fulnam@other.org>" declare -a fields=([0]="bla@some.com" [1]="john@home.com" [2]="Full Name <fulnam@other.org>")
readis the quickiest way to do the split, because there is no forks and no external resources called.From there, you could use the syntax you already know for processing each field:
for x in "${fields[@]}";do echo "> [$x]" done > [bla@some.com] > [john@home.com] > [Full Name <fulnam@other.org>]or drop each field after processing (I like this shifting approach):
while [ "$fields" ] ;do echo "> [$fields]" fields=("${fields[@]:1}") done > [bla@some.com] > [john@home.com] > [Full Name <fulnam@other.org>]or even for simple printout (shorter syntax):
printf "> [%s]\n" "${fields[@]}" > [bla@some.com] > [john@home.com] > [Full Name <fulnam@other.org>]Split string based on delimiter in shellBut if you would write something usable under many shells, you have to not use bashisms .
There is a syntax, used in many shells, for splitting a string across first or last occurrence of a substring:
${var#*SubStr} # will drop begin of string up to first occur of `SubStr` ${var##*SubStr} # will drop begin of string up to last occur of `SubStr` ${var%SubStr*} # will drop part of string from last occur of `SubStr` to the end ${var%%SubStr*} # will drop part of string from first occur of `SubStr` to the end(The missing of this is the main reason of my answer publication ;)
As pointed out by Score_Under :
#and%delete the shortest possible matching string, and
##and%%delete the longest possible.This little sample script work well under bash , dash , ksh , busybox and was tested under Mac-OS's bash too:
var="bla@some.com;john@home.com;Full Name <fulnam@other.org>" while [ "$var" ] ;do iter=${var%%;*} echo "> [$iter]" [ "$var" = "$iter" ] && \ var='' || \ var="${var#*;}" done > [bla@some.com] > [john@home.com] > [Full Name <fulnam@other.org>]Have fun!
Score_Under , Apr 28, 2015 at 16:58
The#,##,%, and%%substitutions have what is IMO an easier explanation to remember (for how much they delete):#and%delete the shortest possible matching string, and##and%%delete the longest possible. – Score_Under Apr 28 '15 at 16:58sorontar , Oct 26, 2016 at 4:36
TheIFS=\; read -a fields <<<"$var"fails on newlines and add a trailing newline. The other solution removes a trailing empty field. – sorontar Oct 26 '16 at 4:36Eric Chen , Aug 30, 2017 at 17:50
The shell delimiter is the most elegant answer, period. – Eric Chen Aug 30 '17 at 17:50sancho.s , Oct 4 at 3:42
Could the last alternative be used with a list of field separators set somewhere else? For instance, I mean to use this as a shell script, and pass a list of field separators as a positional parameter. – sancho.s Oct 4 at 3:42F. Hauri , Oct 4 at 7:47
Yes, in a loop:for sep in "#" "ł" "@" ; do ... var="${var#*$sep}" ...– F. Hauri Oct 4 at 7:47DougW , Apr 27, 2015 at 18:20
I've seen a couple of answers referencing thecutcommand, but they've all been deleted. It's a little odd that nobody has elaborated on that, because I think it's one of the more useful commands for doing this type of thing, especially for parsing delimited log files.In the case of splitting this specific example into a bash script array,
tris probably more efficient, butcutcan be used, and is more effective if you want to pull specific fields from the middle.Example:
$ echo "bla@some.com;john@home.com" | cut -d ";" -f 1 bla@some.com $ echo "bla@some.com;john@home.com" | cut -d ";" -f 2 john@home.comYou can obviously put that into a loop, and iterate the -f parameter to pull each field independently.
This gets more useful when you have a delimited log file with rows like this:
2015-04-27|12345|some action|an attribute|meta data
cutis very handy to be able tocatthis file and select a particular field for further processing.MisterMiyagi , Nov 2, 2016 at 8:42
Kudos for usingcut, it's the right tool for the job! Much cleared than any of those shell hacks. – MisterMiyagi Nov 2 '16 at 8:42uli42 , Sep 14, 2017 at 8:30
This approach will only work if you know the number of elements in advance; you'd need to program some more logic around it. It also runs an external tool for every element. – uli42 Sep 14 '17 at 8:30Louis Loudog Trottier , May 10 at 4:20
Excatly waht i was looking for trying to avoid empty string in a csv. Now i can point the exact 'column' value as well. Work with IFS already used in a loop. Better than expected for my situation. – Louis Loudog Trottier May 10 at 4:20, May 28, 2009 at 10:31
How about this approach:IN="bla@some.com;john@home.com" set -- "$IN" IFS=";"; declare -a Array=($*) echo "${Array[@]}" echo "${Array[0]}" echo "${Array[1]}"Yzmir Ramirez , Sep 5, 2011 at 1:06
+1 ... but I wouldn't name the variable "Array" ... pet peev I guess. Good solution. – Yzmir Ramirez Sep 5 '11 at 1:06ata , Nov 3, 2011 at 22:33
+1 ... but the "set" and declare -a are unnecessary. You could as well have used justIFS";" && Array=($IN)– ata Nov 3 '11 at 22:33Luca Borrione , Sep 3, 2012 at 9:26
+1 Only a side note: shouldn't it be recommendable to keep the old IFS and then restore it? (as shown by stefanB in his edit3) people landing here (sometimes just copying and pasting a solution) might not think about this – Luca Borrione Sep 3 '12 at 9:26Charles Duffy , Jul 6, 2013 at 14:44
-1: First, @ata is right that most of the commands in this do nothing. Second, it uses word-splitting to form the array, and doesn't do anything to inhibit glob-expansion when doing so (so if you have glob characters in any of the array elements, those elements are replaced with matching filenames). – Charles Duffy Jul 6 '13 at 14:44John_West , Jan 8, 2016 at 12:29
Suggest to use$'...':IN=$'bla@some.com;john@home.com;bet <d@\ns* kl.com>'. Thenecho "${Array[2]}"will print a string with newline.set -- "$IN"is also neccessary in this case. Yes, to prevent glob expansion, the solution should includeset -f. – John_West Jan 8 '16 at 12:29Steven Lizarazo , Aug 11, 2016 at 20:45
This worked for me:string="1;2" echo $string | cut -d';' -f1 # output is 1 echo $string | cut -d';' -f2 # output is 2Pardeep Sharma , Oct 10, 2017 at 7:29
this is sort and sweet :) – Pardeep Sharma Oct 10 '17 at 7:29space earth , Oct 17, 2017 at 7:23
Thanks...Helped a lot – space earth Oct 17 '17 at 7:23mojjj , Jan 8 at 8:57
cut works only with a single char as delimiter. – mojjj Jan 8 at 8:57lothar , May 28, 2009 at 2:12
echo "bla@some.com;john@home.com" | sed -e 's/;/\n/g' bla@some.com john@home.comLuca Borrione , Sep 3, 2012 at 10:08
-1 what if the string contains spaces? for exampleIN="this is first line; this is second line" arrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) )will produce an array of 8 elements in this case (an element for each word space separated), rather than 2 (an element for each line semi colon separated) – Luca Borrione Sep 3 '12 at 10:08lothar , Sep 3, 2012 at 17:33
@Luca No the sed script creates exactly two lines. What creates the multiple entries for you is when you put it into a bash array (which splits on white space by default) – lothar Sep 3 '12 at 17:33Luca Borrione , Sep 4, 2012 at 7:09
That's exactly the point: the OP needs to store entries into an array to loop over it, as you can see in his edits. I think your (good) answer missed to mention to usearrIN=( $( echo "$IN" | sed -e 's/;/\n/g' ) )to achieve that, and to advice to change IFS toIFS=$'\n'for those who land here in the future and needs to split a string containing spaces. (and to restore it back afterwards). :) – Luca Borrione Sep 4 '12 at 7:09lothar , Sep 4, 2012 at 16:55
@Luca Good point. However the array assignment was not in the initial question when I wrote up that answer. – lothar Sep 4 '12 at 16:55Ashok , Sep 8, 2012 at 5:01
This also works:IN="bla@some.com;john@home.com" echo ADD1=`echo $IN | cut -d \; -f 1` echo ADD2=`echo $IN | cut -d \; -f 2`Be careful, this solution is not always correct. In case you pass "bla@some.com" only, it will assign it to both ADD1 and ADD2.
fersarr , Mar 3, 2016 at 17:17
You can use -s to avoid the mentioned problem: superuser.com/questions/896800/ "-f, --fields=LIST select only these fields; also print any line that contains no delimiter character, unless the -s option is specified" – fersarr Mar 3 '16 at 17:17Tony , Jan 14, 2013 at 6:33
I think AWK is the best and efficient command to resolve your problem. AWK is included in Bash by default in almost every Linux distribution.echo "bla@some.com;john@home.com" | awk -F';' '{print $1,$2}'will give
bla@some.com john@home.comOf course your can store each email address by redefining the awk print field.
Jaro , Jan 7, 2014 at 21:30
Or even simpler: echo "bla@some.com;john@home.com" | awk 'BEGIN{RS=";"} {print}' – Jaro Jan 7 '14 at 21:30Aquarelle , May 6, 2014 at 21:58
@Jaro This worked perfectly for me when I had a string with commas and needed to reformat it into lines. Thanks. – Aquarelle May 6 '14 at 21:58Eduardo Lucio , Aug 5, 2015 at 12:59
It worked in this scenario -> "echo "$SPLIT_0" | awk -F' inode=' '{print $1}'"! I had problems when trying to use atrings (" inode=") instead of characters (";"). $ 1, $ 2, $ 3, $ 4 are set as positions in an array! If there is a way of setting an array... better! Thanks! – Eduardo Lucio Aug 5 '15 at 12:59Tony , Aug 6, 2015 at 2:42
@EduardoLucio, what I'm thinking about is maybe you can first replace your delimiterinode=into;for example bysed -i 's/inode\=/\;/g' your_file_to_process, then define-F';'when applyawk, hope that can help you. – Tony Aug 6 '15 at 2:42nickjb , Jul 5, 2011 at 13:41
A different take on Darron's answer , this is how I do it:IN="bla@some.com;john@home.com" read ADDR1 ADDR2 <<<$(IFS=";"; echo $IN)ColinM , Sep 10, 2011 at 0:31
This doesn't work. – ColinM Sep 10 '11 at 0:31nickjb , Oct 6, 2011 at 15:33
I think it does! Run the commands above and then "echo $ADDR1 ... $ADDR2" and i get "bla@some.com ... john@home.com" output – nickjb Oct 6 '11 at 15:33Nick , Oct 28, 2011 at 14:36
This worked REALLY well for me... I used it to itterate over an array of strings which contained comma separated DB,SERVER,PORT data to use mysqldump. – Nick Oct 28 '11 at 14:36dubiousjim , May 31, 2012 at 5:28
Diagnosis: theIFS=";"assignment exists only in the$(...; echo $IN)subshell; this is why some readers (including me) initially think it won't work. I assumed that all of $IN was getting slurped up by ADDR1. But nickjb is correct; it does work. The reason is thatecho $INcommand parses its arguments using the current value of $IFS, but then echoes them to stdout using a space delimiter, regardless of the setting of $IFS. So the net effect is as though one had calledread ADDR1 ADDR2 <<< "bla@some.com john@home.com"(note the input is space-separated not ;-separated). – dubiousjim May 31 '12 at 5:28sorontar , Oct 26, 2016 at 4:43
This fails on spaces and newlines, and also expand wildcards*in theecho $INwith an unquoted variable expansion. – sorontar Oct 26 '16 at 4:43gniourf_gniourf , Jun 26, 2014 at 9:11
In Bash, a bullet proof way, that will work even if your variable contains newlines:IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in")Look:
$ in=$'one;two three;*;there is\na newline\nin this field' $ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in") $ declare -p array declare -a array='([0]="one" [1]="two three" [2]="*" [3]="there is a newline in this field")'The trick for this to work is to use the
-doption ofread(delimiter) with an empty delimiter, so thatreadis forced to read everything it's fed. And we feedreadwith exactly the content of the variablein, with no trailing newline thanks toprintf. Note that's we're also putting the delimiter inprintfto ensure that the string passed toreadhas a trailing delimiter. Without it,readwould trim potential trailing empty fields:$ in='one;two;three;' # there's an empty field $ IFS=';' read -d '' -ra array < <(printf '%s;\0' "$in") $ declare -p array declare -a array='([0]="one" [1]="two" [2]="three" [3]="")'the trailing empty field is preserved.
Update for Bash≥4.4Since Bash 4.4, the builtin
mapfile(akareadarray) supports the-doption to specify a delimiter. Hence another canonical way is:mapfile -d ';' -t array < <(printf '%s;' "$in")John_West , Jan 8, 2016 at 12:10
I found it as the rare solution on that list that works correctly with\n, spaces and*simultaneously. Also, no loops; array variable is accessible in the shell after execution (contrary to the highest upvoted answer). Note,in=$'...', it does not work with double quotes. I think, it needs more upvotes. – John_West Jan 8 '16 at 12:10Darron , Sep 13, 2010 at 20:10
How about this one liner, if you're not using arrays:IFS=';' read ADDR1 ADDR2 <<<$INdubiousjim , May 31, 2012 at 5:36
Consider usingread -r ...to ensure that, for example, the two characters "\t" in the input end up as the same two characters in your variables (instead of a single tab char). – dubiousjim May 31 '12 at 5:36Luca Borrione , Sep 3, 2012 at 10:07
-1 This is not working here (ubuntu 12.04). Addingecho "ADDR1 $ADDR1"\n echo "ADDR2 $ADDR2"to your snippet will outputADDR1 bla@some.com john@home.com\nADDR2(\n is newline) – Luca Borrione Sep 3 '12 at 10:07chepner , Sep 19, 2015 at 13:59
This is probably due to a bug involvingIFSand here strings that was fixed inbash4.3. Quoting$INshould fix it. (In theory,$INis not subject to word splitting or globbing after it expands, meaning the quotes should be unnecessary. Even in 4.3, though, there's at least one bug remaining--reported and scheduled to be fixed--so quoting remains a good idea.) – chepner Sep 19 '15 at 13:59sorontar , Oct 26, 2016 at 4:55
This breaks if $in contain newlines even if $IN is quoted. And adds a trailing newline. – sorontar Oct 26 '16 at 4:55kenorb , Sep 11, 2015 at 20:54
Here is a clean 3-liner:in="foo@bar;bizz@buzz;fizz@buzz;buzz@woof" IFS=';' list=($in) for item in "${list[@]}"; do echo $item; donewhere
IFSdelimit words based on the separator and()is used to create an array . Then[@]is used to return each item as a separate word.If you've any code after that, you also need to restore
$IFS, e.g.unset IFS.sorontar , Oct 26, 2016 at 5:03
The use of$inunquoted allows wildcards to be expanded. – sorontar Oct 26 '16 at 5:03user2720864 , Sep 24 at 13:46
+ for the unset command – user2720864 Sep 24 at 13:46Emilien Brigand , Aug 1, 2016 at 13:15
Without setting the IFSIf you just have one colon you can do that:
a="foo:bar" b=${a%:*} c=${a##*:}you will get:
b = foo c = barVictor Choy , Sep 16, 2015 at 3:34
There is a simple and smart way like this:echo "add:sfff" | xargs -d: -i echo {}But you must use gnu xargs, BSD xargs cant support -d delim. If you use apple mac like me. You can install gnu xargs :
brew install findutilsthen
echo "add:sfff" | gxargs -d: -i echo {}Halle Knast , May 24, 2017 at 8:42
The following Bash/zsh function splits its first argument on the delimiter given by the second argument:split() { local string="$1" local delimiter="$2" if [ -n "$string" ]; then local part while read -d "$delimiter" part; do echo $part done <<< "$string" echo $part fi }For instance, the command
$ split 'a;b;c' ';'yields
a b cThis output may, for instance, be piped to other commands. Example:
$ split 'a;b;c' ';' | cat -n 1 a 2 b 3 cCompared to the other solutions given, this one has the following advantages:
IFSis not overriden: Due to dynamic scoping of even local variables, overridingIFSover a loop causes the new value to leak into function calls performed from within the loop.- Arrays are not used: Reading a string into an array using
readrequires the flag-ain Bash and-Ain zsh.If desired, the function may be put into a script as follows:
#!/usr/bin/env bash split() { # ... } split "$@"sandeepkunkunuru , Oct 23, 2017 at 16:10
works and neatly modularized. – sandeepkunkunuru Oct 23 '17 at 16:10Prospero , Sep 25, 2011 at 1:09
This is the simplest way to do it.spo='one;two;three' OIFS=$IFS IFS=';' spo_array=($spo) IFS=$OIFS echo ${spo_array[*]}rashok , Oct 25, 2016 at 12:41
IN="bla@some.com;john@home.com" IFS=';' read -a IN_arr <<< "${IN}" for entry in "${IN_arr[@]}" do echo $entry doneOutput
bla@some.com john@home.comSystem : Ubuntu 12.04.1
codeforester , Jan 2, 2017 at 5:37
IFS is not getting set in the specific context ofreadhere and hence it can upset rest of the code, if any. – codeforester Jan 2 '17 at 5:37shuaihanhungry , Jan 20 at 15:54
you can apply awk to many situationsecho "bla@some.com;john@home.com"|awk -F';' '{printf "%s\n%s\n", $1, $2}'also you can use this
echo "bla@some.com;john@home.com"|awk -F';' '{print $1,$2}' OFS="\n"ghost , Apr 24, 2013 at 13:13
If no space, Why not this?IN="bla@some.com;john@home.com" arr=(`echo $IN | tr ';' ' '`) echo ${arr[0]} echo ${arr[1]}eukras , Oct 22, 2012 at 7:10
There are some cool answers here (errator esp.), but for something analogous to split in other languages -- which is what I took the original question to mean -- I settled on this:IN="bla@some.com;john@home.com" declare -a a="(${IN/;/ })";Now
${a[0]},${a[1]}, etc, are as you would expect. Use${#a[*]}for number of terms. Or to iterate, of course:for i in ${a[*]}; do echo $i; doneIMPORTANT NOTE:
This works in cases where there are no spaces to worry about, which solved my problem, but may not solve yours. Go with the
$IFSsolution(s) in that case.olibre , Oct 7, 2013 at 13:33
Does not work whenINcontains more than two e-mail addresses. Please refer to same idea (but fixed) at palindrom's answer – olibre Oct 7 '13 at 13:33sorontar , Oct 26, 2016 at 5:14
Better use${IN//;/ }(double slash) to make it also work with more than two values. Beware that any wildcard (*?[) will be expanded. And a trailing empty field will be discarded. – sorontar Oct 26 '16 at 5:14jeberle , Apr 30, 2013 at 3:10
Use thesetbuilt-in to load up the$@array:IN="bla@some.com;john@home.com" IFS=';'; set $IN; IFS=$' \t\n'Then, let the party begin:
echo $# for a; do echo $a; done ADDR1=$1 ADDR2=$2sorontar , Oct 26, 2016 at 5:17
Better useset -- $INto avoid some issues with "$IN" starting with dash. Still, the unquoted expansion of$INwill expand wildcards (*?[). – sorontar Oct 26 '16 at 5:17NevilleDNZ , Sep 2, 2013 at 6:30
Two bourne-ish alternatives where neither require bash arrays:Case 1 : Keep it nice and simple: Use a NewLine as the Record-Separator... eg.
IN="bla@some.com john@home.com" while read i; do # process "$i" ... eg. echo "[email:$i]" done <<< "$IN"Note: in this first case no sub-process is forked to assist with list manipulation.
Idea: Maybe it is worth using NL extensively internally , and only converting to a different RS when generating the final result externally .
Case 2 : Using a ";" as a record separator... eg.
NL=" " IRS=";" ORS=";" conv_IRS() { exec tr "$1" "$NL" } conv_ORS() { exec tr "$NL" "$1" } IN="bla@some.com;john@home.com" IN="$(conv_IRS ";" <<< "$IN")" while read i; do # process "$i" ... eg. echo -n "[email:$i]$ORS" done <<< "$IN"In both cases a sub-list can be composed within the loop is persistent after the loop has completed. This is useful when manipulating lists in memory, instead storing lists in files. {p.s. keep calm and carry on B-) }
fedorqui , Jan 8, 2015 at 10:21
Apart from the fantastic answers that were already provided, if it is just a matter of printing out the data you may consider usingawk:awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN"This sets the field separator to
Test;, so that it can loop through the fields with aforloop and print accordingly.$ IN="bla@some.com;john@home.com" $ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "$IN" > [bla@some.com] > [john@home.com]With another input:
$ awk -F";" '{for (i=1;i<=NF;i++) printf("> [%s]\n", $i)}' <<< "a;b;c d;e_;f" > [a] > [b] > [c d] > [e_] > [f]18446744073709551615 , Feb 20, 2015 at 10:49
In Android shell, most of the proposed methods just do not work:$ IFS=':' read -ra ADDR <<<"$PATH" /system/bin/sh: can't create temporary file /sqlite_stmt_journals/mksh.EbNoR10629: No such file or directoryWhat does work is:
$ for i in ${PATH//:/ }; do echo $i; done /sbin /vendor/bin /system/sbin /system/bin /system/xbinwhere
//means global replacement.sorontar , Oct 26, 2016 at 5:08
Fails if any part of $PATH contains spaces (or newlines). Also expands wildcards (asterisk *, question mark ? and braces [ ]). – sorontar Oct 26 '16 at 5:08Eduardo Lucio , Apr 4, 2016 at 19:54
Okay guys!Here's my answer!
DELIMITER_VAL='=' read -d '' F_ABOUT_DISTRO_R <<"EOF" DISTRIB_ID=Ubuntu DISTRIB_RELEASE=14.04 DISTRIB_CODENAME=trusty DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS" NAME="Ubuntu" VERSION="14.04.4 LTS, Trusty Tahr" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 14.04.4 LTS" VERSION_ID="14.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" EOF SPLIT_NOW=$(awk -F$DELIMITER_VAL '{for(i=1;i<=NF;i++){printf "%s\n", $i}}' <<<"${F_ABOUT_DISTRO_R}") while read -r line; do SPLIT+=("$line") done <<< "$SPLIT_NOW" for i in "${SPLIT[@]}"; do echo "$i" doneWhy this approach is "the best" for me?
Because of two reasons:
- You do not need to escape the delimiter;
- You will not have problem with blank spaces . The value will be properly separated in the array!
[]'s
gniourf_gniourf , Jan 30, 2017 at 8:26
FYI,/etc/os-releaseand/etc/lsb-releaseare meant to be sourced, and not parsed. So your method is really wrong. Moreover, you're not quite answering the question about spiltting a string on a delimiter. – gniourf_gniourf Jan 30 '17 at 8:26Michael Hale , Jun 14, 2012 at 17:38
A one-liner to split a string separated by ';' into an array is:IN="bla@some.com;john@home.com" ADDRS=( $(IFS=";" echo "$IN") ) echo ${ADDRS[0]} echo ${ADDRS[1]}This only sets IFS in a subshell, so you don't have to worry about saving and restoring its value.
Luca Borrione , Sep 3, 2012 at 10:04
-1 this doesn't work here (ubuntu 12.04). it prints only the first echo with all $IN value in it, while the second is empty. you can see it if you put echo "0: "${ADDRS[0]}\n echo "1: "${ADDRS[1]} the output is0: bla@some.com;john@home.com\n 1:(\n is new line) – Luca Borrione Sep 3 '12 at 10:04Luca Borrione , Sep 3, 2012 at 10:05
please refer to nickjb's answer at for a working alternative to this idea stackoverflow.com/a/6583589/1032370 – Luca Borrione Sep 3 '12 at 10:05Score_Under , Apr 28, 2015 at 17:09
-1, 1. IFS isn't being set in that subshell (it's being passed to the environment of "echo", which is a builtin, so nothing is happening anyway). 2.$INis quoted so it isn't subject to IFS splitting. 3. The process substitution is split by whitespace, but this may corrupt the original data. – Score_Under Apr 28 '15 at 17:09ajaaskel , Oct 10, 2014 at 11:33
IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)' set -f oldifs="$IFS" IFS=';'; arrayIN=($IN) IFS="$oldifs" for i in "${arrayIN[@]}"; do echo "$i" done set +fOutput:
bla@some.com john@home.com Charlie Brown <cbrown@acme.com !"#$%&/()[]{}*? are no problem simple is beautiful :-)Explanation: Simple assignment using parenthesis () converts semicolon separated list into an array provided you have correct IFS while doing that. Standard FOR loop handles individual items in that array as usual. Notice that the list given for IN variable must be "hard" quoted, that is, with single ticks.
IFS must be saved and restored since Bash does not treat an assignment the same way as a command. An alternate workaround is to wrap the assignment inside a function and call that function with a modified IFS. In that case separate saving/restoring of IFS is not needed. Thanks for "Bize" for pointing that out.
gniourf_gniourf , Feb 20, 2015 at 16:45
!"#$%&/()[]{}*? are no problemwell... not quite:[]*?are glob characters. So what about creating this directory and file: `mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem' and running your command? simple may be beautiful, but when it's broken, it's broken. – gniourf_gniourf Feb 20 '15 at 16:45ajaaskel , Feb 25, 2015 at 7:20
@gniourf_gniourf The string is stored in a variable. Please see the original question. – ajaaskel Feb 25 '15 at 7:20gniourf_gniourf , Feb 25, 2015 at 7:26
@ajaaskel you didn't fully understand my comment. Go in a scratch directory and issue these commands:mkdir '!"#$%&'; touch '!"#$%&/()[]{} got you hahahaha - are no problem'. They will only create a directory and a file, with weird looking names, I must admit. Then run your commands with the exactINyou gave:IN='bla@some.com;john@home.com;Charlie Brown <cbrown@acme.com;!"#$%&/()[]{}*? are no problem;simple is beautiful :-)'. You'll see that you won't get the output you expect. Because you're using a method subject to pathname expansions to split your string. – gniourf_gniourf Feb 25 '15 at 7:26gniourf_gniourf , Feb 25, 2015 at 7:29
This is to demonstrate that the characters*,?,[...]and even, ifextglobis set,!(...),@(...),?(...),+(...)are problems with this method! – gniourf_gniourf Feb 25 '15 at 7:29ajaaskel , Feb 26, 2015 at 15:26
@gniourf_gniourf Thanks for detailed comments on globbing. I adjusted the code to have globbing off. My point was however just to show that rather simple assignment can do the splitting job. – ajaaskel Feb 26 '15 at 15:26> , Dec 19, 2013 at 21:39
Maybe not the most elegant solution, but works with*and spaces:IN="bla@so me.com;*;john@home.com" for i in `delims=${IN//[^;]}; seq 1 $((${#delims} + 1))` do echo "> [`echo $IN | cut -d';' -f$i`]" doneOutputs
> [bla@so me.com] > [*] > [john@home.com]Other example (delimiters at beginning and end):
IN=";bla@so me.com;*;john@home.com;" > [] > [bla@so me.com] > [*] > [john@home.com] > []Basically it removes every character other than
;makingdelimseg.;;;. Then it doesforloop from1tonumber-of-delimitersas counted by${#delims}. The final step is to safely get the$ith part usingcut.
|
|
|
Oct 17, 2018 | linuxconfig.org
Create indexed arrays on the fly We can create indexed arrays with a more concise syntax, by simply assign them some values:
$ my_array=(foo bar)In this case we assigned multiple items at once to the array, but we can also insert one value at a time, specifying its index:$ my_array[0]=fooArray operations Once an array is created, we can perform some useful operations on it, like displaying its keys and values or modifying it by appending or removing elements: Print the values of an array To display all the values of an array we can use the following shell expansion syntax:${my_array[@]}Or even:${my_array[*]}Both syntax let us access all the values of the array and produce the same results, unless the expansion it's quoted. In this case a difference arises: in the first case, when using@, the expansion will result in a word for each element of the array. This becomes immediately clear when performing afor loop. As an example, imagine we have an array with two elements, "foo" and "bar":$ my_array=(foo bar)Performing aforloop on it will produce the following result:$ for i in "${my_array[@]}"; do echo "$i"; done foo barWhen using*, and the variable is quoted, instead, a single "result" will be produced, containing all the elements of the array:$ for i in "${my_array[*]}"; do echo "$i"; done foo bar
Print the keys of an array It's even possible to retrieve and print the keys used in an indexed or associative array, instead of their respective values. The syntax is almost identical, but relies on the use of the!operator:$ my_array=(foo bar baz) $ for index in "${!my_array[@]}"; do echo "$index"; done 0 1 2The same is valid for associative arrays:$ declare -A my_array $ my_array=([foo]=bar [baz]=foobar) $ for key in "${!my_array[@]}"; do echo "$key"; done baz fooAs you can see, being the latter an associative array, we can't count on the fact that retrieved values are returned in the same order in which they were declared. Getting the size of an array We can retrieve the size of an array (the number of elements contained in it), by using a specific shell expansion:$ my_array=(foo bar baz) $ echo "the array contains ${#my_array[@]} elements" the array contains 3 elementsWe have created an array which contains three elements, "foo", "bar" and "baz", then by using the syntax above, which differs from the one we saw before to retrieve the array values only for the#character before the array name, we retrieved the number of the elements in the array instead of its content. Adding elements to an array As we saw, we can add elements to an indexed or associative array by specifying respectively their index or associative key. In the case of indexed arrays, we can also simply add an element, by appending to the end of the array, using the+=operator:$ my_array=(foo bar) $ my_array+=(baz)If we now print the content of the array we see that the element has been added successfully:$ echo "${my_array[@]}" foo bar bazMultiple elements can be added at a time:$ my_array=(foo bar) $ my_array+=(baz foobar) $ echo "${my_array[@]}" foo bar baz foobarTo add elements to an associative array, we are bound to specify also their associated keys:$ declare -A my_array # Add single element $ my_array[foo]="bar" # Add multiple elements at a time $ my_array+=([baz]=foobar [foobarbaz]=baz)
Deleting an element from the array To delete an element from the array we need to know it's index or its key in the case of an associative array, and use theunsetcommand. Let's see an example:$ my_array=(foo bar baz) $ unset my_array[1] $ echo ${my_array[@]} foo bazWe have created a simple array containing three elements, "foo", "bar" and "baz", then we deleted "bar" from it runningunsetand referencing the index of "bar" in the array: in this case we know it was1, since bash arrays start at 0. If we check the indexes of the array, we can now see that1is missing:$ echo ${!my_array[@]} 0 2The same thing it's valid for associative arrays:$ declare -A my_array $ my_array+=([foo]=bar [baz]=foobar) $ unset my_array[foo] $ echo ${my_array[@]} foobarIn the example above, the value referenced by the "foo" key has been deleted, leaving only "foobar" in the array.Deleting an entire array, it's even simpler: we just pass the array name as an argument to the
unsetcommand without specifying any index or key:$ unset my_array $ echo ${!my_array[@]}After executingunsetagainst the entire array, when trying to print its content an empty result is returned: the array doesn't exist anymore. Conclusions In this tutorial we saw the difference between indexed and associative arrays in bash, how to initialize them and how to perform fundamental operations, like displaying their keys and values and appending or removing items. Finally we saw how to unset them completely. Bash syntax can sometimes be pretty weird, but using arrays in scripts can be really useful. When a script starts to become more complex than expected, my advice is, however, to switch to a more capable scripting language such as python.
|
|
|
Oct 13, 2018 | gist.github.com
Instantly share code, notes, and snippets.
mbadran / gist:130469 Created
Jun 16, 2009 Embed
What would you like to do? Learn more about clone URLs Download ZIP replace cd in bash to (silent) pushd Raw gistfile1.sh
alias cd= " pushd $@ > /dev/null " 
This comment has been minimized. Show comment Hide comment
Copy link 
bobbydavidSep 19, 2012 One annoyance with this alias is that simply typing "cd" will twiddle the directory stack instead of bringing you to your home directory. bobbydavid commented
Copy link Sep 19, 2012
One annoyance with this alias is that simply typing "cd" will twiddle the directory stack instead of bringing you to your home directory. 
This comment has been minimized. Show comment Hide comment
Copy link 
didelerMar 9, 2013 @bobbydavid makes a good point. This would be better as a function. function cd { if (("$#" > 0)); then pushd "$@" > /dev/null else cd $HOME fi }By the way, I found this gist by googling "silence pushd".
dideler commented
Copy link Mar 9, 2013
@bobbydavid makes a good point. This would be better as a function. function cd { if (("$#" > 0)); then pushd "$@" > /dev/null else cd $HOME fi }By the way, I found this gist by googling "silence pushd".
This comment has been minimized. Show comment Hide comment
Copy link 
ghostMay 30, 2013 Don't you miss something? function cd { if (("$#" > 0)); then if [ "$1" == "-" ]; then popd > /dev/null else pushd "$@" > /dev/null fi else cd $HOME fi }You can always mimic the "cd -" functionality by using pushd alone.
Btw, I also found this gist by googling "silent pushd" ;)ghost commented
Copy link May 30, 2013
Don't you miss something? function cd { if (("$#" > 0)); then if [ "$1" == "-" ]; then popd > /dev/null else pushd "$@" > /dev/null fi else cd $HOME fi }You can always mimic the "cd -" functionality by using pushd alone.
Btw, I also found this gist by googling "silent pushd" ;)
This comment has been minimized. Show comment Hide comment
Copy link 
craJul 1, 2014 And thanks to your last comment, I found this gist by googling "silent cd -" :) cra commented
Copy link Jul 1, 2014
And thanks to your last comment, I found this gist by googling "silent cd -" :) 
This comment has been minimized. Show comment Hide comment
Copy link 
keltrothJun 25, 2015 With bash completion activated a can't get rid of this error :
"bash: pushd: cd: No such file or directory"...Any clue ?
keltroth commented
Copy link Jun 25, 2015
With bash completion activated a can't get rid of this error :
"bash: pushd: cd: No such file or directory"...Any clue ?
This comment has been minimized. Show comment Hide comment
Copy link keltrothJun 25, 2015 Got it !
One have to add :complete -d cdAfter making the alias !
My complete code here :
function _cd { if (("$#" > 0)); then if [ "$1" == "-" ]; then popd > /dev/null else pushd "$@" > /dev/null fi else cd $HOME fi } alias cd=_cd complete -d cdkeltroth commented
Copy link Jun 25, 2015
Got it !
One have to add :complete -d cdAfter making the alias !
My complete code here :
function _cd { if (("$#" > 0)); then if [ "$1" == "-" ]; then popd > /dev/null else pushd "$@" > /dev/null fi else cd $HOME fi } alias cd=_cd complete -d cd
This comment has been minimized. Show comment Hide comment
Copy link 
jan-warcholNov 29, 2015 I wanted to be able to go back by a given number of history items by typing cd -n, and I came up with this:function _cd { # typing just `_cd` will take you $HOME ;) if [ "$1" == "" ]; then pushd "$HOME" > /dev/null # use `_cd -` to visit previous directory elif [ "$1" == "-" ]; then pushd $OLDPWD > /dev/null # use `_cd -n` to go n directories back in history elif [[ "$1" =~ ^-[0-9]+$ ]]; then for i in `seq 1 ${1/-/}`; do popd > /dev/null done # use `_cd -- <path>` if your path begins with a dash elif [ "$1" == "--" ]; then shift pushd -- "$@" > /dev/null # basic case: move to a dir and add it to history else pushd "$@" > /dev/null fi } # replace standard `cd` with enhanced version, ensure tab-completion works alias cd=_cd complete -d cdI think you may find this interesting.
jan-warchol commented
Copy link Nov 29, 2015
I wanted to be able to go back by a given number of history items by typing cd -n, and I came up with this:function _cd { # typing just `_cd` will take you $HOME ;) if [ "$1" == "" ]; then pushd "$HOME" > /dev/null # use `_cd -` to visit previous directory elif [ "$1" == "-" ]; then pushd $OLDPWD > /dev/null # use `_cd -n` to go n directories back in history elif [[ "$1" =~ ^-[0-9]+$ ]]; then for i in `seq 1 ${1/-/}`; do popd > /dev/null done # use `_cd -- <path>` if your path begins with a dash elif [ "$1" == "--" ]; then shift pushd -- "$@" > /dev/null # basic case: move to a dir and add it to history else pushd "$@" > /dev/null fi } # replace standard `cd` with enhanced version, ensure tab-completion works alias cd=_cd complete -d cdI think you may find this interesting.
This comment has been minimized. Show comment Hide comment
Copy link 
3v1n0Oct 25, 2017 Another improvement over @jan-warchol version, to make cd -to alternatively usepushd $OLDPWDandpopddepending on what we called before.This allows to avoid to fill your history with elements when you often do
cd -; cd - # repeated as long you want. This could be applied when using this alias also for$OLDPWD, but in that case it might be that you want it repeated there, so I didn't touch it.Also added
cd -las alias fordir -vand usecd -g Xto go to theXth directory in your history (without popping, that's possible too of course, but it' something more an addition in this case).# Replace cd with pushd https://gist.github.com/mbadran/130469 function push_cd() { # typing just `push_cd` will take you $HOME ;) if [ -z "$1" ]; then push_cd "$HOME" # use `push_cd -` to visit previous directory elif [ "$1" == "-" ]; then if [ "$(dirs -p | wc -l)" -gt 1 ]; then current_dir="$PWD" popd > /dev/null pushd -n $current_dir > /dev/null elif [ -n "$OLDPWD" ]; then push_cd $OLDPWD fi # use `push_cd -l` or `push_cd -s` to print current stack of folders elif [ "$1" == "-l" ] || [ "$1" == "-s" ]; then dirs -v # use `push_cd -l N` to go to the Nth directory in history (pushing) elif [ "$1" == "-g" ] && [[ "$2" =~ ^[0-9]+$ ]]; then indexed_path=$(dirs -p | sed -n $(($2+1))p) push_cd $indexed_path # use `push_cd +N` to go to the Nth directory in history (pushing) elif [[ "$1" =~ ^+[0-9]+$ ]]; then push_cd -g ${1/+/} # use `push_cd -N` to go n directories back in history elif [[ "$1" =~ ^-[0-9]+$ ]]; then for i in `seq 1 ${1/-/}`; do popd > /dev/null done # use `push_cd -- <path>` if your path begins with a dash elif [ "$1" == "--" ]; then shift pushd -- "$@" > /dev/null # basic case: move to a dir and add it to history else pushd "$@" > /dev/null if [ "$1" == "." ] || [ "$1" == "$PWD" ]; then popd -n > /dev/null fi fi if [ -n "$CD_SHOW_STACK" ]; then dirs -v fi } # replace standard `cd` with enhanced version, ensure tab-completion works alias cd=push_cd complete -d cd```3v1n0 commented
Copy link Oct 25, 2017 • edited
Another improvement over @jan-warchol version, to make cd -to alternatively usepushd $OLDPWDandpopddepending on what we called before.This allows to avoid to fill your history with elements when you often do
cd -; cd - # repeated as long you want. This could be applied when using this alias also for$OLDPWD, but in that case it might be that you want it repeated there, so I didn't touch it.Also added
cd -las alias fordir -vand usecd -g Xto go to theXth directory in your history (without popping, that's possible too of course, but it' something more an addition in this case).# Replace cd with pushd https://gist.github.com/mbadran/130469 function push_cd() { # typing just `push_cd` will take you $HOME ;) if [ -z "$1" ]; then push_cd "$HOME" # use `push_cd -` to visit previous directory elif [ "$1" == "-" ]; then if [ "$(dirs -p | wc -l)" -gt 1 ]; then current_dir="$PWD" popd > /dev/null pushd -n $current_dir > /dev/null elif [ -n "$OLDPWD" ]; then push_cd $OLDPWD fi # use `push_cd -l` or `push_cd -s` to print current stack of folders elif [ "$1" == "-l" ] || [ "$1" == "-s" ]; then dirs -v # use `push_cd -l N` to go to the Nth directory in history (pushing) elif [ "$1" == "-g" ] && [[ "$2" =~ ^[0-9]+$ ]]; then indexed_path=$(dirs -p | sed -n $(($2+1))p) push_cd $indexed_path # use `push_cd +N` to go to the Nth directory in history (pushing) elif [[ "$1" =~ ^+[0-9]+$ ]]; then push_cd -g ${1/+/} # use `push_cd -N` to go n directories back in history elif [[ "$1" =~ ^-[0-9]+$ ]]; then for i in `seq 1 ${1/-/}`; do popd > /dev/null done # use `push_cd -- <path>` if your path begins with a dash elif [ "$1" == "--" ]; then shift pushd -- "$@" > /dev/null # basic case: move to a dir and add it to history else pushd "$@" > /dev/null if [ "$1" == "." ] || [ "$1" == "$PWD" ]; then popd -n > /dev/null fi fi if [ -n "$CD_SHOW_STACK" ]; then dirs -v fi } # replace standard `cd` with enhanced version, ensure tab-completion works alias cd=push_cd complete -d cd```
|
|
|
Oct 10, 2018 | www.cyberciti.biz
- Abhijeet Vaidya says: March 11, 2010 at 11:41 am End single quote is missing.
Correct command is:
echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bash_profile- izaak says: March 12, 2010 at 11:06 am I would also add
$ echo 'export HISTSIZE=10000' >> ~/.bash_profileIt's really useful, I think.
- Dariusz says: March 12, 2010 at 2:31 pm you can add it to /etc/profile so it is available to all users. I also add:
# Make sure all terminals save history
shopt -s histappend histreedit histverify
shopt -s no_empty_cmd_completion # bash>=2.04 only# Whenever displaying the prompt, write the previous line to disk:
PROMPT_COMMAND='history -a'#Use GREP color features by default: This will highlight the matched words / regexes
export GREP_OPTIONS='–color=auto'
export GREP_COLOR='1;37;41′- Babar Haq says: March 15, 2010 at 6:25 am Good tip. We have multiple users connecting as root using ssh and running different commands. Is there a way to log the IP that command was run from?
Thanks in advance.
- Anthony says: August 21, 2014 at 9:01 pm Just for anyone who might still find this thread (like I did today):
export HISTTIMEFORMAT="%F %T : $(echo $SSH_CONNECTION | cut -d\ -f1) : "
will give you the time format, plus the IP address culled from the ssh_connection environment variable (thanks for pointing that out, Cadrian, I never knew about that before), all right there in your history output.
You could even add in $(whoami)@ right to get if you like (although if everyone's logging in with the root account that's not helpful).
- cadrian says: March 16, 2010 at 5:55 pm Yup, you can export one of this
env | grep SSH
SSH_CLIENT=192.168.78.22 42387 22
SSH_TTY=/dev/pts/0
SSH_CONNECTION=192.168.78.22 42387 192.168.36.76 22As their bash history filename
set |grep -i hist
HISTCONTROL=ignoreboth
HISTFILE=/home/cadrian/.bash_history
HISTFILESIZE=1000000000
HISTSIZE=10000000So in profile you can so something like HISTFILE=/root/.bash_history_$(echo $SSH_CONNECTION| cut -d\ -f1)
- TSI says: March 21, 2010 at 10:29 am bash 4 can syslog every command bat afaik, you have to recompile it (check file config-top.h). See the news file of bash: http://tiswww.case.edu/php/chet/bash/NEWS
If you want to safely export history of your luser, you can ssl-syslog them to a central syslog server.- Dinesh Jadhav says: November 12, 2010 at 11:00 am This is good command, It helps me a lot.
- Indie says: September 19, 2011 at 11:41 am You only need to use
export HISTTIMEFORMAT='%F %T 'in your .bash_profile
- lalit jain says: October 3, 2011 at 9:58 am -- show history with date & time
# HISTTIMEFORMAT='%c '
#history- Sohail says: January 13, 2012 at 7:05 am Hi
Nice trick but unfortunately, the commands which were executed in the past few days also are carrying the current day's (today's) timestamp.Please advice.
Regards
- Raymond says: March 15, 2012 at 9:05 am Hi Sohail,
Yes indeed that will be the behavior of the system since you have just enabled on that day the HISTTIMEFORMAT feature. In other words, the system recall or record the commands which were inputted prior enabling of this feature. Hope this answers your concern.
Thanks!
- Raymond says: March 15, 2012 at 9:08 am Hi Sohail,
Yes, that will be the behavior of the system since you have just enabled on that day the HISTTIMEFORMAT feature. In other words, the system can't recall or record the commands which were inputted prior enabling of this feature, thus it will just reflect on the printed output (upon execution of "history") the current day and time. Hope this answers your concern.
Thanks!
- Sohail says: February 24, 2012 at 6:45 am Hi
The command only lists the current date (Today) even for those commands which were executed on earlier days.
Any solutions ?
Regards
- nitiratna nikalje says: August 24, 2012 at 5:24 pm hi vivek.do u know any openings for freshers in linux field? I m doing rhce course from rajiv banergy. My samba,nfs-nis,dhcp,telnet,ftp,http,ssh,squid,cron,quota and system administration is over.iptables ,sendmail and dns is remaining.
-9029917299(Nitiratna)
- JMathew says: August 26, 2012 at 10:51 pm Hi,
Is there anyway to log username also along with the Command Which we typed
Thanks in Advance
- suresh says: May 22, 2013 at 1:42 pm How can i get full comman along with data and path as we het in history command.
- rajesh says: December 6, 2013 at 5:56 am Thanks it worked..
- Krishan says: February 7, 2014 at 6:18 am The command is not working properly. It is displaying the date and time of todays for all the commands where as I ran the some command three before.
How come it is displaying the today date
- PR says: April 29, 2014 at 5:18 pm Hi..
I want to collect the history of particular user everyday and want to send an email.I wrote below script.
for collecting everyday history by time shall i edit .profile file of that user
echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bash_profile
Script:#!/bin/bash #This script sends email of particular user history >/tmp/history if [ -s /tmp/history ] then mailx -s "history 29042014" </tmp/history fi rm /tmp/history #END OF THE SCRIPTCan any one suggest better way to collect particular user history for everyday
- lefty.crupps says: October 24, 2014 at 7:10 pm Love it, but using the ISO date format is always recommended (YYYY-MM-DD), just as every other sorted group goes from largest sorting (Year) to smallest sorting (day)
https://en.wikipedia.org/wiki/ISO_8601#Calendar_datesIn that case, myne looks like this:
echo 'export HISTTIMEFORMAT="%YY-%m-%d/ %T "' >> ~/.bashrcThanks for the tip!
- lefty.crupps says: October 24, 2014 at 7:11 pm please delete post 33, my command is messed up.
- lefty.crupps says: October 24, 2014 at 7:11 pm Love it, but using the ISO date format is always recommended (YYYY-MM-DD), just as every other sorted group goes from largest sorting (Year) to smallest sorting (day)
https://en.wikipedia.org/wiki/ISO_8601#Calendar_datesIn that case, myne looks like this:
echo ‘export HISTTIMEFORMAT=â€%Y-%m-%d %T “‘ >> ~/.bashrcThanks for the tip!
- Vanathu says: October 30, 2014 at 1:01 am its show only current date for all the command history
- lefty.crupps says: October 30, 2014 at 2:08 am it's marking all of your current history with today's date. Try checking again in a few days.
- tinu says: October 14, 2015 at 3:30 pm Hi All,
I Have enabled my history with the command given :
echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bash_profilei need to know how i can add the ip's also , from which the commands are fired to the system.
|
|
|
Jul 04, 2018 | stackoverflow.com
Lawrence Johnston ,Oct 10, 2008 at 16:57
Say, I have a script that gets called with this line:./myscript -vfd ./foo/bar/someFile -o /fizz/someOtherFileor this one:
./myscript -v -f -d -o /fizz/someOtherFile ./foo/bar/someFileWhat's the accepted way of parsing this such that in each case (or some combination of the two)
$v,$f, and$dwill all be set totrueand$outFilewill be equal to/fizz/someOtherFile?Inanc Gumus ,Apr 15, 2016 at 19:11
See my very easy and no-dependency answer here: stackoverflow.com/a/33826763/115363 – Inanc Gumus Apr 15 '16 at 19:11dezza ,Aug 2, 2016 at 2:13
For zsh-users there's a great builtin called zparseopts which can do:zparseopts -D -E -M -- d=debug -debug=dAnd have both-dand--debugin the$debugarrayecho $+debug[1]will return 0 or 1 if one of those are used. Ref: zsh.org/mla/users/2011/msg00350.html – dezza Aug 2 '16 at 2:13Bruno Bronosky ,Jan 7, 2013 at 20:01
Preferred Method: Using straight bash without getopt[s]I originally answered the question as the OP asked. This Q/A is getting a lot of attention, so I should also offer the non-magic way to do this. I'm going to expand upon guneysus's answer to fix the nasty sed and include Tobias Kienzler's suggestion .
Two of the most common ways to pass key value pair arguments are:
Straight Bash Space SeparatedUsage
./myscript.sh -e conf -s /etc -l /usr/lib /etc/hosts#!/bin/bash POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -e|--extension) EXTENSION="$2" shift # past argument shift # past value ;; -s|--searchpath) SEARCHPATH="$2" shift # past argument shift # past value ;; -l|--lib) LIBPATH="$2" shift # past argument shift # past value ;; --default) DEFAULT=YES shift # past argument ;; *) # unknown option POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters echo FILE EXTENSION = "${EXTENSION}" echo SEARCH PATH = "${SEARCHPATH}" echo LIBRARY PATH = "${LIBPATH}" echo DEFAULT = "${DEFAULT}" echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l) if [[ -n $1 ]]; then echo "Last line of file specified as non-opt/last argument:" tail -1 "$1" fiStraight Bash Equals SeparatedUsage
./myscript.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts#!/bin/bash for i in "$@" do case $i in -e=*|--extension=*) EXTENSION="${i#*=}" shift # past argument=value ;; -s=*|--searchpath=*) SEARCHPATH="${i#*=}" shift # past argument=value ;; -l=*|--lib=*) LIBPATH="${i#*=}" shift # past argument=value ;; --default) DEFAULT=YES shift # past argument with no value ;; *) # unknown option ;; esac done echo "FILE EXTENSION = ${EXTENSION}" echo "SEARCH PATH = ${SEARCHPATH}" echo "LIBRARY PATH = ${LIBPATH}" echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l) if [[ -n $1 ]]; then echo "Last line of file specified as non-opt/last argument:" tail -1 $1 fiTo better understand
Using getopt[s]${i#*=}search for "Substring Removal" in this guide . It is functionally equivalent to`sed 's/[^=]*=//' <<< "$i"`which calls a needless subprocess or`echo "$i" | sed 's/[^=]*=//'`which calls two needless subprocesses.from: http://mywiki.wooledge.org/BashFAQ/035#getopts
Never use getopt(1).
getoptcannot handle empty arguments strings, or arguments with embedded whitespace. Please forget that it ever existed.The POSIX shell (and others) offer
getoptswhich is safe to use instead. Here is a simplisticgetoptsexample:#!/bin/sh # A POSIX variable OPTIND=1 # Reset in case getopts has been used previously in the shell. # Initialize our own variables: output_file="" verbose=0 while getopts "h?vf:" opt; do case "$opt" in h|\?) show_help exit 0 ;; v) verbose=1 ;; f) output_file=$OPTARG ;; esac done shift $((OPTIND-1)) [ "${1:-}" = "--" ] && shift echo "verbose=$verbose, output_file='$output_file', Leftovers: $@" # End of fileThe advantages of
getoptsare:
- It's portable, and will work in e.g. dash.
- It can handle things like
-vf filenamein the expected Unix way, automatically.The disadvantage of
getoptsis that it can only handle short options (-h, not--help) without trickery.There is a getopts tutorial which explains what all of the syntax and variables mean. In bash, there is also
help getopts, which might be informative.Livven ,Jun 6, 2013 at 21:19
Is this really true? According to Wikipedia there's a newer GNU enhanced version ofgetoptwhich includes all the functionality ofgetoptsand then some.man getopton Ubuntu 13.04 outputsgetopt - parse command options (enhanced)as the name, so I presume this enhanced version is standard now. – Livven Jun 6 '13 at 21:19szablica ,Jul 17, 2013 at 15:23
That something is a certain way on your system is a very weak premise to base asumptions of "being standard" on. – szablica Jul 17 '13 at 15:23Stephane Chazelas ,Aug 20, 2014 at 19:55
@Livven, thatgetoptis not a GNU utility, it's part ofutil-linux. – Stephane Chazelas Aug 20 '14 at 19:55Nicolas Mongrain-Lacombe ,Jun 19, 2016 at 21:22
If you use-gt 0, remove yourshiftafter theesac, augment all theshiftby 1 and add this case:*) break;;you can handle non optionnal arguments. Ex: pastebin.com/6DJ57HTc – Nicolas Mongrain-Lacombe Jun 19 '16 at 21:22kolydart ,Jul 10, 2017 at 8:11
You do not echo–default. In the first example, I notice that if–defaultis the last argument, it is not processed (considered as non-opt), unlesswhile [[ $# -gt 1 ]]is set aswhile [[ $# -gt 0 ]]– kolydart Jul 10 '17 at 8:11Robert Siemer ,Apr 20, 2015 at 17:47
No answer mentions enhanced getopt . And the top-voted answer is misleading: It ignores-vfdstyle short options (requested by the OP), options after positional arguments (also requested by the OP) and it ignores parsing-errors. Instead:
- Use enhanced
getoptfrom util-linux or formerly GNU glibc . 1- It works with
getopt_long()the C function of GNU glibc.- Has all useful distinguishing features (the others don't have them):
- handles spaces, quoting characters and even binary in arguments 2
- it can handle options at the end:
script.sh -o outFile file1 file2 -v- allows
=-style long options:script.sh --outfile=fileOut --infile fileIn- Is so old already 3 that no GNU system is missing this (e.g. any Linux has it).
- You can test for its existence with:
getopt --test→ return value 4.- Other
getoptor shell-builtingetoptsare of limited use.The following calls
myscript -vfd ./foo/bar/someFile -o /fizz/someOtherFile myscript -v -f -d -o/fizz/someOtherFile -- ./foo/bar/someFile myscript --verbose --force --debug ./foo/bar/someFile -o/fizz/someOtherFile myscript --output=/fizz/someOtherFile ./foo/bar/someFile -vfd myscript ./foo/bar/someFile -df -v --output /fizz/someOtherFileall return
verbose: y, force: y, debug: y, in: ./foo/bar/someFile, out: /fizz/someOtherFilewith the following
myscript#!/bin/bash getopt --test > /dev/null if [[ $? -ne 4 ]]; then echo "I'm sorry, `getopt --test` failed in this environment." exit 1 fi OPTIONS=dfo:v LONGOPTIONS=debug,force,output:,verbose # -temporarily store output to be able to check for errors # -e.g. use "--options" parameter by name to activate quoting/enhanced mode # -pass arguments only via -- "$@" to separate them correctly PARSED=$(getopt --options=$OPTIONS --longoptions=$LONGOPTIONS --name "$0" -- "$@") if [[ $? -ne 0 ]]; then # e.g. $? == 1 # then getopt has complained about wrong arguments to stdout exit 2 fi # read getopt's output this way to handle the quoting right: eval set -- "$PARSED" # now enjoy the options in order and nicely split until we see -- while true; do case "$1" in -d|--debug) d=y shift ;; -f|--force) f=y shift ;; -v|--verbose) v=y shift ;; -o|--output) outFile="$2" shift 2 ;; --) shift break ;; *) echo "Programming error" exit 3 ;; esac done # handle non-option arguments if [[ $# -ne 1 ]]; then echo "$0: A single input file is required." exit 4 fi echo "verbose: $v, force: $f, debug: $d, in: $1, out: $outFile"
1 enhanced getopt is available on most "bash-systems", including Cygwin; on OS X try brew install gnu-getopt
2 the POSIXexec()conventions have no reliable way to pass binary NULL in command line arguments; those bytes prematurely end the argument
3 first version released in 1997 or before (I only tracked it back to 1997)johncip ,Jan 12, 2017 at 2:00
Thanks for this. Just confirmed from the feature table at en.wikipedia.org/wiki/Getopts , if you need support for long options, and you're not on Solaris,getoptis the way to go. – johncip Jan 12 '17 at 2:00Kaushal Modi ,Apr 27, 2017 at 14:02
I believe that the only caveat withgetoptis that it cannot be used conveniently in wrapper scripts where one might have few options specific to the wrapper script, and then pass the non-wrapper-script options to the wrapped executable, intact. Let's say I have agrepwrapper calledmygrepand I have an option--foospecific tomygrep, then I cannot domygrep --foo -A 2, and have the-A 2passed automatically togrep; I need to domygrep --foo -- -A 2. Here is my implementation on top of your solution. – Kaushal Modi Apr 27 '17 at 14:02bobpaul ,Mar 20 at 16:45
Alex, I agree and there's really no way around that since we need to know the actual return value ofgetopt --test. I'm a big fan of "Unofficial Bash Strict mode", (which includesset -e), and I just put the check for getopt ABOVEset -euo pipefailandIFS=$'\n\t'in my script. – bobpaul Mar 20 at 16:45Robert Siemer ,Mar 21 at 9:10
@bobpaul Oh, there is a way around that. And I'll edit my answer soon to reflect my collections regarding this issue (set -e)... – Robert Siemer Mar 21 at 9:10Robert Siemer ,Mar 21 at 9:16
@bobpaul Your statement about util-linux is wrong and misleading as well: the package is marked "essential" on Ubuntu/Debian. As such, it is always installed. – Which distros are you talking about (where you say it needs to be installed on purpose)? – Robert Siemer Mar 21 at 9:16guneysus ,Nov 13, 2012 at 10:31
from :digitalpeer.comwith minor modificationsUsage
myscript.sh -p=my_prefix -s=dirname -l=libname#!/bin/bash for i in "$@" do case $i in -p=*|--prefix=*) PREFIX="${i#*=}" ;; -s=*|--searchpath=*) SEARCHPATH="${i#*=}" ;; -l=*|--lib=*) DIR="${i#*=}" ;; --default) DEFAULT=YES ;; *) # unknown option ;; esac done echo PREFIX = ${PREFIX} echo SEARCH PATH = ${SEARCHPATH} echo DIRS = ${DIR} echo DEFAULT = ${DEFAULT}To better understand
${i#*=}search for "Substring Removal" in this guide . It is functionally equivalent to`sed 's/[^=]*=//' <<< "$i"`which calls a needless subprocess or`echo "$i" | sed 's/[^=]*=//'`which calls two needless subprocesses.Tobias Kienzler ,Nov 12, 2013 at 12:48
Neat! Though this won't work for space-separated arguments à lamount -t tempfs .... One can probably fix this via something likewhile [ $# -ge 1 ]; do param=$1; shift; case $param in; -p) prefix=$1; shift;;etc – Tobias Kienzler Nov 12 '13 at 12:48Robert Siemer ,Mar 19, 2016 at 15:23
This can't handle-vfdstyle combined short options. – Robert Siemer Mar 19 '16 at 15:23bekur ,Dec 19, 2017 at 23:27
link is broken! – bekur Dec 19 '17 at 23:27Matt J ,Oct 10, 2008 at 17:03
getopt()/getopts()is a good option. Stolen from here :The simple use of "getopt" is shown in this mini-script:
#!/bin/bash echo "Before getopt" for i do echo $i done args=`getopt abc:d $*` set -- $args echo "After getopt" for i do echo "-->$i" doneWhat we have said is that any of -a, -b, -c or -d will be allowed, but that -c is followed by an argument (the "c:" says that).
If we call this "g" and try it out:
bash-2.05a$ ./g -abc foo Before getopt -abc foo After getopt -->-a -->-b -->-c -->foo -->--We start with two arguments, and "getopt" breaks apart the options and puts each in its own argument. It also added "--".
Robert Siemer ,Apr 16, 2016 at 14:37
Using$*is broken usage ofgetopt. (It hoses arguments with spaces.) See my answer for proper usage. – Robert Siemer Apr 16 '16 at 14:37SDsolar ,Aug 10, 2017 at 14:07
Why would you want to make it more complicated? – SDsolar Aug 10 '17 at 14:07thebunnyrules ,Jun 1 at 1:57
@Matt J, the first part of the script (for i) would be able to handle arguments with spaces in them if you use "$i" instead of $i. The getopts does not seem to be able to handle arguments with spaces. What would be the advantage of using getopt over the for i loop? – thebunnyrules Jun 1 at 1:57bronson ,Jul 15, 2015 at 23:43
At the risk of adding another example to ignore, here's my scheme.
- handles
-n argand--name=arg- allows arguments at the end
- shows sane errors if anything is misspelled
- compatible, doesn't use bashisms
- readable, doesn't require maintaining state in a loop
Hope it's useful to someone.
while [ "$#" -gt 0 ]; do case "$1" in -n) name="$2"; shift 2;; -p) pidfile="$2"; shift 2;; -l) logfile="$2"; shift 2;; --name=*) name="${1#*=}"; shift 1;; --pidfile=*) pidfile="${1#*=}"; shift 1;; --logfile=*) logfile="${1#*=}"; shift 1;; --name|--pidfile|--logfile) echo "$1 requires an argument" >&2; exit 1;; -*) echo "unknown option: $1" >&2; exit 1;; *) handle_argument "$1"; shift 1;; esac donerhombidodecahedron ,Sep 11, 2015 at 8:40
What is the "handle_argument" function? – rhombidodecahedron Sep 11 '15 at 8:40bronson ,Oct 8, 2015 at 20:41
Sorry for the delay. In my script, the handle_argument function receives all the non-option arguments. You can replace that line with whatever you'd like, maybe*) die "unrecognized argument: $1"or collect the args into a variable*) args+="$1"; shift 1;;. – bronson Oct 8 '15 at 20:41Guilherme Garnier ,Apr 13 at 16:10
Amazing! I've tested a couple of answers, but this is the only one that worked for all cases, including many positional parameters (both before and after flags) – Guilherme Garnier Apr 13 at 16:10Shane Day ,Jul 1, 2014 at 1:20
I'm about 4 years late to this question, but want to give back. I used the earlier answers as a starting point to tidy up my old adhoc param parsing. I then refactored out the following template code. It handles both long and short params, using = or space separated arguments, as well as multiple short params grouped together. Finally it re-inserts any non-param arguments back into the $1,$2.. variables. I hope it's useful.#!/usr/bin/env bash # NOTICE: Uncomment if your script depends on bashisms. #if [ -z "$BASH_VERSION" ]; then bash $0 $@ ; exit $? ; fi echo "Before" for i ; do echo - $i ; done # Code template for parsing command line parameters using only portable shell # code, while handling both long and short params, handling '-f file' and # '-f=file' style param data and also capturing non-parameters to be inserted # back into the shell positional parameters. while [ -n "$1" ]; do # Copy so we can modify it (can't modify $1) OPT="$1" # Detect argument termination if [ x"$OPT" = x"--" ]; then shift for OPT ; do REMAINS="$REMAINS \"$OPT\"" done break fi # Parse current opt while [ x"$OPT" != x"-" ] ; do case "$OPT" in # Handle --flag=value opts like this -c=* | --config=* ) CONFIGFILE="${OPT#*=}" shift ;; # and --flag value opts like this -c* | --config ) CONFIGFILE="$2" shift ;; -f* | --force ) FORCE=true ;; -r* | --retry ) RETRY=true ;; # Anything unknown is recorded for later * ) REMAINS="$REMAINS \"$OPT\"" break ;; esac # Check for multiple short options # NOTICE: be sure to update this pattern to match valid options NEXTOPT="${OPT#-[cfr]}" # try removing single short opt if [ x"$OPT" != x"$NEXTOPT" ] ; then OPT="-$NEXTOPT" # multiple short opts, keep going else break # long form, exit inner loop fi done # Done with that param. move to next shift done # Set the non-parameters back into the positional parameters ($1 $2 ..) eval set -- $REMAINS echo -e "After: \n configfile='$CONFIGFILE' \n force='$FORCE' \n retry='$RETRY' \n remains='$REMAINS'" for i ; do echo - $i ; doneRobert Siemer ,Dec 6, 2015 at 13:47
This code can't handle options with arguments like this:-c1. And the use of=to separate short options from their arguments is unusual... – Robert Siemer Dec 6 '15 at 13:47sfnd ,Jun 6, 2016 at 19:28
I ran into two problems with this useful chunk of code: 1) the "shift" in the case of "-c=foo" ends up eating the next parameter; and 2) 'c' should not be included in the "[cfr]" pattern for combinable short options. – sfnd Jun 6 '16 at 19:28Inanc Gumus ,Nov 20, 2015 at 12:28
More succinct wayscript.sh
#!/bin/bash while [[ "$#" > 0 ]]; do case $1 in -d|--deploy) deploy="$2"; shift;; -u|--uglify) uglify=1;; *) echo "Unknown parameter passed: $1"; exit 1;; esac; shift; done echo "Should deploy? $deploy" echo "Should uglify? $uglify"Usage:
./script.sh -d dev -u # OR: ./script.sh --deploy dev --uglifyhfossli ,Apr 7 at 20:58
This is what I am doing. Have towhile [[ "$#" > 1 ]]if I want to support ending the line with a boolean flag./script.sh --debug dev --uglify fast --verbose. Example: gist.github.com/hfossli/4368aa5a577742c3c9f9266ed214aa58 – hfossli Apr 7 at 20:58hfossli ,Apr 7 at 21:09
I sent an edit request. I just tested this and it works perfectly. – hfossli Apr 7 at 21:09hfossli ,Apr 7 at 21:10
Wow! Simple and clean! This is how I'm using this: gist.github.com/hfossli/4368aa5a577742c3c9f9266ed214aa58 – hfossli Apr 7 at 21:10Ponyboy47 ,Sep 8, 2016 at 18:59
My answer is largely based on the answer by Bruno Bronosky , but I sort of mashed his two pure bash implementations into one that I use pretty frequently.# As long as there is at least one more argument, keep looping while [[ $# -gt 0 ]]; do key="$1" case "$key" in # This is a flag type option. Will catch either -f or --foo -f|--foo) FOO=1 ;; # Also a flag type option. Will catch either -b or --bar -b|--bar) BAR=1 ;; # This is an arg value type option. Will catch -o value or --output-file value -o|--output-file) shift # past the key and to the value OUTPUTFILE="$1" ;; # This is an arg=value type option. Will catch -o=value or --output-file=value -o=*|--output-file=*) # No need to shift here since the value is part of the same string OUTPUTFILE="${key#*=}" ;; *) # Do whatever you want with extra options echo "Unknown option '$key'" ;; esac # Shift after checking all the cases to get the next option shift doneThis allows you to have both space separated options/values, as well as equal defined values.
So you could run your script using:
./myscript --foo -b -o /fizz/file.txtas well as:
./myscript -f --bar -o=/fizz/file.txtand both should have the same end result.
PROS:
- Allows for both -arg=value and -arg value
- Works with any arg name that you can use in bash
- Meaning -a or -arg or --arg or -a-r-g or whatever
- Pure bash. No need to learn/use getopt or getopts
CONS:
- Can't combine args
- Meaning no -abc. You must do -a -b -c
These are the only pros/cons I can think of off the top of my head
bubla ,Jul 10, 2016 at 22:40
I have found the matter to write portable parsing in scripts so frustrating that I have written Argbash - a FOSS code generator that can generate the arguments-parsing code for your script plus it has some nice features:RichVel ,Aug 18, 2016 at 5:34
Thanks for writing argbash, I just used it and found it works well. I mostly went for argbash because it's a code generator supporting the older bash 3.x found on OS X 10.11 El Capitan. The only downside is that the code-generator approach means quite a lot of code in your main script, compared to calling a module. – RichVel Aug 18 '16 at 5:34bubla ,Aug 23, 2016 at 20:40
You can actually use Argbash in a way that it produces tailor-made parsing library just for you that you can have included in your script or you can have it in a separate file and just source it. I have added an example to demonstrate that and I have made it more explicit in the documentation, too. – bubla Aug 23 '16 at 20:40RichVel ,Aug 24, 2016 at 5:47
Good to know. That example is interesting but still not really clear - maybe you can change name of the generated script to 'parse_lib.sh' or similar and show where the main script calls it (like in the wrapping script section which is more complex use case). – RichVel Aug 24 '16 at 5:47bubla ,Dec 2, 2016 at 20:12
The issues were addressed in recent version of argbash: Documentation has been improved, a quickstart argbash-init script has been introduced and you can even use argbash online at argbash.io/generate – bubla Dec 2 '16 at 20:12Alek ,Mar 1, 2012 at 15:15
I think this one is simple enough to use:#!/bin/bash # readopt='getopts $opts opt;rc=$?;[ $rc$opt == 0? ]&&exit 1;[ $rc == 0 ]||{ shift $[OPTIND-1];false; }' opts=vfdo: # Enumerating options while eval $readopt do echo OPT:$opt ${OPTARG+OPTARG:$OPTARG} done # Enumerating arguments for arg do echo ARG:$arg doneInvocation example:
./myscript -v -do /fizz/someOtherFile -f ./foo/bar/someFile OPT:v OPT:d OPT:o OPTARG:/fizz/someOtherFile OPT:f ARG:./foo/bar/someFileerm3nda ,May 20, 2015 at 22:50
I read all and this one is my preferred one. I don't like to use-a=1as argc style. I prefer to put first the main option -options and later the special ones with single spacing-o option. Im looking for the simplest-vs-better way to read argvs. – erm3nda May 20 '15 at 22:50erm3nda ,May 20, 2015 at 23:25
It's working really well but if you pass an argument to a non a: option all the following options would be taken as arguments. You can check this line./myscript -v -d fail -o /fizz/someOtherFile -f ./foo/bar/someFilewith your own script. -d option is not set as d: – erm3nda May 20 '15 at 23:25unsynchronized ,Jun 9, 2014 at 13:46
Expanding on the excellent answer by @guneysus, here is a tweak that lets user use whichever syntax they prefer, egcommand -x=myfilename.ext --another_switchvs
command -x myfilename.ext --another_switchThat is to say the equals can be replaced with whitespace.
This "fuzzy interpretation" might not be to your liking, but if you are making scripts that are interchangeable with other utilities (as is the case with mine, which must work with ffmpeg), the flexibility is useful.
STD_IN=0 prefix="" key="" value="" for keyValue in "$@" do case "${prefix}${keyValue}" in -i=*|--input_filename=*) key="-i"; value="${keyValue#*=}";; -ss=*|--seek_from=*) key="-ss"; value="${keyValue#*=}";; -t=*|--play_seconds=*) key="-t"; value="${keyValue#*=}";; -|--stdin) key="-"; value=1;; *) value=$keyValue;; esac case $key in -i) MOVIE=$(resolveMovie "${value}"); prefix=""; key="";; -ss) SEEK_FROM="${value}"; prefix=""; key="";; -t) PLAY_SECONDS="${value}"; prefix=""; key="";; -) STD_IN=${value}; prefix=""; key="";; *) prefix="${keyValue}=";; esac donevangorra ,Feb 12, 2015 at 21:50
getopts works great if #1 you have it installed and #2 you intend to run it on the same platform. OSX and Linux (for example) behave differently in this respect.Here is a (non getopts) solution that supports equals, non-equals, and boolean flags. For example you could run your script in this way:
./script --arg1=value1 --arg2 value2 --shouldClean # parse the arguments. COUNTER=0 ARGS=("$@") while [ $COUNTER -lt $# ] do arg=${ARGS[$COUNTER]} let COUNTER=COUNTER+1 nextArg=${ARGS[$COUNTER]} if [[ $skipNext -eq 1 ]]; then echo "Skipping" skipNext=0 continue fi argKey="" argVal="" if [[ "$arg" =~ ^\- ]]; then # if the format is: -key=value if [[ "$arg" =~ \= ]]; then argVal=$(echo "$arg" | cut -d'=' -f2) argKey=$(echo "$arg" | cut -d'=' -f1) skipNext=0 # if the format is: -key value elif [[ ! "$nextArg" =~ ^\- ]]; then argKey="$arg" argVal="$nextArg" skipNext=1 # if the format is: -key (a boolean flag) elif [[ "$nextArg" =~ ^\- ]] || [[ -z "$nextArg" ]]; then argKey="$arg" argVal="" skipNext=0 fi # if the format has not flag, just a value. else argKey="" argVal="$arg" skipNext=0 fi case "$argKey" in --source-scmurl) SOURCE_URL="$argVal" ;; --dest-scmurl) DEST_URL="$argVal" ;; --version-num) VERSION_NUM="$argVal" ;; -c|--clean) CLEAN_BEFORE_START="1" ;; -h|--help|-help|--h) showUsage exit ;; esac doneakostadinov ,Jul 19, 2013 at 7:50
This is how I do in a function to avoid breaking getopts run at the same time somewhere higher in stack:function waitForWeb () { local OPTIND=1 OPTARG OPTION local host=localhost port=8080 proto=http while getopts "h:p:r:" OPTION; do case "$OPTION" in h) host="$OPTARG" ;; p) port="$OPTARG" ;; r) proto="$OPTARG" ;; esac done ... }Renato Silva ,Jul 4, 2016 at 16:47
EasyOptions does not require any parsing:## Options: ## --verbose, -v Verbose mode ## --output=FILE Output filename source easyoptions || exit if test -n "${verbose}"; then echo "output file is ${output}" echo "${arguments[@]}" fiOleksii Chekulaiev ,Jul 1, 2016 at 20:56
I give you The Functionparse_paramsthat will parse params:
- Without polluting global scope.
- Effortlessly returns to you ready to use variables so that you could build further logic on them
- Amount of dashes before params does not matter (
--allequals-allequalsall=all)The script below is a copy-paste working demonstration. See
show_usefunction to understand how to useparse_params.Limitations:
- Does not support space delimited params (
-d 1)- Param names will lose dashes so
--any-paramand-anyparamare equivalenteval $(parse_params "$@")must be used inside bash function (it will not work in the global scope)
#!/bin/bash # Universal Bash parameter parsing # Parse equal sign separated params into named local variables # Standalone named parameter value will equal its param name (--force creates variable $force=="force") # Parses multi-valued named params into an array (--path=path1 --path=path2 creates ${path[*]} array) # Parses un-named params into ${ARGV[*]} array # Additionally puts all named params into ${ARGN[*]} array # Additionally puts all standalone "option" params into ${ARGO[*]} array # @author Oleksii Chekulaiev # @version v1.3 (May-14-2018) parse_params () { local existing_named local ARGV=() # un-named params local ARGN=() # named params local ARGO=() # options (--params) echo "local ARGV=(); local ARGN=(); local ARGO=();" while [[ "$1" != "" ]]; do # Escape asterisk to prevent bash asterisk expansion _escaped=${1/\*/\'\"*\"\'} # If equals delimited named parameter if [[ "$1" =~ ^..*=..* ]]; then # Add to named parameters array echo "ARGN+=('$_escaped');" # key is part before first = local _key=$(echo "$1" | cut -d = -f 1) # val is everything after key and = (protect from param==value error) local _val="${1/$_key=}" # remove dashes from key name _key=${_key//\-} # search for existing parameter name if (echo "$existing_named" | grep "\b$_key\b" >/dev/null); then # if name already exists then it's a multi-value named parameter # re-declare it as an array if needed if ! (declare -p _key 2> /dev/null | grep -q 'declare \-a'); then echo "$_key=(\"\$$_key\");" fi # append new value echo "$_key+=('$_val');" else # single-value named parameter echo "local $_key=\"$_val\";" existing_named=" $_key" fi # If standalone named parameter elif [[ "$1" =~ ^\-. ]]; then # Add to options array echo "ARGO+=('$_escaped');" # remove dashes local _key=${1//\-} echo "local $_key=\"$_key\";" # non-named parameter else # Escape asterisk to prevent bash asterisk expansion _escaped=${1/\*/\'\"*\"\'} echo "ARGV+=('$_escaped');" fi shift done } #--------------------------- DEMO OF THE USAGE ------------------------------- show_use () { eval $(parse_params "$@") # -- echo "${ARGV[0]}" # print first unnamed param echo "${ARGV[1]}" # print second unnamed param echo "${ARGN[0]}" # print first named param echo "${ARG0[0]}" # print first option param (--force) echo "$anyparam" # print --anyparam value echo "$k" # print k=5 value echo "${multivalue[0]}" # print first value of multi-value echo "${multivalue[1]}" # print second value of multi-value [[ "$force" == "force" ]] && echo "\$force is set so let the force be with you" } show_use "param 1" --anyparam="my value" param2 k=5 --force --multi-value=test1 --multi-value=test2Oleksii Chekulaiev ,Sep 28, 2016 at 12:55
To use the demo to parse params that come into your bash script you just doshow_use "$@"– Oleksii Chekulaiev Sep 28 '16 at 12:55Oleksii Chekulaiev ,Sep 28, 2016 at 12:58
Basically I found out that github.com/renatosilva/easyoptions does the same in the same way but is a bit more massive than this function. – Oleksii Chekulaiev Sep 28 '16 at 12:58galmok ,Jun 24, 2015 at 10:54
I'd like to offer my version of option parsing, that allows for the following:-s p1 --stage p1 -w somefolder --workfolder somefolder -sw p1 somefolder -e=helloAlso allows for this (could be unwanted):
-s--workfolder p1 somefolder -se=hello p1 -swe=hello p1 somefolderYou have to decide before use if = is to be used on an option or not. This is to keep the code clean(ish).
while [[ $# > 0 ]] do key="$1" while [[ ${key+x} ]] do case $key in -s*|--stage) STAGE="$2" shift # option has parameter ;; -w*|--workfolder) workfolder="$2" shift # option has parameter ;; -e=*) EXAMPLE="${key#*=}" break # option has been fully handled ;; *) # unknown option echo Unknown option: $key #1>&2 exit 10 # either this: my preferred way to handle unknown options break # or this: do this to signal the option has been handled (if exit isn't used) ;; esac # prepare for next option in this key, if any [[ "$key" = -? || "$key" == --* ]] && unset key || key="${key/#-?/-}" done shift # option(s) fully processed, proceed to next input argument doneLuca Davanzo ,Nov 14, 2016 at 17:56
what's the meaning for "+x" on ${key+x} ? – Luca Davanzo Nov 14 '16 at 17:56galmok ,Nov 15, 2016 at 9:10
It is a test to see if 'key' is present or not. Further down I unset key and this breaks the inner while loop. – galmok Nov 15 '16 at 9:10Mark Fox ,Apr 27, 2015 at 2:42
Mixing positional and flag-based arguments --param=arg (equals delimited)Freely mixing flags between positional arguments:
./script.sh dumbo 127.0.0.1 --environment=production -q -d ./script.sh dumbo --environment=production 127.0.0.1 --quiet -dcan be accomplished with a fairly concise approach:
# process flags pointer=1 while [[ $pointer -le $# ]]; do param=${!pointer} if [[ $param != "-"* ]]; then ((pointer++)) # not a parameter flag so advance pointer else case $param in # paramter-flags with arguments -e=*|--environment=*) environment="${param#*=}";; --another=*) another="${param#*=}";; # binary flags -q|--quiet) quiet=true;; -d) debug=true;; esac # splice out pointer frame from positional list [[ $pointer -gt 1 ]] \ && set -- ${@:1:((pointer - 1))} ${@:((pointer + 1)):$#} \ || set -- ${@:((pointer + 1)):$#}; fi done # positional remain node_name=$1 ip_address=$2--param arg (space delimited)It's usualy clearer to not mix
--flag=valueand--flag valuestyles../script.sh dumbo 127.0.0.1 --environment production -q -dThis is a little dicey to read, but is still valid
./script.sh dumbo --environment production 127.0.0.1 --quiet -dSource
# process flags pointer=1 while [[ $pointer -le $# ]]; do if [[ ${!pointer} != "-"* ]]; then ((pointer++)) # not a parameter flag so advance pointer else param=${!pointer} ((pointer_plus = pointer + 1)) slice_len=1 case $param in # paramter-flags with arguments -e|--environment) environment=${!pointer_plus}; ((slice_len++));; --another) another=${!pointer_plus}; ((slice_len++));; # binary flags -q|--quiet) quiet=true;; -d) debug=true;; esac # splice out pointer frame from positional list [[ $pointer -gt 1 ]] \ && set -- ${@:1:((pointer - 1))} ${@:((pointer + $slice_len)):$#} \ || set -- ${@:((pointer + $slice_len)):$#}; fi done # positional remain node_name=$1 ip_address=$2schily ,Oct 19, 2015 at 13:59
Note thatgetopt(1)was a short living mistake from AT&T.getopt was created in 1984 but already buried in 1986 because it was not really usable.
A proof for the fact that
getoptis very outdated is that thegetopt(1)man page still mentions"$*"instead of"$@", that was added to the Bourne Shell in 1986 together with thegetopts(1)shell builtin in order to deal with arguments with spaces inside.BTW: if you are interested in parsing long options in shell scripts, it may be of interest to know that the
getopt(3)implementation from libc (Solaris) andksh93both added a uniform long option implementation that supports long options as aliases for short options. This causesksh93and theBourne Shellto implement a uniform interface for long options viagetopts.An example for long options taken from the Bourne Shell man page:
getopts "f:(file)(input-file)o:(output-file)" OPTX "$@"shows how long option aliases may be used in both Bourne Shell and ksh93.
See the man page of a recent Bourne Shell:
http://schillix.sourceforge.net/man/man1/bosh.1.html
and the man page for getopt(3) from OpenSolaris:
http://schillix.sourceforge.net/man/man3c/getopt.3c.html
and last, the getopt(1) man page to verify the outdated $*:
Volodymyr M. Lisivka ,Jul 9, 2013 at 16:51
Use module "arguments" from bash-modulesExample:
#!/bin/bash . import.sh log arguments NAME="world" parse_arguments "-n|--name)NAME;S" -- "$@" || { error "Cannot parse command line." exit 1 } info "Hello, $NAME!"Mike Q ,Jun 14, 2014 at 18:01
This also might be useful to know, you can set a value and if someone provides input, override the default with that value..myscript.sh -f ./serverlist.txt or just ./myscript.sh (and it takes defaults)
#!/bin/bash # --- set the value, if there is inputs, override the defaults. HOME_FOLDER="${HOME}/owned_id_checker" SERVER_FILE_LIST="${HOME_FOLDER}/server_list.txt" while [[ $# > 1 ]] do key="$1" shift case $key in -i|--inputlist) SERVER_FILE_LIST="$1" shift ;; esac done echo "SERVER LIST = ${SERVER_FILE_LIST}"phk ,Oct 17, 2015 at 21:17
Another solution without getopt[s], POSIX, old Unix styleSimilar to the solution Bruno Bronosky posted this here is one without the usage of
getopt(s).Main differentiating feature of my solution is that it allows to have options concatenated together just like
Code with example optionstar -xzf foo.tar.gzis equal totar -x -z -f foo.tar.gz. And just like intar,psetc. the leading hyphen is optional for a block of short options (but this can be changed easily). Long options are supported as well (but when a block starts with one then two leading hyphens are required).#!/bin/sh echo echo "POSIX-compliant getopt(s)-free old-style-supporting option parser from phk@[se.unix]" echo print_usage() { echo "Usage: $0 {a|b|c} [ARG...] Options: --aaa-0-args -a Option without arguments. --bbb-1-args ARG -b ARG Option with one argument. --ccc-2-args ARG1 ARG2 -c ARG1 ARG2 Option with two arguments. " >&2 } if [ $# -le 0 ]; then print_usage exit 1 fi opt= while :; do if [ $# -le 0 ]; then # no parameters remaining -> end option parsing break elif [ ! "$opt" ]; then # we are at the beginning of a fresh block # remove optional leading hyphen and strip trailing whitespaces opt=$(echo "$1" | sed 's/^-\?\([a-zA-Z0-9\?-]*\)/\1/') fi # get the first character -> check whether long option first_chr=$(echo "$opt" | awk '{print substr($1, 1, 1)}') [ "$first_chr" = - ] && long_option=T || long_option=F # note to write the options here with a leading hyphen less # also do not forget to end short options with a star case $opt in -) # end of options shift break ;; a*|-aaa-0-args) echo "Option AAA activated!" ;; b*|-bbb-1-args) if [ "$2" ]; then echo "Option BBB with argument '$2' activated!" shift else echo "BBB parameters incomplete!" >&2 print_usage exit 1 fi ;; c*|-ccc-2-args) if [ "$2" ] && [ "$3" ]; then echo "Option CCC with arguments '$2' and '$3' activated!" shift 2 else echo "CCC parameters incomplete!" >&2 print_usage exit 1 fi ;; h*|\?*|-help) print_usage exit 0 ;; *) if [ "$long_option" = T ]; then opt=$(echo "$opt" | awk '{print substr($1, 2)}') else opt=$first_chr fi printf 'Error: Unknown option: "%s"\n' "$opt" >&2 print_usage exit 1 ;; esac if [ "$long_option" = T ]; then # if we had a long option then we are going to get a new block next shift opt= else # if we had a short option then just move to the next character opt=$(echo "$opt" | awk '{print substr($1, 2)}') # if block is now empty then shift to the next one [ "$opt" ] || shift fi done echo "Doing something..." exit 0For the example usage please see the examples further below.
Position of options with argumentsFor what its worth there the options with arguments don't be the last (only long options need to be). So while e.g. in
Multiple options with argumentstar(at least in some implementations) thefoptions needs to be last because the file name follows (tar xzf bar.tar.gzworks buttar xfz bar.tar.gzdoes not) this is not the case here (see the later examples).As another bonus the option parameters are consumed in the order of the options by the parameters with required options. Just look at the output of my script here with the command line
abc X Y Z(or-abc X Y Z):Option AAA activated! Option BBB with argument 'X' activated! Option CCC with arguments 'Y' and 'Z' activated!Long options concatenated as wellAlso you can also have long options in option block given that they occur last in the block. So the following command lines are all equivalent (including the order in which the options and its arguments are being processed):
-cba Z Y Xcba Z Y X-cb-aaa-0-args Z Y X-c-bbb-1-args Z Y X -a--ccc-2-args Z Y -ba Xc Z Y b X a-c Z Y -b X -a--ccc-2-args Z Y --bbb-1-args X --aaa-0-argsAll of these lead to:
Option CCC with arguments 'Z' and 'Y' activated! Option BBB with argument 'X' activated! Option AAA activated! Doing something...Not in this solution Optional argumentsOptions with optional arguments should be possible with a bit of work, e.g. by looking forward whether there is a block without a hyphen; the user would then need to put a hyphen in front of every block following a block with a parameter having an optional parameter. Maybe this is too complicated to communicate to the user so better just require a leading hyphen altogether in this case.
Things get even more complicated with multiple possible parameters. I would advise against making the options trying to be smart by determining whether the an argument might be for it or not (e.g. with an option just takes a number as an optional argument) because this might break in the future.
I personally favor additional options instead of optional arguments.
Option arguments introduced with an equal signJust like with optional arguments I am not a fan of this (BTW, is there a thread for discussing the pros/cons of different parameter styles?) but if you want this you could probably implement it yourself just like done at http://mywiki.wooledge.org/BashFAQ/035#Manual_loop with a
Other notes--long-with-arg=?*case statement and then stripping the equal sign (this is BTW the site that says that making parameter concatenation is possible with some effort but "left [it] as an exercise for the reader" which made me take them at their word but I started from scratch).POSIX-compliant, works even on ancient Busybox setups I had to deal with (with e.g.
cut,headandgetoptsmissing).Noah ,Aug 29, 2016 at 3:44
Solution that preserves unhandled arguments. Demos Included.Here is my solution. It is VERY flexible and unlike others, shouldn't require external packages and handles leftover arguments cleanly.
Usage is:
./myscript -flag flagvariable -otherflag flagvar2All you have to do is edit the validflags line. It prepends a hyphen and searches all arguments. It then defines the next argument as the flag name e.g.
./myscript -flag flagvariable -otherflag flagvar2 echo $flag $otherflag flagvariable flagvar2The main code (short version, verbose with examples further down, also a version with erroring out):
#!/usr/bin/env bash #shebang.io validflags="rate time number" count=1 for arg in $@ do match=0 argval=$1 for flag in $validflags do sflag="-"$flag if [ "$argval" == "$sflag" ] then declare $flag=$2 match=1 fi done if [ "$match" == "1" ] then shift 2 else leftovers=$(echo $leftovers $argval) shift fi count=$(($count+1)) done #Cleanup then restore the leftovers shift $# set -- $leftoversThe verbose version with built in echo demos:
#!/usr/bin/env bash #shebang.io rate=30 time=30 number=30 echo "all args $@" validflags="rate time number" count=1 for arg in $@ do match=0 argval=$1 # argval=$(echo $@ | cut -d ' ' -f$count) for flag in $validflags do sflag="-"$flag if [ "$argval" == "$sflag" ] then declare $flag=$2 match=1 fi done if [ "$match" == "1" ] then shift 2 else leftovers=$(echo $leftovers $argval) shift fi count=$(($count+1)) done #Cleanup then restore the leftovers echo "pre final clear args: $@" shift $# echo "post final clear args: $@" set -- $leftovers echo "all post set args: $@" echo arg1: $1 arg2: $2 echo leftovers: $leftovers echo rate $rate time $time number $numberFinal one, this one errors out if an invalid -argument is passed through.
#!/usr/bin/env bash #shebang.io rate=30 time=30 number=30 validflags="rate time number" count=1 for arg in $@ do argval=$1 match=0 if [ "${argval:0:1}" == "-" ] then for flag in $validflags do sflag="-"$flag if [ "$argval" == "$sflag" ] then declare $flag=$2 match=1 fi done if [ "$match" == "0" ] then echo "Bad argument: $argval" exit 1 fi shift 2 else leftovers=$(echo $leftovers $argval) shift fi count=$(($count+1)) done #Cleanup then restore the leftovers shift $# set -- $leftovers echo rate $rate time $time number $number echo leftovers: $leftoversPros: What it does, it handles very well. It preserves unused arguments which a lot of the other solutions here don't. It also allows for variables to be called without being defined by hand in the script. It also allows prepopulation of variables if no corresponding argument is given. (See verbose example).
Cons: Can't parse a single complex arg string e.g. -xcvf would process as a single argument. You could somewhat easily write additional code into mine that adds this functionality though.
Daniel Bigham ,Aug 8, 2016 at 12:42
The top answer to this question seemed a bit buggy when I tried it -- here's my solution which I've found to be more robust:boolean_arg="" arg_with_value="" while [[ $# -gt 0 ]] do key="$1" case $key in -b|--boolean-arg) boolean_arg=true shift ;; -a|--arg-with-value) arg_with_value="$2" shift shift ;; -*) echo "Unknown option: $1" exit 1 ;; *) arg_num=$(( $arg_num + 1 )) case $arg_num in 1) first_normal_arg="$1" shift ;; 2) second_normal_arg="$1" shift ;; *) bad_args=TRUE esac ;; esac done # Handy to have this here when adding arguments to # see if they're working. Just edit the '0' to be '1'. if [[ 0 == 1 ]]; then echo "first_normal_arg: $first_normal_arg" echo "second_normal_arg: $second_normal_arg" echo "boolean_arg: $boolean_arg" echo "arg_with_value: $arg_with_value" exit 0 fi if [[ $bad_args == TRUE || $arg_num < 2 ]]; then echo "Usage: $(basename "$0") <first-normal-arg> <second-normal-arg> [--boolean-arg] [--arg-with-value VALUE]" exit 1 fiphyatt ,Sep 7, 2016 at 18:25
This example shows how to usegetoptandevalandHEREDOCandshiftto handle short and long parameters with and without a required value that follows. Also the switch/case statement is concise and easy to follow.#!/usr/bin/env bash # usage function function usage() { cat << HEREDOC Usage: $progname [--num NUM] [--time TIME_STR] [--verbose] [--dry-run] optional arguments: -h, --help show this help message and exit -n, --num NUM pass in a number -t, --time TIME_STR pass in a time string -v, --verbose increase the verbosity of the bash script --dry-run do a dry run, don't change any files HEREDOC } # initialize variables progname=$(basename $0) verbose=0 dryrun=0 num_str= time_str= # use getopt and store the output into $OPTS # note the use of -o for the short options, --long for the long name options # and a : for any option that takes a parameter OPTS=$(getopt -o "hn:t:v" --long "help,num:,time:,verbose,dry-run" -n "$progname" -- "$@") if [ $? != 0 ] ; then echo "Error in command line arguments." >&2 ; usage; exit 1 ; fi eval set -- "$OPTS" while true; do # uncomment the next line to see how shift is working # echo "\$1:\"$1\" \$2:\"$2\"" case "$1" in -h | --help ) usage; exit; ;; -n | --num ) num_str="$2"; shift 2 ;; -t | --time ) time_str="$2"; shift 2 ;; --dry-run ) dryrun=1; shift ;; -v | --verbose ) verbose=$((verbose + 1)); shift ;; -- ) shift; break ;; * ) break ;; esac done if (( $verbose > 0 )); then # print out all the parameters we read in cat <<-EOM num=$num_str time=$time_str verbose=$verbose dryrun=$dryrun EOM fi # The rest of your script belowThe most significant lines of the script above are these:
OPTS=$(getopt -o "hn:t:v" --long "help,num:,time:,verbose,dry-run" -n "$progname" -- "$@") if [ $? != 0 ] ; then echo "Error in command line arguments." >&2 ; exit 1 ; fi eval set -- "$OPTS" while true; do case "$1" in -h | --help ) usage; exit; ;; -n | --num ) num_str="$2"; shift 2 ;; -t | --time ) time_str="$2"; shift 2 ;; --dry-run ) dryrun=1; shift ;; -v | --verbose ) verbose=$((verbose + 1)); shift ;; -- ) shift; break ;; * ) break ;; esac doneShort, to the point, readable, and handles just about everything (IMHO).
Hope that helps someone.
Emeric Verschuur ,Feb 20, 2017 at 21:30
I have write a bash helper to write a nice bash toolproject home: https://gitlab.mbedsys.org/mbedsys/bashopts
example:
#!/bin/bash -ei # load the library . bashopts.sh # Enable backtrace dusplay on error trap 'bashopts_exit_handle' ERR # Initialize the library bashopts_setup -n "$0" -d "This is myapp tool description displayed on help message" -s "$HOME/.config/myapprc" # Declare the options bashopts_declare -n first_name -l first -o f -d "First name" -t string -i -s -r bashopts_declare -n last_name -l last -o l -d "Last name" -t string -i -s -r bashopts_declare -n display_name -l display-name -t string -d "Display name" -e "\$first_name \$last_name" bashopts_declare -n age -l number -d "Age" -t number bashopts_declare -n email_list -t string -m add -l email -d "Email adress" # Parse arguments bashopts_parse_args "$@" # Process argument bashopts_process_argswill give help:
NAME: ./example.sh - This is myapp tool description displayed on help message USAGE: [options and commands] [-- [extra args]] OPTIONS: -h,--help Display this help -n,--non-interactive true Non interactive mode - [$bashopts_non_interactive] (type:boolean, default:false) -f,--first "John" First name - [$first_name] (type:string, default:"") -l,--last "Smith" Last name - [$last_name] (type:string, default:"") --display-name "John Smith" Display name - [$display_name] (type:string, default:"$first_name $last_name") --number 0 Age - [$age] (type:number, default:0) --email Email adress - [$email_list] (type:string, default:"")enjoy :)
Josh Wulf ,Jun 24, 2017 at 18:07
I get this on Mac OS X: ``` lib/bashopts.sh: line 138: declare: -A: invalid option declare: usage: declare [-afFirtx] [-p] [name[=value] ...] Error in lib/bashopts.sh:138. 'declare -x -A bashopts_optprop_name' exited with status 2 Call tree: 1: lib/controller.sh:4 source(...) Exiting with status 1 ``` – Josh Wulf Jun 24 '17 at 18:07Josh Wulf ,Jun 24, 2017 at 18:17
You need Bash version 4 to use this. On Mac, the default version is 3. You can use home brew to install bash 4. – Josh Wulf Jun 24 '17 at 18:17a_z ,Mar 15, 2017 at 13:24
Here is my approach - using regexp.
- no getopts
- it handles block of short parameters
-qwerty- it handles short parameters
-q -w -e- it handles long options
--qwerty- you can pass attribute to short or long option (if you are using block of short options, attribute is attached to the last option)
- you can use spaces or
=to provide attributes, but attribute matches until encountering hyphen+space "delimiter", so in--q=qwe tyqwe tyis one attribute- it handles mix of all above so
-o a -op attr ibute --option=att ribu te --op-tion attribute --option att-ributeis validscript:
#!/usr/bin/env sh help_menu() { echo "Usage: ${0##*/} [-h][-l FILENAME][-d] Options: -h, --help display this help and exit -l, --logfile=FILENAME filename -d, --debug enable debug " } parse_options() { case $opt in h|help) help_menu exit ;; l|logfile) logfile=${attr} ;; d|debug) debug=true ;; *) echo "Unknown option: ${opt}\nRun ${0##*/} -h for help.">&2 exit 1 esac } options=$@ until [ "$options" = "" ]; do if [[ $options =~ (^ *(--([a-zA-Z0-9-]+)|-([a-zA-Z0-9-]+))(( |=)(([\_\.\?\/\\a-zA-Z0-9]?[ -]?[\_\.\?a-zA-Z0-9]+)+))?(.*)|(.+)) ]]; then if [[ ${BASH_REMATCH[3]} ]]; then # for --option[=][attribute] or --option[=][attribute] opt=${BASH_REMATCH[3]} attr=${BASH_REMATCH[7]} options=${BASH_REMATCH[9]} elif [[ ${BASH_REMATCH[4]} ]]; then # for block options -qwert[=][attribute] or single short option -a[=][attribute] pile=${BASH_REMATCH[4]} while (( ${#pile} > 1 )); do opt=${pile:0:1} attr="" pile=${pile/${pile:0:1}/} parse_options done opt=$pile attr=${BASH_REMATCH[7]} options=${BASH_REMATCH[9]} else # leftovers that don't match opt=${BASH_REMATCH[10]} options="" fi parse_options fi donemauron85 ,Jun 21, 2017 at 6:03
Like this one. Maybe just add -e param to echo with new line. – mauron85 Jun 21 '17 at 6:03John ,Oct 10, 2017 at 22:49
Assume we create a shell script namedtest_args.shas follow#!/bin/sh until [ $# -eq 0 ] do name=${1:1}; shift; if [[ -z "$1" || $1 == -* ]] ; then eval "export $name=true"; else eval "export $name=$1"; shift; fi done echo "year=$year month=$month day=$day flag=$flag"After we run the following command:
sh test_args.sh -year 2017 -flag -month 12 -day 22The output would be:
year=2017 month=12 day=22 flag=trueWill Barnwell ,Oct 10, 2017 at 23:57
This takes the same approach as Noah's answer , but has less safety checks / safeguards. This allows us to write arbitrary arguments into the script's environment and I'm pretty sure your use of eval here may allow command injection. – Will Barnwell Oct 10 '17 at 23:57Masadow ,Oct 6, 2015 at 8:53
Here is my improved solution of Bruno Bronosky's answer using variable arrays.it lets you mix parameters position and give you a parameter array preserving the order without the options
#!/bin/bash echo $@ PARAMS=() SOFT=0 SKIP=() for i in "$@" do case $i in -n=*|--skip=*) SKIP+=("${i#*=}") ;; -s|--soft) SOFT=1 ;; *) # unknown option PARAMS+=("$i") ;; esac done echo "SKIP = ${SKIP[@]}" echo "SOFT = $SOFT" echo "Parameters:" echo ${PARAMS[@]}Will output for example:
$ ./test.sh parameter -s somefile --skip=.c --skip=.obj parameter -s somefile --skip=.c --skip=.obj SKIP = .c .obj SOFT = 1 Parameters: parameter somefileJason S ,Dec 3, 2017 at 1:01
You use shift on the known arguments and not on the unknown ones so your remaining$@will be all but the first two arguments (in the order they are passed in), which could lead to some mistakes if you try to use$@later. You don't need the shift for the = parameters, since you're not handling spaces and you're getting the value with the substring removal#*=– Jason S Dec 3 '17 at 1:01Masadow ,Dec 5, 2017 at 9:17
You're right, in fact, since I build a PARAMS variable, I don't need to use shift at all – Masadow Dec 5 '17 at 9:17
|
|
|
Jun 09, 2018 | opensource.com
Changing an executed command
historyalso allows you to rerun a command with different syntax. For example, if I wanted to change my previous commandhistory | grep dnftohistory | grep ssh, I can execute the following at the prompt:$ ^dnf^ssh^Removing history
historywill rerun the command, but replacednfwithssh, and execute it.There may come a time that you want to remove some or all the commands in your history file. If you want to delete a particular command, enter
history -d <line number>. To clear the entire contents of the history file, executehistory -c.The history file is stored in a file that you can modify, as well. Bash shell users will find it in their Home directory as
Next steps.bash_history.There are a number of other things that you can do with
history:
- Set the size of your history buffer to a certain number of commands
- Record the date and time for each line in history
- Prevent certain commands from being recorded in history
For more information about the
historycommand and other interesting things you can do with it, take a look at the GNU Bash Manual .
|
|
|
Jun 01, 2018 | opensource.com
... ... ...
Looping through arrays
Although in the examples above we used integer indices in our arrays, let's consider two occasions when that won't be the case: First, if we wanted the
Looping through array elements$i-th element of the array, where$iis a variable containing the index of interest, we can retrieve that element using:echo ${allThreads[$i]}. Second, to output all the elements of an array, we replace the numeric index with the@symbol (you can think of@as standing forall):echo ${allThreads[@]}.With that in mind, let's loop through
for t in ${allThreads[@]} ; do$allThreadsand launch the pipeline for each value of--threads:
. / pipeline --threads $t
doneLooping through array indices
Next, let's consider a slightly different approach. Rather than looping over array elements , we can loop over array indices :
for i in ${!allThreads[@]} ; do
. / pipeline --threads ${allThreads[$i]}
doneLet's break that down: As we saw above,
${allThreads[@]}represents all the elements in our array. Adding an exclamation mark to make it${!allThreads[@]}will return the list of all array indices (in our case 0 to 7). In other words, theforloop is looping through all indices$iand reading the$i-th element from$allThreadsto set the value of the--threadsparameter.This is much harsher on the eyes, so you may be wondering why I bother introducing it in the first place. That's because there are times where you need to know both the index and the value within a loop, e.g., if you want to ignore the first element of an array, using indices saves you from creating an additional variable that you then increment inside the loop.
Populating arraysSo far, we've been able to launch the pipeline for each
Some useful syntax--threadsof interest. Now, let's assume the output to our pipeline is the runtime in seconds. We would like to capture that output at each iteration and save it in another array so we can do various manipulations with it at the end.But before diving into the code, we need to introduce some more syntax. First, we need to be able to retrieve the output of a Bash command. To do so, use the following syntax:
output=$( ./my_script.sh ), which will store the output of our commands into the variable$output.The second bit of syntax we need is how to append the value we just retrieved to an array. The syntax to do that will look familiar:
myArray+=( "newElement1" "newElement2" )The parameter sweepPutting everything together, here is our script for launching our parameter sweep:
allThreads = ( 1 2 4 8 16 32 64 128 )
allRuntimes = ()
for t in ${allThreads[@]} ; do
runtime =$ ( . / pipeline --threads $t )
allRuntimes+= ( $runtime )
doneAnd voilà!
What else you got?In this article, we covered the scenario of using arrays for parameter sweeps. But I promise there are more reasons to use Bash arrays -- here are two more examples.
Log alertingIn this scenario, your app is divided into modules, each with its own log file. We can write a cron job script to email the right person when there are signs of trouble in certain modules:
# List of logs and who should be notified of issues
logPaths = ( "api.log" "auth.log" "jenkins.log" "data.log" )
logEmails = ( "jay@email" "emma@email" "jon@email" "sophia@email" )# Look for signs of trouble in each log
for i in ${!logPaths[@]} ;
do
log = ${logPaths[$i]}
stakeholder = ${logEmails[$i]}
numErrors =$ ( tail -n 100 " $log " | grep "ERROR" | wc -l )# Warn stakeholders if recently saw > 5 errors
API queries
if [[ " $numErrors " -gt 5 ]] ;
then
emailRecipient = " $stakeholder "
emailSubject = "WARNING: ${log} showing unusual levels of errors"
emailBody = " ${numErrors} errors found in log ${log} "
echo " $emailBody " | mailx -s " $emailSubject " " $emailRecipient "
fi
doneSay you want to generate some analytics about which users comment the most on your Medium posts. Since we don't have direct database access, SQL is out of the question, but we can use APIs!
To avoid getting into a long discussion about API authentication and tokens, we'll instead use JSONPlaceholder , a public-facing API testing service, as our endpoint. Once we query each post and retrieve the emails of everyone who commented, we can append those emails to our results array:
endpoint = "https://jsonplaceholder.typicode.com/comments"
allEmails = ()# Query first 10 posts
for postId in { 1 .. 10 } ;
do
# Make API call to fetch emails of this posts's commenters
response =$ ( curl " ${endpoint} ?postId= ${postId} " )# Use jq to parse the JSON response into an array
allEmails+= ( $ ( jq '.[].email' <<< " $response " ) )
doneNote here that I'm using the
jqtool to parse JSON from the command line. The syntax ofjqis beyond the scope of this article, but I highly recommend you look into it.As you might imagine, there are countless other scenarios in which using Bash arrays can help, and I hope the examples outlined in this article have given you some food for thought. If you have other examples to share from your own work, please leave a comment below.
But wait, there's more!Since we covered quite a bit of array syntax in this article, here's a summary of what we covered, along with some more advanced tricks we did not cover:
One last thought
Syntax Result arr=()Create an empty array arr=(1 2 3)Initialize array ${arr[2]}Retrieve third element ${arr[@]}Retrieve all elements ${!arr[@]}Retrieve array indices ${#arr[@]}Calculate array size arr[0]=3Overwrite 1st element arr+=(4)Append value(s) str=$(ls)Save lsoutput as a stringarr=( $(ls) )Save lsoutput as an array of files${arr[@]:s:n}Retrieve elements at indices ntos+nAs we've discovered, Bash arrays sure have strange syntax, but I hope this article convinced you that they are extremely powerful. Once you get the hang of the syntax, you'll find yourself using Bash arrays quite often.
... ... ...
Robert Aboukhalil is a Bioinformatics Software Engineer. In his work, he develops cloud applications for the analysis and interactive visualization of genomics data. Robert holds a Ph.D. in Bioinformatics from Cold Spring Harbor Laboratory and a B.Eng. in Computer Engineering from McGill.
|
|
|
May 28, 2018 | www.tecmint.com
In this article, we will share a number of Bash command-line shortcuts useful for any Linux user. These shortcuts allow you to easily and in a fast manner, perform certain activities such as accessing and running previously executed commands, opening an editor, editing/deleting/changing text on the command line, moving the cursor, controlling processes etc. on the command line.
Although this article will mostly benefit Linux beginners getting their way around with command line basics, those with intermediate skills and advanced users might also find it practically helpful. We will group the bash keyboard shortcuts according to categories as follows.
Launch an EditorOpen a terminal and press
Ctrl+XandCtrl+Eto open an editor ( nano editor ) with an empty buffer. Bash will try to launch the editor defined by the $EDITOR environment variable.
Nano Editor Controlling The Screen
These shortcuts are used to control terminal screen output:
Move Cursor on The Command Line
Ctrl+L– clears the screen (same effect as the " clear " command).Ctrl+S– pause all command output to the screen. If you have executed a command that produces verbose, long output, use this to pause the output scrolling down the screen.Ctrl+Q– resume output to the screen after pausing it with Ctrl+S .The next shortcuts are used for moving the cursor within the command-line:
Search Through Bash History
Ctrl+AorHome– moves the cursor to the start of a line.Ctrl+EorEnd– moves the cursor to the end of the line.Ctrl+BorLeft Arrow– moves the cursor back one character at a time.Ctrl+ForRight Arrow– moves the cursor forward one character at a time.Ctrl+Left ArroworAlt+BorEscand thenB– moves the cursor back one word at a time.Ctrl+Right ArroworAlt+CorEscand thenF– moves the cursor forward one word at a time.The following shortcuts are used for searching for commands in the bash history:
Delete Text on the Command Line
Up arrow key– retrieves the previous command. If you press it constantly, it takes you through multiple commands in history, so you can find the one you want. Use the Down arrow to move in the reverse direction through the history.Ctrl+PandCtrl+N– alternatives for the Up and Down arrow keys, respectively.Ctrl+R– starts a reverse search, through the bash history, simply type characters that should be unique to the command you want to find in the history.Ctrl+S– launches a forward search, through the bash history.Ctrl+G– quits reverse or forward search, through the bash history.The following shortcuts are used for deleting text on the command line:
Transpose Text or Change Case on the Command Line
Ctrl+DorDelete– remove or deletes the character under the cursor.Ctrl+K– removes all text from the cursor to the end of the line.Ctrl+Xand thenBackspace– removes all the text from the cursor to the beginning of the line.These shortcuts will transpose or change the case of letters or words on the command line:
Working With Processes in Linux
Ctrl+T– transposes the character before the cursor with the character under the cursor.Escand thenT– transposes the two words immediately before (or under) the cursor.Escand thenU– transforms the text from the cursor to the end of the word to uppercase.Escand thenL– transforms the text from the cursor to the end of the word to lowercase.Escand thenC– changes the letter under the cursor (or the first letter of the next word) to uppercase, leaving the rest of the word unchanged.The following shortcuts help you to control running Linux processes.
Ctrl+Z– suspend the current foreground process. This sends the SIGTSTP signal to the process. You can get the process back to the foreground later using the fg process_name (or %bgprocess_number like %1 , %2 and so on) command.Ctrl+C– interrupt the current foreground process, by sending the SIGINT signal to it. The default behavior is to terminate a process gracefully, but the process can either honor or ignore it.Ctrl+D– exit the bash shell (same as running the exit command).Learn more about: All You Need To Know About Processes in Linux [Comprehensive Guide]
Bash Bang (!) CommandsIn the final part of this article, we will explain some useful
!(bang) operations:
!!– execute last command.!top– execute the most recent command that starts with 'top' (e.g. ! ).!top:p– displays the command that !top would run (also adds it as the latest command in the command history).!$– execute the last word of the previous command (same as Alt + ., e.g. if last command is ' cat tecmint.txt ', then !$ would try to run ' tecmint.txt ').!$:p– displays the word that !$ would execute.!*– displays the last word of the previous command.!*:p– displays the last word that !* would substitute.For more information, see the bash man page:
$ man bashThat's all for now! In this article, we shared some common and useful Bash command-line shortcuts and operations. Use the comment form below to make any additions or ask questions.
|
|
|
Apr 26, 2018 | opensource.com
Bash completion is a functionality through which Bash helps users type their commands more quickly and easily. It does this by presenting possible options when users press the Tab key while typing a command.
$ git < tab >< tab >
How it works
git git-receive-pack git-upload-archive
gitk git-shell git-upload-pack
$ git-s < tab >
$ git-shellMore Linux resources
The completion script is code that uses the builtin Bash command
- What is Linux?
- What are Linux containers?
- Download Now: Linux commands cheat sheet
- Advanced Linux commands cheat sheet
- Our latest Linux articles
completeto define which completion suggestions can be displayed for a given executable . The nature of the completion options vary, from simple static to highly sophisticated. Why bother?This functionality helps users by:
Hands-on
- saving them from typing text when it can be auto-completed
- helping them know the available continuations to their commands
- preventing errors and improving their experience by hiding or showing options based on what they have already typed
Here's what we will do in this tutorial:
We will first create a dummy executable script called
dothis. All it does is execute the command that resides on the number that was passed as an argument in the user's history. For example, the following command will simply execute thels -acommand, given that it exists in history with number235:dothis 235Then we will create a Bash completion script that will display commands along with their number from the user's history, and we will "bind" it to the
$ dothis < tab >< tab >dothisexecutable.
215 ls
216 ls -la
217 cd ~
218 man history
219 git status
220 history | cut -c 8 - bash_screen.png
You can see a gif demonstrating the functionality at this tutorial's code repository on GitHub .
Let the show begin.
Creating the executable scriptCreate a file named
if [ -z "$1" ] ; thendothisin your working directory and add the following code:
echo "No command number passed"
exit 2
fiexists =$ ( fc -l -1000 | grep ^ $1 -- 2 >/ dev / null )
if [ -n " $exists " ] ; then
fc -s -- "$1"
else
echo "Command with number $1 was not found in recent history"
exit 2
fiNotes:
- We first check if the script was called with an argument
- We then check if the specific number is included in the last 1000 commands
- if it exists, we execute the command using the
fcfunctionality- otherwise, we display an error message
Make the script executable with:
chmod +x ./dothisWe will execute this script many times in this tutorial, so I suggest you place it in a folder that is included in your path so that we can access it from anywhere by typing
dothis.I installed it in my home bin folder using:
install ./dothis ~/bin/dothisYou can do the same given that you have a
~/binfolder and it is included in yourPATHvariable.Check to see if it's working:
dothisYou should see this:
$ dothis
No command number passedDone.
Creating the completion scriptCreate a file named
dothis-completion.bash. From now on, we will refer to this file with the term completion script .Once we add some code to it, we will
sourceit to allow the completion to take effect. We mustsourcethis file every single time we change something in it .Later in this tutorial, we will discuss our options for registering this script whenever a Bash shell opens.
Static completionSuppose that the
dothisprogram supported a list of commands, for example:
nowtomorrowneverLet's use the
completecommand to register this list for completion. To use the proper terminology, we say we use thecompletecommand to define a completion specification ( compspec ) for our program.Add this to the completion script.
#/usr/bin/env bash
complete -W "now tomorrow never" dothisHere's what we specified with the
completecommand above:
- we used the
-W( wordlist ) option to provide a list of words for completion.- we defined to which "program" these completion words will be used (the
dothisparameter)Source the file:
source ./dothis-completion.bashNow try pressing Tab twice in the command line, as shown below:
$ dothis < tab >< tab >
never now tomorrowTry again after typing the
$ dothis n < tab >< tab >n:
never nowMagic! The completion options are automatically filtered to match only those starting with
n.Note: The options are not displayed in the order that we defined them in the word list; they are automatically sorted.
There are many other options to be used instead of the
-Wthat we used in this section. Most produce completions in a fixed manner, meaning that we don't intervene dynamically to filter their output.For example, if we want to have directory names as completion words for the
dothisprogram, we would change the complete command to the following:complete -A directory dothisPressing Tab after the
$ dothis < tab >< tab >dothisprogram would get us a list of the directories in the current directory from which we execute the script:
dir1 / dir2 / dir3 /Find the complete list of the available flags in the Bash Reference Manual .
Dynamic completionWe will be producing the completions of the
dothisexecutable with the following logic:
- If the user presses the Tab key right after the command, we will show the last 50 executed commands along with their numbers in history
- If the user presses the Tab key after typing a number that matches more than one command from history, we will show only those commands along with their numbers in history
- If the user presses the Tab key after a number that matches exactly one command in history, we auto-complete the number without appending the command's literal (if this is confusing, no worries -- you will understand later)
Let's start by defining a function that will execute each time the user requests completion on a
#/usr/bin/env bashdothiscommand. Change the completion script to this:
_dothis_completions ()
{
COMPREPLY+= ( "now" )
COMPREPLY+= ( "tomorrow" )
COMPREPLY+= ( "never" )
}complete -F _dothis_completions dothis
Note the following:
- we used the
-Fflag in the complete command defining that the_dothis_completionsis the function that will provide the completions of thedothisexecutableCOMPREPLYis an array variable used to store the completions -- the completion mechanism uses this variable to display its contents as completionsNow source the script and go for completion:
$ dothis < tab >< tab >
never now tomorrowPerfect. We produce the same completions as in the previous section with the word list. Or not? Try this:
$ dothis nev < tab >< tab >
never now tomorrowAs you can see, even though we type nev and then request for completion, the available options are always the same and nothing gets completed automatically. Why is this happening?
- The contents of the
COMPREPLYvariable are always displayed. The function is now responsible for adding/removing entries from there.- If the
COMPREPLYvariable had only one element, then that word would be automatically completed in the command. Since the current implementation always returns the same three words, this will not happen.Enter
compgen: a builtin command that generates completions supporting most of the options of thecompletecommand (ex.-Wfor word list,-dfor directories) and filtering them based on what the user has already typed.Don't worry if you feel confused; everything will become clear later.
Type the following in the console to better understand what
$ compgen -W "now tomorrow never"compgendoes:
now
tomorrow
never
$ compgen -W "now tomorrow never" n
now
never
$ compgen -W "now tomorrow never" t
tomorrowSo now we can use it, but we need to find a way to know what has been typed after the
dothiscommand. We already have the way: The Bash completion facilities provide Bash variables related to the completion taking place. Here are the more important ones:
COMP_WORDS: an array of all the words typed after the name of the program thecompspecbelongs toCOMP_CWORD: an index of theCOMP_WORDSarray pointing to the word the current cursor is at -- in other words, the index of the word the cursor was when the tab key was pressedCOMP_LINE: the current command lineTo access the word just after the
dothisword, we can use the value ofCOMP_WORDS[1]Change the completion script again:
#/usr/bin/env bash
_dothis_completions ()
{
COMPREPLY = ( $ ( compgen -W "now tomorrow never" " ${COMP_WORDS[1]} " ))
}complete -F _dothis_completions dothis
Source, and there you are:
$ dothis
never now tomorrow
$ dothis n
never nowNow, instead of the words now, tomorrow, never , we would like to see actual numbers from the command history.
The
fc -lcommand followed by a negative number-ndisplays the last n commands. So we will use:fc -l -50which lists the last 50 executed commands along with their numbers. The only manipulation we need to do is replace tabs with spaces to display them properly from the completion mechanism.
sedto the rescue.Change the completion script as follows:
#/usr/bin/env bash
_dothis_completions ()
{
COMPREPLY = ( $ ( compgen -W " $(fc -l -50 | sed 's/\t//') " -- " ${COMP_WORDS[1]} " ))
}complete -F _dothis_completions dothis
Source and test in the console:
$ dothis < tab >< tab >
632 source dothis-completion.bash 649 source dothis-completion.bash 666 cat ~ / .bash_profile
633 clear 650 clear 667 cat ~ / .bashrc
634 source dothis-completion.bash 651 source dothis-completion.bash 668 clear
635 source dothis-completion.bash 652 source dothis-completion.bash 669 install . / dothis ~ / bin / dothis
636 clear 653 source dothis-completion.bash 670 dothis
637 source dothis-completion.bash 654 clear 671 dothis 6546545646
638 clear 655 dothis 654 672 clear
639 source dothis-completion.bash 656 dothis 631 673 dothis
640 source dothis-completion.bash 657 dothis 150 674 dothis 651
641 source dothis-completion.bash 658 dothis 675 source dothis-completion.bash
642 clear 659 clear 676 dothis 651
643 dothis 623 ls -la 660 dothis 677 dothis 659
644 clear 661 install . / dothis ~ / bin / dothis 678 clear
645 source dothis-completion.bash 662 dothis 679 dothis 665
646 clear 663 install . / dothis ~ / bin / dothis 680 clear
647 source dothis-completion.bash 664 dothis 681 clear
648 clear 665 cat ~ / .bashrcNot bad.
We do have a problem, though. Try typing a number as you see it in your completion list and then press the key again.
$ dothis 623 < tab >
$ dothis 623 ls 623 ls -la
...
$ dothis 623 ls 623 ls 623 ls 623 ls 623 ls -laThis is happening because in our completion script, we used the
${COMP_WORDS[1]}to always check the first typed word after thedothiscommand (the number623in the above snippet). Hence the completion continues to suggest the same completion again and again when the Tab key is pressed.To fix this, we will not allow any kind of completion to take place if at least one argument has already been typed. We will add a condition in our function that checks the size of the aforementioned
#/usr/bin/env bashCOMP_WORDSarray.
_dothis_completions ()
{
if [ " ${#COMP_WORDS[@]} " ! = "2" ] ; then
return
fiCOMPREPLY = ( $ ( compgen -W " $(fc -l -50 | sed 's/\t//') " -- " ${COMP_WORDS[1]} " ))
}complete -F _dothis_completions dothis
Source and retry.
$ dothis 623 < tab >
$ dothis 623 ls -la < tab > # SUCCESS: nothing happens hereThere is another thing we don't like, though. We do want to display the numbers along with the corresponding commands to help users decide which one is desired, but when there is only one completion suggestion and it gets automatically picked by the completion mechanism, we shouldn't append the command literal too .
In other words, our
dothisexecutable accepts only a number, and we haven't added any functionality to check or expect other arguments. When our completion function gives only one result, we should trim the command literal and respond only with the command number.To accomplish this, we will keep the response of the
compgencommand in an array variable, and if its size is 1 , we will trim the one and only element to keep just the number. Otherwise, we'll let the array as is.Change the completion script to this:
#/usr/bin/env bash
_dothis_completions ()
{
if [ " ${#COMP_WORDS[@]} " ! = "2" ] ; then
return
fi# keep the suggestions in a local variable
local suggestions = ( $ ( compgen -W " $(fc -l -50 | sed 's/\t/ /') " -- " ${COMP_WORDS[1]} " ))if [ " ${#suggestions[@]} " == "1" ] ; then
# if there's only one match, we remove the command literal
# to proceed with the automatic completion of the number
local number =$ ( echo ${suggestions[0]/%\ */} )
COMPREPLY = ( " $number " )
else
# more than one suggestions resolved,
# respond with the suggestions intact
COMPREPLY = ( " ${suggestions[@]} " )
fi
}complete -F _dothis_completions dothis
Registering the completion scriptIf you want to enable the completion just for you on your machine, all you have to do is add a line in your
.bashrcfile sourcing the script:source <path-to-your-script>/dothis-completion.bashIf you want to enable the completion for all users, you can just copy the script under
Fine-tuning the completion script/etc/bash_completion.d/and it will automatically be loaded by Bash.Here are some extra steps for better results:
Displaying each entry in a new lineIn the Bash completion script I was working on, I too had to present suggestions consisting of two parts. I wanted to display the first part in the default color and the second part in gray to distinguish it as help text. In this tutorial's example, it would be nice to present the numbers in the default color and the command literal in a less fancy one.
Unfortunately, this is not possible, at least for now, because the completions are displayed as plain text and color directives are not processed (for example:
\e[34mBlue).What we can do to improve the user experience (or not) is to display each entry in a new line. This solution is not that obvious since we can't just append a new line character in each
COMPREPLYentry. We will follow a rather hackish method and pad suggestion literals to a width that fills the terminal.Enter
#/usr/bin/env bashprintf. If you want to display each suggestion on each own line, change the completion script to the following:
_dothis_completions ()
{
if [ " ${#COMP_WORDS[@]} " ! = "2" ] ; then
return
filocal IFS =$ '\n'
local suggestions = ( $ ( compgen -W " $(fc -l -50 | sed 's/\t//') " -- " ${COMP_WORDS[1]} " ))if [ " ${#suggestions[@]} " == "1" ] ; then
local number = " ${suggestions[0]/%\ */} "
COMPREPLY = ( " $number " )
else
for i in " ${!suggestions[@]} " ; do
suggestions [ $i ] = " $(printf '%*s' "-$COLUMNS" "${suggestions[$i]}") "
doneCOMPREPLY = ( " ${suggestions[@]} " )
fi
}complete -F _dothis_completions dothis
Source and test:
dothis < tab >< tab >
...
499 source dothis-completion.bash
500 clear
...
503 dothis 500Customizable behaviorIn our case, we hard-coded to display the last 50 commands for completion. This is not a good practice. We should first respect what each user might prefer. If he/she hasn't made any preference, we should default to 50.
To accomplish that, we will check if an environment variable
DOTHIS_COMPLETION_COMMANDS_NUMBERhas been set.Change the completion script one last time:
#/usr/bin/env bash
_dothis_completions ()
{
if [ " ${#COMP_WORDS[@]} " ! = "2" ] ; then
return
filocal commands_number = ${DOTHIS_COMPLETION_COMMANDS_NUMBER:-50}
local IFS =$ '\n'
local suggestions = ( $ ( compgen -W " $(fc -l -$commands_number | sed 's/\t//') " -- " ${COMP_WORDS[1]} " ))if [ " ${#suggestions[@]} " == "1" ] ; then
local number = " ${suggestions[0]/%\ */} "
COMPREPLY = ( " $number " )
else
for i in " ${!suggestions[@]} " ; do
suggestions [ $i ] = " $(printf '%*s' "-$COLUMNS" "${suggestions[$i]}") "
doneCOMPREPLY = ( " ${suggestions[@]} " )
fi
}complete -F _dothis_completions dothis
Source and test:
export DOTHIS_COMPLETION_COMMANDS_NUMBER = 5
$ dothis < tab >< tab >
505 clear
506 source . / dothis-completion.bash
507 dothis clear
508 clear
509 export DOTHIS_COMPLETION_COMMANDS_NUMBER = 5 Useful linksCode and comments
- Git's completion script
- Bash Reference Manual: Programmable Completion
- Bash Reference Manual: Programmable Completion Builtins
- Bash Reference Manual: A Programmable Completion Example
- Bash Reference Manual: Bash Variables
You can find the code of this tutorial on GitHub .
For feedback, comments, typos, etc., please open an issue in the repository.
Lazarus Lazaridis - I am an open source enthusiast and I like helping developers with tutorials and tools . I usually code in Ruby especially when it's on Rails but I also speak Java, Go, bash & C#. I have studied CS at Athens University of Economics and Business and I live in Athens, Greece. My nickname is iridakos and I publish tech related posts on my personal blog iridakos.com .
|
|
|
Apr 26, 2018 | linuxhint.com
Bash Range: How to iterate over sequences generated on the shell 2 days ago You can iterate the sequence of numbers in bash by two ways. One is by using seq command and another is by specifying range in for loop. In seq command, the sequence starts from one, the number increments by one in each step and print each number in each line up to the upper limit by default. If the number starts from upper limit then it decrements by one in each step. Normally, all numbers are interpreted as floating point but if the sequence starts from integer then the list of decimal integers will print. If seq command can execute successfully then it returns 0, otherwise it returns any non-zero number. You can also iterate the sequence of numbers using for loop with range. Both seq command and for loop with range are shown in this tutorial by using examples.
The options of seq command:
You can use seq command by using the following options.
Examples of seq command:
- -w This option is used to pad the numbers with leading zeros to print all numbers with equal width.
- -f format This option is used to print number with particular format. Floating number can be formatted by using %f, %g and %e as conversion characters. %g is used as default.
- -s string This option is used to separate the numbers with string. The default value is newline ('\n').
You can apply seq command by three ways. You can use only upper limit or upper and lower limit or upper and lower limit with increment or decrement value of each step . Different uses of the seq command with options are shown in the following examples.
Example-1: seq command without optionWhen only upper limit is used then the number will start from 1 and increment by one in each step. The following command will print the number from 1 to 4.
$ seq 4When the two values are used with seq command then first value will be used as starting number and second value will be used as ending number. The following command will print the number from 7 to 15.
$ seq 7 15When you will use three values with seq command then the second value will be used as increment or decrement value for each step. For the following command, the starting number is 10, ending number is 1 and each step will be counted by decrementing 2.
$ seq 10 -2 1Example-2: seq with –w optionThe following command will print the output by adding leading zero for the number from 1 to 9.
$ seq -w 0110Example-3: seq with –s optionThe following command uses "-" as separator for each sequence number. The sequence of numbers will print by adding "-" as separator.
$ seq -s - 8Example-4: seq with -f option
The following command will print 10 date values starting from 1. Here, "%g" option is used to add sequence number with other string value.
$ seq -f "%g/04/2018" 10The following command is used to generate the sequence of floating point number using "%f" . Here, the number will start from 3 and increment by 0.8 in each step and the last number will be less than or equal to 6.
$ seq -f "%f" 3 0.8 6Example-5: Write the sequence in a file
If you want to save the sequence of number into a file without printing in the console then you can use the following commands. The first command will print the numbers to a file named " seq.txt ". The number will generate from 5 to 20 and increment by 10 in each step. The second command is used to view the content of " seq.txt" file.
seq 5 10 20 | cat > seq.txt cat seq.txtExample-6: Using seq in for loop
Suppose, you want to create files named fn1 to fn10 using for loop with seq. Create a file named "sq1.bash" and add the following code. For loop will iterate for 10 times using seq command and create 10 files in the sequence fn1, fn2,fn3 ..fn10.
#!/bin/bash
for i in ` seq 10 ` ; do touch fn. $i doneRun the following commands to execute the code of the bash file and check the files are created or not.
bash sq1.bash lsExamples of for loop with range: Example-7: For loop with range
The alternative of seq command is range. You can use range in for loop to generate sequence of numbers like seq. Write the following code in a bash file named " sq2.bash ". The loop will iterate for 5 times and print the square root of each number in each step.
#!/bin/bash
for n in { 1 .. 5 } ; do (( result =n * n ))
echo $n square = $result
doneRun the command to execute the script of the file.
bash sq2.bashExample-8: For loop with range and increment value
By default, the number is increment by one in each step in range like seq. You can also change the increment value in range. Write the following code in a bash file named " sq3.bash ". The for loop in the script will iterate for 5 times, each step is incremented by 2 and print all odd numbers between 1 to 10.
#!/bin/bash
echo "all odd numbers from 1 to 10 are"
for i in { 1 .. 10 .. 2 }; do echo $i ; doneRun the command to execute the script of the file.
bash sq3.bashIf you want to work with the sequence of numbers then you can use any of the options that are shown in this tutorial. After completing this tutorial, you will be able to use seq command and for loop with range more efficiently in your bash script.
|
|
|
Mar 19, 2018 | www.tecmint.com
For example, if you have a directory ~/Documents/Phone-Backup/Linux-Docs/Ubuntu/ , using gogo , you can create an alias (a shortcut name), for instance
Ubuntuto access it without typing the whole path anymore. No matter your current working directory, you can move into ~/cd Documents/Phone-Backup/Linux-Docs/Ubuntu/ by simply using the aliasUbuntu.Read Also : bd – Quickly Go Back to a Parent Directory Instead of Typing "cd ../../.." Redundantly
In addition, it also allows you to create aliases for connecting directly into directories on remote Linux servers.
How to Install Gogo in Linux SystemsTo install Gogo , first clone the gogo repository from Github and then copy the
gogo.pyto any directory in your PATH environmental variable (if you already have the~/bin/directory, you can place it here, otherwise create it).$ git clone https://github.com/mgoral/gogo.git $ cd gogo/ $ mkdir -p ~/bin #run this if you do not have ~/bin directory $ cp gogo.py ~/bin/... ... ...
To start using gogo , you need to logout and login back to use it. Gogo stores its configuration in~/.config/gogo/gogo.conffile (which should be auto created if it doesn't exist) and has the following syntax.# Comments are lines that start from '#' character. default = ~/something alias = /desired/path alias2 = /desired/path with space alias3 = "/this/also/works" zażółć = "unicode/is/also/supported/zażółć gęślą jaźń"If you run gogo run without any arguments, it will go to the directory specified in default; this alias is always available, even if it's not in the configuration file, and points to $HOME directory.
To display the current aliases, use the
-lswitch
|
|
|
Jan 08, 2018 | www.linuxjournal.com
Triggering scripts with incron and systemd.
It is, at times, important to know when things change in the Linux OS. The uses to which systems are placed often include high-priority data that must be processed as soon as it is seen. The conventional method of finding and processing new file data is to poll for it, usually with cron. This is inefficient, and it can tax performance unreasonably if too many polling events are forked too often.
Linux has an efficient method for alerting user-space processes to changes impacting files of interest. The inotify Linux system calls were first discussed here in Linux Journal in a 2005 article by Robert Love who primarily addressed the behavior of the new features from the perspective of C.
However, there also are stable shell-level utilities and new classes of monitoring dæmons for registering filesystem watches and reporting events. Linux installations using systemd also can access basic inotify functionality with path units. The inotify interface does have limitations -- it can't monitor remote, network-mounted filesystems (that is, NFS); it does not report the userid involved in the event; it does not work with /proc or other pseudo-filesystems; and mmap() operations do not trigger it, among other concerns. Even with these limitations, it is a tremendously useful feature.
This article completes the work begun by Love and gives everyone who can write a Bourne shell script or set a crontab the ability to react to filesystem changes.
The inotifywait UtilityWorking under Oracle Linux 7 (or similar versions of Red Hat/CentOS/Scientific Linux), the inotify shell tools are not installed by default, but you can load them with yum:
# yum install inotify-tools Loaded plugins: langpacks, ulninfo ol7_UEKR4 | 1.2 kB 00:00 ol7_latest | 1.4 kB 00:00 Resolving Dependencies --> Running transaction check ---> Package inotify-tools.x86_64 0:3.14-8.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================== Package Arch Version Repository Size ============================================================== Installing: inotify-tools x86_64 3.14-8.el7 ol7_latest 50 k Transaction Summary ============================================================== Install 1 Package Total download size: 50 k Installed size: 111 k Is this ok [y/d/N]: y Downloading packages: inotify-tools-3.14-8.el7.x86_64.rpm | 50 kB 00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : inotify-tools-3.14-8.el7.x86_64 1/1 Verifying : inotify-tools-3.14-8.el7.x86_64 1/1 Installed: inotify-tools.x86_64 0:3.14-8.el7 Complete!The package will include two utilities (inotifywait and inotifywatch), documentation and a number of libraries. The inotifywait program is of primary interest.
Some derivatives of Red Hat 7 may not include inotify in their base repositories. If you find it missing, you can obtain it from Fedora's EPEL repository , either by downloading the inotify RPM for manual installation or adding the EPEL repository to yum.
Any user on the system who can launch a shell may register watches -- no special privileges are required to use the interface. This example watches the /tmp directory:
$ inotifywait -m /tmp Setting up watches. Watches established.If another session on the system performs a few operations on the files in /tmp:
$ touch /tmp/hello $ cp /etc/passwd /tmp $ rm /tmp/passwd $ touch /tmp/goodbye $ rm /tmp/hello /tmp/goodbyethose changes are immediately visible to the user running inotifywait:
/tmp/ CREATE hello /tmp/ OPEN hello /tmp/ ATTRIB hello /tmp/ CLOSE_WRITE,CLOSE hello /tmp/ CREATE passwd /tmp/ OPEN passwd /tmp/ MODIFY passwd /tmp/ CLOSE_WRITE,CLOSE passwd /tmp/ DELETE passwd /tmp/ CREATE goodbye /tmp/ OPEN goodbye /tmp/ ATTRIB goodbye /tmp/ CLOSE_WRITE,CLOSE goodbye /tmp/ DELETE hello /tmp/ DELETE goodbyeA few relevant sections of the manual page explain what is happening:
$ man inotifywait | col -b | sed -n '/diagnostic/,/helpful/p' inotifywait will output diagnostic information on standard error and event information on standard output. The event output can be config- ured, but by default it consists of lines of the following form: watched_filename EVENT_NAMES event_filename watched_filename is the name of the file on which the event occurred. If the file is a directory, a trailing slash is output. EVENT_NAMES are the names of the inotify events which occurred, separated by commas. event_filename is output only when the event occurred on a directory, and in this case the name of the file within the directory which caused this event is output. By default, any special characters in filenames are not escaped in any way. This can make the output of inotifywait difficult to parse in awk scripts or similar. The --csv and --format options will be helpful in this case.It also is possible to filter the output by registering particular events of interest with the
-eoption, the list of which is shown here:
access create move_self attrib delete moved_to close_write delete_self moved_from close_nowrite modify open close move unmount A common application is testing for the arrival of new files. Since inotify must be given the name of an existing filesystem object to watch, the directory containing the new files is provided. A trigger of interest is also easy to provide -- new files should be complete and ready for processing when the
close_writetrigger fires. Below is an example script to watch for these events:#!/bin/sh unset IFS # default of space, tab and nl # Wait for filesystem events inotifywait -m -e close_write \ /tmp /var/tmp /home/oracle/arch-orcl/ | while read dir op file do [[ "${dir}" == '/tmp/' && "${file}" == *.txt ]] && echo "Import job should start on $file ($dir $op)." [[ "${dir}" == '/var/tmp/' && "${file}" == CLOSE_WEEK*.txt ]] && echo Weekly backup is ready. [[ "${dir}" == '/home/oracle/arch-orcl/' && "${file}" == *.ARC ]] && su - oracle -c 'ORACLE_SID=orcl ~oracle/bin/log_shipper' & [[ "${dir}" == '/tmp/' && "${file}" == SHUT ]] && break ((step+=1)) done echo We processed $step events.There are a few problems with the script as presented -- of all the available shells on Linux, only ksh93 (that is, the AT&T Korn shell) will report the "step" variable correctly at the end of the script. All the other shells will report this variable as null.
The reason for this behavior can be found in a brief explanation on the manual page for Bash: "Each command in a pipeline is executed as a separate process (i.e., in a subshell)." The MirBSD clone of the Korn shell has a slightly longer explanation:
# man mksh | col -b | sed -n '/The parts/,/do so/p' The parts of a pipeline, like below, are executed in subshells. Thus, variable assignments inside them fail. Use co-processes instead. foo | bar | read baz # will not change $baz foo | bar |& read -p baz # will, however, do soAnd, the pdksh documentation in Oracle Linux 5 (from which MirBSD mksh emerged) has several more mentions of the subject:
General features of at&t ksh88 that are not (yet) in pdksh: - the last command of a pipeline is not run in the parent shell - `echo foo | read bar; echo $bar' prints foo in at&t ksh, nothing in pdksh (ie, the read is done in a separate process in pdksh). - in pdksh, if the last command of a pipeline is a shell builtin, it is not executed in the parent shell, so "echo a b | read foo bar" does not set foo and bar in the parent shell (at&t ksh will). This may get fixed in the future, but it may take a while. $ man pdksh | col -b | sed -n '/BTW, the/,/aware/p' BTW, the most frequently reported bug is echo hi | read a; echo $a # Does not print hi I'm aware of this and there is no need to report it.This behavior is easy enough to demonstrate -- running the script above with the default bash shell and providing a sequence of example events:
$ cp /etc/passwd /tmp/newdata.txt $ cp /etc/group /var/tmp/CLOSE_WEEK20170407.txt $ cp /etc/passwd /tmp/SHUTgives the following script output:
# ./inotify.sh Setting up watches. Watches established. Import job should start on newdata.txt (/tmp/ CLOSE_WRITE,CLOSE). Weekly backup is ready. We processed events.Examining the process list while the script is running, you'll also see two shells, one forked for the control structure:
$ function pps { typeset a IFS=\| ; ps ax | while read a do case $a in *$1*|+([!0-9])) echo $a;; esac; done } $ pps inot PID TTY STAT TIME COMMAND 3394 pts/1 S+ 0:00 /bin/sh ./inotify.sh 3395 pts/1 S+ 0:00 inotifywait -m -e close_write /tmp /var/tmp 3396 pts/1 S+ 0:00 /bin/sh ./inotify.shAs it was manipulated in a subshell, the "step" variable above was null when control flow reached the echo. Switching this from #/bin/sh to #/bin/ksh93 will correct the problem, and only one shell process will be seen:
# ./inotify.ksh93 Setting up watches. Watches established. Import job should start on newdata.txt (/tmp/ CLOSE_WRITE,CLOSE). Weekly backup is ready. We processed 2 events. $ pps inot PID TTY STAT TIME COMMAND 3583 pts/1 S+ 0:00 /bin/ksh93 ./inotify.sh 3584 pts/1 S+ 0:00 inotifywait -m -e close_write /tmp /var/tmpAlthough ksh93 behaves properly and in general handles scripts far more gracefully than all of the other Linux shells, it is rather large:
$ ll /bin/[bkm]+([aksh93]) /etc/alternatives/ksh -rwxr-xr-x. 1 root root 960456 Dec 6 11:11 /bin/bash lrwxrwxrwx. 1 root root 21 Apr 3 21:01 /bin/ksh -> /etc/alternatives/ksh -rwxr-xr-x. 1 root root 1518944 Aug 31 2016 /bin/ksh93 -rwxr-xr-x. 1 root root 296208 May 3 2014 /bin/mksh lrwxrwxrwx. 1 root root 10 Apr 3 21:01 /etc/alternatives/ksh -> /bin/ksh93The mksh binary is the smallest of the Bourne implementations above (some of these shells may be missing on your system, but you can install them with yum). For a long-term monitoring process, mksh is likely the best choice for reducing both processing and memory footprint, and it does not launch multiple copies of itself when idle assuming that a coprocess is used. Converting the script to use a Korn coprocess that is friendly to mksh is not difficult:
#!/bin/mksh unset IFS # default of space, tab and nl # Wait for filesystem events inotifywait -m -e close_write \ /tmp/ /var/tmp/ /home/oracle/arch-orcl/ \ 2</dev/null |& # Launch as Korn coprocess while read -p dir op file # Read from Korn coprocess do [[ "${dir}" == '/tmp/' && "${file}" == *.txt ]] && print "Import job should start on $file ($dir $op)." [[ "${dir}" == '/var/tmp/' && "${file}" == CLOSE_WEEK*.txt ]] && print Weekly backup is ready. [[ "${dir}" == '/home/oracle/arch-orcl/' && "${file}" == *.ARC ]] && su - oracle -c 'ORACLE_SID=orcl ~oracle/bin/log_shipper' & [[ "${dir}" == '/tmp/' && "${file}" == SHUT ]] && break ((step+=1)) done echo We processed $step events.Note that the Korn and Bolsky reference on the Korn shell outlines the following requirements in a program operating as a coprocess:
Caution: The co-process must:
- Send each output message to standard output.
- Have a Newline at the end of each message.
- Flush its standard output whenever it writes a message.
An
fflush(NULL)is found in the main processing loop of the inotifywait source, and these requirements appear to be met.The mksh version of the script is the most reasonable compromise for efficient use and correct behavior, and I have explained it at some length here to save readers trouble and frustration -- it is important to avoid control structures executing in subshells in most of the Borne family. However, hopefully all of these ersatz shells someday fix this basic flaw and implement the Korn behavior correctly.
A Practical Application -- Oracle Log ShippingOracle databases that are configured for hot backups produce a stream of "archived redo log files" that are used for database recovery. These are the most critical backup files that are produced in an Oracle database.
These files are numbered sequentially and are written to a log directory configured by the DBA. An inotifywatch can trigger activities to compress, encrypt and/or distribute the archived logs to backup and disaster recovery servers for safekeeping. You can configure Oracle RMAN to do most of these functions, but the OS tools are more capable, flexible and simpler to use.
There are a number of important design parameters for a script handling archived logs:
- A "critical section" must be established that allows only a single process to manipulate the archived log files at a time. Oracle will sometimes write bursts of log files, and inotify might cause the handler script to be spawned repeatedly in a short amount of time. Only one instance of the handler script can be allowed to run -- any others spawned during the handler's lifetime must immediately exit. This will be achieved with a textbook application of the flock program from the util-linux package.
- The optimum compression available for production applications appears to be lzip . The author claims that the integrity of his archive format is superior to many more well known utilities , both in compression ability and also structural integrity. The lzip binary is not in the standard repository for Oracle Linux -- it is available in EPEL and is easily compiled from source.
- Note that 7-Zip uses the same LZMA algorithm as lzip, and it also will perform AES encryption on the data after compression. Encryption is a desirable feature, as it will exempt a business from breach disclosure laws in most US states if the backups are lost or stolen and they contain "Protected Personal Information" (PPI), such as birthdays or Social Security Numbers. The author of lzip does have harsh things to say regarding the quality of 7-Zip archives using LZMA2, and the
openssl encprogram can be used to apply AES encryption after compression to lzip archives or any other type of file, as I discussed in a previous article . I'm foregoing file encryption in the script below and using lzip for clarity.- The current log number will be recorded in a dot file in the Oracle user's home directory. If a log is skipped for some reason (a rare occurrence for an Oracle database), log shipping will stop. A missing log requires an immediate and full database backup (either cold or hot) -- successful recoveries of Oracle databases cannot skip logs.
- The
scpprogram will be used to copy the log to a remote server, and it should be called repeatedly until it returns successfully.- I'm calling the genuine '93 Korn shell for this activity, as it is the most capable scripting shell and I don't want any surprises.
Given these design parameters, this is an implementation:
# cat ~oracle/archutils/process_logs #!/bin/ksh93 set -euo pipefail IFS=$'\n\t' # http://redsymbol.net/articles/unofficial-bash-strict-mode/ ( flock -n 9 || exit 1 # Critical section-allow only one process. ARCHDIR=~oracle/arch-${ORACLE_SID} APREFIX=${ORACLE_SID}_1_ ASUFFIX=.ARC CURLOG=$(<~oracle/.curlog-$ORACLE_SID) File="${ARCHDIR}/${APREFIX}${CURLOG}${ASUFFIX}" [[ ! -f "$File" ]] && exit while [[ -f "$File" ]] do ((NEXTCURLOG=CURLOG+1)) NextFile="${ARCHDIR}/${APREFIX}${NEXTCURLOG}${ASUFFIX}" [[ ! -f "$NextFile" ]] && sleep 60 # Ensure ARCH has finished nice /usr/local/bin/lzip -9q "$File" until scp "${File}.lz" "yourcompany.com:~oracle/arch-$ORACLE_SID" do sleep 5 done CURLOG=$NEXTCURLOG File="$NextFile" done echo $CURLOG > ~oracle/.curlog-$ORACLE_SID ) 9>~oracle/.processing_logs-$ORACLE_SIDThe above script can be executed manually for testing even while the inotify handler is running, as the flock protects it.
A standby server, or a DataGuard server in primitive standby mode, can apply the archived logs at regular intervals. The script below forces a 12-hour delay in log application for the recovery of dropped or damaged objects, so inotify cannot be easily used in this case -- cron is a more reasonable approach for delayed file processing, and a run every 20 minutes will keep the standby at the desired recovery point:
# cat ~oracle/archutils/delay-lock.sh #!/bin/ksh93 ( flock -n 9 || exit 1 # Critical section-only one process. WINDOW=43200 # 12 hours LOG_DEST=~oracle/arch-$ORACLE_SID OLDLOG_DEST=$LOG_DEST-applied function fage { print $(( $(date +%s) - $(stat -c %Y "$1") )) } # File age in seconds - Requires GNU extended date & stat cd $LOG_DEST of=$(ls -t | tail -1) # Oldest file in directory [[ -z "$of" || $(fage "$of") -lt $WINDOW ]] && exit for x in $(ls -rt) # Order by ascending file mtime do if [[ $(fage "$x") -ge $WINDOW ]] then y=$(basename $x .lz) # lzip compression is optional [[ "$y" != "$x" ]] && /usr/local/bin/lzip -dkq "$x" $ORACLE_HOME/bin/sqlplus '/ as sysdba' > /dev/null 2>&1 <<-EOF recover standby database; $LOG_DEST/$y cancel quit EOF [[ "$y" != "$x" ]] && rm "$y" mv "$x" $OLDLOG_DEST fi done ) 9> ~oracle/.recovering-$ORACLE_SIDI've covered these specific examples here because they introduce tools to control concurrency, which is a common issue when using inotify, and they advance a few features that increase reliability and minimize storage requirements. Hopefully enthusiastic readers will introduce many improvements to these approaches.
The incron SystemLukas Jelinek is the author of the incron package that allows users to specify tables of inotify events that are executed by the master incrond process. Despite the reference to "cron", the package does not schedule events at regular intervals -- it is a tool for filesystem events, and the cron reference is slightly misleading.
The incron package is available from EPEL . If you have installed the repository, you can load it with yum:
# yum install incron Loaded plugins: langpacks, ulninfo Resolving Dependencies --> Running transaction check ---> Package incron.x86_64 0:0.5.10-8.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ================================================================= Package Arch Version Repository Size ================================================================= Installing: incron x86_64 0.5.10-8.el7 epel 92 k Transaction Summary ================================================================== Install 1 Package Total download size: 92 k Installed size: 249 k Is this ok [y/d/N]: y Downloading packages: incron-0.5.10-8.el7.x86_64.rpm | 92 kB 00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : incron-0.5.10-8.el7.x86_64 1/1 Verifying : incron-0.5.10-8.el7.x86_64 1/1 Installed: incron.x86_64 0:0.5.10-8.el7 Complete!On a systemd distribution with the appropriate service units, you can start and enable incron at boot with the following commands:
# systemctl start incrond # systemctl enable incrond Created symlink from /etc/systemd/system/multi-user.target.wants/incrond.service to /usr/lib/systemd/system/incrond.service.In the default configuration, any user can establish incron schedules. The incrontab format uses three fields:
<path> <mask> <command>Below is an example entry that was set with the
-eoption:$ incrontab -e #vi session follows $ incrontab -l /tmp/ IN_ALL_EVENTS /home/luser/myincron.sh $@ $% $#You can record a simple script and mark it with execute permission:
$ cat myincron.sh #!/bin/sh echo -e "path: $1 op: $2 \t file: $3" >> ~/op $ chmod 755 myincron.shThen, if you repeat the original /tmp file manipulations at the start of this article, the script will record the following output:
$ cat ~/op path: /tmp/ op: IN_ATTRIB file: hello path: /tmp/ op: IN_CREATE file: hello path: /tmp/ op: IN_OPEN file: hello path: /tmp/ op: IN_CLOSE_WRITE file: hello path: /tmp/ op: IN_OPEN file: passwd path: /tmp/ op: IN_CLOSE_WRITE file: passwd path: /tmp/ op: IN_MODIFY file: passwd path: /tmp/ op: IN_CREATE file: passwd path: /tmp/ op: IN_DELETE file: passwd path: /tmp/ op: IN_CREATE file: goodbye path: /tmp/ op: IN_ATTRIB file: goodbye path: /tmp/ op: IN_OPEN file: goodbye path: /tmp/ op: IN_CLOSE_WRITE file: goodbye path: /tmp/ op: IN_DELETE file: hello path: /tmp/ op: IN_DELETE file: goodbyeWhile the
IN_CLOSE_WRITEevent on a directory object is usually of greatest interest, most of the standard inotify events are available within incron, which also offers several unique amalgams:$ man 5 incrontab | col -b | sed -n '/EVENT SYMBOLS/,/child process/p' EVENT SYMBOLS These basic event mask symbols are defined: IN_ACCESS File was accessed (read) (*) IN_ATTRIB Metadata changed (permissions, timestamps, extended attributes, etc.) (*) IN_CLOSE_WRITE File opened for writing was closed (*) IN_CLOSE_NOWRITE File not opened for writing was closed (*) IN_CREATE File/directory created in watched directory (*) IN_DELETE File/directory deleted from watched directory (*) IN_DELETE_SELF Watched file/directory was itself deleted IN_MODIFY File was modified (*) IN_MOVE_SELF Watched file/directory was itself moved IN_MOVED_FROM File moved out of watched directory (*) IN_MOVED_TO File moved into watched directory (*) IN_OPEN File was opened (*) When monitoring a directory, the events marked with an asterisk (*) above can occur for files in the directory, in which case the name field in the returned event data identifies the name of the file within the directory. The IN_ALL_EVENTS symbol is defined as a bit mask of all of the above events. Two additional convenience symbols are IN_MOVE, which is a com- bination of IN_MOVED_FROM and IN_MOVED_TO, and IN_CLOSE, which combines IN_CLOSE_WRITE and IN_CLOSE_NOWRITE. The following further symbols can be specified in the mask: IN_DONT_FOLLOW Don't dereference pathname if it is a symbolic link IN_ONESHOT Monitor pathname for only one event IN_ONLYDIR Only watch pathname if it is a directory Additionally, there is a symbol which doesn't appear in the inotify sym- bol set. It is IN_NO_LOOP. This symbol disables monitoring events until the current one is completely handled (until its child process exits).The incron system likely presents the most comprehensive interface to inotify of all the tools researched and listed here. Additional configuration options can be set in /etc/incron.conf to tweak incron's behavior for those that require a non-standard configuration.
Path Units under systemdWhen your Linux installation is running systemd as PID 1, limited inotify functionality is available through "path units" as is discussed in a lighthearted article by Paul Brown at OCS-Mag .
The relevant manual page has useful information on the subject:
$ man systemd.path | col -b | sed -n '/Internally,/,/systems./p' Internally, path units use the inotify(7) API to monitor file systems. Due to that, it suffers by the same limitations as inotify, and for example cannot be used to monitor files or directories changed by other machines on remote NFS file systems.Note that when a systemd path unit spawns a shell script, the
$HOMEand tilde (~) operator for the owner's home directory may not be defined. Using the tilde operator to reference another user's home directory (for example, ~nobody/) does work, even when applied to the self-same user running the script. The Oracle script above was explicit and did not reference ~ without specifying the target user, so I'm using it as an example here.Using inotify triggers with systemd path units requires two files. The first file specifies the filesystem location of interest:
$ cat /etc/systemd/system/oralog.path [Unit] Description=Oracle Archivelog Monitoring Documentation=http://docs.yourserver.com [Path] PathChanged=/home/oracle/arch-orcl/ [Install] WantedBy=multi-user.targetThe
PathChangedparameter above roughly corresponds to theclose-writeevent used in my previous direct inotify calls. The full collection of inotify events is not (currently) supported by systemd -- it is limited toPathExists,PathChangedandPathModified, which are described inman systemd.path.The second file is a service unit describing a program to be executed. It must have the same name, but a different extension, as the path unit:
$ cat /etc/systemd/system/oralog.service [Unit] Description=Oracle Archivelog Monitoring Documentation=http://docs.yourserver.com [Service] Type=oneshot Environment=ORACLE_SID=orcl ExecStart=/bin/sh -c '/root/process_logs >> /tmp/plog.txt 2>&1'The
oneshotparameter above alerts systemd that the program that it forks is expected to exit and should not be respawned automatically -- the restarts are limited to triggers from the path unit. The above service configuration will provide the best options for logging -- divert them to /dev/null if they are not needed.Use
systemctl starton the path unit to begin monitoring -- a common error is using it on the service unit, which will directly run the handler only once. Enable the path unit if the monitoring should survive a reboot.Although this limited functionality may be enough for some casual uses of inotify, it is a shame that the full functionality of inotifywait and incron are not represented here. Perhaps it will come in time.
ConclusionAlthough the inotify tools are powerful, they do have limitations. To repeat them, inotify cannot monitor remote (NFS) filesystems; it cannot report the userid involved in a triggering event; it does not work with /proc or other pseudo-filesystems; mmap() operations do not trigger it; and the inotify queue can overflow resulting in lost events, among other concerns.
Even with these weaknesses, the efficiency of inotify is superior to most other approaches for immediate notifications of filesystem activity. It also is quite flexible, and although the close-write directory trigger should suffice for most usage, it has ample tools for covering special use cases.
In any event, it is productive to replace polling activity with inotify watches, and system administrators should be liberal in educating the user community that the classic crontab is not an appropriate place to check for new files. Recalcitrant users should be confined to Ultrix on a VAX until they develop sufficient appreciation for modern tools and approaches, which should result in more efficient Linux systems and happier administrators.
Sidenote: Archiving /etc/passwdTracking changes to the password file involves many different types of inotify triggering events. The
vipwutility commonly will make changes to a temporary file, then clobber the original with it. This can be seen when the inode number changes:# ll -i /etc/passwd 199720973 -rw-r--r-- 1 root root 3928 Jul 7 12:24 /etc/passwd # vipw [ make changes ] You are using shadow passwords on this system. Would you like to edit /etc/shadow now [y/n]? n # ll -i /etc/passwd 203784208 -rw-r--r-- 1 root root 3956 Jul 7 12:24 /etc/passwdThe destruction and replacement of /etc/passwd even occurs with setuid binaries called by unprivileged users:
$ ll -i /etc/passwd 203784196 -rw-r--r-- 1 root root 3928 Jun 29 14:55 /etc/passwd $ chsh Changing shell for fishecj. Password: New shell [/bin/bash]: /bin/csh Shell changed. $ ll -i /etc/passwd 199720970 -rw-r--r-- 1 root root 3927 Jul 7 12:23 /etc/passwdFor this reason, all inotify triggering events should be considered when tracking this file. If there is concern with an inotify queue overflow (in which events are lost), then the
OPEN,ACCESSandCLOSE_NOWRITE,CLOSEtriggers likely can be immediately ignored.All other inotify events on /etc/passwd might run the following script to version the changes into an RCS archive and mail them to an administrator:
#!/bin/sh # This script tracks changes to the /etc/passwd file from inotify. # Uses RCS for archiving. Watch for UID zero. PWMAILS=Charlie.Root@openbsd.org TPDIR=~/track_passwd cd $TPDIR if diff -q /etc/passwd $TPDIR/passwd then exit # they are the same else sleep 5 # let passwd settle diff /etc/passwd $TPDIR/passwd 2>&1 | # they are DIFFERENT mail -s "/etc/passwd changes $(hostname -s)" "$PWMAILS" cp -f /etc/passwd $TPDIR # copy for checkin # "SCCS, the source motel! Programs check in and never check out!" # -- Ken Thompson rcs -q -l passwd # lock the archive ci -q -m_ passwd # check in new ver co -q passwd # drop the new copy fi > /dev/null 2>&1Here is an example email from the script for the above
chfnoperation:-----Original Message----- From: root [mailto:root@myhost.com] Sent: Thursday, July 06, 2017 2:35 PM To: Fisher, Charles J. <Charles.Fisher@myhost.com>; Subject: /etc/passwd changes myhost 57c57 < fishecj:x:123:456:Fisher, Charles J.:/home/fishecj:/bin/bash --- > fishecj:x:123:456:Fisher, Charles J.:/home/fishecj:/bin/cshFurther processing on the third column of /etc/passwd might detect UID zero (a root user) or other important user classes for emergency action. This might include a rollback of the file from RCS to /etc and/or SMS messages to security contacts. ______________________
Charles Fisher has an electrical engineering degree from the University of Iowa and works as a systems and database administrator for a Fortune 500 mining and manufacturing corporation.
|
|
|
Dec 09, 2017 | stackoverflow.com
,
That line defines what program will execute the given script. Forshnormally that line should start with the # character as so:#!/bin/sh -eThe -e flag's long name is
errexit, causing the script to immediately exit on the first error.
|
|
|
Dec 02, 2017 | www.cyberciti.biz
BASH Shell: How To Redirect stderr To stdout ( redirect stderr to a File ) Posted on March 12, 2008 March 12, 2008 in Categories BASH Shell , Linux , UNIX last updated March 12, 2008 Q. How do I redirect stderr to stdout? How do I redirect stderr to a file?
A. Bash and other modern shell provides I/O redirection facility. There are 3 default standard files (standard streams) open:
[a] stdin – Use to get input (keyboard) i.e. data going into a program.
[b] stdout – Use to write information (screen)
[c] stderr – Use to write error message (screen)
Understanding I/O streams numbersThe Unix / Linux standard I/O streams with numbers:
Redirecting the standard error stream to a file
Handle Name Description 0 stdin Standard input 1 stdout Standard output 2 stderr Standard error The following will redirect program error message to a file called error.log:
Redirecting the standard error (stderr) and stdout to file
$ program-name 2> error.log
$ command1 2> error.logUse the following syntax:
Redirect stderr to stdout
$ command-name &>file
OR
$ command > file-name 2>&1
Another useful example:
# find /usr/home -name .profile 2>&1 | moreUse the command as follows:
$ command-name 2>&1
|
|
|
Nov 01, 2017 | sanctum.geek.nz
A more flexible method for defining custom commands for an interactive shell (or within a script) is to use a shell function. We could declare our
llfunction in a Bash startup file as a function instead of an alias like so:# Shortcut to call ls(1) with the -l flag ll() { command ls -l "$@" }Note the use of the
commandbuiltin here to specify that thellfunction should invoke the program namedls, and not any function namedls. This is particularly important when writing a function wrapper around a command, to stop an infinite loop where the function calls itself indefinitely:# Always add -q to invocations of gdb(1) gdb() { command gdb -q "$@" }In both examples, note also the use of the
"$@"expansion, to add to the final command line any arguments given to the function. We wrap it in double quotes to stop spaces and other shell metacharacters in the arguments causing problems. This means that thellcommand will work correctly if you were to pass it further options and/or one or more directories as arguments:$ ll -a $ ll ~/.configShell functions declared in this way are specified by POSIX for Bourne-style shells, so they should work in your shell of choice, including Bash,
dash, Korn shell, and Zsh. They can also be used within scripts, allowing you to abstract away multiple instances of similar commands to improve the clarity of your script, in much the same way the basics of functions work in general-purpose programming languages.Functions are a good and portable way to approach adding features to your interactive shell; written carefully, they even allow you to port features you might like from other shells into your shell of choice. I'm fond of taking commands I like from Korn shell or Zsh and implementing them in Bash or POSIX shell functions, such as Zsh's
varedor its two-argumentcdfeatures.If you end up writing a lot of shell functions, you should consider putting them into separate configuration subfiles to keep your shell's primary startup file from becoming unmanageably large.
Examples from the authorYou can take a look at some of the shell functions I have defined here that are useful to me in general shell usage; a lot of these amount to implementing convenience features that I wish my shell had, especially for quick directory navigation, or adding options to commands:
Other examples Variables in shell functionsYou can manipulate variables within shell functions, too:
# Print the filename of a path, stripping off its leading path and # extension fn() { name=$1 name=${name##*/} name=${name%.*} printf '%s\n' "$name" }This works fine, but the catch is that after the function is done, the value for
namewill still be defined in the shell, and will overwrite whatever was in there previously:$ printf '%s\n' "$name" foobar $ fn /home/you/Task_List.doc Task_List $ printf '%s\n' "$name" Task_ListThis may be desirable if you actually want the function to change some aspect of your current shell session, such as managing variables or changing the working directory. If you don't want that, you will probably want to find some means of avoiding name collisions in your variables.
If your function is only for use with a shell that provides the
local(Bash) ortypeset(Ksh) features, you can declare the variable as local to the function to remove its global scope, to prevent this happening:# Bash-like fn() { local name name=$1 name=${name##*/} name=${name%.*} printf '%s\n' "$name" } # Ksh-like # Note different syntax for first line function fn { typeset name name=$1 name=${name##*/} name=${name%.*} printf '%s\n' "$name" }If you're using a shell that lacks these features, or you want to aim for POSIX compatibility, things are a little trickier, since local function variables aren't specified by the standard. One option is to use a subshell , so that the variables are only defined for the duration of the function:
# POSIX; note we're using plain parentheses rather than curly brackets, for # a subshell fn() ( name=$1 name=${name##*/} name=${name%.*} printf '%s\n' "$name" ) # POSIX; alternative approach using command substitution: fn() { printf '%s\n' "$( name=$1 name=${name##*/} name=${name%.*} printf %s "$name" )" }This subshell method also allows you to change directory with
cdwithin a function without changing the working directory of the user's interactive shell, or to change shell options withsetor Bash options withshoptonly temporarily for the purposes of the function.Another method to deal with variables is to manipulate the positional parameters directly (
$1,$2) withset, since they are local to the function call too:# POSIX; using positional parameters fn() { set -- "${1##*/}" set -- "${1%.*}" printf '%s\n' "$1" }These methods work well, and can sometimes even be combined, but they're awkward to write, and harder to read than the modern shell versions. If you only need your functions to work with your modern shell, I recommend just using
Keeping functions for laterlocalortypeset. The Bash Guide on Greg's Wiki has a very thorough breakdown of functions in Bash, if you want to read about this and other aspects of functions in more detail.As you get comfortable with defining and using functions during an interactive session, you might define them in ad-hoc ways on the command line for calling in a loop or some other similar circumstance, just to solve a task in that moment.
As an example, I recently made an ad-hoc function called
monitto run a set of commands for its hostname argument that together established different types of monitoring system checks, using an existing script callednmfs:$ monit() { nmfs "$1" Ping Y ; nmfs "$1" HTTP Y ; nmfs "$1" SNMP Y ; } $ for host in webhost{1..10} ; do > monit "$host" > doneAfter that task was done, I realized I was likely to use the
monitcommand interactively again, so I decided to keep it. Shell functions only last as long as the current shell, so if you want to make them permanent, you need to store their definitions somewhere in your startup files. If you're using Bash, and you're content to just add things to the end of your~/.bashrcfile, you could just do something like this:$ declare -f monit >> ~/.bashrcThat would append the existing definition of
monitin parseable form to your~/.bashrcfile, and themonitfunction would then be loaded and available to you for future interactive sessions. Later on, I ended up convertingmonitinto a shell script, as its use wasn't limited to just an interactive shell.If you want a more robust approach to keeping functions like this for Bash permanently, I wrote a tool called Bashkeep , which allows you to quickly store functions and variables defined in your current shell into separate and appropriately-named files, including viewing and managing the list of names conveniently:
$ keep monit $ keep monit $ ls ~/.bashkeep.d monit.bash $ keep -d monit
|
|
|
February 27, 2012 sanctum.geek.nz
For tools like
diffthat work with multiple files as parameters, it can be useful to work with not just files on the filesystem, but also potentially with the output of arbitrary commands. Say, for example, you wanted to compare the output ofpsandps -ewithdiff -u. An obvious way to do this is to write files to compare the output:$ ps > ps.out $ ps -e > pse.out $ diff -u ps.out pse.outThis works just fine, but Bash provides a shortcut in the form of process substitution , allowing you to treat the standard output of commands as files. This is done with the
<()and>()operators. In our case, we want to direct the standard output of two commands into place as files:$ diff -u <(ps) <(ps -e)This is functionally equivalent, except it's a little tidier because it doesn't leave files lying around. This is also very handy for elegantly comparing files across servers, using
ssh:$ diff -u .bashrc <(ssh remote cat .bashrc)Conversely, you can also use the
>()operator to direct from a filename context to the standard input of a command. This is handy for setting up in-place filters for things like logs. In the following example, I'm making a call torsync, specifying that it should make a log of its actions inlog.txt, but filter it throughgrep -vF .tmpfirst to remove anything matching the fixed string.tmp:$ rsync -arv --log-file=>(grep -vF .tmp >log.txt) src/ host::dst/Combined with
teethis syntax is a way of simulating multiple filters for astdoutstream, transforming output from a command in as many ways as you see fit:$ ps -ef | tee >(awk '$1=="tom"' >toms-procs.txt) \ >(awk '$1=="root"' >roots-procs.txt) \ >(awk '$1!="httpd"' >not-apache-procs.txt) \ >(awk 'NR>1{print $1}' >pids-only.txt)In general, the idea is that wherever on the command line you could specify a file to be read from or written to, you can instead use this syntax to make an implicit named pipe for the text stream.
Thanks to Reddit user Rhomboid for pointing out an incorrect assertion about this syntax necessarily abstracting
mkfifocalls, which I've since removed.
|
|
|
Mar 05, 2012 | sanctum.geek.nz
With judicious use of tricks like pipes, redirects, and process substitution in modern shells, it's very often possible to avoid using temporary files, doing everything inline and keeping them quite neat. However when manipulating a lot of data into various formats you do find yourself occasionally needing a temporary file, just to hold data temporarily.
A common way to deal with this is to create a temporary file in your home directory, with some arbitrary name, something like
testorworking:$ ps -ef >~/testIf you want to save the information indefinitely for later use, this makes sense, although it would be better to give it a slightly more instructive name than just
test.If you really only needed the data temporarily, however, you're much better to use the temporary files directory. This is usually
/tmp, but for good practice's sake it's better to check the value ofTMPDIRfirst, and only use/tmpas a default:$ ps -ef >"${TMPDIR:-/tmp}"/testThis is getting better, but there is still a significant problem: there's no built-in check that the
testfile doesn't already exist, perhaps being used by some other user or program, particularly another running instance of the same script.To that end, we have the
mktempprogram, which creates an empty temporary file in the appropriate directory for you without overwriting anything, and prints the filename it created. This allows you to use the file inline in both shell scripts and one-liners, and is much safer than specifying hardcoded paths:$ mktemp /tmp/tmp.yezXn0evDf $ procsfile=$(mktemp) $ printf '%s\n' "$procsfile" /tmp/tmp.9rBjzWYaSU $ ps -ef >"$procsfile"If you're going to create several such files for related purposes, you could also create a directory in which to put them using the
-doption:$ procsdir=$(mktemp -d) $ printf '%s\n' "$procsdir" /tmp/tmp.HMAhM2RBSOOn GNU/Linux systems, files of a sufficient age in
TMPDIRare cleared on boot (controlled in/etc/default/rcSon Debian-derived systems,/etc/cron.daily/tmpwatchon Red Hat ones), making/tmpuseful as a general scratchpad as well as for a kind of relatively reliable inter-process communication without cluttering up users' home directories.In some cases, there may be additional advantages in using
/tmpfor its designed purpose as some administrators choose to mount it as atmpfsfilesystem, so it operates in RAM and works very quickly. It's also common practice to set thenoexecflag on the mount to prevent malicious users from executing any code they manage to find or save in the directory.
|
|
|
Jan 24, 2012 | sanctum.geek.nz
One of my favourite technical presentations I've read online has been Hal Pomeranz's Unix Command-Line Kung Fu , a catalogue of shortcuts and efficient methods of doing very clever things with the Bash shell. None of these are grand arcane secrets, but they're things that are often forgotten in the course of daily admin work, when you find yourself typing something you needn't, or pressing up repeatedly to find something you wrote for which you could simply search your command history.
I highly recommend reading the whole thing, as I think even the most experienced shell users will find there are useful tidbits in there that would make their lives easier and their time with the shell more productive, beyond simpler things like tab completion.
Here, I'll recap two of the things I thought were the most simple and useful items in the presentation for general shell usage, and see if I can add a little value to them with reference to the Bash manual.
History with Ctrl+RFor many shell users, finding a command in history means either pressing the up arrow key repeatedly, or perhaps piping a
historycall throughgrep. It turns out there's a much nicer way to do this, using Bash's built-in history searching functionality; if you press Ctrl+R and start typing a search pattern, the most recent command matching that pattern will automatically be inserted on your current line, at which point you can adapt it as you need, or simply press Enter to run it again. You can keep pressing Ctrl+R to move further back in your history to the next-most recent match. On my shell, if I search through my history forgit, I can pull up what I typed for a previous commit:(reverse-i-search)`git': git commit -am "Pulled up-to-date colors."This functionality isn't actually exclusive to Bash; you can establish a history search function in quite a few tools that use GNU Readline, including the MySQL client command line.
You can search forward through history in the same way with Ctrl+S, but it's likely you'll have to fix up a couple of terminal annoyances first.
Additionally, if like me you're a Vim user and you don't really like having to reach for the arrow keys, or if you're on a terminal where those keys are broken for whatever reason, you can browse back and forth within your command history with Ctrl+P (previous) and Ctrl+N (next). These are just a few of the Emacs-style shortcuts that GNU Readline provides; check here for a more complete list .
Repeating commands with !!The last command you ran in Bash can be abbreviated on the next line with two exclamation marks:
$ echo "Testing." Testing. $ !! Testing.You can use this to simply repeat a command over and over again, although for that you really should be using
watch, but more interestingly it turns out this is very handy for building complex pipes in stages. Suppose you were building a pipeline to digest some data generated from a program likenetstat, perhaps to determine the top 10 IP addresses that are holding open the most connections to a server. You might be able to build a pipeline like this:# netstat -ant # !! | awk '{print $5}' # !! | sort # !! | uniq -c # !! | sort -rn # !! | sed 10qSimilarly, you can repeat the last argument from the previous command line using
!$, which is useful if you're doing a set of operations on one file, such as checking it out via RCS, editing it, and checking it back in:$ co -l file.txt $ vim !$ $ ci -u !$Or if you happen to want to work on a set of arguments, you can repeat all of the arguments from the previous command using
!*:$ touch a.txt b.txt c.txt $ rm !*When you remember to user these three together, they can save you a lot of typing, and will really increase your accuracy because you won't be at risk of mistyping any of the commands or arguments. Naturally, however, it pays to be careful what you're running through
rm!
|
|
|
Mar 16, 2012 | sanctum.geek.nz
When you have some spare time, something instructive to do that can help fill gaps in your Unix knowledge and to get a better idea of the programs installed on your system and what they can do is a simple
whatiscall, run over all the executable files in your/binand/usr/bindirectories.This will give you a one-line summary of the file's function if available from
manpages.tom@conan:/bin$ whatis * bash (1) - GNU Bourne-Again SHell bunzip2 (1) - a block-sorting file compressor, v1.0.4 busybox (1) - The Swiss Army Knife of Embedded Linux bzcat (1) - decompresses files to stdout ... tom@conan:/usr/bin$ whatis * [ (1) - check file types and compare values 2to3 (1) - Python2 to Python3 converter 2to3-2.7 (1) - Python2 to Python3 converter 411toppm (1) - convert Sony Mavica .411 image to ppm ...It also works on many of the files in other directories, such as
/etc:tom@conan:/etc$ whatis * acpi (1) - Shows battery status and other ACPI information adduser.conf (5) - configuration file for adduser(8) and addgroup(8) adjtime (3) - correct the time to synchronize the system clock aliases (5) - Postfix local alias database format ...Because packages often install more than one binary and you're only in the habit of using one or two of them, this process can tell you about programs on your system that you may have missed, particularly standard tools that solve common problems. As an example, I first learned about
watchthis way, having used a clunky solution withforloops withsleepcalls to do the same thing many times before.
|
|
|
Oct 28, 2013 | sanctum.geek.nz
In Bash scripting (and shell scripting in general), we often want to check the exit value of a command to decide an action to take after it completes, likely for the purpose of error handling. For example, to determine whether a particular regular expression
regexwas present somewhere in a fileoptions, we might applygrep(1)with its POSIX-qoption to suppress output and just use the exit value:grep -q regex optionsAn approach sometimes taken is then to test the exit value with the
$?parameter, usingifto check if it's non-zero, which is not very elegant and a bit hard to read:# Bad practice grep -q regex options if (($? > 0)); then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fiBecause the
ifconstruct by design tests the exit value of commands , it's better to test the command directly , making the expansion of$?unnecessary:# Better if grep -q regex options; then # Do nothing : else printf '%s\n' 'myscript: Pattern not found!\n' >&2 exit 1 fiWe can precede the command to be tested with
!to negate the test as well, to prevent us having to useelseas well:# Best if ! grep -q regex options; then printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 fiAn alternative syntax is to use
&&and||to performifandelsetests with grouped commands between braces, but these tend to be harder to read:# Alternative grep -q regex options || { printf '%s\n' 'myscript: Pattern not found!' >&2 exit 1 }With this syntax, the two commands in the block are only executed if the
grep(1)call exits with a non-zero status. We can apply&&instead to execute commands if it does exit with zero.That syntax can be convenient for quickly short-circuiting failures in scripts, for example due to nonexistent commands, particularly if the command being tested already outputs its own error message. This therefore cuts the script off if the given command fails, likely due to
ffmpeg(1)being unavailable on the system:hash ffmpeg || exit 1Note that the braces for a grouped command are not needed here, as there's only one command to be run in case of failure, the
exitcall.Calls to
cdare another good use case here, as running a script in the wrong directory if a call tocdfails could have really nasty effects:cd wherever || exit 1In general, you'll probably only want to test
$?when you have specific non-zero error conditions to catch. For example, if we were using the--max-deleteoption forrsync(1), we could check a call's return value to see whetherrsync(1)hit the threshold for deleted file count and write a message to a logfile appropriately:rsync --archive --delete --max-delete=5 source destination if (($? == 25)); then printf '%s\n' 'Deletion limit was reached' >"$logfile" fiIt may be tempting to use the
errexitfeature in the hopes of stopping a script as soon as it encounters any error, but there are some problems with its usage that make it a bit error-prone. It's generally more straightforward to simply write your own error handling using the methods above.For a really thorough breakdown of dealing with conditionals in Bash, take a look at the relevant chapter of the Bash Guide .
|
|
|
Jan 30, 2015 | sanctum.geek.nz
Large shell startup scripts (
.bashrc,.profile) over about fifty lines or so with a lot of options, aliases, custom functions, and similar tweaks can get cumbersome to manage over time, and if you keep your dotfiles under version control it's not terribly helpful to see large sets of commits just editing the one file when it could be more instructive if broken up into files by section.Given that shell configuration is just shell code, we can apply the
sourcebuiltin (or the.builtin for POSIXsh) to load several files at the end of a.bashrc, for example:source ~/.bashrc.options source ~/.bashrc.aliases source ~/.bashrc.functionsThis is a better approach, but it still binds us into using those filenames; we still have to edit the
~/.bashrcfile if we want to rename them, or remove them, or add new ones.Fortunately, UNIX-like systems have a common convention for this, the
.ddirectory suffix, in which sections of configuration can be stored to be read by a main configuration file dynamically. In our case, we can create a new directory~/.bashrc.d:$ ls ~/.bashrc.d options.bash aliases.bash functions.bashWith a slightly more advanced snippet at the end of
~/.bashrc, we can then load every file with the suffix.bashin this directory:# Load any supplementary scripts for config in "$HOME"/.bashrc.d/*.bash ; do source "$config" done unset -v configNote that we unset the
configvariable after we're done, otherwise it'll be in the namespace of our shell where we don't need it. You may also wish to check for the existence of the~/.bashrc.ddirectory, check there's at least one matching file inside it, or check that the file is readable before attempting to source it, depending on your preference.The same method can be applied with
.profileto load all scripts with the suffix.shin~/.profile.d, if we want to write in POSIXsh, with some slightly different syntax:# Load any supplementary scripts for config in "$HOME"/.profile.d/*.sh ; do . "$config" done unset -v configAnother advantage of this method is that if you have your dotfiles under version control, you can arrange to add extra snippets on a per-machine basis unversioned, without having to update your
.bashrcfile.Here's my implementation of the above method, for both
.bashrcand.profile:Thanks to commenter oylenshpeegul for correcting the syntax of the loops.
|
|
|
Feb 21, 2012 | sanctum.geek.nz
By default, the Bash shell keeps the history of your most recent session in the
.bash_historyfile, and the commands you've issued in your current session are also available with ahistorycall. These defaults are useful for keeping track of what you've been up to in the shell on any given machine, but with disks much larger and faster than they were when Bash was designed, a little tweaking in your.bashrcfile can record history more permanently, consistently, and usefully. Append history instead of rewriting itYou should start by setting the
histappendoption, which will mean that when you close a session, your history will be appended to the.bash_historyfile rather than overwriting what's in there.shopt -s histappendAllow a larger history fileThe default maximum number of commands saved into the
.bash_historyfile is a rather meager 500. If you want to keep history further back than a few weeks or so, you may as well bump this up by explicitly setting$HISTSIZEto a much larger number in your.bashrc. We can do the same thing with the$HISTFILESIZEvariable.HISTFILESIZE=1000000 HISTSIZE=1000000The
manpage for Bash says thatHISTFILESIZEcan beunsetto stop truncation entirely, but unfortunately this doesn't work in.bashrcfiles due to the order in which variables are set; it's therefore more straightforward to simply set it to a very large number.If you're on a machine with resource constraints, it might be a good idea to occasionally archive old
Don't store specific lines.bash_historyfiles to speed up login and reduce memory footprint.You can prevent commands that start with a space from going into history by setting
$HISTCONTROLtoignorespace. You can also ignore duplicate commands, for example repeatedducalls to watch a file grow, by addingignoredups. There's a shorthand to set both inignoreboth.HISTCONTROL=ignorebothYou might also want to remove the use of certain commands from your history, whether for privacy or readability reasons. This can be done with the
$HISTIGNOREvariable. It's common to use this to excludelscalls, job control builtins likebgandfg, and calls tohistoryitself:HISTIGNORE='ls:bg:fg:history'Record timestampsIf you set
$HISTTIMEFORMATto something useful, Bash will record the timestamp of each command in its history. In this variable you can specify the format in which you want this timestamp displayed when viewed withhistory. I find the full date and time to be useful, because it can be sorted easily and works well with tools likecutandawk.HISTTIMEFORMAT='%F %T 'Use one command per lineTo make your
.bash_historyfile a little easier to parse, you can force commands that you entered on more than one line to be adjusted to fit on only one with thecmdhistoption:shopt -s cmdhistStore history immediatelyBy default, Bash only records a session to the
.bash_historyfile on disk when the session terminates. This means that if you crash or your session terminates improperly, you lose the history up to that point. You can fix this by recording each line of history as you issue it, through the$PROMPT_COMMANDvariable:PROMPT_COMMAND='history -a'
|
|
|
Aug 16, 2012 | sanctum.geek.nz
Setting the Bash option
histexpandallows some convenient typing shortcuts using Bash history expansion . The option can be set with either of these:$ set -H $ set -o histexpandIt's likely that this option is already set for all interactive shells, as it's on by default. The manual,
man bash, describes these features as follows:-H Enable ! style history substitution. This option is on by default when the shell is interactive.You may have come across this before, perhaps to your annoyance, in the following error message that comes up whenever
!is used in a double-quoted string, or without being escaped with a backslash:$ echo "Hi, this is Tom!" bash: !": event not foundIf you don't want the feature and thereby make
!into a normal character, it can be disabled with either of these:$ set +H $ set +o histexpandHistory expansion is actually a very old feature of shells, having been available in
cshbefore Bash usage became common.This article is a good followup to Better Bash history , which among other things explains how to include dates and times in
Basic history expansionhistoryoutput, as these examples do.Perhaps the best known and most useful of these expansions is using
!!to refer to the previous command. This allows repeating commands quickly, perhaps to monitor the progress of a long process, such as disk space being freed while deleting a large file:$ rm big_file & [1] 23608 $ du -sh . 3.9G . $ !! du -sh . 3.3G .It can also be useful to specify the full filesystem path to programs that aren't in your
$PATH:$ hdparm -bash: hdparm: command not found $ /sbin/!! /sbin/hdparmIn each case, note that the command itself is printed as expanded, and then run to print the output on the following line.
History by absolute indexHowever,
!!is actually a specific example of a more general form of history expansion. For example, you can supply the history item number of a specific command to repeat it, after looking it up withhistory:$ history | grep expand 3951 2012-08-16 15:58:53 set -o histexpand $ !3951 set -o histexpandYou needn't enter the
!3951on a line by itself; it can be included as any part of the command, for example to add a prefix likesudo:$ sudo !3850If you include the escape string
History by relative index\!as part of your Bash prompt , you can include the current command number in the prompt before the command, making repeating commands by index a lot easier as long as they're still visible on the screen.It's also possible to refer to commands relative to the current command. To subtitute the second-to-last command, we can type
!-2. For example, to check whether truncating a file withsedworked correctly:$ wc -l bigfile.txt 267 bigfile.txt $ printf '%s\n' '11,$d' w | ed -s bigfile.txt $ !-2 wc -l bigfile.txt 10 bigfile.txtThis works further back into history, with
Expanding for historical arguments!-3,!-4, and so on.In each of the above cases, we're substituting for the whole command line. There are also ways to get specific tokens, or words , from the command if we want that. To get the first argument of a particular command in the history, use the
!^token:$ touch a.txt b.txt c.txt $ ls !^ ls a.txt a.txtTo get the last argument, add
!$:$ touch a.txt b.txt c.txt $ ls !$ ls c.txt c.txtTo get all arguments (but not the command itself), use
!*:$ touch a.txt b.txt c.txt $ ls !* ls a.txt b.txt c.txt a.txt b.txt c.txtThis last one is particularly handy when performing several operations on a group of files; we could run
duandwcover them to get their size and character count, and then perhaps decide to delete them based on the output:$ du a.txt b.txt c.txt 4164 a.txt 5184 b.txt 8356 c.txt $ wc !* wc a.txt b.txt c.txt 16689 94038 4250112 a.txt 20749 117100 5294592 b.txt 33190 188557 8539136 c.txt 70628 399695 18083840 total $ rm !* rm a.txt b.txt c.txtThese work not just for the preceding command in history, but also absolute and relative command numbers:
$ history 3 3989 2012-08-16 16:30:59 wc -l b.txt 3990 2012-08-16 16:31:05 du -sh c.txt 3991 2012-08-16 16:31:12 history 3 $ echo !3989^ echo -l -l $ echo !3990$ echo c.txt c.txt $ echo !-1* echo c.txt c.txtMore generally, you can use the syntax
!n:wto refer to any specific argument in a history item by number. In this case, the first word, usually a command or builtin, is word0:$ history | grep bash 4073 2012-08-16 20:24:53 man bash $ !4073:0 man What manual page do you want? $ !4073:1 bashYou can even select ranges of words by separating their indices with a hyphen:
$ history | grep apt-get 3663 2012-08-15 17:01:30 sudo apt-get install gnome $ !3663:0-1 purge !3663:3 sudo apt-get purge gnomeYou can include
Expanding history by string^and$as start and endpoints for these ranges, too.3*is a shorthand for3-$, meaning "all arguments from the third to the last."You can also refer to a previous command in the history that starts with a specific string with the syntax
!string:$ !echo echo c.txt c.txt $ !history history 3 4011 2012-08-16 16:38:28 rm a.txt b.txt c.txt 4012 2012-08-16 16:42:48 echo c.txt 4013 2012-08-16 16:42:51 history 3If you want to match any part of the command line, not just the start, you can use
!?string?:$ !?bash? man bashBe careful when using these, if you use them at all. By default it will run the most recent command matching the string immediately , with no prompting, so it might be a problem if it doesn't match the command you expect.
Checking history expansions before runningIf you're paranoid about this, Bash allows you to audit the command as expanded before you enter it, with the
histverifyoption:$ shopt -s histverify $ !rm $ rm a.txt b.txt c.txtThis option works for any history expansion, and may be a good choice for more cautious administrators. It's a good thing to add to one's
.bashrcif so.If you don't need this set all the time, but you do have reservations at some point about running a history command, you can arrange to print the command without running it by adding a
:psuffix:$ !rm:p rm important-fileIn this instance, the command was expanded, but thankfully not actually run.
Substituting strings in history expansionsTo get really in-depth, you can also perform substitutions on arbitrary commands from the history with
!!:gs/pattern/replacement/. This is getting pretty baroque even for Bash, but it's possible you may find it useful at some point:$ !!:gs/txt/mp3/ rm a.mp3 b.mp3 c.mp3If you only want to replace the first occurrence, you can omit the
g:$ !!:s/txt/mp3/ rm a.mp3 b.txt c.txtStripping leading directories or trailing filesIf you want to chop a filename off a long argument to work with the directory, you can do this by adding an
:hsuffix, kind of like adirnamecall in Perl:$ du -sh /home/tom/work/doc.txt $ cd !$:h cd /home/tom/workTo do the opposite, like a
basenamecall in Perl, use:t:$ ls /home/tom/work/doc.txt $ document=!$:t document=doc.txtStripping extensions or base namesA bit more esoteric, but still possibly useful; to strip a file's extension, use
:r:$ vi /home/tom/work/doc.txt $ stripext=!$:r stripext=/home/tom/work/docTo do the opposite, to get only the extension, use
:e:$ vi /home/tom/work/doc.txt $ extonly=!$:e extonly=.txtQuoting historyIf you're performing substitution not to execute a command or fragment but to use it as a string, it's likely you'll want to quote it. For example, if you've just found through experiment and trial and error an ideal
ffmpegcommand line to accomplish some task, you might want to save it for later use by writing it to a script:$ ffmpeg -f alsa -ac 2 -i hw:0,0 -f x11grab -r 30 -s 1600x900 \ > -i :0.0+1600,0 -acodec pcm_s16le -vcodec libx264 -preset ultrafast \ > -crf 0 -threads 0 "$(date +%Y%m%d%H%M%S)".mkvTo make sure all the escaping is done correctly, you can write the command into the file with the
:qmodifier:$ echo '#!/usr/bin/env bash' >ffmpeg.sh $ echo !ffmpeg:q >>ffmpeg.shIn this case, this will prevent Bash from executing the command expansion
"$(date ... )", instead writing it literally to the file as desired. If you build a lot of complex commands interactively that you later write to scripts once completed, this feature is really helpful and saves a lot of cutting and pasting.Thanks to commenter Mihai Maruseac for pointing out a bug in the examples.
|
|
|
Nov 07, 2014 | sanctum.geek.nz
The common default of some variant of
\h:\w\$for a Bash promptPS1string includes the\wescape character, so that the user's current working directory appears in the prompt, but with$HOMEshortened to a tilde:tom@sanctum:~$ tom@sanctum:~/Documents$ tom@sanctum:/usr/local/nagios$This is normally very helpful, particularly if you leave your shell for a time and forget where you are, though of course you can always call the
pwdshell builtin. However it can get annoying for very deep directory hierarchies, particularly if you're using a smaller terminal window:tom@sanctum:/chroot/apache/usr/local/perl/app-library/lib/App/Library/Class:~$If you're using Bash version 4.0 or above (
bash --version), you can save a bit of terminal space by setting thePROMPT_DIRTRIMvariable for the shell. This limits the length of the tail end of the\wand\Wexpansions to that number of path elements:tom@sanctum:/chroot/apache/usr/local/app-library/lib/App/Library/Class$ PROMPT_DIRTRIM=3 tom@sanctum:.../App/Library/Class$This is a good thing to include in your
~/.bashrcfile if you often find yourself deep in directory trees where the upper end of the hierarchy isn't of immediate interest to you. You can remove the effect again by unsetting the variable:tom@sanctum:.../App/Library/Class$ unset PROMPT_DIRTRIM tom@sanctum:/chroot/apache/usr/local/app-library/lib/App/Library/Class$
|
|
|
Oct 25, 2017 | linuxconfig.org
Trap syntax is very simple and easy to understand: first we must call the trap builtin, followed by the action(s) to be executed, then we must specify the signal(s) we want to react to:
trap [-lp] [[arg] sigspec]Let's see what the possibletrapoptions are for.When used with the
-lflag, the trap command will just display a list of signals associated with their numbers. It's the same output you can obtain running thekill -lcommand:$ trap -l 1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 7) SIGBUS 8) SIGFPE 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT 17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP 21) SIGTTIN 22) SIGTTOU 23) SIGURG 24) SIGXCPU 25) SIGXFSZ 26) SIGVTALRM 27) SIGPROF 28) SIGWINCH 29) SIGIO 30) SIGPWR 31) SIGSYS 34) SIGRTMIN 35) SIGRTMIN+1 36) SIGRTMIN+2 37) SIGRTMIN+3 38) SIGRTMIN+4 39) SIGRTMIN+5 40) SIGRTMIN+6 41) SIGRTMIN+7 42) SIGRTMIN+8 43) SIGRTMIN+9 44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9 56) SIGRTMAX-8 57) SIGRTMAX-7 58) SIGRTMAX-6 59) SIGRTMAX-5 60) SIGRTMAX-4 61) SIGRTMAX-3 62) SIGRTMAX-2 63) SIGRTMAX-1 64) SIGRTMAXIt's really important to specify that it's possible to react only to signals which allows the script to respond: theSIGKILLandSIGSTOPsignals cannot be caught, blocked or ignored.Apart from signals, traps can also react to some
pseudo-signalsuch as EXIT, ERR or DEBUG, but we will see them in detail later. For now just remember that a signal can be specified either by its number or by its name, even without theSIGprefix.About the
-poption now. This option has sense only when a command is not provided (otherwise it will produce an error). When trap is used with it, a list of the previously set traps will be displayed. If the signal name or number is specified, only the trap set for that specific signal will be displayed, otherwise no distinctions will be made, and all the traps will be displayed:$ trap 'echo "SIGINT caught!"' SIGINTWe set a trap to catch the SIGINT signal: it will just display the "SIGINT caught" message onscreen when given signal will be received by the shell. If we now use trap with the -p option, it will display the trap we just defined:$ trap -p trap -- 'echo "SIGINT caught!"' SIGINTBy the way, the trap is now "active", so if we send a SIGINT signal, either using the kill command, or with the CTRL-c shortcut, the associated command in the trap will be executed (^C is just printed because of the key combination):^CSIGINT caught!Trap in action We now will write a simple script to show trap in action, here it is:#!/usr/bin/env bash # # A simple script to demonstrate how trap works # set -e set -u set -o pipefail trap 'echo "signal caught, cleaning..."; rm -i linux_tarball.tar.xz' SIGINT SIGTERM echo "Downloading tarball..." wget -O linux_tarball.tar.xz https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.13.5.tar.xz &> /dev/nullThe above script just tries to download the latest linux kernel tarball into the directory from what it is launched usingwget. During the task, if the SIGINT or SIGTERM signals are received (notice how you can specify more than one signal on the same line), the partially downloaded file will be deleted.In this case the command are actually two: the first is the
echowhich prints the message onscreen, and the second is the actualrmcommand (we provided the -i option to it, so it will ask user confirmation before removing), and they are separated by a semicolon. Instead of specifying commands this way, you can also call functions: this would give you more re-usability. Notice that if you don't provide any command the signal(s) will just be ignored!This is the output of the script above when it receives a SIGINT signal:
$ ./fetchlinux.sh Downloading tarball... ^Csignal caught, cleaning... rm: remove regular file 'linux_tarball.tar.xz'?A very important thing to remember is that when a script is terminated by a signal, like above, its exist status will be the result of128 + the signal number. As you can see, the script above, being terminated by a SIGINT, has an exit status of130:$ echo $? 130Lastly, you can disable a trap just by callingtrapfollowed by the-sign, followed by the signal(s) name or number:trap - SIGINT SIGTERMThe signals will take back the value they had upon the entrance to shell. Pseudo-signals As already mentioned above, trap can be set not only for signals which allows the script to respond but also to what we can call "pseudo-signals". They are not technically signals, but correspond to certain situations that can be specified: EXIT WhenEXITis specified in a trap, the command of the trap will be execute on exit from the shell. ERR This will cause the argument of the trap to be executed when a command returns a non-zero exit status, with some exceptions (the same of the shell errexit option): the command must not be part of awhileoruntilloop; it must not be part of anifconstruct, nor part of a&&or||list, and its value must not be inverted by using the!operator. DEBUG This will cause the argument of the trap to be executed before every simple command,for,caseorselectcommands, and before the first command in shell functions RETURN The argument of the trap is executed after a function or a script sourced by usingsourceor the.command.
|
|
|
Oct 20, 2017 | stackoverflow.com
Amit , Jun 7, 2011 at 19:18
I have a couple of variables and I want to check the following condition (written out in words, then my failed attempt at bash scripting):if varA EQUALS 1 AND ( varB EQUALS "t1" OR varB EQUALS "t2" ) then do something done.And in my failed attempt, I came up with:
if (($varA == 1)) && ( (($varB == "t1")) || (($varC == "t2")) ); then scale=0.05 fiBest answer Gilles
What you've written actually almost works (it would work if all the variables were numbers), but it's not an idiomatic way at all.
( )parentheses indicate a subshell . What's inside them isn't an expression like in many other languages. It's a list of commands (just like outside parentheses). These commands are executed in a separate subprocess, so any redirection, assignment, etc. performed inside the parentheses has no effect outside the parentheses.
- With a leading dollar sign,
$( )is a command substitution : there is a command inside the parentheses, and the output from the command is used as part of the command line (after extra expansions unless the substitution is between double quotes, but that's another story ).
{ }braces are like parentheses in that they group commands, but they only influence parsing, not grouping. The programx=2; { x=4; }; echo $xprints 4, whereasx=2; (x=4); echo $xprints 2. (Also braces require spaces around them and a semicolon before closing, whereas parentheses don't. That's just a syntax quirk.)
- With a leading dollar sign,
${VAR}is a parameter expansion , expanding to the value of a variable, with possible extra transformations.
(( ))double parentheses surround an arithmetic instruction , that is, a computation on integers, with a syntax resembling other programming languages. This syntax is mostly used for assignments and in conditionals.
- The same syntax is used in arithmetic expressions
$(( )), which expand to the integer value of the expression.
[[ ]]double brackets surround conditional expressions . Conditional expressions are mostly built on operators such as-n $variableto test if a variable is empty and-e $fileto test if a file exists. There are also string equality operators:"$string1" = "$string2"(beware that the right-hand side is a pattern, e.g.[[ $foo = a* ]]tests if$foostarts withawhile[[ $foo = "a*" ]]tests if$foois exactlya*), and the familiar!,&&and||operators for negation, conjunction and disjunction as well as parentheses for grouping.
- Note that you need a space around each operator (e.g.
[[ "$x" = "$y" ]], not), and a space or a character like[[ "$x"="$y" ]];both inside and outside the brackets (e.g.[[ -n $foo ]], not).[[-n $foo]]
[ ]single brackets are an alternate form of conditional expressions with more quirks (but older and more portable). Don't write any for now; start worrying about them when you find scripts that contain them.This is the idiomatic way to write your test in bash:
if [[ $varA = 1 && ($varB = "t1" || $varC = "t2") ]]; thenIf you need portability to other shells, this would be the way (note the additional quoting and the separate sets of brackets around each individual test):
if [ "$varA" = 1 ] && { [ "$varB" = "t1" ] || [ "$varC" = "t2" ]; }; thenWill Sheppard , Jun 19, 2014 at 11:07
It's better to use==to differentiate the comparison from assigning a variable (which is also=) – Will Sheppard Jun 19 '14 at 11:07Cbhihe , Apr 3, 2016 at 8:05
+1 @WillSheppard for yr reminder of proper style. Gilles, don't you need a semicolon after yr closing curly bracket and before "then" ? I always thoughtif,then,elseandficould not be on the same line... As in:
if [ "$varA" = 1 ] && { [ "$varB" = "t1" ] || [ "$varC" = "t2" ]; }; thenRockallite , Jan 19 at 2:41
Backquotes (` `) are old-style form of command substitution, with some differences: in this form, backslash retains its literal meaning except when followed by$,`, or\, and the first backquote not preceded by a backslash terminates the command substitution; whereas in the$( )form, all characters between the parentheses make up the command, none are treated specially.Peter A. Schneider , Aug 28 at 13:16
You could emphasize that single brackets have completely different semantics inside and outside of double brackets. (Because you start with explicitly pointing out the subshell semantics but then only as an aside mention the grouping semantics as part of conditional expressions. Was confusing to me for a second when I looked at your idiomatic example.) – Peter A. Schneider Aug 28 at 13:16matchew , Jun 7, 2011 at 19:29
very closeif (( $varA == 1 )) && [[ $varB == 't1' || $varC == 't2' ]]; then scale=0.05 fishould work.
breaking it down
(( $varA == 1 ))is an integer comparison where as
$varB == 't1'is a string comparison. otherwise, I am just grouping the comparisons correctly.
Double square brackets delimit a Conditional Expression. And, I find the following to be a good reading on the subject: "(IBM) Demystify test, [, [[, ((, and if-then-else"
Peter A. Schneider , Aug 28 at 13:21
Just to be sure: The quoting in 't1' is unnecessary, right? Because as opposed to arithmetic instructions in double parentheses, where t1 would be a variable, t1 in a conditional expression in double brackets is just a literal string.I.e.,
[[ $varB == 't1' ]]is exactly the same as[[ $varB == t1 ]], right? – Peter A. Schneider Aug 28 at 13:21
|
|
|
Oct 20, 2017 | unix.stackexchange.com
OR in `expr match` up vote down vote favorite
stracktracer , Dec 14, 2015 at 13:54
I'm confused as to why this does not match:
expr match Unauthenticated123 '^(Unauthenticated|Authenticated).*'it outputs 0.
Charles Duffy , Dec 14, 2015 at 18:22
As an aside, if you were using bash for this, the preferred alternative would be the=~operator in[[ ]], ie.[[ Unauthenticated123 =~ ^(Unauthenticated|Authenticated) ]]– Charles Duffy Dec 14 '15 at 18:22Charles Duffy , Dec 14, 2015 at 18:25
...and if you weren't targeting a known/fixed operating system, usingcaserather than a regex match is very much the better practice, since the accepted answer depends on behavior POSIX doesn't define. – Charles Duffy Dec 14 '15 at 18:25Gilles , Dec 14, 2015 at 23:43
See Why does my regular expression work in X but not in Y? – Gilles Dec 14 '15 at 23:43Lambert , Dec 14, 2015 at 14:04
Your command should be:expr match Unauthenticated123 'Unauthenticated\|Authenticated'If you want the number of characters matched.
To have the part of the string (Unauthenticated) returned use:
expr match Unauthenticated123 '\(Unauthenticated\|Authenticated\)'From
info coreutils 'expr invocation':
STRING : REGEX' Perform pattern matching. The arguments are converted to strings and the second is considered to be a (basic, a la GNUgrep') regular expression, with a `^' implicitly prepended. The first argument is then matched against this regular expression.If the match succeeds and REGEX uses `\(' and `\)', the `:' expression returns the part of STRING that matched the subexpression; otherwise, it returns the number of characters matched. If the match fails, the `:' operator returns the null string if `\(' and `\)' are used in REGEX, otherwise 0. Only the first `\( ... \)' pair is relevant to the return value; additional pairs are meaningful only for grouping the regular expression operators. In the regular expression, `\+', `\?', and `\|' are operators which respectively match one or more, zero or one, or separate alternatives. SunOS and other `expr''s treat these as regular characters. (POSIX allows either behavior.) *Note Regular Expression Library: (regex)Top, for details of regular expression syntax. Some examples are in *note Examples of expr::.stracktracer , Dec 14, 2015 at 14:18
Thanks escaping the | worked. Weird, normally I'd expect it if I wanted to match the literal |... – stracktracer Dec 14 '15 at 14:18reinierpost , Dec 14, 2015 at 15:34
Regular expression syntax, including the use of backquoting, is different for different tools. Always look it up. – reinierpost Dec 14 '15 at 15:34Stéphane Chazelas , Dec 14, 2015 at 14:49
Note that bothmatchand\|are GNU extensions (and the behaviour for:(thematchstandard equivalent) when the pattern starts with^varies with implementations). Standardly, you'd do:expr " $string" : " Authenticated" '|' " $string" : " Unauthenticated"The leading space is to avoid problems with values of
$stringthat start with-or areexproperators, but that means it adds one to the number of characters being matched.With GNU
expr, you'd write it:expr + "$string" : 'Authenticated\|Unauthenticated'The
+forces$stringto be taken as a string even if it happens to be aexproperator.exprregular expressions are basic regular expressions which don't have an alternation operator (and where|is not special). The GNU implementation has it as\|though as an extension.If all you want is to check whether
$stringstarts withAuthenticatedorUnauthenticated, you'd better use:case $string in (Authenticated* | Unauthenticated*) do-something esacnetmonk , Dec 14, 2015 at 14:06
$ expr match "Unauthenticated123" '^\(Unauthenticated\|Authenticated\).*'you have to escape with\the parenthesis and the pipe.mikeserv , Dec 14, 2015 at 14:18
and the^may not mean what some would think depending on theexpr. it is implied anyway. – mikeserv Dec 14 '15 at 14:18Stéphane Chazelas , Dec 14, 2015 at 14:34
@mikeserv,matchand\|are GNU extensions anyway. This Q&A seems to be about GNUexpranyway (where^is guaranteed to mean match at the beginning of the string ). – Stéphane Chazelas Dec 14 '15 at 14:34mikeserv , Dec 14, 2015 at 14:49
@StéphaneChazelas - i didn't know they were strictly GNU. i think i remember them being explicitly officially unspecified - but i don't useexprtoo often anyway and didn't know that. thank you. – mikeserv Dec 14 '15 at 14:49Random832 , Dec 14, 2015 at 16:13
It's not "strictly GNU" - it's present in a number of historical implementations (even System V had it, undocumented, though it didn't have the others like substr/length/index), which is why it's explicitly unspecified. I can't find anything about\|being an extension. – Random832 Dec 14 '15 at 16:13
|
|
|
Oct 19, 2017 | www.bashoneliners.com
Kill a process running on port 8080 $ lsof -i :8080 | awk 'NR > 1 {print $2}' | xargs --no-run-if-empty kill-- by Janos on Sept. 1, 2017, 8:31 p.m.
Make a new folder and cd into it. $ mkcd(){ NAME=$1; mkdir -p "$NAME"; cd "$NAME"; }-- by PrasannaNatarajan on Aug. 3, 2017, 6:49 a.m.
Go up to a particular folder $ alias ph='cd ${PWD%/public_html*}/public_html'-- by Jab2870 on July 18, 2017, 6:07 p.m.
ExplanationI work on a lot of websites and often need to go up to the
public_htmlfolder.This command creates an alias so that however many folders deep I am, I will be taken up to the correct folder.
alias ph='....': This creates a shortcut so that when command ph is typed, the part between the quotes is executed
cd ...: This changes directory to the directory specified
PWD: This is a global bash variable that contains the current directory
${...%/public_html*}: This removes/public_htmland anything after it from the specified stringFinally,
/public_htmlat the end is appended onto the string.So, to sum up, when ph is run, we ask bash to change the directory to the current working directory with anything after public_html removed.
Open another terminal at current location $ $TERMINAL & disown-- by Jab2870 on July 18, 2017, 3:04 p.m.
ExplanationOpens another terminal window at the current location.
Use Case
I often cd into a directory and decide it would be useful to open another terminal in the same folder, maybe for an editor or something. Previously, I would open the terminal and repeat the CD command.
I have aliased this command to open so I just type
openand I get a new terminal already in my desired folder.The
& disownpart of the command stops the new terminal from being dependant on the first meaning that you can still use the first and if you close the first, the second will remain open. LimitationsIt relied on you having the $TERMINAL global variable set. If you don't have this set you could easily change it to something like the following:
gnome-terminal & disownorkonsole & disown
Preserve your fingers from cd ..; cd ..; cd..; cd..; $ up(){ DEEP=$1; for i in $(seq 1 ${DEEP:-"1"}); do cd ../; done; }-- by alireza6677 on June 28, 2017, 5:40 p.m.
Generate a sequence of numbers $ echo {01..10}-- by Elkku on March 1, 2015, 12:04 a.m.
ExplanationThis example will print:
01 02 03 04 05 06 07 08 09 10While the original one-liner is indeed IMHO the canonical way to loop over numbers, the brace expansion syntax of Bash 4.x has some kick-ass features such as correct padding of the number with leading zeros. Limitations
The zero-padding feature works only in Bash >=4.
Related one-liners
Generate a sequence of numbers $ for ((i=1; i<=10; ++i)); do echo $i; done-- by Janos on Nov. 4, 2014, 12:29 p.m.
ExplanationThis is similar to
seq, but portable.seqdoes not exist in all systems and is not recommended today anymore. Other variations to emulate various uses withseq:# seq 1 2 10 for ((i=1; i<=10; i+=2)); do echo $i; done # seq -w 5 10 for ((i=5; i<=10; ++i)); do printf '%02d\n' $i; done
Find recent logs that contain the string "Exception" $ find . -name '*.log' -mtime -2 -exec grep -Hc Exception {} \; | grep -v :0$-- by Janos on July 19, 2014, 7:53 a.m.
ExplanationThe
find:
-name '*.log'-- match files ending with.log-mtime -2-- match files modified within the last 2 days-exec CMD ARGS \;-- for each file found, execute command, where{}inARGSwill be replaced with the file's pathThe
grep:
-cis to print the count of the matches instead of the matches themselves-His to print the name of the file, asgrepnormally won't print it when there is only one filename argument- The output lines will be in the format
path:count. Files that didn't match "Exception" will still be printed, with 0 as count- The second
grepfilters the output of the first, excluding lines that end with:0(= the files that didn't contain matches)Extra tips:
- Change "Exception" to the typical relevant failure indicator of your application
- Add
-iforgrepto make the search case insensitive- To make the
findmatch strictly only files, add-type f- Schedule this as a periodic job, and pipe the output to a mailer, for example
| mailx -s 'error counts' yourmail@example.com
Remove offending key from known_hosts file with one swift move $ sed -i 18d .ssh/known_hosts-- by EvaggelosBalaskas on Jan. 16, 2013, 2:29 p.m.
ExplanationUsing sed to remove a specific line.
The
-iparameter is to edit the file in-place. LimitationsThis works as posted in GNU
sed. In BSDsed, the-iflag requires a parameter to use as the suffix of a backup file. You can set it to empty to not use a backup file:
|
|
|
Feb 15, 2010 | stackoverflow.com
assassin , Feb 15, 2010 at 7:02
Is there a way in bash to convert a string into a lower case string?For example, if I have:
a="Hi all"I want to convert it to:
"hi all"ghostdog74 , Feb 15, 2010 at 7:43
The are various ways: tr$ echo "$a" | tr '[:upper:]' '[:lower:]' hi allAWK$ echo "$a" | awk '{print tolower($0)}' hi allBash 4.0$ echo "${a,,}" hi allPerl$ echo "$a" | perl -ne 'print lc' hi allBashlc(){ case "$1" in [A-Z]) n=$(printf "%d" "'$1") n=$((n+32)) printf \\$(printf "%o" "$n") ;; *) printf "%s" "$1" ;; esac } word="I Love Bash" for((i=0;i<${#word};i++)) do ch="${word:$i:1}" lc "$ch" donejangosteve , Jan 14, 2012 at 21:58
Am I missing something, or does your last example (in Bash) actually do something completely different? It works for "ABX", but if you instead makeword="Hi All"like the other examples, it returnsha, nothi all. It only works for the capitalized letters and skips the already-lowercased letters. – jangosteve Jan 14 '12 at 21:58Richard Hansen , Feb 3, 2012 at 18:55
Note that only thetrandawkexamples are specified in the POSIX standard. – Richard Hansen Feb 3 '12 at 18:55Richard Hansen , Feb 3, 2012 at 18:58
tr '[:upper:]' '[:lower:]'will use the current locale to determine uppercase/lowercase equivalents, so it'll work with locales that use letters with diacritical marks. – Richard Hansen Feb 3 '12 at 18:58Adam Parkin , Sep 25, 2012 at 18:01
How does one get the output into a new variable? Ie say I want the lowercased string into a new variable? – Adam Parkin Sep 25 '12 at 18:01Tino , Nov 14, 2012 at 15:39
@Adam:b="$(echo $a | tr '[A-Z]' '[a-z]')"– Tino Nov 14 '12 at 15:39Dennis Williamson , Feb 15, 2010 at 10:31
In Bash 4:To lowercase
$ string="A FEW WORDS" $ echo "${string,}" a FEW WORDS $ echo "${string,,}" a few words $ echo "${string,,[AEIUO]}" a FeW WoRDS $ string="A Few Words" $ declare -l string $ string=$string; echo "$string" a few wordsTo uppercase
$ string="a few words" $ echo "${string^}" A few words $ echo "${string^^}" A FEW WORDS $ echo "${string^^[aeiou]}" A fEw wOrds $ string="A Few Words" $ declare -u string $ string=$string; echo "$string" A FEW WORDSToggle (undocumented, but optionally configurable at compile time)
$ string="A Few Words" $ echo "${string~~}" a fEW wORDS $ string="A FEW WORDS" $ echo "${string~}" a FEW WORDS $ string="a few words" $ echo "${string~}" A few wordsCapitalize (undocumented, but optionally configurable at compile time)
$ string="a few words" $ declare -c string $ string=$string $ echo "$string" A few wordsTitle case:
$ string="a few words" $ string=($string) $ string="${string[@]^}" $ echo "$string" A Few Words $ declare -c string $ string=(a few words) $ echo "${string[@]}" A Few Words $ string="a FeW WOrdS" $ string=${string,,} $ string=${string~} $ echo "$string"To turn off a
declareattribute, use+. For example,declare +c string. This affects subsequent assignments and not the current value.The
declareoptions change the attribute of the variable, but not the contents. The reassignments in my examples update the contents to show the changes.Edit:
Added "toggle first character by word" (
${var~}) as suggested by ghostdog74Edit: Corrected tilde behavior to match Bash 4.3.
ghostdog74 , Feb 15, 2010 at 10:52
there's also${string~}– ghostdog74 Feb 15 '10 at 10:52Hubert Kario , Jul 12, 2012 at 16:48
Quite bizzare, "^^" and ",," operators don't work on non-ASCII characters but "~~" does... Sostring="łódź"; echo ${string~~}will return "ŁÓDŹ", butecho ${string^^}returns "łóDź". Even inLC_ALL=pl_PL.utf-8. That's using bash 4.2.24. – Hubert Kario Jul 12 '12 at 16:48Dennis Williamson , Jul 12, 2012 at 18:20
@HubertKario: That's weird. It's the same for me in Bash 4.0.33 with the same string inen_US.UTF-8. It's a bug and I've reported it. – Dennis Williamson Jul 12 '12 at 18:20Dennis Williamson , Jul 13, 2012 at 0:44
@HubertKario: Tryecho "$string" | tr '[:lower:]' '[:upper:]'. It will probably exhibit the same failure. So the problem is at least partly not Bash's. – Dennis Williamson Jul 13 '12 at 0:44Dennis Williamson , Jul 14, 2012 at 14:27
@HubertKario: The Bash maintainer has acknowledged the bug and stated that it will be fixed in the next release. – Dennis Williamson Jul 14 '12 at 14:27shuvalov , Feb 15, 2010 at 7:13
echo "Hi All" | tr "[:upper:]" "[:lower:]"Richard Hansen , Feb 3, 2012 at 19:00
+1 for not assuming english – Richard Hansen Feb 3 '12 at 19:00Hubert Kario , Jul 12, 2012 at 16:56
@RichardHansen:trdoesn't work for me for non-ACII characters. I do have correct locale set and locale files generated. Have any idea what could I be doing wrong? – Hubert Kario Jul 12 '12 at 16:56wasatchwizard , Oct 23, 2014 at 16:42
FYI: This worked on Windows/Msys. Some of the other suggestions did not. – wasatchwizard Oct 23 '14 at 16:42Ignacio Vazquez-Abrams , Feb 15, 2010 at 7:03
tr :a="$(tr [A-Z] [a-z] <<< "$a")"AWK :{ print tolower($0) }sed :y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/Sandeepan Nath , Feb 2, 2011 at 11:12
+1a="$(tr [A-Z] [a-z] <<< "$a")"looks easiest to me. I am still a beginner... – Sandeepan Nath Feb 2 '11 at 11:12Haravikk , Oct 19, 2013 at 12:54
I strongly recommend thesedsolution; I've been working in an environment that for some reason doesn't havetrbut I've yet to find a system withoutsed, plus a lot of the time I want to do this I've just done something else insedanyway so can chain the commands together into a single (long) statement. – Haravikk Oct 19 '13 at 12:54Dennis , Nov 6, 2013 at 19:49
The bracket expressions should be quoted. Intr [A-Z] [a-z] A, the shell may perform filename expansion if there are filenames consisting of a single letter or nullgob is set.tr "[A-Z]" "[a-z]" Awill behave properly. – Dennis Nov 6 '13 at 19:49Haravikk , Jun 15, 2014 at 10:51
@CamiloMartin it's a BusyBox system where I'm having that problem, specifically Synology NASes, but I've encountered it on a few other systems too. I've been doing a lot of cross-platform shell scripting lately, and with the requirement that nothing extra be installed it makes things very tricky! However I've yet to encounter a system withoutsed– Haravikk Jun 15 '14 at 10:51fuz , Jan 31, 2016 at 14:54
Note thattr [A-Z] [a-z]is incorrect in almost all locales. for example, in theen-USlocale,A-Zis actually the intervalAaBbCcDdEeFfGgHh...XxYyZ. – fuz Jan 31 '16 at 14:54nettux443 , May 14, 2014 at 9:36
I know this is an oldish post but I made this answer for another site so I thought I'd post it up here:UPPER -> lower : use python:
b=`echo "print '$a'.lower()" | python`Or Ruby:
b=`echo "print '$a'.downcase" | ruby`Or Perl (probably my favorite):
b=`perl -e "print lc('$a');"`Or PHP:
b=`php -r "print strtolower('$a');"`Or Awk:
b=`echo "$a" | awk '{ print tolower($1) }'`Or Sed:
b=`echo "$a" | sed 's/./\L&/g'`Or Bash 4:
b=${a,,}Or NodeJS if you have it (and are a bit nuts...):
b=`echo "console.log('$a'.toLowerCase());" | node`You could also use
dd(but I wouldn't!):b=`echo "$a" | dd conv=lcase 2> /dev/null`lower -> UPPER
use python:
b=`echo "print '$a'.upper()" | python`Or Ruby:
b=`echo "print '$a'.upcase" | ruby`Or Perl (probably my favorite):
b=`perl -e "print uc('$a');"`Or PHP:
b=`php -r "print strtoupper('$a');"`Or Awk:
b=`echo "$a" | awk '{ print toupper($1) }'`Or Sed:
b=`echo "$a" | sed 's/./\U&/g'`Or Bash 4:
b=${a^^}Or NodeJS if you have it (and are a bit nuts...):
b=`echo "console.log('$a'.toUpperCase());" | node`You could also use
dd(but I wouldn't!):b=`echo "$a" | dd conv=ucase 2> /dev/null`Also when you say 'shell' I'm assuming you mean
bashbut if you can usezshit's as easy asb=$a:lfor lower case and
b=$a:ufor upper case.
JESii , May 28, 2015 at 21:42
Neither the sed command nor the bash command worked for me. – JESii May 28 '15 at 21:42nettux443 , Nov 20, 2015 at 14:33
@JESii both work for me upper -> lower and lower-> upper. I'm using sed 4.2.2 and Bash 4.3.42(1) on 64bit Debian Stretch. – nettux443 Nov 20 '15 at 14:33JESii , Nov 21, 2015 at 17:34
Hi, @nettux443... I just tried the bash operation again and it still fails for me with the error message "bad substitution". I'm on OSX using homebrew's bash: GNU bash, version 4.3.42(1)-release (x86_64-apple-darwin14.5.0) – JESii Nov 21 '15 at 17:34tripleee , Jan 16, 2016 at 11:45
Do not use! All of the examples which generate a script are extremely brittle; if the value ofacontains a single quote, you have not only broken behavior, but a serious security problem. – tripleee Jan 16 '16 at 11:45Scott Smedley , Jan 27, 2011 at 5:37
In zsh:echo $a:uGotta love zsh!
Scott Smedley , Jan 27, 2011 at 5:39
or $a:l for lower case conversion – Scott Smedley Jan 27 '11 at 5:39biocyberman , Jul 24, 2015 at 23:26
Add one more case:echo ${(C)a} #Upcase the first char only– biocyberman Jul 24 '15 at 23:26devnull , Sep 26, 2013 at 15:45
Using GNUsed:sed 's/.*/\L&/'Example:
$ foo="Some STRIng"; $ foo=$(echo "$foo" | sed 's/.*/\L&/') $ echo "$foo" some stringtechnosaurus , Jan 21, 2012 at 10:27
For a standard shell (without bashisms) using only builtins:uppers=ABCDEFGHIJKLMNOPQRSTUVWXYZ lowers=abcdefghijklmnopqrstuvwxyz lc(){ #usage: lc "SOME STRING" -> "some string" i=0 while ([ $i -lt ${#1} ]) do CUR=${1:$i:1} case $uppers in *$CUR*)CUR=${uppers%$CUR*};OUTPUT="${OUTPUT}${lowers:${#CUR}:1}";; *)OUTPUT="${OUTPUT}$CUR";; esac i=$((i+1)) done echo "${OUTPUT}" }And for upper case:
uc(){ #usage: uc "some string" -> "SOME STRING" i=0 while ([ $i -lt ${#1} ]) do CUR=${1:$i:1} case $lowers in *$CUR*)CUR=${lowers%$CUR*};OUTPUT="${OUTPUT}${uppers:${#CUR}:1}";; *)OUTPUT="${OUTPUT}$CUR";; esac i=$((i+1)) done echo "${OUTPUT}" }Dereckson , Nov 23, 2014 at 19:52
I wonder if you didn't let some bashism in this script, as it's not portable on FreeBSD sh: ${1:$...}: Bad substitution – Dereckson Nov 23 '14 at 19:52tripleee , Apr 14, 2015 at 7:09
Indeed; substrings with${var:1:1}are a Bashism. – tripleee Apr 14 '15 at 7:09Derek Shaw , Jan 24, 2011 at 13:53
Regular expressionI would like to take credit for the command I wish to share but the truth is I obtained it for my own use from http://commandlinefu.com . It has the advantage that if you
cdto any directory within your own home folder that is it will change all files and folders to lower case recursively please use with caution. It is a brilliant command line fix and especially useful for those multitudes of albums you have stored on your drive.find . -depth -exec rename 's/(.*)\/([^\/]*)/$1\/\L$2/' {} \;You can specify a directory in place of the dot(.) after the find which denotes current directory or full path.
I hope this solution proves useful the one thing this command does not do is replace spaces with underscores - oh well another time perhaps.
Wadih M. , Nov 29, 2011 at 1:31
thanks for commandlinefu.com – Wadih M. Nov 29 '11 at 1:31John Rix , Jun 26, 2013 at 15:58
This didn't work for me for whatever reason, though it looks fine. I did get this to work as an alternative though: find . -exec /bin/bash -c 'mv {} `tr [A-Z] [a-z] <<< {}`' \; – John Rix Jun 26 '13 at 15:58Tino , Dec 11, 2015 at 16:27
This needsprenamefromperl:dpkg -S "$(readlink -e /usr/bin/rename)"givesperl: /usr/bin/prename– Tino Dec 11 '15 at 16:27c4f4t0r , Aug 21, 2013 at 10:21
In bash 4 you can use typesetExample:
A="HELLO WORLD" typeset -l A=$Acommunity wiki, Jan 16, 2016 at 12:26
Pre Bash 4.0Bash Lower the Case of a string and assign to variable
VARIABLE=$(echo "$VARIABLE" | tr '[:upper:]' '[:lower:]') echo "$VARIABLE"Tino , Dec 11, 2015 at 16:23
No need forechoand pipes: use$(tr '[:upper:]' '[:lower:]' <<<"$VARIABLE")– Tino Dec 11 '15 at 16:23tripleee , Jan 16, 2016 at 12:28
@Tino The here string is also not portable back to really old versions of Bash; I believe it was introduced in v3. – tripleee Jan 16 '16 at 12:28Tino , Jan 17, 2016 at 14:28
@tripleee You are right, it was introduced in bash-2.05b - however that's the oldest bash I was able to find on my systems – Tino Jan 17 '16 at 14:28Bikesh M Annur , Mar 23 at 6:48
You can try thiss="Hello World!" echo $s # Hello World! a=${s,,} echo $a # hello world! b=${s^^} echo $b # HELLO WORLD!ref : http://wiki.workassis.com/shell-script-convert-text-to-lowercase-and-uppercase/
Orwellophile , Mar 24, 2013 at 13:43
For Bash versions earlier than 4.0, this version should be fastest (as it doesn't fork/exec any commands):function string.monolithic.tolower { local __word=$1 local __len=${#__word} local __char local __octal local __decimal local __result for (( i=0; i<__len; i++ )) do __char=${__word:$i:1} case "$__char" in [A-Z] ) printf -v __decimal '%d' "'$__char" printf -v __octal '%03o' $(( $__decimal ^ 0x20 )) printf -v __char \\$__octal ;; esac __result+="$__char" done REPLY="$__result" }technosaurus's answer had potential too, although it did run properly for mee.
Stephen M. Harris , Mar 22, 2013 at 22:42
If using v4, this is baked-in . If not, here is a simple, widely applicable solution. Other answers (and comments) on this thread were quite helpful in creating the code below.# Like echo, but converts to lowercase echolcase () { tr [:upper:] [:lower:] <<< "${*}" } # Takes one arg by reference (var name) and makes it lowercase lcase () { eval "${1}"=\'$(echo ${!1//\'/"'\''"} | tr [:upper:] [:lower:] )\' }Notes:
- Doing:
a="Hi All"and then:lcase awill do the same thing as:a=$( echolcase "Hi All" )- In the lcase function, using
${!1//\'/"'\''"}instead of${!1}allows this to work even when the string has quotes.JaredTS486 , Dec 23, 2015 at 17:37
In spite of how old this question is and similar to this answer by technosaurus . I had a hard time finding a solution that was portable across most platforms (That I Use) as well as older versions of bash. I have also been frustrated with arrays, functions and use of prints, echos and temporary files to retrieve trivial variables. This works very well for me so far I thought I would share. My main testing environments are:
- GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
- GNU bash, version 3.2.57(1)-release (sparc-sun-solaris2.10)
lcs="abcdefghijklmnopqrstuvwxyz" ucs="ABCDEFGHIJKLMNOPQRSTUVWXYZ" input="Change Me To All Capitals" for (( i=0; i<"${#input}"; i++ )) ; do : for (( j=0; j<"${#lcs}"; j++ )) ; do : if [[ "${input:$i:1}" == "${lcs:$j:1}" ]] ; then input="${input/${input:$i:1}/${ucs:$j:1}}" fi done doneSimple C-style for loop to iterate through the strings. For the line below if you have not seen anything like this before this is where I learned this . In this case the line checks if the char ${input:$i:1} (lower case) exists in input and if so replaces it with the given char ${ucs:$j:1} (upper case) and stores it back into input.
input="${input/${input:$i:1}/${ucs:$j:1}}"Gus Neves , May 16 at 10:04
Many answers using external programs, which is not really usingBash.If you know you will have Bash4 available you should really just use the
${VAR,,}notation (it is easy and cool). For Bash before 4 (My Mac still uses Bash 3.2 for example). I used the corrected version of @ghostdog74 's answer to create a more portable version.One you can call
lowercase 'my STRING'and get a lowercase version. I read comments about setting the result to a var, but that is not really portable inBash, since we can't return strings. Printing it is the best solution. Easy to capture with something likevar="$(lowercase $str)".How this works
The way this works is by getting the ASCII integer representation of each char with
printfand thenadding 32ifupper-to->lower, orsubtracting 32iflower-to->upper. Then useprintfagain to convert the number back to a char. From'A' -to-> 'a'we have a difference of 32 chars.Using
printfto explain:$ printf "%d\n" "'a" 97 $ printf "%d\n" "'A" 65
97 - 65 = 32And this is the working version with examples.
Please note the comments in the code, as they explain a lot of stuff:#!/bin/bash # lowerupper.sh # Prints the lowercase version of a char lowercaseChar(){ case "$1" in [A-Z]) n=$(printf "%d" "'$1") n=$((n+32)) printf \\$(printf "%o" "$n") ;; *) printf "%s" "$1" ;; esac } # Prints the lowercase version of a sequence of strings lowercase() { word="$@" for((i=0;i<${#word};i++)); do ch="${word:$i:1}" lowercaseChar "$ch" done } # Prints the uppercase version of a char uppercaseChar(){ case "$1" in [a-z]) n=$(printf "%d" "'$1") n=$((n-32)) printf \\$(printf "%o" "$n") ;; *) printf "%s" "$1" ;; esac } # Prints the uppercase version of a sequence of strings uppercase() { word="$@" for((i=0;i<${#word};i++)); do ch="${word:$i:1}" uppercaseChar "$ch" done } # The functions will not add a new line, so use echo or # append it if you want a new line after printing # Printing stuff directly lowercase "I AM the Walrus!"$'\n' uppercase "I AM the Walrus!"$'\n' echo "----------" # Printing a var str="A StRing WITH mixed sTUFF!" lowercase "$str"$'\n' uppercase "$str"$'\n' echo "----------" # Not quoting the var should also work, # since we use "$@" inside the functions lowercase $str$'\n' uppercase $str$'\n' echo "----------" # Assigning to a var myLowerVar="$(lowercase $str)" myUpperVar="$(uppercase $str)" echo "myLowerVar: $myLowerVar" echo "myUpperVar: $myUpperVar" echo "----------" # You can even do stuff like if [[ 'option 2' = "$(lowercase 'OPTION 2')" ]]; then echo "Fine! All the same!" else echo "Ops! Not the same!" fi exit 0And the results after running this:
$ ./lowerupper.sh i am the walrus! I AM THE WALRUS! ---------- a string with mixed stuff! A STRING WITH MIXED STUFF! ---------- a string with mixed stuff! A STRING WITH MIXED STUFF! ---------- myLowerVar: a string with mixed stuff! myUpperVar: A STRING WITH MIXED STUFF! ---------- Fine! All the same!This should only work for ASCII characters though .
For me it is fine, since I know I will only pass ASCII chars to it.
I am using this for some case-insensitive CLI options, for example.nitinr708 , Jul 8, 2016 at 9:20
To store the transformed string into a variable. Following worked for me -$SOURCE_NAMEto$TARGET_NAMETARGET_NAME="`echo $SOURCE_NAME | tr '[:upper:]' '[:lower:]'`"
|
|
|
Oct 16, 2017 | www.safaribooksonline.com
Indenting Here-Documents Problem
The here-document is great, but it's messing up your shell script's formatting. You want to be able to indent for readability. Solution
Use <<- and then you can use tab characters (only!) at the beginning of lines to indent this portion of your shell script.
$ cat myscript.sh ... grep $1 <<-'EOF' lots of data can go here it's indented with tabs to match the script's indenting but the leading tabs are discarded when read EOF ls ... $DiscussionThe hyphen just after the << is enough to tell bash to ignore the leading tab characters. This is for tab characters only and not arbitrary white space. This is especially important with the
EOFor any other marker designation. If you have spaces there, it will not recognize theEOFas your ending marker, and the "here" data will continue through to the end of the file (swallowing the rest of your script). Therefore, you may want to always left-justify theEOF(or other marker) just to be safe, and let the formatting go on this one line.
|
|
|
Oct 16, 2017 | prefetch.net
The Bourne shell provides here documents to allow block of data to be passed to a process through STDIN. The typical format for a here document is something similar to this:
command <<ARBITRARY_TAG
data to pass 1
data to pass 2
ARBITRARY_TAGThis will send the data between the ARBITRARY_TAG statements to the standard input of the process. In order for this to work, you need to make sure that the data is not indented. If you indent it for readability, you will get a syntax error similar to the following:
./test: line 12: syntax error: unexpected end of file
To allow your here documents to be indented, you can append a "-" to the end of the redirection strings like so:
if [ "${STRING}" = "SOMETHING" ] then somecommand <<-EOF this is a string1 this is a string2 this is a string3 EOF fiYou will need to use tabs to indent the data, but that is a small price to pay for added readability. Nice!
|
|
|
Oct 07, 2017 | www.tecmint.com
... ... ..To enable automatic user logout, we will be using the
TMOUTshell variable, which terminates a user's login shell in case there is no activity for a given number of seconds that you can specify.To enable this globally (system-wide for all users), set the above variable in the
/etc/profileshell initialization file.
|
|
|
Sep 27, 2017 | mywiki.wooledge.org
Bash has several different ways to say we want to do arithmetic instead of string operations. Let's look at them one by one.
The first way is the let command:
$ unset a; a=4+5 $ echo $a 4+5 $ let a=4+5 $ echo $a 9You may use spaces, parentheses and so forth, if you quote the expression:
$ let a='(5+2)*3'For a full list of operators availabile, see help let or the manual.
Next, the actual arithmetic evaluation compound command syntax:
$ ((a=(5+2)*3))This is equivalent to let , but we can also use it as a command , for example in an if statement:
$ if (($a == 21)); then echo 'Blackjack!'; fiOperators such as == , < , > and so on cause a comparison to be performed, inside an arithmetic evaluation. If the comparison is "true" (for example, 10 > 2 is true in arithmetic -- but not in strings!) then the compound command exits with status 0. If the comparison is false, it exits with status 1. This makes it suitable for testing things in a script.
Although not a compound command, an arithmetic substitution (or arithmetic expression ) syntax is also available:
$ echo "There are $(($rows * $columns)) cells"Inside $((...)) is an arithmetic context , just like with ((...)) , meaning we do arithmetic (multiplying things) instead of string manipulations (concatenating $rows , space, asterisk, space, $columns ). $((...)) is also portable to the POSIX shell, while ((...)) is not.
Readers who are familiar with the C programming language might wish to know that ((...)) has many C-like features. Among them are the ternary operator:
$ ((abs = (a >= 0) ? a : -a))and the use of an integer value as a truth value:
$ if ((flag)); then echo "uh oh, our flag is up"; fiNote that we used variables inside ((...)) without prefixing them with $ -signs. This is a special syntactic shortcut that Bash allows inside arithmetic evaluations and arithmetic expressions.
There is one final thing we must mention about ((flag)) . Because the inside of ((...)) is C-like, a variable (or expression) that evaluates to zero will be considered false for the purposes of the arithmetic evaluation. Then, because the evaluation is false, it will exit with a status of 1. Likewise, if the expression inside ((...)) is non-zero , it will be considered true ; and since the evaluation is true, it will exit with status 0. This is potentially very confusing, even to experts, so you should take some time to think about this. Nevertheless, when things are used the way they're intended, it makes sense in the end:
$ flag=0 # no error $ while read line; do > if [[ $line = *err* ]]; then flag=1; fi > done < inputfile $ if ((flag)); then echo "oh no"; fi
|
|
|
Sep 01, 2017 | stackoverflow.com
down vote favorite 234Peter Mortensen , asked Oct 5 '09 at 17:52
How do I iterate through each line of a text file with Bash ?With this script
echo "Start!" for p in (peptides.txt) do echo "${p}" doneI get this output on the screen:
Start! ./runPep.sh: line 3: syntax error near unexpected token `(' ./runPep.sh: line 3: `for p in (peptides.txt)'(Later I want to do something more complicated with $p than just output to the screen.)
The environment variable SHELL is (from env):
SHELL=/bin/bash
/bin/bash --versionoutput:GNU bash, version 3.1.17(1)-release (x86_64-suse-linux-gnu) Copyright (C) 2005 Free Software Foundation, Inc.
cat /proc/versionoutput:Linux version 2.6.18.2-34-default (geeko@buildhost) (gcc version 4.1.2 20061115 (prerelease) (SUSE Linux)) #1 SMP Mon Nov 27 11:46:27 UTC 2006The file peptides.txt contains:
RKEKNVQ IPKKLLQK QYFHQLEKMNVK IPKKLLQK GDLSTALEVAIDCYEK QYFHQLEKMNVKIPENIYR RKEKNVQ VLAKHGKLQDAIN ILGFMK LEDVALQILLBruno De Fraine , answered Oct 5 '09 at 18:00
One way to do it is:while read p; do echo $p done <peptides.txt
Exceptionally, if the loop body may read from standard input , you can open the file using a different file descriptor:
while read -u 10 p; do ... done 10<peptides.txtHere, 10 is just an arbitrary number (different from 0, 1, 2).
Warren Young , answered Oct 5 '09 at 17:54
cat peptides.txt | while read line do # do something with $line here doneStan Graves , answered Oct 5 '09 at 18:18
Option 1a: While loop: Single line at a time: Input redirection#!/bin/bash filename='peptides.txt' echo Start while read p; do echo $p done < $filenameOption 1b: While loop: Single line at a time:
Open the file, read from a file descriptor (in this case file descriptor #4).#!/bin/bash filename='peptides.txt' exec 4<$filename echo Start while read -u4 p ; do echo $p doneOption 2: For loop: Read file into single variable and parse.
This syntax will parse "lines" based on any white space between the tokens. This still works because the given input file lines are single work tokens. If there were more than one token per line, then this method would not work as well. Also, reading the full file into a single variable is not a good strategy for large files.#!/bin/bash filename='peptides.txt' filelines=`cat $filename` echo Start for line in $filelines ; do echo $line donemightypile , answered Oct 4 '13 at 13:30
This is no better than other answers, but is one more way to get the job done in a file without spaces (see comments). I find that I often need one-liners to dig through lists in text files without the extra step of using separate script files.for word in $(cat peptides.txt); do echo $word; doneThis format allows me to put it all in one command-line. Change the "echo $word" portion to whatever you want and you can issue multiple commands separated by semicolons. The following example uses the file's contents as arguments into two other scripts you may have written.
for word in $(cat peptides.txt); do cmd_a.sh $word; cmd_b.py $word; doneOr if you intend to use this like a stream editor (learn sed) you can dump the output to another file as follows.
for word in $(cat peptides.txt); do cmd_a.sh $word; cmd_b.py $word; done > outfile.txtI've used these as written above because I have used text files where I've created them with one word per line. (See comments) If you have spaces that you don't want splitting your words/lines, it gets a little uglier, but the same command still works as follows:
OLDIFS=$IFS; IFS=$'\n'; for line in $(cat peptides.txt); do cmd_a.sh $line; cmd_b.py $line; done > outfile.txt; IFS=$OLDIFSThis just tells the shell to split on newlines only, not spaces, then returns the environment back to what it was previously. At this point, you may want to consider putting it all into a shell script rather than squeezing it all into a single line, though.
Best of luck!
Jahid , answered Jun 9 '15 at 15:09
Use a while loop, like this:while IFS= read -r line; do echo "$line" done <fileNotes:
- If you don't set the
IFSproperly, you will lose indentation.- You should almost always use the -r option with read.
- Don't read lines with
forcodeforester , answered Jan 14 at 3:30
A few more things not covered by other answers: Reading from a delimited file# ':' is the delimiter here, and there are three fields on each line in the file # IFS set below is restricted to the context of `read`, it doesn't affect any other code while IFS=: read -r field1 field2 field3; do # process the fields # if the line has less than three fields, the missing fields will be set to an empty string # if the line has more than three fields, `field3` will get all the values, including the third field plus the delimiter(s) done < input.txtReading from more than one file at a timewhile read -u 3 -r line1 && read -u 4 -r line2; do # process the lines # note that the loop will end when we reach EOF on either of the files, because of the `&&` done 3< input1.txt 4< input2.txtReading a whole file into an array (Bash version 4+)readarray -t my_array < my_fileor
mapfile -t my_array < my_fileAnd then
for line in "${my_array[@]}"; do # process the lines doneAnjul Sharma , answered Mar 8 '16 at 16:10
If you don't want your read to be broken by newline character, use -#!/bin/bash while IFS='' read -r line || [[ -n "$line" ]]; do echo "$line" done < "$1"Then run the script with file name as parameter.
Sine , answered Nov 14 '13 at 14:23
#!/bin/bash # # Change the file name from "test" to desired input file # (The comments in bash are prefixed with #'s) for x in $(cat test.txt) do echo $x donedawg , answered Feb 3 '16 at 19:15
Suppose you have this file:$ cat /tmp/test.txt Line 1 Line 2 has leading space Line 3 followed by blank line Line 5 (follows a blank line) and has trailing space Line 6 has no ending CRThere are four elements that will alter the meaning of the file output read by many Bash solutions:
- The blank line 4;
- Leading or trailing spaces on two lines;
- Maintaining the meaning of individual lines (i.e., each line is a record);
- The line 6 not terminated with a CR.
If you want the text file line by line including blank lines and terminating lines without CR, you must use a while loop and you must have an alternate test for the final line.
Here are the methods that may change the file (in comparison to what
catreturns):1) Lose the last line and leading and trailing spaces:
$ while read -r p; do printf "%s\n" "'$p'"; done </tmp/test.txt 'Line 1' 'Line 2 has leading space' 'Line 3 followed by blank line' '' 'Line 5 (follows a blank line) and has trailing space'(If you do
while IFS= read -r p; do printf "%s\n" "'$p'"; done </tmp/test.txtinstead, you preserve the leading and trailing spaces but still lose the last line if it is not terminated with CR)2) Using process substitution with
catwill reads the entire file in one gulp and loses the meaning of individual lines:$ for p in "$(cat /tmp/test.txt)"; do printf "%s\n" "'$p'"; done 'Line 1 Line 2 has leading space Line 3 followed by blank line Line 5 (follows a blank line) and has trailing space Line 6 has no ending CR'(If you remove the
"from$(cat /tmp/test.txt)you read the file word by word rather than one gulp. Also probably not what is intended...)
The most robust and simplest way to read a file line-by-line and preserve all spacing is:
$ while IFS= read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/test.txt 'Line 1' ' Line 2 has leading space' 'Line 3 followed by blank line' '' 'Line 5 (follows a blank line) and has trailing space ' 'Line 6 has no ending CR'If you want to strip leading and trading spaces, remove the
IFS=part:$ while read -r line || [[ -n $line ]]; do printf "'%s'\n" "$line"; done </tmp/test.txt 'Line 1' 'Line 2 has leading space' 'Line 3 followed by blank line' '' 'Line 5 (follows a blank line) and has trailing space' 'Line 6 has no ending CR'(A text file without a terminating
\n, while fairly common, is considered broken under POSIX. If you can count on the trailing\nyou do not need|| [[ -n $line ]]in thewhileloop.)More at the BASH FAQ
,
Here is my real life example how to loop lines of another program output, check for substrings, drop double quotes from variable, use that variable outside of the loop. I guess quite many is asking these questions sooner or later.##Parse FPS from first video stream, drop quotes from fps variable ## streams.stream.0.codec_type="video" ## streams.stream.0.r_frame_rate="24000/1001" ## streams.stream.0.avg_frame_rate="24000/1001" FPS=unknown while read -r line; do if [[ $FPS == "unknown" ]] && [[ $line == *".codec_type=\"video\""* ]]; then echo ParseFPS $line FPS=parse fi if [[ $FPS == "parse" ]] && [[ $line == *".r_frame_rate="* ]]; then echo ParseFPS $line FPS=${line##*=} FPS="${FPS%\"}" FPS="${FPS#\"}" fi done <<< "$(ffprobe -v quiet -print_format flat -show_format -show_streams -i "$input")" if [ "$FPS" == "unknown" ] || [ "$FPS" == "parse" ]; then echo ParseFPS Unknown frame rate fi echo Found $FPSDeclare variable outside of the loop, set value and use it outside of loop requires done <<< "$(...)" syntax. Application need to be run within a context of current console. Quotes around the command keeps newlines of output stream.
Loop match for substrings then reads name=value pair, splits right-side part of last = character, drops first quote, drops last quote, we have a clean value to be used elsewhere.
|
|
|
Aug 29, 2017 | askubuntu.com
If you actually need the output of the
.bash_historyfile , replacehistorywith
cat ~/.bash_historyin all of the commands below.If you actually want the commands without numbers in front, use this command instead of
history:history | cut -d' ' -f 4-
|
|
|
Jul 29, 2017 | stackoverflow.com
getmizanur , asked Sep 10 '11 at 20:35
Is there any directory bookmarking utility for bash to allow move around faster on the command line?UPDATE
Thanks guys for the feedback however I created my own simple shell script (feel free to modify/expand it)
function cdb() { USAGE="Usage: cdb [-c|-g|-d|-l] [bookmark]" ; if [ ! -e ~/.cd_bookmarks ] ; then mkdir ~/.cd_bookmarks fi case $1 in # create bookmark -c) shift if [ ! -f ~/.cd_bookmarks/$1 ] ; then echo "cd `pwd`" > ~/.cd_bookmarks/"$1" ; else echo "Try again! Looks like there is already a bookmark '$1'" fi ;; # goto bookmark -g) shift if [ -f ~/.cd_bookmarks/$1 ] ; then source ~/.cd_bookmarks/"$1" else echo "Mmm...looks like your bookmark has spontaneously combusted. What I mean to say is that your bookmark does not exist." ; fi ;; # delete bookmark -d) shift if [ -f ~/.cd_bookmarks/$1 ] ; then rm ~/.cd_bookmarks/"$1" ; else echo "Oops, forgot to specify the bookmark" ; fi ;; # list bookmarks -l) shift ls -l ~/.cd_bookmarks/ ; ;; *) echo "$USAGE" ; ;; esac }INSTALL
1./ create a file ~/.cdb and copy the above script into it.
2./ in your ~/.bashrc add the following
if [ -f ~/.cdb ]; then source ~/.cdb fi3./ restart your bash session
USAGE
1./ to create a bookmark
$cd my_project $cdb -c project12./ to goto a bookmark
$cdb -g project13./ to list bookmarks
$cdb -l4./ to delete a bookmark
$cdb -d project15./ where are all my bookmarks stored?
$cd ~/.cd_bookmarksFredrik Pihl , answered Sep 10 '11 at 20:47
Also, have a look at CDPATHA colon-separated list of search paths available to the cd command, similar in function to the $PATH variable for binaries. The $CDPATH variable may be set in the local ~/.bashrc file.
ash$ cd bash-doc bash: cd: bash-doc: No such file or directory bash$ CDPATH=/usr/share/doc bash$ cd bash-doc /usr/share/doc/bash-doc bash$ echo $PWD /usr/share/doc/bash-docand
cd -It's the command-line equivalent of the back button (takes you to the previous directory you were in).
ajreal , answered Sep 10 '11 at 20:41
In bash script/command,
you can usepushdandpopdSave and then change the current directory. With no arguments, pushd exchanges the top two directories.
Usage
cd /abc pushd /xxx <-- save /abc to environment variables and cd to /xxx pushd /zzz pushd +1 <-- cd /xxxpopd is to remove the variable (reverse manner)
fgm , answered Sep 11 '11 at 8:28
bookmarks.sh provides a bookmark management system for the Bash version 4.0+. It can also use a Midnight Commander hotlist.Dmitry Frank , answered Jun 16 '15 at 10:22
Thanks for sharing your solution, and I'd like to share mine as well, which I find more useful than anything else I've came across before.The engine is a great, universal tool: command-line fuzzy finder by Junegunn.
It primarily allows you to "fuzzy-find" files in a number of ways, but it also allows to feed arbitrary text data to it and filter this data. So, the shortcuts idea is simple: all we need is to maintain a file with paths (which are shortcuts), and fuzzy-filter this file. Here's how it looks: we type
cdgcommand (from "cd global", if you like), get a list of our bookmarks, pick the needed one in just a few keystrokes, and press Enter. Working directory is changed to the picked item:It is extremely fast and convenient: usually I just type 3-4 letters of the needed item, and all others are already filtered out. Additionally, of course we can move through list with arrow keys or with vim-like keybindings
Ctrl+j/Ctrl+k.Article with details: Fuzzy shortcuts for your shell .
It is possible to use it for GUI applications as well (via xterm): I use that for my GUI file manager Double Commander . I have plans to write an article about this use case, too.
return42 , answered Feb 6 '15 at 11:56
Inspired by the question and answers here, I added the lines below to my~/.bashrcfile.With this you have a
favdircommand (function) to manage your favorites and a autocompletion function to select an item from these favorites.# --------- # Favorites # --------- __favdirs_storage=~/.favdirs __favdirs=( "$HOME" ) containsElement () { local e for e in "${@:2}"; do [[ "$e" == "$1" ]] && return 0; done return 1 } function favdirs() { local cur local IFS local GLOBIGNORE case $1 in list) echo "favorite folders ..." printf -- ' - %s\n' "${__favdirs[@]}" ;; load) if [[ ! -e $__favdirs_storage ]] ; then favdirs save fi # mapfile requires bash 4 / my OS-X bash vers. is 3.2.53 (from 2007 !!?!). # mapfile -t __favdirs < $__favdirs_storage IFS=$'\r\n' GLOBIGNORE='*' __favdirs=($(< $__favdirs_storage)) ;; save) printf -- '%s\n' "${__favdirs[@]}" > $__favdirs_storage ;; add) cur=${2-$(pwd)} favdirs load if containsElement "$cur" "${__favdirs[@]}" ; then echo "'$cur' allready exists in favorites" else __favdirs+=( "$cur" ) favdirs save echo "'$cur' added to favorites" fi ;; del) cur=${2-$(pwd)} favdirs load local i=0 for fav in ${__favdirs[@]}; do if [ "$fav" = "$cur" ]; then echo "delete '$cur' from favorites" unset __favdirs[$i] favdirs save break fi let i++ done ;; *) echo "Manage favorite folders." echo "" echo "usage: favdirs [ list | load | save | add | del ]" echo "" echo " list : list favorite folders" echo " load : load favorite folders from $__favdirs_storage" echo " save : save favorite directories to $__favdirs_storage" echo " add : add directory to favorites [default pwd $(pwd)]." echo " del : delete directory from favorites [default pwd $(pwd)]." esac } && favdirs load function __favdirs_compl_command() { COMPREPLY=( $( compgen -W "list load save add del" -- ${COMP_WORDS[COMP_CWORD]})) } && complete -o default -F __favdirs_compl_command favdirs function __favdirs_compl() { local IFS=$'\n' COMPREPLY=( $( compgen -W "${__favdirs[*]}" -- ${COMP_WORDS[COMP_CWORD]})) } alias _cd='cd' complete -F __favdirs_compl _cdWithin the last two lines, an alias to change the current directory (with autocompletion) is created. With this alias (
_cd) you are able to change to one of your favorite directories. May you wan't to change this alias to something which fits your needs .With the function
favdirsyou can manage your favorites (see usage).$ favdirs Manage favorite folders. usage: favdirs [ list | load | save | add | del ] list : list favorite folders load : load favorite folders from ~/.favdirs save : save favorite directories to ~/.favdirs add : add directory to favorites [default pwd /tmp ]. del : delete directory from favorites [default pwd /tmp ].Zied , answered Mar 12 '14 at 9:53
Yes there is DirB: Directory Bookmarks for Bash well explained in this Linux Journal articleAn example from the article:
% cd ~/Desktop % s d # save(bookmark) ~/Desktop as d % cd /tmp # go somewhere % pwd /tmp % g d # go to the desktop % pwd /home/DesktopAl Conrad , answered Sep 4 '15 at 16:10
@getmizanur I used your cdb script. I enhanced it slightly by adding bookmarks tab completion. Here's my version of your cdb script._cdb() { local _script_commands=$(ls -1 ~/.cd_bookmarks/) local cur=${COMP_WORDS[COMP_CWORD]} COMPREPLY=( $(compgen -W "${_script_commands}" -- $cur) ) } complete -F _cdb cdb function cdb() { local USAGE="Usage: cdb [-h|-c|-d|-g|-l|-s] [bookmark]\n \t[-h or no args] - prints usage help\n \t[-c bookmark] - create bookmark\n \t[-d bookmark] - delete bookmark\n \t[-g bookmark] - goto bookmark\n \t[-l] - list bookmarks\n \t[-s bookmark] - show bookmark location\n \t[bookmark] - same as [-g bookmark]\n Press tab for bookmark completion.\n" if [ ! -e ~/.cd_bookmarks ] ; then mkdir ~/.cd_bookmarks fi case $1 in # create bookmark -c) shift if [ ! -f ~/.cd_bookmarks/$1 ] ; then echo "cd `pwd`" > ~/.cd_bookmarks/"$1" complete -F _cdb cdb else echo "Try again! Looks like there is already a bookmark '$1'" fi ;; # goto bookmark -g) shift if [ -f ~/.cd_bookmarks/$1 ] ; then source ~/.cd_bookmarks/"$1" else echo "Mmm...looks like your bookmark has spontaneously combusted. What I mean to say is that your bookmark does not exist." ; fi ;; # show bookmark -s) shift if [ -f ~/.cd_bookmarks/$1 ] ; then cat ~/.cd_bookmarks/"$1" else echo "Mmm...looks like your bookmark has spontaneously combusted. What I mean to say is that your bookmark does not exist." ; fi ;; # delete bookmark -d) shift if [ -f ~/.cd_bookmarks/$1 ] ; then rm ~/.cd_bookmarks/"$1" ; else echo "Oops, forgot to specify the bookmark" ; fi ;; # list bookmarks -l) shift ls -1 ~/.cd_bookmarks/ ; ;; -h) echo -e $USAGE ; ;; # goto bookmark by default *) if [ -z "$1" ] ; then echo -e $USAGE elif [ -f ~/.cd_bookmarks/$1 ] ; then source ~/.cd_bookmarks/"$1" else echo "Mmm...looks like your bookmark has spontaneously combusted. What I mean to say is that your bookmark does not exist." ; fi ;; esac }tobimensch , answered Jun 5 '16 at 21:31
Yes, one that I have written, that is called anc.https://github.com/tobimensch/anc
Anc stands for anchor, but anc's anchors are really just bookmarks.
It's designed for ease of use and there're multiple ways of navigating, either by giving a text pattern, using numbers, interactively, by going back, or using [TAB] completion.
I'm actively working on it and open to input on how to make it better.
Allow me to paste the examples from anc's github page here:
# make the current directory the default anchor: $ anc s # go to /etc, then /, then /usr/local and then back to the default anchor: $ cd /etc; cd ..; cd usr/local; anc # go back to /usr/local : $ anc b # add another anchor: $ anc a $HOME/test # view the list of anchors (the default one has the asterisk): $ anc l (0) /path/to/first/anchor * (1) /home/usr/test # jump to the anchor we just added: # by using its anchor number $ anc 1 # or by jumping to the last anchor in the list $ anc -1 # add multiple anchors: $ anc a $HOME/projects/first $HOME/projects/second $HOME/documents/first # use text matching to jump to $HOME/projects/first $ anc pro fir # use text matching to jump to $HOME/documents/first $ anc doc fir # add anchor and jump to it using an absolute path $ anc /etc # is the same as $ anc a /etc; anc -1 # add anchor and jump to it using a relative path $ anc ./X11 #note that "./" is required for relative paths # is the same as $ anc a X11; anc -1 # using wildcards you can add many anchors at once $ anc a $HOME/projects/* # use shell completion to see a list of matching anchors # and select the one you want to jump to directly $ anc pro[TAB]Cảnh Toàn Nguyễn , answered Feb 20 at 5:41
Bashmarks is an amazingly simple and intuitive utility. In short, after installation, the usage is:s <bookmark_name> - Saves the current directory as "bookmark_name" g <bookmark_name> - Goes (cd) to the directory associated with "bookmark_name" p <bookmark_name> - Prints the directory associated with "bookmark_name" d <bookmark_name> - Deletes the bookmark l - Lists all available bookmarks,
For short term shortcuts, I have a the following in my respective init script (Sorry. I can't find the source right now and didn't bother then):function b() { alias $1="cd `pwd -P`" }Usage:
In any directory that you want to bookmark type
b THEDIR # <THEDIR> being the name of your 'bookmark'It will create an alias to cd (back) to here.
To return to a 'bookmarked' dir type
THEDIRIt will run the stored alias and cd back there.
Caution: Use only if you understand that this might override existing shell aliases and what that means.
|
|
|
Jul 29, 2017 | unix.stackexchange.com
Amelio Vazquez-Reina asked May 19 '14I read here that the purpose of
exportin a shell is to make the variable available to sub-processes started from the shell.However, I have also read here and here that "Processes inherit their environment from their parent (the process which started them)."
If this is the case, why do we need
export? What am I missing?Are shell variables not part of the environment by default? What is the difference?
Your assumption is that all shell variables are in the environment . This is incorrect. The
exportcommand is what defines a name to be in the environment at all. Thus:a=1 b=2 export bresults in the current shell knowing that
$aexpands to 1 and$bto 2, but subprocesses will not know anything aboutabecause it is not part of the environment (even in the current shell).Some useful tools:
set: Useful for viewing the current shell's parameters, exported-or-notset -k: Sets assigned args in the environment. Considerf() { set -k; env; }; f a=1export: Tells the shell to put a name in the environment. Export and assignment are two entirely different operations.env: As an external command,envcan only tell you about the inherited environment, thus, it's useful for sanity checking.env -i: Useful for clearing the environment before starting a subprocess.Alternatives to
export:====
name=val command# Assignment before command exports that name to the command.declare/local -x name# Exports name, particularly useful in shell functions when you want to avoid exposing the name to outside scope.There's a difference between shell variables and environment variables. If you define a shell variable without
exporting it, it is not added to the processes environment and thus not inherited to its children.Using
exportyou tell the shell to add the shell variable to the environment. You can test this usingprintenv(which just prints its environment tostdout, since it's a child-process you see the effect ofexporting variables):#!/bin/sh MYVAR="my cool variable" echo "Without export:" printenv | grep MYVAR echo "With export:" export MYVAR printenv | grep MYVARA variable, once exported, is part of the environment.PATHis exported in the shell itself, while custom variables can be exported as needed.... ... ..
|
|
|
Jul 29, 2017 | superuser.com
up vote down vote favorite 1I am using startxto start the graphical environment. I have a very simple.xinitrcwhich I will add things to as I set up the environment, but for now it is as follows:catwm & # Just a basic window manager, for testing. xtermThe reason I background the WM and foreground terminal and not the other way around as often is done, is because I would like to be able to come back to the virtual text console after typing
exitinxterm. This appears to work as described.The problem is that the
PS1variable that currently is set to my preference in/etc/profile.d/user.sh(which is sourced from/etc/profilesupplied by distro), does not appear to propagate to the environment of thextermmentioned above. The relevant process tree is as follows:\_ bash \_ xinit home user /. xinitrc -- etc X11 xinit xserverrc auth tmp serverauth ggJna3I0vx \_ usr bin nolisten tcp auth tmp serverauth ggJna3I0vx vt1 \_ sh home user /. xinitrc \_ home user catwm \_ xterm \_ bashThe shell started by
xtermappears to be interactive, the shell executing.xinitrchowever is not. I am ok with both, the assumptions about interactivity seem to be perfectly valid, but now I have a non-interactive shell that spawns an interactive shell indirectly, and the interactive shell has no chance to automatically inherit the prompt, because the prompt was unset or otherwise made unavailable higher up the process tree.How do I go about getting my prompt back? bash environment-variables sh
down vote accepted
share improve this question edited Oct 21 '13 at 11:39 asked Oct 21 '13 at 9:51 amn 453 12 29 Commands
envandexportlist only variables which are exported.$PS1is usually not exported. Tryecho $PS1in your shell to see actual value of$PS1.Non-interactive shells usually do not have
$PS1. Non-interactivebashexplicitly unsets$PS1. 1 You can check ifbashis interactive byecho $-. If the output containsithen it is interactive. You can explicitly start interactive shell by using the option on the command line:bash -i. Shell started with-cis not interactive.The
/etc/profilescript is read for a login shell. You can start the shell as a login shell by:bash -l.With
bashshell the scripts/etc/bash.bashrcand~/.bashrcare usually used to set$PS1. Those scripts are sourced when interactive non-login shell is started. It is your case in thexterm.See Setting the PS? Strings Permanently
Possible solutions
- Start the shell inside
xtermas a login shellbash -l. Check if/etc/profileand~/.profiledo not contain code which should be executed only after login. Maybe slight modifications of the scripts will be needed.- Use a different shell. For example
dashdoes not unset$PS1. You can use such a shell just as the non-interactive shell which will run the scripts up toxterm.- Give up the strict POSIX compliance and use the bash-standard place for setting
$PS1:/etc/bash.bashrcor~/.bashrc.- Give up the strict POSIX compliance and source your own startup script like:
bash --rcfile <(echo "PS1=$PS1save") -i- Start the intermediate shells from
startxtillxtermas interactive shells (bash -i). Unfortunately this can have some side-effect and I would not do this.
share improve this answer edited Oct 22 '13 at 16:45 answered Oct 21 '13 at 11:19 pabouk 4,250 25 40
I am specifically avoiding to set PS1in.bashrcor/etc/bash.bashrc(which is executed as well), to retain POSIX shell compatibility. These do not set or unsetPS1.PS1is set in/etc/profile.d/user.sh, which is sourced by/etc/profile. Indeed, this file is only executed for login shells, however I do exportPS1from/etc/profile.d/user.shexactly because I want propagation of my preferred value down the process tree. So it shouldn't matter which subshells are login and/or interactive ones then, should it? – amn Oct 21 '13 at 11:32
It seems that bashremoves thePS1variable. What exactly do you want to achieve by "POSIX shell compatibility"? Do you want to be able to replacebashby a different POSIX-compliant shell and retain the same functionality? Based on my testsbashremovesPS1when it is started as non-interactive. I think of two simple solutions: 1. start the shell as a login shell with the-loption (attention for actions in the startup scripts which should be started only at login) 2. start the intermediate shells as interactive with the-ioption. – pabouk Oct 21 '13 at 12:00
I try to follow interfaces and specifications, not implementations - hence POSIX compatibility. That's important (to me). I already have one login shell - the one started by /usr/bin/login. I understand that a non-interactive shell doesn't need prompt, but unsetting a variable is too much - I need the prompt in an interactive shell (spawned and used byxterm) later on. What am I doing wrong? I guess most people set their prompt in.bashrcwhich is sourced by bash anyway, and so the prompt survives. I try to avoid.bashrchowever. – amn Oct 22 '13 at 12:12
@amn: I have added various possible solutions to the reply. – pabouk Oct 22 '13 at 16:46
|
|
|
Jul 29, 2017 | stackoverflow.com
user3718463 , asked Sep 27 '14 at 21:41
The Learning Bash Book mention that a subshell will inherit only environment variabels and file descriptors , ...etc and that it will not inherit variables that are not exported of$ var=15 $ (echo $var) 15 $ ./file # this file include the same command echo $var $As i know the shell will create two subshells for () case and for ./file, but why in () case the subshell identified the var variable although it is not exported and in the ./file case it did not identify it ?
...I tried to use strace to figure out how this happens and surprisingly i found that bash will use the same arguments for the clone system call so this means that the both forked process in () and ./file should have the same process address space of the parent, so why in () case the variable is visible to the subshell and the same does not happen for ./file case although the same arguments is based with clone system call ?
Alfe , answered Sep 27 '14 at 23:16
The subshell created using parentheses does not use anexecve()call for the new process, the calling of the script does. At this point the variables from the parent shell are handled differently: Theexecve()passes a deliberate set of variables (the script-calling case) while not callingexecve()(the parentheses case) leaves the complete set of variables intact.Your probing using
straceshould have shown exactly that difference; if you did not see it, I can only assume that you made one of several possible mistakes. I will just strip down what I did to show the difference, then you can decide for yourself where your error was.... ... ...
Nicolas Albert , answered Sep 27 '14 at 21:43
You have to export yourvarfor child process:export var 15Once exported, the variable is used for all children process at the launch time (not export time).
var 15 export varis same as
export var var 15is same as
export var 15Export can be cancelled using
unset. Sample:unset var.user3718463 , answered Sep 27 '14 at 23:11
The solution for this mystery is that subshells inherit everything from the parent shell including all shell variables because they are simply called with fork or clone so they share the same memory space with the parent shell , that's why this will work$ var=15 $ (echo $var) 15But in the ./file , the subshell will be later followed by exec or execv system call which will clear all the previous parent variables but we still have the environment variables you can check this out using strace using -f to monitor the child subshell and you will find that there is a call to execv
|
|
|
Mar 03, 2014 | www.digitalocean.com
Introduction
When interacting with your server through a shell session, there are many pieces of information that your shell compiles to determine its behavior and access to resources. Some of these settings are contained within configuration settings and others are determined by user input.
One way that the shell keeps track of all of these settings and details is through an area it maintains called the environment . The environment is an area that the shell builds every time that it starts a session that contains variables that define system properties.
In this guide, we will discuss how to interact with the environment and read or set environmental and shell variables interactively and through configuration files. We will be using an Ubuntu 12.04 VPS as an example, but these details should be relevant on any Linux system.
How the Environment and Environmental Variables Work
Every time a shell session spawns, a process takes place to gather and compile information that should be available to the shell process and its child processes. It obtains the data for these settings from a variety of different files and settings on the system.
Basically the environment provides a medium through which the shell process can get or set settings and, in turn, pass these on to its child processes.
The environment is implemented as strings that represent key-value pairs. If multiple values are passed, they are typically separated by colon (:) characters. Each pair will generally will look something like this:
KEY value1 value2:...If the value contains significant white-space, quotations are used:
KEY =" value with spaces "The keys in these scenarios are variables. They can be one of two types, environmental variables or shell variables.
Environmental variables are variables that are defined for the current shell and are inherited by any child shells or processes. Environmental variables are used to pass information into processes that are spawned from the shell.
Shell variables are variables that are contained exclusively within the shell in which they were set or defined. They are often used to keep track of ephemeral data, like the current working directory.
By convention, these types of variables are usually defined using all capital letters. This helps users distinguish environmental variables within other contexts.
Printing Shell and Environmental Variables
Each shell session keeps track of its own shell and environmental variables. We can access these in a few different ways.
We can see a list of all of our environmental variables by using the
envorprintenvcommands. In their default state, they should function exactly the same:printenv
SHELL=/bin/bash TERM=xterm USER=demouser LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca:... MAIL=/var/mail/demouser PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games PWD=/home/demouser LANG=en_US.UTF-8 SHLVL=1 HOME=/home/demouser LOGNAME=demouser LESSOPEN=| /usr/bin/lesspipe %s LESSCLOSE=/usr/bin/lesspipe %s %s _=/usr/bin/printenvThis is fairly typical of the output of both
printenvandenv. The difference between the two commands is only apparent in their more specific functionality. For instance, withprintenv, you can requests the values of individual variables:printenv SHELL
/bin/bashOn the other hand,
envlet's you modify the environment that programs run in by passing a set of variable definitions into a command like this:env VAR1="blahblah" command_to_run command_optionsSince, as we learned above, child processes typically inherit the environmental variables of the parent process, this gives you the opportunity to override values or add additional variables for the child.
As you can see from the output of our
printenvcommand, there are quite a few environmental variables set up through our system files and processes without our input.These show the environmental variables, but how do we see shell variables?
The
setcommand can be used for this. If we typesetwithout any additional parameters, we will get a list of all shell variables, environmental variables, local variables, and shell functions:set
BASH=/bin/bash BASHOPTS=checkwinsize:cmdhist:expand_aliases:extglob:extquote:force_fignore:histappend:interactive_comments:login_shell:progcomp:promptvars:sourcepath BASH_ALIASES=() BASH_ARGC=() BASH_ARGV=() BASH_CMDS=() . . .This is usually a huge list. You probably want to pipe it into a pager program to deal with the amount of output easily:
set | lessThe amount of additional information that we receive back is a bit overwhelming. We probably do not need to know all of the bash functions that are defined, for instance.
We can clean up the output by specifying that
setshould operate in POSIX mode, which won't print the shell functions. We can execute this in a sub-shell so that it does not change our current environment:(set -o posix; set)This will list all of the environmental and shell variables that are defined.
We can attempt to compare this output with the output of the
envorprintenvcommands to try to get a list of only shell variables, but this will be imperfect due to the different ways that these commands output information:comm -23 <(set -o posix; set | sort) <(env | sort)This will likely still include a few environmental variables, due to the fact that the
setcommand outputs quoted values, while theprintenvandenvcommands do not quote the values of strings.This should still give you a good idea of the environmental and shell variables that are set in your session.
These variables are used for all sorts of things. They provide an alternative way of setting persistent values for the session between processes, without writing changes to a file.
Common Environmental and Shell Variables
Some environmental and shell variables are very useful and are referenced fairly often.
Here are some common environmental variables that you will come across:
- SHELL : This describes the shell that will be interpreting any commands you type in. In most cases, this will be bash by default, but other values can be set if you prefer other options.
- TERM : This specifies the type of terminal to emulate when running the shell. Different hardware terminals can be emulated for different operating requirements. You usually won't need to worry about this though.
- USER : The current logged in user.
- PWD : The current working directory.
- OLDPWD : The previous working directory. This is kept by the shell in order to switch back to your previous directory by running
cd -.- LS_COLORS : This defines color codes that are used to optionally add colored output to the
lscommand. This is used to distinguish different file types and provide more info to the user at a glance.- MAIL : The path to the current user's mailbox.
- PATH : A list of directories that the system will check when looking for commands. When a user types in a command, the system will check directories in this order for the executable.
- LANG : The current language and localization settings, including character encoding.
- HOME : The current user's home directory.
- : The most recent previously executed command.
In addition to these environmental variables, some shell variables that you'll often see are:
Setting Shell and Environmental Variables
- BASHOPTS : The list of options that were used when bash was executed. This can be useful for finding out if the shell environment will operate in the way you want it to.
- BASH_VERSION : The version of bash being executed, in human-readable form.
- BASH_VERSINFO : The version of bash, in machine-readable output.
- COLUMNS : The number of columns wide that are being used to draw output on the screen.
- DIRSTACK : The stack of directories that are available with the
pushdandpopdcommands.- HISTFILESIZE : Number of lines of command history stored to a file.
- HISTSIZE : Number of lines of command history allowed in memory.
- HOSTNAME : The hostname of the computer at this time.
- IFS : The internal field separator to separate input on the command line. By default, this is a space.
- PS1 : The primary command prompt definition. This is used to define what your prompt looks like when you start a shell session. The
PS2is used to declare secondary prompts for when a command spans multiple lines.- SHELLOPTS : Shell options that can be set with the
setoption.- UID : The UID of the current user.
To better understand the difference between shell and environmental variables, and to introduce the syntax for setting these variables, we will do a small demonstration.
Creating Shell Variables
We will begin by defining a shell variable within our current session. This is easy to accomplish; we only need to specify a name and a value. We'll adhere to the convention of keeping all caps for the variable name, and set it to a simple string.
TEST_VAR='Hello World!'Here, we've used quotations since the value of our variable contains a space. Furthermore, we've used single quotes because the exclamation point is a special character in the bash shell that normally expands to the bash history if it is not escaped or put into single quotes.
We now have a shell variable. This variable is available in our current session, but will not be passed down to child processes.
We can see this by grepping for our new variable within the
setoutput:set | grep TEST_VAR
TEST_VAR='Hello World!'We can verify that this is not an environmental variable by trying the same thing with
printenv:printenv | grep TEST_VARNo out should be returned.
Let's take this as an opportunity to demonstrate a way of accessing the value of any shell or environmental variable.
echo $TEST_VAR
Hello World!As you can see, reference the value of a variable by preceding it with a
$sign. The shell takes this to mean that it should substitute the value of the variable when it comes across this.So now we have a shell variable. It shouldn't be passed on to any child processes. We can spawn a new bash shell from within our current one to demonstrate:
bash echo $TEST_VARIf we type
bashto spawn a child shell, and then try to access the contents of the variable, nothing will be returned. This is what we expected.Get back to our original shell by typing
exit:Creating Environmental Variablesexit
Now, let's turn our shell variable into an environmental variable. We can do this by exporting the variable. The command to do so is appropriately named:
export TEST_VARThis will change our variable into an environmental variable. We can check this by checking our environmental listing again:
printenv | grep TEST_VAR
TEST_VAR=Hello World!This time, our variable shows up. Let's try our experiment with our child shell again:
bash echo $TEST_VAR
Hello World!Great! Our child shell has received the variable set by its parent. Before we exit this child shell, let's try to export another variable. We can set environmental variables in a single step like this:
export NEW_VAR="Testing export"Test that it's exported as an environmental variable:
printenv | grep NEW_VAR
NEW_VAR=Testing exportNow, let's exit back into our original shell:
exitLet's see if our new variable is available:
echo $NEW_VARNothing is returned.
This is because environmental variables are only passed to child processes. There isn't a built-in way of setting environmental variables of the parent shell. This is good in most cases and prevents programs from affecting the operating environment from which they were called.
The
Demoting and Unsetting VariablesNEW_VARvariable was set as an environmental variable in our child shell. This variable would be available to itself and any of its child shells and processes. When we exited back into our main shell, that environment was destroyed.
We still have our
TEST_VARvariable defined as an environmental variable. We can change it back into a shell variable by typing:export -n TEST_VARIt is no longer an environmental variable:
printenv | grep TEST_VARHowever, it is still a shell variable:
set | grep TEST_VAR
TEST_VAR='Hello World!'If we want to completely unset a variable, either shell or environmental, we can do so with the
unsetcommand:unset TEST_VARWe can verify that it is no longer set:
echo $TEST_VARNothing is returned because the variable has been unset.
Setting Environmental Variables at Login
We've already mentioned that many programs use environmental variables to decide the specifics of how to operate. We do not want to have to set important variables up every time we start a new shell session, and we have already seen how many variables are already set upon login, so how do we make and define variables automatically?
This is actually a more complex problem than it initially seems, due to the numerous configuration files that the bash shell reads depending on how it is started.
The Difference between Login, Non-Login, Interactive, and Non-Interactive Shell Sessions
The bash shell reads different configuration files depending on how the session is started.
One distinction between different sessions is whether the shell is being spawned as a "login" or "non-login" session.
A login shell is a shell session that begins by authenticating the user. If you are signing into a terminal session or through SSH and authenticate, your shell session will be set as a "login" shell.
If you start a new shell session from within your authenticated session, like we did by calling the
bashcommand from the terminal, a non-login shell session is started. You were were not asked for your authentication details when you started your child shell.Another distinction that can be made is whether a shell session is interactive, or non-interactive.
An interactive shell session is a shell session that is attached to a terminal. A non-interactive shell session is one is not attached to a terminal session.
So each shell session is classified as either login or non-login and interactive or non-interactive.
A normal session that begins with SSH is usually an interactive login shell. A script run from the command line is usually run in a non-interactive, non-login shell. A terminal session can be any combination of these two properties.
Whether a shell session is classified as a login or non-login shell has implications on which files are read to initialize the shell session.
A session started as a login session will read configuration details from the
/etc/profilefile first. It will then look for the first login shell configuration file in the user's home directory to get user-specific configuration details.It reads the first file that it can find out of
~/.bash_profile,~/.bash_login, and~/.profileand does not read any further files.In contrast, a session defined as a non-login shell will read
/etc/bash.bashrcand then the user-specific~/.bashrcfile to build its environment.Non-interactive shells read the environmental variable called
Implementing Environmental VariablesBASH_ENVand read the file specified to define the new environment.
As you can see, there are a variety of different files that we would usually need to look at for placing our settings.
This provides a lot of flexibility that can help in specific situations where we want certain settings in a login shell, and other settings in a non-login shell. However, most of the time we will want the same settings in both situations.
Fortunately, most Linux distributions configure the login configuration files to source the non-login configuration files. This means that you can define environmental variables that you want in both inside the non-login configuration files. They will then be read in both scenarios.
We will usually be setting user-specific environmental variables, and we usually will want our settings to be available in both login and non-login shells. This means that the place to define these variables is in the
~/.bashrcfile.Open this file now:
nano ~/.bashrcThis will most likely contain quite a bit of data already. Most of the definitions here are for setting bash options, which are unrelated to environmental variables. You can set environmental variables just like you would from the command line:
export VARNAME=valueWe can then save and close the file. The next time you start a shell session, your environmental variable declaration will be read and passed on to the shell environment. You can force your current session to read the file now by typing:
source ~/.bashrcIf you need to set system-wide variables, you may want to think about adding them to
Conclusion/etc/profile,/etc/bash.bashrc, or/etc/environment.
Environmental and shell variables are always present in your shell sessions and can be very useful. They are an interesting way for a parent process to set configuration details for its children, and are a way of setting options outside of files.
This has many advantages in specific situations. For instance, some deployment mechanisms rely on environmental variables to configure authentication information. This is useful because it does not require keeping these in files that may be seen by outside parties.
There are plenty of other, more mundane, but more common scenarios where you will need to read or alter the environment of your system. These tools and techniques should give you a good foundation for making these changes and using them correctly.
By Justin Ellingwood
|
|
|
Jul 29, 2017 | stackoverflow.com
up vote 130 down vote favorite 717
Adam Rosenfield , asked Jan 6 '09 at 3:58
I've used a number of different *nix-based systems of the years, and it seems like every flavor of Bash I use has a different algorithm for deciding which startup scripts to run. For the purposes of tasks like setting up environment variables and aliases and printing startup messages (e.g. MOTDs), which startup script is the appropriate place to do these?What's the difference between putting things in
.bashrc,.bash_profile, and.environment? I've also seen other files such as.login,.bash_login, and.profile; are these ever relevant? What are the differences in which ones get run when logging in physically, logging in remotely via ssh, and opening a new terminal window? Are there any significant differences across platforms (including Mac OS X (and its Terminal.app) and Cygwin Bash)?Cos , answered Jan 6 '09 at 4:18
The main difference with shell config files is that some are only read by "login" shells (eg. when you login from another host, or login at the text console of a local unix machine). these are the ones called, say,.loginor.profileor.zlogin(depending on which shell you're using).Then you have config files that are read by "interactive" shells (as in, ones connected to a terminal (or pseudo-terminal in the case of, say, a terminal emulator running under a windowing system). these are the ones with names like
.bashrc,.tcshrc,.zshrc, etc.
bashcomplicates this in that.bashrcis only read by a shell that's both interactive and non-login , so you'll find most people end up telling their.bash_profileto also read.bashrcwith something like
[[ -r ~/.bashrc ]] && . ~/.bashrcOther shells behave differently - eg with
zsh,.zshrcis always read for an interactive shell, whether it's a login one or not.The manual page for bash explains the circumstances under which each file is read. Yes, behaviour is generally consistent between machines.
.profileis simply the login script filename originally used by/bin/sh.bash, being generally backwards-compatible with/bin/sh, will read.profileif one exists.Johannes Schaub - litb , answered Jan 6 '09 at 15:21
That's simple. It's explained inman bash:... ... ...Login shells are the ones that are the one you login (so, they are not executed when merely starting up xterm, for example). There are other ways to login. For example using an X display manager. Those have other ways to read and export environment variables at login time.
Also read the
INVOCATIONchapter in the manual. It says "The following paragraphs describe how bash executes its startup files." , i think that's a spot-on :) It explains what an "interactive" shell is too.Bash does not know about
.environment. I suspect that's a file of your distribution, to set environment variables independent of the shell that you drive.Jonathan Leffler , answered Jan 6 '09 at 4:13
Classically,~/.profileis used by Bourne Shell, and is probably supported by Bash as a legacy measure. Again,~/.loginand~/.cshrcwere used by C Shell - I'm not sure that Bash uses them at all.The
~/.bash_profilewould be used once, at login. The~/.bashrcscript is read every time a shell is started. This is analogous to/.cshrcfor C Shell.One consequence is that stuff in
~/.bashrcshould be as lightweight (minimal) as possible to reduce the overhead when starting a non-login shell.I believe the
~/.environmentfile is a compatibility file for Korn Shell.Filip Ekberg , answered Jan 6 '09 at 4:03
I found information about .bashrc and .bash_profile here to sum it up:.bash_profile is executed when you login. Stuff you put in there might be your PATH and other important environment variables.
.bashrc is used for non login shells. I'm not sure what that means. I know that RedHat executes it everytime you start another shell (su to this user or simply calling bash again) You might want to put aliases in there but again I am not sure what that means. I simply ignore it myself.
.profile is the equivalent of .bash_profile for the root. I think the name is changed to let other shells (csh, sh, tcsh) use it as well. (you don't need one as a user)
There is also .bash_logout wich executes at, yeah good guess...logout. You might want to stop deamons or even make a little housekeeping . You can also add "clear" there if you want to clear the screen when you log out.
Also there is a complete follow up on each of the configurations files here
These are probably even distro.-dependant, not all distros choose to have each configuraton with them and some have even more. But when they have the same name, they usualy include the same content.
Rose Perrone , answered Feb 27 '12 at 0:22
According to Josh Staiger , Mac OS X's Terminal.app actually runs a login shell rather than a non-login shell by default for each new terminal window, calling .bash_profile instead of .bashrc.He recommends:
Most of the time you don't want to maintain two separate config files for login and non-login shells ! when you set a PATH, you want it to apply to both. You can fix this by sourcing .bashrc from your .bash_profile file, then putting PATH and common settings in .bashrc.
To do this, add the following lines to .bash_profile:
if ~/.bashrc ]; then source ~/.bashrc fiNow when you login to your machine from a console .bashrc will be called.
PolyThinker , answered Jan 6 '09 at 4:06
A good place to look at is the man page of bash. Here 's an online version. Look for "INVOCATION" section.seismick , answered May 21 '12 at 10:42
I have used Debian-family distros which appear to execute.profile, but not.bash_profile, whereas RHEL derivatives execute.bash_profilebefore.profile.It seems to be a mess when you have to set up environment variables to work in any Linux OS.
|
|
|
Jul 29, 2017 | unix.stackexchange.com
Oli , asked Aug 26 '10 at 13:04
I consistently have more than one terminal open. Anywhere from two to ten, doing various bits and bobs. Now let's say I restart and open up another set of terminals. Some remember certain things, some forget.I want a history that:
- Remembers everything from every terminal
- Is instantly accessible from every terminal (eg if I
lsin one, switch to another already-running terminal and then press up,lsshows up)- Doesn't forget command if there are spaces at the front of the command.
Anything I can do to make bash work more like that?
Pablo R. , answered Aug 26 '10 at 14:37
# Avoid duplicates export HISTCONTROL=ignoredups:erasedups # When the shell exits, append to the history file instead of overwriting it shopt -s histappend # After each command, append to the history file and reread it export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"kch , answered Sep 19 '08 at 17:49
So, this is all my history-related.bashrcthing:export HISTCONTROL=ignoredups:erasedups # no duplicate entries export HISTSIZE=100000 # big big history export HISTFILESIZE=100000 # big big history shopt -s histappend # append to history, don't overwrite it # Save and reload the history after each command finishes export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"Tested with bash 3.2.17 on Mac OS X 10.5, bash 4.1.7 on 10.6.
lesmana , answered Jun 16 '10 at 16:11
Here is my attempt at Bash session history sharing. This will enable history sharing between bash sessions in a way that the history counter does not get mixed up and history expansion like!numberwill work (with some constraints).Using Bash version 4.1.5 under Ubuntu 10.04 LTS (Lucid Lynx).
HISTSIZE=9000 HISTFILESIZE=$HISTSIZE HISTCONTROL=ignorespace:ignoredups _bash_history_sync() { builtin history -a #1 HISTFILESIZE=$HISTSIZE #2 builtin history -c #3 builtin history -r #4 } history() { #5 _bash_history_sync builtin history "$@" } PROMPT_COMMAND=_bash_history_syncExplanation:More explanation:
- Append the just entered line to the
$HISTFILE(default is.bash_history). This will cause$HISTFILEto grow by one line.- Setting the special variable
$HISTFILESIZEto some value will cause Bash to truncate$HISTFILEto be no longer than$HISTFILESIZElines by removing the oldest entries.- Clear the history of the running session. This will reduce the history counter by the amount of
$HISTSIZE.- Read the contents of
$HISTFILEand insert them in to the current running session history. this will raise the history counter by the amount of lines in$HISTFILE. Note that the line count of$HISTFILEis not necessarily$HISTFILESIZE.- The
history()function overrides the builtin history to make sure that the history is synchronised before it is displayed. This is necessary for the history expansion by number (more about this later).About the constraints of the history expansion:
- Step 1 ensures that the command from the current running session gets written to the global history file.
- Step 4 ensures that the commands from the other sessions gets read in to the current session history.
- Because step 4 will raise the history counter, we need to reduce the counter in some way. This is done in step 3.
- In step 3 the history counter is reduced by
$HISTSIZE. In step 4 the history counter is raised by the number of lines in$HISTFILE. In step 2 we make sure that the line count of$HISTFILEis exactly$HISTSIZE(this means that$HISTFILESIZEmust be the same as$HISTSIZE).When using history expansion by number, you should always look up the number immediately before using it. That means no bash prompt display between looking up the number and using it. That usually means no enter and no ctrl+c.
Generally, once you have more than one Bash session, there is no guarantee whatsoever that a history expansion by number will retain its value between two Bash prompt displays. Because when
PROMPT_COMMANDis executed the history from all other Bash sessions are integrated in the history of the current session. If any other bash session has a new command then the history numbers of the current session will be different.I find this constraint reasonable. I have to look the number up every time anyway because I can't remember arbitrary history numbers.
Usually I use the history expansion by number like this
$ history | grep something #note number $ !numberI recommend using the following Bash options.
## reedit a history substitution line if it failed shopt -s histreedit ## edit a recalled history line before executing shopt -s histverifyStrange bugs:Running the history command piped to anything will result that command to be listed in the history twice. For example:
$ history | head $ history | tail $ history | grep foo $ history | true $ history | falseAll will be listed in the history twice. I have no idea why.
Ideas for improvements:
- Modify the function
_bash_history_sync()so it does not execute every time. For example it should not execute after aCTRL+Con the prompt. I often useCTRL+Cto discard a long command line when I decide that I do not want to execute that line. Sometimes I have to useCTRL+Cto stop a Bash completion script.- Commands from the current session should always be the most recent in the history of the current session. This will also have the side effect that a given history number keeps its value for history entries from this session.
Maciej Piechotka , answered Aug 26 '10 at 13:20
I'm not aware of any way usingbash. But it's one of the most popular features ofzsh.
Personally I preferzshoverbashso I recommend trying it.Here's the part of my
.zshrcthat deals with history:SAVEHIST=10000 # Number of entries HISTSIZE=10000 HISTFILE=~/.zsh/history # File setopt APPEND_HISTORY # Don't erase history setopt EXTENDED_HISTORY # Add additional data to history like timestamp setopt INC_APPEND_HISTORY # Add immediately setopt HIST_FIND_NO_DUPS # Don't show duplicates in search setopt HIST_IGNORE_SPACE # Don't preserve spaces. You may want to turn it off setopt NO_HIST_BEEP # Don't beep setopt SHARE_HISTORY # Share history between session/terminalsChris Down , answered Nov 25 '11 at 15:46
To do this, you'll need to add two lines to your~/.bashrc:shopt -s histappend PROMPT_COMMAND="history -a;history -c;history -r;" $PROMPT_COMMANDFrom
man bash:If the histappend shell option is enabled (see the description of shopt under SHELL BUILTIN COMMANDS below), the lines are appended to the history file, otherwise the history file is over-written.
Schof , answered Sep 19 '08 at 19:38
You can edit your BASH prompt to run the "history -a" and "history -r" that Muerr suggested:savePS1=$PS1(in case you mess something up, which is almost guaranteed)
PS1=$savePS1`history -a;history -r`(note that these are back-ticks; they'll run history -a and history -r on every prompt. Since they don't output any text, your prompt will be unchanged.
Once you've got your PS1 variable set up the way you want, set it permanently it in your ~/.bashrc file.
If you want to go back to your original prompt while testing, do:
PS1=$savePS1I've done basic testing on this to ensure that it sort of works, but can't speak to any side-effects from running
history -a;history -ron every prompt.pts , answered Mar 25 '11 at 17:40
If you need a bash or zsh history synchronizing solution which also solves the problem below, then see it at http://ptspts.blogspot.com/2011/03/how-to-automatically-synchronize-shell.htmlThe problem is the following: I have two shell windows A and B. In shell window A, I run
sleep 9999, and (without waiting for the sleep to finish) in shell window B, I want to be able to seesleep 9999in the bash history.The reason why most other solutions here won't solve this problem is that they are writing their history changes to the the history file using
PROMPT_COMMANDorPS1, both of which are executing too late, only after thesleep 9999command has finished.jtimberman , answered Sep 19 '08 at 17:38
You can usehistory -ato append the current session's history to the histfile, then usehistory -ron the other terminals to read the histfile.jmanning2k , answered Aug 26 '10 at 13:59
I can offer a fix for that last one: make sure the env variable HISTCONTROL does not specify "ignorespace" (or "ignoreboth").But I feel your pain with multiple concurrent sessions. It simply isn't handled well in bash.
Toby , answered Nov 20 '14 at 14:53
Here's an alternative that I use. It's cumbersome but it addresses the issue that @axel_c mentioned where sometimes you may want to have a separate history instance in each terminal (one for make, one for monitoring, one for vim, etc).I keep a separate appended history file that I constantly update. I have the following mapped to a hotkey:
history | grep -v history >> ~/master_history.txtThis appends all history from the current terminal to a file called master_history.txt in your home dir.
I also have a separate hotkey to search through the master history file:
cat /home/toby/master_history.txt | grep -iI use cat | grep because it leaves the cursor at the end to enter my regex. A less ugly way to do this would be to add a couple of scripts to your path to accomplish these tasks, but hotkeys work for my purposes. I also periodically will pull history down from other hosts I've worked on and append that history to my master_history.txt file.
It's always nice to be able to quickly search and find that tricky regex you used or that weird perl one-liner you came up with 7 months ago.
Yarek T , answered Jul 23 '15 at 9:05
Right, So finally this annoyed me to find a decent solution:# Write history after each command _bash_history_append() { builtin history -a } PROMPT_COMMAND="_bash_history_append; $PROMPT_COMMAND"What this does is sort of amalgamation of what was said in this thread, except that I don't understand why would you reload the global history after every command. I very rarely care about what happens in other terminals, but I always run series of commands, say in one terminal:
make ls -lh target/*.foo scp target/artifact.foo vm:~/(Simplified example)
And in another:
pv ~/test.data | nc vm:5000 >> output less output mv output output.backup1No way I'd want the command to be shared
rouble , answered Apr 15 at 17:43
Here is my enhancement to @lesmana's answer . The main difference is that concurrent windows don't share history. This means you can keep working in your windows, without having context from other windows getting loaded into your current windows.If you explicitly type 'history', OR if you open a new window then you get the history from all previous windows.
Also, I use this strategy to archive every command ever typed on my machine.
# Consistent and forever bash history HISTSIZE=100000 HISTFILESIZE=$HISTSIZE HISTCONTROL=ignorespace:ignoredups _bash_history_sync() { builtin history -a #1 HISTFILESIZE=$HISTSIZE #2 } _bash_history_sync_and_reload() { builtin history -a #1 HISTFILESIZE=$HISTSIZE #2 builtin history -c #3 builtin history -r #4 } history() { #5 _bash_history_sync_and_reload builtin history "$@" } export HISTTIMEFORMAT="%y/%m/%d %H:%M:%S " PROMPT_COMMAND='history 1 >> ${HOME}/.bash_eternal_history' PROMPT_COMMAND=_bash_history_sync;$PROMPT_COMMANDsimotek , answered Jun 1 '14 at 6:02
I have written a script for setting a history file per session or task its based off the following.# write existing history to the old file history -a # set new historyfile export HISTFILE="$1" export HISET=$1 # touch the new file to make sure it exists touch $HISTFILE # load new history file history -r $HISTFILEIt doesn't necessary save every history command but it saves the ones that i care about and its easier to retrieve them then going through every command. My version also lists all history files and provides the ability to search through them all.
Full source: https://github.com/simotek/scripts-config/blob/master/hiset.sh
Litch , answered Aug 11 '15 at 0:15
I chose to put history in a file-per-tty, as multiple people can be working on the same server - separating each session's commands makes it easier to audit.# Convert /dev/nnn/X or /dev/nnnX to "nnnX" HISTSUFFIX=`tty | sed 's/\///g;s/^dev//g'` # History file is now .bash_history_pts0 HISTFILE=".bash_history_$HISTSUFFIX" HISTTIMEFORMAT="%y-%m-%d %H:%M:%S " HISTCONTROL=ignoredups:ignorespace shopt -s histappend HISTSIZE=1000 HISTFILESIZE=5000History now looks like:
user@host:~# test 123 user@host:~# test 5451 user@host:~# history 1 15-08-11 10:09:58 test 123 2 15-08-11 10:10:00 test 5451 3 15-08-11 10:10:02 historyWith the files looking like:
user@host:~# ls -la .bash* -rw------- 1 root root 4275 Aug 11 09:42 .bash_history_pts0 -rw------- 1 root root 75 Aug 11 09:49 .bash_history_pts1 -rw-r--r-- 1 root root 3120 Aug 11 10:09 .bashrcfstang , answered Sep 10 '16 at 19:30
Here I will point out one problem withexport PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"and
PROMPT_COMMAND="$PROMPT_COMMAND;history -a; history -n"If you run source ~/.bashrc, the $PROMPT_COMMAND will be like
"history -a; history -c; history -r history -a; history -c; history -r"and
"history -a; history -n history -a; history -n"This repetition occurs each time you run 'source ~/.bashrc'. You can check PROMPT_COMMAND after each time you run 'source ~/.bashrc' by running 'echo $PROMPT_COMMAND'.
You could see some commands are apparently broken: "history -n history -a". But the good news is that it still works, because other parts still form a valid command sequence (Just involving some extra cost due to executing some commands repetitively. And not so clean.)
Personally I use the following simple version:
shopt -s histappend PROMPT_COMMAND="history -a; history -c; history -r"which has most of the functionalities while no such issue as mentioned above.
Another point to make is: there is really nothing magic . PROMPT_COMMAND is just a plain bash environment variable. The commands in it get executed before you get bash prompt (the $ sign). For example, your PROMPT_COMMAND is "echo 123", and you run "ls" in your terminal. The effect is like running "ls; echo 123".
$ PROMPT_COMMAND="echo 123"output (Just like running 'PROMPT_COMMAND="echo 123"; $PROMPT_COMMAND'):
123Run the following:
$ echo 3output:
3 123"history -a" is used to write the history commands in memory to ~/.bash_history
"history -c" is used to clear the history commands in memory
"history -r" is used to read history commands from ~/.bash_history to memory
See history command explanation here: http://ss64.com/bash/history.html
PS: As other users have pointed out, export is unnecessary. See: using export in .bashrc
Hopping Bunny , answered May 13 '15 at 4:48
Here is the snippet from my .bashrc and short explanations wherever needed:# The following line ensures that history logs screen commands as well shopt -s histappend # This line makes the history file to be rewritten and reread at each bash prompt PROMPT_COMMAND="$PROMPT_COMMAND;history -a; history -n" # Have lots of history HISTSIZE=100000 # remember the last 100000 commands HISTFILESIZE=100000 # start truncating commands after 100000 lines HISTCONTROL=ignoreboth # ignoreboth is shorthand for ignorespace and ignoredupsThe HISTFILESIZE and HISTSIZE are personal preferences and you can change them as per your tastes.
Mulki , answered Jul 24 at 20:49
This works for ZSH############################################################################## # History Configuration for ZSH ############################################################################## HISTSIZE=10000 #How many lines of history to keep in memory HISTFILE=~/.zsh_history #Where to save history to disk SAVEHIST=10000 #Number of history entries to save to disk #HISTDUP=erase #Erase duplicates in the history file setopt appendhistory #Append history to the history file (no overwriting) setopt sharehistory #Share history across terminals setopt incappendhistory #Immediately append to the history file, not just when a term is killed
|
|
|
Jul 29, 2017 | stackoverflow.com
user1284631 , asked Jun 5 '13 at 8:44
Following some issues with scp (it did not like the presence of the bash bind command in my.bashrcfile, apparently), I followed the advice of a clever guy on the Internet (I just cannot find that post right now) that put at the top of its.bashrcfile this:[[ ${-#*} != ${-} ]] || returnin order to make sure that the bash initialization is NOT executed unless in interactive session.
Now, that works. However, I am not able to figure how it works. Could you enlighten me?
According to this answer , the
$-is the current options set for the shell and I know that the${}is the so-called "substring" syntax for expanding variables.However, I do not understand the
${-#*i}part. And why$-#*iis not the same as${-#*i}.blue , answered Jun 5 '13 at 8:49
parameter#word} $parameter##word}The word is expanded to produce a pattern just as in filename expansion. If the pattern matches the beginning of the expanded value of parameter, then the result of the expansion is the expanded value of parameter with the shortest matching pattern (the '#' case) or the longest matching pattern (the '##' case) deleted.
If parameter is '@' or ' ', the pattern removal operation is applied to each positional parameter in turn, and the expansion is the resultant list. If parameter is an array variable subscripted with '@' or ' ', the pattern removal operation is applied to each member of the array in turn, and the expansion is the resultant list.
Source: http://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html
So basically what happens in
${-#*i}is that*iis expanded, and if it matches the beginning of the value of$-, then the result of the whole expansion is$-with the shortest matching pattern between*iand$-deleted.Example
VAR "baioasd" echo ${VAR#*i};outputs
oasd.In your case
If shell is interactive,
$-will contain the letter 'i', so when you strip the variable$-of the pattern*iyou will get a string that is different from the original$-([[ ${-#*i} != ${-} ]]yelds true). If shell is not interactive,$-does not contain the letter 'i' so the pattern*idoes not match anything in$-and[[ ${-#*i} != $- ]]yelds false, and thereturnstatement is executed.perreal , answered Jun 5 '13 at 8:53
See this :To determine within a startup script whether or not Bash is running interactively, test the value of the '-' special parameter. It contains i when the shell is interactive
Your substitution removes the string up to, and including the
iand tests if the substituted version is equal to the original string. They will be different if there isiin the${-}.
|
|
|
Jul 28, 2017 | stackoverflow.com
Possible Duplicate: What's the difference between .bashrc, .bash_profile, and .environment?
It seems that if I use
alias ls 'ls -F'inside of .bashrc on Mac OS X, then the newly created shell will not have that alias. I need to type
bashagain and that alias will be in effect.And if I log into Linux on the hosting company, the
.bashrcfile has a comment line that says:For non-login shell
and the
.bash_profilefile has a comment that saysfor login shell
So where should aliases be written in? How come we separate the login shell and non-login shell?
Some webpage say use
.bash_aliases, but it doesn't work on Mac OS X, it seems.Maggyero edited Apr 25 '16 at 16:24
The reason you separate the login and non-login shell is because the.bashrcfile is reloaded every time you start a new copy of Bash.The
.profilefile is loaded only when you either log in or use the appropriate flag to tell Bash to act as a login shell.Personally,
- I put my
PATHsetup into a.profilefile (because I sometimes use other shells);- I put my Bash aliases and functions into my
.bashrcfile;- I put this
#!/bin/bash # CRM .bash_profile Time-stamp: "2008-12-07 19:42" # echo "Loading ${HOME}/.bash_profile" source ~/. profile # get my PATH setup source ~/. bashrc # get my Bash aliasesin my
.bash_profilefile.Oh, and the reason you need to type
bashagain to get the new alias is that Bash loads your.bashrcfile when it starts but it doesn't reload it unless you tell it to. You can reload the.bashrcfile (and not need a second shell) by typingsource ~/. bashrcwhich loads the
.bashrcfile as if you had typed the commands directly to Bash.lhunath answered May 24 '09 at 6:22
Check out http://mywiki.wooledge.org/DotFiles for an excellent resource on the topic aside fromAdam Rosenfield May 24 '09 at 2:46man bash.Summary:
- You only log in once, and that's when
~/.bash_profileor~/.profileis read and executed. Since everything you run from your login shell inherits the login shell's environment, you should put all your environment variables in there. LikeLESS,PATH,MANPATH,LC_*, ... For an example, see: My.profile
- Once you log in, you can run several more shells. Imagine logging in, running X, and in X starting a few terminals with bash shells. That means your login shell started X, which inherited your login shell's environment variables, which started your terminals, which started your non-login bash shells. Your environment variables were passed along in the whole chain, so your non-login shells don't need to load them anymore. Non-login shells only execute
~/.bashrc, not/.profileor~/.bash_profile, for this exact reason, so in there define everything that only applies to bash . That's functions, aliases, bash-only variables like HISTSIZE (this is not an environment variable, don't export it!) , shell options withsetandshopt, etc. For an example, see: My.bashrc
- Now, as part of UNIX peculiarity, a login-shell does NOT execute
~/.bashrcbut only~/.profileor~/.bash_profile, so you should source that one manually from the latter. You'll see me do that in my~/.profiletoo:source ~/.bashrc.From the bash manpage:When bash is invoked as an interactive login shell, or as a non-interactive shell with the
--loginoption, it first reads and executes commands from the file/etc/profile, if that file exists. After reading that file, it looks for~/.bash_profile,~/.bash_login, and~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The--noprofileoption may be used when the shell is started to inhibit this behavior.When a login shell exits, bash reads and executes commands from the file
~/.bash_logout, if it exists.When an interactive shell that is not a login shell is started, bash reads and executes commands from
~/.bashrc, if that file exists. This may be inhibited by using the--norcoption. The--rcfilefile option will force bash to read and execute commands from file instead of~/.bashrc.Thus, if you want to get the same behavior for both login shells and interactive non-login shells, you should put all of your commands in either
.bashrcor.bash_profile, and then have the other file source the first one.Adam Rosenfield May 24 '09 at 2:46
.bash_profileis loaded for a "login shell". I am not sure what that would be on OS X, but on Linux that is either X11 or a virtual terminal.
.bashrcis loaded every time you run Bash. That is where you should put stuff you want loaded whenever you open a new Terminal.app window.I personally put everything in
.bashrcso that I don't have to restart the application for changes to take effect.
|
|
|
www.linuxquestions.org
Mohtek
I feel stupid: declare not found in bash scripting? I was anxious to get my feet wet, and I'm only up to my toes before I'm stuck...this seems very very easy but I'm not sure what I've done wrong. Below is the script and its output. What the heck am I missing?
______________________________________________________
#!/bin/bash
declare -a PROD[0]="computers" PROD[1]="HomeAutomation"
printf "${ PROD[*]}"
_______________________________________________________products.sh: 6: declare: not found
products.sh: 8: Syntax error: Bad substitutionI ran what you posted (but at the command line, not in a script, though that should make no significant difference), and got this:
Code:
-bash: ${ PROD[*]}: bad substitution
In other words, I couldn't reproduce your first problem, the "declare: not found" error. Try the declare command by itself, on the command line.
And I got rid of the "bad substitution" problem when I removed the space which is between the ${ and the PROD on the printf line.
Hope this helps.
blackhole54
The previous poster identified your second problem.
As far as your first problem goes ... I am not a bash guru although I have written a number of bash scripts. So far I have found no need for declare statements. I suspect that you might not need it either. But if you do want to use it, the following does work:
Code:
#!/bin/bashdeclare -a PROD
PROD[0]="computers"
PROD[1]="HomeAutomation"
printf "${PROD[*]}\n"EDIT: My original post was based on an older version of bash. When I tried the declare statement you posted I got an error message, but one that was different from yours. I just tried it on a newer version of bash, and your declare statement worked fine. So it might depend on the version of bash you are running. What I posted above runs fine on both versions.
|
|
|
Jul 26, 2017 | unix.stackexchange.com
Ron Burk :
Obviously cut out of a much more complex script that was more meaningful:
bash array readonly#!/bin/bash function InitializeConfig(){ declare -r -g -A SHCFG_INIT=( [a]=b ) declare -r -g -A SHCFG_INIT=( [c]=d ) echo "This statement never gets executed" } set -o xtrace InitializeConfig echo "Back from function"The output looks like this:ronburk@ubuntu:~/ubucfg$ bash bug.sh + InitializeConfig + SHCFG_INIT=([a]=b) + declare -r -g -A SHCFG_INIT + SHCFG_INIT=([c]=d) + echo 'Back from function' Back from functionBash seems to silently execute a function return upon the second declare statement. Starting to think this really is a new bug, but happy to learn otherwise.Other details:
Machine: x86_64 OS: linux-gnu Compiler: gcc Compilation CFLAGS: -DPROGRAM='bash' -DCONF_HOSTTYPE='x86_64' -DCONF_OSTYPE='linux-gnu' -DCONF_MACHTYPE='x86_64-pc-linux-gn$ uname output: Linux ubuntu 3.16.0-38-generic #52~14.04.1-Ubuntu SMP Fri May 8 09:43:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Lin$ Machine Type: x86_64-pc-linux-gnu Bash Version: 4.3 Patch Level: 11 Release Status: release
share improve this question edited Jun 14 '15 at 17:43 asked Jun 14 '15 at 7:05 118
add a comment |
Weird. Doesn't happen in bash 4.2.53(1). – choroba Jun 14 '15 at 7:22
I can reproduce this problem with bash version 4.3.11 (Ubuntu 14.04.1 LTS). It works fine with bash 4.2.8 (Ubuntu 11.04). – Cyrus Jun 14 '15 at 7:34
Maybe related: unix.stackexchange.com/q/56815/116972 I can get expected result with declare -r -g -A 'SHCFG_INIT=( [a]=b )'. – yaegashi Jun 14 '15 at 23:22By gum, you're right! Then I get readonly warning on second declare, which is reasonable, and the function completes. The xtrace output is also interesting; implies
declarewithout single quotes is really treated as two steps. Ready to become superstitious about always single-quoting the argument todeclare. Hard to see how popping the function stack can be anything but a bug, though. – Ron Burk Jun 14 '15 at 23:58I found this thread in bug-bash@gnu.org related to
test -von an assoc array. In short, bash implicitly didtest -v SHCFG_INIT[0]in your script. I'm not sure this behavior got introduced in 4.3.You might want to use
declare -pto workaround this...if declare p SHCFG_INIT >/dev/null >& ; then echo "looks like SHCFG_INIT not defined" fi ====
Well, rats. I think your answer is correct, but also reveals I'm really asking two separate questions when I thought they were probably the same issue. Since the title better reflects what turns out to be the "other" question, I'll leave this up for a while and see if anybody knows what's up with the mysterious implicit function return... Thanks! – Ron Burk Jun 14 '15 at 17:01
Edited question to focus on the remaining issue. Thanks again for the answer on the "-v" issue with associative arrays. – Ron Burk Jun 14 '15 at 17:55
Accepting this answer. Complete answer is here plus your comments above plus (IMHO) there's a bug in this version of bash (can't see how there can be any excuse for popping the function stack without warning). Thanks for your excellent research on this! – Ron Burk Jun 21 '15 at 19:31
|
|
|
Jul 26, 2017 | www.tldp.org
The declare or typeset builtins , which are exact synonyms, permit modifying the properties of variables. This is a very weak form of the typing [1] available in certain programming languages. The declare command is specific to version 2 or later of Bash. The typeset command also works in ksh scripts.
declare/typeset optionsExample 9-10. Using declare to type variables
- -r readonly
- ( declare -r var1 works the same as readonly var1 )
This is the rough equivalent of the C const type qualifier. An attempt to change the value of a readonly variable fails with an error message.
- -i integer
Certain arithmetic operations are permitted for declared integer variables without the need for expr or let .
- -a array
The variable indices will be treated as an array .
- -f function(s)
A declare -f line with no arguments in a script causes a listing of all the functions previously defined in that script.
A declare -f function_name in a script lists just the function named.
- -x export
This declares a variable as available for exporting outside the environment of the script itself.
- -x var=$value
The declare command permits assigning a value to a variable in the same statement as setting its properties.
#!/bin/bash func1 () { echo This is a function. } declare -f # Lists the function above. echo declare -i var1 # var1 is an integer. var1=2367 echo "var1 declared as $var1" var1=var1+1 # Integer declaration eliminates the need for 'let'. echo "var1 incremented by 1 is $var1." # Attempt to change variable declared as integer. echo "Attempting to change var1 to floating point value, 2367.1." var1=2367.1 # Results in error message, with no change to variable. echo "var1 is still $var1" echo declare -r var2=13.36 # 'declare' permits setting a variable property #+ and simultaneously assigning it a value. echo "var2 declared as $var2" # Attempt to change readonly variable. var2=13.37 # Generates error message, and exit from script. echo "var2 is still $var2" # This line will not execute. exit 0 # Script will not exit here.9.2.1. Another use for declare
Using the declare builtin restricts the scope of a variable.
foo () { FOO="bar" } bar () { foo echo $FOO } bar # Prints bar.However . . .
foo (){ declare FOO="bar" } bar () { foo echo $FOO } bar # Prints nothing. # Thank you, Michael Iatrou, for pointing this out.The declare command can be helpful in identifying variables, environmental or otherwise. This can be especially useful with arrays .
Notes
bash$ declare | grep HOME HOME=/home/bozo bash$ zzy=68 bash$ declare | grep zzy zzy=68 bash$ Colors=([0]="purple" [1]="reddish-orange" [2]="light green") bash$ echo ${Colors[@]} purple reddish-orange light green bash$ declare | grep Colors Colors=([0]="purple" [1]="reddish-orange" [2]="light green")
[1] In this context, typing a variable means to classify it and restrict its properties. For example, a variable declared or typed as an integer is no longer available for string operations .
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
Script execution Your perfect Bash script executes with syntax errors If you write Bash scripts with Bash specific syntax and features, run them with Bash , and run them with Bash in native mode .
Wrong
- no shebang
- the interpreter used depends on the OS implementation and current shell
- can be run by calling bash with the script name as an argument, e.g.
bash myscript#!/bin/shshebang
- depends on what
/bin/shactually is, for a Bash it means compatiblity mode, not native modeSee also:
Your script named "test" doesn't execute Give it another name. The executabletestalready exists.In Bash it's a builtin. With other shells, it might be an executable file. Either way, it's bad name choice!
Workaround: You can call it using the pathname:
/home/user/bin/testGlobbing Brace expansion is not globbing The following command line is not related to globbing (filename expansion):
# YOU EXPECT # -i1.vob -i2.vob -i3.vob .... echo -i{*.vob,} # YOU GET # -i*.vob -iWhy? The brace expansion is simple text substitution. All possible text formed by the prefix, the postfix and the braces themselves are generated. In the example, these are only two:-i*.voband-i. The filename expansion happens after that, so there is a chance that-i*.vobis expanded to a filename - if you have files like-ihello.vob. But it definitely doesn't do what you expected.Please see:
Test-command
if [ $foo ]if [-d $dir]Please see:
Variables Setting variables The Dollar-Sign There is no$(dollar-sign) when you reference the name of a variable! Bash is not PHP!# THIS IS WRONG! $myvar="Hello world!"A variable name preceeded with a dollar-sign always means that the variable gets expanded . In the example above, it might expand to nothing (because it wasn't set), effectively resulting in
="Hello world!"which definitely is wrong !When you need the name of a variable, you write only the name , for example
- (as shown above) to set variables:
picture=/usr/share/images/foo.png- to name variables to be used by the
readbuiltin command:read picture- to name variables to be unset:
unset pictureWhen you need the content of a variable, you prefix its name with a dollar-sign , like
Whitespace Putting spaces on either or both sides of the equal-sign (
- echo "The used picture is: $picture"
=) when assigning a value to a variable will fail.# INCORRECT 1 example = Hello # INCORRECT 2 example= Hello # INCORRECT 3 example =HelloThe only valid form is no spaces between the variable name and assigned value
# CORRECT 1 example=Hello # CORRECT 2 example=" Hello"Expanding (using) variables A typical beginner's trap is quoting.
As noted above, when you want to expand a variable i.e. "get the content", the variable name needs to be prefixed with a dollar-sign. But, since Bash knows various ways to quote and does word-splitting, the result isn't always the same.
Let's define an example variable containing text with spaces:
example="Hello world"
Used form result number of words $exampleHello world2 "$example"Hello world1 \$example$example1 '$example'$example1 If you use parameter expansion, you must use the name (
PATH) of the referenced variables/parameters. i.e. not ($PATH):# WRONG! echo "The first character of PATH is ${$PATH:0:1}" # CORRECT echo "The first character of PATH is ${PATH:0:1}"Note that if you are using variables in arithmetic expressions , then the bare name is allowed:
((a=$a+7)) # Add 7 to a ((a = a + 7)) # Add 7 to a. Identical to the previous command. ((a += 7)) # Add 7 to a. Identical to the previous command. a=$((a+7)) # POSIX-compatible version of previous code.Please see:
Exporting Exporting a variable means to give newly created (child-)processes a copy of that variable. not copy a variable created in a child process to the parent process. The following example does not work, since the variablehellois set in a child process (the process you execute to start that script./script.sh):$ cat script.sh export hello=world $ ./script.sh $ echo $hello $Exporting is one-way. The direction is parent process to child process, not the reverse. The above example will work, when you don't execute the script, but include ("source") it:
$ source ./script.sh $ echo $hello world $In this case, the export command is of no use.Please see:
Exit codes Reacting to exit codes If you just want to react to an exit code, regardless of its specific value, you don't need to use$?in a test command like this:grep ^root: etc passwd >/ dev null >& if $? -neq then echo "root was not found - check the pub at the corner" fiThis can be simplified to:
if grep ^root: etc passwd >/ dev null >& then echo "root was not found - check the pub at the corner" fiOr, simpler yet:
grep ^root: etc passwd >/ dev null >& || echo "root was not found - check the pub at the corner"If you need the specific value of
$?, there's no other choice. But if you need only a "true/false" exit indication, there's no need for$?.See also:
Output vs. Return Value It's important to remember the different ways to run a child command, and whether you want the output, the return value, or neither.When you want to run a command (or a pipeline) and save (or print) the output , whether as a string or an array, you use Bash's
$(command)syntax:$(ls -l /tmp) newvariable=$(printf "foo")When you want to use the return value of a command, just use the command, or add ( ) to run a command or pipeline in a subshell:
if grep someuser /etc/passwd ; then # do something fi if ( w | grep someuser | grep sqlplus ) ; then # someuser is logged in and running sqlplus fiMake sure you're using the form you intended:
# WRONG! if $(grep ERROR /var/log/messages) ; then # send alerts fi
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
Purpose An array is a parameter that holds mappings from keys to values. Arrays are used to store a collection of parameters into a parameter. Arrays (in any programming language) are a useful and common composite data structure, and one of the most important scripting features in Bash and other shells.
Here is an abstract representation of an array named
NAMES. The indexes go from 0 to 3.NAMES 0: Peter 1: Anna 2: Greg 3: JanInstead of using 4 separate variables, multiple related variables are grouped grouped together into elements of the array, accessible by their key . If you want the second name, ask for index 1 of the array
NAMES. Indexing Bash supports two different types of ksh-like one-dimensional arrays. Multidimensional arrays are not implemented .Syntax Referencing To accommodate referring to array variables and their individual elements, Bash extends the parameter naming scheme with a subscript suffix. Any valid ordinary scalar parameter name is also a valid array name:
- Indexed arrays use positive integer numbers as keys. Indexed arrays are always sparse , meaning indexes are not necessarily contiguous. All syntax used for both assigning and dereferencing indexed arrays is an arithmetic evaluation context (see Referencing ). As in C and many other languages, the numerical array indexes start at 0 (zero). Indexed arrays are the most common, useful, and portable type. Indexed arrays were first introduced to Bourne-like shells by ksh88. Similar, partially compatible syntax was inherited by many derivatives including Bash. Indexed arrays always carry the
-aattribute.- Associative arrays (sometimes known as a "hash" or "dict") use arbitrary nonempty strings as keys. In other words, associative arrays allow you to look up a value from a table based upon its corresponding string label. Associative arrays are always unordered , they merely associate key-value pairs. If you retrieve multiple values from the array at once, you can't count on them coming out in the same order you put them in. Associative arrays always carry the
-Aattribute, and unlike indexed arrays, Bash requires that they always be declared explicitly (as indexed arrays are the default, see declaration ). Associative arrays were first introduced in ksh93, and similar mechanisms were later adopted by Zsh and Bash version 4. These three are currently the only POSIX-compatible shells with any associative array support.[[:alpha:]_][[:alnum:]_]*. The parameter name may be followed by an optional subscript enclosed in square brackets to refer to a member of the array.The overall syntax is
arrname[subscript]- where for indexed arrays,subscriptis any valid arithmetic expression, and for associative arrays, any nonempty string. Subscripts are first processed for parameter and arithmetic expansions, and command and process substitutions. When used within parameter expansions or as an argument to the unset builtin, the special subscripts*and@are also accepted which act upon arrays analogously to the way the@and*special parameters act upon the positional parameters. In parsing the subscript, bash ignores any text that follows the closing bracket up to the end of the parameter name.With few exceptions, names of this form may be used anywhere ordinary parameter names are valid, such as within arithmetic expressions , parameter expansions , and as arguments to builtins that accept parameter names. An array is a Bash parameter that has been given the
-a(for indexed) or-A(for associative) attributes . However, any regular (non-special or positional) parameter may be validly referenced using a subscript, because in most contexts, referring to the zeroth element of an array is synonymous with referring to the array name without a subscript.# "x" is an ordinary non-array parameter. $ x=hi; printf '%s ' "$x" "${x[0]}"; echo "${_[0]}" hi hi hiThe only exceptions to this rule are in a few cases where the array variable's name refers to the array as a whole. This is the case for the
unsetbuiltin (see destruction ) and when declaring an array without assigning any values (see declaration ). Declaration The following explicitly give variables array attributes, making them arrays:Storing values Storing values in arrays is quite as simple as storing values in normal variables.
Syntax Description ARRAY=()Declares an indexed array ARRAYand initializes it to be empty. This can also be used to empty an existing array.ARRAY[0]=Generally sets the first element of an indexed array. If no array ARRAYexisted before, it is created.declare -a ARRAYDeclares an indexed array ARRAY. An existing array is not initialized.declare -A ARRAYDeclares an associative array ARRAY. This is the one and only way to create associative arrays.
Syntax Description ARRAY[N]=VALUESets the element Nof the indexed arrayARRAYtoVALUE.Ncan be any valid arithmetic expressionARRAY[STRING]=VALUESets the element indexed by STRINGof the associative arrayARRAY.ARRAY=VALUEAs above. If no index is given, as a default the zeroth element is set to VALUE. Careful, this is even true of associative arrays - there is no error if no key is specified, and the value is assigned to string index "0".ARRAY=(E1 E2 )Compound array assignment - sets the whole array ARRAYto the given list of elements indexed sequentially starting at zero. The array is unset before assignment unless the += operator is used. When the list is empty (ARRAY=()), the array will be set to an empty array. This method obviously does not use explicit indexes. An associative array can not be set like that! Clearing an associative array usingARRAY=()works.ARRAY=([X]=E1 [Y]=E2 )Compound assignment for indexed arrays with index-value pairs declared individually (here for example XandY). X and Y are arithmetic expressions. This syntax can be combined with the above - elements declared without an explicitly specified index are assigned sequentially starting at either the last element with an explicit index, or zero.ARRAY=([S1]=E1 [S2]=E2 )Individual mass-setting for associative arrays . The named indexes (here: S1andS2) are strings.ARRAY+=(E1 E2 )Append to ARRAY. As of now, arrays can't be exported. Getting values article about parameter expansion and check the notes about arrays.
Syntax Description ${ARRAY[N]}Expands to the value of the index Nin the indexed arrayARRAY. IfNis a negative number, it's treated as the offset from the maximum assigned index (can't be used for assignment) - 1${ARRAY[S]}Expands to the value of the index Sin the associative arrayARRAY."${ARRAY[@]}"
${ARRAY[@]}
"${ARRAY[*]}"
${ARRAY[*]}Similar to mass-expanding positional parameters , this expands to all elements. If unquoted, both subscripts *and@expand to the same result, if quoted,@expands to all elements individually quoted,*expands to all elements quoted as a whole."${ARRAY[@]:N:M}"
${ARRAY[@]:N:M}
"${ARRAY[*]:N:M}"
${ARRAY[*]:N:M}Similar to what this syntax does for the characters of a single string when doing substring expansion , this expands to Melements starting with elementN. This way you can mass-expand individual indexes. The rules for quoting and the subscripts*and@are the same as above for the other mass-expansions.For clarification: When you use the subscripts
@or*for mass-expanding, then the behaviour is exactly what it is for$@and$*when mass-expanding the positional parameters . You should read this article to understand what's going on. MetadataDestruction The unset builtin command is used to destroy (unset) arrays or individual elements of arrays.
Syntax Description ${#ARRAY[N]}Expands to the length of an individual array member at index N( stringlength${#ARRAY[STRING]}Expands to the length of an individual associative array member at index STRING( stringlength )${#ARRAY[@]}
${#ARRAY[*]}Expands to the number of elements in ARRAY${!ARRAY[@]}
${!ARRAY[*]}Expands to the indexes in ARRAYsince BASH 3.0
Syntax Description unset -v ARRAY
unset -v ARRAY[@]
unset -v ARRAY[*]Destroys a complete array unset -v ARRAY[N]Destroys the array element at index Nunset -v ARRAY[STRING]Destroys the array element of the associative array at index STRINGIt is best to explicitly specify -v when unsetting variables with unset.
pathname expansion to occur due to the presence of glob characters.Example: You are in a directory with a file named
x1, and you want to destroy an array elementx[1], withunset x[1]then pathname expansion will expand to the filenamex1and break your processing!Even worse, if
nullglobis set, your array/index will disappear.To avoid this, always quote the array name and index:
unset -v 'x[1]'This applies generally to all commands which take variable names as arguments. Single quotes preferred.
Usage Numerical Index Numerical indexed arrays are easy to understand and easy to use. The Purpose and Indexing chapters above more or less explain all the needed background theory.
Now, some examples and comments for you.
Let's say we have an array
sentencewhich is initialized as follows:sentence=(Be liberal in what you accept, and conservative in what you send)Since no special code is there to prevent word splitting (no quotes), every word there will be assigned to an individual array element. When you count the words you see, you should get 12. Now let's see if Bash has the same opinion:
$ echo ${#sentence[@]} 12Yes, 12. Fine. You can take this number to walk through the array. Just subtract 1 from the number of elements, and start your walk at 0 (zero)
((n_elements=${#sentence[@]}, max_index=n_elements - 1)) for ((i = 0; i <= max_index; i++)); do echo "Element $i: '${sentence[i]}'" doneYou always have to remember that, it seems newbies have problems sometimes. Please understand that numerical array indexing begins at 0 (zero)
The method above, walking through an array by just knowing its number of elements, only works for arrays where all elements are set, of course. If one element in the middle is removed, then the calculation is nonsense, because the number of elements doesn't correspond to the highest used index anymore (we call them " sparse arrays "). Associative (Bash 4) Associative arrays (or hash tables ) are not much more complicated than numerical indexed arrays. The numerical index value (in Bash a number starting at zero) just is replaced with an arbitrary string:
# declare -A, introduced with Bash 4 to declare an associative array declare -A sentence sentence[Begin]='Be liberal in what' sentence[Middle]='you accept, and conservative' sentence[End]='in what you send' sentence['Very end']=...Beware: don't rely on the fact that the elements are ordered in memory like they were declared, it could look like this:
# output from 'set' command sentence=([End]="in what you send" [Middle]="you accept, and conservative " [Begin]="Be liberal in what " ["Very end"]="...")This effectively means, you can get the data back with"${sentence[@]}", of course (just like with numerical indexing), but you can't rely on a specific order. If you want to store ordered data, or re-order data, go with numerical indexes. For associative arrays, you usually query known index values:for element in Begin Middle End "Very end"; do printf "%s" "${sentence[$element]}" done printf "\n"A nice code example: Checking for duplicate files using an associative array indexed with the SHA sum of the files:
# Thanks to Tramp in #bash for the idea and the code unset flist; declare -A flist; while read -r sum fname; do if [[ ${flist[$sum]} ]]; then printf 'rm -- "%s" # Same as >%s<\n' "$fname" "${flist[$sum]}" else flist[$sum]="$fname" fi done < <(find . -type f -exec sha256sum {} +) >rmdupsInteger arrays Any type attributes applied to an array apply to all elements of the array. If the integer attribute is set for either indexed or associative arrays, then values are considered as arithmetic for both compound and ordinary assignment, and the += operator is modified in the same way as for ordinary integer variables.
~ $ ( declare -ia 'a=(2+4 [2]=2+2 [a[2]]="a[2]")' 'a+=(42 [a[4]]+=3)'; declare -p a ) declare -ai a='([0]="6" [2]="4" [4]="7" [5]="42")'
a[0]is assigned to the result of2+4.a[1]gets the result of2+2. The last index in the first assignment is the result ofa[2], which has already been assigned as4, and its value is also givena[2].This shows that even though any existing arrays named
ain the current scope have already been unset by using=instead of+=to the compound assignment, arithmetic variables within keys can self-reference any elements already assigned within the same compound-assignment. With integer arrays this also applies to expressions to the right of the=. (See evaluation order , the right side of an arithmetic assignment is typically evaluated first in Bash.)The second compound assignment argument to declare uses
+=, so it appends after the last element of the existing array rather than deleting it and creating a new array, soa[5]gets42.Lastly, the element whose index is the value of
a[4](4), gets3added to its existing value, makinga[4]==7. Note that having the integer attribute set this time causes += to add, rather than append a string, as it would for a non-integer array.The single quotes force the assignments to be evaluated in the environment of
declare. This is important because attributes are only applied to the assignment after assignment arguments are processed. Without them the+=compound assignment would have been invalid, and strings would have been inserted into the integer array without evaluating the arithmetic. A special-case of this is shown in the next section.eval, but there are differences.)'Todo:' Discuss this in detail.Indirection Arrays can be expanded indirectly using the indirect parameter expansion syntax. Parameters whose values are of the form:
name[index],name[@], orname[*]when expanded indirectly produce the expected results. This is mainly useful for passing arrays (especially multiple arrays) by name to a function.This example is an "isSubset"-like predicate which returns true if all key-value pairs of the array given as the first argument to isSubset correspond to a key-value of the array given as the second argument. It demonstrates both indirect array expansion and indirect key-passing without eval using the aforementioned special compound assignment expansion.
isSubset() { local -a 'xkeys=("${!'"$1"'[@]}")' 'ykeys=("${!'"$2"'[@]}")' set -- "${@/%/[key]}" (( ${#xkeys[@]} <= ${#ykeys[@]} )) || return 1 local key for key in "${xkeys[@]}"; do [[ ${!2+_} && ${!1} == ${!2} ]] || return 1 done } main() { # "a" is a subset of "b" local -a 'a=({0..5})' 'b=({0..10})' isSubset a b echo $? # true # "a" contains a key not in "b" local -a 'a=([5]=5 {6..11})' 'b=({0..10})' isSubset a b echo $? # false # "a" contains an element whose value != the corresponding member of "b" local -a 'a=([5]=5 6 8 9 10)' 'b=({0..10})' isSubset a b echo $? # false } mainThis script is one way of implementing a crude multidimensional associative array by storing array definitions in an array and referencing them through indirection. The script takes two keys and dynamically calls a function whose name is resolved from the array.
callFuncs() { # Set up indirect references as positional parameters to minimize local name collisions. set -- "${@:1:3}" ${2+'a["$1"]' "$1"'["$2"]'} # The only way to test for set but null parameters is unfortunately to test each individually. local x for x; do [[ $x ]] || return 0 done local -A a=( [foo]='([r]=f [s]=g [t]=h)' [bar]='([u]=i [v]=j [w]=k)' [baz]='([x]=l [y]=m [z]=n)' ) ${4+${a["$1"]+"${1}=${!3}"}} # For example, if "$1" is "bar" then define a new array: bar=([u]=i [v]=j [w]=k) ${4+${a["$1"]+"${!4-:}"}} # Now just lookup the new array. for inputs: "bar" "v", the function named "j" will be called, which prints "j" to stdout. } main() { # Define functions named {f..n} which just print their own names. local fun='() { echo "$FUNCNAME"; }' x for x in {f..n}; do eval "${x}${fun}" done callFuncs "$@" } main "$@"Bugs and Portability Considerations
Bugs
- Arrays are not specified by POSIX. One-dimensional indexed arrays are supported using similar syntax and semantics by most Korn-like shells.
- Associative arrays are supported via
typeset -Ain Bash 4, Zsh, and Ksh93.- In Ksh93, arrays whose types are not given explicitly are not necessarily indexed. Arrays defined using compound assignments which specify subscripts are associative by default. In Bash, associative arrays can only be created by explicitly declaring them as associative, otherwise they are always indexed. In addition, ksh93 has several other compound structures whose types can be determined by the compound assignment syntax used to create them.
- In Ksh93, using the
=compound assignment operator unsets the array, including any attributes that have been set on the array prior to assignment. In order to preserve attributes, you must use the+=operator. However, declaring an associative array, then attempting ana=( )style compound assignment without specifying indexes is an error. I can't explain this inconsistency.$ ksh -c 'function f { typeset -a a; a=([0]=foo [1]=bar); typeset -p a; }; f' # Attribute is lost, and since subscripts are given, we default to associative. typeset -A a=([0]=foo [1]=bar) $ ksh -c 'function f { typeset -a a; a+=([0]=foo [1]=bar); typeset -p a; }; f' # Now using += gives us the expected results. typeset -a a=(foo bar) $ ksh -c 'function f { typeset -A a; a=(foo bar); typeset -p a; }; f' # On top of that, the reverse does NOT unset the attribute. No idea why. ksh: f: line 1: cannot append index array to associative array a- Only Bash and mksh support compound assignment with mixed explicit subscripts and automatically incrementing subscripts. In ksh93, in order to specify individual subscripts within a compound assignment, all subscripts must be given (or none). Zsh doesn't support specifying individual subscripts at all.
- Appending to a compound assignment is a fairly portable way to append elements after the last index of an array. In Bash, this also sets append mode for all individual assignments within the compound assignment, such that if a lower subscript is specified, subsequent elements will be appended to previous values. In ksh93, it causes subscripts to be ignored, forcing appending everything after the last element. (Appending has different meaning due to support for multi-dimensional arrays and nested compound datastructures.)
$ ksh -c 'function f { typeset -a a; a+=(foo bar baz); a+=([3]=blah [0]=bork [1]=blarg [2]=zooj); typeset -p a; }; f' # ksh93 forces appending to the array, disregarding subscripts typeset -a a=(foo bar baz '[3]=blah' '[0]=bork' '[1]=blarg' '[2]=zooj') $ bash -c 'function f { typeset -a a; a+=(foo bar baz); a+=(blah [0]=bork blarg zooj); typeset -p a; }; f' # Bash applies += to every individual subscript. declare -a a='([0]="foobork" [1]="barblarg" [2]="bazzooj" [3]="blah")' $ mksh -c 'function f { typeset -a a; a+=(foo bar baz); a+=(blah [0]=bork blarg zooj); typeset -p a; }; f' # Mksh does like Bash, but clobbers previous values rather than appending. set -A a typeset a[0]=bork typeset a[1]=blarg typeset a[2]=zooj typeset a[3]=blah- In Bash and Zsh, the alternate value assignment parameter expansion (
${arr[idx]:=foo}) evaluates the subscript twice, first to determine whether to expand the alternate, and second to determine the index to assign the alternate to. See evaluation order .$ : ${_[$(echo $RANDOM >&2)1]:=$(echo hi >&2)} 13574 hi 14485- In Zsh, arrays are indexed starting at 1 in its default mode. Emulation modes are required in order to get any kind of portability.
- Zsh and mksh do not support compound assignment arguments to
typeset.- Ksh88 didn't support modern compound array assignment syntax. The original (and most portable) way to assign multiple elements is to use the
set -A name arg1 arg2syntax. This is supported by almost all shells that support ksh-like arrays except for Bash. Additionally, these shells usually support an optional-sargument tosetwhich performs lexicographic sorting on either array elements or the positional parameters. Bash has no built-in sorting ability other than the usual comparison operators.$ ksh -c 'set -A arr -- foo bar bork baz; typeset -p arr' # Classic array assignment syntax typeset -a arr=(foo bar bork baz) $ ksh -c 'set -sA arr -- foo bar bork baz; typeset -p arr' # Native sorting! typeset -a arr=(bar baz bork foo) $ mksh -c 'set -sA arr -- foo "[3]=bar" "[2]=baz" "[7]=bork"; typeset -p arr' # Probably a bug. I think the maintainer is aware of it. set -A arr typeset arr[2]=baz typeset arr[3]=bar typeset arr[7]=bork typeset arr[8]=foo- Evaluation order for assignments involving arrays varies significantly depending on context. Notably, the order of evaluating the subscript or the value first can change in almost every shell for both expansions and arithmetic variables. See evaluation order for details.
- Bash 4.1.* and below cannot use negative subscripts to address array indexes relative to the highest-numbered index. You must use the subscript expansion, i.e.
"${arr[@]:(-n):1}", to expand the nth-last element (or the next-highest indexed afternifarr[n]is unset). In Bash 4.2, you may expand (but not assign to) a negative index. In Bash 4.3, ksh93, and zsh, you may both assign and expand negative offsets.- ksh93 also has an additional slice notation:
"${arr[n..m]}"wherenandmare arithmetic expressions. These are needed for use with multi-dimensional arrays.- Assigning or referencing negative indexes in mksh causes wrap-around. The max index appears to be
UINT_MAX, which would be addressed byarr[-1].- So far, Bash's
-v vartest doesn't support individual array subscripts. You may supply an array name to test whether an array is defined, but can't check an element. ksh93's-vsupports both. Other shells lack a-vtest.Evaluation order Here are some of the nasty details of array assignment evaluation order. You can use this testcase code to generate these results.
- Fixed in 4.3 Bash 4.2.* and earlier considers each chunk of a compound assignment, including the subscript for globbing. The subscript part is considered quoted, but any unquoted glob characters on the right-hand side of the
[ ]=will be clumped with the subscript and counted as a glob. Therefore, you must quote anything on the right of the=sign. This is fixed in 4.3, so that each subscript assignment statement is expanded following the same rules as an ordinary assignment. This also works correctly in ksh93.$ touch '[1]=a'; bash -c 'a=([1]=*); echo "${a[@]}"' [1]=amksh has a similar but even worse problem in that the entire subscript is considered a glob.$ touch 1=a; mksh -c 'a=([123]=*); print -r -- "${a[@]}"' 1=a- Fixed in 4.3 In addition to the above globbing issue, assignments preceding "declare" have an additional effect on brace and pathname expansion.
$ set -x; foo=bar declare arr=( {1..10} ) + foo=bar + declare 'a=(1)' 'a=(2)' 'a=(3)' 'a=(4)' 'a=(5)' $ touch xy=foo $ declare x[y]=* + declare 'x[y]=*' $ foo=bar declare x[y]=* + foo=bar + declare xy=fooEach word (the entire assignment) is subject to globbing and brace expansion. This appears to trigger the same strange expansion mode aslet,eval, other declaration commands, and maybe more.- Fixed in 4.3 Indirection combined with another modifier expands arrays to a single word.
$ a=({a..c}) b=a[@]; printf '<%s> ' "${!b}"; echo; printf '<%s> ' "${!b/%/foo}"; echo <a> <b> <c> <a b cfoo>- Fixed in 4.3 Process substitutions are evaluated within array indexes. Zsh and ksh don't do this in any arithmetic context.
# print "moo" dev=fd=1 _[1<(echo moo >&2)]= # Fork bomb ${dev[${dev='dev[1>(${dev[dev]})]'}]}Each testcase prints evaluation order for indexed array assignment contexts. Each context is tested for expansions (represented by digits) and arithmetic (letters), ordered from left to right within the expression. The output corresponds to the way evaluation is re-ordered for each shell: a[ $1 a ]=${b[ $2 b ]:=${c[ $3 c ]}} No attributes a[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia a a[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia b a[ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} typeset -ia a b (( a[ $1 a ] = b[ $2 b ] ${c[ $3 c ]} )) No attributes (( a[ $1 a ] = ${b[ $2 b ]:=c[ $3 c ]} )) typeset -ia b a+=( [ $1 a ]=${b[ $2 b ]:=${c[ $3 c ]}} [ $4 d ]=$(( $5 e )) ) typeset -a a a+=( [ $1 a ]=${b[ $2 b ]:=c[ $3 c ]} [ $4 d ]=${5}e ) typeset -ia a bash: 4.2.42(1)-release 2 b 3 c 2 b 1 a 2 b 3 2 b 1 a c 2 b 3 2 b c 1 a 2 b 3 2 b c 1 a c 1 2 3 c b a 1 2 b 3 2 b c c a 1 2 b 3 c 2 b 4 5 e a d 1 2 b 3 2 b 4 5 a c d e ksh93: Version AJM 93v- 2013-02-22 1 2 b b a 1 2 b b a 1 2 b b a 1 2 b b a 1 2 3 c b a 1 2 b b a 1 2 b b a 4 5 e d 1 2 b b a 4 5 d e mksh: @(#)MIRBSD KSH R44 2013/02/24 2 b 3 c 1 a 2 b 3 1 a c 2 b 3 c 1 a 2 b 3 c 1 a 1 2 3 c a b 1 2 b 3 c a 1 2 b 3 c 4 5 e a d 1 2 b 3 4 5 a c d e zsh: 5.0.2 2 b 3 c 2 b 1 a 2 b 3 2 b 1 a c 2 b 1 a 2 b 1 a 1 2 3 c b a 1 2 b a 1 2 b 3 c 2 b 4 5 e 1 2 b 3 2 b 4 5See also
- Parameter expansion (contains sections for arrays)
- The classic for-loop (contains some examples to iterate over arrays)
- The declare builtin command
- BashFAQ 005 - How can I use array variables? - A very detailed discussion on arrays with many examples.
- BashSheet - Arrays - Bashsheet quick-reference on Greycat's wiki.
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
Intro The day will come when you want to give arguments to your scripts. These arguments are known as positional parameters . Some relevant special parameters are described below:
Parameter(s) Description $0the first positional parameter, equivalent to argv[0]in C, see the first argument$FUNCNAMEthe function name ( attention : inside a function, $0is still the$0of the shell, not the function name)$1 $9the argument list elements from 1 to 9 ${10} ${N}the argument list elements beyond 9 (note the parameter expansion syntax!) $*all positional parameters except $0, see mass usage$@all positional parameters except $0, see mass usage$#the number of arguments, not counting $0These positional parameters reflect exactly what was given to the script when it was called.
Option-switch parsing (e.g.
-hfor displaying help) is not performed at this point.See also the dictionary entry for "parameter" . The first argument The very first argument you can access is referenced as
$0. It is usually set to the script's name exactly as called, and it's set on shell initialization:Testscript - it just echos
$0:#!/bin/bash echo "$0"You see,$0is always set to the name the script is called with ($is the prompt ):> ./testscript ./testscript> /usr/bin/testscript /usr/bin/testscriptHowever, this isn't true for login shells:
> echo "$0" -bashIn other terms,
$0is not a positional parameter, it's a special parameter independent from the positional parameter list. It can be set to anything. In the ideal case it's the pathname of the script, but since this gets set on invocation, the invoking program can easily influence it (theloginprogram does that for login shells, by prefixing a dash, for example).Inside a function,
$0still behaves as described above. To get the function name, use$FUNCNAME. Shifting The builtin commandshiftis used to change the positional parameter values:
$1will be discarded$2will become$1$3will become$2- in general:
$Nwill become$N-1The command can take a number as argument: Number of positions to shift. e.g.
shift 4shifts$5to$1. Using them Enough theory, you want to access your script-arguments. Well, here we go. One by one One way is to access specific parameters:#!/bin/bash echo "Total number of arguments: $#" echo "Argument 1: $1" echo "Argument 2: $2" echo "Argument 3: $3" echo "Argument 4: $4" echo "Argument 5: $5"While useful in another situation, this way is lacks flexibility. The maximum number of arguments is a fixedvalue - which is a bad idea if you write a script that takes many filenames as arguments.
⇒ forget that one Loops There are several ways to loop through the positional parameters.
You can code a C-style for-loop using
$#as the end value. On every iteration, theshift-command is used to shift the argument list:numargs=$# for ((i=1 ; i <= numargs ; i++)) do echo "$1" shift doneNot very stylish, but usable. The
numargsvariable is used to store the initial value of$#because the shift command will change it as the script runs.
Another way to iterate one argument at a time is the
forloop without a given wordlist. The loop uses the positional parameters as a wordlist:for arg do echo "$arg" doneAdvantage: The positional parameters will be preserved
The next method is similar to the first example (the
forloop), but it doesn't test for reaching$#. It shifts and checks if$1still expands to something, using the test command :while [ "$1" ] do echo "$1" shift doneLooks nice, but has the disadvantage of stopping when
$1is empty (null-string). Let's modify it to run as long as$1is defined (but may be null), using parameter expansion for an alternate value :while [ "${1+defined}" ]; do echo "$1" shift doneGetopts There is a small tutorial dedicated to ''getopts'' ( under construction ). Mass usage All Positional Parameters Sometimes it's necessary to just "relay" or "pass" given arguments to another program. It's very inefficient to do that in one of these loops, as you will destroy integrity, most likely (spaces!).
The shell developers created
$*and$@for this purpose.As overview:
Syntax Effective result $*$1 $2 $3 ${N}$@$1 $2 $3 ${N}"$*""$1c$2c$3c c${N}""$@""$1" "$2" "$3" "${N}"Without being quoted (double quotes), both have the same effect: All positional parameters from
$1to the last one used are expanded without any special handling.When the
$*special parameter is double quoted, it expands to the equivalent of:"$1c$2c$3c$4c ..$N", where 'c' is the first character ofIFS.But when the
$@special parameter is used inside double quotes, it expands to the equivanent of
"$1" "$2" "$3" "$4" .. "$N"which reflects all positional parameters as they were set initially and passed to the script or function. If you want to re-use your positional parameters to call another program (for example in a wrapper-script), then this is the choice for you, use double quoted
"$@".Well, let's just say: You almost always want a quoted
"$@"! Range Of Positional Parameters Another way to mass expand the positional parameters is similar to what is possible for a range of characters using substring expansion on normal parameters and the mass expansion range of arrays .
${@:START:COUNT}
${*:START:COUNT}
"${@:START:COUNT}"
"${*:START:COUNT}"The rules for using
@or*and quoting are the same as above. This will expandCOUNTnumber of positional parameters beginning atSTART.COUNTcan be omitted (${@:START}), in which case, all positional parameters beginning atSTARTare expanded.If
STARTis negative, the positional parameters are numbered in reverse starting with the last one.
COUNTmay not be negative, i.e. the element count may not be decremented.Example: START at the last positional parameter:
echo "${@: -1}"Attention : As of Bash 4, a
STARTof0includes the special parameter$0, i.e. the shell name or whatever $0 is set to, when the positional parameters are in use. ASTARTof1begins at$1. In Bash 3 and older, both0and1began at$1. Setting Positional Parameters Setting positional parameters with command line arguments, is not the only way to set them. The builtin command, set may be used to "artificially" change the positional parameters from inside the script or function:set "This is" my new "set of" positional parameters # RESULTS IN # $1: This is # $2: my # $3: new # $4: set of # $5: positional # $6: parametersIt's wise to signal "end of options" when setting positional parameters this way. If not, the dashes might be interpreted as an option switch by
setitself:# both ways work, but behave differently. See the article about the set command! set -- ... set - ...Alternately this will also preserve any verbose (-v) or tracing (-x) flags, which may otherwise be reset by
setset -$- ...Production examples Using a while loop To make your program accept options as standard command syntax:
COMMAND [options] <params># Like 'cat -A file.txt'See simple option parsing code below. It's not that flexible. It doesn't auto-interpret combined options (-fu USER) but it works and is a good rudimentary way to parse your arguments.
#!/bin/sh # Keeping options in alphabetical order makes it easy to add more. while : do case "$1" in -f | --file) file="$2" # You may want to check validity of $2 shift 2 ;; -h | --help) display_help # Call your function # no shifting needed here, we're done. exit 0 ;; -u | --user) username="$2" # You may want to check validity of $2 shift 2 ;; -v | --verbose) # It's better to assign a string, than a number like "verbose=1" # because if you're debugging the script with "bash -x" code like this: # # if [ "$verbose" ] ... # # You will see: # # if [ "verbose" ] ... # # Instead of cryptic # # if [ "1" ] ... # verbose="verbose" shift ;; --) # End of all options shift break; -*) echo "Error: Unknown option: $1" >&2 exit 1 ;; *) # No more options break ;; esac done # End of fileFilter unwanted options with a wrapper script This simple wrapper enables filtering unwanted options (here:
-aand–allforls) out of the command line. It reads the positional parameters and builds a filtered array consisting of them, then callslswith the new option set. It also respects the–as "end of options" forlsand doesn't change anything after it:#!/bin/bash # simple ls(1) wrapper that doesn't allow the -a option options=() # the buffer array for the parameters eoo=0 # end of options reached while [[ $1 ]] do if ! ((eoo)); then case "$1" in -a) shift ;; --all) shift ;; -[^-]*a*|-a?*) options+=("${1//a}") shift ;; --) eoo=1 options+=("$1") shift ;; *) options+=("$1") shift ;; esac else options+=("$1") # Another (worse) way of doing the same thing: # options=("${options[@]}" "$1") shift fi done /bin/ls "${options[@]}"Using getopts There is a small tutorial dedicated to ''getopts'' ( under construction ). See also
Discussion 2010/04/14 14:20
- Internal: Small getopts tutorial
- Internal: The while-loop
- Internal: The C-style for-loop
- Internal: Arrays (for equivalent syntax for mass-expansion)
- Internal: Substring expansion on a parameter (for equivalent syntax for mass-expansion)
- Dictionary, internal: Parameter
The shell-developers invented $* and $@ for this purpose.Without being quoted (double-quoted), both have the same effect: All positional parameters from $1 to the last used one >are expanded, separated by the first character of IFS (represented by "c" here, but usually a space):
$1c$2c$3c$4c........$NWithout double quotes, $* and $@ are expanding the positional parameters separated by only space, not by IFS.
#!/bin/bash export IFS='-' echo -e $* echo -e $@$./test "This is" 2 3 This is 2 3 This is 2 32011/02/18 16:11 #!/bin/bashOLDIFS="$IFS" IFS='-' #export IFS='-'
#echo -e $* #echo -e $@ #should be echo -e "$*" echo -e "$@" IFS="$OLDIFS"
2011/02/18 16:14 #should be echo -e "$*"2012/04/20 10:32 Here's yet another non-getopts way.
http://bsdpants.blogspot.de/2007/02/option-ize-your-shell-scripts.html
2012/07/16 14:48 Hi there!What if I use "$@" in subsequent function calls, but arguments are strings?
I mean, having:
#!/bin/bash echo "$@" echo n: $#If you use it
mypc$ script arg1 arg2 "asd asd" arg4 arg1 arg2 asd asd arg4 n: 4But having
#!/bin/bash myfunc() { echo "$@" echo n: $# } echo "$@" echo n: $# myfunc "$@"you get:
mypc$ myscrpt arg1 arg2 "asd asd" arg4 arg1 arg2 asd asd arg4 4 arg1 arg2 asd asd arg4 5As you can see, there is no way to make know the function that a parameter is a string and not a space separated list of arguments.
Any idea of how to solve it? I've test calling functions and doing expansion in almost all ways with no results.
2012/08/12 09:11 I don't know why it fails for you. It should work if you use
"$@", of course.See the example I used your second script with:
$ ./args1 a b c "d e" f a b c d e f n: 5 a b c d e f n: 5
|
|
|
Jul 25, 2017 | artofsoftware.org
Sep 2, 2011
Posted by craig in Tools Tags Once upon a time I was playing with Windows Power Shell (WPSH) and discovered a very useful function for changing to commonly visited directories. The function, called "go", which was written by Peter Provost , grew on me as I used WPSH ! so much so that I decided to implement it in bash after my WPSH experiments ended.The problem is simple. Users of command line interfaces tend to visit the same directories repeatedly over the course of their work, and having a way to get to these oft-visited places without a lot of typing is nice.
The solution entails maintaining a map of key-value pairs, where each key is an alias to a value, which is itself a commonly visited directory. The "go" function will, when given a string input, look that string up in the map, and if the key is found, move to the directory indicated by the value.
The map itself is just a specially formatted text file with one key-value entry per line, while each entry is separated into key-value components by the first encountered colon, with the left side being interpreted as the entry's key and the right side as its value.
Keys are typically short easily typed strings, while values can be arbitrary path names, and even contain references to environment variables. The effect of this is that "go" can respond dynamically to the environment.
Finally, the "go" function finds the map file by referring to an environment variable called "GO_FILE", which should have as its value the full path to the map.
Before I ran into this idea I had maintained a number of shell aliases, (i.e. alias dwork='cd $WORK_DIR'), to achieve a similar end, but every time I wanted to add a new location I was forced to edit my .bashrc file. Then I would subsequently have to resource it or enter the alias again on the command line. Since I typically keep multiple shells open this is just a pain, and so I didn't add new aliases very often. With this method, a new entry in the "go file" is immediately available to all open shells without any extra finagling.
This functionality is related to CDPATH, but they are not replacements for one another. Indeed CDPATH is the more appropriate solution when you want to be able to "cd" to all or most of the sub-directories of some parent. On the other hand, "go" works very well for getting to a single directory easily. For example you might not want "/usr/local" in your CDPATH and still want an abbreviated way of getting to "/usr/local/share".
The code for the go function, as well as some brief documentation follows.
############################################## # GO # # Inspired by some Windows Power Shell code # from Peter Provost (peterprovost.org) # # Here are some examples entries: # work:${WORK_DIR} # source:${SOURCE_DIR} # dev:/c/dev # object:${USER_OBJECT_DIR} # debug:${USER_OBJECT_DIR}/debug ############################################### export GO_FILE=~/.go_locations function go { if [ -z "$GO_FILE" ] then echo "The variable GO_FILE is not set." return fi if [ ! -e "$GO_FILE" ] then echo "The 'go file': '$GO_FILE' does not exist." return fi dest="" oldIFS=${IFS} IFS=$'\n' for entry in `cat ${GO_FILE}` do if [ "$1" = ${entry%%:*} ] then #echo $entry dest=${entry##*:} break fi done if [ -n "$dest" ] then # Expand variables in the go file. #echo $dest cd `eval echo $dest` else echo "Invalid location, valid locations are:" cat $GO_FILE fi export IFS=${oldIFS} }
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
local to a function:
- Using the
localkeyword, or- Using
declare(which will detect when it was called from within a function and make the variable(s) local).myfunc () local var =VALUE # alternative, only when used INSIDE a function declare var =VALUE ...The local keyword (or declaring a variable using the
declarecommand) tags a variable to be treated completely local and separate inside the function where it was declared:foo =external printvalue () local foo =internal echo $foo # this will print "external" echo $foo # this will print "internal" printvalue # this will print - again - "external" echo $foo
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
The environment space is not directly related to the topic about scope, but it's worth mentioning.
Every UNIX® process has a so-called environment . Other items, in addition to variables, are saved there, the so-called environment variables . When a child process is created (in Bash e.g. by simply executing another program, say
lsto list files), the whole environment including the environment variables is copied to the new process. Reading that from the other side means: Only variables that are part of the environment are available in the child process.A variable can be tagged to be part of the environment using the
exportcommand:# create a new variable and set it: # -> This is a normal shell variable, not an environment variable! myvariable "Hello world." # make the variable visible to all child processes: # -> Make it an environment variable: "export" it export myvariableRemember that the exported variable is a copy . There is no provision to "copy it back to the parent." See the article about Bash in the process tree !
1) under specific circumstances, also by the shell itself
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
:(colon) and input redirection. The:does nothing, it's a pseudo command, so it does not care about standard input. In the following code example, you want to test mail and logging, but not dump the database, or execute a shutdown:#!/bin/bash # Write info mails, do some tasks and bring down the system in a safe way echo "System halt requested" mail -s "System halt" netadmin example.com logger -t SYSHALT "System halt requested" ##### The following "code block" is effectively ignored : << "SOMEWORD" etc init.d mydatabase clean_stop mydatabase_dump var db db1 mnt fsrv0 backups db1 logger -t SYSHALT "System halt: pre-shutdown actions done, now shutting down the system" shutdown -h NOW SOMEWORD ##### The ignored codeblock ends hereWhat happened? The:pseudo command was given some input by redirection (a here-document) - the pseudo command didn't care about it, effectively, the entire block was ignored.The here-document-tag was quoted here to avoid substitutions in the "commented" text! Check redirection with here-documents for more
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
- Simple locking (against parallel run)
- Rudimentary config files for your scripts
- Editing files with ed(1)
- Collapsing Functions
- Illustrated Redirection Tutorial
- Calculate with dc(1)
- Introduction to pax - the POSIX archiver
- Small getopts tutorial ( under construction! )
- Dissect a bad oneliner An example of a bad oneliner, breakdown and fix (by
kojoro)- Write tests for ./your-script.sh by using bashtest util
|
|
|
Jul 25, 2017 | wiki.bash-hackers.org
Bash changes page for new stuff introduced.
Besides many bugfixes since Bash 3.2, Bash 4 will bring some interesting new features for shell users and scripters. See also Bash changes for a small general overview with more details.
Not all of the changes and news are included here, just the biggest or most interesting ones. The changes to completion, and the readline component are not covered. Though, if you're familiar with these parts of Bash (and Bash 4), feel free to write a chapter here.
The complete list of fixes and changes is in the CHANGES or NEWS file of your Bash 4 distribution.
The current available stable version is 4.2 release (February 13, 2011): New or changed commands and keywords The new "coproc" keyword Bash 4 introduces the concepts of coprocesses, a well known feature of other shells. The basic concept is simple: It will start any command in the background and set up an array that is populated with accessible files that represent the filedescriptors of the started process.In other words: It lets you start a process in background and communicate with its input and output data streams.
See The coproc keyword The new "mapfile" builtin The
mapfilebuiltin is able to map the lines of a file directly into an array. This avoids having to fill an array yourself using a loop. It enables you to define the range of lines to read, and optionally call a callback, for example to display a progress bar.See: The mapfile builtin command Changes to the "case" keyword The
caseconstruct understands two new action list terminators:The
;&terminator causes execution to continue with the next action list (rather than terminate thecaseconstruct).The
;;&terminator causes thecaseconstruct to test the next given pattern instead of terminating the whole execution.See The case statement Changes to the "declare" builtin The
-poption now prints all attributes and values of declared variables (or functions, when used with-f). The output is fully re-usable as input.The new option
-ldeclares a variable in a way that the content is converted to lowercase on assignment. For uppercase, the same applies to-u. The option-ccauses the content to be capitalized before assignment.
declare -Adeclares associative arrays (see below). Changes to the "read" builtin Thereadbuiltin command has some interesting new features.The
-toption to specify a timeout value has been slightly tuned. It now accepts fractional values and the special value 0 (zero). When-t 0is specified,readimmediately returns with an exit status indicating if there's data waiting or not. However, when a timeout is given, and thereadbuiltin times out, any partial data recieved up to the timeout is stored in the given variable, rather than lost. When a timeout is hit,readexits with a code greater than 128.A new option,
-i, was introduced to be able to preload the input buffer with some text (when Readline is used, with-e). The user is able to change the text, or press return to accept it.See The read builtin command Changes to the "help" builtin The builtin itself didn't change much, but the data displayed is more structured now. The help texts are in a better format, much easier to read.
There are two new options:
-ddisplays the summary of a help text,-mdisplays a manpage-like format. Changes to the "ulimit" builtin Besides the use of the 512 bytes blocksize everywhere in POSIX mode,ulimitsupports two new limits:-bfor max socket buffer size and-Tfor max number of threads. Expansions Brace Expansion The brace expansion was tuned to provide expansion results with leading zeros when requesting a row of numbers.See Brace expansion Parameter Expansion Methods to modify the case on expansion time have been added.
On expansion time you can modify the syntax by adding operators to the parameter name.
See Case modification on parameter expansion Substring expansion When using substring expansion on the positional parameters, a starting index of 0 now causes $0 to be prepended to the list (if the positional parameters are used). Before, this expansion started with $1:
# this should display $0 on Bash v4, $1 on Bash v3 echo ${@:0:1}Globbing There's a new shell option
globstar. When enabled, Bash will perform recursive globbing on**– this means it matches all directories and files from the current position in the filesystem, rather than only the current level.The new shell option
dirspellenables spelling corrections on directory names during globbing.See Pathname expansion (globbing) Associative Arrays Besides the classic method of integer indexed arrays, Bash 4 supports associative arrays.
An associative array is an array indexed by an arbitrary string, something like
declare -A ASSOC ASSOC[First]="first element" ASSOC[Hello]="second element" ASSOC[Peter Pan]="A weird guy"See Arrays Redirection There is a new
&>>redirection operator, which appends the standard output and standard error to the named file. This is the same as the good old>>FILE 2>&1notation.The parser now understands
|&as a synonym for2>&1 |, which redirects the standard error for a command through a pipe.See Redirection Interesting new shell variables
Variable Description BASHPID contains the PID of the current shell (this is different than what $$does!)PROMPT_DIRTRIM specifies the max. level of unshortened pathname elements in the prompt FUNCNEST control the maximum number of shell function recursions See Special parameters and shell variables Interesting new Shell Options The mentioned shell options are off by default unless otherwise mentioned.
Option Description checkjobscheck for and report any running jobs at shell exit compat*set compatiblity modes for older shell versions (influences regular expression matching in [[ ... ]]dirspellenables spelling corrections on directory names during globbing globstarenables recursive globbing with **lastpipe(4.2) to execute the last command in a pipeline in the current environment See List of shell options Misc
- If a command is not found, the shell attempts to execute a shell function named
command_not_found_handle, supplying the command words as the function arguments. This can be used to display userfriendly messages or perform different command searches- The behaviour of the
set -e(errexit) mode was changed, it now acts more intuitive (and is better documented in the manpage).- The output target for the
xtrace(set -x/set +x) feature is configurable since Bash 4.1 (previously, it was fixed tostderr): a variable named BASH_XTRACEFD can be set to the filedescriptor that should get the output- Bash 4.1 is able to log the history to syslog (only to be enabled at compile time in
config-top.h)
|
|
|
Jul 25, 2017 | eli.thegreenplace.net
June 11, 2013 at 19:27 Tags Linux , Software & Tools
Update (Jan 26, 2016): I posted a short update about my usage of persistent history.
For someone spending most of his time in front of a Linux terminal, history is very important. But traditional bash history has a number of limitations, especially when multiple terminals are involved (I sometimes have dozens open). Also it's not very good at preserving just the history you're interested in across reboots.
There are many approaches to improve the situation; here I want to discuss one I've been using very successfully in the past few months - a simple "persistent history" that keeps track of history across terminal instances, saving it into a dot-file in my home directory ( ~/.persistent_history ). All commands, from all terminal instances, are saved there, forever. I found this tremendously useful in my work - it saves me time almost every day.
Why does it go into a separate history and not the main one which is accessible by all the existing history manipulation tools? Because IMHO the latter is still worthwhile to be kept separate for the simple need of bringing up recent commands in a single terminal, without mixing up commands from other terminals. While the terminal is open, I want the press "Up" and get the previous command, even if I've executed a 1000 other commands in other terminal instances in the meantime.
Persistent history is very easy to set up. Here's the relevant portion of my ~/.bashrc :
log_bash_persistent_history() { [[ $(history 1) =~ ^\ *[0-9]+\ +([^\ ]+\ [^\ ]+)\ +(.*)$ ]] local date_part="${BASH_REMATCH[1]}" local command_part="${BASH_REMATCH[2]}" if [ "$command_part" != "$PERSISTENT_HISTORY_LAST" ] then echo $date_part "|" "$command_part" >> ~/.persistent_history export PERSISTENT_HISTORY_LAST="$command_part" fi } # Stuff to do on PROMPT_COMMAND run_on_prompt_command() { log_bash_persistent_history } PROMPT_COMMAND="run_on_prompt_command"The format of the history file created by this is:
2013-06-09 17:48:11 | cat ~/.persistent_history 2013-06-09 17:49:17 | vi /home/eliben/.bashrc 2013-06-09 17:49:23 | lsNote that an environment variable is used to avoid useless duplication (i.e. if I run ls twenty times in a row, it will only be recorded once).
OK, so we have ~/.persistent_history , how do we use it? First, I should say that it's not used very often, which kind of connects to the point I made earlier about separating it from the much higher-use regular command history. Sometimes I just look into the file with vi or tail , but mostly this alias does the trick for me:
alias phgrep='cat ~/.persistent_history|grep --color'The alias name mirrors another alias I've been using for ages:
alias hgrep='history|grep --color'Another tool for managing persistent history is a trimmer. I said earlier this file keeps the history "forever", which is a scary word - what if it grows too large? Well, first of all - worry not. At work my history file grew to about 2 MB after 3 months of heavy usage, and 2 MB is pretty small these days. Appending to the end of a file is very, very quick (I'm pretty sure it's a constant-time operation) so the size doesn't matter much. But trimming is easy:
tail -20000 ~/.persistent_history | tee ~/.persistent_historyTrims to the last 20000 lines. This should be sufficient for at least a couple of months of history, and your workflow should not really rely on more than that :-)
Finally, what's the use of having a tool like this without employing it to collect some useless statistics. Here's a histogram of the 15 most common commands I've used on my home machine's terminal over the past 3 months:
ls : 865 vi : 863 hg : 741 cd : 512 ll : 289 pss : 245 hst : 200 python : 168 make : 167 git : 148 time : 94 python3 : 88 ./python : 88 hpu : 82 cat : 80Some explanation: hst is an alias for hg st . hpu is an alias for hg pull -u . pss is my awesome pss tool , and is the reason why you don't see any calls to grep and find in the list. The proportion of Mercurial vs. git commands is likely to change in the very
|
|
|
history -a to preserve history
from multiple terminals. This is a very neat trick !!!get=
Bash history handling with multiple terminals
The bash session that is saved is the one for the terminal that is closed the latest. If you want to save the commands for every session, you could use the trick explained here.
export PROMPT_COMMAND='history -a'To quote the manpage: "If set, the value is executed as a command prior to issuing each primary prompt."
So every time my command has finished, it appends the unwritten history item to
~/.bashATTENTION: If you use multiple shell sessions and do not use this trick, you need to write the history manually to preserver it using the command history -a
See also:
|
|
|
Jul 07, 2017 | opensource.com
7 comments your Bash prompt.Anyone who has started a terminal in Linux is familiar with the default Bash prompt:
[ user @ $host ~ ] $But did you know is that this is completely customizable and can contain some very useful information? Here are a few hidden treasures you can use to customize your Bash prompt.
How is the Bash prompt set?The Bash prompt is set by the environment variable PS1 (Prompt String 1), which is used for interactive shell prompts. There is also a PS2 variable, which is used when more input is required to complete a Bash command.
[ dneary @ dhcp- 41 - 137 ~ ] $ export PS1 = "[Linux Rulez]$ "
[ Linux Rulez ] export PS2 = "... "
[ Linux Rulez ] if true ; then
... echo "Success!"
... fi
Success ! Where is the value of PS1 set?PS1 is a regular environment variable.
The system default value is set in /etc/bashrc . On my system, the default prompt is set with this line:
[ " $PS1 " = "\\s-\ \v \\ \$ " ] && PS1 = "[\u@\h \W]\ \$ "This tests whether the value of PS1 is \s-\v$ (the system default value), and if it is, it sets PS1 to the value [\u@\h \W]\\$ .
If you want to see a custom prompt, however, you should not be editing /etc/bashrc . You should instead add it to .bashrc in your Home directory.
What do \u, \h, \W, \s, and \v mean? More Linux resources
- What is Linux?
- What are Linux containers?
- Download Now: Linux commands cheat sheet
- Advanced Linux commands cheat sheet
- Our latest Linux articles
In the PROMPTING section of man bash , you can find a description of all the special characters in PS1 and PS2 . The following are the default options:
What other special strings can I use in the prompts?
- \u : Username
- \h : Short hostname
- \W : Basename of the current working directory ( ~ for home, the end of the current directory elsewhere)
- \s : Shell name ( bash or sh , depending on how the shell is called)
- \v : The shell's version
There are a number of special strings that can be useful.
- \d : Expands to the date in the format "Tue Jun 27"
- \D{fmt} : Allows custom date formats!see man strftime for the available options
- \D{%c} : Gives the date and time in the current locale
- \n : Include a new line (see multi-line prompts below)
- \w : The full path of the current working directory
- \H : The full hostname for the current machine
- \! : History number!you can run any previous command with its history number by using the shell history event designator ! followed by the number for the specific command you are interested in. (Using Linux history is yet another tutorial...)
There are many other special characters!you can see the full list in the PROMPTING section of the Bash man page .
Multi-line promptsIf you use longer prompts (say if you include \H or \w or a full date-time ), you may want to break things over two lines. Here is an example of a multi-line prompt, with the date, time, and current working directory on one line, and username @hostname on the second line:
Are there any other interesting things I can do?PS1 = "\D{%c} \w \n [\u@\H]$ "One thing people occasionally do is create colorful prompts. While I find them annoying and distracting, you may like them. For example, to change the date-time above to display in red text, the directory in cyan, and your username on a yellow background, you could try this:
PS1 = "\[\e[31m\]\D{%c}\[\e[0m\]
\[\e[36m\]\w\[\e[0m\] \n [\[\e[1;43m\]\u\[\e[0m\]@\H]$ "To dissect this:
- \[..\] declares some non-printed characters
- \e[.. is an escape character. What follows is a special escape sequence to change the color (or other characteristic) in the terminal
- 31m is red text ( 41m would be a red background)
- 36m is cyan text
- 1;43m declares a yellow background ( 1;33m would be yellow text)
- \[\e[0m\] at the end resets the colors to the terminal defaults
You can find more colors and tips in the Bash prompt HOWTO . You can even make text inverted or blinking! Why on earth anyone would want to do this, I don't know. But you can!
What are your favorite Bash prompt customizations? And which ones have you seen that drive you crazy? Let me know in the comments. Ben Cotton on 07 Jul 2017 Permalink I really like the Bash-Beautify setup by Chris Albrecht:
https://github.com/KeyboardCowboy/Bash-Beautify/blob/master/.bash_beautifyWhen you're in a version-controlled directory, it includes the VCS information (e.g. the git branch and status), which is really handy if you do development. Victorhck on 07 Jul 2017 Permalink An easy drag and drop interface to build your own .bashrc/PS1 configuration
've phun!
How Docker Is Growing Its Container Business (Apr 21, 2017, 07:00)
VIDEO: Ben Golub, CEO of Docker Inc., discusses the business of containers and where Docker is headed.
Understanding Shell Initialization Files and User Profiles in Linux (Apr 22, 2017, 10:00)
tecmint: Learn about shell initialization files in relation to user profiles for local user management in Linux.Cockpit An Easy Way to Administer Multiple Remote Linux Servers via a Web Browser (Apr 23, 2017, 18:00)
Cockpit is a free and open source web-based system management tool where users can easily monitor and manage multiple remote Linux servers.The Story of Getting SSH Port 22 (Apr 24, 2017, 13:00)
It's no coincidence that the SSH protocol got assigned to port 22.How To Suspend A Process And Resume It Later In Linux (Apr 24, 2017, 11:00)
This brief tutorial describes how to suspend or pause a running process and resume it later in Unix-like operating systems.
ShellCheck -A Tool That Shows Warnings and Suggestions for Shell Scripts (Apr 25, 2017, 06:00)
tecmint: ShellCheck is a static analysis tool that shows warnings and suggestions concerning bad code in bash/sh shell scripts.Quick guide for Linux check disk space (Apr 26, 2017, 14:00)
Do you know how much space is left on your Linux system?
|
|
|
Jul 16, 2017 | www.ostechnix.com
Today, I have stumbled upon a collection of useful BASH scripts for heavy commandline users. These scripts, known as Bash-Snippets , might be quite helpful for those who live in Terminal all day. Want to check the weather of a place where you live? This script will do that for you. Wondering what is the Stock prices? You can run the script that displays the current details of a stock. Feel bored? You can watch some youtube videos. All from commandline. You don't need to install any heavy memory consumable GUI applications.Bash-Snippets provides the following 12 useful tools:
Bash-Snippets – A Collection Of Useful BASH Scripts For Heavy Commandline Users Installation
- currency – Currency converter.
- stocks – Provides certain Stock details.
- weather – Displays weather details of your place.
- crypt – Encrypt and decrypt files.
- movies – Search and display a movie details.
- taste – Recommendation engine that provides three similar items like the supplied item (The items can be books, music, artists, movies, and games etc).
- short – URL Shortner
- geo – Provides the details of wan, lan, router, dns, mac, and ip.
- cheat – Provides cheat-sheets for various Linux commands .
- ytview – Watch YouTube from Terminal.
- cloudup – A tool to backup your GitHub repositories to bitbucket.
- qrify – Turns the given string into a qr code.
You can install these scripts on any OS that supports BASH.
First, clone the GIT repository using command:
git clone https://github.com/alexanderepstein/Bash-SnippetsSample output would be:
Cloning into 'Bash-Snippets'... remote: Counting objects: 1103, done. remote: Compressing objects: 100% (45/45), done. remote: Total 1103 (delta 40), reused 55 (delta 23), pack-reused 1029 Receiving objects: 100% (1103/1103), 1.92 MiB | 564.00 KiB/s, done. Resolving deltas: 100% (722/722), done.Go to the cloned directory:
cd Bash-Snippets/Git checkout to the latest stable release:
git checkout v1.11.0Finally, install the Bash-Snippets using command:
sudo ./install.shThis will ask you which scripts to install. Just type Y and press ENTER key to install the respective script. If you don't want to install a particular script, type N and hit ENTER.
|
|
|
Jul 16, 2017 | github.com
If i'm not wrong, all our download folder is pretty Sloppy compare with others because most of the downloaded files are sitting over there and we can't delete blindly, which leads to lose some important files. Also not possible to create bunch of folders based on the files and move appropriate files into folder manually.So, what to do to avoid this ? Better to organize files with help of classifier, later we can delete unnecessary files easily. Classifier app was written in Python.
How to Organize directory ? Simple navigate to corresponding directory, where you want to organize/classify your files and run the
classifiercommand, it will take few mins or more depends on the directory files count or quantity.Make a note, there is no undo option, if you want to go back. So, finalize before run classifier in directory. Also, it wont move folders.
Install Classifier in Linux through pippip is a recommended tool for installing Python packages in Linux. Use pip command instead of package manager to get latest build.
For Debian based systems.
$ sudo apt-get install python-pipFor RHEL/CentOS based systems.
$ sudo yum install python-pipFor Fedora
$ sudo dnf install python-pipFor openSUSE
$ sudo zypper install python-pipFor Arch Linux based systems
$ sudo pacman -S python-pipFinally run the pip tool to install Classifier on Linux.
$ sudo pip install classifierOrganize pattern files into specific foldersFirst i will go with default option which will organize pattern files into specific folders. This will create bunch of directories based on the file types and move them into specific folders.
See my directory, how its looking now (Before run classifier command).
$ pwd /home/magi/classifier $ ls -lh total 139M -rw-r--r-- 1 magi magi 4.5M Mar 21 21:21 Aaluma_Doluma.mp3 -rw-r--r-- 1 magi magi 26K Mar 21 21:12 battery-monitor_0.4-xenial_all.deb -rw-r--r-- 1 magi magi 24K Mar 21 21:12 buku-command-line-bookmark-manager-linux.png -rw-r--r-- 1 magi magi 0 Mar 21 21:43 config.php -rw-r--r-- 1 magi magi 25 Mar 21 21:13 core.py -rw-r--r-- 1 magi magi 101K Mar 21 21:12 drawing.svg -rw-r--r-- 1 magi magi 86M Mar 21 21:12 go1.8.linux-amd64.tar.gz -rw-r--r-- 1 magi magi 28 Mar 21 21:13 index.html -rw-r--r-- 1 magi magi 27 Mar 21 21:13 index.php -rw-r--r-- 1 magi magi 48M Apr 30 2016 Kabali Tamil Movie _ Official Teaser _ Rajinikanth _ Radhika Apte _ Pa Ranjith-9mdJV5-eias.webm -rw-r--r-- 1 magi magi 28 Mar 21 21:12 magi1.txt -rw-r--r-- 1 magi magi 66 Mar 21 21:12 ppa.py -rw-r--r-- 1 magi magi 1.1K Mar 21 21:12 Release.html -rw-r--r-- 1 magi magi 45K Mar 21 21:12 v0.4.zipNavigate to corresponding directory where you want to organize files, then run
classifiercommand without any option to achieve it.$ classifier Scanning Files Done!See the Directory look, after run classifier command
$ ls -lh total 44K drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Archives -rw-r--r-- 1 magi magi 0 Mar 21 21:43 config.php -rw-r--r-- 1 magi magi 25 Mar 21 21:13 core.py drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 DEBPackages drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Documents -rw-r--r-- 1 magi magi 28 Mar 21 21:13 index.html -rw-r--r-- 1 magi magi 27 Mar 21 21:13 index.php drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Music drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Pictures -rw-r--r-- 1 magi magi 66 Mar 21 21:12 ppa.py -rw-r--r-- 1 magi magi 1.1K Mar 21 21:12 Release.html drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 VideosMake a note, this will organize only general category files such docs, audio, video, pictures, archive, etc and wont organize .py, .html, .php, etc.,.
Classify specific file types into specific folderTo Classify specific file types into specific folder, just add
-st(mention the file type) &-sf(folder name) followed by classifier command.For best understanding, i'm going to move
.py,.html&.phpfiles intoDevelopmentfolder. See the exact command to achieve it.$ classifier -st .py .html .php -sf "Development" Scanning Files Done!If the folder doesn't exit, it will create the new one and organize the files into that. See the following output. It created
Developmentdirectory and moved all the files inside the directory.$ ls -lh total 28K drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Archives drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 DEBPackages drwxr-xr-x 2 magi magi 4.0K Mar 21 21:51 Development drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Documents drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Music drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 Pictures drwxr-xr-x 2 magi magi 4.0K Mar 21 21:28 VideosFor better clarification, i have listed Development folder files.
$ ls -lh Development/ total 12K -rw-r--r-- 1 magi magi 0 Mar 21 21:43 config.php -rw-r--r-- 1 magi magi 25 Mar 21 21:13 core.py -rw-r--r-- 1 magi magi 28 Mar 21 21:13 index.html -rw-r--r-- 1 magi magi 27 Mar 21 21:13 index.php -rw-r--r-- 1 magi magi 0 Mar 21 21:43 ppa.py -rw-r--r-- 1 magi magi 0 Mar 21 21:43 Release.htmlTo Organize files by Date. It will organize current directory files based on the date.
$ classifier -dtTo save organized files in different location, add
-d(source directory) &-o(destination directory) followed by classifier command.$ classifier -d /home/magi/organizer -o /home/magi/2g
|
|
|
Jul 06, 2017 | www.ibm.com
Demystify test, [, [[, ((, and if-then-else
Ian Shields
Published on February 20, 2007 > > >
|
|
|
economistsview.typepad.com
Do you sometimes wonder how to use parameters with your scripts, and how to pass them to internal functions or other scripts? Do you need to do simple validity tests on parameters or options, or perform simple extraction and replacement operations on the parameter strings? This tip helps you with parameter use and the various parameter expansions available in the bash shell.
|
|
|
Jun 18, 2017 | opensource.com
About conditional, substring, and substitution parameter expansion operators Conditional parameter expansionConditional parameter expansion allows branching on whether the parameter is unset, empty, or has content. Based on these conditions, the parameter can be expanded to its value, a default value, or an alternate value; throw a customizable error; or reassign the parameter to a default value. The following table shows the conditional parameter expansions-each row shows a parameter expansion using an operator to potentially modify the expansion, with the columns showing the result of that expansion given the parameter's status as indicated in the column headers. Operators with the ':' prefix treat parameters with empty values as if they were unset.
parameter expansion unset var var="" var="gnu" ${var-default} default - gnu ${var:-default} default default gnu ${var+alternate} - alternate alternate ${var:+alternate} - - alternate ${var?error} error - gnu ${var:?error} error error gnu The = and := operators in the table function identically to - and :- , respectively, except that the = variants rebind the variable to the result of the expansion.
As an example, let's try opening a user's editor on a file specified by the OUT_FILE variable. If either the EDITOR environment variable or our OUT_FILE variable is not specified, we will have a problem. Using a conditional expansion, we can ensure that when the EDITOR variable is expanded, we get the specified value or at least a sane default:
$ echo ${ EDITOR } /usr/bin/vi $ echo ${ EDITOR :- $( which nano ) } /usr/bin/vi $ unset EDITOR $ echo ${ EDITOR :- $( which nano ) } /usr/bin/nanoBuilding on the above, we can run the editor command and abort with a helpful error at runtime if there's no filename specified:
$ ${ EDITOR :- $( which nano ) } ${ OUT_FILE :? Missing filename } bash: OUT_FILE: Missing filenameSubstring parameter expansionParameters can be expanded to just part of their contents, either by offset or by removing content matching a pattern. When specifying a substring offset, a length may optionally be specified. If running Bash version 4.2 or greater, negative numbers may be used as offsets from the end of the string. Note the parentheses used around the negative offset, which ensure that Bash does not parse the expansion as having the conditional default expansion operator from above:
$ location = " CA 90095 " $ echo " Zip Code: ${ location : 3 } " Zip Code: 90095 $ echo " Zip Code: ${ location : (-5) } " Zip Code: 90095 $ echo " State: ${ location : 0 : 2 } " State: CAAnother way to take a substring is to remove characters from the string matching a pattern, either from the left edge with the # and ## operators or from the right edge with the % and %% operators. A useful mnemonic is that # appears left of a comment and % appears right of a number. When the operator is doubled, it matches greedily, as opposed to the single version, which removes the most minimal set of characters matching the pattern.
var="open source" parameter expansion offset of 5
length of 4${var:offset} source ${var:offset:length} sour pattern of *o? ${var#pattern} en source ${var##pattern} rce pattern of ?e* ${var%pattern} open sour ${var%%pattern} o The pattern-matching used is the same as with filename globbing: * matches zero or more of any character, ? matches exactly one of any character, [...] brackets introduce a character class match against a single character, supporting negation ( ^ ), as well as the posix character classes, e.g. . By excising characters from our string in this manner, we can take a substring without first knowing the offset of the data we need:
$ echo $ PATH /usr/local/bin:/usr/bin:/bin $ echo " Lowest priority in PATH: ${ PATH ## *: } " Lowest priority in PATH: /bin $ echo " Everything except lowest priority: ${ PATH % :* } " Everything except lowest priority: /usr/local/bin:/usr/bin $ echo " Highest priority in PATH: ${ PATH %% :* } " Highest priority in PATH: /usr/local/binSubstitution in parameter expansionThe same types of patterns are used for substitution in parameter expansion. Substitution is introduced with the / or // operators, followed by two arguments separated by another / representing the pattern and the string to substitute. The pattern matching is always greedy, so the doubled version of the operator, in this case, causes all matches of the pattern to be replaced in the variable's expansion, while the singleton version replaces only the leftmost.
var="free and open" parameter expansion pattern of
string of _${var/pattern/string} free_and open ${var//pattern/string} free_and_open The wealth of parameter expansion modifiers transforms Bash variables and other parameters into powerful tools beyond simple value stores. At the very least, it is important to understand how parameter expansion works when reading Bash scripts, but I suspect that not unlike myself, many of you will enjoy the conciseness and expressiveness that these expansion modifiers bring to your scripts as well as your interactive sessions.
|
|
|
Apr 26, 2017 | www.tecmint.com
by Aaron Kili | Published: April 24, 2017 | April 24, 2017
Download Your Free eBooks NOW - 10 Free Linux eBooks for Administrators | 4 Free Shell Scripting eBooksShellCheck is a static analysis tool that shows warnings and suggestions concerning bad code in bash/sh shell scripts. It can be used in several ways: from the web by pasting your shell script in an online editor (Ace – a standalone code editor written in JavaScript) in https://www.shellcheck.net (it is always synchronized to the latest git commit, and is the simplest way to give ShellCheck a go) for instant feedback. Alternatively, you can install it on your machine and run it from the terminal, integrate it with your text editor as well as in your build or test suites.
There are three things ShellCheck does primarily:
- It points out and explains typical beginner's syntax issues that cause a shell to give cryptic error messages.
- It points out and explains typical intermediate level semantic problems that cause a shell to behave strangely and counter-intuitively.
- It also points out subtle caveats, corner cases and pitfalls that may cause an advanced user's otherwise working script to fail under future circumstances.
In this article, we will show how to install and use ShellCheck in the various ways to find bugs or bad code in your shell scripts in Linux.
How to Install and Use ShellCheck in LinuxShellCheck can be easily installed locally through your package manager as shown.
On Debian/Ubuntu# apt-get install shellcheckOn RHEL/CentOS# yum -y install epel-release # yum install ShellCheckOn Fedora# dnf install ShellCheckOnce ShellCheck installed, let's take a look at how to use ShellCheck in the various methods we mentioned before.
Using ShellCheck From the WebGo to https://www.shellcheck.net and paste your script in the Ace editor provided, you will view the output at the bottom of the editor as shown in the screen shot below.
In the following example, the test shell script consists of the following lines:
#!/bin/bash #declare variables MINARGS=2 E_NOTROOT=50 E_MINARGS=100 #echo values of variables echo $MINARGS echo $E_NONROOT exit 0;
ShellCheck – Online Shell Script Analysis Tool
From the screenshot above, the first two variables E_NOTROOT and E_MINARGS have been declared but are unused, ShellCheck reports these as "suggestive errors":
SC2034: E_NOTROOT appears unused. Verify it or export it. SC2034: E_MINARGS appears unused. Verify it or export it.Then secondly, the wrong name (in the statement echo $E_NONROOT ) was used to echo variable E_NOTROOT , that is why ShellCheck shows the error:
SC2153: Possible misspelling: E_NONROOT may not be assigned, but E_NOTROOT isAgain when you look at the echo commands , the variables have not been double quoted (helps to prevent globbing and word splitting), therefore Shell Check shows the warning:
SC2086: Double quote to prevent globbing and word splitting.Using ShellCheck From the TerminalYou can also run ShellCheck from the command-line, we'll use the same shell script above as follows:
$ shellcheck test.sh
ShellCheck – Checks Bad Code in Shell Scripts Using ShellCheck From the Text Editor
You can also view ShellCheck suggestions and warnings directly in a variety of editors, this is probably a more efficient way of using ShellCheck, once you save a files, it shows you any errors in the code.
In Vim , use ALE or Syntastic (we will use this):
Start by installing Pathogen so that it's easy to install syntastic. Run the commands below to get the pathogen.vim file and the directories it needs:
# mkdir -p ~/.vim/autoload ~/.vim/bundle && curl -LSso ~/.vim/autoload/pathogen.vim https://tpo.pe/pathogen.vimThen add this to your ~/.vimrc file:
execute pathogen#infect()Once you have installed pathogen, and you now can put syntastic into