Piping Vim Buffer Through Unix Filters
( ! and !! Commands)

• Introduction
• Use of shell as a filter for execution of command or command sequences vi
• Using vi as a simple program generator
• Range command
• Putting visual block boundaries into the command line for filtering

Introduction

Everybody knows how pipes work at the command prompt. Text originates from some source, is processed via one or more filters and output goes either to the console display or is redirected to a file.

VI takes this same paradigm of pipes and filters and wraps it in a editor user interface in which the pipe is applied to editing buffer both as a source and as a destination. A VI pipe is thus can alter the buffer using standard Unix filters that instantly become a part of editor toolbox. This is an extremely elegant idea. The ability to pass nearly arbitrary chunks of text through any UNIX filter adds incredible flexibility  at no "additional cost" in size or performance of the editor. 

That was a stoke in genius in design on vi.  And still remains one of the most advanced features vi(and by extension VIM) has. Which, unfortunately, few people understand and use to the full extent.

Pipes can be used both from command line commands:

• ex commands

• vi-mode commands. 

With the vi filter command

:[address-range] ! external-command-name

you can process a range of lines with an external program. This enables you to do much more effective editing. For example, the command:

:1,$ ! indent

will beautify your program using  standard Unix beautifier (indent).  This is a classic example of using piping in vi. You can also create a macro for this operation using keystroke remapping. Instead of indent you can use any available batch beautifier most suitable for the language that you are using.

Without any filter, the command !, prompts for the name of a UNIX command (which should be a filter), then passes selected lines through the filter, replacing those selected line in the vi buffer with the output of the filter command.

Actually the ex  % operator is the easiest way to filter all the lines in your buffer, and this classic vi idiom should look like:

:%!indent

To edit the current line you need to use:

!!command

To edit the next paragraph: 

!}!command

If you use any keystrokes that move cursor you need to use them so that they move cursor more than one line ( G, { }, ( ), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line.

You can also use a slash (/) followed by a regular expression  and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

The entire sequence can be preceded by a number to repetitions:

20!!grep -v '^#'

or:

!20!grep -v '^#'

NOTE: To move one paragraph down you can use !}. That's a very convenient, often  used idiom.  

Among Unix filters that can be used I would like to mention AWK and  Perl. Of course, any filter can be used, for example  tr can be used for case conversion of deletion of  some characters (tr '[a-z]' '[A-Z]').

As most Unix/Linux sysadmin know some Perl along with bash it is natural choice that gradually replaces AWK (although AWK remains attractive for Python users because it is a more simple and still quite powerful language). 

For example,  to pipe all the buffer you need to specify the filer (which can be written by you Perl script) after ! command:

:1,$ ! perltidy

will beautify your program using perltidy.  Beautifying your script is a classic example of using piping in vi. You can also create a macro using keystroke remapping. Instead of beautifier you can use any other filter. This way for example you can temporary remove all comments from the script as it simplifies understanding  and then reread the file with comments back into the buffer:

To do the same for selection you need to provide boundary of selection to the command. The easiest way to do this is to use visual mode:  v, V, or Ctrl-v commands to select the part of your script that you want to process. In this  case boundary of your selection are defined by two mark '< and '> which represent upper and lower boundary correspondingly. They will be put on the command like automatically as soon as you enter : .  For example:

'<,'>! perl -nle 'print "\#".$_'

(this operation is called "commenting out" and it can also be performed via a plugin, but it is a still a good illustration of using Perl filters with piping).

You can return to original text is you pipe worked incorrectly

Vim will run pass your selection to the script as STDIN  and insert standard output in place of it. If you incorrectly overwrite your selection or the whole buffer you can restore it using undo command (u). As buffer is an "in-memory" mapping of the file, rereading file from the disk using r command also allows you to restore the original text.

Repeating previous pipe command

To repeat the previous pipe command, you need to type:

! object !

Use of shell as a filter for execution of command or command sequences vi

Shell can be used as a filter, which gives you an ability to replace shell command that you typed in the current line (lines) with their output.  That's actually great, unappreciated and underutilized vi capability.

Here is a relevant quote from Vim tutorial (Vim Color Editor HOW-TO (Vi Improved with syntax color highlighting) Vi Tutorial)

Create a line in your file containing just the word who and absolutely no other text. Put the cursor on this line, and press !! This command is analogous to dd, cc, or yy, but instead of deleting, changing, or yanking the current line, it filters the current line. When you press the second !, the cursor drops down to the lower left corner of the screen and a single ! is displayed, prompting you to enter the name of a filter.

As the filter name, type sh and press the Return key. sh (the Bourne shell) is a filter! It reads standard input, does some processing of its input (that is, executes commands), and sends its output (the output of those commands) to standard output. Filtering the line containing who through sh causes the line containing who to be replaced with a list of the current users on the system - right in your file!

Try repeating this process with date. That is, create a line containing nothing but the word date, then put the cursor on the line, and press !!sh and the Return key. The line containing date is replaced with the output of the date command.

Put your cursor on the first line of the output of who. Count the number of lines. Suppose, for example, the number is six. Then select those six lines to be filtered through sort; press 6!!sort and the Return key. The six lines will be passed through sort, and sort's output replaces the original six lines.

The filter command can only be used on complete lines, not on characters or words.

Some other filter commands (here, < CR > means press Return):

• !/the < CR > sort < CR > : Sort from the current line up to and including the next line containing the
• !1Ggrep the < CR > : Replace from the current line to and including Line 1 with just the lines that contain the
• !Gawk '{print $1}' < CR > : From the current line to the end of file, replace every line with just its first word.

For example, you need to insert into document or script you are editing the  IP address and netmask of the server on which you are working. you can just execute 'ifconfig eth0 | grep Mask' using ! command and get it directly in the place where it is needed. For example:

25 ! bash

Using vi as a simple program generator

You can use internal editor piping for a lot of interesting stuff. For example you can read list of files in the current directory into the buffer, convert then into some commands and then execute them in shell:

$ vim
:r! ls *.c
:%s/\(.*\).c/mv & \1.bla
:w !sh
:q!

or as one liner:

:r! ls *.c :%s/\(.*\).c/mv & \1.bla :w 
!sh :q!

You can format text without the fmt program using instead perl's Text::Wrap module (especially useful if you are working in Cygwin):

:% ! perl -00 -MText::Wrap -ne 'BEGIN{$Text::Wrap::columns=40} 
print wrap("\t","",$_)'

Numbering items with pattern matches in vi

:! type foo.html | perl -pe"BEGIN{$i=1;} ++$i if s:<foo>:<bar$i>:;" > bar.html

Use filtering with a tool like perl to get variable interpolation into search patterns, unless you are lucky enough to have compiled-in support for perl or other tools that allow you to do this.

Range command

Another quite useful and powerful Vim command is range command:

:range g[lobal][!]/pattern/cmd

The global commands work by first scanning through the [range] of of the lines and marking each line where a match occurs. In a second scan the [cmd] is executed for each marked line with its line number prepended. If a line is changed or deleted its mark disappears. The default for the [range] is the whole file.

Note: Non-Ex commands (normal mode commands) can be also executed from the command line using :norm[al]non-ex command  mechanism.

Putting visual block boundaries into the command line for filtering

In the previous section, we saw that we could execute shell commands from within Vim, causing Vim to be moved to a background process while the command executed. There are more practical use cases for this feature, mainly, having the ability to manipulate our content via external filters.

Let’s use as an example tr command. This command translates character set to another. For example we might need to convert some characters that appear in the text copied from the WEB to "normal" characters acceptable to particular scripting language interpreter.

If we selected visual block and types ":" after will  we’d enter COMMAND-LINE mode with boundaries on the block already put into the command line. After that  we can use tr command

:'<,'> ! tr '[:lower:]' '[:upper:]'

NEWS CONTENTS

• 20181130 : Filtering Text Through a Command ( Nov 30, 2018 )
• 20010807 : Tip #95 - How do I pipe the output from ex commands into the text buffer? ( Tip #95 - How do I pipe the output from ex commands into the text buffer?, Aug 7, 2001 )
• 20010807 : Questions & Answers about using tags with Vim ( Questions & Answers about using tags with Vim, )
• 20010807 : Chapter 30 vi Tips and Tricks ( Chapter 30 vi Tips and Tricks, )

Old News ;-)

[Nov 30, 2018] Filtering Text Through a Command

... ...

In vi, text is filtered through a Unix command by typing an exclamation mark followed by any of vi's movement keystrokes that indicate a block of text, and then by the Unix command line to be executed. For example:

!)command

will pass the next sentence through command.

There are a few unusual aspects of the way vi acts when you use this feature:

• The exclamation mark doesn't appear on your screen right away. When you type the keystroke(s) for the text object you want to filter, the exclamation mark appears at the bottom of the screen, but the character you type to reference the object does not.
• Text blocks must be more than one line, so you can use only the keystrokes that would move more than one line ( G, { }, (), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next 10 lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash (/) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command.
• Entire lines are affected. For example, if your cursor is in the middle of a line and you issue a command to go to the end of the next sentence, the entire lines containing the beginning and end of the sentence will be changed, not just the sentence itself.^[32]
• There is a special text object that can be used only with this command syntax: you can specify the current line by entering a second exclamation mark:

  !!command

  Remember that either the entire sequence or the text object can be preceded by a number to repeat the effect. For instance, to change lines 96 through 99 as in the previous example, you could position the cursor on line 96 and enter either:

  4!!sort

  or:

  !4!sort

As another example, assume you have a portion of text in a file that you want to change from lowercase to uppercase letters. You could process that portion with the tr command to change the case. In this example, the second sentence is the block of text that will be filtered through the command:

 One sentence before.With a screen editor you can scroll the page
 move the cursor, delete lines, insert characters,
 and more, while seeing the results of your edits
 as you make them.
 One sentence after.

Keystrokes	Results
!)	One sentence after. ~ ~ ~ ! An exclamation mark appears on the last line to prompt you for the Unix command. The ) indicates that a sentence is the unit of text to be filtered.
tr '[:lower:]' '[:upper:]'	One sentence before.WITH A SCREEN EDITOR YOU CAN SCROLL THE PAGE MOVE THE CURSOR, DELETE LINES, INSERT CHARACTERS, AND MORE, WHILE SEEING THE RESULTS OF YOUR EDITS AS YOU MAKE THEM. One sentence after. Enter the Unix command and press ENTER. The input is replaced by the output.

To repeat the previous command, the syntax is:

!object !

It is sometimes useful to send sections of a coded document to nroff to be replaced by formatted output. (Or, when editing electronic mail, you might use the fmt program to "beautify" your text before sending the message.) Remember that the "original" input is replaced by the output. Fortunately, if there is a mistake-such as an error message being sent instead of the expected output-you can undo the command and restore the lines.

[Aug 7, 2001] Tip #95 - How do I pipe the output from ex commands into the text buffer?

This is a *request* for a tip. I need to be able to pipe the output of a :blah ex command into the vim text buffer for editing. I wanted to do this many times for different reasons and could never find a way! I would just love to be able to do :hi --> textBuffer and examine the output at my own leasure scrolling up and down and using vim search commands on it. Same thing for :set all, and other things. Considering that cut and paste is horrible in windows, I can't for example do :set guioptions? then cut and paste! So I have to retype it, or cut and paste from the help manual. I really want to be able to pipe the output of ex commands into the text buffer. Can someone help me?

Questions & Answers about using tags with Vim

Yegappan, August 7, 2001 11:45

You can use the :redir command to redirect the output of an ex command to a register and then paste the contents of the register into a Vim buffer.

For example: :redir @a
:set all
:redir END Now, register 'a' will have the output of the "set all" ex command. You
can paste this into a Vim buffer. You can also write a Vim function
to do the above. For more information, read :help redir Anonymous, August 7, 2001 14:13 Wow!!! That's awesome!! Exactly what I want!

[email protected], July 25, 2002 11:28

This may be obvious to experts, but it took me a very long time to figure it out, because Google searches on terms like 'pipe', 'buffer', 'shell', etc never brought it to my attention.

However, you can pipe the contents of the file currently being edited (the current buffer) to a shell command, and replace the current file/buffer with the _output_ of that command, using this: :%! [cmd] ie, if you didn't know the :retab command (as for a long time I didn't), you could expand tabs using basic unix commands like ":%! expand -t 4". Wish I'd known this a long time ago, so I'm posting it here in the hopes that others might find it :-)

[email protected], February 18, 2004 14:01

The answer is (for ex.): :read !ls ~ and :help :read for more info :-)

Grateful, September 27, 2004 12:10

Thanks Anonymous and Yegappan, I've long wanted to do this too, but never known how. Great initiative Anonymous!

Chapter 30 vi Tips and Tricks

From the old edition of Unix Power Tools

30.37 Neatening Lines

Have you made edits that left some of your lines too short or long? The fmt utility can clean that up. Here's an example. Let's say you're editing a file (email message, whatever) in vi and the lines aren't even. They look like this:

This file is a mess
with some short lines
and some lines that are too long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

You put your cursor on the first line and type (in command mode):

5!!fmt

which means " filter 5 lines through fmt." Then the lines will look like this:

This file is a mess with some short lines and some lines that are too
long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

This is handiest for formatting paragraphs. Put your cursor on the first line of the paragraph and type (in command mode):

!}fmt

If you don't have any text in your file that needs to be kept as is, you can neaten the whole file at once by typing:

:%!fmt

There are a few different versions of fmt, some fancier than others. Most of the articles in the chapter about editing-related tools can be handy too. For example,

• recomment (35.4) reformats program comment blocks.

• cut (35.14) can remove columns, fields, or shorten lines;

• tr (35.11) can do other transformations.

To neaten columns, try filtering through with the setup in article 35.22. In general, if the utility will read its standard input and write converted text to its standard output, you can use the utility as a vi filter.

- JP

Recommended Links

Softpanorama Recommended

Top articles

Sites

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: July 28, 2019