Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Scriptorama: A Slightly Skeptical View
on Scripting Languages

News Introduction Recommended Books Recommended Links Programming Languages Design Recommended Papers Scripting Languages for Java vs. Pure Java
Software Engineering Anti-OO John Ousterhout Larry Wall Shell Giants Software Prototyping Software Life Cycle Models
Shells AWK Perl Perl Warts and Quirks Python PHP Javascript
Ruby Tcl/Tk R programming language Rexx Lua S-lang JVM-based scripting languages
Pipes Regex Program understanding Beautifiers and Pretty Printers Neatbash -- a simple bash prettyprinter Neatperl -- a simple Perl prettyprinter  
Brooks law Conway Law KISS Principle Featuritis Software Prototyping Unix Component Model  
Programming as a Profession Programming style Language design and programming quotes History Humor Random Findings Etc

This is the central page of the Softpanorama WEB site because I am strongly convinced that the development of scripting languages, not the replication of the efforts of BSD group undertaken by Stallman and Torvalds is the central part of open source. See Scripting languages as VHLL for more details.

 
Ordinarily technology changes fast. But programming languages are different: programming languages are not just technology, but what programmers think in.

They're half technology and half religion. And so the median language, meaning whatever language the median programmer uses, moves as slow as an iceberg.

Paul Graham: Beating the Averages

Libraries are more important that the language.

Donald Knuth


Introduction

A fruitful way to think about language development is to consider it a to be special type of theory building. Peter Naur suggested that programming in general is theory building activity in his 1985 paper "Programming as Theory Building". But idea is especially applicable to compilers and interpreters. What Peter Naur failed to understand was that design of programming languages has religious overtones and sometimes represent an activity, which is pretty close to the process of creating a new, obscure cult ;-). Clueless academics publishing junk papers at obscure conferences are high priests of the church of programming languages. Some, like Niklaus Wirth and Edsger W. Dijkstra, (temporary) reached the status close to (false) prophets :-).

On a deep conceptual level building of a new language is a human way of solving complex problems. That means that complier construction in probably the most underappreciated paradigm of programming of large systems. Much more so then greatly oversold object-oriented programming. OO benefits are greatly overstated.

For users, programming languages distinctly have religious aspects, so decisions about what language to use are often far from being rational and are mainly cultural.  Indoctrination at the university plays a very important role. Recently they were instrumental in making Java a new Cobol.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. The latter includes libraries, IDE, books, level of adoption at universities,  popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things.

A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked.  This is  a story behind success of  Java and PHP. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all the most interesting Perl features surgically removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

Progress in programming languages has been very uneven and contain several setbacks. Currently this progress is mainly limited to development of so called scripting languages.  Traditional high level languages field is stagnant for many decades.  From 2000 to 2017 we observed the huge sucess of Javascript; Python encroached in Perl territory (including genomics/bioinformatics) and R in turn start squeezing Python in several areas. At the same time Ruby despite initial success remained niche language.  PHP still holds its own in web-site design.

Some observations about scripting language design and  usage

At the same time there are some mysterious, unanswered question about factors that help the particular scripting language to increase its user base, or fail in popularity. Among them:

Nothing succeed like success

Those are difficult questions to answer without some way of classifying languages into different categories. Several such classifications exists. First of all like with natural languages, the number of people who speak a given language is a tremendous force that can overcome any real of perceived deficiencies of the language. In programming languages, like in natural languages nothing succeed like success.

The second interesting category is number of applications written in particular language that became part of Linux or, at least, are including in standard RHEL/FEDORA/CENTOS or Debian/Ubuntu repository.

The third relevant category is the number and quality of books for the particular language.

Complexity Curse

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that a language with simpler core, or even simplistic core Basic, Pascal) have better chances to acquire high level of popularity.

The underlying fact here probably is that most programmers are at best mediocre and such programmers tend on intuitive level to avoid more complex, more rich languages and prefer, say, Pascal to PL/1 and PHP to Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because of the progress of hardware, availability of compilers and not the least, because it was associated with OO exactly at the time OO became a mainstream fashion).

Complex non-orthogonal languages can succeed only as a result of a long period of language development (which usually adds complexly -- just compare Fortran IV with Fortran 99; or PHP 3 with PHP 5 ) from a smaller core. Attempts to ride some fashionable new trend extending existing popular language to this new "paradigm" also proved to be relatively successful (OO programming in case of C++, which is a superset of C).

Historically, few complex languages were successful (PL/1, Ada, Perl, C++), but even if they were successful, their success typically was temporary rather then permanent  (PL/1, Ada, Perl). As Professor Wilkes noted   (iee90):

Things move slowly in the computer language field but, over a sufficiently long period of time, it is possible to discern trends. In the 1970s, there was a vogue among system programmers for BCPL, a typeless language. This has now run its course, and system programmers appreciate some typing support. At the same time, they like a language with low level features that enable them to do things their way, rather than the compiler’s way, when they want to.

They continue, to have a strong preference for a lean language. At present they tend to favor C in its various versions. For applications in which flexibility is important, Lisp may be said to have gained strength as a popular programming language.

Further progress is necessary in the direction of achieving modularity. No language has so far emerged which exploits objects in a fully satisfactory manner, although C++ goes a long way. ADA was progressive in this respect, but unfortunately it is in the process of collapsing under its own great weight.

ADA is an example of what can happen when an official attempt is made to orchestrate technical advances. After the experience with PL/1 and ALGOL 68, it should have been clear that the future did not lie with massively large languages.

I would direct the reader’s attention to Modula-3, a modest attempt to build on the appeal and success of Pascal and Modula-2 [12].

Complexity of the compiler/interpreter also matter as it affects portability: this is one thing that probably doomed PL/1 (and later Ada), although those days a new language typically come with open source compiler (or in case of scripting languages, an interpreter) and this is less of a problem.

Here is an interesting take on language design from the preface to The D programming language book 9D language failed to achieve any significant level of popularity):

Programming language design seeks power in simplicity and, when successful, begets beauty.

Choosing the trade-offs among contradictory requirements is a difficult task that requires good taste from the language designer as much as mastery of theoretical principles and of practical implementation matters. Programming language design is software-engineering-complete.

D is a language that attempts to consistently do the right thing within the constraints it chose: system-level access to computing resources, high performance, and syntactic similarity with C-derived languages. In trying to do the right thing, D sometimes stays with tradition and does what other languages do, and other times it breaks tradition with a fresh, innovative solution. On occasion that meant revisiting the very constraints that D ostensibly embraced. For example, large program fragments or indeed entire programs can be written in a well-defined memory-safe subset of D, which entails giving away a small amount of system-level access for a large gain in program debuggability.

You may be interested in D if the following values are important to you:

The role of fashion

At the initial, the most difficult stage of language development the language should solve an important problem that was inadequately solved by currently popular languages.  But at the same time the language has few chances to succeed unless it perfectly fits into the current software fashion. This "fashion factor" is probably as important as several other factors combined. With the notable exclusion of "language sponsor" factor.  The latter can make or break the language.

Like in woman dress fashion rules in language design.  And with time this trend became more and more pronounced.  A new language should simultaneously represent the current fashionable trend.  For example OO-programming was a visit card into the world of "big, successful languages" since probably early 90th (C++, Java, Python).  Before that "structured programming" and "verification" (Pascal, Modula) played similar role.

Programming environment and the role of "powerful sponsor" in language success

PL/1, Java, C#, Ada, Python are languages that had powerful sponsors. Pascal, Basic, Forth, partially Perl (O'Reilly was a sponsor for a short period of time)  are examples of the languages that had no such sponsor during the initial period of development.  C and C++ are somewhere in between.

But language itself is not enough. Any language now need a "programming environment" which consists of a set of libraries, debugger and other tools (make tool, lint, pretty-printer, etc). The set of standard" libraries and debugger are probably two most important elements. They cost  lot of time (or money) to develop and here the role of powerful sponsor is difficult to underestimate.

While this is not the necessary condition for becoming popular, it really helps: other things equal the weight of the sponsor of the language does matter. For example Java, being a weak, inconsistent language (C-- with garbage collection and OO) was pushed through the throat on the strength of marketing and huge amount of money spend on creating Java programming environment. 

The same was partially true for  C# and Python. That's why Python, despite its "non-Unix" origin is more viable scripting language now then, say, Perl (which is better integrated with Unix and has pretty innovative for scripting languages support of pointers and regular expressions), or Ruby (which has support of coroutines from day 1, not as "bolted on" feature like in Python).

Like in political campaigns, negative advertizing also matter. For example Perl suffered greatly from blackmail comparing programs in it with "white noise". And then from withdrawal of O'Reilly from the role of sponsor of the language (although it continue to milk that Perl book publishing franchise ;-)

People proved to be pretty gullible and in this sense language marketing is not that different from woman clothing marketing :-)

Language level and success

One very important classification of programming languages is based on so called the level of the language.  Essentially after there is at least one language that is successful on a given level, the success of other languages on the same level became more problematic. Higher chances for success are for languages that have even slightly higher, but still higher level then successful predecessors.

The level of the language informally can be described as the number of statements (or, more correctly, the number of  lexical units (tokens)) needed to write a solution of a particular problem in one language versus another. This way we can distinguish several levels of programming languages:

 "Nanny languages" vs "Sharp razor" languages

Some people distinguish between "nanny languages" and "sharp razor" languages. The latter do not attempt to protect user from his errors while the former usually go too far... Right compromise is extremely difficult to find.

For example, I consider the explicit availability of pointers as an important feature of the language that greatly increases its expressive power and far outweighs risks of errors in hands of unskilled practitioners.  In other words attempts to make the language "safer" often misfire.

Expressive style of the languages

Another useful typology is based in expressive style of the language:

Those categories are not pure and somewhat overlap. For example, it's possible to program in an object-oriented style in C, or even assembler. Some scripting languages like Perl have built-in regular expressions engines that are a part of the language so they have functional component despite being procedural. Some relatively low level languages (Algol-style languages) implement garbage collection. A good example is Java. There are scripting languages that compile into common language framework which was designed for high level languages. For example, Iron Python compiles into .Net.

Weak correlation between quality of design and popularity

Popularity of the programming languages is not strongly connected to their quality. Some languages that look like a collection of language designer blunders (PHP, Java ) became quite popular. Java became  a new Cobol and PHP dominates dynamic Web sites construction. The dominant technology for such Web sites is often called LAMP, which means Linux - Apache -MySQL- PHP. Being a highly simplified but badly constructed subset of Perl ( kind of new Basic for dynamic Web sites) PHP provides the most depressing experience. I was unpleasantly surprised when I had learnt that the Wikipedia engine was rewritten in PHP from Perl some time ago, but this fact quite illustrates the trend.  The number of mediocre programmer outweigh  the number of talented programmers by factor of 100 or higher.

So language design quality has little to do with the language success in the marketplace. Simpler languages have more wide appeal as success of PHP (which at the beginning was at the expense of Perl) suggests. In addition much depends whether the language has powerful sponsor like was the case with Java (Sun and IBM), PHO (Facebook). This is partially true for Python (Google) but it was after the designers of the language spend many years fighting for survival. 

Progress in programming languages has been very uneven and contain several setbacks like Java and PHP (and partially C++). Currently this progress is usually associated with scripting languages. History of programming languages raises interesting general questions about "laws" of programming language design. First let's reproduce several notable quotes:

  1. Knuth law of optimization: "Premature optimization is the root of all evil (or at least most of it) in programming." - Donald Knuth
  2. "Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." - Phil Greenspun
  3. "The key to performance is elegance, not battalions of special cases." - Jon Bentley and Doug McIlroy
  4. "Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people." - Matz, LL2
  5. Most papers in computer science describe how their author learned what someone else already knew. - Peter Landin
  6. "The only way to learn a new programming language is by writing programs in it." - Kernighan and Ritchie
  7. "If I had a nickel for every time I've written "for (i = 0; i < N; i++)" in C, I'd be a millionaire." - Mike Vanier
  8. "Language designers are not intellectuals. They're not as interested in thinking as you might hope. They just want to get a language done and start using it." - Dave Moon
  9. "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
  10. "Programs must be written for people to read, and only incidentally for machines to execute." - Abelson & Sussman, SICP, preface to the first edition

Please note that one thing is to read language manual and appreciate how good the concepts are, and another to bet your project on a new, unproved language without good debuggers, manuals and, what is very important, libraries. Debugger is very important but standard libraries are crucial: they represent a factor that makes or breaks new languages.

In this sense languages are much like cars. For many people car is the thing that they use get to work and shopping mall and they are not very interesting is engine inline or V-type and the use of fuzzy logic in the transmission. What they care is safety, reliability, mileage, insurance and the size of trunk. In this sense "Worse is better" is very true. I already mentioned the importance of the debugger. The other important criteria is quality and availability of libraries. Actually libraries are what make 80% of the usability of the language, moreover in a sense libraries are more important than the language...

A popular belief that scripting is "unsafe" or "second rate" or "prototype" solution is completely wrong. If a project had died than it does not matter what was the implementation language, so for any successful project and tough schedules scripting language (especially in dual scripting language+C combination, for example TCL+C) is an optimal blend that for a large class of tasks. Such an approach helps to separate architectural decisions from implementation details much better that any OO model does.

Moreover even for tasks that handle a fair amount of computations and data (computationally intensive tasks) such languages as Python and Perl are often (but not always !) competitive with C++, C# and, especially, Java.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. the latter includes libraries, IDE, books, level of adoption at universities, popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things. A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked. This is a story behind success of Java. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all most interesting Perl features removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that languages with simpler core, or even simplistic core has more chanced to acquire high level of popularity. The underlying fact here probably is that most programmers are at best mediocre and such programmer tend on intuitive level to avoid more complex, more rich languages like, say, PL/1 and Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because OO became a fashion). Complex non-orthogonal languages can succeed only as a result on long period of language development from a smaller core or with the banner of some fashionable new trend (OO programming in case of C++).

Programming Language Development Timeline

Here is modified from Byte the timeline of Programming Languages (for the original see BYTE.com September 1995 / 20th Anniversary /)

Forties

ca. 1946


1949

Fifties


1951


1952

1957


1958


1959

Sixties


1960


1962


1963


1964


1965


1966


1967



1969

Seventies


1970


1972


1974


1975


1976


1977


1978


1979

Eighties


1980


1981


1982


1983


1984


1985


1986


1987


1988


1989

Nineties


1990


1991


1992


1993


1994


1995


1996


1997


2000


2006

2007 

2008:

2009:

2011

2017:

Special note on Scripting languages

Scripting helps to avoid OO trap that is pushed by
  "a hoard of practically illiterate researchers
publishing crap papers in junk conferences."

Despite the fact that scripting languages are really important computer science phenomena, they are usually happily ignored in university curriculums.  Students are usually indoctrinated (or in less politically correct terms  "brainwashed")  in Java and OO programming ;-)

This site tries to give scripting languages proper emphasis and  promotes scripting languages as an alternative to mainstream reliance on "Java as a new Cobol" approach for software development. Please read my introduction to the topic that was recently converted into the article: A Slightly Skeptical View on Scripting Languages.  

The tragedy of scripting language designer is that there is no way to overestimate the level of abuse of any feature of the language.  Half of the programmers by definition is below average and it is this half that matters most in enterprise environment.  In a way the higher is the level of programmer, the less relevant for him are limitations of the language. That's why statements like "Perl is badly suitable for large project development" are plain vanilla silly. With proper discipline it is perfectly suitable and programmers can be more productive with Perl than with Java. The real question is "What is the team quality and quantity?".  

Scripting is a part of Unix cultural tradition and Unix was the initial development platform for most of mainstream scripting languages with the exception of REXX. But they are portable and now all can be used in Windows and other OSes.

List of Softpanorama pages related to scripting languages

Standard topics

Main Representatives of the family Related topics History Etc

Different scripting languages provide different level of integration with base OS API (for example, Unix or Windows). For example Iron Python compiles into .Net and provides pretty high level of integration with Windows. The same is true about Perl and Unix: almost all Unix system calls are available directly from Perl. Moreover Perl integrates most of Unix API in a very natural way, making it perfect replacement of shell for coding complex scripts. It also have very good debugger. The latter is weak point of shells like bash and ksh93

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also an extremely powerful programming paradigm.

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also extremely powerful programming paradigm.

There are also several separate pages devoted to scripting in different applications. The main emphasis is on shells and Perl. Right now I am trying to convert my old Perl lecture notes into a eBook Nikolai Bezroukov. Introduction to Perl for Unix System Administrators.

Along with pages devoted to major scripting languages this site has many pages devoted to scripting in different applications.  There are more then a dozen of "Perl/Scripting tools for a particular area" type of pages. The most well developed and up-to-date pages of this set are probably Shells and Perl. This page main purpose is to follow the changes in programming practices that can be called the "rise of  scripting," as predicted in the famous John Ousterhout article Scripting: Higher Level Programming for the 21st Century in IEEE COMPUTER (1998). In this brilliant paper he wrote:

...Scripting languages such as Perl and Tcl represent a very different style of programming than system programming languages such as C or Java. Scripting languages are designed for "gluing" applications; they use typeless approaches to achieve a higher level of programming and more rapid application development than system programming languages. Increases in computer speed and changes in the application mix are making scripting languages more and more important for applications of the future.

...Scripting languages and system programming languages are complementary, and most major computing platforms since the 1960's have provided both kinds of languages. The languages are typically used together in component frameworks, where components are created with system programming languages and glued together with scripting languages. However, several recent trends, such as faster machines, better scripting languages, the increasing importance of graphical user interfaces and component architectures, and the growth of the Internet, have greatly increased the applicability of scripting languages. These trends will continue over the next decade, with more and more new applications written entirely in scripting languages and system programming languages used primarily for creating components.

My e-book Portraits of Open Source Pioneers contains several chapters on scripting (most are in early draft stage) that expand on this topic. 

The reader must understand that the treatment of the scripting languages in press, and especially academic press is far from being fair: entrenched academic interests often promote old or commercially supported paradigms until they retire, so change of paradigm often is possible only with the change of generations. And people tend to live longer those days... Please also be aware that even respectable academic magazines like Communications of ACM and IEEE Software often promote "Cargo cult software engineering" like Capability Maturity (CMM) model.

Dr. Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

2007 2006 2005 2004 2003 2002 2001 2000 1999

[Nov 13, 2019] python - Running shell command and capturing the output

Nov 22, 2012 | stackoverflow.com

Vartec's answer doesn't read all lines, so I made a version that did:

def run_command(command):
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

Usage is the same as the accepted answer:

command = 'mysqladmin create test -uroot -pmysqladmin12'.split()
for line in run_command(command):
    print(line)
share python - Running shell command and capturing the output - Stack Overflow Share a link to this answer Copy link | improve this answer edited May 23 '17 at 11:33 Community ♦ 1 1 1 silver badge answered Oct 30 '12 at 9:24 Max Ekman Max Ekman 769 5 5 silver badges 5 5 bronze badges

[Nov 13, 2019] Execute shell commands in Python

Nov 13, 2019 | unix.stackexchange.com

Execute shell commands in Python Ask Question Asked 4 years ago Active 2 months ago Viewed 557k times 67 32


fooot ,Nov 8, 2017 at 21:39

I'm currently studying penetration testing and Python programming. I just want to know how I would go about executing a Linux command in Python. The commands I want to execute are:
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080

If I just use print in Python and run it in the terminal will it do the same as executing it as if you was typing it yourself and pressing Enter ?

binarysubstrate ,Feb 28 at 19:58

You can use os.system() , like this:
import os
os.system('ls')

Or in your case:

os.system('echo 1 > /proc/sys/net/ipv4/ip_forward')
os.system('iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080')

Better yet, you can use subprocess's call, it is safer, more powerful and likely faster:

from subprocess import call
call('echo "I like potatos"', shell=True)

Or, without invoking shell:

call(['echo', 'I like potatos'])

If you want to capture the output, one way of doing it is like this:

import subprocess
cmd = ['echo', 'I like potatos']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

o, e = proc.communicate()

print('Output: ' + o.decode('ascii'))
print('Error: '  + e.decode('ascii'))
print('code: ' + str(proc.returncode))

I highly recommend setting a timeout in communicate , and also to capture the exceptions you can get when calling it. This is a very error-prone code, so you should expect errors to happen and handle them accordingly.

https://docs.python.org/3/library/subprocess.html

jordanm ,Oct 23, 2015 at 15:43

The first command simply writes to a file. You wouldn't execute that as a shell command because python can read and write to files without the help of a shell:
with open('/proc/sys/net/ipv4/ip_forward', 'w') as f:
    f.write("1")

The iptables command is something you may want to execute externally. The best way to do this is to use the subprocess module .

import subprocess
subprocess.check_call(['iptables', '-t', 'nat', '-A',
                       'PREROUTING', '-p', 'tcp', 
                       '--destination-port', '80',
                       '-j', 'REDIRECT', '--to-port', '8080'])

Note that this method also does not use a shell, which is unnecessary overhead.

Tom Hunt ,Oct 23, 2015 at 15:41

The quickest way:
import os
os.system("your command here")

This isn't the most flexible approach; if you need any more control over your process than "run it once, to completion, and block until it exits", then you should use the subprocess module instead.

jordanm ,Apr 5, 2018 at 9:23

As a general rule, you'd better use python bindings whenever possible (better Exception catching, among other advantages.)

For the echo command, it's obviously better to use python to write in the file as suggested in @jordanm's answer.

For the iptables command, maybe python-iptables ( PyPi page , GitHub page with description and doc ) would provide what you need (I didn't check your specific command).

This would make you depend on an external lib, so you have to weight the benefits. Using subprocess works, but if you want to use the output, you'll have to parse it yourself, and deal with output changes in future iptables versions.

,

A python version of your shell. Be careful, I haven't tested it.
from subprocess import run

def bash(command):
        run(command.split())

>>> bash('find / -name null')
/dev/null
/sys/fs/selinux/null
/sys/devices/virtual/mem/null
/sys/class/mem/null
/usr/lib/kbd/consoletrans/null

[Nov 13, 2019] How fast is Perl s smartmatch operator when searching for a scalar in an array

Nov 13, 2019 | stackoverflow.com

Paul Tomblin ,Oct 19, 2010 at 13:38

I want to repeatedly search for values in an array that does not change.

So far, I have been doing it this way: I put the values in a hash (so I have an array and a hash with essentially the same contents) and I search the hash using exists .

I don't like having two different variables (the array and the hash) that both store the same thing; however, the hash is much faster for searching.

I found out that there is a ~~ (smartmatch) operator in Perl 5.10. How efficient is it when searching for a scalar in an array?

> ,

If you want to search for a single scalar in an array, you can use List::Util 's first subroutine. It stops as soon as it knows the answer. I don't expect this to be faster than a hash lookup if you already have the hash , but when you consider creating the hash and having it in memory, it might be more convenient for you to just search the array you already have.

As for the smarts of the smart-match operator, if you want to see how smart it is, test it. :)

There are at least three cases you want to examine. The worst case is that every element you want to find is at the end. The best case is that every element you want to find is at the beginning. The likely case is that the elements you want to find average out to being in the middle.

Now, before I start this benchmark, I expect that if the smart match can short circuit (and it can; its documented in perlsyn ), that the best case times will stay the same despite the array size, while the other ones get increasingly worse. If it can't short circuit and has to scan the entire array every time, there should be no difference in the times because every case involves the same amount of work.

Here's a benchmark:

#!perl
use 5.12.2;
use strict;
use warnings;

use Benchmark qw(cmpthese);

my @hits = qw(A B C);
my @base = qw(one two three four five six) x ( $ARGV[0] || 1 );

my @at_end       = ( @base, @hits );
my @at_beginning = ( @hits, @base );

my @in_middle = @base;
splice @in_middle, int( @in_middle / 2 ), 0, @hits;

my @random = @base;
foreach my $item ( @hits ) {
    my $index = int rand @random;
    splice @random, $index, 0, $item;
    }

sub count {
    my( $hits, $candidates ) = @_;

    my $count;
    foreach ( @$hits ) { when( $candidates ) { $count++ } }
    $count;
    }

cmpthese(-5, {
    hits_beginning => sub { my $count = count( \@hits, \@at_beginning ) },
    hits_end       => sub { my $count = count( \@hits, \@at_end ) },
    hits_middle    => sub { my $count = count( \@hits, \@in_middle ) },
    hits_random    => sub { my $count = count( \@hits, \@random ) },
    control        => sub { my $count = count( [], [] ) },
  }
);
div class="answercell post-layout--right

,

Here's how the various parts did. Note that this is a logarithmic plot on both axes, so the slopes of the plunging lines aren't as close as they look:

So, it looks like the smart match operator is a bit smart, but that doesn't really help you because you still might have to scan the entire array. You probably don't know ahead of time where you'll find your elements. I expect a hash will perform the same as the best case smart match, even if you have to give up some memory for it.


Okay, so the smart match being smart times two is great, but the real question is "Should I use it?". The alternative is a hash lookup, and it's been bugging me that I haven't considered that case.

As with any benchmark, I start off thinking about what the results might be before I actually test them. I expect that if I already have the hash, looking up a value is going to be lightning fast. That case isn't a problem. I'm more interested in the case where I don't have the hash yet. How quickly can I make the hash and lookup a key? I expect that to perform not so well, but is it still better than the worst case smart match?

Before you see the benchmark, though, remember that there's almost never enough information about which technique you should use just by looking at the numbers. The context of the problem selects the best technique, not the fastest, contextless micro-benchmark. Consider a couple of cases that would select different techniques:

Now, keeping those in mind, I add to my previous program:

my %old_hash = map {$_,1} @in_middle; 

cmpthese(-5, {
    ...,
    new_hash       => sub { 
        my %h = map {$_,1} @in_middle; 
        my $count = 0;
        foreach ( @hits ) { $count++ if exists $h{$_} }
        $count;
        },
    old_hash       => sub { 
        my $count = 0;
        foreach ( @hits ) { $count++ if exists $old_hash{$_} }
        $count;
        },
    control_hash   => sub { 
        my $count = 0;
        foreach ( @hits ) { $count++ }
        $count;
        },
    }
);

Here's the plot. The colors are a bit difficult to distinguish. The lowest line there is the case where you have to create the hash any time you want to search it. That's pretty poor. The highest two (green) lines are the control for the hash (no hash actually there) and the existing hash lookup. This is a log/log plot; those two cases are faster than even the smart match control (which just calls a subroutine).

There are a few other things to note. The lines for the "random" case are a bit different. That's understandable because each benchmark (so, once per array scale run) randomly places the hit elements in the candidate array. Some runs put them a bit earlier and some a bit later, but since I only make the @random array once per run of the entire program, they move around a bit. That means that the bumps in the line aren't significant. If I tried all positions and averaged, I expect that "random" line to be the same as the "middle" line.

Now, looking at these results, I'd say that a smart-match is much faster in its worst case than the hash lookup is in its worst case. That makes sense. To create a hash, I have to visit every element of the array and also make the hash, which is a lot of copying. There's no copying with the smart match.

Here's a further case I won't examine though. When does the hash become better than the smart match? That is, when does the overhead of creating the hash spread out enough over repeated searches that the hash is the better choice?

,

Fast for small numbers of potential matches, but not faster than the hash. Hashes are really the right tool for testing set membership. Since hash access is O(log n) and smartmatch on an array is still O(n) linear scan (albeit short-circuiting, unlike grep), with larger numbers of values in the allowed matches, smartmatch gets relatively worse. Benchmark code (matching against 3 values):
#!perl
use 5.12.0;
use Benchmark qw(cmpthese);

my @hits = qw(one two three);
my @candidates = qw(one two three four five six); # 50% hit rate
my %hash;
@hash{@hits} = ();

sub count_hits_hash {
  my $count = 0;
  for (@_) {
    $count++ if exists $hash{$_};
  }
  $count;
}

sub count_hits_smartmatch {
  my $count = 0;
  for (@_) {
    $count++ when @hits;
  }
  $count;
}

say count_hits_hash(@candidates);
say count_hits_smartmatch(@candidates);

cmpthese(-5, {
    hash => sub { count_hits_hash((@candidates) x 1000) },
    smartmatch => sub { count_hits_smartmatch((@candidates) x 1000) },
  }
);
Benchmark results:
             Rate smartmatch       hash
smartmatch  404/s         --       -65%
hash       1144/s       183%         --

[Nov 13, 2019] Static code analysis module in Perl - Stack Overflow

Nov 13, 2019 | stackoverflow.com

Static code analysis module in Perl Ask Question Asked 7 years, 5 months ago Active 1 year, 7 months ago Viewed 835 times 0

DavidO ,Jun 12, 2012 at 9:13

Is there any static code analysis module in Perl except B::Lint and Perl::Critic? How effective is Module::Checkstyle?

> ,

There is a post on perlmonks.org asking if PPI can be used for static analysis. PPI is the power behind Perl::Critic, according to the reviews of this module. (I have not used it yet).

Then there is perltidy .

[Nov 12, 2019] c - python (conditional-ternary) operator for assignments

Jul 30, 2014 | stackoverflow.com
This question already has an answer here:

karadoc ,May 14, 2013 at 13:01

Python has such an operator:
variable = something if condition else something_else

Alternatively, although not recommended (see @karadoc's comment):

variable = (condition and something) or something_else

[Nov 12, 2019] Static code analysis module in Perl - Stack Overflow

Nov 12, 2019 | stackoverflow.com

Static code analysis module in Perl Ask Question Asked 7 years, 5 months ago Active 1 year, 7 months ago Viewed 835 times 0

DavidO ,Jun 12, 2012 at 9:13

Is there any static code analysis module in Perl except B::Lint and Perl::Critic? How effective is Module::Checkstyle?

> ,

There is a post on perlmonks.org asking if PPI can be used for static analysis. PPI is the power behind Perl::Critic, according to the reviews of this module. (I have not used it yet).

Then there is perltidy .

[Nov 11, 2019] python - How to access environment variable values

Nov 11, 2019 | stackoverflow.com
Environment variables are accessed through os.environ
import os
print(os.environ['HOME'])

Or you can see a list of all the environment variables using:

os.environ

As sometimes you might need to see a complete list!

# using get will return `None` if a key is not present rather than raise a `KeyError`
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))

# os.getenv is equivalent, and can also give a default value instead of `None`
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))

Python default installation on Windows is C:\Python . If you want to find out while running python you can do:

import sys
print(sys.prefix)

,

import sys
print sys.argv[0]

This will print foo.py for python foo.py , dir/foo.py for python dir/foo.py , etc. It's the first argument to python . (Note that after py2exe it would be foo.exe .)

[Nov 11, 2019] How can I find the current OS in Python

Nov 11, 2019 | stackoverflow.com

> ,May 29, 2012 at 21:57

Possible Duplicate:
Python: What OS am I running on?

As the title says, how can I find the current operating system in python?

Shital Shah ,Sep 23 at 23:34

I usually use sys.platform ( docs ) to get the platform. sys.platform will distinguish between linux, other unixes, and OS X, while os.name is " posix " for all of them.

For much more detailed information, use the platform module . This has cross-platform functions that will give you information on the machine architecture, OS and OS version, version of Python, etc. Also it has os-specific functions to get things like the particular linux distribution.

xssChauhan ,Sep 9 at 7:34

If you want user readable data but still detailed, you can use platform.platform()
>>> import platform
>>> platform.platform()
'Linux-3.3.0-8.fc16.x86_64-x86_64-with-fedora-16-Verne'

platform also has some other useful methods:

>>> platform.system()
'Windows'
>>> platform.release()
'XP'
>>> platform.version()
'5.1.2600'

Here's a few different possible calls you can make to identify where you are

import platform
import sys

def linux_distribution():
  try:
    return platform.linux_distribution()
  except:
    return "N/A"

print("""Python version: %s
dist: %s
linux_distribution: %s
system: %s
machine: %s
platform: %s
uname: %s
version: %s
mac_ver: %s
""" % (
sys.version.split('\n'),
str(platform.dist()),
linux_distribution(),
platform.system(),
platform.machine(),
platform.platform(),
platform.uname(),
platform.version(),
platform.mac_ver(),))

The outputs of this script ran on a few different systems (Linux, Windows, Solaris, MacOS) and architectures (x86, x64, Itanium, power pc, sparc) is available here: https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version

Steg ,Mar 31, 2015 at 15:13

import os
print os.name

This gives you the essential information you will usually need. To distinguish between, say, different editions of Windows, you will have to use a platform-specific method.

UnkwnTech ,Sep 21, 2008 at 6:17

https://docs.python.org/library/os.html

To complement Greg's post, if you're on a posix system, which includes MacOS, Linux, Unix, etc. you can use os.uname() to get a better feel for what kind of system it is.

> ,

Something along the lines:
import os
if (os.name == "posix"):
    print os.system("uname -a")
# insert other possible OSes here
# ...
else:
    print "unknown OS"

[Nov 11, 2019] Python read file from current line

Nov 11, 2019 | stackoverflow.com

Ask Question Asked 6 years, 8 months ago Active 6 years, 8 months ago Viewed 2k times 0


dylanoo ,Feb 18, 2013 at 17:09

I have one problem regarding using python to process the trace file (it contains billion lines of data).

What I want to do is, the program will find one specific line in the file (say it is line# x), and it needs to find another symbol from this (line# x) in the file. Once it finds the line, starts from (line# x) again to search another one.

What I did now, is as following, but the problem is it always needs to reopen the file and read from the beginning to find the match ones (line # > x, and contain the symbol I want). For one big trace file, it takes too long to processing.

1.

    for line in file.readlines()
      i++ #update the line number
      if i > x:
          if (line.find()):

or:

   for i, line in enumerate(open(file)):
      if i > x:
          if ....

Anyone can give me one hint on better ideas?

Thanks

dylanoo ,Feb 18, 2013 at 20:24

If the file is otherwise stable, use fileobj.tell() to remember your position in the file, then next time use fileobj.seek(pos) to return to that same position in the file.

This only works if you do not use the fileobject as an iterator (no for line in fileobject) or next(fileobject) ) as that uses a read-ahead buffer that will obscure the exact position.

Instead, use:

for line in iter(fileobj.readline, ''):

to still use fileobj in an iteration context.

Martijn Pieters ♦ ,Feb 18, 2013 at 17:30

I suggest you use random access, and record where your line started. Something like:
index = []

fh = open(gash.txt)

for line in fh:
    if target in line:
        index.append(fh.tell() - len(line))

Then, when you want to recall the contents, use fh.seek(index[n]) .

A couple of "gotchas":

  1. Notice that the index position will not be the same as the line number. If you need the line number then maybe use a dictionary, with the line number as the key.
  2. On Windows, you will have to adjust the file position by -1. This is because the "\r" is stripped out and does not appear in the len(line) .

[Nov 11, 2019] How fast is Perl's smartmatch operator when searching for a scalar in an array - Stack Overflow

Nov 11, 2019 | stackoverflow.com

[Nov 10, 2019] Perl to Python Function translation [closed]

Feb 01, 2014 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 5 years, 8 months ago Viewed 303 times -3


Jim Garrison ,Feb 1, 2014 at 22:24

I am trying to translate a Perl function into a Python function, but I am having trouble figuring out what some of the Perl to Python function equivalents.

Perl function:

sub reverse_hex {

 my $HEXDATE = shift;
 my @bytearry=();
 my $byte_cnt = 0;
 my $max_byte_cnt = 8;
 my $byte_offset = 0;
 while($byte_cnt < $max_byte_cnt) {
   my $tmp_str = substr($HEXDATE,$byte_offset,2);
    push(@bytearry,$tmp_str);
   $byte_cnt++;
   $byte_offset+=2;
 }
   return join('',reverse(@bytearry));
}

I am not sure what "push", "shift", and "substr" are doing here that would be the same in Python.

Any help will be much appreciated.

Kenosis ,Feb 1, 2014 at 22:17

The Perl subroutine seems rather complicated for what it does, viz., taking chunks of two chars at a time (the first 16 chars) from the sent string and then reverses it. Another Perl option is:
sub reverse_hex {
    return join '', reverse unpack 'A2' x 8, $_[0];
}

First, unpack here takes two characters at a time (eight times) and produces a list. That list is reverse d and join ed to produce the final string.

Here's a Python subroutine to accomplish this:

def reverse_hex(HEXDATE):
    hexVals = [HEXDATE[i:i + 2] for i in xrange(0, 16, 2)]
    reversedHexVals = hexVals[::-1]
    return ''.join(reversedHexVals)

The list comprehension produces eight elements of two characters each. [::-1] reverses the list's elements and the result is join ed and returned.

Hope this helps!

MikeMayer67 ,Feb 2, 2014 at 2:10

I realize that you are asking about the perl to python translation, but if you have any control over the perl, I would like to point out that this function is a lot more complicated than it needs to be.

The entire thing could be replaced with:

sub reverse_hex
{
  my $hexdate = shift;
  my @bytes = $hexdate =~ /../g;  # break $hexdate into array of character pairs
  return join '', reverse(@bytes);
}

Not only is this shorter, it is much easier to get your head around. Of course, if you have no control over the perl, you are stuck with what you were dealt.

[Nov 08, 2019] Is losing BDFL a death sentence for open source projects such as Python? by Jason Baker

Jul 16, 2018 | opensource.com
What happens when a Benevolent Dictator For Life moves on from an open source project? up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 08, 2019] How to escape unicode characters in bash prompt correctly - Stack Overflow

Nov 08, 2019 | stackoverflow.com

How to escape unicode characters in bash prompt correctly Ask Question Asked 8 years, 2 months ago Active 9 months ago Viewed 6k times 7 2


Andy Ray ,Aug 18, 2011 at 19:08

I have a specific method for my bash prompt, let's say it looks like this:
CHAR="༇ "
my_function="
    prompt=\" \[\$CHAR\]\"
    echo -e \$prompt"

PS1="\$(${my_function}) \$ "

To explain the above, I'm builidng my bash prompt by executing a function stored in a string, which was a decision made as the result of this question . Let's pretend like it works fine, because it does, except when unicode characters get involved

I am trying to find the proper way to escape a unicode character, because right now it messes with the bash line length. An easy way to test if it's broken is to type a long command, execute it, press CTRL-R and type to find it, and then pressing CTRL-A CTRL-E to jump to the beginning / end of the line. If the text gets garbled then it's not working.

I have tried several things to properly escape the unicode character in the function string, but nothing seems to be working.

Special characters like this work:

COLOR_BLUE=$(tput sgr0 && tput setaf 6)

my_function="
    prompt="\\[\$COLOR_BLUE\\] \"
    echo -e \$prompt"

Which is the main reason I made the prompt a function string. That escape sequence does NOT mess with the line length, it's just the unicode character.

Andy Ray ,Aug 23, 2011 at 2:09

The \[...\] sequence says to ignore this part of the string completely, which is useful when your prompt contains a zero-length sequence, such as a control sequence which changes the text color or the title bar, say. But in this case, you are printing a character, so the length of it is not zero. Perhaps you could work around this by, say, using a no-op escape sequence to fool Bash into calculating the correct line length, but it sounds like that way lies madness.

The correct solution would be for the line length calculations in Bash to correctly grok UTF-8 (or whichever Unicode encoding it is that you are using). Uhm, have you tried without the \[...\] sequence?

Edit: The following implements the solution I propose in the comments below. The cursor position is saved, then two spaces are printed, outside of \[...\] , then the cursor position is restored, and the Unicode character is printed on top of the two spaces. This assumes a fixed font width, with double width for the Unicode character.

PS1='\['"`tput sc`"'\]  \['"`tput rc`"'༇ \] \$ '

At least in the OSX Terminal, Bash 3.2.17(1)-release, this passes cursory [sic] testing.

In the interest of transparency and legibility, I have ignored the requirement to have the prompt's functionality inside a function, and the color coding; this just changes the prompt to the character, space, dollar prompt, space. Adapt to suit your somewhat more complex needs.

tripleee ,Aug 23, 2011 at 7:01

@tripleee wins it, posting the final solution here because it's a pain to post code in comments:
CHAR="༇"
my_function="
    prompt=\" \\[`tput sc`\\]  \\[`tput rc`\\]\\[\$CHAR\\] \"
    echo -e \$prompt"

PS1="\$(${my_function}) \$ "

The trick as pointed out in @tripleee's link is the use of the commands tput sc and tput rc which save and then restore the cursor position. The code is effectively saving the cursor position, printing two spaces for width, restoring the cursor position to before the spaces, then printing the special character so that the width of the line is from the two spaces, not the character.

> ,

(Not the answer to your problem, but some pointers and general experience related to your issue.)

I see the behaviour you describe about cmd-line editing (Ctrl-R, ... Cntrl-A Ctrl-E ...) all the time, even without unicode chars.

At one work-site, I spent the time to figure out the diff between the terminals interpretation of the TERM setting VS the TERM definition used by the OS (well, stty I suppose).

NOW, when I have this problem, I escape out of my current attempt to edit the line, bring the line up again, and then immediately go to the 'vi' mode, which opens the vi editor. (press just the 'v' char, right?). All the ease of use of a full-fledged session of vi; why go with less ;-)?

Looking again at your problem description, when you say

my_function="
    prompt=\" \[\$CHAR\]\"
    echo -e \$prompt"

That is just a string definition, right? and I'm assuming your simplifying the problem definition by assuming this is the output of your my_function . It seems very likely in the steps of creating the function definition, calling the function AND using the values returned are a lot of opportunities for shell-quoting to not work the way you want it to.

If you edit your question to include the my_function definition, and its complete use (reducing your function to just what is causing the problem), it may be easier for others to help with this too. Finally, do you use set -vx regularly? It can help show how/wnen/what of variable expansions, you may find something there.

Failing all of those, look at Orielly termcap & terminfo . You may need to look at the man page for your local systems stty and related cmds AND you may do well to look for user groups specific to you Linux system (I'm assuming you use a Linux variant).

I hope this helps.

[Nov 07, 2019] Is BDFL a death sentence Opensource.com

Nov 07, 2019 | opensource.com

What happens when a Benevolent Dictator For Life moves on from an open source project? 16 Jul 2018 Jason Baker (Red Hat) Feed 131 up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 02, 2019] Copied variable changes the original

Nov 14, 2011 | stackoverflow.com

Copied variable changes the original? Ask Question Asked 7 years, 11 months ago Active 2 years, 9 months ago Viewed 61k times 46 17


André Freitas ,Nov 14, 2011 at 13:56

I have a simple problem in Python that is very very strange.
def estExt(matriz,erro):
    # (1) Determinar o vector X das soluções
    print ("Matrix after:");
    print(matriz);

    aux=matriz;
    x=solucoes(aux); # IF aux is a copy of matrix, why the matrix is changed??

    print ("Matrix before: ");
    print(matriz)

...

As you see below, the matrix matriz is changed in spite of the fact that aux is the one being changed by the function solucoes() .

Matrix before:
[[7, 8, 9, 24], [8, 9, 10, 27], [9, 10, 8, 27]]

Matrix after:
[[7, 8, 9, 24], [0.0, -0.14285714285714235, -0.2857142857142847, -0.42857142857142705], [0.0, 0.0, -3.0, -3.0000000000000018]]

André Freitas ,Nov 14, 2011 at 17:16

The line
aux=matriz;

Does not make a copy of matriz , it merely creates a new reference to matriz named aux . You probably want

aux=matriz[:]

Which will make a copy, assuming matriz is a simple data structure. If it is more complex, you should probably use copy.deepcopy

aux = copy.deepcopy(matriz)

As an aside, you don't need semi-colons after each statement, python doesn't use them as EOL markers.

André Freitas ,Nov 15, 2011 at 8:49

Use copy module
aux = copy.deepcopy(matriz) # there is copy.copy too for shallow copying

Minor one: semicolons are not needed.

aux is not a copy of matrix , it's just a different name that refers to the same object.

[Oct 25, 2019] unix - Remove a file on Linux using the inode number - Super User

Oct 25, 2019 | superuser.com

,

ome other methods include:

escaping the special chars:

[~]$rm \"la\*

use the find command and only search the current directory. The find command can search for inode numbers, and has a handy -delete switch:

[~]$ls -i
7404301 "la*

[~]$find . -maxdepth 1 -type f -inum 7404301
./"la*

[~]$find . -maxdepth 1 -type f -inum 7404301 -delete
[~]$ls -i
[~]$

,

Maybe I'm missing something, but...
rm '"la*'

Anyways, filenames don't have inodes, files do. Trying to remove a file without removing all filenames that point to it will damage your filesystem.

[Oct 22, 2019] Perl vs Python log processing performance

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 6 years, 11 months ago Active 6 years, 11 months ago Viewed 2k times 0 0

texasbruce ,Nov 11, 2012 at 2:05

I am working on a web-based log management system that will be built on the Grails framework and I am going to use one of the text processing languages like Python or Perl. I have created Python and Perl scripts that load log files and parse each line to save them to a MySQL database (the file contains about 40,000 lines, about 7MB). It took 1 min 2 secs using Perl and only 17 secs using Python .

I had supposed that Perl would be faster than Python, as Perl is the original text processing language (my suspicions also coming from different blogs where I was reading about Perl text processing performance).

Also I was not expecting a 47 second difference between Perl and Python. Why is Perl taking more time than Python to process my log file? Is it because I am using some wrong db module or my code and regular expression for Perl can be improved?

Note: I am a Java and Groovy developer and I have no experience with Perl (I am using Strawberry Perl v5.16). Also I have made this test with Java (1 min 5 secs) and Groovy (1 min 7 secs) but more than 1 min to process the log file is too much, so both languages are out and now I want to choose between Perl and Python.

PERL Code

use DBI;
use DBD::mysql;
# make connection to database
$connection = DBI->connect("dbi:mysql:logs:localhost:3306","root","") || die      "Cannot connect: $DBI::errstr";

# set the value of your SQL query
$query = "insert into logs (line_number, dated, time_stamp, thread, level, logger, user, message)
        values (?, ?, ?, ?, ?, ?, ?, ?) ";

# prepare your statement for connecting to the database
$statement = $connection->prepare($query); 

$runningTime = time;

# open text file
open (LOG,'catalina2.txt') || die "Cannot read logfile!\n";;

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}  

# close the open text file
close(LOG);

# close database connection
$connection->disconnect;

$runningTime = time - $runningTime;
printf("\n\nTotal running time: %02d:%02d:%02d\n\n", int($runningTime / 3600),   int(($runningTime % 3600) / 60), int($runningTime % 60));

# exit the script
exit;

PYTHON Code

import re
import mysql.connector
import time

file = open("D:\catalina2.txt","r")
rexp = re.compile('^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$')
conn = mysql.connector.connect(user='root',host='localhost',database='logs')
cursor = conn.cursor()

tic = time.clock()

increment  = 1
for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

conn.commit()
cursor.close()
conn.close()

toc = time.clock()
print "Total time: %s" % (toc - tic)

David-SkyMesh ,Nov 11, 2012 at 1:35

It is not a fair comparison:

You are only calling cursor.execute once in Python:

for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

But you are calling $statement->execute many times in Perl:

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}

By the way, for the Python version, calling cursor.execute once for every row will be slow. You can make it faster by using cursor.executemany :

sql = 'insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)'
args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))

cursor.executemany(sql, args)

If there are too many lines in the log file, you may need to break this up into blocks:

args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))
    if increment % 1000 == 0:
        cursor.executemany(sql, args)
        args = []
if args:
    cursor.executemany(sql, args)

(Also, don't use file.readlines() because this creates a list (which may be huge). file is an iterator which spits out one line at a time, so for text in file suffices.)

[Oct 22, 2019] Is there an advantage to using Bash over Perl or Python?

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 8 years, 5 months ago Active 8 years, 5 months ago Viewed 19k times 23 10


> ,May 2, 2011 at 18:58

Hey I've been using Linux for a while and thought it was time to finally dive into shell scripting.

The problem is I've failed to find any significant advantage of using Bash over something like Perl or Python. Are there any performance or power differences between the two? I'd figure Python/Perl would be more well suited as far as power and efficiency goes.

Sebastian ,May 2, 2011 at 15:21

Two advantages come to mind:

By the way, I usually have some python calls in my bash scripts (e.g. for plotting). Use whatever is best for the task!

Mario Peshev ,May 2, 2011 at 15:16

Perl scripts are usually (if not 100% of the times) faster than bash.

A discussion on that: Perl vs Bash

reinierpost ,May 7, 2011 at 12:16

bash isn't a language so much as a command interpreter that's been hacked to death to allow for things that make it look like a scripting language. It's great for the simplest 1-5 line one-off tasks, but things that are dead simple in Perl or Python like array manipulation are horribly ugly in bash. I also find that bash tends not to pass two critical rules of thumb:
  1. The 6-month rule, which says you should be able to easily discern the purpose and basic mechanics of a script you wrote but haven't looked at in 6 months.
  2. The 'WTF per minute' rule. Everyone has their limit, and mine is pretty small. Once I get to 3 WTFs/min, I'm looking elsewhere.

As for 'shelling out' in scripting languages like Perl and Python, I find that I almost never need to do this, fwiw (disclaimer: I code almost 100% in Python). The Python os and shutil modules have most of what I need most of the time, and there are built-in modules for handling tarfiles, gzip files, zip files, etc. There's a glob module, an fnmatch module... there's a lot of stuff there. If you come across something you need to parallelize, then indent your code a level, put it in a 'run()' method, put that in a class that extends either threading.Thread or multiprocessing.Process, instantiate as many of those as you want, calling 'start()' on each one. Less than 5 minutes to get parallel execution generally.

Best of luck. Hope this helps.

daotoad ,May 2, 2011 at 17:40

For big projects use a language like Perl.

There are a few things you can only do in bash (for example, alter the calling environment (when a script is sourced rather than run). Also, shell scripting is commonplace. It is worthwhile to learn the basics and learn your way around the available docs.

Plus there are times when knowing a shell well can save your bacon (on a fork-bombed system where you can't start any new processes, or if /usr/bin and or /usr/local/bin fail to mount).

Sebastian ,May 3, 2011 at 8:47

The advantage is that it's right there. Unless you use Python (or Perl) as your shell, writing a script to do a simple loop is a bunch of extra work.

For short, simple scripts that call other programs, I'll use Bash. If I want to keep the output, odds are good that I'll trade up to Python.

For example:

for file in *; do process $file ; done

where process is a program I want to run on each file, or...

while true; do program_with_a_tendency_to_fail ; done

Doing either of those in Python or Perl is overkill.

For actually writing a program that I expect to maintain and use over time, Bash is rarely the right tool for the job. Particularly since most modern Unices come with both Perl and Python.

tchrist ,May 4, 2011 at 11:01

The most important advantage of POSIX shell scripts over Python or Perl scripts is that a POSIX shell is available on virtually every Unix machine. (There are also a few tasks shell scripts happen to be slightly more convenient for, but that's not a major issue.) If the portability is not an issue for you, I don't see much need to learn shell scripting.

tchrist ,May 3, 2011 at 23:50

If you want to execute programs installed on the machine, nothing beats bash. You can always make a system call from Perl or Python, but I find it to be a hassle to read return values, etc.

And since you know it will work pretty much anywhere throughout all of of time...

Alexandr Ciornii ,May 3, 2011 at 8:26

The advantage of shell scripting is that it's globally present on *ix boxes, and has a relatively stable core set of features you can rely on to run everywhere. With Perl and Python you have to worry about whether they're available and if so what version, as there have been significant syntactical incompatibilities throughout their lifespans. (Especially if you include Python 3 and Perl 6.)

The disadvantage of shell scripting is everything else. Shell scripting languages are typically lacking in expressiveness, functionality and performance. And hacking command lines together from strings in a language without strong string processing features and libraries, to ensure the escaping is correct, invites security problems. Unless there's a compelling compatibility reason you need to go with shell, I would personally plump for a scripting language every time.

[Oct 22, 2019] Python Code Glitch May Have Caused Errors In Over 100 Published Studies

Oct 22, 2019 | science.slashdot.org

(vice.com) 121

An anonymous reader quotes Motherboard: The glitch caused results of a common chemistry computation to vary depending on the operating system used, causing discrepancies among Mac, Windows, and Linux systems. The researchers published the revelation and a debugged version of the script, which amounts to roughly 1,000 lines of code, on Tuesday in the journal Organic Letters .

"This simple glitch in the original script calls into question the conclusions of a significant number of papers on a wide range of topics in a way that cannot be easily resolved from published information because the operating system is rarely mentioned," the new paper reads. "Authors who used these scripts should certainly double-check their results and any relevant conclusions using the modified scripts in the [supplementary information]."

Yuheng Luo, a graduate student at the University of Hawaii at Manoa, discovered the glitch this summer when he was verifying the results of research conducted by chemistry professor Philip Williams on cyanobacteria... Under supervision of University of Hawaii at Manoa assistant chemistry professor Rui Sun, Luo used a script written in Python that was published as part of a 2014 paper by Patrick Willoughby, Matthew Jansma, and Thomas Hoye in the journal Nature Protocols . The code computes chemical shift values for NMR, or nuclear magnetic resonance spectroscopy, a common technique used by chemists to determine the molecular make-up of a sample. Luo's results did not match up with the NMR values that Williams' group had previously calculated, and according to Sun, when his students ran the code on their computers, they realized that different operating systems were producing different results.

Sun then adjusted the code to fix the glitch, which had to do with how different operating systems sort files.
The researcher who wrote the flawed script told Motherboard that the new study was "a beautiful example of science working to advance the work we reported in 2014. They did a tremendous service to the community in figuring this out."

Sun described the original authors as "very gracious," saying they encouraged the publication of the findings.

[Oct 22, 2019] Difference in regex behavior between Perl and Python?

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 10 years, 6 months ago Active 10 years, 6 months ago Viewed 2k times 3 1


Gumbo ,Apr 16, 2009 at 18:42

I have a couple email addresses, 'support@company.com' and '1234567@tickets.company.com' .

In perl, I could take the To: line of a raw email and find either of the above addresses with

/\w+@(tickets\.)?company\.com/i

In python, I simply wrote the above regex as '\w+@(tickets\.)?company\.com' expecting the same result. However, support@company.com isn't found at all and a findall on the second returns a list containing only 'tickets.' . So clearly the '(tickets\.)?' is the problem area, but what exactly is the difference in regular expression rules between Perl and Python that I'm missing?

Axeman ,Apr 16, 2009 at 21:10

The documentation for re.findall :
findall(pattern, string, flags=0)
    Return a list of all non-overlapping matches in the string.

    If one or more groups are present in the pattern, return a
    list of groups; this will be a list of tuples if the pattern
    has more than one group.

    Empty matches are included in the result.

Since (tickets\.) is a group, findall returns that instead of the whole match. If you want the whole match, put a group around the whole pattern and/or use non-grouping matches, i.e.

r'(\w+@(tickets\.)?company\.com)'
r'\w+@(?:tickets\.)?company\.com'

Note that you'll have to pick out the first element of each tuple returned by findall in the first case.

chaos ,Apr 16, 2009 at 18:45

I think the problem is in your expectations of extracted values. Try using this in your current Python code:
'(\w+@(?:tickets\.)?company\.com)'

Jason Coon ,Apr 16, 2009 at 18:46

Two problems jump out at me:
  1. You need to use a raw string to avoid having to escape " \ "
  2. You need to escape " . "

So try:

r'\w+@(tickets\.)?company\.com'

EDIT

Sample output:

>>> import re
>>> exp = re.compile(r'\w+@(tickets\.)?company\.com')
>>> bool(exp.match("s@company.com"))
True
>>> bool(exp.match("1234567@tickets.company.com"))
True

,

There isn't a difference in the regexes, but there is a difference in what you are looking for. Your regex is capturing only "tickets." if it exists in both regexes. You probably want something like this
#!/usr/bin/python

import re

regex = re.compile("(\w+@(?:tickets\.)?company\.com)");

a = [
    "foo@company.com", 
    "foo@tickets.company.com", 
    "foo@ticketsacompany.com",
    "foo@compant.org"
];

for string in a:
    print regex.findall(string)

[Oct 22, 2019] Python for a Perl programmer

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 9 years, 8 months ago Active 11 months ago Viewed 22k times 53 47


Hamish Grubijan ,Feb 17, 2010 at 17:56

I am an experienced Perl developer with some degree of experience and/or familiarity with other languages (working experience with C/C++, school experience with Java and Scheme, and passing familiarity with many others).

I might need to get some web work done in Python (most immediately, related to Google App Engine). As such, I'd like to ask SO overmind for good references on how to best learn Python for someone who's coming from Perl background (e.g. the emphasis would be on differences between the two and how to translate perl idiomatics into Python idiomatics, as opposed to generic Python references). Something also centered on Web development is even better. I'll take anything - articles, tutorials, books, sample apps?

Thanks!

FMc ,Dec 19, 2014 at 17:50

I've recently had to make a similar transition for work reasons, and it's been pretty painful. For better or worse, Python has a very different philosophy and way of working than Perl, and getting used to that can be frustrating. The things I've found most useful have been

Personally, I found Dive Into Python annoying and patronising, but it's freely available online, so you can form your own judgment on that.

Philip Durbin ,Feb 18, 2010 at 18:12

If you happen to be a fan of The Perl Cookbook , you might be interested in checking out PLEAC, the Programming Language Examples Alike Cookbook , specifically the section that shows the Perl Cookbook code translated into Python .

larley ,Feb 18, 2010 at 6:16

Being a hardcore Perl programmer, all I can say is DO NOT BUY O'Reilly's "Learning Python". It is nowhere NEAR as good as "Learning Perl", and there's no equivalent I know of to Larry Wall's "Programming Perl", which is simply unbeatable.

I've had the most success taking past Perl programs and translating them into Python, trying to make use of as many new techniques as possible.

Mike Graham ,Feb 17, 2010 at 18:02

Check out the official tutorial , which is actually pretty good. If you are interested in web development you should be ready at that point to jump right in to the documentation of the web framework you will be working with; Python has many to choose from, with zope, cherrypy, pylons, and werkzeug all having good reputations.

I would not try to search for things specifically meant to help you transition from Perl, which are not to be of as high of quality as references that can be useful for more people.

ghostdog74 ,Feb 18, 2010 at 1:17

This is the site you should really go to. There's a section called Getting Started which you should take a look. There are also recommendations on books. On top of that, you might also be interested in this on "idioms"

sateesh ,Feb 17, 2010 at 18:08

If what you are looking at is succinct, concise reference to python then the book Python Essential Reference might be helpful.

Robert P ,May 31, 2013 at 22:39

I wouldn't try to compare Perl and Python too much in order to learn Python, especially since you have working knowledge of other languages. If you are unfamiliar with OOP/Functional programming aspects and just looking to work procedurally like in Perl, start learning the Python language constructs / syntax and then do a couple examples. if you are making a switch to OO or functional style paradigms, I would read up on OO fundamentals first, then start on Python syntax and examples...so you have a sort of mental blueprint of how things can be constructed before you start working with the actual materials. this is just my humble opinion however..

[Oct 21, 2019] Differences between Perl and PHP [closed]

Notable quotes:
"... Perl has native regular expression support, ..."
"... Perl has quite a few more operators , including matching ..."
"... In PHP, new is an operator. In Perl, it's the conventional name of an object creation subroutine defined in packages, nothing special as far as the language is concerned. ..."
"... Perl logical operators return their arguments, while they return booleans in PHP. ..."
"... Perl gives access to the symbol table ..."
"... Note that "references" has a different meaning in PHP and Perl. In PHP, references are symbol table aliases. In Perl, references are smart pointers. ..."
"... Perl has different types for integer-indexed collections (arrays) and string indexed collections (hashes). In PHP, they're the same type: an associative array/ordered map ..."
"... Perl arrays aren't sparse ..."
"... Perl supports hash and array slices natively, ..."
Nov 23, 2013 | stackoverflow.com

jholster ,Nov 23, 2013 at 21:20

I'm planning to learn Perl 5 and as I have only used PHP until now, I wanted to know a bit about how the languages differ from each other.

As PHP started out as a set of "Perl hacks" it has obviously cloned some of Perls features.

hobbs ,Jan 17, 2013 at 8:36

Perl and PHP are more different than alike. Let's consider Perl 5, since Perl 6 is still under development. Some differences, grouped roughly by subject:

PHP was inspired by Perl the same way Phantom of the Paradise was inspired by Phantom of the Opera , or Strange Brew was inspired by Hamlet . It's best to put the behavior specifics of PHP out of your mind when learning Perl, else you'll get tripped up.

My brain hurts now, so I'm going to stop.

Your Common Sense ,Mar 29, 2010 at 2:19

When PHP came to the scene, everyone were impressed with main differences from Perl:
  1. Input variables already in the global scope, no boring parsing.
  2. HTML embedding. Just <?php ... ?> anywhere. No boring templates.
  3. On-screen error messages. No boring error log peeks.
  4. Easy to learn. No boring book reading.

As the time passed, everyone learned that they were not a benefit, hehe...

Quentin ,Jan 15, 2016 at 3:27

I've noticed that most PHP vs. Perl pages seem to be of the

PHP is better than Perl because <insert lame reason here>

ilk, and rarely make reasonable comparisons.

Syntax-wise, you will find PHP is often easier to understand than Perl, particularly when you have little experience. For example, trimming a string of leading and trailing whitespace in PHP is simply

$string = trim($string);

In Perl it is the somewhat more cryptic

$string =~ s/^\s+//;
$string =~ s/\s+$//;

(I believe this is slightly more efficient than a single line capture and replace, and also a little more understandable.) However, even though PHP is often more English-like, it sometimes still shows its roots as a wrapper for low level C, for example, strpbrk and strspn are probably rarely used, because most PHP dabblers write their own equivalent functions for anything too esoteric, rather than spending time exploring the manual. I also wonder about programmers for whom English is a second language, as everybody is on equal footing with things such as Perl, having to learn it from scratch.

I have already mentioned the manual. PHP has a fine online manual, and unfortunately it needs it. I still refer to it from time to time for things that should be simple, such as order of parameters or function naming convention. With Perl, you will probably find you are referring to the manual a lot as you get started and then one day you will have an a-ha moment and never need it again. Well, at least not until you're more advanced and realize that not only is there more than one way, there is probably a better way, somebody else has probably already done it that better way, and perhaps you should just visit CPAN.

Perl does have a lot more options and ways to express things. This is not necessarily a good thing, although it allows code to be more readable if used wisely and at least one of the ways you are likely to be familiar with. There are certain styles and idioms that you will find yourself falling into, and I can heartily recommend reading Perl Best Practices (sooner rather than later), along with Perl Cookbook, Second Edition to get up to speed on solving common problems.

I believe the reason Perl is used less often in shared hosting environments is that historically the perceived slowness of CGI and hosts' unwillingness to install mod_perl due to security and configuration issues has made PHP a more attractive option. The cycle then continued, more people learned to use PHP because more hosts offered it, and more hosts offered it because that's what people wanted to use. The speed differences and security issues are rendered moot by FastCGI these days, and in most cases PHP is run out of FastCGI as well, rather than leaving it in the core of the web server.

Whether or not this is the case or there are other reasons, PHP became popular and a myriad of applications have been written in it. For the majority of people who just want an entry-level website with a simple blog or photo gallery, PHP is all they need so that's what the hosts promote. There should be nothing stopping you from using Perl (or anything else you choose) if you want.

At an enterprise level, I doubt you would find too much PHP in production (and please, no-one point at Facebook as a counter-example, I said enterprise level).

Leon Timmermans ,Mar 28, 2010 at 22:15

Perl is used plenty for websites, no less than Python and Ruby for example. That said, PHP is used way more often than any of those. I think the most important factors in that are PHP's ease of deployment and the ease to start with it.

The differences in syntax are too many to sum up here, but generally it is true that it has more ways to express yourself (this is know as TIMTWOTDI, There Is More Than One Way To Do It).

Brad Gilbert ,Mar 29, 2010 at 4:04

My favorite thing about Perl is the way it handles arrays/lists. Here's an example of how you would make and use a Perl function (or "subroutine"), which makes use of this for arguments:
sub multiply
{
    my ($arg1, $arg2) = @_; # @_ is the array of arguments
    return $arg1 * $arg2;
}

In PHP you could do a similar thing with list() , but it's not quite the same; in Perl lists and arrays are actually treated the same (usually). You can also do things like:

$week_day_name = ("Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday")[$week_day_index];

And another difference that you MUST know about, is numerical/string comparison operators. In Perl, if you use < , > , == , != , <=> , and so on, Perl converts both operands to numbers. If you want to convert as strings instead, you have to use lt , gt , eq , ne , cmp (the respective equivalents of the operators listed previously). Examples where this will really get you:

if ("a" == "b") { ... } # This is true.
if ("a" == 0) { ... } # This is also true, for the same reason.

Sorin Postelnicu, Aug 5, 2015 at 15:44

I do not need add anything to outis's fantastic answer, i want only show the answer for you question:

Why is Perl not used for dynamic websites very often anymore? What made PHP gain more popularity than it?

Please check first some "Job Trends" sites - and you can make the judgement alone.

as you can see, perl is still a leader - but preferable for real applications not for toys. :)

[Oct 21, 2019] Python variable reference assignment

Oct 21, 2019 | stackoverflow.com

Python variable reference assignment Ask Question Asked 7 years, 3 months ago Active 3 years, 11 months ago Viewed 54k times 42 20


viji ,Jun 27, 2012 at 8:43

In the code
y = 7
x = y    
x = 8

Now, y will be 7 and x will be 8. But actually I wanna change y. Can I assign the reference of y and do that ?

For example, in C++ the same thing can be achieved as,

int y = 8;
int &x = y;
x = 9;

Now both y & x will be 9

Daniel Roseman ,Jun 27, 2012 at 9:20

No, you cannot. As other answer point out, you can (ab?)use aliasing of mutable objects to achieve a similar effect. However, that's not the same thing as C++ references, and I want to explain what actually happens to avoid any misconceptions.

You see, in C++ (and other languages), a variable (and object fields, and entries in collections, etc.) is a storage location and you write a value (for instance, an integer, an object, or a pointer) to that location. In this model, references are an alias for a storage location (of any kind) - when you assign to a non-reference variable, you copy a value (even if it's just a pointer, it's still a value) to the storage location; when you assign to a reference, you copy to a storage location somewhere else. Note that you cannot change a reference itself - once it is bound (and it has to as soon as you create one) all assignments to it alter not the reference but whatever is referred to.

In Python (and other languages), a variable (and object fields, and entries in collections, etc.) is a just a name. Values are somewhere else (e.g. sprinkled all over the heap), and a variable refers (not in the sense of C++ references, more like a pointer minus the pointer arithmetic) to a value. Multiple names can refer to the same value (which is generally a good thing). Python (and other languages) calls whatever is needed to refer to a value a reference, despite being pretty unrelated to things like C++ references and pass-by-reference. Assigning to a variable (or object field, or ...) simply makes it refer to another value. The whole model of storage locations does not apply to Python, the programmer never handles storage locations for values. All he stores and shuffles around are Python references, and those are not values in Python, so they cannot be target of other Python references.

All of this is independent of mutability of the value - it's the same for ints and lists, for instance. You cannot take a variable that refers to either, and overwrite the object it points to. You can only tell the object to modify parts of itself - say, change some reference it contains.

Is this a more restrictive model? Perhaps, but it's powerful enough most of the time. And when it isn't you can work around it, either with a custom class like the one given below, or (equivalent, but less obvious) a single-element collection.

class Reference:
    def __init__(self, val):
        self._value = val # just refers to val, no copy

    def get(self):
        return self._value

    def set(self, val):
        self._value = val

That still won't allow you to alias a "regular" variable or object field, but you can have multiple variables referring to the same Reference object (ditto for the mutable-singleton-collection alternative). You just have to be careful to always use .get() / .set() (or [0] ).

rbaleksandar ,Sep 24, 2013 at 19:34

No, Python doesn't have this feature.

If you had a list (or any other mutable object) you could do what you want by mutating the object that both x and y are bound to:

>>> x = [7]
>>> y = x
>>> y[0] = 8
>>> print x
[8]

See it working online: ideone

tauseef_CuriousGuy ,Sep 7, 2017 at 8:10

You should use a mutable object for this.

In python x & y are just references to objects so y = 7 means y points to the object 7 . x=y means x too points to 7 , but as 7 is immutable so changing the value of x simply changes the object 7 and y still remains pointing to 7 .

>>> y = [7]
>>> x = y
>>> x[0] = 8 # here you're changing [7] not x or y, x & y are just references to [7]
>>> y
[8]

,

Alternatively, you could use a self crafted container.
class Value(object):
    def __init__(self, value): self.value = value

y = Value(7)
x = y
x.value = 8
print y.value

[Oct 20, 2019] Using command line arguments to R CMD BATCH

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 6 years, 9 months ago Active 1 year, 7 months ago Viewed 112k times 93 54


Bryce Thomas ,Jan 5, 2013 at 0:26

I have been using R CMD BATCH my_script.R from a terminal to execute an R script. I am now at the point where I would like to pass an argument to the command, but am having some issues getting it working. If I do R CMD BATCH my_script.R blabla then blabla becomes the output file, rather than being interpreted as an argument available to the R script being executed.

I have tried Rscript my_script.R blabla which seems to pass on blabla correctly as an argument, but then I don't get the my_script.Rout output file that I get with R CMD BATCH (I want the .Rout file). While I could redirect the output of a call to Rscript to a file name of my choosing, I would not be getting the R input commands included in the file in the way R CMD BATCH does in the .Rout file.

So, ideally, I'm after a way to pass arguments to an R script being executed via the R CMD BATCH method, though would be happy with an approach using Rscript if there is a way to make it produce a comparable .Rout file.

Bryce Thomas ,Feb 11, 2015 at 20:59

My impression is that R CMD BATCH is a bit of a relict. In any case, the more recent Rscript executable (available on all platforms), together with commandArgs() makes processing command line arguments pretty easy.

As an example, here is a little script -- call it "myScript.R" :

## myScript.R
args <- commandArgs(trailingOnly = TRUE)
rnorm(n=as.numeric(args[1]), mean=as.numeric(args[2]))

And here is what invoking it from the command line looks like

> Rscript myScript.R 5 100
[1]  98.46435 100.04626  99.44937  98.52910 100.78853

Edit:

Not that I'd recommend it, but ... using a combination of source() and sink() , you could get Rscript to produce an .Rout file like that produced by R CMD BATCH . One way would be to create a little R script -- call it RscriptEcho.R -- which you call directly with Rscript. It might look like this:

## RscriptEcho.R
args <- commandArgs(TRUE)
srcFile <- args[1]
outFile <- paste0(make.names(date()), ".Rout")
args <- args[-1]

sink(outFile, split = TRUE)
source(srcFile, echo = TRUE)

To execute your actual script, you would then do:

Rscript RscriptEcho.R myScript.R 5 100
[1]  98.46435 100.04626  99.44937  98.52910 100.78853

which will execute myScript.R with the supplied arguments and sink interleaved input, output, and messages to a uniquely named .Rout .

Edit2:
You can run Rscript verbosely and place the verbose output in a file.

Rscript --verbose myScript.R 5 100 > myScript.Rout

d2a2d ,Apr 9 at 22:33

After trying the options described here, I found this post from Forester in r-bloggers . I think it is a clean option to consider.

I put his code here:

From command line

$ R CMD BATCH --no-save --no-restore '--args a=1 b=c(2,5,6)' test.R test.out &

Test.R

##First read in the arguments listed at the command line
args=(commandArgs(TRUE))

##args is now a list of character vectors
## First check to see if arguments are passed.
## Then cycle through each element of the list and evaluate the expressions.
if(length(args)==0){
    print("No arguments supplied.")
    ##supply default values
    a = 1
    b = c(1,1,1)
}else{
    for(i in 1:length(args)){
      eval(parse(text=args[[i]]))
    }
}

print(a*2)
print(b*3)

In test.out

> print(a*2)
[1] 2
> print(b*3)
[1]  6 15 18

Thanks to Forester !

user1563570 ,Apr 5, 2013 at 13:15

In your R script, called test.R :
args <- commandArgs(trailingOnly = F)
myargument <- args[length(args)]
myargument <- sub("-","",myargument)
print(myargument)
q(save="no")

From the command line run:

R CMD BATCH -4 test.R

Your output file, test.Rout, will show that the argument 4 has been successfully passed to R:

cat test.Rout

> args <- commandArgs(trailingOnly = F)
> myargument <- args[length(args)]
> myargument <- sub("-","",myargument)
> print(myargument)
[1] "4"
> q(save="no")
> proc.time()
user  system elapsed 
0.222   0.022   0.236

Bryce Thomas ,Oct 23, 2014 at 20:57

You need to put arguments before my_script.R and use - on the arguments, e.g.
R CMD BATCH -blabla my_script.R

commandArgs() will receive -blabla as a character string in this case. See the help for details:

$ R CMD BATCH --help
Usage: R CMD BATCH [options] infile [outfile]

Run R non-interactively with input from infile and place output (stdout
and stderr) to another file.  If not given, the name of the output file
is the one of the input file, with a possible '.R' extension stripped,
and '.Rout' appended.

Options:
  -h, --help        print short help message and exit
  -v, --version     print version info and exit
  --no-timing           do not report the timings
  --            end processing of options

Further arguments starting with a '-' are considered as options as long
as '--' was not encountered, and are passed on to the R process, which
by default is started with '--restore --save --no-readline'.
See also help('BATCH') inside R.

ClementWalter ,Mar 16, 2016 at 9:52

I add an answer because I think a one line solution is always good! Atop of your myRscript.R file, add the following line:
eval(parse(text=paste(commandArgs(trailingOnly = TRUE), collapse=";")))

Then submit your script with something like:

R CMD BATCH [options] '--args arguments you want to supply' myRscript.R &

For example:

R CMD BATCH --vanilla '--args N=1 l=list(a=2, b="test") name="aname"' myscript.R &

Then:

> ls()
[1] "N"    "l"    "name"

Dagremu ,Dec 15, 2016 at 0:50

Here's another way to process command line args, using R CMD BATCH . My approach, which builds on an earlier answer here , lets you specify arguments at the command line and, in your R script, give some or all of them default values.

Here's an R file, which I name test.R :

defaults <- list(a=1, b=c(1,1,1)) ## default values of any arguments we might pass

## parse each command arg, loading it into global environment
for (arg in commandArgs(TRUE))
  eval(parse(text=arg))

## if any variable named in defaults doesn't exist, then create it
## with value from defaults
for (nm in names(defaults))
  assign(nm, mget(nm, ifnotfound=list(defaults[[nm]]))[[1]])

print(a)
print(b)

At the command line, if I type

R CMD BATCH --no-save --no-restore '--args a=2 b=c(2,5,6)' test.R

then within R we'll have a = 2 and b = c(2,5,6) . But I could, say, omit b , and add in another argument c :

R CMD BATCH --no-save --no-restore '--args a=2 c="hello"' test.R

Then in R we'll have a = 2 , b = c(1,1,1) (the default), and c = "hello" .

Finally, for convenience we can wrap the R code in a function, as long as we're careful about the environment:

## defaults should be either NULL or a named list
parseCommandArgs <- function(defaults=NULL, envir=globalenv()) {
  for (arg in commandArgs(TRUE))
    eval(parse(text=arg), envir=envir)

  for (nm in names(defaults))
    assign(nm, mget(nm, ifnotfound=list(defaults[[nm]]), envir=envir)[[1]], pos=envir)
}

## example usage:
parseCommandArgs(list(a=1, b=c(1,1,1)))

[Oct 20, 2019] How can I use multiple library paths?

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 2 years, 10 months ago Active 2 years, 10 months ago Viewed 747 times 2 0

user797963 ,Nov 28, 2016 at 16:11

I'm trying to set up an easy to use R development environment for multiple users. R is installed along with a set of other dev tools on an NFS mount.

I want to create a core set of R packages that also live on NFS so n users don't need to install their own copies of the same packages n times. Then, I was hoping users can install one off packages to a local R library. Has anyone worked with an R setup like this before? From the doc, it looks doable by adding both the core package and personal package file paths to .libPaths() .

,

You want to use the .Renviron file (see ?Startup ).

There are three places to put the file:

In this file you can specify R_LIBS and the R_LIBS_SITE environment variables.

For your particular problem, you probably want to add the NFS drive location to R_LIBS_SITE in the R_HOME/etc/Renviron.site file.


## To get R_HOME
Sys.getenv("R_HOME")

[Oct 20, 2019] How to update a package in R?

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 10 months ago Viewed 23k times 23 4


Joshua Ulrich ,Jan 31, 2014 at 8:15

I would like to upgrade one R package to the newer version which is already available. I tried
update.packages(c("R2jags"))

but it does nothing! No output on console, no error, nothing. I used the same syntax as for install.packages but perhaps I'm doing something wrong. I have been looking at ?update.packages but I was not been able to figure out how it works, where to specify the package(s) etc. There is no example. I also tried to update the package using install.packages to "install" it again but that says "Warning: package 'R2jags' is in use and will not be installed" .

TMS ,Jan 30, 2014 at 16:47

You can't do this I'm afraid, well, not with update.packages() . You need to call install.packages("R2jags") instead.

You can't install R2jags in the current session because you have already loaded the current version into the session. If you need to, save any objects you can't easily recreate, and quit out of R. Then start a new R session, immediately run install.packages("R2jags") , then once finished, load the package and reload in any previously saved objects. You could try to unload the package with:

detach(package:R2jags, unload = TRUE)

but it is quite complex to do this cleanly unless the package cleans up after itself.

update.packages() exists to update all outdated packages in a stated library location. That library location is given by the first argument (if not supplied it works on all known library locations for the current R session). Hence you were asking it the update the packages in library location R2jags which is most unlikely to exist on your R installation.

amzu ,Jan 30, 2014 at 16:36

Additionally, you can install RStudio and update all packages by going to the Tools menu and selecting Check for Package Updates .

DJ6968 ,Dec 13, 2018 at 11:42

# The following two commands remove any previously installed H2O packages for R.
if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }

# Next, we download packages that H2O depends on.
pkgs <- c("RCurl","jsonlite")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }
}

# Now we download, install and initialize the H2O package for R.
install.packages("h2o", type="source", repos="http://h2o-release.s3.amazonaws.com/h2o/rel-xia/2/R")

# Finally, let's load H2O and start up an H2O cluster
library(h2o)`enter code here`
h2o.init()

[Oct 16, 2019] R libraries installation - Stack Overflow

Oct 31, 2016 | stackoverflow.com

R libraries installation Ask Question Asked 2 years, 11 months ago Active 2 years, 10 months ago Viewed 783 times 0

horseshoe ,Oct 31, 2016 at 14:47

I am using a computer where I have read only rights for the R library folder. When I am installing new packages I therefore use
libpath <- "c:/R/mylibraries"
.libPaths( c( .libPaths(), libpath) )
install.packages("htmltools",   lib=libpath)

always when I am installing a new packages with dependecies (like e.g. htmltools depends on lme4), I get erros like:

Error in .requirePackage(package) : 
  unable to find required package 'lme4'

although lme4 is installed and I used it before.... also other errors/warnings like:

Warning in install.packages :
  cannot remove prior installation of package 'Rcpp'

or:

Warning in install.packages :
  unable to move temporary installation 'c:\...\file17b033a54a21\jsonlite' to 'c:\...\jsonlite'

occur. If I install them twice they usually work but sometimes dependencies to packages that worked before are lost and I have to reinstall them again. Is there a way to circumvent this?

> ,

Put this in a file named .REnviron in your Documents folder and restart R:
R_LIBS=c:/R/mylibraries

From then on, you should be able to install packages into that location automatically, without having to fiddle around with .libPaths .

[Oct 16, 2019] R libraries installation - Stack Overflow

Oct 16, 2019 | stackoverflow.com

R libraries installation Ask Question Asked 2 years, 11 months ago Active 2 years, 10 months ago Viewed 783 times 0

horseshoe ,Oct 31, 2016 at 14:47

I am using a computer where I have read only rights for the R library folder. When I am installing new packages I therefore use
libpath <- "c:/R/mylibraries"
.libPaths( c( .libPaths(), libpath) )
install.packages("htmltools",   lib=libpath)

always when I am installing a new packages with dependecies (like e.g. htmltools depends on lme4), I get erros like:

Error in .requirePackage(package) : 
  unable to find required package 'lme4'

although lme4 is installed and I used it before.... also other errors/warnings like:

Warning in install.packages :
  cannot remove prior installation of package 'Rcpp'

or:

Warning in install.packages :
  unable to move temporary installation 'c:\...\file17b033a54a21\jsonlite' to 'c:\...\jsonlite'

occur. If I install them twice they usually work but sometimes dependencies to packages that worked before are lost and I have to reinstall them again. Is there a way to circumvent this?

> ,

Put this in a file named .REnviron in your Documents folder and restart R:
R_LIBS=c:/R/mylibraries

From then on, you should be able to install packages into that location automatically, without having to fiddle around with .libPaths .

[Oct 16, 2019] R setting library path via R_LIBS

Oct 16, 2019 | stackoverflow.com

Ask Question Asked 3 years, 4 months ago Active 3 years, 1 month ago Viewed 5k times 2

Xiongbing Jin ,Jun 12, 2016 at 12:42

I have read the R FAQS and other posts but I am a bit confused and would be grateful to know whether I did everything correctly.

In Windows, in order to modify the default library folder I created a file Renviron.site and put inside E:/Programs/R-3.3.0/etc . The file has only one line saying

R_LIBS=E:/Rlibrary

When I open R and run .libPaths() I see E:/Rlibrary as [1] and the default R library E:/Programs/R-3.3.0/library as [2].

This should mean that from now on all packages I will install will go in E:/Rlibrary but at the same time I will be able to load and use both packages in this folder and those in the default location. Am I correct?

,

When you load a package via library , it will go through each directory in .libPaths() in turn to find the required package. If the package hasn't been found, you will get an error. This means you can have multiple versions of a package (in different directories), but the package that will be used is determined by the order of .libPaths() .

Regarding how .libPaths() is constructed, from ?.R_LIBS

The library search path is initialized at startup from the environment variable 'R_LIBS' (which should be a colon-separated list of directories at which R library trees are rooted) followed by those in environment variable 'R_LIBS_USER'. Only directories which exist at the time will be included.

[Oct 16, 2019] The .Rprofile file

Oct 16, 2019 | csgillespie.github.io

3.3 R startup

Every time R starts, a number of files are read, in a particular order. The contents of these files determine how R performs for the duration of the session. Note that these files should only be changed with caution, as they may make your R version behave differently to other R installations. This could reduce the reproducibility of your code.

Files in three folders are important in this process:

It is important to know the location of the .Rprofile and .Renviron set-up files that are being used out of these three options. R only uses one .Rprofile and one .Renviron in any session: if you have a .Rprofile file in your current project, R will ignore .Rprofile in R_HOME and HOME . Likewise, .Rprofile in HOME overrides .Rprofile in R_HOME . The same applies to .Renviron : you should remember that adding project specific environment variables with .Renviron will de-activate other .Renviron files.

To create a project-specific start-up script, simply create a .Rprofile file in the project's root directory and start adding R code, e.g. via file.edit(".Rprofile") . Remember that this will make .Rprofile in the home directory be ignored. The following commands will open your .Rprofile from within an R editor:

file.edit(file.path("~", ".Rprofile")) # edit .Rprofile in HOME
file.edit(".Rprofile") # edit project specific .Rprofile

Note that editing the .Renviron file in the same locations will have the same effect. The following code will create a user specific .Renviron file (where API keys and other cross-project environment variables can be stored), without overwriting any existing file.

user_renviron = path.expand(file.path("~", ".Renviron"))
if(!file.exists(user_renviron)) # check to see if the file already exists
  file.create(user_renviron)
file.edit(user_renviron) # open with another text editor if this fails

The location, contents and uses of each is outlined in more detail below. 3.3.1 The .Rprofile file

By default R looks for and runs .Rprofile files in the three locations described above, in a specific order. .Rprofile files are simply R scripts that run each time R runs and they can be found within R_HOME , HOME and the project's home directory, found with getwd() . To check if you have a site-wide .Rprofile , which will run for all users on start-up, run:

site_path = R.home(component = "home")
fname = file.path(site_path, "etc", "Rprofile.site")
file.exists(fname)

The above code checks for the presence of Rprofile.site in that directory. As outlined above, the .Rprofile located in your home directory is user-specific. Again, we can test whether this file exists using

file.exists("~/.Rprofile")

We can use R to create and edit .Rprofile (warning: do not overwrite your previous .Rprofile - we suggest you try project-specific .Rprofile first):

if(!file.exists("~/.Rprofile")) # only create if not already there
  file.create("~/.Rprofile")    # (don't overwrite it)
file.edit("~/.Rprofile")
3.3.2 Example .Rprofile settings

An .Rprofile file is just an R script that is run at start-up. The examples at the bottom of the .Rprofile help file

help("Rprofile")

give clues as to the types of things we could place in our profile.

3.3.2.1 Setting options

The function options is a list that contains a number of default options. See help("options") or simply type options() to get an idea of what we can configure. In my .Rprofile file, we have the line

options(prompt="R> ", digits=4, show.signif.stars=FALSE)

This changes three features.

Typically we want to avoid adding options to the start-up file that make our code non-portable. For example, adding

options(stringsAsFactors=FALSE)

to your start-up script has knock-on effects for read.table and related functions including read.csv , making them convert text strings into characters rather than into factors as is default. This may be useful for you, but it is dangerous as it may make your code less portable. 3.3.2.2 Setting the CRAN mirror

To avoid setting the CRAN mirror each time you run install.packages you can permanently set the mirror in your .Rprofile .

## local creates a new, empty environment
## This avoids polluting the global environment with
## the object r
local({
  r = getOption("repos")             
  r["CRAN"] = "https://cran.rstudio.com/"
  options(repos = r)
})

The RStudio mirror is a virtual machine run by Amazon's EC2 service, and it syncs with the main CRAN mirror in Austria once per day. Since RStudio is using Amazon's CloudFront, the repository is automatically distributed around the world, so no matter where you are in the world, the data doesn't need to travel very far, and is therefore fast to download. 3.3.2.3 The fortunes package

This section illustrate what .Rprofile does with reference to a package that was developed for fun. The code below could easily be altered to automatically connect to a database, or ensure that the latest packages have been downloaded.

The fortunes package contains a number of memorable quotes that the community has collected over many years, called R fortunes. Each fortune has a number. To get fortune number , for example, enter

fortunes::fortune(50)

It is easy to make R print out one of these nuggets of truth each time you start a session, by adding the following to ~/.Rprofile :

if(interactive()) 
  try(fortunes::fortune(), silent=TRUE)

The interactive function tests whether R is being used interactively in a terminal. The fortune function is called within try . If the fortunes package is not available, we avoid raising an error and move on. By using :: we avoid adding the fortunes package to our list of attached packages..

The function .Last , if it exists in the .Rprofile , is always run at the end of the session. We can use it to install the fortunes package if needed. To load the package, we use require , since if the package isn't installed, the require function returns FALSE and raises a warning.

.Last = function() {
  cond = suppressWarnings(!require(fortunes, quietly=TRUE))
  if(cond) 
    try(install.packages("fortunes"), silent=TRUE)
  message("Goodbye at ", date(), "\n")
}
3.3.2.4 Useful functions

You can also load useful functions in .Rprofile . For example, we could load the following two functions for examining data frames:

## ht == headtail
ht = function(d, n=6) rbind(head(d, n), tail(d, n))
  
## Show the first 5 rows & first 5 columns of a data frame
hh = function(d) d[1:5, 1:5]

and a function for setting a nice plotting window:

setnicepar = function(mar = c(3, 3, 2, 1), mgp = c(2, 0.4, 0), 
                      tck = -.01, cex.axis = 0.9, 
                      las = 1, mfrow = c(1, 1), ...) {
    par(mar = mar, mgp = mgp, tck = tck,
        cex.axis = cex.axis, las = las, 
        mfrow = mfrow, ...)
}

Note that these functions are for personal use and are unlikely to interfere with code from other people. For this reason even if you use a certain package every day, we don't recommend loading it in your .Rprofile . Also beware the dangers of loading many functions by default: it may make your code less portable. Another downside of putting functions in your .Rprofile is that it can clutter-up your work space: when you run the ls() command, your .Rprofile functions will appear. Also if you run rm(list=ls()) , your functions will be deleted.

One neat trick to overcome this issue is to use hidden objects and environments. When an object name starts with . , by default it doesn't appear in the output of the ls() function

.obj = 1
".obj" %in% ls()
## [1] FALSE

This concept also works with environments. In the .Rprofile file we can create a hidden environment

.env = new.env()

and then add functions to this environment

.env$ht = function(d, n = 6) rbind(head(d, n), tail(d, n))

At the end of the .Rprofile file, we use attach , which makes it possible to refer to objects in the environment by their names alone.

attach(.env)
3.3.3 The .Renviron file

The .Renviron file is used to store system variables. It follows a similar start up routine to the .Rprofile file: R first looks for a global .Renviron file, then for local versions. A typical use of the .Renviron file is to specify the R_LIBS path

## Linux
R_LIBS=~/R/library

## Windows
R_LIBS=C:/R/library

This variable points to a directory where R packages will be installed. When install.packages is called, new packages will be stored in R_LIBS .

Another common use of .Renviron is to store API keys that will be available from one session to another. 4 The following line in .Renviron , for example, sets the ZEIT_KEY environment variable which is used in the package diezeit package:

ZEIT_KEY=PUT_YOUR_KEY_HERE

You will need to sign-in and start a new R session for the environment variable (accessed by Sys.getenv ) to be visible. To test if the example API key has been successfully added as an environment variable, run the following:

Sys.getenv("ZEIT_KEY")

Use of the .Renviron file for storing settings such as library paths and API keys is efficient because it reduces the need to update your settings for every R session. Furthermore, the same .Renviron file will work across different platforms so keep it stored safely. 3.3.4 Exercises

  1. What are the three locations where they are stored? Where are these locations on your computer?
  2. For each location, does a .Rprofile or .Renviron file exist?
  3. Create a .Rprofile file in your current working directory that prints the message Happy efficient R programming each time you start R at this location.

[Oct 16, 2019] Python Variable Declaration - Stack Overflow

Oct 16, 2019 | stackoverflow.com

Python Variable Declaration Ask Question Asked 7 years, 4 months ago Active 20 days ago Viewed 249k times 73 48


jpaugh ,Feb 28, 2017 at 23:52

Learning Python , and has some basic doubts.

1.I have seen variable declaration (path here) as

class writer:
    path = ""

sometimes, no explicit declaration but initialize through __init__ .

def __init__(self, name):
    self.name = name

I understand the purpose of __init__ , but is it advisable to declare variable in any other functions.

2.How can I create variable to hold a custom type?

class writer:
    path = "" # string value
    customObj = ??

Martijn Pieters ♦ ,Mar 9 at 3:52

Okay, first things first.

There is no such thing as "variable declaration" or "variable initialization" in Python.

There is simply what we call "assignment", but should probably just call "naming".

Assignment means "this name on the left-hand side now refers to the result of evaluating the right-hand side, regardless of what it referred to before (if anything)".

foo = 'bar' # the name 'foo' is now a name for the string 'bar'
foo = 2 * 3 # the name 'foo' stops being a name for the string 'bar',
# and starts being a name for the integer 6, resulting from the multiplication

As such, Python's names (a better term than "variables", arguably) don't have associated types; the values do. You can re-apply the same name to anything regardless of its type, but the thing still has behaviour that's dependent upon its type. The name is simply a way to refer to the value (object). This answers your second question: You don't create variables to hold a custom type. You don't create variables to hold any particular type. You don't "create" variables at all. You give names to objects.

Second point: Python follows a very simple rule when it comes to classes, that is actually much more consistent than what languages like Java, C++ and C# do: everything declared inside the class block is part of the class . So, functions ( def ) written here are methods, i.e. part of the class object (not stored on a per-instance basis), just like in Java, C++ and C#; but other names here are also part of the class. Again, the names are just names, and they don't have associated types, and functions are objects too in Python. Thus:

class Example:
    data = 42
    def method(self): pass

Classes are objects too , in Python.

So now we have created an object named Example , which represents the class of all things that are Example s. This object has two user-supplied attributes (In C++, "members"; in C#, "fields or properties or methods"; in Java, "fields or methods"). One of them is named data , and it stores the integer value 42 . The other is named method , and it stores a function object. (There are several more attributes that Python adds automatically.)

These attributes still aren't really part of the object, though. Fundamentally, an object is just a bundle of more names (the attribute names), until you get down to things that can't be divided up any more. Thus, values can be shared between different instances of a class, or even between objects of different classes, if you deliberately set that up.

Let's create an instance:

x = Example()

Now we have a separate object named x , which is an instance of Example . The data and method are not actually part of the object, but we can still look them up via x because of some magic that Python does behind the scenes. When we look up method , in particular, we will instead get a "bound method" (when we call it, x gets passed automatically as the self parameter, which cannot happen if we look up Example.method directly).

What happens when we try to use x.data ?

When we examine it, it's looked up in the object first. If it's not found in the object, Python looks in the class.

However, when we assign to x.data , Python will create an attribute on the object. It will not replace the class' attribute.

This allows us to do object initialization. Python will automatically call the class' __init__ method on new instances when they are created, if present. In this method, we can simply assign to attributes to set initial values for that attribute on each object:

class Example:
    name = "Ignored"
    def __init__(self, name):
        self.name = name
    # rest as before

Now we must specify a name when we create an Example , and each instance has its own name . Python will ignore the class attribute Example.name whenever we look up the .name of an instance, because the instance's attribute will be found first.

One last caveat: modification (mutation) and assignment are different things!

In Python, strings are immutable. They cannot be modified. When you do:

a = 'hi '
b = a
a += 'mom'

You do not change the original 'hi ' string. That is impossible in Python. Instead, you create a new string 'hi mom' , and cause a to stop being a name for 'hi ' , and start being a name for 'hi mom' instead. We made b a name for 'hi ' as well, and after re-applying the a name, b is still a name for 'hi ' , because 'hi ' still exists and has not been changed.

But lists can be changed:

a = [1, 2, 3]
b = a
a += [4]

Now b is [1, 2, 3, 4] as well, because we made b a name for the same thing that a named, and then we changed that thing. We did not create a new list for a to name, because Python simply treats += differently for lists.

This matters for objects because if you had a list as a class attribute, and used an instance to modify the list, then the change would be "seen" in all other instances. This is because (a) the data is actually part of the class object, and not any instance object; (b) because you were modifying the list and not doing a simple assignment, you did not create a new instance attribute hiding the class attribute.

detly ,Jun 13, 2012 at 3:38

There's no need to declare new variables in Python. If we're talking about variables in functions or modules, no declaration is needed. Just assign a value to a name where you need it: mymagic = "Magic" . Variables in Python can hold values of any type, and you can't restrict that.

Your question specifically asks about classes, objects and instance variables though. The idiomatic way to create instance variables is in the __init__ method and nowhere else -- while you could create new instance variables in other methods, or even in unrelated code, it's just a bad idea. It'll make your code hard to reason about or to maintain.

So for example:

class Thing(object):

    def __init__(self, magic):
        self.magic = magic

Easy. Now instances of this class have a magic attribute:

thingo = Thing("More magic")
# thingo.magic is now "More magic"

Creating variables in the namespace of the class itself leads to different behaviour altogether. It is functionally different, and you should only do it if you have a specific reason to. For example:

class Thing(object):

    magic = "Magic"

    def __init__(self):
        pass

Now try:

thingo = Thing()
Thing.magic = 1
# thingo.magic is now 1

Or:

class Thing(object):

    magic = ["More", "magic"]

    def __init__(self):
        pass

thing1 = Thing()
thing2 = Thing()
thing1.magic.append("here")
# thing1.magic AND thing2.magic is now ["More", "magic", "here"]

This is because the namespace of the class itself is different to the namespace of the objects created from it. I'll leave it to you to research that a bit more.

The take-home message is that idiomatic Python is to (a) initialise object attributes in your __init__ method, and (b) document the behaviour of your class as needed. You don't need to go to the trouble of full-blown Sphinx-level documentation for everything you ever write, but at least some comments about whatever details you or someone else might need to pick it up.

O. Aroesti ,Jul 4, 2018 at 14:35

This might be 6 years late, but in Python 3.5 and above, you declare a variable type like this:
variable_name: type_name

or this:

variable_name # type: shinyType

So in your case(if you have a CustomObject class defined), you can do:

customObj: CustomObject

See this or that for more info.

redhot ,Jun 22, 2017 at 15:29

For scoping purpose, I use:
custom_object = None

,

Variables have scope, so yes it is appropriate to have variables that are specific to your function. You don't always have to be explicit about their definition; usually you can just use them. Only if you want to do something specific to the type of the variable, like append for a list, do you need to define them before you start using them. Typical example of this.
list = []
for i in stuff:
  list.append(i)

By the way, this is not really a good way to setup the list. It would be better to say:

list = [i for i in stuff] # list comprehension

...but I digress.

Your other question. The custom object should be a class itself.

class CustomObject(): # always capitalize the class name...this is not syntax, just style.
  pass
customObj = CustomObject()

[Oct 15, 2019] Perl to Python Function translation [closed]

Feb 01, 2014 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 5 years, 8 months ago Viewed 303 times -3


Jim Garrison ,Feb 1, 2014 at 22:24

I am trying to translate a Perl function into a Python function, but I am having trouble figuring out what some of the Perl to Python function equivalents.

Perl function:

sub reverse_hex {

 my $HEXDATE = shift;
 my @bytearry=();
 my $byte_cnt = 0;
 my $max_byte_cnt = 8;
 my $byte_offset = 0;
 while($byte_cnt < $max_byte_cnt) {
   my $tmp_str = substr($HEXDATE,$byte_offset,2);
    push(@bytearry,$tmp_str);
   $byte_cnt++;
   $byte_offset+=2;
 }
   return join('',reverse(@bytearry));
}

I am not sure what "push", "shift", and "substr" are doing here that would be the same in Python.

Any help will be much appreciated.

Kenosis ,Feb 1, 2014 at 22:17

The Perl subroutine seems rather complicated for what it does, viz., taking chunks of two chars at a time (the first 16 chars) from the sent string and then reverses it. Another Perl option is:
sub reverse_hex {
    return join '', reverse unpack 'A2' x 8, $_[0];
}

First, unpack here takes two characters at a time (eight times) and produces a list. That list is reverse d and join ed to produce the final string.

Here's a Python subroutine to accomplish this:

def reverse_hex(HEXDATE):
    hexVals = [HEXDATE[i:i + 2] for i in xrange(0, 16, 2)]
    reversedHexVals = hexVals[::-1]
    return ''.join(reversedHexVals)

The list comprehension produces eight elements of two characters each. [::-1] reverses the list's elements and the result is join ed and returned.

Hope this helps!

MikeMayer67 ,Feb 2, 2014 at 2:10

I realize that you are asking about the perl to python translation, but if you have any control over the perl, I would like to point out that this function is a lot more complicated than it needs to be.

The entire thing could be replaced with:

sub reverse_hex
{
  my $hexdate = shift;
  my @bytes = $hexdate =~ /../g;  # break $hexdate into array of character pairs
  return join '', reverse(@bytes);
}

Not only is this shorter, it is much easier to get your head around. Of course, if you have no control over the perl, you are stuck with what you were dealt.

[Oct 15, 2019] What's the easiest way to install a missing Perl module

Oct 15, 2019 | stackoverflow.com

ikegami ,Dec 13, 2017 at 21:21

I get this error:

Can't locate Foo.pm in @INC

Is there an easier way to install it than downloading, untarring, making, etc?

brian d foy ,Jun 10, 2014 at 21:52

On Unix :

usually you start cpan in your shell:

# cpan

and type

install Chocolate::Belgian

or in short form:

cpan Chocolate::Belgian

On Windows :

If you're using ActivePerl on Windows, the PPM (Perl Package Manager) has much of the same functionality as CPAN.pm.

Example:

# ppm
ppm> search net-smtp
ppm> install Net-SMTP-Multipart

see How do I install Perl modules? in the CPAN FAQ

Many distributions ship a lot of perl modules as packages.

You should always prefer them as you benefit from automatic (security) updates and the ease of removal . This can be pretty tricky with the cpan tool itself.

For Gentoo there's a nice tool called g-cpan which builds/installs the module from CPAN and creates a Gentoo package ( ebuild ) for you.

Chas. Owens ,Jan 30, 2017 at 21:08

Try App::cpanminus :
# cpanm Chocolate::Belgian

It's great for just getting stuff installed. It provides none of the more complex functionality of CPAN or CPANPLUS, so it's easy to use, provided you know which module you want to install. If you haven't already got cpanminus, just type:

# cpan App::cpanminus

to install it.

It is also possible to install it without using cpan at all. The basic bootstrap procedure is,

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

For more information go to the App::cpanminus page and look at the section on installation.

isomorphismes ,Mar 22, 2011 at 16:03

I note some folks suggesting one run cpan under sudo. That used to be necessary to install into the system directory, but modern versions of the CPAN shell allow you to configure it to use sudo just for installing. This is much safer, since it means that tests don't run as root.

If you have an old CPAN shell, simply install the new cpan ("install CPAN") and when you reload the shell, it should prompt you to configure these new directives.

Nowadays, when I'm on a system with an old CPAN, the first thing I do is update the shell and set it up to do this so I can do most of my cpan work as a normal user.

Also, I'd strongly suggest that Windows users investigate strawberry Perl . This is a version of Perl that comes packaged with a pre-configured CPAN shell as well as a compiler. It also includes some hard-to-compile Perl modules with their external C library dependencies, notably XML::Parser. This means that you can do the same thing as every other Perl user when it comes to installing modules, and things tend to "just work" a lot more often.

Ivan X ,Sep 19 at 19:32

If you're on Ubuntu and you want to install the pre-packaged perl module (for example, geo::ipfree) try this:
    $ apt-cache search perl geo::ipfree
    libgeo-ipfree-perl - A look up country of ip address Perl module

    $ sudo apt-get install libgeo-ipfree-perl

brian d foy ,Sep 15, 2008 at 22:47

A couple of people mentioned the cpan utility, but it's more than just starting a shell. Just give it the modules that you want to install and let it do it's work.
$prompt> cpan Foo::Bar

If you don't give it any arguments it starts the CPAN.pm shell. This works on Unix, Mac, and should be just fine on Windows (especially Strawberry Perl).

There are several other things that you can do with the cpan tool as well. Here's a summary of the current features (which might be newer than the one that comes with CPAN.pm and perl):

-a
Creates the CPAN.pm autobundle with CPAN::Shell->autobundle.

-A module [ module ... ]
Shows the primary maintainers for the specified modules

-C module [ module ... ]
Show the Changes files for the specified modules

-D module [ module ... ]
Show the module details. This prints one line for each out-of-date module (meaning,
modules locally installed but have newer versions on CPAN). Each line has three columns:
module name, local version, and CPAN version.

-L author [ author ... ]
List the modules by the specified authors.

-h
Prints a help message.

-O
Show the out-of-date modules.

-r
Recompiles dynamically loaded modules with CPAN::Shell->recompile.

-v
Print the script version and CPAN.pm version.

wytten ,Apr 25 at 19:26

sudo perl -MCPAN -e 'install Foo'

Corion ,Sep 16, 2008 at 6:36

Also see Yes, even you can use CPAN . It shows how you can use CPAN without having root or sudo access.

mikegrb ,Sep 16, 2008 at 19:25

Otto made a good suggestion . This works for Debian too, as well as any other Debian derivative. The missing piece is what to do when apt-cache search doesn't find something.
$ sudo apt-get install dh-make-perl build-essential apt-file
$ sudo apt-file update

Then whenever you have a random module you wish to install:

$ cd ~/some/path
$ dh-make-perl --build --cpan Some::Random::Module
$ sudo dpkg -i libsome-random-module-perl-0.01-1_i386.deb

This will give you a deb package that you can install to get Some::Random::Module. One of the big benefits here is man pages and sample scripts in addition to the module itself will be placed in your distro's location of choice. If the distro ever comes out with an official package for a newer version of Some::Random::Module, it will automatically be installed when you apt-get upgrade.

community wiki
jm666
,May 22, 2011 at 18:19

Already answered and accepted answer - but anyway:

IMHO the easiest way installing CPAN modules (on unix like systems, and have no idea about the wondows) is:

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

The above is installing the "zero configuration CPAN modules installer" called cpanm . (Can take several minutes to install - don't break the process)

and after - simply:

cpanm Foo
cpanm Module::One
cpanm Another::Module

brian d foy ,Oct 8, 2008 at 7:26

Lots of recommendation for CPAN.pm , which is great, but if you're using Perl 5.10 then you've also got access to CPANPLUS.pm which is like CPAN.pm but better.

And, of course, it's available on CPAN for people still using older versions of Perl. Why not try:

$ cpan CPANPLUS

IgorGanapolsky ,Jan 30, 2017 at 21:09

Many times it does happen that cpan install command fails with the message like "make test had returned bad status, won't install without force"

In that case following is the way to install the module:

perl -MCPAN -e "CPAN::Shell->force(qw(install Foo::Bar));"

community wiki
2 revs, 2 users 80%
,Apr 13, 2015 at 14:50

On ubuntu most perl modules are already packaged, so installing is much faster than most other systems which have to compile.

To install Foo::Bar at a commmand prompt for example usually you just do:

sudo apt-get install libfoo-bar-perl

Sadly not all modules follow that naming convention.

community wiki
3 revs, 3 users 50%
,Apr 13, 2015 at 14:52

Even it should work:
cpan -i module_name

community wiki
3 revs, 2 users 97%
,Apr 13, 2015 at 16:43

Use cpan command as cpan Modulename
$ cpan HTML::Parser

To install dependencies automatically follow the below

$ perl -MCPAN -e shell
cpan[1]>  o conf prerequisites_policy follow
cpan[2]>  o conf commit
exit

I prefer App::cpanminus , it installs dependencies automatically. Just do

$ cpanm HTML::Parser

brian d foy ,Sep 27, 2008 at 18:58

2 ways that I know of :

USING PPM :

With Windows (ActivePerl) I've used ppm

from the command line type ppm. At the ppm prompt ...

ppm> install foo

or

ppm> search foo

to get a list of foo modules available. Type help for all the commands

USING CPAN :

you can also use CPAN like this ( *nix systems ) :

perl -MCPAN -e 'shell'

gets you a prompt

cpan>

at the prompt ...

cpan> install foo  (again to install the foo module)

type h to get a list of commands for cpan

community wiki
Bruce Alderman
,Nov 21, 2008 at 19:59

On Fedora you can use
# yum install foo

as long as Fedora has an existing package for the module.

community wiki
2 revs, 2 users 89%
,Apr 13, 2015 at 14:51

On Fedora Linux or Enterprise Linux , yum also tracks perl library dependencies. So, if the perl module is available, and some rpm package exports that dependency, it will install the right package for you.
yum install 'perl(Chocolate::Belgian)'

(most likely perl-Chocolate-Belgian package, or even ChocolateFactory package)

community wiki
Mister X
,Dec 28, 2016 at 11:16

Easiest way for me is this:
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install DateTime::TimeZone'

a) automatic recursive dependency detection/resolving/installing

b) it's a shell onliner, good for setup-scripts

community wiki
venkrao
,Sep 11, 2013 at 18:06

If you want to put the new module into a custom location that your cpan shell isn't configured to use, then perhaps, the following will be handy.
 #wget <URL to the module.tgz>
 ##unpack
 perl Build.PL
./Build destdir=$HOME install_base=$HOME
./Build destdir=$HOME install_base=$HOME install

community wiki
2 revs, 2 users 67%
,Apr 13, 2015 at 14:50

Sometimes you can use the yum search foo to search the relative perl module, then use yum install xxx to install.

PW. ,Sep 15, 2008 at 19:26

On Windows with the ActiveState distribution of Perl, use the ppm command.

community wiki
Kamal Nayan
,Oct 1, 2015 at 9:56

Simply executing cpan Foo::Bar on shell would serve the purpose.

community wiki
Ed Dunn
,Nov 4, 2016 at 15:26

Seems like you've already got your answer but I figured I'd chime in. This is what I do in some scripts on an Ubuntu (or debian server)
#!/usr/bin/perl

use warnings;
use strict;

#I've gotten into the habit of setting this on all my scripts, prevents weird path issues if the script is not being run by root
$ENV{'PATH'} = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin';

#Fill this with the perl modules required for your project
my @perl = qw(LWP::Simple XML::LibXML MIME::Lite DBI DateTime Config::Tiny Proc::ProcessTable);

chomp(my $curl = `which curl`);

if(!$curl){ system('apt-get install curl -y > /dev/null'); }

chomp(my $cpanm = system('/bin/bash', '-c', 'which cpanm &>/dev/null'));

#installs cpanm if missing
if($cpanm){ system('curl -s -L http://cpanmin.us | perl - --sudo App::cpanminus'); }

#loops through required modules and installs them if missing
foreach my $x (@perl){
    eval "use $x";
    if($@){
        system("cpanm $x");
        eval "use $x";
    }
}

This works well for me, maybe there is something here you can use.

[Oct 13, 2019] What is the system function in python

Oct 13, 2019 | stackoverflow.com

what is the system function in python Ask Question Asked 9 years, 3 months ago Active 9 years, 3 months ago Viewed 6k times 1


Eva Feldman ,Jul 6, 2010 at 15:55

I want to play with system command in python . for example we have this function in perl : system("ls -la"); and its run ls -la what is the system function in python ? Thanks in Advance .

Felix Kling ,Jul 6, 2010 at 15:58

It is os.system :
import os
os.system('ls -la')

But this won't give you any output. So subprocess.check_output is probably more what you want:

>>> import subprocess
>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

KLee1 ,Jul 6, 2010 at 16:00

import os
os.system("")

From here

> ,

In the os module there is os.system() .

But if you want to do more advanced things with subprocesses the subprocess module provides a higher level interface with more possibilities that is usually preferable.

[Oct 13, 2019] python - Find size and free space of the filesystem containing a given file - Stack Overflow

Oct 13, 2019 | stackoverflow.com

Find size and free space of the filesystem containing a given file Ask Question Asked 8 years, 10 months ago Active 6 months ago Viewed 110k times 67 21


Piskvor ,Aug 21, 2013 at 7:19

I'm using Python 2.6 on Linux. What is the fastest way:

Sven Marnach ,May 5, 2016 at 11:11

If you just need the free space on a device, see the answer using os.statvfs() below.

If you also need the device name and mount point associated with the file, you should call an external program to get this information. df will provide all the information you need -- when called as df filename it prints a line about the partition that contains the file.

To give an example:

import subprocess
df = subprocess.Popen(["df", "filename"], stdout=subprocess.PIPE)
output = df.communicate()[0]
device, size, used, available, percent, mountpoint = \
    output.split("\n")[1].split()

Note that this is rather brittle, since it depends on the exact format of the df output, but I'm not aware of a more robust solution. (There are a few solutions relying on the /proc filesystem below that are even less portable than this one.)

Halfgaar ,Feb 9, 2017 at 10:41

This doesn't give the name of the partition, but you can get the filesystem statistics directly using the statvfs Unix system call. To call it from Python, use os.statvfs('/home/foo/bar/baz') .

The relevant fields in the result, according to POSIX :

unsigned long f_frsize   Fundamental file system block size. 
fsblkcnt_t    f_blocks   Total number of blocks on file system in units of f_frsize. 
fsblkcnt_t    f_bfree    Total number of free blocks. 
fsblkcnt_t    f_bavail   Number of free blocks available to 
                         non-privileged process.

So to make sense of the values, multiply by f_frsize :

import os
statvfs = os.statvfs('/home/foo/bar/baz')

statvfs.f_frsize * statvfs.f_blocks     # Size of filesystem in bytes
statvfs.f_frsize * statvfs.f_bfree      # Actual number of free bytes
statvfs.f_frsize * statvfs.f_bavail     # Number of free bytes that ordinary users
                                        # are allowed to use (excl. reserved space)

Halfgaar ,Feb 9, 2017 at 10:44

import os

def get_mount_point(pathname):
    "Get the mount point of the filesystem containing pathname"
    pathname= os.path.normcase(os.path.realpath(pathname))
    parent_device= path_device= os.stat(pathname).st_dev
    while parent_device == path_device:
        mount_point= pathname
        pathname= os.path.dirname(pathname)
        if pathname == mount_point: break
        parent_device= os.stat(pathname).st_dev
    return mount_point

def get_mounted_device(pathname):
    "Get the device mounted at pathname"
    # uses "/proc/mounts"
    pathname= os.path.normcase(pathname) # might be unnecessary here
    try:
        with open("/proc/mounts", "r") as ifp:
            for line in ifp:
                fields= line.rstrip('\n').split()
                # note that line above assumes that
                # no mount points contain whitespace
                if fields[1] == pathname:
                    return fields[0]
    except EnvironmentError:
        pass
    return None # explicit

def get_fs_freespace(pathname):
    "Get the free space of the filesystem containing pathname"
    stat= os.statvfs(pathname)
    # use f_bfree for superuser, or f_bavail if filesystem
    # has reserved space for superuser
    return stat.f_bfree*stat.f_bsize

Some sample pathnames on my computer:

path 'trash':
  mp /home /dev/sda4
  free 6413754368
path 'smov':
  mp /mnt/S /dev/sde
  free 86761562112
path '/usr/local/lib':
  mp / rootfs
  free 2184364032
path '/proc/self/cmdline':
  mp /proc proc
  free 0
PS

if on Python ≥3.3, there's shutil.disk_usage(path) which returns a named tuple of (total, used, free) expressed in bytes.

Xiong Chiamiov ,Sep 30, 2016 at 20:39

As of Python 3.3, there an easy and direct way to do this with the standard library:
$ cat free_space.py 
#!/usr/bin/env python3

import shutil

total, used, free = shutil.disk_usage(__file__)
print(total, used, free)

$ ./free_space.py 
1007870246912 460794834944 495854989312

These numbers are in bytes. See the documentation for more info.

Giampaolo Rodolà ,Aug 16, 2017 at 9:08

This should make everything you asked:
import os
from collections import namedtuple

disk_ntuple = namedtuple('partition',  'device mountpoint fstype')
usage_ntuple = namedtuple('usage',  'total used free percent')

def disk_partitions(all=False):
    """Return all mountd partitions as a nameduple.
    If all == False return phyisical partitions only.
    """
    phydevs = []
    f = open("/proc/filesystems", "r")
    for line in f:
        if not line.startswith("nodev"):
            phydevs.append(line.strip())

    retlist = []
    f = open('/etc/mtab', "r")
    for line in f:
        if not all and line.startswith('none'):
            continue
        fields = line.split()
        device = fields[0]
        mountpoint = fields[1]
        fstype = fields[2]
        if not all and fstype not in phydevs:
            continue
        if device == 'none':
            device = ''
        ntuple = disk_ntuple(device, mountpoint, fstype)
        retlist.append(ntuple)
    return retlist

def disk_usage(path):
    """Return disk usage associated with path."""
    st = os.statvfs(path)
    free = (st.f_bavail * st.f_frsize)
    total = (st.f_blocks * st.f_frsize)
    used = (st.f_blocks - st.f_bfree) * st.f_frsize
    try:
        percent = ret = (float(used) / total) * 100
    except ZeroDivisionError:
        percent = 0
    # NB: the percentage is -5% than what shown by df due to
    # reserved blocks that we are currently not considering:
    # http://goo.gl/sWGbH
    return usage_ntuple(total, used, free, round(percent, 1))


if __name__ == '__main__':
    for part in disk_partitions():
        print part
        print "    %s\n" % str(disk_usage(part.mountpoint))

On my box the code above prints:

giampaolo@ubuntu:~/dev$ python foo.py 
partition(device='/dev/sda3', mountpoint='/', fstype='ext4')
    usage(total=21378641920, used=4886749184, free=15405903872, percent=22.9)

partition(device='/dev/sda7', mountpoint='/home', fstype='ext4')
    usage(total=30227386368, used=12137168896, free=16554737664, percent=40.2)

partition(device='/dev/sdb1', mountpoint='/media/1CA0-065B', fstype='vfat')
    usage(total=7952400384, used=32768, free=7952367616, percent=0.0)

partition(device='/dev/sr0', mountpoint='/media/WB2PFRE_IT', fstype='iso9660')
    usage(total=695730176, used=695730176, free=0, percent=100.0)

partition(device='/dev/sda6', mountpoint='/media/Dati', fstype='fuseblk')
    usage(total=914217758720, used=614345637888, free=299872120832, percent=67.2)

AK47 ,Jul 7, 2016 at 10:37

The simplest way to find out it.
import os
from collections import namedtuple

DiskUsage = namedtuple('DiskUsage', 'total used free')

def disk_usage(path):
    """Return disk usage statistics about the given path.

    Will return the namedtuple with attributes: 'total', 'used' and 'free',
    which are the amount of total, used and free space, in bytes.
    """
    st = os.statvfs(path)
    free = st.f_bavail * st.f_frsize
    total = st.f_blocks * st.f_frsize
    used = (st.f_blocks - st.f_bfree) * st.f_frsize
    return DiskUsage(total, used, free)

tzot ,Aug 8, 2011 at 10:11

For the first point, you can try using os.path.realpath to get a canonical path, check it against /etc/mtab (I'd actually suggest calling getmntent , but I can't find a normal way to access it) to find the longest match. (to be sure, you should probably stat both the file and the presumed mountpoint to verify that they are in fact on the same device)

For the second point, use os.statvfs to get block size and usage information.

(Disclaimer: I have tested none of this, most of what I know came from the coreutils sources)

andrew ,Dec 15, 2017 at 0:55

For the second part of your question, "get usage statistics of the given partition", psutil makes this easy with the disk_usage(path) function. Given a path, disk_usage() returns a named tuple including total, used, and free space expressed in bytes, plus the percentage usage.

Simple example from documentation:

>>> import psutil
>>> psutil.disk_usage('/')
sdiskusage(total=21378641920, used=4809781248, free=15482871808, percent=22.5)

Psutil works with Python versions from 2.6 to 3.6 and on Linux, Windows, and OSX among other platforms.

Donald Duck ,Jan 12, 2018 at 18:28

import os

def disk_stat(path):
    disk = os.statvfs(path)
    percent = (disk.f_blocks - disk.f_bfree) * 100 / (disk.f_blocks -disk.f_bfree + disk.f_bavail) + 1
    return percent


print disk_stat('/')
print disk_stat('/data')

> ,

Usually the /proc directory contains such information in Linux, it is a virtual filesystem. For example, /proc/mounts gives information about current mounted disks; and you can parse it directly. Utilities like top , df all make use of /proc .

I haven't used it, but this might help too, if you want a wrapper: http://bitbucket.org/chrismiles/psi/wiki/Home

[Oct 09, 2019] oop - Perl Importing Variables From Calling Module - Stack Overflow

Oct 09, 2019 | stackoverflow.com

Perl Importing Variables From Calling Module Ask Question Asked 9 years, 1 month ago Active 9 years, 1 month ago Viewed 4k times 0 1


Russell C. ,Aug 31, 2010 at 20:31

I have a Perl module (Module.pm) that initializes a number of variables, some of which I'd like to import ($VAR2, $VAR3) into additional submodules that it might load during execution.

The way I'm currently setting up Module.pm is as follows:

package Module;

use warnings;
use strict;

use vars qw($SUBMODULES $VAR1 $VAR2 $VAR3);

require Exporter;
our @ISA = qw(Exporter);
our @EXPORT = qw($VAR2 $VAR3);

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    return $self;
}

sub SubModules1 {
    my $self = shift;
    if($SUBMODULES->{'1'}) { return $SUBMODULES->{'1'}; }

    # Load & cache submodule
    require Module::SubModule1;
    $SUBMODULES->{'1'} = Module::SubModule1->new(@_);    
    return $SUBMODULES->{'1'};
}

sub SubModules2 {
    my $self = shift;
    if($SUBMODULES->{'2'}) { return $SUBMODULES->{'2'}; }

    # Load & cache submodule
    require Module::SubModule2;
    $SUBMODULES->{'2'} = Module::SubModule2->new(@_);    
    return $SUBMODULES->{'2'};
}

Each submodule is structured as follows:

package Module::SubModule1;

use warnings;
use strict;
use Carp;

use vars qw();

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    return $self;
}

I want to be able to import the $VAR2 and $VAR3 variables into each of the submodules without having to reference them as $Module::VAR2 and $Module::VAR3. I noticed that the calling script is able to access both the variables that I have exported in Module.pm in the desired fashion but SubModule1.pm and SubModule2.pm still have to reference the variables as being from Module.pm.

I tried updating each submodule as follows which unfortunately didn't work I was hoping:

package Module::SubModule1;

use warnings;
use strict;
use Carp;

use vars qw($VAR2 $VAR3);

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    $VAR2 = $Module::VAR2;
    $VAR3 = $Module::VAR3;
    return $self;
}

Please let me know how I can successfully export $VAR2 and $VAR3 from Module.pm into each Submodule. Thanks in advance for your help!

Russell C. ,Aug 31, 2010 at 22:37

In your submodules, are you forgetting to say
use Module;

? Calling use Module from another package (say Module::Submodule9 ) will try to run the Module::import method. Since you don't have that method, it will call the Exporter::import method, and that is where the magic that exports Module 's variables into the Module::Submodule9 namespace will happen.


In your program there is only one Module namespace and only one instance of the (global) variable $Module::VAR2 . Exporting creates aliases to this variable in other namespaces, so the same variable can be accessed in different ways. Try this in a separate script:

package Whatever;
use Module;
use strict;
use vars qw($VAR2);

$Module::VAR2 = 5;
print $Whatever::VAR2;    # should be 5.
$VAR2 = 14;               # same as $Whatever::VAR2 = 14
print $Module::VAR2;      # should be 14

Russell C. ,Aug 31, 2010 at 21:38

Well there is the easy way:

In M.pm:

package M;

use strict;
use warnings;

#our is better than "use vars" for creating package variables
#it creates an alias to $M::foo named $foo in the current lexical scope 
our $foo = 5;

sub inM { print "$foo\n" }

1;

In M/S.pm

package M;

#creates an alias to $M::foo that will last for the entire scope,
#in this case the entire file
our $foo;

package M::S;

use strict;
use warnings;

sub inMS { print "$foo\n" }

1;

In the script:

#!/usr/bin/perl

use strict;
use warnings;

use M;
use M::S;

M::inM();
M::S::inMS();

But I would advise against this. Global variables are not a good practice, and sharing global variables between modules is even worse.

[Oct 09, 2019] gzip - How can I recover files from a corrupted .tar.gz archive - Stack Overflow

Oct 09, 2019 | stackoverflow.com

15


George ,Jun 24, 2016 at 2:49

Are you sure that it is a gzip file? I would first run 'file SMS.tar.gz' to validate that.

Then I would read the The gzip Recovery Toolkit page.

JohnEye ,Oct 4, 2016 at 11:27

Recovery is possible but it depends on what caused the corruption.

If the file is just truncated, getting some partial result out is not too hard; just run

gunzip < SMS.tar.gz > SMS.tar.partial

which will give some output despite the error at the end.

If the compressed file has large missing blocks, it's basically hopeless after the bad block.

If the compressed file is systematically corrupted in small ways (e.g. transferring the binary file in ASCII mode, which smashes carriage returns and newlines throughout the file), it is possible to recover but requires quite a bit of custom programming, it's really only worth it if you have absolutely no other recourse (no backups) and the data is worth a lot of effort. (I have done it successfully.) I mentioned this scenario in a previous question .

The answers for .zip files differ somewhat, since zip archives have multiple separately-compressed members, so there's more hope (though most commercial tools are rather bogus, they eliminate warnings by patching CRCs, not by recovering good data). But your question was about a .tar.gz file, which is an archive with one big member.

,

Here is one possible scenario that we encountered. We had a tar.gz file that would not decompress, trying to unzip gave the error:
gzip -d A.tar.gz
gzip: A.tar.gz: invalid compressed data--format violated

I figured out that the file may been originally uploaded over a non binary ftp connection (we don't know for sure).

The solution was relatively simple using the unix dos2unix utility

dos2unix A.tar.gz
dos2unix: converting file A.tar.gz to UNIX format ...
tar -xvf A.tar
file1.txt
file2.txt 
....etc.

It worked! This is one slim possibility, and maybe worth a try - it may help somebody out there.

[Oct 09, 2019] scope - What is the difference between my and our in Perl - Stack Overflow

Oct 09, 2019 | stackoverflow.com

Asked 10 years, 5 months ago Active 3 years, 1 month ago Viewed 107k times 180 56


Nathan Fellman ,May 10, 2009 at 10:24

I know what my is in Perl. It defines a variable that exists only in the scope of the block in which it is defined. What does our do? How does our differ from my ?

Nathan Fellman ,Nov 20, 2016 at 1:15

Great question: How does our differ from my and what does our do?

In Summary:

Available since Perl 5, my is a way to declare:


On the other hand, our variables are:


Declaring a variable with our allows you to predeclare variables in order to use them under use strict without getting typo warnings or compile-time errors. Since Perl 5.6, it has replaced the obsolete use vars , which was only file-scoped, and not lexically scoped as is our .

For example, the formal, qualified name for variable $x inside package main is $main::x . Declaring our $x allows you to use the bare $x variable without penalty (i.e., without a resulting error), in the scope of the declaration, when the script uses use strict or use strict "vars" . The scope might be one, or two, or more packages, or one small block.

Georg ,Oct 1, 2016 at 6:41

The PerlMonks and PerlDoc links from cartman and Olafur are a great reference - below is my crack at a summary:

my variables are lexically scoped within a single block defined by {} or within the same file if not in {} s. They are not accessible from packages/subroutines defined outside of the same lexical scope / block.

our variables are scoped within a package/file and accessible from any code that use or require that package/file - name conflicts are resolved between packages by prepending the appropriate namespace.

Just to round it out, local variables are "dynamically" scoped, differing from my variables in that they are also accessible from subroutines called within the same block.

Nathan Fellman ,Nov 20, 2015 at 18:46

An example:
use strict;

for (1 .. 2){
    # Both variables are lexically scoped to the block.
    our ($o);  # Belongs to 'main' package.
    my  ($m);  # Does not belong to a package.

    # The variables differ with respect to newness.
    $o ++;
    $m ++;
    print __PACKAGE__, " >> o=$o m=$m\n";  # $m is always 1.

    # The package has changed, but we still have direct,
    # unqualified access to both variables, because the
    # lexical scope has not changed.
    package Fubb;
    print __PACKAGE__, " >> o=$o m=$m\n";
}

# The our() and my() variables differ with respect to privacy.
# We can still access the variable declared with our(), provided
# that we fully qualify its name, but the variable declared
# with my() is unavailable.
print __PACKAGE__, " >> main::o=$main::o\n";  # 2
print __PACKAGE__, " >> main::m=$main::m\n";  # Undefined.

# Attempts to access the variables directly won't compile.
# print __PACKAGE__, " >> o=$o\n";
# print __PACKAGE__, " >> m=$m\n";

# Variables declared with use vars() are like those declared
# with our(): belong to a package; not private; and not new.
# However, their scoping is package-based rather than lexical.
for (1 .. 9){
    use vars qw($uv);
    $uv ++;
}

# Even though we are outside the lexical scope where the
# use vars() variable was declared, we have direct access
# because the package has not changed.
print __PACKAGE__, " >> uv=$uv\n";

# And we can access it from another package.
package Bubb;
print __PACKAGE__, " >> main::uv=$main::uv\n";

daotoad ,May 10, 2009 at 16:37

Coping with Scoping is a good overview of Perl scoping rules. It's old enough that our is not discussed in the body of the text. It is addressed in the Notes section at the end.

The article talks about package variables and dynamic scope and how that differs from lexical variables and lexical scope.

Chas. Owens ,Oct 7, 2013 at 14:02

my is used for local variables, where as our is used for global variables. More reading over Variable Scoping in Perl: the basics .

ruffin ,Feb 10, 2015 at 19:47

It's an old question, but I ever met some pitfalls about lexical declarations in Perl that messed me up, which are also related to this question, so I just add my summary here:

1. definition or declaration?

local $var = 42; 
print "var: $var\n";

The output is var: 42 . However we couldn't tell if local $var = 42; is a definition or declaration. But how about this:

use strict;
use warnings;

local $var = 42;
print "var: $var\n";

The second program will throw an error:

Global symbol "$var" requires explicit package name.

$var is not defined, which means local $var; is just a declaration! Before using local to declare a variable, make sure that it is defined as a global variable previously.

But why this won't fail?

use strict;
use warnings;

local $a = 42;
print "var: $a\n";

The output is: var: 42 .

That's because $a , as well as $b , is a global variable pre-defined in Perl. Remember the sort function?

2. lexical or global?

I was a C programmer before starting using Perl, so the concept of lexical and global variables seems straightforward to me: just corresponds to auto and external variables in C. But there're small differences:

In C, an external variable is a variable defined outside any function block. On the other hand, an automatic variable is a variable defined inside a function block. Like this:

int global;

int main(void) {
    int local;
}

While in Perl, things are subtle:

sub main {
    $var = 42;
}

&main;

print "var: $var\n";

The output is var: 42 , $var is a global variable even it's defined in a function block! Actually in Perl, any variable is declared as global by default.

The lesson is to always add use strict; use warnings; at the beginning of a Perl program, which will force the programmer to declare the lexical variable explicitly, so that we don't get messed up by some mistakes taken for granted.

Ólafur Waage ,May 10, 2009 at 10:25

The perldoc has a good definition of our.

Unlike my, which both allocates storage for a variable and associates a simple name with that storage for use within the current scope, our associates a simple name with a package variable in the current package, for use within the current scope. In other words, our has the same scoping rules as my, but does not necessarily create a variable.

Cosmicnet ,Nov 22, 2014 at 13:57

This is only somewhat related to the question, but I've just discovered a (to me) obscure bit of perl syntax that you can use with "our" (package) variables that you can't use with "my" (local) variables.
#!/usr/bin/perl

our $foo = "BAR";

print $foo . "\n";
${"foo"} = "BAZ";
print $foo . "\n";

Output:

BAR
BAZ

This won't work if you change 'our' to 'my'.

Okuma.Scott ,Sep 6, 2014 at 20:13

print "package is: " . __PACKAGE__ . "\n";
our $test = 1;
print "trying to print global var from main package: $test\n";

package Changed;

{
        my $test = 10;
        my $test1 = 11;
        print "trying to print local vars from a closed block: $test, $test1\n";
}

&Check_global;

sub Check_global {
        print "trying to print global var from a function: $test\n";
}
print "package is: " . __PACKAGE__ . "\n";
print "trying to print global var outside the func and from \"Changed\" package:     $test\n";
print "trying to print local var outside the block $test1\n";

Will Output this:

package is: main
trying to print global var from main package: 1
trying to print local vars from a closed block: 10, 11
trying to print global var from a function: 1
package is: Changed
trying to print global var outside the func and from "Changed" package: 1
trying to print local var outside the block

In case using "use strict" will get this failure while attempting to run the script:

Global symbol "$test1" requires explicit package name at ./check_global.pl line 24.
Execution of ./check_global.pl aborted due to compilation errors.

Nathan Fellman ,Nov 5, 2015 at 14:03

Just try to use the following program :
#!/usr/local/bin/perl
use feature ':5.10';
#use warnings;
package a;
{
my $b = 100;
our $a = 10;


print "$a \n";
print "$b \n";
}

package b;

#my $b = 200;
#our $a = 20 ;

print "in package b value of  my b $a::b \n";
print "in package b value of our a  $a::a \n";

Nathan Fellman ,May 16, 2013 at 11:07

#!/usr/bin/perl -l

use strict;

# if string below commented out, prints 'lol' , if the string enabled, prints 'eeeeeeeee'
#my $lol = 'eeeeeeeeeee' ;
# no errors or warnings at any case, despite of 'strict'

our $lol = eval {$lol} || 'lol' ;

print $lol;

Evgeniy ,Jan 27, 2016 at 4:57

Let us think what an interpreter actually is: it's a piece of code that stores values in memory and lets the instructions in a program that it interprets access those values by their names, which are specified inside these instructions. So, the big job of an interpreter is to shape the rules of how we should use the names in those instructions to access the values that the interpreter stores.

On encountering "my", the interpreter creates a lexical variable: a named value that the interpreter can access only while it executes a block, and only from within that syntactic block. On encountering "our", the interpreter makes a lexical alias of a package variable: it binds a name, which the interpreter is supposed from then on to process as a lexical variable's name, until the block is finished, to the value of the package variable with the same name.

The effect is that you can then pretend that you're using a lexical variable and bypass the rules of 'use strict' on full qualification of package variables. Since the interpreter automatically creates package variables when they are first used, the side effect of using "our" may also be that the interpreter creates a package variable as well. In this case, two things are created: a package variable, which the interpreter can access from everywhere, provided it's properly designated as requested by 'use strict' (prepended with the name of its package and two colons), and its lexical alias.

Sources:

[Oct 09, 2019] Perl Import Package in different Namespace - Stack Overflow

Oct 09, 2019 | stackoverflow.com

Perl Import Package in different Namespace Ask Question Asked 1 year ago Active 7 months ago Viewed 150 times We're doing things differently. View all 8 job openings! 2


choroba ,Sep 28, 2018 at 22:17

is it possible to import ( use ) a perl module within a different namespace?

Let's say I have a Module A (XS Module with no methods Exported @EXPORT is empty) and I have no way of changing the module.

This Module has a Method A::open

currently I can use that Module in my main program (package main) by calling A::open I would like to have that module inside my package main so that I can directly call open

I tried to manually push every key of %A:: into %main:: however that did not work as expected.

The only way that I know to achieve what I want is by using package A; inside my main program, effectively changing the package of my program from main to A . Im not satisfied with this. I would really like to keep my program inside package main.

Is there any way to achieve this and still keep my program in package main?

Offtopic: Yes I know usually you would not want to import everything into your namespace but this module is used by us extensively and we don't want to type A:: (well the actual module name is way longer which isn't making the situation better)in front of hundreds or thousands of calls

Grinnz ,Oct 1, 2018 at 6:26

This is one of those "impossible" situations, where the clear solution -- to rework that module -- is off limits.

But, you can alias that package's subs names, from its symbol table, to the same names in main . Worse than being rude, this comes with a glitch: it catches all names that that package itself imported in any way. However, since this package is a fixed quantity it stands to reason that you can establish that list (and even hard-code it). It is just this one time, right?

main

use warnings;
use strict;
use feature 'say';

use OffLimits;

GET_SUBS: {
    # The list of names to be excluded
    my $re_exclude = qr/^(?:BEGIN|import)$/;  # ...
    my @subs = grep { !/$re_exclude/ } sort keys %OffLimits::;
    no strict 'refs';
    for my $sub_name (@subs) {
        *{ $sub_name } = \&{ 'OffLimits::' . $sub_name };
    }   
};

my $name = name('name() called from ' . __PACKAGE__);
my $id   = id('id() called from ' . __PACKAGE__);

say "name() returned: $name";
say "id()   returned: $id";

with OffLimits.pm

package OffLimits;    
use warnings;
use strict;

sub name { return "In " .  __PACKAGE__ . ": @_" }
sub id   { return "In " .  __PACKAGE__ . ": @_" }

1;

It prints

name() returned: In OffLimits: name() called from  main
id()   returned: In OffLimits: id() called from  main

You may need that code in a BEGIN block, depending on other details.

Another option is of course to hard-code the subs to be "exported" (in @subs ). Given that the module is in practice immutable this option is reasonable and more reliable.


This can also be wrapped in a module, so that you have the normal, selective, importing.

WrapOffLimits.pm

package WrapOffLimits;
use warnings;
use strict;

use OffLimits;

use Exporter qw(import);

our @sub_names;
our @EXPORT_OK   = @sub_names;
our %EXPORT_TAGS = (all => \@sub_names);

BEGIN { 
    # Or supply a hard-coded list of all module's subs in @sub_names
    my $re_exclude = qr/^(?:BEGIN|import)$/;  # ...
    @sub_names = grep { !/$re_exclude/ } sort keys %OffLimits::;

    no strict 'refs';
    for my $sub_name (@sub_names) {
        *{ $sub_name } = \&{ 'OffLimits::' . $sub_name };
    }   
};
1;

and now in the caller you can import either only some subs

use WrapOffLimits qw(name);

or all

use WrapOffLimits qw(:all);

with otherwise the same main as above for a test.

The module name is hard-coded, which should be OK as this is meant only for that module.


The following is added mostly for completeness.

One can pass the module name to the wrapper by writing one's own import sub, which is what gets used then. The import list can be passed as well, at the expense of an awkward interface of the use statement.

It goes along the lines of

package WrapModule;
use warnings;
use strict;

use OffLimits;

use Exporter qw();  # will need our own import 

our ($mod_name, @sub_names);

our @EXPORT_OK   = @sub_names;
our %EXPORT_TAGS = (all => \@sub_names);

sub import {
    my $mod_name = splice @_, 1, 1;  # remove mod name from @_ for goto

    my $re_exclude = qr/^(?:BEGIN|import)$/;  # etc

    no strict 'refs';
    @sub_names = grep { !/$re_exclude/ } sort keys %{ $mod_name . '::'};    
    for my $sub_name (@sub_names) {    
        *{ $sub_name } = \&{ $mod_name . '::' . $sub_name };
    }   

    push @EXPORT_OK, @sub_names;

    goto &Exporter::import;
}
1;

what can be used as

use WrapModule qw(OffLimits name id);  # or (OffLimits :all)

or, with the list broken-up so to remind the user of the unusual interface

use WrapModule 'OffLimits', qw(name id);

When used with the main above this prints the same output.

The use statement ends up using the import sub defined in the module, which exports symbols by writing to the caller's symbol table. (If no import sub is written then the Exporter 's import method is nicely used, which is how this is normally done.)

This way we are able to unpack the arguments and have the module name supplied at use invocation. With the import list supplied as well now we have to push manually to @EXPORT_OK since this can't be in the BEGIN phase. In the end the sub is replaced by Exporter::import via the (good form of) goto , to complete the job.

Simerax ,Sep 30, 2018 at 10:19

You can forcibly "import" a function into main using glob assignment to alias the subroutine (and you want to do it in BEGIN so it happens at compile time, before calls to that subroutine are parsed later in the file):
use strict;
use warnings;
use Other::Module;

BEGIN { *open = \&Other::Module::open }

However, another problem you might have here is that open is a builtin function, which may cause some problems . You can add use subs 'open'; to indicate that you want to override the built-in function in this case, since you aren't using an actual import function to do so.

Grinnz ,Sep 30, 2018 at 17:33

Here is what I now came up with. Yes this is hacky and yes I also feel like I opened pandoras box with this. However at least a small dummy program ran perfectly fine.

I renamed the module in my code again. In my original post I used the example A::open actually this module does not contain any method/variable reserved by the perl core. This is why I blindly import everything here.

BEGIN {
    # using the caller to determine the parent. Usually this is main but maybe we want it somewhere else in some cases
    my ($parent_package) = caller;

    package A;

    foreach (keys(%A::)) {
        if (defined $$_) {
            eval '*'.$parent_package.'::'.$_.' = \$A::'.$_;
        }
        elsif (%$_) {
            eval '*'.$parent_package.'::'.$_.' = \%A::'.$_;
        }
        elsif (@$_) {
            eval '*'.$parent_package.'::'.$_.' = \@A::'.$_;
        }
        else {
            eval '*'.$parent_package.'::'.$_.' = \&A::'.$_;
        }
    }
}

[Oct 08, 2019] Perl constant array

Oct 08, 2019 | stackoverflow.com

Ask Question Asked 6 years, 1 month ago Active 4 years ago Viewed 5k times 4 1


Alec ,Sep 5, 2018 at 8:25

use constant {
    COLUMNS => qw/ TEST1 TEST2 TEST3 /,
}

Can I store an array using the constant package in Perl?

Whenever I go on to try to use the array like my @attr = (COLUMNS); , it does not contain the values.

Сухой27 ,Aug 12, 2013 at 13:37

use constant {
  COLUMNS => [qw/ TEST1 TEST2 TEST3 /],
};

print @{+COLUMNS};

> ,

Or remove the curly braces as the docs show :-
  1 use strict;
  2 use constant COLUMNS => qw/ TEST1 TEST2 TEST3 /;
  3 
  4 my @attr = (COLUMNS);
  5 print @attr;

which gives :-

 % perl test.pl
TEST1TEST2TEST3

Your code actually defines two constants COLUMNS and TEST2 :-

use strict;
use constant { COLUMNS => qw/ TEST1 TEST2 TEST3 /, };

my @attr = (COLUMNS);
print @attr;
print TEST2

and gives :-

% perl test.pl
TEST1TEST3

[Oct 07, 2019] How to commit to remote git repository

Apr 28, 2012 | stackoverflow.com

Ahmed ,Apr 28, 2012 at 14:32

I am new to git.
I have done a clone of remote repo as follows
git clone https://myusername@something.com/repo.git

then I did

git checkout master

made some changes and committed these changes to my local repository like below..

git add .

git commit -m "my changes"

Now I have to push these changes to the remote repository. I am not sure what to do.

Would I do a merge of my repo to remote ? what steps do I need to take ?

I have git bash and git gui

please advise,
thanks,

zafarkhaja ,Apr 28, 2012 at 14:39

All You have to do is git push origin master , where origin is the default name (alias) of Your remote repository and master is the remote branch You want to push Your changes to.

You may also want to check these out:

  1. http://gitimmersion.com/
  2. http://progit.org/book/

Sergey K. ,Apr 28, 2012 at 14:54

You just need to make sure you have the rights to push to the remote repository and do
git push origin master

or simply

git push

haziz ,Apr 28, 2012 at 21:30

git push

or

git push server_name master

should do the trick, after you have made a commit to your local repository.

Bryan Oakley ,Apr 28, 2012 at 14:34

Have you tried git push ? gitref.org has a nice section dealing with remote repositories .

You can also get help from the command line using the --help option. For example:

% git push --help
GIT-PUSH(1)                             Git Manual                             GIT-PUSH(1)



NAME
       git-push - Update remote refs along with associated objects

SYNOPSIS
       git push [--all | --mirror | --tags] [-n | --dry-run] [--receive-pack=<git-receive-pack>]
                  [--repo=<repository>] [-f | --force] [-v | --verbose] [-u | --set-upstream]
                  [<repository> [<refspec>...]]
...

[Sep 30, 2019] Get command line arguments as string

Sep 30, 2019 | stackoverflow.com

KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

[Sep 30, 2019] How can I convert a string to a number in Perl?

Sep 30, 2019 | stackoverflow.com

Ask Question Asked 10 years, 10 months ago Active 4 years, 7 months ago Viewed 264k times 81 7


TimK ,Mar 1, 2016 at 22:51

How would I convert a string holding a number into its numeric value in Perl?

OrangeDog ,May 22, 2014 at 15:24

You don't need to convert it at all:
% perl -e 'print "5.45" + 0.1;'
5.55

,

This is a simple solution:

Example 1

my $var1 = "123abc";
print $var1 + 0;

Result

123

Example 2

my $var2 = "abc123";
print $var2 + 0;

Result

0

[Sep 28, 2019] Python quoting conventions

Sep 28, 2019 | stackoverflow.com

Share a link to this answer Copy link edited Jul 7 '11 at 14:11 answered Sep 11 '08 at 10:06 Will Harris Will Harris 20.4k 11 11 gold badges 60 60 silver badges 63 63 bronze badges

| show 2 more comments 96 votes

mlissner ,Aug 6, 2011 at 6:19

Quoting the official docs at https://docs.python.org/2.0/ref/strings.html :

In plain English: String literals can be enclosed in matching single quotes (') or double quotes (").

So there is no difference. Instead, people will tell you to choose whichever style that matches the context, and to be consistent . And I would agree - adding that it is pointless to try to come up with "conventions" for this sort of thing because you'll only end up confusing any newcomers.

eksortso ,Jun 10, 2013 at 23:36

I used to prefer ' , especially for '''docstrings''' , as I find """this creates some fluff""" . Also, ' can be typed without the Shift key on my Swiss German keyboard.

I have since changed to using triple quotes for """docstrings""" , to conform to PEP 257 .

Garth Kidd ,Sep 11, 2008 at 10:21

I'm with Will:

I'll stick with that even if it means a lot of escaping.

I get the most value out of single quoted identifiers standing out because of the quotes. The rest of the practices are there just to give those single quoted identifiers some standing room.

Tony Meyer ,Sep 11, 2008 at 8:33

If the string you have contains one, then you should use the other. For example, "You're able to do this" , or 'He said "Hi!"' . Other than that, you should simply be as consistent as you can (within a module, within a package, within a project, within an organisation).

If your code is going to be read by people who work with C/C++ (or if you switch between those languages and Python), then using '' for single-character strings, and "" for longer strings might help ease the transition. (Likewise for following other languages where they are not interchangeable).

The Python code I've seen in the wild tends to favour " over ' , but only slightly. The one exception is that """these""" are much more common than '''these''' , from what I have seen.

jblocksom ,Jul 30, 2009 at 20:35

Triple quoted comments are an interesting subtopic of this question. PEP 257 specifies triple quotes for doc strings . I did a quick check using Google Code Search and found that triple double quotes in Python are about 10x as popular as triple single quotes -- 1.3M vs 131K occurrences in the code Google indexes. So in the multi line case your code is probably going to be more familiar to people if it uses triple double quotes.

Paolo ,Sep 30, 2013 at 15:39

"If you're going to use apostrophes, 
       ^

you'll definitely want to use double quotes".
   ^

For that simple reason, I always use double quotes on the outside. Always

Speaking of fluff, what good is streamlining your string literals with ' if you're going to have to use escape characters to represent apostrophes? Does it offend coders to read novels? I can't imagine how painful high school English class was for you!

dolma33 ,Mar 17, 2014 at 23:13

Python uses quotes something like this:
mystringliteral1="this is a string with 'quotes'"
mystringliteral2='this is a string with "quotes"'
mystringliteral3="""this is a string with "quotes" and more 'quotes'"""
mystringliteral4='''this is a string with 'quotes' and more "quotes"'''
mystringliteral5='this is a string with \"quotes\"'
mystringliteral6='this is a string with \042quotes\042'
mystringliteral6='this is a string with \047quotes\047'

print mystringliteral1
print mystringliteral2
print mystringliteral3
print mystringliteral4
print mystringliteral5
print mystringliteral6

Which gives the following output:

this is a string with 'quotes'
this is a string with "quotes"
this is a string with "quotes" and more 'quotes'
this is a string with 'quotes' and more "quotes"
this is a string with "quotes"
this is a string with 'quotes'

Matt Sheppard ,Sep 11, 2008 at 8:40

I use double quotes in general, but not for any specific reason - Probably just out of habit from Java.

I guess you're also more likely to want apostrophes in an inline literal string than you are to want double quotes.

schwa ,Sep 28, 2008 at 3:35

Personally I stick with one or the other. It doesn't matter. And providing your own meaning to either quote is just to confuse other people when you collaborate.

maxpolk ,Sep 21, 2013 at 17:45

It's probably a stylistic preference more than anything. I just checked PEP 8 and didn't see any mention of single versus double quotes.

I prefer single quotes because its only one keystroke instead of two. That is, I don't have to mash the shift key to make single quote.

stivlo ,Mar 11, 2012 at 4:25

In Perl you want to use single quotes when you have a string which doesn't need to interpolate variables or escaped characters like \n, \t, \r, etc.

PHP makes the same distinction as Perl: content in single quotes will not be interpreted (not even \n will be converted), as opposed to double quotes which can contain variables to have their value printed out.

Python does not, I'm afraid. Technically seen, there is no $ token (or the like) to separate a name/text from a variable in Python. Both features make Python more readable, less confusing, after all. Single and double quotes can be used interchangeably in Python.

Andrew Dalke ,Mar 10, 2009 at 6:25

I chose to use double quotes because they are easier to see.

Alphy

add a comment ,Jul 22, 2009 at 19:50
I just use whatever strikes my fancy at the time; it's convenient to be able to switch between the two at a whim!

Of course, when quoting quote characetrs, switching between the two might not be so whimsical after all...

Vinko Vrsalovic ,Sep 11, 2008 at 8:25

Your team's taste or your project's coding guidelines.

If you are in a multilanguage environment, you might wish to encourage the use of the same type of quotes for strings that the other language uses, for instance. Else, I personally like best the look of '

Mario F ,Sep 11, 2008 at 8:28

None as far as I know. Although if you look at some code, " " is commonly used for strings of text (I guess ' is more common inside text than "), and ' ' appears in hashkeys and things like that.

Acumenus ,Apr 16, 2013 at 22:32

I aim to minimize both pixels and surprise. I typically prefer ' in order to minimize pixels, but " instead if the string has an apostrophe, again to minimize pixels. For a docstring, however, I prefer """ over ''' because the latter is non-standard, uncommon, and therefore surprising. If now I have a bunch of strings where I used " per the above logic, but also one that can get away with a ' , I may still use " in it to preserve consistency, only to minimize surprise.

Perhaps it helps to think of the pixel minimization philosophy in the following way. Would you rather that English characters looked like A B C or AA BB CC ? The latter choice wastes 50% of the non-empty pixels.

Philipp ,Jul 5, 2010 at 13:06

I use double quotes because I have been doing so for years in most languages (C++, Java, VB ) except Bash, because I also use double quotes in normal text and because I'm using a (modified) non-English keyboard where both characters require the shift key.

Adam Smith ,Dec 16, 2013 at 23:38

' = "

/ = \ = \\

example :

f = open('c:\word.txt', 'r')
f = open("c:\word.txt", "r")
f = open("c:/word.txt", "r")
f = open("c:\\\word.txt", "r")

Results are the same

=>> no, they're not the same. A single backslash will escape characters. You just happen to luck out in that example because \k and \w aren't valid escapes like \t or \n or \\ or \"

If you want to use single backslashes (and have them interpreted as such), then you need to use a "raw" string. You can do this by putting an ' r ' in front of the string

im_raw = r'c:\temp.txt'
non_raw = 'c:\\temp.txt'
another_way = 'c:/temp.txt'

As far as paths in Windows are concerned, forward slashes are interpreted the same way. Clearly the string itself is different though. I wouldn't guarantee that they're handled this way on an external device though.

[Sep 28, 2019] python - Get command line arguments as string - Stack Overflow

Sep 28, 2019 | stackoverflow.com

Get command line arguments as string Ask Question Asked 3 years, 3 months ago Active 2 months ago Viewed 21k times 7 1


KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

> ,

Or can do format , and multiply the string by length of sys.argv :
(' {}'*len(sys.argv)).lstrip().format(*sys.argv)

Or can do % :

(' %s'*len(sys.argv)).lstrip()%sys.argv

[Sep 24, 2019] How can I compile my Perl script so it can be executed on systems without perl installed

Sep 24, 2019 | stackoverflow.com
Install PAR::Packer . Example for *nix:
sudo cpan -i PAR::Packer

For Strawberry Perl for Windows or for ActivePerl and MSVC installed:

cpan -i PAR::Packer

Pack it with pp . It will create an executable named "example" or "example.exe" on Windows.

pp -o example example.pl

This would work only on the OS where it was built.

P.S. It is really hard to find a Unix clone without Perl. Did you mean Windows?

[Sep 24, 2019] How to properly use the try catch in perl that error.pm provides?

Sep 24, 2019 | stackoverflow.com

Sinan Ünür ,Apr 28, 2012 at 18:07

Last I checked, Error was deprecated. But here's how you would do it without that module:
eval {
    die "Oops!";
    1;
} or do {
    my $e = $@;
    print("Something went wrong: $e\n");
};

Basically, use eval instead of try , die instead of throw , and look for the exception in $@ . The true value at the end of the eval block is part of an idiom to prevent $@ from unintentionally changing before it is used again in Perl versions older than 5.14, see P::C::P::ErrorHandling::RequireCheckingReturnValueOfEval for details. For example, this code suffers from this flaw.

# BAD, DO NOT USE WITH PERLS OLDER THAN 5.14
eval {
    die "Oops!";
};
if (my $e = $@) {
    print("Something went wrong: $e\n");
}
# BAD, DO NOT USE WITH PERLS OLDER THAN 5.14

But note that many Perl operations do not raise exceptions when they fail; they simply return an error code. This behavior can be altered via autodie for builtins and standard modules. If you're using autodie , then the standard way of doing try/catch is this (straight out of the autodie perldoc):

use feature qw(switch);

eval {
   use autodie;

   open(my $fh, '<', $some_file);

   my @records = <$fh>;

   # Do things with @records...

   close($fh);

};

given ($@) {
   when (undef)   { say "No error";                    }
   when ('open')  { say "Error from open";             }
   when (':io')   { say "Non-open, IO error.";         }
   when (':all')  { say "All other autodie errors."    }
   default        { say "Not an autodie error at all." }
}

For getting a stacktrace, look at Carp .

> ,

If you want something a bit more powerful than Try::Tiny, you might want to try looking at the TryCatch module in CPAN.

[Sep 24, 2019] What is the best way to handle exceptions in Perl - Stack Overflow

Sep 24, 2019 | stackoverflow.com

Michael Carman ,Apr 8 at 9:52

The consensus of the Perl community seems to be that Try::Tiny is the preferred way of doing exception handling. The "lenient policy" you refer to is probably due to a combination of:

Note that the last item means that you'll see a lot of code like this:

eval { something() };
if ($@) {
    warn "Oh no! [$@]\n";
}

That's exception handling even though it doesn't use try/catch syntax. It's fragile, though, and will break in a number of subtle edge cases that most people don't think about.

Try::Tiny and the other exception handling modules on CPAN were written to make it easier to get right.

1. C does have setjmp() and longjmp() , which can be used for a very crude form of exception handling.

,

Never test $@ as is, because it is a global variable, so even the test itself can change it.

General eval-template:

my $result;

eval {
    $result= something();
    # ...
    1;  # ok
} or do {
    my $eval_error= $@ || "error";
    # ...
    die $eval_error;
};  # needs a semicolon

In practice that is the lightest way. It still leaves a tiny room for funny $@ behaviour, but nothing that really concerned me enough.

[Sep 21, 2019] How Did Perl Lose Ground to Bash?

Notable quotes:
"... It baffles me the most because the common objection to Perl is legibility. Even if you assume that the objection is made from ignorance - i.e. not even having looked at some Perl to gauge its legibility - the nonsense you see in a complex bash script is orders of magnitude worse! ..."
"... Maybe it's not reassuring to hear that, but I took an interest in Perl precisely because it's seen as an underdog and "dead" despite having experienced users and a lot of code, kind of like TCL, Prolog, or Ada. ..."
"... There's a long history of bad code written by mediocre developers who became the only one who could maintain the codebase until they no longer worked for the organization. The next poor sap to go in found a mess of a codebase and did their best to not break it further. After a few iterations, the whole thing is ready for /dev/null and Perl gets the blame. ..."
"... All in all, Perl is still my first go-to language, but there are definitely some things I wish it did better. ..."
"... The Perl leadership Osborned itself with Perl6. 20/20 hindsight says the new project should have been given a different name at conception, that way all the "watch this space -- under construction" signage wouldn't have steered people away from perfectly usable Perl5. Again, IMO. ..."
"... I don't observe the premise at all though. Is bash really gaining ground over anything recently? ..."
"... Python again is loved, because "taught by rote" idiots. Now you can give them pretty little packages. And it's no wonder they can do little better than be glorified system admins (which id rather have a real sys admin, since he's likely to understand Perl) ..."
"... Making a new language means lots of new training. Lots of profit in this. Nobody profits from writing new books on old languages. Lots of profit in general from supporting a new language. In the end, owning the language gets you profits. ..."
"... And I still don't get why tab for blocks python is even remotely more readable than Perl. ..."
"... If anything, JavaScript is pretty dang godly at what it does, I understand why that's popular. But I don't get python one bit, except to employ millions of entry level minions who can't think on their own. ..."
"... "Every teacher I know has students using it. We do it because it's an easy language, there's only one way to do it, and with whitespace as syntax it's easy to grade. We don't teach it because it is some powerful or exceptional language. " ..."
Sep 21, 2019 | www.reddit.com

How Did Perl Lose Ground to Bash?

Setting aside Perl vs. Python for the moment, how did Perl lose ground to Bash? It used to be that Bash scripts often got replaced by Perl scripts because Perl was more powerful. Even with very modern versions of Bash, Perl is much more powerful.

The Linux Standards Base (LSB) has helped ensure that certain tools are in predictable locations. Bash has gotten a bit more powerful since the release of 4.x, sure. Arrays, handicapped to 2-D arrays, have improved somewhat. There is a native regex engine in Bash 3.x, which admit is a big deal. There is also support for hash maps.

This is all good stuff for Bash. But, none of this is sufficient to explain why Perl isn't the thing you learn after Bash, or, after Bash and Python; take your pick. Thoughts?

28 comments 75% Upvoted What are your thoughts? Log in or Sign up log in sign up Sort by

oldmanwillow21 9 points · 9 days ago

Because Perl has suffered immensely in the popularity arena and is now viewed as undesirable. It's not that Bash is seen as an adequate replacement for Perl, that's where Python has landed.

emilper 8 points · 8 days ago

How did Perl5 lose ground to anything else?

Thusly

- "thou must use Moose for everything" -> "Perl is too slow" -> rewrite in Python because the architect loves Python -> Python is even slower -> architect shunned by the team and everything new written in Go, nobody dares to complain about speed now because the budget people don't trust them -> Perl is slow

- "globals are bad, singletons are good" -> spaghetti -> Perl is unreadable

- "lets use every single item from the gang of four book" -> insanity -> Perl is bad

- "we must be more OOP" -> everything is a faux object with everything else as attributes -> maintenance team quits and they all take PHP jobs, at least the PHP people know their place in the order of things and do less hype-driven-development -> Perl is not OOP enough

- "CGI is bad" -> app needs 6.54GB of RAM for one worker -> customer refuses to pay for more RAM, fires the team, picks a PHP team to do the next version -> PHP team laughs all the way to the bank, chanting "CGI is king"

recrof 2 points · 8 days ago

"CGI is bad" is real. PSGI or FCGI is much faster for web services, and if there are memory leaks, it's always possible to debug & fix them.

Grinnz 6 points · 8 days ago

CGI is fine, when it's all you need. There are many different use cases out there. Just don't use CGI.pm .

emilper 2 points · 7 days ago

memory leaks

memory leaks ... do huge monoliths count as "memory leaks" ?

Altreus 7 points · 8 days ago

It baffles me the most because the common objection to Perl is legibility. Even if you assume that the objection is made from ignorance - i.e. not even having looked at some Perl to gauge its legibility - the nonsense you see in a complex bash script is orders of magnitude worse!

Not to mention its total lack of common language features like first-class data and... Like, a compiler...

I no longer write bash scripts because it takes about 5 lines to become unmaintainable.

crashorbit 5 points · 9 days ago

Every language that reaches functional equity with Perl is perceived as better than it. Mostly because hey, at least it's not Perl.

oldmanwillow21 15 points · 9 days ago · edited 9 days ago

Jumbled mess of thoughts surely to follow.

When I discuss projects with peers and mention that I chose to develop in Perl, the responses range from passive bemusement, to scorn, to ridicule. The assumption is usually that I'm using a dead language that's crippled in functionality and uses syntax that will surely make everyone's eyes bleed to read. This is the culture everywhere from the casual hackers to the C-suite.

I've proven at work that I can write nontrivial software using Perl. I'm still asked to use Python or Go (edit: or node, ugh) for any project that'll have contributors from other teams, or to containerize apps using Docker to remove the need for Perl knowledge for end-users (no CPAN, carton, etc.). But I'll take what I can get, and now the attitude has gone from "get with the times" or "that's cute", to "ok but I don't expect everyone else to know it".

Perl has got a lot to offer, and I vastly enjoy using it over other languages I work with. I know that all the impassioned figures in the Perl community love it just the same, but the community's got some major fragmentation going on. I understand that everyone's got ideas about the future of the language, but is this really the best time to pull the community apart? I feel like if everyone was able to let go of their ego and put their heads together to bring us to a point of stability, even a place where we're not laughed at for professing our support for the language, it would be a major step in the right direction. I think we're heading to the bottom fast, otherwise.

In that spirit of togetherness, I think the language, particularly the community, needs to be made more accessible to newcomers. Not accessible to one Perl offshoot, but accessible to Perl. It needs to be decided what Perl means in today's day and age. What can it do? Why would I want to use it over another shiny language? What are the definitive places I can go to learn more? Who else will be there? How do I contribute and grow as a Perl developer? There need to be people talking about Perl in places that aren't necessarily hubs for other Perl enthusiasts. It needs to be something business decision-makers can look at and feel confident in using.

I really hope something changes. I'd be pretty sad if I had to spend the rest of my career writing whatever the trendy language of the day is. These are just observations from someone that likes writing Perl and has been watching from the sidelines.

PhloxPaniculata 2 points · 7 days ago

Maybe it's not reassuring to hear that, but I took an interest in Perl precisely because it's seen as an underdog and "dead" despite having experienced users and a lot of code, kind of like TCL, Prolog, or Ada.

Being able to read Modern Perl for free also helped a lot. I'm still lacking experience in Perl and I've yet to write anything of importance in it because I don't see an area in which it's clearly better than anything else, either because of the language, a package, or a framework, and I don't do a lot of text-munging anymore (I'm also a fan of awk so for small tasks it has the priority).

codon011 1 point · 9 days ago

Don't call it Perl. Unfortunately. Also IME multitasking in Perl5 (or the lack thereof and/or severe issues with) has been a detriment to it's standing in a "multithread all the things" world.

crashorbit 4 points · 8 days ago

So often I see people drag themselves down that "thread my app" path. Eventually realize that they are implementing a whole multi-processing operating system inside their app rather than taking advantage of the perfectly good one they are running on.

There are several perfectly good ways to do concurrency, multitasking, async IO and so on in perl. Many work well in the single node case and in the multi-node case. Anyone who tells you that multitasking systems are easy because of some implementation language choice has not made it through the whole Dunning Kruger cycle yet.

codon011 2 points · 8 days ago

Multithreading is never easy. The processors will always manage to do things in a "wrong" order unless you are very careful with your gatekeeping. However, other languages/frameworks have paradigms that make it seem easier such that those race conditions show up much later in your product lifecycle.

codon011 3 points · 9 days ago

There's a long history of bad code written by mediocre developers who became the only one who could maintain the codebase until they no longer worked for the organization. The next poor sap to go in found a mess of a codebase and did their best to not break it further. After a few iterations, the whole thing is ready for /dev/null and Perl gets the blame.

Bash has limitations, but that (usually) means fewer ways to mess it up. There's less domain knowledge to learn, (afaik) no CPAN equivalent, and fewer issues with things like "I need to upgrade this but I can't because this other thing uses this older version which is incompatible with the newer version so now we have to maintain two versions of the library and/or interpreter."

All in all, Perl is still my first go-to language, but there are definitely some things I wish it did better.

crb3 3 points · 9 days ago · edited 9 days ago

*[e:] Consider, not just core here, but CPAN pull-in as well. I had one project clobbered on a smaller-memory machine when I tried to set up a pure-Perl scp transfer -- there wasn't room enough for the full file to transfer if it was larger than about 50k, what with all the CPAN. Shelling to commandline scp worked just fine.

beermad 2 points · 8 days ago

To be fair, wrapping a Perl script around something that's (if I read your comment right) just running SCP is adding a pointless extra layer of complexity anyway.

It's a matter of using the best tool for each particular job, not just sticking with one. My own ~/bin directory has a big mix of Perl and pure shell, depending on the complexity of the job to be done.

crb3 2 points · 8 days ago · edited 7 days ago

Agreed; I brought that example up to illustrate the bulk issue. In it, I was feeling my way, not sure how much finagling I might have to do for the task (backdoor-passing legitimate sparse but possibly quite bulky email from one server to another), which is why I initially went for the pure-Perl approach, so I'd have the mechanics exposed for any needed hackery. The experience taught me to get by more on shelling to precompiled tooling where appropriate... and a healthy respect for CPAN pull-in, [e:] the way that this module depends on that module so it gets pulled in along with its dependencies in turn, and the pileup grows in memory. There was a time or two here and there where I only needed a teeny bit of what a module does, so I went in and studied the code, then implemented it internally as a function without the object's generalities and bulk. The caution learned on ancient x86 boxes now seems appropriate on ARM boards like rPi; what goes around comes around.

minimim 1 point · 4 days ago

wouldn't have steered people away from perfectly usable Perl5

Perl5 development was completely stalled at the time. Perl6 brought not only new blood into it's own effort, it reinvigorated Perl5 in the process.

It's completely backwards to suggest Perl 5 was fine until perl6 came along. It was almost dormant and became a lively language after Perl 6 was announced.

perlancar 2 points · 8 days ago

I don't observe the premise at all though. Is bash really gaining ground over anything recently? l

linearblade 3 points · 8 days ago

Perl is better than pretty much everything g out there at what it does.

But keep in mind,

They say C sharp is loved by everyone, when in reality it's Microsoft pushing their narrative and the army of "learn by rote" engineers In developing countries

Python again is loved, because "taught by rote" idiots. Now you can give them pretty little packages. And it's no wonder they can do little better than be glorified system admins (which id rather have a real sys admin, since he's likely to understand Perl)

Making a new language means lots of new training. Lots of profit in this. Nobody profits from writing new books on old languages. Lots of profit in general from supporting a new language. In the end, owning the language gets you profits.

And I still don't get why tab for blocks python is even remotely more readable than Perl.

If anything, JavaScript is pretty dang godly at what it does, I understand why that's popular. But I don't get python one bit, except to employ millions of entry level minions who can't think on their own.

duo-rotae 6 points · 8 days ago

I know a comp sci professor. I asked why he thought Python was so popular.

"Every teacher I know has students using it. We do it because it's an easy language, there's only one way to do it, and with whitespace as syntax it's easy to grade. We don't teach it because it is some powerful or exceptional language. "

Then he said if he really needs to get something done, it's Perl or C.

linearblade 2 points · 8 days ago

Yep that's pretty much my opinion from using it.

techsnapp 1 point · 2 days ago

So is per harder than python because the lack of everyone else using it?

duo-rotae 1 point · 2 days ago

Perl has a steeper and longer learning with it. curve than Python, and there is more than one way to do anything. And there quite a few that continue coding

[Sep 19, 2019] List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons).

Notable quotes:
"... List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons). ..."
Sep 19, 2019 | stackoverflow.com

List::Util's min and max are fine,

use List::Util qw( min max );
my $min = min @numbers;
my $max = max @numbers;

But List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons).

use List::MoreUtils qw( minmax );
my ($min, $max) = minmax @numbers;

List::Util is part of core, but List::MoreUtils isn't.

--ikegami

[Sep 17, 2019] How can a Perl regex re-use part of the previous match for the next match?

Sep 17, 2019 | stackoverflow.com

Ask Question Asked 10 years, 1 month ago Active 10 years, 1 month ago Viewed 2k times 2


dlw ,Aug 16, 2009 at 3:52

I need some Perl regular expression help. The following snippet of code:
use strict; 
use warnings; 
my $str = "In this example, A plus B equals C, D plus E plus F equals G and H plus I plus J plus K equals L"; 
my $word = "plus"; 
my @results = ();
1 while $str =~ s/(.{2}\b$word\b.{2})/push(@results,"$1\n")/e;
print @results;

Produces the following output:

A plus B
D plus E
2 plus F
H plus I
4 plus J
5 plus K

What I want to see is this, where a character already matched can appear in a new match in a different context:

A plus B
D plus E
E plus F
H plus I
I plus J
J plus K

How do I change the regular expression to get this result? Thanks --- Dan

Michael Carman ,Aug 16, 2009 at 4:11

General advice: Don't use s/// when you want m// . Be specific in what you match.

The answer is pos :

#!/usr/bin/perl -l

use strict;
use warnings;

my $str = 'In this example, ' . 'A plus B equals C, ' .
          'D plus E plus F equals G ' .
          'and H plus I plus J plus K equals L';

my $word = "plus";

my @results;

while ( $str =~ /([A-Z] $word [A-Z])/g ) {
    push @results, $1;
    pos($str) -= 1;
}

print "'$_'" for @results;

Output:

C:\Temp> b
'A plus B'
'D plus E'
'E plus F'
'H plus I'
'I plus J'
'J plus K'

Michael Carman ,Aug 16, 2009 at 2:56

You can use a m//g instead of s/// and assign to the pos function to rewind the match location before the second term:
use strict;
use warnings;

my $str  = 'In this example, A plus B equals C, D plus E plus F equals G and H plus I plus J plus K equals L';
my $word = 'plus';
my @results;

while ($str =~ /(.{2}\b$word\b(.{2}))/g) {
    push @results, "$1\n";
    pos $str -= length $2;
}
print @results;

dlw ,Aug 18, 2009 at 13:00

Another option is to use a lookahead:
use strict; 
use warnings; 
my $str = "In this example, A plus B equals C, D plus E "
        . "plus F equals G and H plus I plus J plus K equals L"; 
my $word = "plus"; 
my $chars = 2;
my @results = ();

push @results, $1 
  while $str =~ /(?=((.{0,$chars}?\b$word\b).{0,$chars}))\2/g;

print "'$_'\n" for @results;

Within the lookahead, capturing group 1 matches the word along with a variable number of leading and trailing context characters, up to whatever maximum you've set. When the lookahead finishes, the backreference \2 matches "for real" whatever was captured by group 2, which is the same as group 1 except that it stops at the end of the word. That sets pos where you want it, without requiring you to calculate how many characters you actually matched after the word.

ysth ,Aug 16, 2009 at 9:01

Given the "Full Disclosure" comment (but assuming .{0,35} , not .{35} ), I'd do
use List::Util qw/max min/;
my $context = 35;
while ( $str =~ /\b$word\b/g ) {
    my $pre = substr( $str, max(0, $-[0] - $context), min( $-[0], $context ) );
    my $post = substr( $str, $+[0], $context );
    my $match = substr( $str, $-[0], $+[0] - $-[0] );
    $pre =~ s/.*\n//s;
    $post =~ s/\n.*//s;
    push @results, "$pre$match$post";
}
print for @results;

You'd skip the substitutions if you really meant (?s:.{0,35}) .

Greg Hewgill ,Aug 16, 2009 at 2:29

Here's one way to do it:
use strict; 
use warnings; 
my $str = "In this example, A plus B equals C, D plus E plus F equals G and H plus I plus J plus K equals L"; 
my $word = "plus"; 
my @results = ();
my $i = 0;
while (substr($str, $i) =~ /(.{2}\b$word\b.{2})/) {
    push @results, "$1\n";
    $i += $-[0] + 1;
}
print @results;

It's not terribly Perl-ish, but it works and it doesn't use too many obscure regular expression tricks. However, you might have to look up the function of the special variable @- in perlvar .

ghostdog74 ,Aug 16, 2009 at 3:44

don't have to use regex. basically, just split up the string, use a loop to go over each items, check for "plus" , then get the word from before and after.
my $str = "In this example, A plus B equals C, D plus E plus F equals G and H plus I plus J plus K equals L"; 
@s = split /\s+/,$str;
for($i=0;$i<=scalar @s;$i++){
    if ( "$s[$i]"  eq "plus" ){
        print "$s[$i-1] plus $s[$i+1]\n";
    }
}

[Sep 16, 2019] How can I find the location of a regex match in Perl?

Notable quotes:
"... The built-in variables @- and @+ hold the start and end positions, respectively, of the last successful match. $-[0] and $+[0] correspond to entire pattern, while $-[N] and $+[N] correspond to the $N ( $1 , $2 , etc.) submatches. ..."
"... Edited to add: to quote from perlvar on $1..$9. "These variables are all read-only and dynamically scoped to the current BLOCK." In other words, if you want to use $1..$9, you cannot use a subroutine to do the matching. ..."
Sep 17, 2008 | stackoverflow.com

Michael Carman ,Sep 17, 2008 at 20:58

I need to write a function that receives a string and a regex. I need to check if there is a match and return the start and end location of a match. (The regex was already compiled by qr// .)

The function might also receive a "global" flag and then I need to return the (start,end) pairs of all the matches.

I cannot change the regex, not even add () around it as the user might use () and \1 . Maybe I can use (?:) .

Example: given "ababab" and the regex qr/ab/ , in the global case I need to get back 3 pairs of (start, end).

Nick T ,Sep 8, 2015 at 19:58

The built-in variables @- and @+ hold the start and end positions, respectively, of the last successful match. $-[0] and $+[0] correspond to entire pattern, while $-[N] and $+[N] correspond to the $N ( $1 , $2 , etc.) submatches.

szabgab ,Sep 17, 2008 at 23:51

Forget my previous post, I've got a better idea.
sub match_positions {
    my ($regex, $string) = @_;
    return if not $string =~ /$regex/;
    return ($-[0], $+[0]);
}
sub match_all_positions {
    my ($regex, $string) = @_;
    my @ret;
    while ($string =~ /$regex/g) {
        push @ret, [ $-[0], $+[0] ];
    }
    return @ret
}

This technique doesn't change the the regex in any way.

Edited to add: to quote from perlvar on $1..$9. "These variables are all read-only and dynamically scoped to the current BLOCK." In other words, if you want to use $1..$9, you cannot use a subroutine to do the matching.

Aftershock ,Dec 23, 2012 at 12:13

The pos function gives you the position of the match. If you put your regex in parentheses you can get the length (and thus the end) using length $1 . Like this
sub match_positions {
    my ($regex, $string) = @_;
    return if not $string =~ /($regex)/;
    return (pos($string), pos($string) + length $1);
}
sub all_match_positions {
    my ($regex, $string) = @_;
    my @ret;
    while ($string =~ /($regex)/g) {
        push @ret, [pos($string), pos($string) + length $1];
    }
    return @ret
}

zigdon ,Sep 17, 2008 at 20:43

You can also use the deprecated $` variable, if you're willing to have all the REs in your program execute slower. From perlvar:
   $'      The string preceding whatever was matched by the last successful pattern match (not
           counting any matches hidden within a BLOCK or eval enclosed by the current BLOCK).
           (Mnemonic: "`" often precedes a quoted string.)  This variable is read-only.

           The use of this variable anywhere in a program imposes a considerable performance penalty
           on all regular expression matches.  See "BUGS".

Shicheng Guo ,Jan 22, 2016 at 0:16

#!/usr/bin/perl

# search the postions for the CpGs in human genome

sub match_positions {
    my ($regex, $string) = @_;
    return if not $string =~ /($regex)/;
    return (pos($string), pos($string) + length $1);
}
sub all_match_positions {
    my ($regex, $string) = @_;
    my @ret;
    while ($string =~ /($regex)/g) {
        push @ret, [(pos($string)-length $1),pos($string)-1];
    }
    return @ret
}

my $regex='CG';
my $string="ACGACGCGCGCG";
my $cgap=3;    
my @pos=all_match_positions($regex,$string);

my @hgcg;

foreach my $pos(@pos){
    push @hgcg,@$pos[1];
}

foreach my $i(0..($#hgcg-$cgap+1)){
my $len=$hgcg[$i+$cgap-1]-$hgcg[$i]+2;
print "$len\n"; 
}

[Sep 16, 2019] How can I capture multiple matches from the same Perl regex - Stack Overflow

Sep 16, 2019 | stackoverflow.com

How can I capture multiple matches from the same Perl regex? Ask Question Asked 9 years, 4 months ago Active 7 years, 4 months ago Viewed 35k times 24 1


brian d foy ,May 22, 2010 at 15:42

I'm trying to parse a single string and get multiple chunks of data out from the same string with the same regex conditions. I'm parsing a single HTML doc that is static (For an undisclosed reason, I can't use an HTML parser to do the job.) I have an expression that looks like:
$string =~ /\<img\ssrc\="(.*)"/;

and I want to get the value of $1. However, in the one string, there are many img tags like this, so I need something like an array returned (@1?) is this possible?

VolatileRig ,Jan 14, 2014 at 19:41

As Jim's answer, use the /g modifier (in list context or in a loop).

But beware of greediness, you dont want the .* to match more than necessary (and dont escape < = , they are not special).

while($string =~ /<img\s+src="(.*?)"/g ) {
  ...
}

Robert Wohlfarth ,May 21, 2010 at 18:44

@list = ($string =~ m/\<img\ssrc\="(.*)"/g);

The g modifier matches all occurences in the string. List context returns all of the matches. See the m// operator in perlop .

dalton ,May 21, 2010 at 18:42

You just need the global modifier /g at the end of the match. Then loop through until there are no matches remaining
my @matches;
while ($string =~ /\<img\ssrc\="(.*)"/g) {
        push(@matches, $1);
}

VolatileRig ,May 24, 2010 at 16:37

Use the /g modifier and list context on the left, as in
@result = $string =~ /\<img\ssrc\="(.*)"/g;

[Sep 16, 2019] Pretty-print for shell script

Sep 16, 2019 | stackoverflow.com

Benoit ,Oct 21, 2010 at 13:19

I'm looking for something similiar to indent but for (bash) scripts. Console only, no colorizing, etc.

Do you know of one ?

Jamie ,Sep 11, 2012 at 3:00

Vim can indent bash scripts. But not reformat them before indenting.
Backup your bash script, open it with vim, type gg=GZZ and indent will be corrected. (Note for the impatient: this overwrites the file, so be sure to do that backup!)

Though, some bugs with << (expecting EOF as first character on a line) e.g.

EDIT: ZZ not ZQ

Daniel Martí ,Apr 8, 2018 at 13:52

A bit late to the party, but it looks like shfmt could do the trick for you.

Brian Chrisman ,Sep 9 at 7:47

In bash I do this:
reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3 | sed -e "s/^\s\s\s\s//"
}

this eliminates comments and reindents the script "bash way".

If you have HEREDOCS in your script, they got ruined by the sed in the previous function.

So use:

reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3"
}

But all your script will have a 4 spaces indentation.

Or you can do:

reindent () 
{ 
    rstr=$(mktemp -u "XXXXXXXXXX");
    source <(echo "Zibri () {";cat "$1"|sed -e "s/^\s\s\s\s/$rstr/"; echo "}");
    echo '#!/bin/bash';
    declare -f Zibri | head --lines=-1 | tail --lines=+3 | sed -e "s/^\s\s\s\s//;s/$rstr/    /"
}

which takes care also of heredocs.

> ,

Found this http://www.linux-kheops.com/doc/perl/perl-aubert/fmt.script .

Very nice, only one thing i took out is the [...]->test substitution.

[Sep 16, 2019] A command-line HTML pretty-printer Making messy HTML readable - Stack Overflow

Notable quotes:
"... Have a look at the HTML Tidy Project: http://www.html-tidy.org/ ..."
Sep 16, 2019 | stackoverflow.com

nisetama ,Aug 12 at 10:33

I'm looking for recommendations for HTML pretty printers which fulfill the following requirements:

> ,

Have a look at the HTML Tidy Project: http://www.html-tidy.org/

The granddaddy of HTML tools, with support for modern standards.

There used to be a fork called tidy-html5 which since became the official thing. Here is its GitHub repository .

Tidy is a console application for Mac OS X, Linux, Windows, UNIX, and more. It corrects and cleans up HTML and XML documents by fixing markup errors and upgrading legacy code to modern standards.

For your needs, here is the command line to call Tidy:

[Sep 11, 2019] string - Extract substring in Bash - Stack Overflow

Sep 11, 2019 | stackoverflow.com

Jeff ,May 8 at 18:30

Given a filename in the form someletters_12345_moreleters.ext , I want to extract the 5 digits and put them into a variable.

So to emphasize the point, I have a filename with x number of characters then a five digit sequence surrounded by a single underscore on either side then another set of x number of characters. I want to take the 5 digit number and put that into a variable.

I am very interested in the number of different ways that this can be accomplished.

Berek Bryan ,Jan 24, 2017 at 9:30

Use cut :
echo 'someletters_12345_moreleters.ext' | cut -d'_' -f 2

More generic:

INPUT='someletters_12345_moreleters.ext'
SUBSTRING=$(echo $INPUT| cut -d'_' -f 2)
echo $SUBSTRING

JB. ,Jan 6, 2015 at 10:13

If x is constant, the following parameter expansion performs substring extraction:
b=${a:12:5}

where 12 is the offset (zero-based) and 5 is the length

If the underscores around the digits are the only ones in the input, you can strip off the prefix and suffix (respectively) in two steps:

tmp=${a#*_}   # remove prefix ending in "_"
b=${tmp%_*}   # remove suffix starting with "_"

If there are other underscores, it's probably feasible anyway, albeit more tricky. If anyone knows how to perform both expansions in a single expression, I'd like to know too.

Both solutions presented are pure bash, with no process spawning involved, hence very fast.

A Sahra ,Mar 16, 2017 at 6:27

Generic solution where the number can be anywhere in the filename, using the first of such sequences:
number=$(echo $filename | egrep -o '[[:digit:]]{5}' | head -n1)

Another solution to extract exactly a part of a variable:

number=${filename:offset:length}

If your filename always have the format stuff_digits_... you can use awk:

number=$(echo $filename | awk -F _ '{ print $2 }')

Yet another solution to remove everything except digits, use

number=$(echo $filename | tr -cd '[[:digit:]]')

sshow ,Jul 27, 2017 at 17:22

In case someone wants more rigorous information, you can also search it in man bash like this
$ man bash [press return key]
/substring  [press return key]
[press "n" key]
[press "n" key]
[press "n" key]
[press "n" key]

Result:

${parameter:offset}
       ${parameter:offset:length}
              Substring Expansion.  Expands to  up  to  length  characters  of
              parameter  starting  at  the  character specified by offset.  If
              length is omitted, expands to the substring of parameter  start‐
              ing at the character specified by offset.  length and offset are
              arithmetic expressions (see ARITHMETIC  EVALUATION  below).   If
              offset  evaluates  to a number less than zero, the value is used
              as an offset from the end of the value of parameter.  Arithmetic
              expressions  starting  with  a - must be separated by whitespace
              from the preceding : to be distinguished from  the  Use  Default
              Values  expansion.   If  length  evaluates to a number less than
              zero, and parameter is not @ and not an indexed  or  associative
              array,  it is interpreted as an offset from the end of the value
              of parameter rather than a number of characters, and the  expan‐
              sion is the characters between the two offsets.  If parameter is
              @, the result is length positional parameters beginning at  off‐
              set.   If parameter is an indexed array name subscripted by @ or
              *, the result is the length members of the array beginning  with
              ${parameter[offset]}.   A  negative  offset is taken relative to
              one greater than the maximum index of the specified array.  Sub‐
              string  expansion applied to an associative array produces unde‐
              fined results.  Note that a negative offset  must  be  separated
              from  the  colon  by  at least one space to avoid being confused
              with the :- expansion.  Substring indexing is zero-based  unless
              the  positional  parameters are used, in which case the indexing
              starts at 1 by default.  If offset  is  0,  and  the  positional
              parameters are used, $0 is prefixed to the list.

Aleksandr Levchuk ,Aug 29, 2011 at 5:51

Building on jor's answer (which doesn't work for me):
substring=$(expr "$filename" : '.*_\([^_]*\)_.*')

kayn ,Oct 5, 2015 at 8:48

I'm surprised this pure bash solution didn't come up:
a="someletters_12345_moreleters.ext"
IFS="_"
set $a
echo $2
# prints 12345

You probably want to reset IFS to what value it was before, or unset IFS afterwards!

zebediah49 ,Jun 4 at 17:31

Here's how i'd do it:
FN=someletters_12345_moreleters.ext
[[ ${FN} =~ _([[:digit:]]{5})_ ]] && NUM=${BASH_REMATCH[1]}

Note: the above is a regular expression and is restricted to your specific scenario of five digits surrounded by underscores. Change the regular expression if you need different matching.

TranslucentCloud ,Jun 16, 2014 at 13:27

Following the requirements

I have a filename with x number of characters then a five digit sequence surrounded by a single underscore on either side then another set of x number of characters. I want to take the 5 digit number and put that into a variable.

I found some grep ways that may be useful:

$ echo "someletters_12345_moreleters.ext" | grep -Eo "[[:digit:]]+" 
12345

or better

$ echo "someletters_12345_moreleters.ext" | grep -Eo "[[:digit:]]{5}" 
12345

And then with -Po syntax:

$ echo "someletters_12345_moreleters.ext" | grep -Po '(?<=_)\d+' 
12345

Or if you want to make it fit exactly 5 characters:

$ echo "someletters_12345_moreleters.ext" | grep -Po '(?<=_)\d{5}' 
12345

Finally, to make it be stored in a variable it is just need to use the var=$(command) syntax.

Darron ,Jan 9, 2009 at 16:13

Without any sub-processes you can:
shopt -s extglob
front=${input%%_+([a-zA-Z]).*}
digits=${front##+([a-zA-Z])_}

A very small variant of this will also work in ksh93.

user2350426

add a comment ,Aug 5, 2014 at 8:11
If we focus in the concept of:
"A run of (one or several) digits"

We could use several external tools to extract the numbers.
We could quite easily erase all other characters, either sed or tr:

name='someletters_12345_moreleters.ext'

echo $name | sed 's/[^0-9]*//g'    # 12345
echo $name | tr -c -d 0-9          # 12345

But if $name contains several runs of numbers, the above will fail:

If "name=someletters_12345_moreleters_323_end.ext", then:

echo $name | sed 's/[^0-9]*//g'    # 12345323
echo $name | tr -c -d 0-9          # 12345323

We need to use regular expresions (regex).
To select only the first run (12345 not 323) in sed and perl:

echo $name | sed 's/[^0-9]*\([0-9]\{1,\}\).*$/\1/'
perl -e 'my $name='$name';my ($num)=$name=~/(\d+)/;print "$num\n";'

But we could as well do it directly in bash (1) :

regex=[^0-9]*([0-9]{1,}).*$; \
[[ $name =~ $regex ]] && echo ${BASH_REMATCH[1]}

This allows us to extract the FIRST run of digits of any length
surrounded by any other text/characters.

Note : regex=[^0-9]*([0-9]{5,5}).*$; will match only exactly 5 digit runs. :-)

(1) : faster than calling an external tool for each short texts. Not faster than doing all processing inside sed or awk for large files.

codist ,May 6, 2011 at 12:50

Here's a prefix-suffix solution (similar to the solutions given by JB and Darron) that matches the first block of digits and does not depend on the surrounding underscores:
str='someletters_12345_morele34ters.ext'
s1="${str#"${str%%[[:digit:]]*}"}"   # strip off non-digit prefix from str
s2="${s1%%[^[:digit:]]*}"            # strip off non-digit suffix from s1
echo "$s2"                           # 12345

Campa ,Oct 21, 2016 at 8:12

I love sed 's capability to deal with regex groups:
> var="someletters_12345_moreletters.ext"
> digits=$( echo $var | sed "s/.*_\([0-9]\+\).*/\1/p" -n )
> echo $digits
12345

A slightly more general option would be not to assume that you have an underscore _ marking the start of your digits sequence, hence for instance stripping off all non-numbers you get before your sequence: s/[^0-9]\+\([0-9]\+\).*/\1/p .


> man sed | grep s/regexp/replacement -A 2
s/regexp/replacement/
    Attempt to match regexp against the pattern space.  If successful, replace that portion matched with replacement.  The replacement may contain the special  character  &  to
    refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding matching sub-expressions in the regexp.

More on this, in case you're not too confident with regexps:

All escapes \ are there to make sed 's regexp processing work.

Dan Dascalescu ,May 8 at 18:28

Given test.txt is a file containing "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cut -b19-20 test.txt > test1.txt # This will extract chars 19 & 20 "ST" 
while read -r; do;
> x=$REPLY
> done < test1.txt
echo $x
ST

Alex Raj Kaliamoorthy ,Jul 29, 2016 at 7:41

My answer will have more control on what you want out of your string. Here is the code on how you can extract 12345 out of your string
str="someletters_12345_moreleters.ext"
str=${str#*_}
str=${str%_more*}
echo $str

This will be more efficient if you want to extract something that has any chars like abc or any special characters like _ or - . For example: If your string is like this and you want everything that is after someletters_ and before _moreleters.ext :

str="someletters_123-45-24a&13b-1_moreleters.ext"

With my code you can mention what exactly you want. Explanation:

#* It will remove the preceding string including the matching key. Here the key we mentioned is _ % It will remove the following string including the matching key. Here the key we mentioned is '_more*'

Do some experiments yourself and you would find this interesting.

Dan Dascalescu ,May 8 at 18:27

similar to substr('abcdefg', 2-1, 3) in php:
echo 'abcdefg'|tail -c +2|head -c 3

olibre ,Nov 25, 2015 at 14:50

Ok, here goes pure Parameter Substitution with an empty string. Caveat is that I have defined someletters and moreletters as only characters. If they are alphanumeric, this will not work as it is.
filename=someletters_12345_moreletters.ext
substring=${filename//@(+([a-z])_|_+([a-z]).*)}
echo $substring
12345

gniourf_gniourf ,Jun 4 at 17:33

There's also the bash builtin 'expr' command:
INPUT="someletters_12345_moreleters.ext"  
SUBSTRING=`expr match "$INPUT" '.*_\([[:digit:]]*\)_.*' `  
echo $SUBSTRING

russell ,Aug 1, 2013 at 8:12

A little late, but I just ran across this problem and found the following:
host:/tmp$ asd=someletters_12345_moreleters.ext 
host:/tmp$ echo `expr $asd : '.*_\(.*\)_'`
12345
host:/tmp$

I used it to get millisecond resolution on an embedded system that does not have %N for date:

set `grep "now at" /proc/timer_list`
nano=$3
fraction=`expr $nano : '.*\(...\)......'`
$debug nano is $nano, fraction is $fraction

> ,Aug 5, 2018 at 17:13

A bash solution:
IFS="_" read -r x digs x <<<'someletters_12345_moreleters.ext'

This will clobber a variable called x . The var x could be changed to the var _ .

input='someletters_12345_moreleters.ext'
IFS="_" read -r _ digs _ <<<"$input"

[Sep 10, 2019] How do I avoid an uninitialized value

Sep 10, 2019 | stackoverflow.com

marto ,Jul 15, 2011 at 16:52

I use this scrub function to clean up output from other functions.
#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;

my %h = (
    a => 1,
    b => 1
    );

print scrub($h{c});

sub scrub {
    my $a = shift;

    return ($a eq '' or $a eq '~' or not defined $a) ? -1 : $a;
}

The problem occurs when I also would like to handle the case, where the key in a hash doesn't exist, which is shown in the example with scrub($h{c}) .

What change should be make to scrub so it can handle this case?

Sandra Schlichting ,Jun 22, 2017 at 19:00

You're checking whether $a eq '' before checking whether it's defined, hence the warning "Use of uninitialized value in string eq". Simply change the order of things in the conditional:
return (!defined($a) or $a eq '' or $a eq '~') ? -1 : $a;

As soon as anything in the chain of 'or's matches, Perl will stop processing the conditional, thus avoiding the erroneous attempt to compare undef to a string.

Sandra Schlichting ,Jul 14, 2011 at 14:34

In scrub it is too late to check, if the hash has an entry for key key . scrub() only sees a scalar, which is undef , if the hash key does not exist. But a hash could have an entry with the value undef also, like this:
my %h = (
 a => 1,
 b => 1,
 c => undef
);

So I suggest to check for hash entries with the exists function.

[Sep 10, 2019] How do I check if a Perl scalar variable has been initialized - Stack Overflow

Sep 10, 2019 | stackoverflow.com

How do I check if a Perl scalar variable has been initialized? Ask Question Asked 8 years, 11 months ago Active 3 years ago Viewed 49k times 33 10


brian d foy ,Sep 18, 2010 at 13:53

Is the following the best way to check if a scalar variable is initialized in Perl, using defined ?
my $var;

if (cond) {
    $var = "string1";
}

# Is this the correct way?
if (defined $var) {
    ...
}

mob ,Sep 25, 2010 at 21:35

Perl doesn't offer a way to check whether or not a variable has been initialized.

However, scalar variables that haven't been explicitly initialized with some value happen to have the value of undef by default. You are right about defined being the right way to check whether or not a variable has a value of undef .

There's several other ways tho. If you want to assign to the variable if it's undef , which your example code seems to indicate, you could, for example, use perl's defined-or operator:

$var //= 'a default value';

vol7ron ,Sep 17, 2010 at 23:17

It depends on what you're trying to do. The proper C way to do things is to initialize variables when they are declared; however, Perl is not C , so one of the following may be what you want:
  1)   $var = "foo" unless defined $var;      # set default after the fact
  2)   $var = defined $var? $var : {...};     # ternary operation
  3)   {...} if !(defined $var);              # another way to write 1)
  4)   $var = $var || "foo";                  # set to $var unless it's falsy, in which case set to 'foo'
  5)   $var ||= "foo";                        # retain value of $var unless it's falsy, in which case set to 'foo' (same as previous line)
  6)   $var = $var // "foo";                  # set to $var unless it's undefined, in which case set to 'foo'
  7)   $var //= "foo";                        # 5.10+ ; retain value of $var unless it's undefined, in which case set to 'foo' (same as previous line)


C way of doing things ( not recommended ):

# initialize the variable to a default value during declaration
#   then test against that value when you want to see if it's been changed
my $var = "foo";
{...}
if ($var eq "foo"){
   ... # do something
} else {
   ... # do something else
}

Another long-winded way of doing this is to create a class and a flag when the variable's been changed, which is unnecessary.

Axeman ,Sep 17, 2010 at 20:39

If you don't care whether or not it's empty, it is. Otherwise you can check
if ( length( $str || '' )) {}

swilliams ,Sep 17, 2010 at 20:53

It depends on what you plan on doing with the variable whether or not it is defined; as of Perl 5.10, you can do this (from perl51000delta ):

A new operator // (defined-or) has been implemented. The following expression:

 $a // $b

is merely equivalent to

defined $a ? $a : $b

and the statement

$c //= $d;

can now be used instead of

$c = $d unless defined $c;

rafl ,Jun 24, 2012 at 7:53

'defined' will return true if a variable has a real value.

As an aside, in a hash, this can be true:

if(exists $h{$e} && !defined $h{$e})

[Sep 10, 2019] logging - Perl - Output the log files - Stack Overflow

Aug 27, 2015 | stackoverflow.com

Perl - Output the log files Ask Question Asked 4 years ago Active 4 years ago Viewed 3k times 1 2


Arunesh Singh ,Aug 27, 2015 at 8:53

I have created a perl that telnet to multiple switches. I would like to check if telnet functions properly by telneting the switch.

This is my code to telnet to the switches:

#!/usr/bin/perl
use warnings;
use Net::Cisco;

open( OUTPUT, ">log.txt" );
open( SWITCHIP, "ip.txt" ) or die "couldn't open ip.txt";

my $count = 0;

while (<SWITCHIP>) {
    chomp($_);
    my $switch = $_;
    my $tl     = 0;
    my $t      = Net::Telnet::Cisco->new(
        Host => $switch,
        Prompt =>
            '/(?m:^(?:[\w.\/]+\:)?[\w.-]+\s?(?:\(config[^\)]*\))?\s?[\$#>]\s?(?:\(enable\))?\s*$)/',
        Timeout => 5,
        Errmode => 'return'
    ) or $tl = 1;

    my @output = ();
    if ( $tl != 1 ) {
        print "$switch Telnet success\n";
    }
    else {
        my $telnetstat = "Telnet Failed";
        print "$switch $telnetstat\n";
    }
    close(OUTPUT);
    $count++;
}

This is my output status after I was testing 7 switches:

10.xxx.3.17 Telnet success
10.xxx.10.12 Telnet success
10.xxx.136.10 Telnet success
10.xxx.136.12 Telnet success
10.xxx.188.188 Telnet Failed
10.xxx.136.13 Telnet success

I would like to convert the telnet result as log file.
How to separate successful and failed telnet results by using perl?

Danny Luk ,Aug 28, 2015 at 8:40

Please Try the following
#!/usr/bin/perl
use warnings;
use Net::Cisco;
################################### S
open( OUTPUTS, ">log_Success.txt" );
open( OUTPUTF, ">log_Fail.txt" );
################################### E
open( SWITCHIP, "ip.txt" ) or die "couldn't open ip.txt";

my $count = 0;

while (<SWITCHIP>) {
    chomp($_);
    my $switch = $_;
    my $tl     = 0;
    my $t      = Net::Telnet::Cisco->new(
        Host => $switch,
        Prompt =>
            '/(?m:^(?:[\w.\/]+\:)?[\w.-]+\s?(?:\(config[^\)]*\))?\s?[\$#>]\s?(?:\(enable\))?\s*$)/',
        Timeout => 5,
        Errmode => 'return'
    ) or $tl = 1;

    my @output = ();
################################### S
    if ( $tl != 1 ) {
        print "$switch Telnet success\n"; # for printing it in screen
        print OUTPUTS "$switch Telnet success\n"; # it will print it in the log_Success.txt
    }
    else {
        my $telnetstat = "Telnet Failed";
        print "$switch $telnetstat\n"; # for printing it in screen
        print OUTPUTF "$switch $telnetstat\n"; # it will print it in the log_Fail.txt
    }
################################### E
    $count++;
}
################################### S
close(SWITCHIP);
close(OUTPUTS);
close(OUTPUTF);
################################### E

Danny Luk ,Aug 28, 2015 at 8:39

In print statement after print just write the filehandle name which is OUTPUT in your code:
print OUTPUT "$switch Telnet success\n";

and

print OUTPUT "$switch $telnetstat\n";

A side note: always use a lexical filehandle and three arguments with error handling to open a file. This line open(OUTPUT, ">log.txt"); you can write like this:

open my $fhout, ">", "log.txt" or die $!;

Sobrique ,Aug 28, 2015 at 8:39

Use Sys::Syslog to write log messages.

But since you're opening a log.txt file with the handle OUTPUT , just change your two print statements to have OUTPUT as the first argument and the string as the next (without a comma).

my $telnetstat;
if($tl != 1) {
  $telnetstat = "Telnet success";
} else {
  $telnetstat = "Telnet Failed";
}
print OUTPUT "$switch $telnetstat\n";

# Or the shorter ternary operator line for all the above:
print OUTPUT $swtich . (!$tl ? " Telnet success\n" : " Telnet failed\n");

You might consider moving close to an END block:

END {
  close(OUTPUT);
}

Not only because it's in your while loop.

[Sep 08, 2019] How to replace spaces in file names using a bash script

Sep 08, 2019 | stackoverflow.com

Ask Question Asked 9 years, 4 months ago Active 2 months ago Viewed 226k times 238 127


Mark Byers ,Apr 25, 2010 at 19:20

Can anyone recommend a safe solution to recursively replace spaces with underscores in file and directory names starting from a given root directory? For example:
$ tree
.
|-- a dir
|   `-- file with spaces.txt
`-- b dir
    |-- another file with spaces.txt
    `-- yet another file with spaces.pdf

becomes:

$ tree
.
|-- a_dir
|   `-- file_with_spaces.txt
`-- b_dir
    |-- another_file_with_spaces.txt
    `-- yet_another_file_with_spaces.pdf

Jürgen Hötzel ,Nov 4, 2015 at 3:03

Use rename (aka prename ) which is a Perl script which may be on your system already. Do it in two steps:
find -name "* *" -type d | rename 's/ /_/g'    # do the directories first
find -name "* *" -type f | rename 's/ /_/g'

Based on Jürgen's answer and able to handle multiple layers of files and directories in a single bound using the "Revision 1.5 1998/12/18 16:16:31 rmb1" version of /usr/bin/rename (a Perl script):

find /tmp/ -depth -name "* *" -execdir rename 's/ /_/g' "{}" \;

oevna ,Jan 1, 2016 at 8:25

I use:
for f in *\ *; do mv "$f" "${f// /_}"; done

Though it's not recursive, it's quite fast and simple. I'm sure someone here could update it to be recursive.

The ${f// /_} part utilizes bash's parameter expansion mechanism to replace a pattern within a parameter with supplied string. The relevant syntax is ${parameter/pattern/string} . See: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html or http://wiki.bash-hackers.org/syntax/pe .

armandino ,Dec 3, 2013 at 20:51

find . -depth -name '* *' \
| while IFS= read -r f ; do mv -i "$f" "$(dirname "$f")/$(basename "$f"|tr ' ' _)" ; done

failed to get it right at first, because I didn't think of directories.

Edmund Elmer ,Jul 3 at 7:12

you can use detox by Doug Harple
detox -r <folder>

Dennis Williamson ,Mar 22, 2012 at 20:33

A find/rename solution. rename is part of util-linux.

You need to descend depth first, because a whitespace filename can be part of a whitespace directory:

find /tmp/ -depth -name "* *" -execdir rename " " "_" "{}" ";"

armandino ,Apr 26, 2010 at 11:49

bash 4.0
#!/bin/bash
shopt -s globstar
for file in **/*\ *
do 
    mv "$file" "${file// /_}"       
done

Itamar ,Jan 31, 2013 at 21:27

you can use this:
    find . -name '* *' | while read fname 

do
        new_fname=`echo $fname | tr " " "_"`

        if [ -e $new_fname ]
        then
                echo "File $new_fname already exists. Not replacing $fname"
        else
                echo "Creating new file $new_fname to replace $fname"
                mv "$fname" $new_fname
        fi
done

yabt ,Apr 26, 2010 at 14:54

Here's a (quite verbose) find -exec solution which writes "file already exists" warnings to stderr:
function trspace() {
   declare dir name bname dname newname replace_char
   [ $# -lt 1 -o $# -gt 2 ] && { echo "usage: trspace dir char"; return 1; }
   dir="${1}"
   replace_char="${2:-_}"
   find "${dir}" -xdev -depth -name $'*[ \t\r\n\v\f]*' -exec bash -c '
      for ((i=1; i<=$#; i++)); do
         name="${@:i:1}"
         dname="${name%/*}"
         bname="${name##*/}"
         newname="${dname}/${bname//[[:space:]]/${0}}"
         if [[ -e "${newname}" ]]; then
            echo "Warning: file already exists: ${newname}" 1>&2
         else
            mv "${name}" "${newname}"
         fi
      done
  ' "${replace_char}" '{}' +
}

trspace rootdir _

degi ,Aug 8, 2011 at 9:10

This one does a little bit more. I use it to rename my downloaded torrents (no special characters (non-ASCII), spaces, multiple dots, etc.).
#!/usr/bin/perl

&rena(`find . -type d`);
&rena(`find . -type f`);

sub rena
{
    ($elems)=@_;
    @t=split /\n/,$elems;

    for $e (@t)
    {
    $_=$e;
    # remove ./ of find
    s/^\.\///;
    # non ascii transliterate
    tr [\200-\377][_];
    tr [\000-\40][_];
    # special characters we do not want in paths
    s/[ \-\,\;\?\+\'\"\!\[\]\(\)\@\#]/_/g;
    # multiple dots except for extension
    while (/\..*\./)
    {
        s/\./_/;
    }
    # only one _ consecutive
    s/_+/_/g;
    next if ($_ eq $e ) or ("./$_" eq $e);
    print "$e -> $_\n";
    rename ($e,$_);
    }
}

Junyeop Lee ,Apr 10, 2018 at 9:44

Recursive version of Naidim's Answers.
find . -name "* *" | awk '{ print length, $0 }' | sort -nr -s | cut -d" " -f2- | while read f; do base=$(basename "$f"); newbase="${base// /_}"; mv "$(dirname "$f")/$(basename "$f")" "$(dirname "$f")/$newbase"; done

ghoti ,Dec 5, 2016 at 21:16

I found around this script, it may be interesting :)
 IFS=$'\n';for f in `find .`; do file=$(echo $f | tr [:blank:] '_'); [ -e $f ] && [ ! -e $file ] && mv "$f" $file;done;unset IFS

ghoti ,Dec 5, 2016 at 21:17

Here's a reasonably sized bash script solution
#!/bin/bash
(
IFS=$'\n'
    for y in $(ls $1)
      do
         mv $1/`echo $y | sed 's/ /\\ /g'` $1/`echo "$y" | sed 's/ /_/g'`
      done
)

user1060059 ,Nov 22, 2011 at 15:15

This only finds files inside the current directory and renames them . I have this aliased.

find ./ -name "* *" -type f -d 1 | perl -ple '$file = $_; $file =~ s/\s+/_/g; rename($_, $file);

Hongtao ,Sep 26, 2014 at 19:30

I just make one for my own purpose. You may can use it as reference.
#!/bin/bash
cd /vzwhome/c0cheh1/dev_source/UB_14_8
for file in *
do
    echo $file
    cd "/vzwhome/c0cheh1/dev_source/UB_14_8/$file/Configuration/$file"
    echo "==> `pwd`"
    for subfile in *\ *; do [ -d "$subfile" ] && ( mv "$subfile" "$(echo $subfile | sed -e 's/ /_/g')" ); done
    ls
    cd /vzwhome/c0cheh1/dev_source/UB_14_8
done

Marcos Jean Sampaio ,Dec 5, 2016 at 20:56

For files in folder named /files
for i in `IFS="";find /files -name *\ *`
do
   echo $i
done > /tmp/list


while read line
do
   mv "$line" `echo $line | sed 's/ /_/g'`
done < /tmp/list

rm /tmp/list

Muhammad Annaqeeb ,Sep 4, 2017 at 11:03

For those struggling through this using macOS, first install all the tools:
 brew install tree findutils rename

Then when needed to rename, make an alias for GNU find (gfind) as find. Then run the code of @Michel Krelin:

alias find=gfind 
find . -depth -name '* *' \
| while IFS= read -r f ; do mv -i "$f" "$(dirname "$f")/$(basename "$f"|tr ' ' _)" ; done

[Sep 07, 2019] As soon as you stop writing code on a regular basis you stop being a programmer. You lose you qualification very quickly. That's a typical tragedy of talented programmers who became mediocre managers or, worse, theoretical computer scientists

Programming skills are somewhat similar to the skills of people who play violin or piano. As soon a you stop playing violin or piano still start to evaporate. First slowly, then quicker. In two yours you probably will lose 80%.
Notable quotes:
"... I happened to look the other day. I wrote 35 programs in January, and 28 or 29 programs in February. These are small programs, but I have a compulsion. I love to write programs and put things into it. ..."
Sep 07, 2019 | archive.computerhistory.org

Dijkstra said he was proud to be a programmer. Unfortunately he changed his attitude completely, and I think he wrote his last computer program in the 1980s. At this conference I went to in 1967 about simulation language, Chris Strachey was going around asking everybody at the conference what was the last computer program you wrote. This was 1967. Some of the people said, "I've never written a computer program." Others would say, "Oh yeah, here's what I did last week." I asked Edsger this question when I visited him in Texas in the 90s and he said, "Don, I write programs now with pencil and paper, and I execute them in my head." He finds that a good enough discipline.

I think he was mistaken on that. He taught me a lot of things, but I really think that if he had continued... One of Dijkstra's greatest strengths was that he felt a strong sense of aesthetics, and he didn't want to compromise his notions of beauty. They were so intense that when he visited me in the 1960s, I had just come to Stanford. I remember the conversation we had. It was in the first apartment, our little rented house, before we had electricity in the house.

We were sitting there in the dark, and he was telling me how he had just learned about the specifications of the IBM System/360, and it made him so ill that his heart was actually starting to flutter.

He intensely disliked things that he didn't consider clean to work with. So I can see that he would have distaste for the languages that he had to work with on real computers. My reaction to that was to design my own language, and then make Pascal so that it would work well for me in those days. But his response was to do everything only intellectually.

So, programming.

I happened to look the other day. I wrote 35 programs in January, and 28 or 29 programs in February. These are small programs, but I have a compulsion. I love to write programs and put things into it. I think of a question that I want to answer, or I have part of my book where I want to present something. But I can't just present it by reading about it in a book. As I code it, it all becomes clear in my head. It's just the discipline. The fact that I have to translate my knowledge of this method into something that the machine is going to understand just forces me to make that crystal-clear in my head. Then I can explain it to somebody else infinitely better. The exposition is always better if I've implemented it, even though it's going to take me more time.

[Sep 07, 2019] Knuth about computer science and money: At that point I made the decision in my life that I wasn't going to optimize my income;

Sep 07, 2019 | archive.computerhistory.org

So I had a programming hat when I was outside of Cal Tech, and at Cal Tech I am a mathematician taking my grad studies. A startup company, called Green Tree Corporation because green is the color of money, came to me and said, "Don, name your price. Write compilers for us and we will take care of finding computers for you to debug them on, and assistance for you to do your work. Name your price." I said, "Oh, okay. $100,000.", assuming that this was In that era this was not quite at Bill Gate's level today, but it was sort of out there.

The guy didn't blink. He said, "Okay." I didn't really blink either. I said, "Well, I'm not going to do it. I just thought this was an impossible number."

At that point I made the decision in my life that I wasn't going to optimize my income; I was really going to do what I thought I could do for well, I don't know. If you ask me what makes me most happy, number one would be somebody saying "I learned something from you". Number two would be somebody saying "I used your software". But number infinity would be Well, no. Number infinity minus one would be "I bought your book". It's not as good as "I read your book", you know. Then there is "I bought your software"; that was not in my own personal value. So that decision came up. I kept up with the literature about compilers. The Communications of the ACM was where the action was. I also worked with people on trying to debug the ALGOL language, which had problems with it. I published a few papers, like "The Remaining Trouble Spots in ALGOL 60" was one of the papers that I worked on. I chaired a committee called "Smallgol" which was to find a subset of ALGOL that would work on small computers. I was active in programming languages.

[Sep 07, 2019] Knuth: maybe 1 in 50 people have the "computer scientist's" type of intellect

Sep 07, 2019 | conservancy.umn.edu

Frana: You have made the comment several times that maybe 1 in 50 people have the "computer scientist's mind." Knuth: Yes. Frana: I am wondering if a large number of those people are trained professional librarians? [laughter] There is some strangeness there. But can you pinpoint what it is about the mind of the computer scientist that is....

Knuth: That is different?

Frana: What are the characteristics?

Knuth: Two things: one is the ability to deal with non-uniform structure, where you have case one, case two, case three, case four. Or that you have a model of something where the first component is integer, the next component is a Boolean, and the next component is a real number, or something like that, you know, non-uniform structure. To deal fluently with those kinds of entities, which is not typical in other branches of mathematics, is critical. And the other characteristic ability is to shift levels quickly, from looking at something in the large to looking at something in the small, and many levels in between, jumping from one level of abstraction to another. You know that, when you are adding one to some number, that you are actually getting closer to some overarching goal. These skills, being able to deal with nonuniform objects and to see through things from the top level to the bottom level, these are very essential to computer programming, it seems to me. But maybe I am fooling myself because I am too close to it.

Frana: It is the hardest thing to really understand that which you are existing within.

Knuth: Yes.

[Sep 07, 2019] Knuth: I can be a writer, who tries to organize other people's ideas into some kind of a more coherent structure so that it is easier to put things together

Sep 07, 2019 | conservancy.umn.edu

Knuth: I can be a writer, who tries to organize other people's ideas into some kind of a more coherent structure so that it is easier to put things together. I can see that I could be viewed as a scholar that does his best to check out sources of material, so that people get credit where it is due. And to check facts over, not just to look at the abstract of something, but to see what the methods were that did it and to fill in holes if necessary. I look at my role as being able to understand the motivations and terminology of one group of specialists and boil it down to a certain extent so that people in other parts of the field can use it. I try to listen to the theoreticians and select what they have done that is important to the programmer on the street; to remove technical jargon when possible.

But I have never been good at any kind of a role that would be making policy, or advising people on strategies, or what to do. I have always been best at refining things that are there and bringing order out of chaos. I sometimes raise new ideas that might stimulate people, but not really in a way that would be in any way controlling the flow. The only time I have ever advocated something strongly was with literate programming; but I do this always with the caveat that it works for me, not knowing if it would work for anybody else.

When I work with a system that I have created myself, I can always change it if I don't like it. But everybody who works with my system has to work with what I give them. So I am not able to judge my own stuff impartially. So anyway, I have always felt bad about if anyone says, 'Don, please forecast the future,'...

[Sep 06, 2019] Knuth: Programming and architecture are interrelated and it is impossible to create good architecure wthout actually programming at least of a prototype

Notable quotes:
"... When you're writing a document for a human being to understand, the human being will look at it and nod his head and say, "Yeah, this makes sense." But then there's all kinds of ambiguities and vagueness that you don't realize until you try to put it into a computer. Then all of a sudden, almost every five minutes as you're writing the code, a question comes up that wasn't addressed in the specification. "What if this combination occurs?" ..."
"... When you're faced with implementation, a person who has been delegated this job of working from a design would have to say, "Well hmm, I don't know what the designer meant by this." ..."
Sep 06, 2019 | archive.computerhistory.org

...I showed the second version of this design to two of my graduate students, and I said, "Okay, implement this, please, this summer. That's your summer job." I thought I had specified a language. I had to go away. I spent several weeks in China during the summer of 1977, and I had various other obligations. I assumed that when I got back from my summer trips, I would be able to play around with TeX and refine it a little bit. To my amazement, the students, who were outstanding students, had not competed [it]. They had a system that was able to do about three lines of TeX. I thought, "My goodness, what's going on? I thought these were good students." Well afterwards I changed my attitude to saying, "Boy, they accomplished a miracle."

Because going from my specification, which I thought was complete, they really had an impossible task, and they had succeeded wonderfully with it. These students, by the way, [were] Michael Plass, who has gone on to be the brains behind almost all of Xerox's Docutech software and all kind of things that are inside of typesetting devices now, and Frank Liang, one of the key people for Microsoft Word.

He did important mathematical things as well as his hyphenation methods which are quite used in all languages now. These guys were actually doing great work, but I was amazed that they couldn't do what I thought was just sort of a routine task. Then I became a programmer in earnest, where I had to do it. The reason is when you're doing programming, you have to explain something to a computer, which is dumb.

When you're writing a document for a human being to understand, the human being will look at it and nod his head and say, "Yeah, this makes sense." But then there's all kinds of ambiguities and vagueness that you don't realize until you try to put it into a computer. Then all of a sudden, almost every five minutes as you're writing the code, a question comes up that wasn't addressed in the specification. "What if this combination occurs?"

It just didn't occur to the person writing the design specification. When you're faced with implementation, a person who has been delegated this job of working from a design would have to say, "Well hmm, I don't know what the designer meant by this."

If I hadn't been in China they would've scheduled an appointment with me and stopped their programming for a day. Then they would come in at the designated hour and we would talk. They would take 15 minutes to present to me what the problem was, and then I would think about it for a while, and then I'd say, "Oh yeah, do this. " Then they would go home and they would write code for another five minutes and they'd have to schedule another appointment.

I'm probably exaggerating, but this is why I think Bob Floyd's Chiron compiler never got going. Bob worked many years on a beautiful idea for a programming language, where he designed a language called Chiron, but he never touched the programming himself. I think this was actually the reason that he had trouble with that project, because it's so hard to do the design unless you're faced with the low-level aspects of it, explaining it to a machine instead of to another person.

Forsythe, I think it was, who said, "People have said traditionally that you don't understand something until you've taught it in a class. The truth is you don't really understand something until you've taught it to a computer, until you've been able to program it." At this level, programming was absolutely important

[Sep 06, 2019] Knuth: No, I stopped going to conferences. It was too discouraging. Computer programming keeps getting harder because more stuff is discovered

Sep 06, 2019 | conservancy.umn.edu

Knuth: No, I stopped going to conferences. It was too discouraging. Computer programming keeps getting harder because more stuff is discovered. I can cope with learning about one new technique per day, but I can't take ten in a day all at once. So conferences are depressing; it means I have so much more work to do. If I hide myself from the truth I am much happier.

[Sep 06, 2019] How TAOCP was hatched

Notable quotes:
"... Also, Addison-Wesley was the people who were asking me to do this book; my favorite textbooks had been published by Addison Wesley. They had done the books that I loved the most as a student. For them to come to me and say, "Would you write a book for us?", and here I am just a secondyear gradate student -- this was a thrill. ..."
"... But in those days, The Art of Computer Programming was very important because I'm thinking of the aesthetical: the whole question of writing programs as something that has artistic aspects in all senses of the word. The one idea is "art" which means artificial, and the other "art" means fine art. All these are long stories, but I've got to cover it fairly quickly. ..."
Sep 06, 2019 | archive.computerhistory.org

Knuth: This is, of course, really the story of my life, because I hope to live long enough to finish it. But I may not, because it's turned out to be such a huge project. I got married in the summer of 1961, after my first year of graduate school. My wife finished college, and I could use the money I had made -- the $5000 on the compiler -- to finance a trip to Europe for our honeymoon.

We had four months of wedded bliss in Southern California, and then a man from Addison-Wesley came to visit me and said "Don, we would like you to write a book about how to write compilers."

The more I thought about it, I decided "Oh yes, I've got this book inside of me."

I sketched out that day -- I still have the sheet of tablet paper on which I wrote -- I sketched out 12 chapters that I thought ought to be in such a book. I told Jill, my wife, "I think I'm going to write a book."

As I say, we had four months of bliss, because the rest of our marriage has all been devoted to this book. Well, we still have had happiness. But really, I wake up every morning and I still haven't finished the book. So I try to -- I have to -- organize the rest of my life around this, as one main unifying theme. The book was supposed to be about how to write a compiler. They had heard about me from one of their editorial advisors, that I knew something about how to do this. The idea appealed to me for two main reasons. One is that I did enjoy writing. In high school I had been editor of the weekly paper. In college I was editor of the science magazine, and I worked on the campus paper as copy editor. And, as I told you, I wrote the manual for that compiler that we wrote. I enjoyed writing, number one.

Also, Addison-Wesley was the people who were asking me to do this book; my favorite textbooks had been published by Addison Wesley. They had done the books that I loved the most as a student. For them to come to me and say, "Would you write a book for us?", and here I am just a secondyear gradate student -- this was a thrill.

Another very important reason at the time was that I knew that there was a great need for a book about compilers, because there were a lot of people who even in 1962 -- this was January of 1962 -- were starting to rediscover the wheel. The knowledge was out there, but it hadn't been explained. The people who had discovered it, though, were scattered all over the world and they didn't know of each other's work either, very much. I had been following it. Everybody I could think of who could write a book about compilers, as far as I could see, they would only give a piece of the fabric. They would slant it to their own view of it. There might be four people who could write about it, but they would write four different books. I could present all four of their viewpoints in what I would think was a balanced way, without any axe to grind, without slanting it towards something that I thought would be misleading to the compiler writer for the future. I considered myself as a journalist, essentially. I could be the expositor, the tech writer, that could do the job that was needed in order to take the work of these brilliant people and make it accessible to the world. That was my motivation. Now, I didn't have much time to spend on it then, I just had this page of paper with 12 chapter headings on it. That's all I could do while I'm a consultant at Burroughs and doing my graduate work. I signed a contract, but they said "We know it'll take you a while." I didn't really begin to have much time to work on it until 1963, my third year of graduate school, as I'm already finishing up on my thesis. In the summer of '62, I guess I should mention, I wrote another compiler. This was for Univac; it was a FORTRAN compiler. I spent the summer, I sold my soul to the devil, I guess you say, for three months in the summer of 1962 to write a FORTRAN compiler. I believe that the salary for that was $15,000, which was much more than an assistant professor. I think assistant professors were getting eight or nine thousand in those days.

Feigenbaum: Well, when I started in 1960 at [University of California] Berkeley, I was getting $7,600 for the nine-month year.

Knuth: Knuth: Yeah, so you see it. I got $15,000 for a summer job in 1962 writing a FORTRAN compiler. One day during that summer I was writing the part of the compiler that looks up identifiers in a hash table. The method that we used is called linear probing. Basically you take the variable name that you want to look up, you scramble it, like you square it or something like this, and that gives you a number between one and, well in those days it would have been between 1 and 1000, and then you look there. If you find it, good; if you don't find it, go to the next place and keep on going until you either get to an empty place, or you find the number you're looking for. It's called linear probing. There was a rumor that one of Professor Feller's students at Princeton had tried to figure out how fast linear probing works and was unable to succeed. This was a new thing for me. It was a case where I was doing programming, but I also had a mathematical problem that would go into my other [job]. My winter job was being a math student, my summer job was writing compilers. There was no mix. These worlds did not intersect at all in my life at that point. So I spent one day during the summer while writing the compiler looking at the mathematics of how fast does linear probing work. I got lucky, and I solved the problem. I figured out some math, and I kept two or three sheets of paper with me and I typed it up. ["Notes on 'Open' Addressing', 7/22/63] I guess that's on the internet now, because this became really the genesis of my main research work, which developed not to be working on compilers, but to be working on what they call analysis of algorithms, which is, have a computer method and find out how good is it quantitatively. I can say, if I got so many things to look up in the table, how long is linear probing going to take. It dawned on me that this was just one of many algorithms that would be important, and each one would lead to a fascinating mathematical problem. This was easily a good lifetime source of rich problems to work on. Here I am then, in the middle of 1962, writing this FORTRAN compiler, and I had one day to do the research and mathematics that changed my life for my future research trends. But now I've gotten off the topic of what your original question was.

Feigenbaum: We were talking about sort of the.. You talked about the embryo of The Art of Computing. The compiler book morphed into The Art of Computer Programming, which became a seven-volume plan.

Knuth: Exactly. Anyway, I'm working on a compiler and I'm thinking about this. But now I'm starting, after I finish this summer job, then I began to do things that were going to be relating to the book. One of the things I knew I had to have in the book was an artificial machine, because I'm writing a compiler book but machines are changing faster than I can write books. I have to have a machine that I'm totally in control of. I invented this machine called MIX, which was typical of the computers of 1962.

In 1963 I wrote a simulator for MIX so that I could write sample programs for it, and I taught a class at Caltech on how to write programs in assembly language for this hypothetical computer. Then I started writing the parts that dealt with sorting problems and searching problems, like the linear probing idea. I began to write those parts, which are part of a compiler, of the book. I had several hundred pages of notes gathering for those chapters for The Art of Computer Programming. Before I graduated, I've already done quite a bit of writing on The Art of Computer Programming.

I met George Forsythe about this time. George was the man who inspired both of us [Knuth and Feigenbaum] to come to Stanford during the '60s. George came down to Southern California for a talk, and he said, "Come up to Stanford. How about joining our faculty?" I said "Oh no, I can't do that. I just got married, and I've got to finish this book first." I said, "I think I'll finish the book next year, and then I can come up [and] start thinking about the rest of my life, but I want to get my book done before my son is born." Well, John is now 40-some years old and I'm not done with the book. Part of my lack of expertise is any good estimation procedure as to how long projects are going to take. I way underestimated how much needed to be written about in this book. Anyway, I started writing the manuscript, and I went merrily along writing pages of things that I thought really needed to be said. Of course, it didn't take long before I had started to discover a few things of my own that weren't in any of the existing literature. I did have an axe to grind. The message that I was presenting was in fact not going to be unbiased at all. It was going to be based on my own particular slant on stuff, and that original reason for why I should write the book became impossible to sustain. But the fact that I had worked on linear probing and solved the problem gave me a new unifying theme for the book. I was going to base it around this idea of analyzing algorithms, and have some quantitative ideas about how good methods were. Not just that they worked, but that they worked well: this method worked 3 times better than this method, or 3.1 times better than this method. Also, at this time I was learning mathematical techniques that I had never been taught in school. I found they were out there, but they just hadn't been emphasized openly, about how to solve problems of this kind.

So my book would also present a different kind of mathematics than was common in the curriculum at the time, that was very relevant to analysis of algorithm. I went to the publishers, I went to Addison Wesley, and said "How about changing the title of the book from 'The Art of Computer Programming' to 'The Analysis of Algorithms'." They said that will never sell; their focus group couldn't buy that one. I'm glad they stuck to the original title, although I'm also glad to see that several books have now come out called "The Analysis of Algorithms", 20 years down the line.

But in those days, The Art of Computer Programming was very important because I'm thinking of the aesthetical: the whole question of writing programs as something that has artistic aspects in all senses of the word. The one idea is "art" which means artificial, and the other "art" means fine art. All these are long stories, but I've got to cover it fairly quickly.

I've got The Art of Computer Programming started out, and I'm working on my 12 chapters. I finish a rough draft of all 12 chapters by, I think it was like 1965. I've got 3,000 pages of notes, including a very good example of what you mentioned about seeing holes in the fabric. One of the most important chapters in the book is parsing: going from somebody's algebraic formula and figuring out the structure of the formula. Just the way I had done in seventh grade finding the structure of English sentences, I had to do this with mathematical sentences.

Chapter ten is all about parsing of context-free language, [which] is what we called it at the time. I covered what people had published about context-free languages and parsing. I got to the end of the chapter and I said, well, you can combine these ideas and these ideas, and all of a sudden you get a unifying thing which goes all the way to the limit. These other ideas had sort of gone partway there. They would say "Oh, if a grammar satisfies this condition, I can do it efficiently." "If a grammar satisfies this condition, I can do it efficiently." But now, all of a sudden, I saw there was a way to say I can find the most general condition that can be done efficiently without looking ahead to the end of the sentence. That you could make a decision on the fly, reading from left to right, about the structure of the thing. That was just a natural outgrowth of seeing the different pieces of the fabric that other people had put together, and writing it into a chapter for the first time. But I felt that this general concept, well, I didn't feel that I had surrounded the concept. I knew that I had it, and I could prove it, and I could check it, but I couldn't really intuit it all in my head. I knew it was right, but it was too hard for me, really, to explain it well.

So I didn't put in The Art of Computer Programming. I thought it was beyond the scope of my book. Textbooks don't have to cover everything when you get to the harder things; then you have to go to the literature. My idea at that time [is] I'm writing this book and I'm thinking it's going to be published very soon, so any little things I discover and put in the book I didn't bother to write a paper and publish in the journal because I figure it'll be in my book pretty soon anyway. Computer science is changing so fast, my book is bound to be obsolete.

It takes a year for it to go through editing, and people drawing the illustrations, and then they have to print it and bind it and so on. I have to be a little bit ahead of the state-of-the-art if my book isn't going to be obsolete when it comes out. So I kept most of the stuff to myself that I had, these little ideas I had been coming up with. But when I got to this idea of left-to-right parsing, I said "Well here's something I don't really understand very well. I'll publish this, let other people figure out what it is, and then they can tell me what I should have said." I published that paper I believe in 1965, at the end of finishing my draft of the chapter, which didn't get as far as that story, LR(k). Well now, textbooks of computer science start with LR(k) and take off from there. But I want to give you an idea of

[Sep 06, 2019] Python vs. Ruby Which is best for web development Opensource.com

Sep 06, 2019 | opensource.com

Python was developed organically in the scientific space as a prototyping language that easily could be translated into C++ if a prototype worked. This happened long before it was first used for web development. Ruby, on the other hand, became a major player specifically because of web development; the Rails framework extended Ruby's popularity with people developing complex websites.

Which programming language best suits your needs? Here is a quick overview of each language to help you choose:

Approach: one best way vs. human-language Python

Python takes a direct approach to programming. Its main goal is to make everything obvious to the programmer. In Python, there is only one "best" way to do something. This philosophy has led to a language strict in layout.

Python's core philosophy consists of three key hierarchical principles:

This regimented philosophy results in Python being eminently readable and easy to learn -- and why Python is great for beginning coders. Python has a big foothold in introductory programming courses . Its syntax is very simple, with little to remember. Because its code structure is explicit, the developer can easily tell where everything comes from, making it relatively easy to debug.

Python's hierarchy of principles is evident in many aspects of the language. Its use of whitespace to do flow control as a core part of the language syntax differs from most other languages, including Ruby. The way you indent code determines the meaning of its action. This use of whitespace is a prime example of Python's "explicit" philosophy, the shape a Python app takes spells out its logic and how the app will act.

Ruby

In contrast to Python, Ruby focuses on "human-language" programming, and its code reads like a verbal language rather than a machine-based one, which many programmers, both beginners and experts, like. Ruby follows the principle of " least astonishment ," and offers myriad ways to do the same thing. These similar methods can have multiple names, which many developers find confusing and frustrating.

Unlike Python, Ruby makes use of "blocks," a first-class object that is treated as a unit within a program. In fact, Ruby takes the concept of OOP (Object-Oriented Programming) to its limit. Everything is an object -- even global variables are actually represented within the ObjectSpace object. Classes and modules are themselves objects, and functions and operators are methods of objects. This ability makes Ruby especially powerful, especially when combined with its other primary strength: functional programming and the use of lambdas.

In addition to blocks and functional programming, Ruby provides programmers with many other features, including fragmentation, hashable and unhashable types, and mutable strings.

Ruby's fans find its elegance to be one of its top selling points. At the same time, Ruby's "magical" features and flexibility can make it very hard to track down bugs.

Communities: stability vs. innovation

Although features and coding philosophy are the primary drivers for choosing a given language, the strength of a developer community also plays an important role. Fortunately, both Python and Ruby boast strong communities.

Python

Python's community already includes a large Linux and academic community and therefore offers many academic use cases in both math and science. That support gives the community a stability and diversity that only grows as Python increasingly is used for web development.

Ruby

However, Ruby's community has focused primarily on web development from the get-go. It tends to innovate more quickly than the Python community, but this innovation also causes more things to break. In addition, while it has gotten more diverse, it has yet to reach the level of diversity that Python has.

Final thoughts

For web development, Ruby has Rails and Python has Django. Both are powerful frameworks, so when it comes to web development, you can't go wrong with either language. Your decision will ultimately come down to your level of experience and your philosophical preferences.

If you plan to focus on building web applications, Ruby is popular and flexible. There is a very strong community built upon it and they are always on the bleeding edge of development.

If you are interested in building web applications and would like to learn a language that's used more generally, try Python. You'll get a diverse community and lots of influence and support from the various industries in which it is used.

Tom Radcliffe - Tom Radcliffe has over 20 years experience in software development and management in both academia and industry. He is a professional engineer (PEO and APEGBC) and holds a PhD in physics from Queen's University at Kingston. Tom brings a passion for quantitative, data-driven processes to ActiveState .

[Sep 03, 2019] bash - How to convert strings like 19-FEB-12 to epoch date in UNIX - Stack Overflow

Feb 11, 2013 | stackoverflow.com

Asked 6 years, 6 months ago Active 2 years, 2 months ago Viewed 53k times 24 4

hellish ,Feb 11, 2013 at 3:45

In UNIX how to convert to epoch milliseconds date strings like:
19-FEB-12
16-FEB-12
05-AUG-09

I need this to compare these dates with the current time on the server.

> ,

To convert a date to seconds since the epoch:
date --date="19-FEB-12" +%s

Current epoch:

date +%s

So, since your dates are in the past:

NOW=`date +%s`
THEN=`date --date="19-FEB-12" +%s`

let DIFF=$NOW-$THEN
echo "The difference is: $DIFF"

Using BSD's date command, you would need

$ date -j -f "%d-%B-%y" 19-FEB-12 +%s

Differences from GNU date :

  1. -j prevents date from trying to set the clock
  2. The input format must be explicitly set with -f
  3. The input date is a regular argument, not an option (viz. -d )
  4. When no time is specified with the date, use the current time instead of midnight.

[Sep 03, 2019] command line - How do I convert an epoch timestamp to a human readable format on the cli - Unix Linux Stack Exchange

Sep 03, 2019 | unix.stackexchange.com

Gilles ,Oct 11, 2010 at 18:14

date -d @1190000000 Replace 1190000000 with your epoch

Stefan Lasiewski ,Oct 11, 2010 at 18:04

$ echo 1190000000 | perl -pe 's/(\d+)/localtime($1)/e' 
Sun Sep 16 20:33:20 2007

This can come in handy for those applications which use epoch time in the logfiles:

$ tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
[Thu May 13 10:15:46 2010] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;HOSTA;check_raid;0;check_raid.pl: OK (Unit 0 on Controller 0 is OK)

Stéphane Chazelas ,Jul 31, 2015 at 20:24

With bash-4.2 or above:
printf '%(%F %T)T\n' 1234567890

(where %F %T is the strftime() -type format)

That syntax is inspired from ksh93 .

In ksh93 however, the argument is taken as a date expression where various and hardly documented formats are supported.

For a Unix epoch time, the syntax in ksh93 is:

printf '%(%F %T)T\n' '#1234567890'

ksh93 however seems to use its own algorithm for the timezone and can get it wrong. For instance, in Britain, it was summer time all year in 1970, but:

$ TZ=Europe/London bash -c 'printf "%(%c)T\n" 0'
Thu 01 Jan 1970 01:00:00 BST
$ TZ=Europe/London ksh93 -c 'printf "%(%c)T\n" "#0"'
Thu Jan  1 00:00:00 1970

DarkHeart ,Jul 28, 2014 at 3:56

Custom format with GNU date :
date -d @1234567890 +'%Y-%m-%d %H:%M:%S'

Or with GNU awk :

awk 'BEGIN { print strftime("%Y-%m-%d %H:%M:%S", 1234567890); }'

Linked SO question: https://stackoverflow.com/questions/3249827/convert-from-unixtime-at-command-line

,

The two I frequently use are:
$ perl -leprint\ scalar\ localtime\ 1234567890
Sat Feb 14 00:31:30 2009

[Sep 02, 2019] bash - Pretty-print for shell script

Oct 21, 2010 | stackoverflow.com

Pretty-print for shell script Ask Question Asked 8 years, 10 months ago Active 30 days ago Viewed 14k times


Benoit ,Oct 21, 2010 at 13:19

I'm looking for something similiar to indent but for (bash) scripts. Console only, no colorizing, etc.

Do you know of one ?

Jamie ,Sep 11, 2012 at 3:00

Vim can indent bash scripts. But not reformat them before indenting.
Backup your bash script, open it with vim, type gg=GZZ and indent will be corrected. (Note for the impatient: this overwrites the file, so be sure to do that backup!)

Though, some bugs with << (expecting EOF as first character on a line) e.g.

EDIT: ZZ not ZQ

Daniel Martí ,Apr 8, 2018 at 13:52

A bit late to the party, but it looks like shfmt could do the trick for you.

Brian Chrisman ,Aug 11 at 4:08

In bash I do this:
reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3 | sed -e "s/^\s\s\s\s//"
}

this eliminates comments and reindents the script "bash way".

If you have HEREDOCS in your script, they got ruined by the sed in the previous function.

So use:

reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3"
}

But all your script will have a 4 spaces indentation.

Or you can do:

reindent () 
{ 
    rstr=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1);
    source <(echo "Zibri () {";cat "$1"|sed -e "s/^\s\s\s\s/$rstr/"; echo "}");
    echo '#!/bin/bash';
    declare -f Zibri | head --lines=-1 | tail --lines=+3 | sed -e "s/^\s\s\s\s//;s/$rstr/    /"
}

which takes care also of heredocs.

Pius Raeder ,Jan 10, 2017 at 8:35

Found this http://www.linux-kheops.com/doc/perl/perl-aubert/fmt.script .

Very nice, only one thing i took out is the [...]->test substitution.

[Sep 02, 2019] Negative regex for Perl string pattern match

Sep 02, 2019 | stackoverflow.com

mirod ,Jun 15, 2011 at 17:21

I have this regex:
if($string =~ m/^(Clinton|[^Bush]|Reagan)/i)
  {print "$string\n"};

I want to match with Clinton and Reagan, but not Bush.

It's not working.

Calvin Taylor ,Jul 14, 2017 at 21:03

Sample text:

Clinton said
Bush used crayons
Reagan forgot

Just omitting a Bush match:

$ perl -ne 'print if /^(Clinton|Reagan)/' textfile
Clinton said
Reagan forgot

Or if you really want to specify:

$ perl -ne 'print if /^(?!Bush)(Clinton|Reagan)/' textfile
Clinton said
Reagan forgot

GuruM ,Oct 27, 2012 at 12:54

Your regex does not work because [] defines a character class, but what you want is a lookahead:
(?=) - Positive look ahead assertion foo(?=bar) matches foo when followed by bar
(?!) - Negative look ahead assertion foo(?!bar) matches foo when not followed by bar
(?<=) - Positive look behind assertion (?<=foo)bar matches bar when preceded by foo
(?<!) - Negative look behind assertion (?<!foo)bar matches bar when NOT preceded by foo
(?>) - Once-only subpatterns (?>\d+)bar Performance enhancing when bar not present
(?(x)) - Conditional subpatterns
(?(3)foo|fu)bar - Matches foo if 3rd subpattern has matched, fu if not
(?#) - Comment (?# Pattern does x y or z)

So try: (?!bush)

[Sep 02, 2019] How to get the current line number of a file open using Perl

Sep 02, 2019 | stackoverflow.com

How to get the current line number of a file open using Perl? Ask Question Asked 8 years, 3 months ago Active 6 months ago Viewed 33k times 25 1


tadmc ,May 8, 2011 at 17:08

open my $fp, '<', $file or die $!;

while (<$fp>) {
    my $line = $_;
    if ($line =~ /$regex/) {
        # How do I find out which line number this match happened at?
    }
}

close $fp;

tchrist ,Apr 22, 2015 at 21:16

Use $. (see perldoc perlvar ).

tchrist ,May 7, 2011 at 16:48

You can also do it through OO interface:
use IO::Handle;
# later on ...
my $n = $fp->input_line_number();

This is in perldoc perlvar , too.

> ,

Don't use $. , nor $_ or any global variable. Use this instead:
while(my $line = <FILE>) {
  print $line unless ${\*FILE}->input_line_number == 1;
}

To avoid this and a lot of others Perl gotchas you can use on Atom or VSCode packages like linter-perl . Stop making Perl a write-only language !

[Aug 31, 2019] Complexity prevent programmer from ever learning the whole language, only subset is learned and used

Aug 31, 2019 | ask.slashdot.org

Re:Neither! (Score 2, Interesting) 817 by M. D. Nahas on Friday December 23, 2005 @06:08PM ( #14329127 ) Attached to: Learning Java or C# as a Next Language? The cleanest languages I've used are C, Java, and OCaml. By "clean", I mean the language has a few concepts that can be completely memorized, which results in less "gotchas" and manual reading. For these languages, you'll see small manuals (e.g., K&R's book for C) which cover the complete language and then lots of pages devoted to the libraries that come with the language. I'd definitely recommend Java (or C, or OCaml) over C# for this reason. C# seems to have combined every feature of C++, Java, and VBA into a single language. It is very complex and has a ton of concepts, for which I could never memorize the whole language. I have a feeling that most programmers will use the subset of C# that is closest to the language they understand, whether it is C++, Java or VBA. You might as well learn Java's style of programming, and then, if needed, switch to C# using its Java-like features.

[Aug 29, 2019] How do I parse command line arguments in Bash - Stack Overflow

Jul 10, 2017 | stackoverflow.com

Livven, Jul 10, 2017 at 8:11

Update: It's been more than 5 years since I started this answer. Thank you for LOTS of great edits/comments/suggestions. In order save maintenance time, I've modified the code block to be 100% copy-paste ready. Please do not post comments like "What if you changed X to Y ". Instead, copy-paste the code block, see the output, make the change, rerun the script, and comment "I changed X to Y and " I don't have time to test your ideas and tell you if they work.
Method #1: Using bash without getopt[s]

Two common ways to pass key-value-pair arguments are:

Bash Space-Separated (e.g., --option argument ) (without getopt[s])

Usage demo-space-separated.sh -e conf -s /etc -l /usr/lib /etc/hosts

cat >/tmp/demo-space-separated.sh <<'EOF'
#!/bin/bash

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -e|--extension)
    EXTENSION="$2"
    shift # past argument
    shift # past value
    ;;
    -s|--searchpath)
    SEARCHPATH="$2"
    shift # past argument
    shift # past value
    ;;
    -l|--lib)
    LIBPATH="$2"
    shift # past argument
    shift # past value
    ;;
    --default)
    DEFAULT=YES
    shift # past argument
    ;;
    *)    # unknown option
    POSITIONAL+=("$1") # save it in an array for later
    shift # past argument
    ;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

echo "FILE EXTENSION  = ${EXTENSION}"
echo "SEARCH PATH     = ${SEARCHPATH}"
echo "LIBRARY PATH    = ${LIBPATH}"
echo "DEFAULT         = ${DEFAULT}"
echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l)
if [[ -n $1 ]]; then
    echo "Last line of file specified as non-opt/last argument:"
    tail -1 "$1"
fi
EOF

chmod +x /tmp/demo-space-separated.sh

/tmp/demo-space-separated.sh -e conf -s /etc -l /usr/lib /etc/hosts

output from copy-pasting the block above:

FILE EXTENSION  = conf
SEARCH PATH     = /etc
LIBRARY PATH    = /usr/lib
DEFAULT         =
Number files in SEARCH PATH with EXTENSION: 14
Last line of file specified as non-opt/last argument:
#93.184.216.34    example.com
Bash Equals-Separated (e.g., --option=argument ) (without getopt[s])

Usage demo-equals-separated.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts

cat >/tmp/demo-equals-separated.sh <<'EOF'
#!/bin/bash

for i in "$@"
do
case $i in
    -e=*|--extension=*)
    EXTENSION="${i#*=}"
    shift # past argument=value
    ;;
    -s=*|--searchpath=*)
    SEARCHPATH="${i#*=}"
    shift # past argument=value
    ;;
    -l=*|--lib=*)
    LIBPATH="${i#*=}"
    shift # past argument=value
    ;;
    --default)
    DEFAULT=YES
    shift # past argument with no value
    ;;
    *)
          # unknown option
    ;;
esac
done
echo "FILE EXTENSION  = ${EXTENSION}"
echo "SEARCH PATH     = ${SEARCHPATH}"
echo "LIBRARY PATH    = ${LIBPATH}"
echo "DEFAULT         = ${DEFAULT}"
echo "Number files in SEARCH PATH with EXTENSION:" $(ls -1 "${SEARCHPATH}"/*."${EXTENSION}" | wc -l)
if [[ -n $1 ]]; then
    echo "Last line of file specified as non-opt/last argument:"
    tail -1 $1
fi
EOF

chmod +x /tmp/demo-equals-separated.sh

/tmp/demo-equals-separated.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts

output from copy-pasting the block above:

FILE EXTENSION  = conf
SEARCH PATH     = /etc
LIBRARY PATH    = /usr/lib
DEFAULT         =
Number files in SEARCH PATH with EXTENSION: 14
Last line of file specified as non-opt/last argument:
#93.184.216.34    example.com

To better understand ${i#*=} search for "Substring Removal" in this guide . It is functionally equivalent to `sed 's/[^=]*=//' <<< "$i"` which calls a needless subprocess or `echo "$i" | sed 's/[^=]*=//'` which calls two needless subprocesses.

Method #2: Using bash with getopt[s]

from: http://mywiki.wooledge.org/BashFAQ/035#getopts

getopt(1) limitations (older, relatively-recent getopt versions):

More recent getopt versions don't have these limitations.

Additionally, the POSIX shell (and others) offer getopts which doesn't have these limitations. I've included a simplistic getopts example.

Usage demo-getopts.sh -vf /etc/hosts foo bar

cat >/tmp/demo-getopts.sh <<'EOF'
#!/bin/sh

# A POSIX variable
OPTIND=1         # Reset in case getopts has been used previously in the shell.

# Initialize our own variables:
output_file=""
verbose=0

while getopts "h?vf:" opt; do
    case "$opt" in
    h|\?)
        show_help
        exit 0
        ;;
    v)  verbose=1
        ;;
    f)  output_file=$OPTARG
        ;;
    esac
done

shift $((OPTIND-1))

[ "${1:-}" = "--" ] && shift

echo "verbose=$verbose, output_file='$output_file', Leftovers: $@"
EOF

chmod +x /tmp/demo-getopts.sh

/tmp/demo-getopts.sh -vf /etc/hosts foo bar

output from copy-pasting the block above:

verbose=1, output_file='/etc/hosts', Leftovers: foo bar

The advantages of getopts are:

  1. It's more portable, and will work in other shells like dash .
  2. It can handle multiple single options like -vf filename in the typical Unix way, automatically.

The disadvantage of getopts is that it can only handle short options ( -h , not --help ) without additional code.

There is a getopts tutorial which explains what all of the syntax and variables mean. In bash, there is also help getopts , which might be informative.

johncip ,Jul 23, 2018 at 15:15

No answer mentions enhanced getopt . And the top-voted answer is misleading: It either ignores -⁠vfd style short options (requested by the OP) or options after positional arguments (also requested by the OP); and it ignores parsing-errors. Instead:

The following calls

myscript -vfd ./foo/bar/someFile -o /fizz/someOtherFile
myscript -v -f -d -o/fizz/someOtherFile -- ./foo/bar/someFile
myscript --verbose --force --debug ./foo/bar/someFile -o/fizz/someOtherFile
myscript --output=/fizz/someOtherFile ./foo/bar/someFile -vfd
myscript ./foo/bar/someFile -df -v --output /fizz/someOtherFile

all return

verbose: y, force: y, debug: y, in: ./foo/bar/someFile, out: /fizz/someOtherFile

with the following myscript

#!/bin/bash
# saner programming env: these switches turn some bugs into errors
set -o errexit -o pipefail -o noclobber -o nounset

# -allow a command to fail with !'s side effect on errexit
# -use return value from ${PIPESTATUS[0]}, because ! hosed $?
! getopt --test > /dev/null 
if [[ ${PIPESTATUS[0]} -ne 4 ]]; then
    echo 'I'm sorry, `getopt --test` failed in this environment.'
    exit 1
fi

OPTIONS=dfo:v
LONGOPTS=debug,force,output:,verbose

# -regarding ! and PIPESTATUS see above
# -temporarily store output to be able to check for errors
# -activate quoting/enhanced mode (e.g. by writing out "--options")
# -pass arguments only via   -- "$@"   to separate them correctly
! PARSED=$(getopt --options=$OPTIONS --longoptions=$LONGOPTS --name "$0" -- "$@")
if [[ ${PIPESTATUS[0]} -ne 0 ]]; then
    # e.g. return value is 1
    #  then getopt has complained about wrong arguments to stdout
    exit 2
fi
# read getopt's output this way to handle the quoting right:
eval set -- "$PARSED"

d=n f=n v=n outFile=-
# now enjoy the options in order and nicely split until we see --
while true; do
    case "$1" in
        -d|--debug)
            d=y
            shift
            ;;
        -f|--force)
            f=y
            shift
            ;;
        -v|--verbose)
            v=y
            shift
            ;;
        -o|--output)
            outFile="$2"
            shift 2
            ;;
        --)
            shift
            break
            ;;
        *)
            echo "Programming error"
            exit 3
            ;;
    esac
done

# handle non-option arguments
if [[ $# -ne 1 ]]; then
    echo "$0: A single input file is required."
    exit 4
fi

echo "verbose: $v, force: $f, debug: $d, in: $1, out: $outFile"

1 enhanced getopt is available on most "bash-systems", including Cygwin; on OS X try brew install gnu-getopt or sudo port install getopt
2 the POSIX exec() conventions have no reliable way to pass binary NULL in command line arguments; those bytes prematurely end the argument
3 first version released in 1997 or before (I only tracked it back to 1997)

Tobias Kienzler ,Mar 19, 2016 at 15:23

from : digitalpeer.com with minor modifications

Usage myscript.sh -p=my_prefix -s=dirname -l=libname

#!/bin/bash
for i in "$@"
do
case $i in
    -p=*|--prefix=*)
    PREFIX="${i#*=}"

    ;;
    -s=*|--searchpath=*)
    SEARCHPATH="${i#*=}"
    ;;
    -l=*|--lib=*)
    DIR="${i#*=}"
    ;;
    --default)
    DEFAULT=YES
    ;;
    *)
            # unknown option
    ;;
esac
done
echo PREFIX = ${PREFIX}
echo SEARCH PATH = ${SEARCHPATH}
echo DIRS = ${DIR}
echo DEFAULT = ${DEFAULT}

To better understand ${i#*=} search for "Substring Removal" in this guide . It is functionally equivalent to `sed 's/[^=]*=//' <<< "$i"` which calls a needless subprocess or `echo "$i" | sed 's/[^=]*=//'` which calls two needless subprocesses.

Robert Siemer ,Jun 1, 2018 at 1:57

getopt() / getopts() is a good option. Stolen from here :

The simple use of "getopt" is shown in this mini-script:

#!/bin/bash
echo "Before getopt"
for i
do
  echo $i
done
args=`getopt abc:d $*`
set -- $args
echo "After getopt"
for i
do
  echo "-->$i"
done

What we have said is that any of -a, -b, -c or -d will be allowed, but that -c is followed by an argument (the "c:" says that).

If we call this "g" and try it out:

bash-2.05a$ ./g -abc foo
Before getopt
-abc
foo
After getopt
-->-a
-->-b
-->-c
-->foo
-->--

We start with two arguments, and "getopt" breaks apart the options and puts each in its own argument. It also added "--".

hfossli ,Jan 31 at 20:05

More succinct way

script.sh

#!/bin/bash

while [[ "$#" -gt 0 ]]; do case $1 in
  -d|--deploy) deploy="$2"; shift;;
  -u|--uglify) uglify=1;;
  *) echo "Unknown parameter passed: $1"; exit 1;;
esac; shift; done

echo "Should deploy? $deploy"
echo "Should uglify? $uglify"

Usage:

./script.sh -d dev -u

# OR:

./script.sh --deploy dev --uglify

bronson ,Apr 27 at 23:22

At the risk of adding another example to ignore, here's my scheme.

Hope it's useful to someone.

while [ "$#" -gt 0 ]; do
  case "$1" in
    -n) name="$2"; shift 2;;
    -p) pidfile="$2"; shift 2;;
    -l) logfile="$2"; shift 2;;

    --name=*) name="${1#*=}"; shift 1;;
    --pidfile=*) pidfile="${1#*=}"; shift 1;;
    --logfile=*) logfile="${1#*=}"; shift 1;;
    --name|--pidfile|--logfile) echo "$1 requires an argument" >&2; exit 1;;

    -*) echo "unknown option: $1" >&2; exit 1;;
    *) handle_argument "$1"; shift 1;;
  esac
done

Robert Siemer ,Jun 6, 2016 at 19:28

I'm about 4 years late to this question, but want to give back. I used the earlier answers as a starting point to tidy up my old adhoc param parsing. I then refactored out the following template code. It handles both long and short params, using = or space separated arguments, as well as multiple short params grouped together. Finally it re-inserts any non-param arguments back into the $1,$2.. variables. I hope it's useful.
#!/usr/bin/env bash

# NOTICE: Uncomment if your script depends on bashisms.
#if [ -z "$BASH_VERSION" ]; then bash $0 $@ ; exit $? ; fi

echo "Before"
for i ; do echo - $i ; done


# Code template for parsing command line parameters using only portable shell
# code, while handling both long and short params, handling '-f file' and
# '-f=file' style param data and also capturing non-parameters to be inserted
# back into the shell positional parameters.

while [ -n "$1" ]; do
        # Copy so we can modify it (can't modify $1)
        OPT="$1"
        # Detect argument termination
        if [ x"$OPT" = x"--" ]; then
                shift
                for OPT ; do
                        REMAINS="$REMAINS \"$OPT\""
                done
                break
        fi
        # Parse current opt
        while [ x"$OPT" != x"-" ] ; do
                case "$OPT" in
                        # Handle --flag=value opts like this
                        -c=* | --config=* )
                                CONFIGFILE="${OPT#*=}"
                                shift
                                ;;
                        # and --flag value opts like this
                        -c* | --config )
                                CONFIGFILE="$2"
                                shift
                                ;;
                        -f* | --force )
                                FORCE=true
                                ;;
                        -r* | --retry )
                                RETRY=true
                                ;;
                        # Anything unknown is recorded for later
                        * )
                                REMAINS="$REMAINS \"$OPT\""
                                break
                                ;;
                esac
                # Check for multiple short options
                # NOTICE: be sure to update this pattern to match valid options
                NEXTOPT="${OPT#-[cfr]}" # try removing single short opt
                if [ x"$OPT" != x"$NEXTOPT" ] ; then
                        OPT="-$NEXTOPT"  # multiple short opts, keep going
                else
                        break  # long form, exit inner loop
                fi
        done
        # Done with that param. move to next
        shift
done
# Set the non-parameters back into the positional parameters ($1 $2 ..)
eval set -- $REMAINS


echo -e "After: \n configfile='$CONFIGFILE' \n force='$FORCE' \n retry='$RETRY' \n remains='$REMAINS'"
for i ; do echo - $i ; done

> ,

I have found the matter to write portable parsing in scripts so frustrating that I have written Argbash - a FOSS code generator that can generate the arguments-parsing code for your script plus it has some nice features:

https://argbash.io

[Aug 29, 2019] shell - An example of how to use getopts in bash - Stack Overflow

The key thing to understand is that getops is just parsing options. You need to shift them as a separate operation:
shift $((OPTIND-1))
May 10, 2013 | stackoverflow.com

An example of how to use getopts in bash Ask Question Asked 6 years, 3 months ago Active 10 months ago Viewed 419k times 288 132

chepner ,May 10, 2013 at 13:42

I want to call myscript file in this way:
$ ./myscript -s 45 -p any_string

or

$ ./myscript -h >>> should display help
$ ./myscript    >>> should display help

My requirements are:

I tried so far this code:

#!/bin/bash
while getopts "h:s:" arg; do
  case $arg in
    h)
      echo "usage" 
      ;;
    s)
      strength=$OPTARG
      echo $strength
      ;;
  esac
done

But with that code I get errors. How to do it with Bash and getopt ?

,

#!/bin/bash

usage() { echo "Usage: $0 [-s <45|90>] [-p <string>]" 1>&2; exit 1; }

while getopts ":s:p:" o; do
    case "${o}" in
        s)
            s=${OPTARG}
            ((s == 45 || s == 90)) || usage
            ;;
        p)
            p=${OPTARG}
            ;;
        *)
            usage
            ;;
    esac
done
shift $((OPTIND-1))

if [ -z "${s}" ] || [ -z "${p}" ]; then
    usage
fi

echo "s = ${s}"
echo "p = ${p}"

Example runs:

$ ./myscript.sh
Usage: ./myscript.sh [-s <45|90>] [-p <string>]

$ ./myscript.sh -h
Usage: ./myscript.sh [-s <45|90>] [-p <string>]

$ ./myscript.sh -s "" -p ""
Usage: ./myscript.sh [-s <45|90>] [-p <string>]

$ ./myscript.sh -s 10 -p foo
Usage: ./myscript.sh [-s <45|90>] [-p <string>]

$ ./myscript.sh -s 45 -p foo
s = 45
p = foo

$ ./myscript.sh -s 90 -p bar
s = 90
p = bar

[Aug 28, 2019] How do I import a perl module outside of @INC that does not end in .pm - Stack Overflow

Aug 22, 2019 | stackoverflow.com


mob ,Aug 22 at 19:47

Background

I am attempting to import a perl module that does not end in .pm with a method similar to this answer :

use lib "/hard/coded/directory"; use scripting;

However, when I attempt to import a module in this way, I get the following error when running perl -c :

Can't locate scripting.pm in @INC (@INC contains: ... ... ... /hard/coded/directory) at name of script line 47.

BEGIN failed--compilation aborted at name of script line 47.

Question

How do I import a perl module outside of @INC that does not have .pm at the end of the file?

ikegami ,Aug 22 at 20:04

If the file has a package directive, the file name and the package directive need to match, so simply fix the file name.

If the file doesn't have a package directive, you don't have a module , and you shouldn't use use or require . This can cause problems.

What you have is sometimes called a library, and you should use do .

do('/hard/coded/directory/scripting')
   or die $@ || $!;

(For proper error checking, the file needs to result in a true value.)

That said, you are probably trying to do something really awful. I'm guessing you're either have a configuration file written in Perl or a poorly written module. Perl is not a suitable choice of language for a configuration file, and avoiding namespaces is just bad programming with no benefit.

ikegami ,Aug 22 at 20:10

If the source file does not define new namespaces or classes and you just want to read the function definitions or data from a file, Perl provides the do and require functions.
do "scripting";
require "scripting";

The difference between them is that require will look for the file to evaluate to a true value (it expects the last statement in the file to resolve to a non-zero, non-empty value), and will emit a fatal error if this does not happen. (You will often see naked 1; statements at the end of modules to satisfy this requirement).

If scripting really contains class code and you do need all the functionality that the use function provides, remember that

use Foo::Bar qw(stuff);

is just syntactic sugar for

BEGIN {
    $file = <find Foo/Bar.pm on @INC>;
    require "$file";
    Foo::Bar->import( qw(stuff) )
}

and suggests how you can workaround your inability to use use :

BEGIN {
    require "scripting";
    scripting->import()
}

In theory, the file scripting might define some other package and begin with a line like package Something::Else; . Then you would load the package in this module with

BEGIN {
    require "scripting";
    Something::Else->import();
}

[Aug 27, 2019] perl defensive programming (die, assert, croak) - Stack Overflow

Aug 27, 2019 | stackoverflow.com

perl defensive programming (die, assert, croak) Ask Question Asked 5 years, 6 months ago Active 5 years, 6 months ago Viewed 645 times 2 0


Zaid ,Feb 23, 2014 at 17:11

What is the best (or recommended) approach to do defensive programming in perl? For example if I have a sub which must be called with a (defined) SCALAR, an ARRAYREF and an optional HASHREF.

Three of the approaches I have seen:

sub test1 {
    die if !(@_ == 2 || @_ == 3);
    my ($scalar, $arrayref, $hashref) = @_;
    die if !defined($scalar) || ref($scalar);
    die if ref($arrayref) ne 'ARRAY';
    die if defined($hashref) && ref($hashref) ne 'HASH';
    #do s.th with scalar, arrayref and hashref
}

sub test2 {
    Carp::assert(@_ == 2 || @_ == 3) if DEBUG;
    my ($scalar, $arrayref, $hashref) = @_;
    if(DEBUG) {
        Carp::assert defined($scalar) && !ref($scalar);
        Carp::assert ref($arrayref) eq 'ARRAY';
        Carp::assert !defined($hashref) || ref($hashref) eq 'HASH';
    }
    #do s.th with scalar, arrayref and hashref
}

sub test3 {
    my ($scalar, $arrayref, $hashref) = @_;
    (@_ == 2 || @_ == 3 && defined($scalar) && !ref($scalar) && ref($arrayref) eq 'ARRAY' && (!defined($hashref) || ref($hashref) eq 'HASH'))
        or Carp::croak 'usage: test3(SCALAR, ARRAYREF, [HASHREF])';
    #do s.th with scalar, arrayref and hashref
}

tobyink ,Feb 23, 2014 at 21:44

use Params::Validate qw(:all);

sub Yada {
   my (...)=validate_pos(@_,{ type=>SCALAR },{ type=>ARRAYREF },{ type=>HASHREF,optional=>1 });
   ...
}

ikegami ,Feb 23, 2014 at 17:33

I wouldn't use any of them. Aside from not not accepting many array and hash references, the checks you used are almost always redundant.
>perl -we"use strict; sub { my ($x) = @_; my $y = $x->[0] }->( 'abc' )"
Can't use string ("abc") as an ARRAY ref nda"strict refs" in use at -e line 1.

>perl -we"use strict; sub { my ($x) = @_; my $y = $x->[0] }->( {} )"
Not an ARRAY reference at -e line 1.

The only advantage to checking is that you can use croak to show the caller in the error message.


Proper way to check if you have an reference to an array:

defined($x) && eval { @$x; 1 }

Proper way to check if you have an reference to a hash:

defined($x) && eval { %$x; 1 }

Borodin ,Feb 23, 2014 at 17:23

None of the options you show display any message to give a reason for the failure, which I think is paramount.

It is also preferable to use croak instead of die from within library subroutines, so that the error is reported from the point of view of the caller.

I would replace all occurrences of if ! with unless . The former is a C programmer's habit.

I suggest something like this

sub test1 {
    croak "Incorrect number of parameters" unless @_ == 2 or @_ == 3;
    my ($scalar, $arrayref, $hashref) = @_;
    croak "Invalid first parameter" unless $scalar and not ref $scalar;
    croak "Invalid second parameter" unless $arrayref eq 'ARRAY';
    croak "Invalid third parameter" if defined $hashref and ref $hashref ne 'HASH';

    # do s.th with scalar, arrayref and hashref
}

[Aug 27, 2019] linux - How to show line number when executing bash script - Stack Overflow

Aug 27, 2019 | stackoverflow.com

How to show line number when executing bash script Ask Question Asked 6 years, 1 month ago Active 1 year, 4 months ago Viewed 47k times 68 31


dspjm ,Jul 23, 2013 at 7:31

I have a test script which has a lot of commands and will generate lots of output, I use set -x or set -v and set -e , so the script would stop when error occurs. However, it's still rather difficult for me to locate which line did the execution stop in order to locate the problem. Is there a method which can output the line number of the script before each line is executed? Or output the line number before the command exhibition generated by set -x ? Or any method which can deal with my script line location problem would be a great help. Thanks.

Suvarna Pattayil ,Jul 28, 2017 at 17:25

You mention that you're already using -x . The variable PS4 denotes the value is the prompt printed before the command line is echoed when the -x option is set and defaults to : followed by space.

You can change PS4 to emit the LINENO (The line number in the script or shell function currently executing).

For example, if your script reads:

$ cat script
foo=10
echo ${foo}
echo $((2 + 2))

Executing it thus would print line numbers:

$ PS4='Line ${LINENO}: ' bash -x script
Line 1: foo=10
Line 2: echo 10
10
Line 3: echo 4
4

http://wiki.bash-hackers.org/scripting/debuggingtips gives the ultimate PS4 that would output everything you will possibly need for tracing:

export PS4='+(${BASH_SOURCE}:${LINENO}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'

Deqing ,Jul 23, 2013 at 8:16

In Bash, $LINENO contains the line number where the script currently executing.

If you need to know the line number where the function was called, try $BASH_LINENO . Note that this variable is an array.

For example:

#!/bin/bash       

function log() {
    echo "LINENO: ${LINENO}"
    echo "BASH_LINENO: ${BASH_LINENO[*]}"
}

function foo() {
    log "$@"
}

foo "$@"

See here for details of Bash variables.

Eliran Malka ,Apr 25, 2017 at 10:14

Simple (but powerful) solution: Place echo around the code you think that causes the problem and move the echo line by line until the messages does not appear anymore on screen - because the script has stop because of an error before.

Even more powerful solution: Install bashdb the bash debugger and debug the script line by line

kklepper ,Apr 2, 2018 at 22:44

Workaround for shells without LINENO

In a fairly sophisticated script I wouldn't like to see all line numbers; rather I would like to be in control of the output.

Define a function

echo_line_no () {
    grep -n "$1" $0 |  sed "s/echo_line_no//" 
    # grep the line(s) containing input $1 with line numbers
    # replace the function name with nothing 
} # echo_line_no

Use it with quotes like

echo_line_no "this is a simple comment with a line number"

Output is

16   "this is a simple comment with a line number"

if the number of this line in the source file is 16.

This basically answers the question How to show line number when executing bash script for users of ash or other shells without LINENO .

Anything more to add?

Sure. Why do you need this? How do you work with this? What can you do with this? Is this simple approach really sufficient or useful? Why do you want to tinker with this at all?

Want to know more? Read reflections on debugging

[Aug 27, 2019] How do I get the filename and line number in Perl - Stack Overflow

Aug 27, 2019 | stackoverflow.com

How do I get the filename and line number in Perl? Ask Question Asked 8 years, 10 months ago Active 8 years, 9 months ago Viewed 6k times 6


Elijah ,Nov 1, 2010 at 17:35

I would like to get the current filename and line number within a Perl script. How do I do this?

For example, in a file call test.pl :

my $foo = 'bar';
print 'Hello World';
print functionForFilename() . ':' . functionForLineNo();

It would output:

Hello World
test.pl:3

tchrist ,Nov 2, 2010 at 19:13

These are available with the __LINE__ and __FILE__ tokens, as documented in perldoc perldata under "Special Literals":

The special literals __FILE__, __LINE__, and __PACKAGE__ represent the current filename, line number, and package name at that point in your program. They may be used only as separate tokens; they will not be interpolated into strings. If there is no current package (due to an empty package; directive), __PACKAGE__ is the undefined value.

Eric Strom ,Nov 1, 2010 at 17:41

The caller function will do what you are looking for:
sub print_info {
   my ($package, $filename, $line) = caller;
   ...
}

print_info(); # prints info about this line

This will get the information from where the sub is called, which is probably what you are looking for. The __FILE__ and __LINE__ directives only apply to where they are written, so you can not encapsulate their effect in a subroutine. (unless you wanted a sub that only prints info about where it is defined)

,

You can use:
print __FILE__. " " . __LINE__;

[Aug 26, 2019] bash - How to prevent rm from reporting that a file was not found

Aug 26, 2019 | stackoverflow.com

How to prevent rm from reporting that a file was not found? Ask Question Asked 7 years, 4 months ago Active 1 year, 4 months ago Viewed 101k times 133 19


pizza ,Apr 20, 2012 at 21:29

I am using rm within a BASH script to delete many files. Sometimes the files are not present, so it reports many errors. I do not need this message. I have searched the man page for a command to make rm quiet, but the only option I found is -f , which from the description, "ignore nonexistent files, never prompt", seems to be the right choice, but the name does not seem to fit, so I am concerned it might have unintended consequences.

Keith Thompson ,Dec 19, 2018 at 13:05

The main use of -f is to force the removal of files that would not be removed using rm by itself (as a special case, it "removes" non-existent files, thus suppressing the error message).

You can also just redirect the error message using

$ rm file.txt 2> /dev/null

(or your operating system's equivalent). You can check the value of $? immediately after calling rm to see if a file was actually removed or not.

vimdude ,May 28, 2014 at 18:10

Yes, -f is the most suitable option for this.

tripleee ,Jan 11 at 4:50

-f is the correct flag, but for the test operator, not rm
[ -f "$THEFILE" ] && rm "$THEFILE"

this ensures that the file exists and is a regular file (not a directory, device node etc...)

mahemoff ,Jan 11 at 4:41

\rm -f file will never report not found.

Idelic ,Apr 20, 2012 at 16:51

As far as rm -f doing "anything else", it does force ( -f is shorthand for --force ) silent removal in situations where rm would otherwise ask you for confirmation. For example, when trying to remove a file not writable by you from a directory that is writable by you.

Keith Thompson ,May 28, 2014 at 18:09

I had same issue for cshell. The only solution I had was to create a dummy file that matched pattern before "rm" in my script.

[Aug 26, 2019] shell - rm -rf return codes

Aug 26, 2019 | superuser.com

rm -rf return codes Ask Question Asked 6 years ago Active 6 years ago Viewed 15k times 8 0


SheetJS ,Aug 15, 2013 at 2:50

Any one can let me know the possible return codes for the command rm -rf other than zero i.e, possible return codes for failure cases. I want to know more detailed reason for the failure of the command unlike just the command is failed(return other than 0).

Adrian Frühwirth ,Aug 14, 2013 at 7:00

To see the return code, you can use echo $? in bash.

To see the actual meaning, some platforms (like Debian Linux) have the perror binary available, which can be used as follows:

$ rm -rf something/; perror $?
rm: cannot remove `something/': Permission denied
OS error code   1:  Operation not permitted

rm -rf automatically suppresses most errors. The most likely error you will see is 1 (Operation not permitted), which will happen if you don't have permissions to remove the file. -f intentionally suppresses most errors

Adrian Frühwirth ,Aug 14, 2013 at 7:21

grabbed coreutils from git....

looking at exit we see...

openfly@linux-host:~/coreutils/src $ cat rm.c | grep -i exit
  if (status != EXIT_SUCCESS)
  exit (status);
  /* Since this program exits immediately after calling 'rm', rm need not
  atexit (close_stdin);
          usage (EXIT_FAILURE);
        exit (EXIT_SUCCESS);
          usage (EXIT_FAILURE);
        error (EXIT_FAILURE, errno, _("failed to get attributes of %s"),
        exit (EXIT_SUCCESS);
  exit (status == RM_ERROR ? EXIT_FAILURE : EXIT_SUCCESS);

Now looking at the status variable....

openfly@linux-host:~/coreutils/src $ cat rm.c | grep -i status
usage (int status)
  if (status != EXIT_SUCCESS)
  exit (status);
  enum RM_status status = rm (file, &x);
  assert (VALID_STATUS (status));
  exit (status == RM_ERROR ? EXIT_FAILURE : EXIT_SUCCESS);

looks like there isn't much going on there with the exit status.

I see EXIT_FAILURE and EXIT_SUCCESS and not anything else.

so basically 0 and 1 / -1

To see specific exit() syscalls and how they occur in a process flow try this

openfly@linux-host:~/ $ strace rm -rf $whatever

fairly simple.

ref:

http://www.unix.com/man-page/Linux/EXIT_FAILURE/exit/

[Aug 26, 2019] debugging - How can I debug a Perl script - Stack Overflow

Jun 27, 2014 | stackoverflow.com

Matthew Lock ,Jun 27, 2014 at 1:01

To run your script under perl debugger you should use -d switch:
perl -d script.pl

But perl is flexible. It supply some hooks and you may force debugger to work as you want

So to use different debuggers you may do:

perl -d:DebugHooks::Terminal script.pl
# OR
perl -d:Trepan script.pl

Look these modules here and here

There are several most interesting perl modules that hook into perl debugger internals: Devel::NYTProf , Devel::Cover

And many others

XXX,

If you want to do remote debug (for cgi or if you don't want to mess output with debug command line) use this:

given test:

use v5.14;
say 1;
say 2;
say 3;

Start a listener on whatever host and port on terminal 1 (here localhost:12345):

$ nc -v -l localhost -p 12345

for readline support use rlwrap (you can use on perl -d too):

$ rlwrap nc -v -l localhost -p 12345

And start the test on another terminal (say terminal 2):

$ PERLDB_OPTS="RemotePort=localhost:12345" perl -d test

Input/Output on terminal 1:

Connection from 127.0.0.1:42994

Loading DB routines from perl5db.pl version 1.49
Editor support available.

Enter h or 'h h' for help, or 'man perldebug' for more help.

main::(test:2): say 1;
  DB<1> n
main::(test:3): say 2;
  DB<1> select $DB::OUT

  DB<2> n
2
main::(test:4): say 3;
  DB<2> n
3
Debugged program terminated.  Use q to quit or R to restart,
use o inhibit_exit to avoid stopping after program termination,
h q, h R or h o to get additional info.  
  DB<2>

Output on terminal 2:

1

Note the sentence if you want output on debug terminal

select $DB::OUT

If you are vim user, install this plugin: dbg.vim which provides basic support for perl

[Aug 26, 2019] D>ebugging - How to use the Perl debugger

Aug 26, 2019 | stackoverflow.com
This is like "please can you give me an example how to drive a car" .

I have explained the basic commands that you will use most often. Beyond this you must read the debugger's inline help and reread the perldebug documentation

The debugger will do a lot more than this, but these are the basic commands that you need to know. You should experiment with them and look at the contents of the help text to get more proficient with the Perl debugger

[Aug 25, 2019] How to check if a variable is set in Bash?

Aug 25, 2019 | stackoverflow.com

Ask Question Asked 8 years, 11 months ago Active 2 months ago Viewed 1.1m times 1339 435


Jens ,Jul 15, 2014 at 9:46

How do I know if a variable is set in Bash?

For example, how do I check if the user gave the first parameter to a function?

function a {
    # if $1 is set ?
}

Graeme ,Nov 25, 2016 at 5:07

(Usually) The right way
if [ -z ${var+x} ]; then echo "var is unset"; else echo "var is set to '$var'"; fi

where ${var+x} is a parameter expansion which evaluates to nothing if var is unset, and substitutes the string x otherwise.

Quotes Digression

Quotes can be omitted (so we can say ${var+x} instead of "${var+x}" ) because this syntax & usage guarantees this will only expand to something that does not require quotes (since it either expands to x (which contains no word breaks so it needs no quotes), or to nothing (which results in [ -z ] , which conveniently evaluates to the same value (true) that [ -z "" ] does as well)).

However, while quotes can be safely omitted, and it was not immediately obvious to all (it wasn't even apparent to the first author of this quotes explanation who is also a major Bash coder), it would sometimes be better to write the solution with quotes as [ -z "${var+x}" ] , at the very small possible cost of an O(1) speed penalty. The first author also added this as a comment next to the code using this solution giving the URL to this answer, which now also includes the explanation for why the quotes can be safely omitted.

(Often) The wrong way
if [ -z "$var" ]; then echo "var is blank"; else echo "var is set to '$var'"; fi

This is often wrong because it doesn't distinguish between a variable that is unset and a variable that is set to the empty string. That is to say, if var='' , then the above solution will output "var is blank".

The distinction between unset and "set to the empty string" is essential in situations where the user has to specify an extension, or additional list of properties, and that not specifying them defaults to a non-empty value, whereas specifying the empty string should make the script use an empty extension or list of additional properties.

The distinction may not be essential in every scenario though. In those cases [ -z "$var" ] will be just fine.

Flow ,Nov 26, 2014 at 13:49

To check for non-null/non-zero string variable, i.e. if set, use
if [ -n "$1" ]

It's the opposite of -z . I find myself using -n more than -z .

You would use it like:

if [ -n "$1" ]; then
  echo "You supplied the first parameter!"
else
  echo "First parameter not supplied."
fi

Jens ,Jan 19, 2016 at 23:30

Here's how to test whether a parameter is unset , or empty ("Null") or set with a value :
+--------------------+----------------------+-----------------+-----------------+
|                    |       parameter      |     parameter   |    parameter    |
|                    |   Set and Not Null   |   Set But Null  |      Unset      |
+--------------------+----------------------+-----------------+-----------------+
| ${parameter:-word} | substitute parameter | substitute word | substitute word |
| ${parameter-word}  | substitute parameter | substitute null | substitute word |
| ${parameter:=word} | substitute parameter | assign word     | assign word     |
| ${parameter=word}  | substitute parameter | substitute null | assign word     |
| ${parameter:?word} | substitute parameter | error, exit     | error, exit     |
| ${parameter?word}  | substitute parameter | substitute null | error, exit     |
| ${parameter:+word} | substitute word      | substitute null | substitute null |
| ${parameter+word}  | substitute word      | substitute word | substitute null |
+--------------------+----------------------+-----------------+-----------------+

Source: POSIX: Parameter Expansion :

In all cases shown with "substitute", the expression is replaced with the value shown. In all cases shown with "assign", parameter is assigned that value, which also replaces the expression.

Dan ,Jul 24, 2018 at 20:16

While most of the techniques stated here are correct, bash 4.2 supports an actual test for the presence of a variable ( man bash ), rather than testing the value of the variable.
[[ -v foo ]]; echo $?
# 1

foo=bar
[[ -v foo ]]; echo $?
# 0

foo=""
[[ -v foo ]]; echo $?
# 0

Notably, this approach will not cause an error when used to check for an unset variable in set -u / set -o nounset mode, unlike many other approaches, such as using [ -z .

chepner ,Sep 11, 2013 at 14:22

There are many ways to do this with the following being one of them:
if [ -z "$1" ]

This succeeds if $1 is null or unset

phkoester ,Feb 16, 2018 at 8:06

To see if a variable is nonempty, I use
if [[ $var ]]; then ...       # `$var' expands to a nonempty string

The opposite tests if a variable is either unset or empty:

if [[ ! $var ]]; then ...     # `$var' expands to the empty string (set or not)

To see if a variable is set (empty or nonempty), I use

if [[ ${var+x} ]]; then ...   # `var' exists (empty or nonempty)
if [[ ${1+x} ]]; then ...     # Parameter 1 exists (empty or nonempty)

The opposite tests if a variable is unset:

if [[ ! ${var+x} ]]; then ... # `var' is not set at all
if [[ ! ${1+x} ]]; then ...   # We were called with no arguments

Palec ,Jun 19, 2017 at 3:25

I always find the POSIX table in the other answer slow to grok, so here's my take on it:
   +----------------------+------------+-----------------------+-----------------------+
   |   if VARIABLE is:    |    set     |         empty         |        unset          |
   +----------------------+------------+-----------------------+-----------------------+
 - |  ${VARIABLE-default} | $VARIABLE  |          ""           |       "default"       |
 = |  ${VARIABLE=default} | $VARIABLE  |          ""           | $(VARIABLE="default") |
 ? |  ${VARIABLE?default} | $VARIABLE  |          ""           |       exit 127        |
 + |  ${VARIABLE+default} | "default"  |       "default"       |          ""           |
   +----------------------+------------+-----------------------+-----------------------+
:- | ${VARIABLE:-default} | $VARIABLE  |       "default"       |       "default"       |
:= | ${VARIABLE:=default} | $VARIABLE  | $(VARIABLE="default") | $(VARIABLE="default") |
:? | ${VARIABLE:?default} | $VARIABLE  |       exit 127        |       exit 127        |
:+ | ${VARIABLE:+default} | "default"  |          ""           |          ""           |
   +----------------------+------------+-----------------------+-----------------------+

Note that each group (with and without preceding colon) has the same set and unset cases, so the only thing that differs is how the empty cases are handled.

With the preceding colon, the empty and unset cases are identical, so I would use those where possible (i.e. use := , not just = , because the empty case is inconsistent).

Headings:

Values:

chepner ,Mar 28, 2017 at 12:26

On a modern version of Bash (4.2 or later I think; I don't know for sure), I would try this:
if [ ! -v SOMEVARIABLE ] #note the lack of a $ sigil
then
    echo "Variable is unset"
elif [ -z "$SOMEVARIABLE" ]
then
    echo "Variable is set to an empty string"
else
    echo "Variable is set to some string"
fi

Gordon Davisson ,May 15, 2015 at 13:53

if [ "$1" != "" ]; then
  echo \$1 is set
else
  echo \$1 is not set
fi

Although for arguments it is normally best to test $#, which is the number of arguments, in my opinion.

if [ $# -gt 0 ]; then
  echo \$1 is set
else
  echo \$1 is not set
fi

Jarrod Chesney ,Dec 9, 2016 at 3:34

You want to exit if it's unset

This worked for me. I wanted my script to exit with an error message if a parameter wasn't set.

#!/usr/bin/env bash

set -o errexit

# Get the value and empty validation check all in one
VER="${1:?You must pass a version of the format 0.0.0 as the only argument}"

This returns with an error when it's run

peek@peek:~$ ./setver.sh
./setver.sh: line 13: 1: You must pass a version of the format 0.0.0 as the only argument
Check only, no exit - Empty and Unset are INVALID

Try this option if you just want to check if the value set=VALID or unset/empty=INVALID.

TSET="good val"
TEMPTY=""
unset TUNSET

if [ "${TSET:-}" ]; then echo "VALID"; else echo "INVALID";fi
# VALID
if [ "${TEMPTY:-}" ]; then echo "VALID"; else echo "INVALID";fi
# INVALID
if [ "${TUNSET:-}" ]; then echo "VALID"; else echo "INVALID";fi
# INVALID

Or, Even short tests ;-)

[ "${TSET:-}"   ] && echo "VALID" || echo "INVALID"
[ "${TEMPTY:-}" ] && echo "VALID" || echo "INVALID"
[ "${TUNSET:-}" ] && echo "VALID" || echo "INVALID"
Check only, no exit - Only empty is INVALID

And this is the answer to the question. Use this if you just want to check if the value set/empty=VALID or unset=INVALID.

NOTE, the "1" in "..-1}" is insignificant, it can be anything (like x)

TSET="good val"
TEMPTY=""
unset TUNSET

if [ "${TSET+1}" ]; then echo "VALID"; else echo "INVALID";fi
# VALID
if [ "${TEMPTY+1}" ]; then echo "VALID"; else echo "INVALID";fi
# VALID
if [ "${TUNSET+1}" ]; then echo "VALID"; else echo "INVALID";fi
# INVALID

Short tests

[ "${TSET+1}"   ] && echo "VALID" || echo "INVALID"
[ "${TEMPTY+1}" ] && echo "VALID" || echo "INVALID"
[ "${TUNSET+1}" ] && echo "VALID" || echo "INVALID"

I dedicate this answer to @mklement0 (comments) who challenged me to answer the question accurately.

Reference http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02

Gilles ,Aug 31, 2010 at 7:30

To check whether a variable is set with a non-empty value, use [ -n "$x" ] , as others have already indicated.

Most of the time, it's a good idea to treat a variable that has an empty value in the same way as a variable that is unset. But you can distinguish the two if you need to: [ -n "${x+set}" ] ( "${x+set}" expands to set if x is set and to the empty string if x is unset).

To check whether a parameter has been passed, test $# , which is the number of parameters passed to the function (or to the script, when not in a function) (see Paul's answer ).

tripleee ,Sep 12, 2015 at 6:33

Read the "Parameter Expansion" section of the bash man page. Parameter expansion doesn't provide a general test for a variable being set, but there are several things you can do to a parameter if it isn't set.

For example:

function a {
    first_arg=${1-foo}
    # rest of the function
}

will set first_arg equal to $1 if it is assigned, otherwise it uses the value "foo". If a absolutely must take a single parameter, and no good default exists, you can exit with an error message when no parameter is given:

function a {
    : ${1?a must take a single argument}
    # rest of the function
}

(Note the use of : as a null command, which just expands the values of its arguments. We don't want to do anything with $1 in this example, just exit if it isn't set)

AlejandroVD ,Feb 8, 2016 at 13:31

In bash you can use -v inside the [[ ]] builtin:
#! /bin/bash -u

if [[ ! -v SOMEVAR ]]; then
    SOMEVAR='hello'
fi

echo $SOMEVAR

Palec ,Nov 16, 2016 at 15:01

For those that are looking to check for unset or empty when in a script with set -u :
if [ -z "${var-}" ]; then
   echo "Must provide var environment variable. Exiting...."
   exit 1
fi

The regular [ -z "$var" ] check will fail with var; unbound variable if set -u but [ -z "${var-}" ] expands to empty string if var is unset without failing.

user1387866 ,Jul 30 at 15:57

Note

I'm giving a heavily Bash-focused answer because of the bash tag.

Short answer

As long as you're only dealing with named variables in Bash, this function should always tell you if the variable has been set, even if it's an empty array.

is-variable-set() {
    declare -p $1 &>dev/null
}
Why this works

In Bash (at least as far back as 3.0), if var is a declared/set variable, then declare -p var outputs a declare command that would set variable var to whatever its current type and value are, and returns status code 0 (success). If var is undeclared, then declare -p var outputs an error message to stderr and returns status code 1 . Using &>/dev/null , redirects both regular stdout and stderr output to /dev/null , never to be seen, and without changing the status code. Thus the function only returns the status code.

Why other methods (sometimes) fail in Bash
  • [ -n "$var" ] : This only checks if ${var[0]} is nonempty. (In Bash, $var is the same as ${var[0]} .)
  • [ -n "${var+x}" ] : This only checks if ${var[0]} is set.
  • [ "${#var[@]}" != 0 ] : This only checks if at least one index of $var is set.
When this method fails in Bash

This only works for named variables (including $_ ), not certain special variables ( $! , $@ , $# , $$ , $* , $? , $- , $0 , $1 , $2 , ..., and any I may have forgotten). Since none of these are arrays, the POSIX-style [ -n "${var+x}" ] works for all of these special variables. But beware of wrapping it in a function since many special variables change values/existence when functions are called.

Shell compatibility note

If your script has arrays and you're trying to make it compatible with as many shells as possible, then consider using typeset -p instead of declare -p . I've read that ksh only supports the former, but haven't been able to test this. I do know that Bash 3.0+ and Zsh 5.5.1 each support both typeset -p and declare -p , differing only in which one is an alternative for the other. But I haven't tested differences beyond those two keywords, and I haven't tested other shells.

If you need your script to be POSIX sh compatible, then you can't use arrays. Without arrays, [ -n "{$var+x}" ] works.

Comparison code for different methods in Bash

This function unsets variable var , eval s the passed code, runs tests to determine if var is set by the eval d code, and finally shows the resulting status codes for the different tests.

I'm skipping test -v var , [ -v var ] , and [[ -v var ]] because they yield identical results to the POSIX standard [ -n "${var+x}" ] , while requiring Bash 4.2+. I'm also skipping typeset -p because it's the same as declare -p in the shells I've tested (Bash 3.0 thru 5.0, and Zsh 5.5.1).

is-var-set-after() {
    # Set var by passed expression.
    unset var
    eval "$1"

    # Run the tests, in increasing order of accuracy.
    [ -n "$var" ] # (index 0 of) var is nonempty
    nonempty=$?
    [ -n "${var+x}" ] # (index 0 of) var is set, maybe empty
    plus=$?
    [ "${#var[@]}" != 0 ] # var has at least one index set, maybe empty
    count=$?
    declare -p var &>/dev/null # var has been declared (any type)
    declared=$?

    # Show test results.
    printf '%30s: %2s %2s %2s %2s\n' "$1" $nonempty $plus $count $declared
}
Test case code

Note that test results may be unexpected due to Bash treating non-numeric array indices as "0" if the variable hasn't been declared as an associative array. Also, associative arrays are only valid in Bash 4.0+.

# Header.
printf '%30s: %2s %2s %2s %2s\n' "test" '-n' '+x' '#@' '-p'
# First 5 tests: Equivalent to setting 'var=foo' because index 0 of an
# indexed array is also the nonindexed value, and non-numerical
# indices in an array not declared as associative are the same as
# index 0.
is-var-set-after "var=foo"                        #  0  0  0  0
is-var-set-after "var=(foo)"                      #  0  0  0  0
is-var-set-after "var=([0]=foo)"                  #  0  0  0  0
is-var-set-after "var=([x]=foo)"                  #  0  0  0  0
is-var-set-after "var=([y]=bar [x]=foo)"          #  0  0  0  0
# '[ -n "$var" ]' fails when var is empty.
is-var-set-after "var=''"                         #  1  0  0  0
is-var-set-after "var=([0]='')"                   #  1  0  0  0
# Indices other than 0 are not detected by '[ -n "$var" ]' or by
# '[ -n "${var+x}" ]'.
is-var-set-after "var=([1]='')"                   #  1  1  0  0
is-var-set-after "var=([1]=foo)"                  #  1  1  0  0
is-var-set-after "declare -A var; var=([x]=foo)"  #  1  1  0  0
# Empty arrays are only detected by 'declare -p'.
is-var-set-after "var=()"                         #  1  1  1  0
is-var-set-after "declare -a var"                 #  1  1  1  0
is-var-set-after "declare -A var"                 #  1  1  1  0
# If 'var' is unset, then it even fails the 'declare -p var' test.
is-var-set-after "unset var"                      #  1  1  1  1
Test output

The test mnemonics in the header row correspond to [ -n "$var" ] , [ -n "${var+x}" ] , [ "${#var[@]}" != 0 ] , and declare -p var , respectively.

                         test: -n +x #@ -p
                      var=foo:  0  0  0  0
                    var=(foo):  0  0  0  0
                var=([0]=foo):  0  0  0  0
                var=([x]=foo):  0  0  0  0
        var=([y]=bar [x]=foo):  0  0  0  0
                       var='':  1  0  0  0
                 var=([0]=''):  1  0  0  0
                 var=([1]=''):  1  1  0  0
                var=([1]=foo):  1  1  0  0
declare -A var; var=([x]=foo):  1  1  0  0
                       var=():  1  1  1  0
               declare -a var:  1  1  1  0
               declare -A var:  1  1  1  0
                    unset var:  1  1  1  1
Summary
  • declare -p var &>/dev/null is (100%?) reliable for testing named variables in Bash since at least 3.0.
  • [ -n "${var+x}" ] is reliable in POSIX compliant situations, but cannot handle arrays.
  • Other tests exist for checking if a variable is nonempty, and for checking for declared variables in other shells. But these tests are suited for neither Bash nor POSIX scripts.

Peregring-lk ,Oct 18, 2014 at 22:09

Using [[ -z "$var" ]] is the easiest way to know if a variable was set or not, but that option -z doesn't distinguish between an unset variable and a variable set to an empty string:
$ set=''
$ [[ -z "$set" ]] && echo "Set" || echo "Unset" 
Unset
$ [[ -z "$unset" ]] && echo "Set" || echo "Unset"
Unset

It's best to check it according to the type of variable: env variable, parameter or regular variable.

For a env variable:

[[ $(env | grep "varname=" | wc -l) -eq 1 ]] && echo "Set" || echo "Unset"

For a parameter (for example, to check existence of parameter $5 ):

[[ $# -ge 5 ]] && echo "Set" || echo "Unset"

For a regular variable (using an auxiliary function, to do it in an elegant way):

function declare_var {
   declare -p "$1" &> /dev/null
}
declare_var "var_name" && echo "Set" || echo "Unset"

Notes:

  • $# : gives you the number of positional parameters.
  • declare -p : gives you the definition of the variable passed as a parameter. If it exists, returns 0, if not, returns 1 and prints an error message.
  • &> /dev/null : suppresses output from declare -p without affecting its return code.

Dennis Williamson ,Nov 27, 2013 at 20:56

You can do:
function a {
        if [ ! -z "$1" ]; then
                echo '$1 is set'
        fi
}

LavaScornedOven ,May 11, 2017 at 13:14

The answers above do not work when Bash option set -u is enabled. Also, they are not dynamic, e.g., how to test is variable with name "dummy" is defined? Try this:
is_var_defined()
{
    if [ $# -ne 1 ]
    then
        echo "Expected exactly one argument: variable name as string, e.g., 'my_var'"
        exit 1
    fi
    # Tricky.  Since Bash option 'set -u' may be enabled, we cannot directly test if a variable
    # is defined with this construct: [ ! -z "$var" ].  Instead, we must use default value
    # substitution with this construct: [ ! -z "${var:-}" ].  Normally, a default value follows the
    # operator ':-', but here we leave it blank for empty (null) string.  Finally, we need to
    # substitute the text from $1 as 'var'.  This is not allowed directly in Bash with this
    # construct: [ ! -z "${$1:-}" ].  We need to use indirection with eval operator.
    # Example: $1="var"
    # Expansion for eval operator: "[ ! -z \${$1:-} ]" -> "[ ! -z \${var:-} ]"
    # Code  execute: [ ! -z ${var:-} ]
    eval "[ ! -z \${$1:-} ]"
    return $?  # Pedantic.
}

Related: In Bash, how do I test if a variable is defined in "-u" mode

Aquarius Power ,Nov 15, 2014 at 17:55

My prefered way is this:
$var=10
$if ! ${var+false};then echo "is set";else echo "NOT set";fi
is set
$unset var
$if ! ${var+false};then echo "is set";else echo "NOT set";fi
NOT set

So basically, if a variable is set, it becomes "a negation of the resulting false " (what will be true = "is set").

And, if it is unset, it will become "a negation of the resulting true " (as the empty result evaluates to true ) (so will end as being false = "NOT set").

kenorb ,Sep 22, 2014 at 13:57

In a shell you can use the -z operator which is True if the length of string is zero.

A simple one-liner to set default MY_VAR if it's not set, otherwise optionally you can display the message:

[[ -z "$MY_VAR" ]] && MY_VAR="default"
[[ -z "$MY_VAR" ]] && MY_VAR="default" || echo "Variable already set."

Zlatan ,Nov 20, 2013 at 18:53

if [[ ${1:+isset} ]]
then echo "It was set and not null." >&2
else echo "It was not set or it was null." >&2
fi

if [[ ${1+isset} ]]
then echo "It was set but might be null." >&2
else echo "It was was not set." >&2
fi

solidsnack ,Nov 30, 2013 at 16:47

I found a (much) better code to do this if you want to check for anything in $@ .
if [[ $1 = "" ]]
then
  echo '$1 is blank'
else
  echo '$1 is filled up'
fi

Why this all? Everything in $@ exists in Bash, but by default it's blank, so test -z and test -n couldn't help you.

Update: You can also count number of characters in a parameters.

if [ ${#1} = 0 ]
then
  echo '$1 is blank'
else
  echo '$1 is filled up'
fi

Steven Penny ,May 11, 2014 at 4:59

[[ $foo ]]

Or

(( ${#foo} ))

Or

let ${#foo}

Or

declare -p foo

Celeo ,Feb 11, 2015 at 20:58

if [[ ${!xx[@]} ]] ; then echo xx is defined; fi

HelloGoodbye ,Nov 29, 2013 at 22:41

I always use this one, based on the fact that it seems easy to be understood by anybody who sees the code for the very first time:
if [ "$variable" = "" ]
    then
    echo "Variable X is empty"
fi

And, if wanting to check if not empty;

if [ ! "$variable" = "" ]
    then
    echo "Variable X is not empty"
fi

That's it.

fr00tyl00p ,Nov 29, 2015 at 20:26

This is what I use every day:
#
# Check if a variable is set
#   param1  name of the variable
#
function is_set()
{
    [[ -n "${1}" ]] && test -n "$(eval "echo "\${${1}+x}"")"
}

This works well under Linux and Solaris down to bash 3.0.

bash-3.00$ myvar="TEST"
bash-3.00$ is_set myvar ; echo $?
0
bash-3.00$ mavar=""
bash-3.00$ is_set myvar ; echo $?
0
bash-3.00$ unset myvar
bash-3.00$ is_set myvar ; echo $?
1

Daniel S ,Mar 1, 2016 at 13:12

I like auxiliary functions to hide the crude details of bash. In this case, doing so adds even more (hidden) crudeness:
# The first ! negates the result (can't use -n to achieve this)
# the second ! expands the content of varname (can't do ${$varname})
function IsDeclared_Tricky
{
  local varname="$1"
  ! [ -z ${!varname+x} ]
}

Because I first had bugs in this implementation (inspired by the answers of Jens and Lionel), I came up with a different solution:

# Ask for the properties of the variable - fails if not declared
function IsDeclared()
{
  declare -p $1 &>/dev/null
}

I find it to be more straight-forward, more bashy and easier to understand/remember. Test case shows it is equivalent:

function main()
{
  declare -i xyz
  local foo
  local bar=
  local baz=''

  IsDeclared_Tricky xyz; echo "IsDeclared_Tricky xyz: $?"
  IsDeclared_Tricky foo; echo "IsDeclared_Tricky foo: $?"
  IsDeclared_Tricky bar; echo "IsDeclared_Tricky bar: $?"
  IsDeclared_Tricky baz; echo "IsDeclared_Tricky baz: $?"

  IsDeclared xyz; echo "IsDeclared xyz: $?"
  IsDeclared foo; echo "IsDeclared foo: $?"
  IsDeclared bar; echo "IsDeclared bar: $?"
  IsDeclared baz; echo "IsDeclared baz: $?"
}

main

The test case also shows that local var does NOT declare var (unless followed by '='). For quite some time I thought i declared variables this way, just to discover now that i merely expressed my intention... It's a no-op, i guess.

IsDeclared_Tricky xyz: 1
IsDeclared_Tricky foo: 1
IsDeclared_Tricky bar: 0
IsDeclared_Tricky baz: 0
IsDeclared xyz: 1
IsDeclared foo: 1
IsDeclared bar: 0
IsDeclared baz: 0

BONUS: usecase

I mostly use this test to give (and return) parameters to functions in a somewhat "elegant" and safe way (almost resembling an interface...):

#auxiliary functions
function die()
{
  echo "Error: $1"; exit 1
}

function assertVariableDeclared()
{
  IsDeclared "$1" || die "variable not declared: $1"
}

function expectVariables()
{
  while (( $# > 0 )); do
    assertVariableDeclared $1; shift
  done
}

# actual example
function exampleFunction()
{
  expectVariables inputStr outputStr
  outputStr="$inputStr world!"
}

function bonus()
{
  local inputStr='Hello'
  local outputStr= # remove this to trigger error
  exampleFunction
  echo $outputStr
}

bonus

If called with all requires variables declared:

Hello world!

else:

Error: variable not declared: outputStr

Hatem Jaber ,Jun 13 at 12:08

After skimming all the answers, this also works:
if [[ -z $SOME_VAR ]]; then read -p "Enter a value for SOME_VAR: " SOME_VAR; fi
echo "SOME_VAR=$SOME_VAR"

if you don't put SOME_VAR instead of what I have $SOME_VAR , it will set it to an empty value; $ is necessary for this to work.

Keith Thompson ,Aug 5, 2013 at 19:10

If you wish to test that a variable is bound or unbound, this works well, even after you've turned on the nounset option:
set -o noun set

if printenv variableName >/dev/null; then
    # variable is bound to a value
else
    # variable is unbound
fi

> ,Jan 30 at 18:23

Functions to check if variable is declared/unset including empty $array=()


The following functions test if the given name exists as a variable

# The first parameter needs to be the name of the variable to be checked.
# (See example below)

var_is_declared() {
    { [[ -n ${!1+anything} ]] || declare -p $1 &>/dev/null;}
}

var_is_unset() {
    { [[ -z ${!1+anything} ]] && ! declare -p $1 &>/dev/null;} 
}

This functions would test as showed in the following conditions:

a;       # is not declared
a=;      # is declared
a="foo"; # is declared
a=();    # is declared
a=("");  # is declared
unset a; # is not declared

a;       # is unset
a=;      # is not unset
a="foo"; # is not unset
a=();    # is not unset
a=("");  # is not unset
unset a; # is unset

.

For more details

and a test script see my answer to the question "How do I check if a variable exists in bash?" .

Remark: The similar usage of declare -p , as it is also shown by Peregring-lk 's answer , is truly coincidental. Otherwise I would of course have credited it!

[Aug 20, 2019] Is it possible to insert separator in midnight commander menu?

Jun 07, 2010 | superuser.com

Ask Question Asked 9 years, 2 months ago Active 7 years, 10 months ago Viewed 363 times 2

okutane ,Jun 7, 2010 at 3:36

I want to insert some items into mc menu (which is opened by F2) grouped together. Is it possible to insert some sort of separator before them or put them into some submenu?
Probably, not.
The format of the menu file is very simple. Lines that start with anything but
space or tab are considered entries for the menu (in order to be able to use
it like a hot key, the first character should be a letter). All the lines that
start with a space or a tab are the commands that will be executed when the
entry is selected.

But MC allows you to make multiple menu entries with same shortcut and title, so you can make a menu entry that looks like separator and does nothing, like:

a hello
  echo world
- --------
b world
  echo hello
- --------
c superuser
  ls /

This will look like:

[Aug 20, 2019] Midnight Commander, using date in User menu

Dec 31, 2013 | unix.stackexchange.com

user2013619 ,Dec 31, 2013 at 0:43

I would like to use MC (midnight commander) to compress the selected dir with date in its name, e.g: dirname_20131231.tar.gz

The command in the User menu is :

tar -czf dirname_`date '+%Y%m%d'`.tar.gz %d

The archive is missing because %m , and %d has another meaning in MC. I made an alias for the date, but it also doesn't work.

Does anybody solved this problem ever?

John1024 ,Dec 31, 2013 at 1:06

To escape the percent signs, double them:
tar -czf dirname_$(date '+%%Y%%m%%d').tar.gz %d

The above would compress the current directory (%d) to a file also in the current directory. If you want to compress the directory pointed to by the cursor rather than the current directory, use %f instead:

tar -czf %f_$(date '+%%Y%%m%%d').tar.gz %f

mc handles escaping of special characters so there is no need to put %f in quotes.

By the way, midnight commander's special treatment of percent signs occurs not just in the user menu file but also at the command line. This is an issue when using shell commands with constructs like ${var%.c} . At the command line, the same as in the user menu file, percent signs can be escaped by doubling them.

[Aug 19, 2019] mc - Is there are any documentation about user-defined menu in midnight-commander - Unix Linux Stack Exchange

Aug 19, 2019 | unix.stackexchange.com

Is there are any documentation about user-defined menu in midnight-commander? Ask Question Asked 5 years, 2 months ago Active 1 year, 2 months ago Viewed 3k times 6 2


login ,Jun 11, 2014 at 13:13

I'd like to create my own user-defined menu for mc ( menu file). I see some lines like
+ t r & ! t t

or

+ t t

What does it mean?

goldilocks ,Jun 11, 2014 at 13:35

It is documented in the help, the node is "Edit Menu File" under "Command Menu"; if you scroll down you should find "Addition Conditions":

If the condition begins with '+' (or '+?') instead of '=' (or '=?') it is an addition condition. If the condition is true the menu entry will be included in the menu. If the condition is false the menu entry will not be included in the menu.

This is preceded by "Default conditions" (the = condition), which determine which entry will be highlighted as the default choice when the menu appears. Anyway, by way of example:

+ t r & ! t t

t r means if this is a regular file ("t(ype) r"), and ! t t means if the file has not been tagged in the interface.

Jarek

On top what has been written above, this page can be browsed in the Internet, when searching for man pages, e.g.: https://www.systutorials.com/docs/linux/man/1-mc/

Search for "Menu File Edit" .

Best regards, Jarek

[Aug 14, 2019] bash - PID background process - Unix Linux Stack Exchange

Aug 14, 2019 | unix.stackexchange.com

PID background process Ask Question Asked 2 years, 8 months ago Active 2 years, 8 months ago Viewed 2k times 2


Raul ,Nov 27, 2016 at 18:21

As I understand pipes and commands, bash takes each command, spawns a process for each one and connects stdout of the previous one with the stdin of the next one.

For example, in "ls -lsa | grep feb", bash will create two processes, and connect the output of "ls -lsa" to the input of "grep feb".

When you execute a background command like "sleep 30 &" in bash, you get the pid of the background process running your command. Surprisingly for me, when I wrote "ls -lsa | grep feb &" bash returned only one PID.

How should this be interpreted? A process runs both "ls -lsa" and "grep feb"? Several process are created but I only get the pid of one of them?

Raul ,Nov 27, 2016 at 19:21

Spawns 2 processes. The & displays the PID of the second process. Example below.
$ echo $$
13358
$ sleep 100 | sleep 200 &
[1] 13405
$ ps -ef|grep 13358
ec2-user 13358 13357  0 19:02 pts/0    00:00:00 -bash
ec2-user 13404 13358  0 19:04 pts/0    00:00:00 sleep 100
ec2-user 13405 13358  0 19:04 pts/0    00:00:00 sleep 200
ec2-user 13406 13358  0 19:04 pts/0    00:00:00 ps -ef
ec2-user 13407 13358  0 19:04 pts/0    00:00:00 grep --color=auto 13358
$

> ,

When you run a job in the background, bash prints the process ID of its subprocess, the one that runs the command in that job. If that job happens to create more subprocesses, that's none of the parent shell's business.

When the background job is a pipeline (i.e. the command is of the form something1 | something2 & , and not e.g. { something1 | something2; } & ), there's an optimization which is strongly suggested by POSIX and performed by most shells including bash: each of the elements of the pipeline are executed directly as subprocesses of the original shell. What POSIX mandates is that the variable $! is set to the last command in the pipeline in this case. In most shells, that last command is a subprocess of the original process, and so are the other commands in the pipeline.

When you run ls -lsa | grep feb , there are three processes involved: the one that runs the left-hand side of the pipe (a subshell that finishes setting up the pipe then executes ls ), the one that runs the right-hand side of the pipe (a subshell that finishes setting up the pipe then executes grep ), and the original process that waits for the pipe to finish.

You can watch what happens by tracing the processes:

$ strace -f -e clone,wait4,pipe,execve,setpgid bash --norc
execve("/usr/local/bin/bash", ["bash", "--norc"], [/* 82 vars */]) = 0
setpgid(0, 24084)                       = 0
bash-4.3$ sleep 10 | sleep 20 &

Note how the second sleep is reported and stored as $! , but the process group ID is the first sleep . Dash has the same oddity, ksh and mksh don't.

[Aug 14, 2019] unix - How to get PID of process by specifying process name and store it in a variable to use further - Stack Overflow

Aug 14, 2019 | stackoverflow.com

Nidhi ,Nov 28, 2014 at 0:54

pids=$(pgrep <name>)

will get you the pids of all processes with the given name. To kill them all, use

kill -9 $pids

To refrain from using a variable and directly kill all processes with a given name issue

pkill -9 <name>

panticz.de ,Nov 11, 2016 at 10:11

On a single line...
pgrep -f process_name | xargs kill -9

flazzarini ,Jun 13, 2014 at 9:54

Another possibility would be to use pidof it usually comes with most distributions. It will return you the PID of a given process by using it's name.
pidof process_name

This way you could store that information in a variable and execute kill -9 on it.

#!/bin/bash
pid=`pidof process_name`
kill -9 $pid

Pawel K ,Dec 20, 2017 at 10:27

use grep [n]ame to remove that grep -v name this is first... Sec using xargs in the way how it is up there is wrong to rnu whatever it is piped you have to use -i ( interactive mode) otherwise you may have issues with the command.

ps axf | grep | grep -v grep | awk '{print "kill -9 " $1}' ? ps aux |grep [n]ame | awk '{print "kill -9 " $2}' ? isnt that better ?

[Aug 14, 2019] linux - How to get PID of background process - Stack Overflow

Highly recommended!
Aug 14, 2019 | stackoverflow.com

How to get PID of background process? Ask Question Asked 9 years, 8 months ago Active 7 months ago Viewed 238k times 336 64


pixelbeat ,Mar 20, 2013 at 9:11

I start a background process from my shell script, and I would like to kill this process when my script finishes.

How to get the PID of this process from my shell script? As far as I can see variable $! contains the PID of the current script, not the background process.

WiSaGaN ,Jun 2, 2015 at 14:40

You need to save the PID of the background process at the time you start it:
foo &
FOO_PID=$!
# do other stuff
kill $FOO_PID

You cannot use job control, since that is an interactive feature and tied to a controlling terminal. A script will not necessarily have a terminal attached at all so job control will not necessarily be available.

Phil ,Dec 2, 2017 at 8:01

You can use the jobs -l command to get to a particular jobL
^Z
[1]+  Stopped                 guard

my_mac:workspace r$ jobs -l
[1]+ 46841 Suspended: 18           guard

In this case, 46841 is the PID.

From help jobs :

-l Report the process group ID and working directory of the jobs.

jobs -p is another option which shows just the PIDs.

Timo ,Dec 2, 2017 at 8:03

Here's a sample transcript from a bash session ( %1 refers to the ordinal number of background process as seen from jobs ):

$ echo $$
3748

$ sleep 100 &
[1] 192

$ echo $!
192

$ kill %1

[1]+  Terminated              sleep 100

lepe ,Dec 2, 2017 at 8:29

An even simpler way to kill all child process of a bash script:
pkill -P $$

The -P flag works the same way with pkill and pgrep - it gets child processes, only with pkill the child processes get killed and with pgrep child PIDs are printed to stdout.

Luis Ramirez ,Feb 20, 2013 at 23:11

this is what I have done. Check it out, hope it can help.
#!/bin/bash
#
# So something to show.
echo "UNO" >  UNO.txt
echo "DOS" >  DOS.txt
#
# Initialize Pid List
dPidLst=""
#
# Generate background processes
tail -f UNO.txt&
dPidLst="$dPidLst $!"
tail -f DOS.txt&
dPidLst="$dPidLst $!"
#
# Report process IDs
echo PID=$$
echo dPidLst=$dPidLst
#
# Show process on current shell
ps -f
#
# Start killing background processes from list
for dPid in $dPidLst
do
        echo killing $dPid. Process is still there.
        ps | grep $dPid
        kill $dPid
        ps | grep $dPid
        echo Just ran "'"ps"'" command, $dPid must not show again.
done

Then just run it as: ./bgkill.sh with proper permissions of course

root@umsstd22 [P]:~# ./bgkill.sh
PID=23757
dPidLst= 23758 23759
UNO
DOS
UID        PID  PPID  C STIME TTY          TIME CMD
root      3937  3935  0 11:07 pts/5    00:00:00 -bash
root     23757  3937  0 11:55 pts/5    00:00:00 /bin/bash ./bgkill.sh
root     23758 23757  0 11:55 pts/5    00:00:00 tail -f UNO.txt
root     23759 23757  0 11:55 pts/5    00:00:00 tail -f DOS.txt
root     23760 23757  0 11:55 pts/5    00:00:00 ps -f
killing 23758. Process is still there.
23758 pts/5    00:00:00 tail
./bgkill.sh: line 24: 23758 Terminated              tail -f UNO.txt
Just ran 'ps' command, 23758 must not show again.
killing 23759. Process is still there.
23759 pts/5    00:00:00 tail
./bgkill.sh: line 24: 23759 Terminated              tail -f DOS.txt
Just ran 'ps' command, 23759 must not show again.
root@umsstd22 [P]:~# ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
root      3937  3935  0 11:07 pts/5    00:00:00 -bash
root     24200  3937  0 11:56 pts/5    00:00:00 ps -f

Phil ,Oct 15, 2013 at 18:22

You might also be able to use pstree:
pstree -p user

This typically gives a text representation of all the processes for the "user" and the -p option gives the process-id. It does not depend, as far as I understand, on having the processes be owned by the current shell. It also shows forks.

Phil ,Dec 4, 2018 at 9:46

pgrep can get you all of the child PIDs of a parent process. As mentioned earlier $$ is the current scripts PID. So, if you want a script that cleans up after itself, this should do the trick:
trap 'kill $( pgrep -P $$ | tr "\n" " " )' SIGINT SIGTERM EXIT

[Aug 10, 2019] How to check the file size in Linux-Unix bash shell scripting by Vivek Gite

Aug 10, 2019 | www.cyberciti.biz

The stat command shows information about the file. The syntax is as follows to get the file size on GNU/Linux stat:

stat -c %s "/etc/passwd"

OR

stat --format=%s "/etc/passwd"

[Aug 10, 2019] bash - How to check size of a file - Stack Overflow

Aug 10, 2019 | stackoverflow.com

[ -n file.txt ] doesn't check its size , it checks that the string file.txt is non-zero length, so it will always succeed.

If you want to say " size is non-zero", you need [ -s file.txt ] .

To get a file's size , you can use wc -c to get the size ( file length) in bytes:

file=file.txt
minimumsize=90000
actualsize=$(wc -c <"$file")
if [ $actualsize -ge $minimumsize ]; then
    echo size is over $minimumsize bytes
else
    echo size is under $minimumsize bytes
fi

In this case, it sounds like that's what you want.

But FYI, if you want to know how much disk space the file is using, you could use du -k to get the size (disk space used) in kilobytes:

file=file.txt
minimumsize=90
actualsize=$(du -k "$file" | cut -f 1)
if [ $actualsize -ge $minimumsize ]; then
    echo size is over $minimumsize kilobytes
else
    echo size is under $minimumsize kilobytes
fi

If you need more control over the output format, you can also look at stat . On Linux, you'd start with something like stat -c '%s' file.txt , and on BSD/Mac OS X, something like stat -f '%z' file.txt .

--Mikel

On Linux, you'd start with something like stat -c '%s' file.txt , and on BSD/Mac OS X, something like stat -f '%z' file.txt .

Oz Solomon ,Jun 13, 2014 at 21:44

It surprises me that no one mentioned stat to check file size. Some methods are definitely better: using -s to find out whether the file is empty or not is easier than anything else if that's all you want. And if you want to find files of a size, then find is certainly the way to go.

I also like du a lot to get file size in kb, but, for bytes, I'd use stat :

size=$(stat -f%z $filename) # BSD stat

size=$(stat -c%s $filename) # GNU stat?
alternative solution with awk and double parenthesis:
FILENAME=file.txt
SIZE=$(du -sb $FILENAME | awk '{ print $1 }')

if ((SIZE<90000)) ; then 
    echo "less"; 
else 
    echo "not less"; 
fi

[Aug 10, 2019] command line - How do I add file and directory comparision option to mc user menu - Ask Ubuntu

Aug 10, 2019 | askubuntu.com

How do I add file and directory comparision option to mc user menu? Ask Question Asked 7 years, 4 months ago Active 7 years, 3 months ago Viewed 664 times 0

sorin ,Mar 30, 2012 at 8:57

I want to add Beyond Compare diff to mc (midnight commmander) user menu.

All I know is that I need to add my custom command to ~/.mc/menu but I have no idea about the syntax to use.

I want to be able to compare two files from the two panes or the directories themselves.

The command that I need to run is bcompare file1 file2 & (some for directories, it will figure it out).

mivk ,Oct 17, 2015 at 15:35

Add this to ~/.mc/menu :
+ t r & ! t t
d       Diff against file of same name in other directory
        if [ "%d" = "%D" ]; then
          echo "The two directores must be different"
          exit 1
        fi
        if [ -f %D/%f ]; then        # if two of them, then
          bcompare %f %D/%f &
        else
          echo %f: No copy in %D/%f
        fi

x       Diff file to file
        if [ -f %D/%F ]; then        # if two of them, then
          bcompare %f %D/%F &
        else
          echo %f: No copy in %D/%f
        fi

D       Diff current directory against other directory
        if [ "%d" = "%D" ]; then
          echo "The two directores must be different"
          exit 1
        fi
        bcompare %d %D &

[Aug 10, 2019] mc - Is there are any documentation about user-defined menu in midnight-commander - Unix Linux Stack Exchange

Aug 10, 2019 | unix.stackexchange.com

Is there are any documentation about user-defined menu in midnight-commander? Ask Question Asked 5 years, 2 months ago Active 1 year, 1 month ago Viewed 3k times 6 2


login ,Jun 11, 2014 at 13:13

I'd like to create my own user-defined menu for mc ( menu file). I see some lines like
+ t r & ! t t

or

+ t t

What does it mean?

goldilocks ,Jun 11, 2014 at 13:35

It is documented in the help, the node is "Edit Menu File" under "Command Menu"; if you scroll down you should find "Addition Conditions":

If the condition begins with '+' (or '+?') instead of '=' (or '=?') it is an addition condition. If the condition is true the menu entry will be included in the menu. If the condition is false the menu entry will not be included in the menu.

This is preceded by "Default conditions" (the = condition), which determine which entry will be highlighted as the default choice when the menu appears. Anyway, by way of example:

+ t r & ! t t

t r means if this is a regular file ("t(ype) r"), and ! t t means if the file has not been tagged in the interface.

> ,

On top what has been written above, this page can be browsed in the Internet, when searching for man pages, e.g.: https://www.systutorials.com/docs/linux/man/1-mc/

Search for "Menu File Edit" .

Best regards, Jarek

[Aug 10, 2019] midnight commander - How to configure coloring of the file names in MC - Super User

If colors are crazy, the simplest way to solve this problem is to turn them off
To turn off color you can also use option mc --nocolor or by by using the -b flag
You can customize the color displayed by defining them in ~/.mc/ini . But that requres some work. Have a look here for an example: http://ajnasz.hu/blog/20080101/midnight-commander-coloring .
Aug 10, 2019 | superuser.com
How to configure coloring of the file names in MC? Ask Question Asked 8 years, 7 months ago Active 1 year, 4 months ago Viewed 4k times 8 3

Mike L. ,Jan 9, 2011 at 17:21

Is it possible to configure the Midnight Commander (Ubuntu 10.10) to show certain file and directory names differently, e.g. all hidden (starting with a period) using grey color?

Mike L. ,Feb 20, 2018 at 5:51

Under Options -> Panel Options select File highlight -> File types .

See man mc in the Colors section for ways to choose particular colors by adding entries in your ~/.config/mc/ini file. Unfortunately, there doesn't appear to be a keyword for hidden files.

[Aug 07, 2019] Find files and tar them (with spaces)

Aug 07, 2019 | stackoverflow.com

Ask Question Asked 8 years, 3 months ago Active 1 month ago Viewed 104k times 106 45


porges ,Sep 6, 2012 at 17:43

Alright, so simple problem here. I'm working on a simple back up code. It works fine except if the files have spaces in them. This is how I'm finding files and adding them to a tar archive:
find . -type f | xargs tar -czvf backup.tar.gz

The problem is when the file has a space in the name because tar thinks that it's a folder. Basically is there a way I can add quotes around the results from find? Or a different way to fix this?

Brad Parks ,Mar 2, 2017 at 18:35

Use this:
find . -type f -print0 | tar -czvf backup.tar.gz --null -T -

It will:

Also see:

czubehead ,Mar 19, 2018 at 11:51

There could be another way to achieve what you want. Basically,
  1. Use the find command to output path to whatever files you're looking for. Redirect stdout to a filename of your choosing.
  2. Then tar with the -T option which allows it to take a list of file locations (the one you just created with find!)
    find . -name "*.whatever" > yourListOfFiles
    tar -cvf yourfile.tar -T yourListOfFiles
    

gsteff ,May 5, 2011 at 2:05

Try running:
    find . -type f | xargs -d "\n" tar -czvf backup.tar.gz

Caleb Kester ,Oct 12, 2013 at 20:41

Why not:
tar czvf backup.tar.gz *

Sure it's clever to use find and then xargs, but you're doing it the hard way.

Update: Porges has commented with a find-option that I think is a better answer than my answer, or the other one: find -print0 ... | xargs -0 ....

Kalibur x ,May 19, 2016 at 13:54

If you have multiple files or directories and you want to zip them into independent *.gz file you can do this. Optional -type f -atime
find -name "httpd-log*.txt" -type f -mtime +1 -exec tar -vzcf {}.gz {} \;

This will compress

httpd-log01.txt
httpd-log02.txt

to

httpd-log01.txt.gz
httpd-log02.txt.gz

Frank Eggink ,Apr 26, 2017 at 8:28

Why not give something like this a try: tar cvf scala.tar `find src -name *.scala`

tommy.carstensen ,Dec 10, 2017 at 14:55

Another solution as seen here :
find var/log/ -iname "anaconda.*" -exec tar -cvzf file.tar.gz {} +

Robino ,Sep 22, 2016 at 14:26

The best solution seem to be to create a file list and then archive files because you can use other sources and do something else with the list.

For example this allows using the list to calculate size of the files being archived:

#!/bin/sh

backupFileName="backup-big-$(date +"%Y%m%d-%H%M")"
backupRoot="/var/www"
backupOutPath=""

archivePath=$backupOutPath$backupFileName.tar.gz
listOfFilesPath=$backupOutPath$backupFileName.filelist

#
# Make a list of files/directories to archive
#
echo "" > $listOfFilesPath
echo "${backupRoot}/uploads" >> $listOfFilesPath
echo "${backupRoot}/extra/user/data" >> $listOfFilesPath
find "${backupRoot}/drupal_root/sites/" -name "files" -type d >> $listOfFilesPath

#
# Size calculation
#
sizeForProgress=`
cat $listOfFilesPath | while read nextFile;do
    if [ ! -z "$nextFile" ]; then
        du -sb "$nextFile"
    fi
done | awk '{size+=$1} END {print size}'
`

#
# Archive with progress
#
## simple with dump of all files currently archived
#tar -czvf $archivePath -T $listOfFilesPath
## progress bar
sizeForShow=$(($sizeForProgress/1024/1024))
echo -e "\nRunning backup [source files are $sizeForShow MiB]\n"
tar -cPp -T $listOfFilesPath | pv -s $sizeForProgress | gzip > $archivePath

user3472383 ,Jun 27 at 1:11

Would add a comment to @Steve Kehlet post but need 50 rep (RIP).

For anyone that has found this post through numerous googling, I found a way to not only find specific files given a time range, but also NOT include the relative paths OR whitespaces that would cause tarring errors. (THANK YOU SO MUCH STEVE.)

find . -name "*.pdf" -type f -mtime 0 -printf "%f\0" | tar -czvf /dir/zip.tar.gz --null -T -
  1. . relative directory
  2. -name "*.pdf" look for pdfs (or any file type)
  3. -type f type to look for is a file
  4. -mtime 0 look for files created in last 24 hours
  5. -printf "%f\0" Regular -print0 OR -printf "%f" did NOT work for me. From man pages:

This quoting is performed in the same way as for GNU ls. This is not the same quoting mechanism as the one used for -ls and -fls. If you are able to decide what format to use for the output of find then it is normally better to use '\0' as a terminator than to use newline, as file names can contain white space and newline characters.

  1. -czvf create archive, filter the archive through gzip , verbosely list files processed, archive name

[Aug 06, 2019] Tar archiving that takes input from a list of files>

Aug 06, 2019 | stackoverflow.com

Tar archiving that takes input from a list of files Ask Question Asked 7 years, 9 months ago Active 6 months ago Viewed 123k times 131 29


Kurt McKee ,Apr 29 at 10:22

I have a file that contain list of files I want to archive with tar. Let's call it mylist.txt

It contains:

file1.txt
file2.txt
...
file10.txt

Is there a way I can issue TAR command that takes mylist.txt as input? Something like

tar -cvf allfiles.tar -[someoption?] mylist.txt

So that it is similar as if I issue this command:

tar -cvf allfiles.tar file1.txt file2.txt file10.txt

Stphane ,May 25 at 0:11

Yes:
tar -cvf allfiles.tar -T mylist.txt

drue ,Jun 23, 2014 at 14:56

Assuming GNU tar (as this is Linux), the -T or --files-from option is what you want.

Stphane ,Mar 1, 2016 at 20:28

You can also pipe in the file names which might be useful:
find /path/to/files -name \*.txt | tar -cvf allfiles.tar -T -

David C. Rankin ,May 31, 2018 at 18:27

Some versions of tar, for example, the default versions on HP-UX (I tested 11.11 and 11.31), do not include a command line option to specify a file list, so a decent work-around is to do this:
tar cvf allfiles.tar $(cat mylist.txt)

Jan ,Sep 25, 2015 at 20:18

On Solaris, you can use the option -I to read the filenames that you would normally state on the command line from a file. In contrast to the command line, this can create tar archives with hundreds of thousands of files (just did that).

So the example would read

tar -cvf allfiles.tar -I mylist.txt

,

For me on AIX, it worked as follows:
tar -L List.txt -cvf BKP.tar

[Aug 06, 2019] Shell command to tar directory excluding certain files-folders

Aug 06, 2019 | stackoverflow.com

Shell command to tar directory excluding certain files/folders Ask Question Asked 10 years, 1 month ago Active 1 month ago Viewed 787k times 720 186


Rekhyt ,Jun 24, 2014 at 16:06

Is there a simple shell command/script that supports excluding certain files/folders from being archived?

I have a directory that need to be archived with a sub directory that has a number of very large files I do not need to backup.

Not quite solutions:

The tar --exclude=PATTERN command matches the given pattern and excludes those files, but I need specific files & folders to be ignored (full file path), otherwise valid files might be excluded.

I could also use the find command to create a list of files and exclude the ones I don't want to archive and pass the list to tar, but that only works with for a small amount of files. I have tens of thousands.

I'm beginning to think the only solution is to create a file with a list of files/folders to be excluded, then use rsync with --exclude-from=file to copy all the files to a tmp directory, and then use tar to archive that directory.

Can anybody think of a better/more efficient solution?

EDIT: Charles Ma 's solution works well. The big gotcha is that the --exclude='./folder' MUST be at the beginning of the tar command. Full command (cd first, so backup is relative to that directory):

cd /folder_to_backup
tar --exclude='./folder' --exclude='./upload/folder2' -zcvf /backup/filename.tgz .

James O'Brien ,Nov 24, 2016 at 9:55

You can have multiple exclude options for tar so
$ tar --exclude='./folder' --exclude='./upload/folder2' -zcvf /backup/filename.tgz .

etc will work. Make sure to put --exclude before the source and destination items.

Johan Soderberg ,Jun 11, 2009 at 23:10

You can exclude directories with --exclude for tar.

If you want to archive everything except /usr you can use:

tar -zcvf /all.tgz / --exclude=/usr

In your case perhaps something like

tar -zcvf archive.tgz arc_dir --exclude=dir/ignore_this_dir

cstamas ,Oct 8, 2018 at 18:02

Possible options to exclude files/directories from backup using tar:

Exclude files using multiple patterns

tar -czf backup.tar.gz --exclude=PATTERN1 --exclude=PATTERN2 ... /path/to/backup

Exclude files using an exclude file filled with a list of patterns

tar -czf backup.tar.gz -X /path/to/exclude.txt /path/to/backup

Exclude files using tags by placing a tag file in any directory that should be skipped

tar -czf backup.tar.gz --exclude-tag-all=exclude.tag /path/to/backup

Anish Ramaswamy ,Apr 1 at 16:18

old question with many answers, but I found that none were quite clear enough for me, so I would like to add my try.

if you have the following structure

/home/ftp/mysite/

with following file/folders

/home/ftp/mysite/file1
/home/ftp/mysite/file2
/home/ftp/mysite/file3
/home/ftp/mysite/folder1
/home/ftp/mysite/folder2
/home/ftp/mysite/folder3

so, you want to make a tar file that contain everyting inside /home/ftp/mysite (to move the site to a new server), but file3 is just junk, and everything in folder3 is also not needed, so we will skip those two.

we use the format

tar -czvf <name of tar file> <what to tar> <any excludes>

where the c = create, z = zip, and v = verbose (you can see the files as they are entered, usefull to make sure none of the files you exclude are being added). and f= file.

so, my command would look like this

cd /home/ftp/
tar -czvf mysite.tar.gz mysite --exclude='file3' --exclude='folder3'

note the files/folders excluded are relatively to the root of your tar (I have tried full path here relative to / but I can not make that work).

hope this will help someone (and me next time I google it)

not2qubit ,Apr 4, 2018 at 3:24

You can use standard "ant notation" to exclude directories relative.
This works for me and excludes any .git or node_module directories.
tar -cvf myFile.tar --exclude=**/.git/* --exclude=**/node_modules/*  -T /data/txt/myInputFile.txt 2> /data/txt/myTarLogFile.txt

myInputFile.txt Contains:

/dev2/java
/dev2/javascript

GeertVc ,Feb 9, 2015 at 13:37

I've experienced that, at least with the Cygwin version of tar I'm using ("CYGWIN_NT-5.1 1.7.17(0.262/5/3) 2012-10-19 14:39 i686 Cygwin" on a Windows XP Home Edition SP3 machine), the order of options is important.

While this construction worked for me:

tar cfvz target.tgz --exclude='<dir1>' --exclude='<dir2>' target_dir

that one didn't work:

tar cfvz --exclude='<dir1>' --exclude='<dir2>' target.tgz target_dir

This, while tar --help reveals the following:

tar [OPTION...] [FILE]

So, the second command should also work, but apparently it doesn't seem to be the case...

Best rgds,

Scott Stensland ,Feb 12, 2015 at 20:55

This exclude pattern handles filename suffix like png or mp3 as well as directory names like .git and node_modules
tar --exclude={*.png,*.mp3,*.wav,.git,node_modules} -Jcf ${target_tarball}  ${source_dirname}

Michael ,May 18 at 23:29

I found this somewhere else so I won't take credit, but it worked better than any of the solutions above for my mac specific issues (even though this is closed):
tar zc --exclude __MACOSX --exclude .DS_Store -f <archive> <source(s)>

J. Lawson ,Apr 17, 2018 at 23:28

For those who have issues with it, some versions of tar would only work properly without the './' in the exclude value.
Tar --version

tar (GNU tar) 1.27.1

Command syntax that work:

tar -czvf ../allfiles-butsome.tar.gz * --exclude=acme/foo

These will not work:

$ tar -czvf ../allfiles-butsome.tar.gz * --exclude=./acme/foo
$ tar -czvf ../allfiles-butsome.tar.gz * --exclude='./acme/foo'
$ tar --exclude=./acme/foo -czvf ../allfiles-butsome.tar.gz *
$ tar --exclude='./acme/foo' -czvf ../allfiles-butsome.tar.gz *
$ tar -czvf ../allfiles-butsome.tar.gz * --exclude=/full/path/acme/foo
$ tar -czvf ../allfiles-butsome.tar.gz * --exclude='/full/path/acme/foo'
$ tar --exclude=/full/path/acme/foo -czvf ../allfiles-butsome.tar.gz *
$ tar --exclude='/full/path/acme/foo' -czvf ../allfiles-butsome.tar.gz *

Jerinaw ,May 6, 2017 at 20:07

For Mac OSX I had to do

tar -zcv --exclude='folder' -f theOutputTarFile.tar folderToTar

Note the -f after the --exclude=

Aaron Votre ,Jul 15, 2016 at 15:56

I agree the --exclude flag is the right approach.
$ tar --exclude='./folder_or_file' --exclude='file_pattern' --exclude='fileA'

A word of warning for a side effect that I did not find immediately obvious: The exclusion of 'fileA' in this example will search for 'fileA' RECURSIVELY!

Example:A directory with a single subdirectory containing a file of the same name (data.txt)

data.txt
config.txt
--+dirA
  |  data.txt
  |  config.docx

Znik ,Nov 15, 2014 at 5:12

To avoid possible 'xargs: Argument list too long' errors due to the use of find ... | xargs ... when processing tens of thousands of files, you can pipe the output of find directly to tar using find ... -print0 | tar --null ... .
# archive a given directory, but exclude various files & directories 
# specified by their full file paths
find "$(pwd -P)" -type d \( -path '/path/to/dir1' -or -path '/path/to/dir2' \) -prune \
   -or -not \( -path '/path/to/file1' -or -path '/path/to/file2' \) -print0 | 
   gnutar --null --no-recursion -czf archive.tar.gz --files-from -
   #bsdtar --null -n -czf archive.tar.gz -T -

Mike ,May 9, 2014 at 21:29

After reading this thread, I did a little testing on RHEL 5 and here are my results for tarring up the abc directory:

This will exclude the directories error and logs and all files under the directories:

tar cvpzf abc.tgz abc/ --exclude='abc/error' --exclude='abc/logs'

Adding a wildcard after the excluded directory will exclude the files but preserve the directories:

tar cvpzf abc.tgz abc/ --exclude='abc/error/*' --exclude='abc/logs/*'

Alex B ,Jun 11, 2009 at 23:03

Use the find command in conjunction with the tar append (-r) option. This way you can add files to an existing tar in a single step, instead of a two pass solution (create list of files, create tar).
find /dir/dir -prune ... -o etc etc.... -exec tar rvf ~/tarfile.tar {} \;

frommelmak ,Sep 10, 2012 at 14:08

You can also use one of the "--exclude-tag" options depending on your needs:

The folder hosting the specified FILE will be excluded.

camh ,Jun 12, 2009 at 5:53

You can use cpio(1) to create tar files. cpio takes the files to archive on stdin, so if you've already figured out the find command you want to use to select the files the archive, pipe it into cpio to create the tar file:
find ... | cpio -o -H ustar | gzip -c > archive.tar.gz

PicoutputCls ,Aug 21, 2018 at 14:13

gnu tar v 1.26 the --exclude needs to come after archive file and backup directory arguments, should have no leading or trailing slashes, and prefers no quotes (single or double). So relative to the PARENT directory to be backed up, it's:

tar cvfz /path_to/mytar.tgz ./dir_to_backup --exclude=some_path/to_exclude

user2553863 ,May 28 at 21:41

After reading all this good answers for different versions and having solved the problem for myself, I think there are very small details that are very important, and rare to GNU/Linux general use , that aren't stressed enough and deserves more than comments.

So I'm not going to try to answer the question for every case, but instead, try to register where to look when things doesn't work.

IT IS VERY IMPORTANT TO NOTICE:

  1. THE ORDER OF THE OPTIONS MATTER: it is not the same put the --exclude before than after the file option and directories to backup. This is unexpected at least to me, because in my experience, in GNU/Linux commands, usually the order of the options doesn't matter.
  2. Different tar versions expects this options in different order: for instance, @Andrew's answer indicates that in GNU tar v 1.26 and 1.28 the excludes comes last, whereas in my case, with GNU tar 1.29, it's the other way.
  3. THE TRAILING SLASHES MATTER : at least in GNU tar 1.29, it shouldn't be any .

In my case, for GNU tar 1.29 on Debian stretch, the command that worked was

tar --exclude="/home/user/.config/chromium" --exclude="/home/user/.cache" -cf file.tar  /dir1/ /home/ /dir3/

The quotes didn't matter, it worked with or without them.

I hope this will be useful to someone.

jørgensen ,Dec 19, 2015 at 11:10

Your best bet is to use find with tar, via xargs (to handle the large number of arguments). For example:
find / -print0 | xargs -0 tar cjf tarfile.tar.bz2

Ashwini Gupta ,Jan 12, 2018 at 10:30

tar -cvzf destination_folder source_folder -X /home/folder/excludes.txt

-X indicates a file which contains a list of filenames which must be excluded from the backup. For Instance, you can specify *~ in this file to not include any filenames ending with ~ in the backup.

George ,Sep 4, 2013 at 22:35

Possible redundant answer but since I found it useful, here it is:

While a FreeBSD root (i.e. using csh) I wanted to copy my whole root filesystem to /mnt but without /usr and (obviously) /mnt. This is what worked (I am at /):

tar --exclude ./usr --exclude ./mnt --create --file - . (cd /mnt && tar xvd -)

My whole point is that it was necessary (by putting the ./ ) to specify to tar that the excluded directories where part of the greater directory being copied.

My €0.02

t0r0X ,Sep 29, 2014 at 20:25

I had no luck getting tar to exclude a 5 Gigabyte subdirectory a few levels deep. In the end, I just used the unix Zip command. It worked a lot easier for me.

So for this particular example from the original post
(tar --exclude='./folder' --exclude='./upload/folder2' -zcvf /backup/filename.tgz . )

The equivalent would be:

zip -r /backup/filename.zip . -x upload/folder/**\* upload/folder2/**\*

(NOTE: Here is the post I originally used that helped me https://superuser.com/questions/312301/unix-zip-directory-but-excluded-specific-subdirectories-and-everything-within-t )

RohitPorwal ,Jul 21, 2016 at 9:56

Check it out
tar cvpzf zip_folder.tgz . --exclude=./public --exclude=./tmp --exclude=./log --exclude=fileName

tripleee ,Sep 14, 2017 at 4:38

The following bash script should do the trick. It uses the answer given here by Marcus Sundman.
#!/bin/bash

echo -n "Please enter the name of the tar file you wish to create with out extension "
read nam

echo -n "Please enter the path to the directories to tar "
read pathin

echo tar -czvf $nam.tar.gz
excludes=`find $pathin -iname "*.CC" -exec echo "--exclude \'{}\'" \;|xargs`
echo $pathin

echo tar -czvf $nam.tar.gz $excludes $pathin

This will print out the command you need and you can just copy and paste it back in. There is probably a more elegant way to provide it directly to the command line.

Just change *.CC for any other common extension, file name or regex you want to exclude and this should still work.

EDIT

Just to add a little explanation; find generates a list of files matching the chosen regex (in this case *.CC). This list is passed via xargs to the echo command. This prints --exclude 'one entry from the list'. The slashes () are escape characters for the ' marks.

[Aug 06, 2019] bash - More efficient way to find tar millions of files - Stack Overflow

Aug 06, 2019 | stackoverflow.com

More efficient way to find & tar millions of files Ask Question Asked 9 years, 3 months ago Active 8 months ago Viewed 25k times 22 13


theomega ,Apr 29, 2010 at 13:51

I've got a job running on my server at the command line prompt for a two days now:
find data/ -name filepattern-*2009* -exec tar uf 2009.tar {} ;

It is taking forever , and then some. Yes, there are millions of files in the target directory. (Each file is a measly 8 bytes in a well hashed directory structure.) But just running...

find data/ -name filepattern-*2009* -print > filesOfInterest.txt

...takes only two hours or so. At the rate my job is running, it won't be finished for a couple of weeks .. That seems unreasonable. Is there a more efficient to do this? Maybe with a more complicated bash script?

A secondary questions is "why is my current approach so slow?"

Stu Thompson ,May 6, 2013 at 1:11

If you already did the second command that created the file list, just use the -T option to tell tar to read the files names from that saved file list. Running 1 tar command vs N tar commands will be a lot better.

Matthew Mott ,Jul 3, 2014 at 19:21

One option is to use cpio to generate a tar-format archive:
$ find data/ -name "filepattern-*2009*" | cpio -ov --format=ustar > 2009.tar

cpio works natively with a list of filenames from stdin, rather than a top-level directory, which makes it an ideal tool for this situation.

bashfu ,Apr 23, 2010 at 10:05

Here's a find-tar combination that can do what you want without the use of xargs or exec (which should result in a noticeable speed-up):
tar --version    # tar (GNU tar) 1.14 

# FreeBSD find (on Mac OS X)
find -x data -name "filepattern-*2009*" -print0 | tar --null --no-recursion -uf 2009.tar --files-from -

# for GNU find use -xdev instead of -x
gfind data -xdev -name "filepattern-*2009*" -print0 | tar --null --no-recursion -uf 2009.tar --files-from -

# added: set permissions via tar
find -x data -name "filepattern-*2009*" -print0 | \
    tar --null --no-recursion --owner=... --group=... --mode=... -uf 2009.tar --files-from -

Stu Thompson ,Apr 28, 2010 at 12:50

There is xargs for this:
find data/ -name filepattern-*2009* -print0 | xargs -0 tar uf 2009.tar

Guessing why it is slow is hard as there is not much information. What is the structure of the directory, what filesystem do you use, how it was configured on creating. Having milions of files in single directory is quite hard situation for most filesystems.

bashfu ,May 1, 2010 at 14:18

To correctly handle file names with weird (but legal) characters (such as newlines, ...) you should write your file list to filesOfInterest.txt using find's -print0:
find -x data -name "filepattern-*2009*" -print0 > filesOfInterest.txt
tar --null --no-recursion -uf 2009.tar --files-from filesOfInterest.txt

Michael Aaron Safyan ,Apr 23, 2010 at 8:47

The way you currently have things, you are invoking the tar command every single time it finds a file, which is not surprisingly slow. Instead of taking the two hours to print plus the amount of time it takes to open the tar archive, see if the files are out of date, and add them to the archive, you are actually multiplying those times together. You might have better success invoking the tar command once, after you have batched together all the names, possibly using xargs to achieve the invocation. By the way, I hope you are using 'filepattern-*2009*' and not filepattern-*2009* as the stars will be expanded by the shell without quotes.

ruffrey ,Nov 20, 2018 at 17:13

There is a utility for this called tarsplitter .
tarsplitter -m archive -i folder/*.json -o archive.tar -p 8

will use 8 threads to archive the files matching "folder/*.json" into an output archive of "archive.tar"

https://github.com/AQUAOSOTech/tarsplitter

syneticon-dj ,Jul 22, 2013 at 8:47

Simplest (also remove file after archive creation):
find *.1  -exec tar czf '{}.tgz' '{}' --remove-files \;

[Aug 06, 2019] backup - Fastest way combine many files into one (tar czf is too slow) - Unix Linux Stack Exchange

Aug 06, 2019 | unix.stackexchange.com

Fastest way combine many files into one (tar czf is too slow) Ask Question Asked 7 years, 11 months ago Active 21 days ago Viewed 32k times 22 5


Gilles ,Nov 5, 2013 at 0:05

Currently I'm running tar czf to combine backup files. The files are in a specific directory.

But the number of files is growing. Using tzr czf takes too much time (more than 20 minutes and counting).

I need to combine the files more quickly and in a scalable fashion.

I've found genisoimage , readom and mkisofs . But I don't know which is fastest and what the limitations are for each of them.

Rufo El Magufo ,Aug 24, 2017 at 7:56

You should check if most of your time are being spent on CPU or in I/O. Either way, there are ways to improve it:

A: don't compress

You didn't mention "compression" in your list of requirements so try dropping the "z" from your arguments list: tar cf . This might be speed up things a bit.

There are other techniques to speed-up the process, like using "-N " to skip files you already backed up before.

B: backup the whole partition with dd

Alternatively, if you're backing up an entire partition, take a copy of the whole disk image instead. This would save processing and a lot of disk head seek time. tar and any other program working at a higher level have a overhead of having to read and process directory entries and inodes to find where the file content is and to do more head disk seeks , reading each file from a different place from the disk.

To backup the underlying data much faster, use:

dd bs=16M if=/dev/sda1 of=/another/filesystem

(This assumes you're not using RAID, which may change things a bit)

,

To repeat what others have said: we need to know more about the files that are being backed up. I'll go with some assumptions here. Append to the tar file

If files are only being added to the directories (that is, no file is being deleted), make sure you are appending to the existing tar file rather than re-creating it every time. You can do this by specifying the existing archive filename in your tar command instead of a new one (or deleting the old one).

Write to a different disk

Reading from the same disk you are writing to may be killing performance. Try writing to a different disk to spread the I/O load. If the archive file needs to be on the same disk as the original files, move it afterwards.

Don't compress

Just repeating what @Yves said. If your backup files are already compressed, there's not much need to compress again. You'll just be wasting CPU cycles.

[Aug 02, 2019] linux - How to tar directory and then remove originals including the directory - Super User

Aug 02, 2019 | superuser.com

How to tar directory and then remove originals including the directory? Ask Question Asked 9 years, 6 months ago Active 4 years, 6 months ago Viewed 124k times 28 7


mit ,Dec 7, 2016 at 1:22

I'm trying to tar a collection of files in a directory called 'my_directory' and remove the originals by using the command:
tar -cvf files.tar my_directory --remove-files

However it is only removing the individual files inside the directory and not the directory itself (which is what I specified in the command). What am I missing here?

EDIT:

Yes, I suppose the 'remove-files' option is fairly literal. Although I too found the man page unclear on that point. (In linux I tend not to really distinguish much between directories and files that much, and forget sometimes that they are not the same thing). It looks like the consensus is that it doesn't remove directories.

However, my major prompting point for asking this question stems from tar's handling of absolute paths. Because you must specify a relative path to a file/s to be compressed, you therefore must change to the parent directory to tar it properly. As I see it using any kind of follow-on 'rm' command is potentially dangerous in that situation. Thus I was hoping to simplify things by making tar itself do the remove.

For example, imagine a backup script where the directory to backup (ie. tar) is included as a shell variable. If that shell variable value was badly entered, it is possible that the result could be deleted files from whatever directory you happened to be in last.

Arjan ,Feb 13, 2016 at 13:08

You are missing the part which says the --remove-files option removes files after adding them to the archive.

You could follow the archive and file-removal operation with a command like,

find /path/to/be/archived/ -depth -type d -empty -exec rmdir {} \;


Update: You may be interested in reading this short Debian discussion on,
Bug 424692: --remove-files complains that directories "changed as we read it" .

Kim ,Feb 13, 2016 at 13:08

Since the --remove-files option only removes files , you could try
tar -cvf files.tar my_directory && rm -R my_directory

so that the directory is removed only if the tar returns an exit status of 0

redburn ,Feb 13, 2016 at 13:08

Have you tried to put --remove-files directive after archive name? It works for me.
tar -cvf files.tar --remove-files my_directory

shellking ,Oct 4, 2010 at 19:58

source={directory argument}

e.g.

source={FULL ABSOLUTE PATH}/my_directory
parent={parent directory of argument}

e.g.

parent={ABSOLUTE PATH of 'my_directory'/
logFile={path to a run log that captures status messages}

Then you could execute something along the lines of:

cd ${parent}

tar cvf Tar_File.`date%Y%M%D_%H%M%S` ${source}

if [ $? != 0 ]

then

 echo "Backup FAILED for ${source} at `date` >> ${logFile}

else

 echo "Backup SUCCESS for ${source} at `date` >> ${logFile}

 rm -rf ${source}

fi

mit ,Nov 14, 2011 at 13:21

This was probably a bug.

Also the word "file" is ambigous in this case. But because this is a command line switch I would it expect to mean also directories, because in unix/lnux everything is a file, also a directory. (The other interpretation is of course also valid, but It makes no sense to keep directories in such a case. I would consider it unexpected and confusing behavior.)

But I have found that in gnu tar on some distributions gnu tar actually removes the directory tree. Another indication that keeping the tree was a bug. Or at least some workaround until they fixed it.

This is what I tried out on an ubuntu 10.04 console:

mit:/var/tmp$ mkdir tree1                                                                                               
mit:/var/tmp$ mkdir tree1/sub1                                                                                          
mit:/var/tmp$ > tree1/sub1/file1                                                                                        

mit:/var/tmp$ ls -la                                                                                                    
drwxrwxrwt  4 root root 4096 2011-11-14 15:40 .                                                                              
drwxr-xr-x 16 root root 4096 2011-02-25 03:15 ..
drwxr-xr-x  3 mit  mit  4096 2011-11-14 15:40 tree1

mit:/var/tmp$ tar -czf tree1.tar.gz tree1/ --remove-files

# AS YOU CAN SEE THE TREE IS GONE NOW:

mit:/var/tmp$ ls -la
drwxrwxrwt  3 root root 4096 2011-11-14 15:41 .
drwxr-xr-x 16 root root 4096 2011-02-25 03:15 ..
-rw-r--r--  1 mit   mit    159 2011-11-14 15:41 tree1.tar.gz                                                                   


mit:/var/tmp$ tar --version                                                                                             
tar (GNU tar) 1.22                                                                                                           
Copyright © 2009 Free Software Foundation, Inc.

If you want to see it on your machine, paste this into a console at your own risk:

tar --version                                                                                             
cd /var/tmp
mkdir -p tree1/sub1                                                                                          
> tree1/sub1/file1                                                                                        
tar -czf tree1.tar.gz tree1/ --remove-files
ls -la

[Jul 31, 2019] Is Ruby moving toward extinction?

Jul 31, 2019 | developers.slashdot.org

timeOday ( 582209 ) , Monday July 29, 2019 @03:44PM ( #59007686 )

Re:ORLY ( Score: 5 , Insightful)

This is what it feels like to actually learn from an article instead of simply having it confirm your existing beliefs.

Here is what it says:

An analysis of Dice job-posting data over the past year shows a startling dip in the number of companies looking for technology professionals who are skilled in Ruby. In 2018, the number of Ruby jobs declined 56 percent. That's a huge warning sign that companies are turning away from Ruby - and if that's the case, the language's user-base could rapidly erode to almost nothing.

Well, what's your evidence-based rebuttal to that?

Wdomburg ( 141264 ) writes:
Re: ( Score: 2 )

If you actually look at the TIOBE rankings, it's #11 (not #12 as claimed in the article), and back on the upswing. If you look at RedMonk, which they say they looked at but don't reference with respect to Ruby, it is a respectable #8, being one of the top languages on GitHub and Stack Overflow.

We are certainly past the glory days of Ruby, when it was the Hot New Thing and everyone was deploying Rails, but to suggest that it is "probably doomed" seems a somewhat hysterical prediction.

OrangeTide ( 124937 ) , Tuesday July 30, 2019 @01:52AM ( #59010348 ) Homepage Journal
Re:ORLY ( Score: 4 , Funny)
How do they know how many Ruby jobs there are? Maybe how many Ruby job openings announced, but not the actual number of jobs. Or maybe they are finding Ruby job-applicants and openings via other means.

Maybe there is a secret list of Ruby job postings only available to the coolest programmers? Man! I never get to hang out with the cool kids.

jellomizer ( 103300 ) , Monday July 29, 2019 @03:48PM ( #59007714 )
Re:ORLY ( Score: 5 , Insightful)

Perhaps the devops/web programmers is a dying field.

But to be fair, Ruby had its peak about 10 years ago. With Ruby on Rails. However the problem is the "Rails" started to get very dated. And Python and Node.JS had taken its place.

whitelabrat ( 469237 ) , Monday July 29, 2019 @03:57PM ( #59007778 )
Re:ORLY ( Score: 5 , Insightful)

I don't see Ruby dying anytime soon, but I do get the feeling that Python is the go-to scripting language for all the things now. I learned Ruby and wish I spent that time learning Python.

Perl is perl. It will live on, but anybody writing new things with it probably needs to have a talkin' to.

phantomfive ( 622387 ) , Monday July 29, 2019 @07:32PM ( #59009188 ) Journal
Re:ORLY ( Score: 4 , Insightful)
I learned Ruby and wish I spent that time learning Python.

Ruby and Python are basically the same thing. With a little google, you can literally start programming in Python today. Search for "print python" and you can easily write a hello world. search for 'python for loop' and suddenly you can do repetitious tasks. Search for "define function python" and you can organize your code.

After that do a search for hash tables and lists in Python and you'll be good enough to pass a coding interview in the language.

[Jul 31, 2019] 5 Programming Languages That Are Probably Doomed

The article is a clickbait. entrenched languages seldom die. But some Slashdot comments are interesting.
Jul 31, 2019 | developers.slashdot.org

NoNonAlphaCharsHere ( 2201864 ) , Monday July 29, 2019 @03:39PM ( #59007638 )

Re:ORLY ( Score: 5 , Funny)

Perl has been "doomed" for over 30 years now, hasn't stopped it.

geekoid ( 135745 ) writes:
Re: ( Score: 2 )

OTOH, it not exactly what it once was.

IMO: if you can't write good readable code in PERL, you should find a new business to work in.

Anonymous Coward writes:
check the job description ( Score: 3 , Funny)

Writing unreadable perl is the business.

ShanghaiBill ( 739463 ) writes:
Re: ( Score: 3 )
Perl has been "doomed" for over 30 years now, hasn't stopped it.

I love Perl, but today it is mostly small throw-away scripts and maintaining legacy apps.

It makes little sense to use Perl for a new project.

Perl won't disappear, but the glory days are in the past.

Anonymous Coward , Monday July 29, 2019 @03:59PM ( #59007794 )
Re:ORLY ( Score: 4 , Interesting)

I write new code in perl all the time. Cleanly written, well formatted and completely maintainable. Simply because YOU can't write perl in such a manner, that doesn't mean others can't.

Anonymous Coward writes:
Re: ORLY ( Score: 2 , Insightful)

Do you have someone else who is saying that about your code or is that your own opinion?

Sarten-X ( 1102295 ) , Monday July 29, 2019 @05:53PM ( #59008624 ) Homepage
Re: ORLY ( Score: 4 , Insightful)

I happen to read a lot of Perl in my day job, involving reverse-engineering a particular Linux-based appliance for integration purposes. I seldom come across scripts that are too actually bad.

It's important to understand that Perl has a different concept of readability. It's more like reading a book than reading a program, because there are so many ways to write any given task. A good Perl programmer will incorporate that flexibility into their style, so intent can be inferred not just from the commands used, but also how the code is arranged. For example, a large block describing a complex function would be written verbosely for detailed clarity.

A trivial statement could be used, if it resolves an edge case.

Conversely, a good Perl reader will be familiar enough with the language to understand the idioms and shorthand used, so they can understand the story as written without being distracted by the ugly bits. Once viewed from that perspective, a Perl program can condense incredible amounts of description into just a few lines, and still be as readily-understood as any decent novel.

Sarten-X ( 1102295 ) writes: on Monday July 29, 2019 @07:06PM ( #59009056 ) Homepage
Re: ORLY ( Score: 4 , Insightful)

Since you brought it up...

In building several dev teams, I have never tried to hire everyone with any particular skill. I aim to have at least two people with each skill, but won't put effort to having more than that at first. After the initial startup of the team, I try to run projects in pairs, with an expert starting the project, then handing it to a junior (in that particular skill) for completion. After a few rounds of that, the junior is close enough to an expert, and somebody else takes the junior role. That way, even with turnover, expertise is shared among the team, and there's always someone who can be the expert.

Back to the subject at hand, though...

My point is that Perl is a more conversational language that others, and its structure reflects that. It is unreasonable to simply look at Perl code, see the variety of structures, and declare it "unreadable" simply because the reader doesn't understand the language.

As an analogy, consider the structural differences between Lord of the Rings and The Cat in the Hat . A reader who is only used to The Cat in the Hat would find Lord of the Rings to be ridiculously complex to the point of being unreadable, when Lord of the Rings is simply making use of structures and capabilities that are not permitted in the language of young children's' books.

This is not to say that other languages are wrong to have a more limited grammar. They are simply different, and learning to read a more flexible language is a skill to be developed like any other. Similar effort must be spent to learn other languages with sufficiently-different structure, like Lisp or Haskell.

phantomfive ( 622387 ) , Monday July 29, 2019 @07:24PM ( #59009128 ) Journal
Re:ORLY ( Score: 3 )

FWIW DuckDuckGo is apparently written primarily in Perl.

fahrbot-bot ( 874524 ) , Monday July 29, 2019 @03:46PM ( #59007696 )
If your career is based on ... ( Score: 3 , Interesting)

From TFA:

Perl: Even if RedMonk has Perl's popularity declining, it's still going to take a long time for the language to flatten out completely, given the sheer number of legacy websites that still feature its code. Nonetheless, a lack of active development, and widespread developer embrace of other languages for things like building websites, means that Perl is going to just fall into increasing disuse.

First, Perl is used for many, many more things than websites -- and the focus in TFA is short-sighted. Second, I've written a LOT of Perl in my many years, but wouldn't say my (or most people's) career is based on it. Yes, I have written applications in Perl, but more often used it for utility, glue and other things that help get things done, monitor and (re)process data. Nothing (or very few things) can beat Perl for a quick knock-off script to do something or another.

Perl's not going anywhere and it will be a useful language to know for quite a while. Languages like Perl (and Python) are great tools to have in your toolbox, ones that you know how to wield well when you need them. Knowing when you need them, and not something else, is important.

TimHunter ( 174406 ) , Monday July 29, 2019 @05:22PM ( #59008400 )
Career based on *a* programming language? ( Score: 4 , Insightful)

Anybody whose career is based on a single programming language is doomed already. Programmers know how to write code. The language they use is beside the point. A good programmer can write code in whatever language is asked of them.

bobbied ( 2522392 ) , Monday July 29, 2019 @04:23PM ( #59007966 )
Re:Diversifying ( Score: 5 , Insightful)
The writer of this article should consider diversifying his skillset at some point, as not all bloggers endure forever and his popularity ranking on Slashdot has recently tanked.

I'd suggest that this writer quit their day job and take up stand up...

Old languages never really die until the platform dies. Languages may fall out of favor, but they don't usually die until the platform they are running on disappears and then the people who used them die. So, FORTRAN, C, C++, and COBOL and more are here to pretty much stay.

Specifically, PERL isn't going anywhere being fundamentally on Linux, neither is Ruby, the rest to varying degrees have been out of favor for awhile now, but none of the languages in the article are dead. They are, however, falling out of favor and because of that, it might be a good idea to be adding other tools to your programmer's tool box if your livelihood depends on one of them.

[Jul 30, 2019] Python is overrated

Notable quotes:
"... R commits a substantial scale crime by being so dependent on memory-resident objects. Python commits major scale crime with its single-threaded primary interpreter loop. ..."
Jul 29, 2019 | developers.slashdot.org

epine ( 68316 ), Monday July 29, 2019 @05:48PM ( #59008600 ) Score: 3 )

Jul 30, 2019 | developers.slashdot.org

I had this naive idea that Python might substantially displace R until I learned more about the Python internals, which are pretty nasty. This is the new generation's big data language? If true, sure sucks to be young again.

Python isn't even really used to do big data. It's mainly used to orchestrate big data flows on top of other libraries or facilities. It has more or less become the lingua franca of high-level hand waving. Any real grunt is far below.

R commits a substantial scale crime by being so dependent on memory-resident objects. Python commits major scale crime with its single-threaded primary interpreter loop.

If I move away from R, it will definitely be Julia for any real work (as Julia matures, if it matures well), and not Python.

[Jul 30, 2019] The difference between tar and tar.gz archives

With tar.gz to extract a file archiver first creates an intermediary tarball x.tar file from x.tar.gz by uncompressing the whole archive then unpack requested files from this intermediary tarball. In tar.gz archive is large unpacking can take several hours or even days.
Jul 30, 2019 | askubuntu.com

[Jul 29, 2019] How do I tar a directory of files and folders without including the directory itself - Stack Overflow

Jan 05, 2017 | stackoverflow.com

How do I tar a directory of files and folders without including the directory itself? Ask Question Asked 10 years, 1 month ago Active 8 months ago Viewed 464k times 348 105


tvanfosson ,Jan 5, 2017 at 12:29

I typically do:
tar -czvf my_directory.tar.gz my_directory

What if I just want to include everything (including any hidden system files) in my_directory, but not the directory itself? I don't want:

my_directory
   --- my_file
   --- my_file
   --- my_file

I want:

my_file
my_file
my_file

PanCrit ,Feb 19 at 13:04

cd my_directory/ && tar -zcvf ../my_dir.tgz . && cd -

should do the job in one line. It works well for hidden files as well. "*" doesn't expand hidden files by path name expansion at least in bash. Below is my experiment:

$ mkdir my_directory
$ touch my_directory/file1
$ touch my_directory/file2
$ touch my_directory/.hiddenfile1
$ touch my_directory/.hiddenfile2
$ cd my_directory/ && tar -zcvf ../my_dir.tgz . && cd ..
./
./file1
./file2
./.hiddenfile1
./.hiddenfile2
$ tar ztf my_dir.tgz
./
./file1
./file2
./.hiddenfile1
./.hiddenfile2

JCotton ,Mar 3, 2015 at 2:46

Use the -C switch of tar:
tar -czvf my_directory.tar.gz -C my_directory .

The -C my_directory tells tar to change the current directory to my_directory , and then . means "add the entire current directory" (including hidden files and sub-directories).

Make sure you do -C my_directory before you do . or else you'll get the files in the current directory.

Digger ,Mar 23 at 6:52

You can also create archive as usual and extract it with:
tar --strip-components 1 -xvf my_directory.tar.gz

jwg ,Mar 8, 2017 at 12:56

Have a look at --transform / --xform , it gives you the opportunity to massage the file name as the file is added to the archive:
% mkdir my_directory
% touch my_directory/file1
% touch my_directory/file2
% touch my_directory/.hiddenfile1
% touch my_directory/.hiddenfile2
% tar -v -c -f my_dir.tgz --xform='s,my_directory/,,' $(find my_directory -type f)
my_directory/file2
my_directory/.hiddenfile1
my_directory/.hiddenfile2
my_directory/file1
% tar -t -f my_dir.tgz 
file2
.hiddenfile1
.hiddenfile2
file1

Transform expression is similar to that of sed , and we can use separators other than / ( , in the above example).
https://www.gnu.org/software/tar/manual/html_section/tar_52.html

Alex ,Mar 31, 2017 at 15:40

TL;DR
find /my/dir/ -printf "%P\n" | tar -czf mydir.tgz --no-recursion -C /my/dir/ -T -

With some conditions (archive only files, dirs and symlinks):

find /my/dir/ -printf "%P\n" -type f -o -type l -o -type d | tar -czf mydir.tgz --no-recursion -C /my/dir/ -T -
Explanation

The below unfortunately includes a parent directory ./ in the archive:

tar -czf mydir.tgz -C /my/dir .

You can move all the files out of that directory by using the --transform configuration option, but that doesn't get rid of the . directory itself. It becomes increasingly difficult to tame the command.

You could use $(find ...) to add a file list to the command (like in magnus' answer ), but that potentially causes a "file list too long" error. The best way is to combine it with tar's -T option, like this:

find /my/dir/ -printf "%P\n" -type f -o -type l -o -type d | tar -czf mydir.tgz --no-recursion -C /my/dir/ -T -

Basically what it does is list all files ( -type f ), links ( -type l ) and subdirectories ( -type d ) under your directory, make all filenames relative using -printf "%P\n" , and then pass that to the tar command (it takes filenames from STDIN using -T - ). The -C option is needed so tar knows where the files with relative names are located. The --no-recursion flag is so that tar doesn't recurse into folders it is told to archive (causing duplicate files).

If you need to do something special with filenames (filtering, following symlinks etc), the find command is pretty powerful, and you can test it by just removing the tar part of the above command:

$ find /my/dir/ -printf "%P\n" -type f -o -type l -o -type d
> textfile.txt
> documentation.pdf
> subfolder2
> subfolder
> subfolder/.gitignore

For example if you want to filter PDF files, add ! -name '*.pdf'

$ find /my/dir/ -printf "%P\n" -type f ! -name '*.pdf' -o -type l -o -type d
> textfile.txt
> subfolder2
> subfolder
> subfolder/.gitignore
Non-GNU find

The command uses printf (available in GNU find ) which tells find to print its results with relative paths. However, if you don't have GNU find , this works to make the paths relative (removes parents with sed ):

find /my/dir/ -type f -o -type l -o -type d | sed s,^/my/dir/,, | tar -czf mydir.tgz --no-recursion -C /my/dir/ -T -

BrainStone ,Dec 21, 2016 at 22:14

This Answer should work in most situations. Notice however how the filenames are stored in the tar file as, for example, ./file1 rather than just file1 . I found that this caused problems when using this method to manipulate tarballs used as package files in BuildRoot .

One solution is to use some Bash globs to list all files except for .. like this:

tar -C my_dir -zcvf my_dir.tar.gz .[^.]* ..?* *

This is a trick I learnt from this answer .

Now tar will return an error if there are no files matching ..?* or .[^.]* , but it will still work. If the error is a problem (you are checking for success in a script), this works:

shopt -s nullglob
tar -C my_dir -zcvf my_dir.tar.gz .[^.]* ..?* *
shopt -u nullglob

Though now we are messing with shell options, we might decide that it is neater to have * match hidden files:

shopt -s dotglob
tar -C my_dir -zcvf my_dir.tar.gz *
shopt -u dotglob

This might not work where your shell globs * in the current directory, so alternatively, use:

shopt -s dotglob
cd my_dir
tar -zcvf ../my_dir.tar.gz *
cd ..
shopt -u dotglob

PanCrit ,Jun 14, 2010 at 6:47

cd my_directory
tar zcvf ../my_directory.tar.gz *

anion ,May 11, 2018 at 14:10

If it's a Unix/Linux system, and you care about hidden files (which will be missed by *), you need to do:
cd my_directory
tar zcvf ../my_directory.tar.gz * .??*

I don't know what hidden files look like under Windows.

gpz500 ,Feb 27, 2014 at 10:46

I would propose the following Bash function (first argument is the path to the dir, second argument is the basename of resulting archive):
function tar_dir_contents ()
{
    local DIRPATH="$1"
    local TARARCH="$2.tar.gz"
    local ORGIFS="$IFS"
    IFS=$'\n'
    tar -C "$DIRPATH" -czf "$TARARCH" $( ls -a "$DIRPATH" | grep -v '\(^\.$\)\|\(^\.\.$\)' )
    IFS="$ORGIFS"
}

You can run it in this way:

$ tar_dir_contents /path/to/some/dir my_archive

and it will generate the archive my_archive.tar.gz within current directory. It works with hidden (.*) elements and with elements with spaces in their filename.

med ,Feb 9, 2017 at 17:19

cd my_directory && tar -czvf ../my_directory.tar.gz $(ls -A) && cd ..

This one worked for me and it's include all hidden files without putting all files in a root directory named "." like in tomoe's answer :

Breno Salgado ,Apr 16, 2016 at 15:42

Use pax.

Pax is a deprecated package but does the job perfectly and in a simple fashion.

pax -w > mydir.tar mydir

asynts ,Jun 26 at 16:40

Simplest way I found:

cd my_dir && tar -czvf ../my_dir.tar.gz *

marcingo ,Aug 23, 2016 at 18:04

# tar all files within and deeper in a given directory
# with no prefixes ( neither <directory>/ nor ./ )
# parameters: <source directory> <target archive file>
function tar_all_in_dir {
    { cd "$1" && find -type f -print0; } \
    | cut --zero-terminated --characters=3- \
    | tar --create --file="$2" --directory="$1" --null --files-from=-
}

Safely handles filenames with spaces or other unusual characters. You can optionally add a -name '*.sql' or similar filter to the find command to limit the files included.

user1456599 ,Feb 13, 2013 at 21:37

 tar -cvzf  tarlearn.tar.gz --remove-files mytemp/*

If the folder is mytemp then if you apply the above it will zip and remove all the files in the folder but leave it alone

 tar -cvzf  tarlearn.tar.gz --remove-files --exclude='*12_2008*' --no-recursion mytemp/*

You can give exclude patterns and also specify not to look into subfolders too

Aaron Digulla ,Jun 2, 2009 at 15:33

tar -C my_dir -zcvf my_dir.tar.gz `ls my_dir`

[Jun 26, 2019] 7,000 Developers Report Their Top Languages: Java, JavaScript, and Python

The article mixes apples and oranges and demonstrates complete ignorance in the of language classification.
Two of the three top language are scripting languages. This is a huge victory. But Python has problems with efficiency (not that they matter everywhere) and is far from being an elegant language. It entered mainstream via the adoption it at universities as the first programming language, displacing Java (which I think might be a mistake -- I think teaching should start with assembler and replicate the history of development -- assembler -- compiled languages -- scripting language)
Perl which essentially heralded the era of scripting languages is now losing its audience and shrinks to its initial purpose -- the tool for Unix system administrators. But I think is such surveys its use is underreported for obvious reasons -- it is not fashionable. But please note that Fortran is still widely used.
Go is just veriant of a "better C" -- statically typed, compiled language. Rust is an attempt to improve C++. Both belong to the class of compiled languages. So complied language still hold their own and are important part of the ecosystem. See also How Rust Compares to Other Programming Languages - The New Stack
Jun 26, 2019 | developers.slashdot.org
The report surveyed about 7,000 developers worldwide, and revealed Python is the most studied programming language, the most loved language , and the third top primary programming language developers are using... The top use cases developers are using Python for include data analysis, web development, machine learning and writing automation scripts, according to the JetBrains report . More developers are also beginning to move over to Python 3, with 9 out of 10 developers using the current version.

The JetBrains report also found while Go is still a young language, it is the most promising programming language. "Go started out with a share of 8% in 2017 and now it has reached 18%. In addition, the biggest number of developers (13%) chose Go as a language they would like to adopt or migrate to," the report stated...

Seventy-three percent of JavaScript developers use TypeScript, which is up from 17 percent last year. Seventy-one percent of Kotlin developers use Kotlin for work. Java 8 is still the most popular programming language, but developers are beginning to migrate to Java 10 and 11.
JetBrains (which designed Kotlin in 2011) also said that 60% of their survey's respondents identified themselves as professional web back-end developers (while 46% said they did web front-end, and 23% developed mobile applications). 41% said they hadn't contributed to open source projects "but I would like to," while 21% said they contributed "several times a year."

"16% of developers don't have any tests in their projects. Among fully-employed senior developers though, that statistic is just 8%. Like last year, about 30% of developers still don't have unit tests in their projects." Other interesting statistics: 52% say they code in their dreams. 57% expect AI to replace developers "partially" in the future. "83% prefer the Dark theme for their editor or IDE. This represents a growth of 6 percentage points since last year for each environment. 47% take public transit to work.

And 97% of respondents using Rust "said they have been using Rust for less than a year. With only 14% using it for work, it's much more popular as a language for personal/side projects." And more than 90% of the Rust developers who responded worked with codebases with less than 300 files.

[Jun 23, 2019] Utilizing multi core for tar+gzip-bzip compression-decompression

Highly recommended!
Notable quotes:
"... There is effectively no CPU time spent tarring, so it wouldn't help much. The tar format is just a copy of the input file with header blocks in between files. ..."
"... You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use. ..."
Jun 23, 2019 | stackoverflow.com

user1118764 , Sep 7, 2012 at 6:58

I normally compress using tar zcvf and decompress using tar zxvf (using gzip due to habit).

I've recently gotten a quad core CPU with hyperthreading, so I have 8 logical cores, and I notice that many of the cores are unused during compression/decompression.

Is there any way I can utilize the unused cores to make it faster?

Warren Severin , Nov 13, 2017 at 4:37

The solution proposed by Xiong Chiamiov above works beautifully. I had just backed up my laptop with .tar.bz2 and it took 132 minutes using only one cpu thread. Then I compiled and installed tar from source: gnu.org/software/tar I included the options mentioned in the configure step: ./configure --with-gzip=pigz --with-bzip2=lbzip2 --with-lzip=plzip I ran the backup again and it took only 32 minutes. That's better than 4X improvement! I watched the system monitor and it kept all 4 cpus (8 threads) flatlined at 100% the whole time. THAT is the best solution. – Warren Severin Nov 13 '17 at 4:37

Mark Adler , Sep 7, 2012 at 14:48

You can use pigz instead of gzip, which does gzip compression on multiple cores. Instead of using the -z option, you would pipe it through pigz:
tar cf - paths-to-archive | pigz > archive.tar.gz

By default, pigz uses the number of available cores, or eight if it could not query that. You can ask for more with -p n, e.g. -p 32. pigz has the same options as gzip, so you can request better compression with -9. E.g.

tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz

user788171 , Feb 20, 2013 at 12:43

How do you use pigz to decompress in the same fashion? Or does it only work for compression?

Mark Adler , Feb 20, 2013 at 16:18

pigz does use multiple cores for decompression, but only with limited improvement over a single core. The deflate format does not lend itself to parallel decompression.

The decompression portion must be done serially. The other cores for pigz decompression are used for reading, writing, and calculating the CRC. When compressing on the other hand, pigz gets close to a factor of n improvement with n cores.

Garrett , Mar 1, 2014 at 7:26

The hyphen here is stdout (see this page ).

Mark Adler , Jul 2, 2014 at 21:29

Yes. 100% compatible in both directions.

Mark Adler , Apr 23, 2015 at 5:23

There is effectively no CPU time spent tarring, so it wouldn't help much. The tar format is just a copy of the input file with header blocks in between files.

Jen , Jun 14, 2013 at 14:34

You can also use the tar flag "--use-compress-program=" to tell tar what compression program to use.

For example use:

tar -c --use-compress-program=pigz -f tar.file dir_to_zip

Valerio Schiavoni , Aug 5, 2014 at 22:38

Unfortunately by doing so the concurrent feature of pigz is lost. You can see for yourself by executing that command and monitoring the load on each of the cores. – Valerio Schiavoni Aug 5 '14 at 22:38

bovender , Sep 18, 2015 at 10:14

@ValerioSchiavoni: Not here, I get full load on all 4 cores (Ubuntu 15.04 'Vivid'). – bovender Sep 18 '15 at 10:14

Valerio Schiavoni , Sep 28, 2015 at 23:41

On compress or on decompress ? – Valerio Schiavoni Sep 28 '15 at 23:41

Offenso , Jan 11, 2017 at 17:26

I prefer tar - dir_to_zip | pv | pigz > tar.file pv helps me estimate, you can skip it. But still it easier to write and remember. – Offenso Jan 11 '17 at 17:26

Maxim Suslov , Dec 18, 2014 at 7:31

Common approach

There is option for tar program:

-I, --use-compress-program PROG
      filter through PROG (must accept -d)

You can use multithread version of archiver or compressor utility.

Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:

$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):

$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz

Input and output of singlethread and multithread are compatible. You can compress using multithread version and decompress using singlethread version and vice versa.

p7zip

For p7zip for compression you need a small shell script like the following:

#!/bin/sh
case $1 in
  -d) 7za -txz -si -so e;;
   *) 7za -txz -si -so a .;;
esac 2>/dev/null

Save it as 7zhelper.sh. Here the example of usage:

$ tar -I 7zhelper.sh -cf OUTPUT_FILE.tar.7z paths_to_archive
$ tar -I 7zhelper.sh -xf OUTPUT_FILE.tar.7z
xz

Regarding multithreaded XZ support. If you are running version 5.2.0 or above of XZ Utils, you can utilize multiple cores for compression by setting -T or --threads to an appropriate value via the environmental variable XZ_DEFAULTS (e.g. XZ_DEFAULTS="-T 0" ).

This is a fragment of man for 5.1.0alpha version:

Multithreaded compression and decompression are not implemented yet, so this option has no effect for now.

However this will not work for decompression of files that haven't also been compressed with threading enabled. From man for version 5.2.2:

Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't even if --block-size=size is used.

Recompiling with replacement

If you build tar from sources, then you can recompile with parameters

--with-gzip=pigz
--with-bzip2=lbzip2
--with-lzip=plzip

After recompiling tar with these options you can check the output of tar's help:

$ tar --help | grep "lbzip2\|plzip\|pigz"
  -j, --bzip2                filter the archive through lbzip2
      --lzip                 filter the archive through plzip
  -z, --gzip, --gunzip, --ungzip   filter the archive through pigz

mpibzip2 , Apr 28, 2015 at 20:57

I just found pbzip2 and mpibzip2 . mpibzip2 looks very promising for clusters or if you have a laptop and a multicore desktop computer for instance. – user1985657 Apr 28 '15 at 20:57

oᴉɹǝɥɔ , Jun 10, 2015 at 17:39

Processing STDIN may in fact be slower. – oᴉɹǝɥɔ Jun 10 '15 at 17:39

selurvedu , May 26, 2016 at 22:13

Plus 1 for xz option. It the simplest, yet effective approach. – selurvedu May 26 '16 at 22:13

panticz.de , Sep 1, 2014 at 15:02

You can use the shortcut -I for tar's --use-compress-program switch, and invoke pbzip2 for bzip2 compression on multiple cores:
tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 DIRECTORY_TO_COMPRESS/

einpoklum , Feb 11, 2017 at 15:59

A nice TL;DR for @MaximSuslov's answer . – einpoklum Feb 11 '17 at 15:59
If you want to have more flexibility with filenames and compression options, you can use:
find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec \
tar -P --transform='s@/my/path/@@g' -cf - {} + | \
pigz -9 -p 4 > myarchive.tar.gz
Step 1: find

find /my/path/ -type f -name "*.sql" -o -name "*.log" -exec

This command will look for the files you want to archive, in this case /my/path/*.sql and /my/path/*.log . Add as many -o -name "pattern" as you want.

-exec will execute the next command using the results of find : tar

Step 2: tar

tar -P --transform='s@/my/path/@@g' -cf - {} +

--transform is a simple string replacement parameter. It will strip the path of the files from the archive so the tarball's root becomes the current directory when extracting. Note that you can't use -C option to change directory as you'll lose benefits of find : all files of the directory would be included.

-P tells tar to use absolute paths, so it doesn't trigger the warning "Removing leading `/' from member names". Leading '/' with be removed by --transform anyway.

-cf - tells tar to use the tarball name we'll specify later

{} + uses everyfiles that find found previously

Step 3: pigz

pigz -9 -p 4

Use as many parameters as you want. In this case -9 is the compression level and -p 4 is the number of cores dedicated to compression. If you run this on a heavy loaded webserver, you probably don't want to use all available cores.

Step 4: archive name

> myarchive.tar.gz

Finally.

[May 24, 2019] How to send keystrokes from one computer to another by USB?

Notable quotes:
"... On a different note, have you considered a purely software/network solution such as TightVNC ? ..."
Aug 05, 2018 | stackoverflow.com

Ask Question


Yehonatan ,Aug 5, 2018 at 6:34

Is there a way to use one computer to send keystrokes to another by usb ?

What i'm looking to do is to capture the usb signal used by a keyboard (with USBTrace for example) and use it with PC-1 to send it to PC-2. So that PC-2 recognize it as a regular keyboard input.

Some leads to do this would be very appreciated.

Lucas ,Jan 16, 2011 at 19:18

What you essentially need is a USB port on PC-1 that will act as a USB device for PC-2.

That is not possible for the vast majority of PC systems because USB is an asymmetric bus, with a host/device (or master/slave, if you wish) architecture. USB controllers (and their ports) on most PCs can only work in host mode and cannot simulate a device.

That is the reason that you cannot network computers through USB without a special cable with specialized electronics.

The only exception is if you somehow have a PC that supports the USB On-The-Go standard that allows for a USB port to act in both host and device mode. USB-OTG devices do exist, but they are usually embedded devices (smartphones etc). I don't know if there is a way to add a USB-OTG port to a commodity PC.

EDIT:

If you do not need a keyboard before the OS on PC-2 boots, you might be able to use a pair of USB Bluetooth dongles - one on each PC. You'd have to use specialised software on PC-1, but it is definitely possible - I've already seen a possible implementation on Linux , and I am reasonably certain that there must be one for Windows. You will also need Bluetooth HID drivers on PC-2, if they are not already installed.

On a different note, have you considered a purely software/network solution such as TightVNC ?

bebbo ,Sep 20, 2017 at 18:14

There is a solution: https://github.com/Flowm/etherkey

This uses a network connection from your computer to the raspi which is connected to a teensy (usb developer board) to send the key strokes.

This solution is not an out-of-the-box product. The required skill is similar to programming some other devices like arduino. But it's a complete and working setup.

Yehonatan ,Jan 25, 2011 at 5:51

The cheapest options are commercial microcontrollers (eg arduino platform, pic, etc) or ready built usb keyboard controllers (eg i-pac, arcade controllers,etc)

Benoit-Pierre DEMAINE ,Oct 27, 2017 at 17:17

SEARCH THIS PROGRAM:

TWedge: Keyboard Wedge Software (RS232, Serial, TCP, Bluetooth)

then, MAKE YOUR OWN CONNECTION CABLE WITH:

(usb <-> rs232) + (NULL MODEM) + (rs232 <-> usb)

Connect 2 computer, write your own program to send signal to your (usb <-> rs232) unit, then you can control another computer under the help of TWedge.

> ,

The above mentionned https://github.com/Flowm/etherkey is one way. The keyboard is emulated from an rPi, but the principle can be used from PC to PC (or Mac to Whatever). The core answer to your question is to use an OTG-capable chip, and then you control this chip via a USB-serial adapter.

https://euer.krebsco.de/a-software-kvm-switch.html uses a very similar method, using an Arduino instead of the Teensy.

The generic answer is: you need an OTG capable, or slave capable device: Arduino, Teensy, Pi 0 (either from Rapberry or Orange brands, both work; only the ZERO models are OTG capable), or, an rPi-A with heavy customisation (since it does not include USB hub, it can theoretically be converted into a slave; never found any public tutorial to do it), or any smartphone (Samsung, Nokia, HTC, Oukitel ... most smartphones are OTG capable). If you go for a Pi or a phone, then, you want to dig around USB Gadget. Cheaper solutions (Arduino/Teensy) need custom firmware.

[May 03, 2019] RedHat local repository and offline updates

Aug 03, 2018 | stackoverflow.com

My company just bought a two redhat license for two physical machines , the machines wont be accessible via internet , so we have an issue here regarding the updates , patches , ... etc .

i am thinking of configuring a local repository to be accessible via internet and have all the necessary updates but there is a problem here that i have only two licenses , is it applicable if i activate the local repository for the updates and one of my two service machines , or is there any other way like if there is some sort of offline package that i can download it separately from redhat and update my machines without internet access ?

thanks in advance

XXX

You have several options:

See How can we regularly update a disconnected system (A system without internet connection)? for details.

[Mar 20, 2019] How do I troubleshoot a yum repository problem that has an error No package available error?

Mar 20, 2019 | unix.stackexchange.com

Kiran ,Jan 2, 2017 at 23:57

I have three RHEL 6.6 servers. One has a yum repository that I know works. The other two servers I will refer to as "yum clients." These two are configured to use the same yum repository (the first server described). When I do yum install httpd on each of these two yum client servers, I get two different results. One server prepares for the installation as normal and prompts me with a y/n prompt. The second server says

No package httpd available.

The /etc/yum.conf files on each of the two servers is identical. The /etc/yum.repos.d/ directories have the same .repo files. Why does one yum client not see the httpd package? I use httpd as an example. One yum client cannot install any package. The other yum client can install anything. Neither have access to the Internet or different servers the other one does not have access to.

XXX

If /etc/yum.conf is identical on all servers, and that package is not listed there in exclude line, check if the repo is enabled on all the servers.

Do

grep enabled /etc/yum.repos.d/filename.repo

and see if it is set to 0 or 1.

value of enabled needs to be set to 1, for yum to use that repo.

If repo is not enabled, you can edit the repo file, and change the enable to 1, or try to run yum with enablerepo switch, to enable it for that operation.

Try to run yum like this.

yum --enablerepo=repo_name install package_name

[Mar 20, 2019] How to I print to STDERR only if STDOUT is a different destination?

Mar 14, 2013 | stackoverflow.com

squiguy, Mar 14, 2013 at 19:06

I would like Perl to write to STDERR only if STDOUT is not the same. For example, if both STDOUT and STDERR would redirect output to the Terminal, then I don't want STDERR to be printed.

Consider the following example (outerr.pl):

#!/usr/bin/perl

use strict;
use warnings;

print STDOUT "Hello standard output!\n";
print STDERR "Hello standard error\n" if ($someMagicalFlag);
exit 0

Now consider this (this is what I would like to achieve):

bash $ outerr.pl
Hello standard output!

However, if I redirect out to a file, I'd like to get:

bash $ outerr.pl > /dev/null
Hello standard error

and similary the other way round:

bash $ outerr.pl 2> /dev/null
Hello standard output!

If I re-direct both out/err to the same file, then only stdout should be displayed:

bash $ outerr.pl > foo.txt 2>&1
bash $ cat foo.txt
Hello standard output!

So is there a way to evaluate / determine whether OUT and ERR and are pointing to the same "thing" (descriptor?)?

tchrist ,Mar 15, 2013 at 5:07

On Unix-style systems, you should be able to do:
my @stat_err = stat STDERR;
my @stat_out = stat STDOUT;

my $stderr_is_not_stdout = (($stat_err[0] != $stat_out[0]) ||
                            ($stat_err[1] != $stat_out[1]));

But that won't work on Windows, which doesn't have real inode numbers. It gives both false positives (thinks they're different when they aren't) and false negatives (thinks they're the same when they aren't).

Jim Stewart ,Mar 14, 2013 at 20:59

You can do that (almost) with -t:
-t STDERR

will be true if it is a terminal, and likewise for STDOUT.

This still would not tell you what terminal, and if you redirect to the same file, you may stilll get both.

Hence, if

-t STDERR && ! (-t STDOUT) || -t STDOUT && !(-t STDERR)

or shorter

-t STDOUT ^ -t STDERR  # thanks to @mob

you know you're okay.

EDIT: Solutions for the case that both STDERR and STDOUT are regular files:

Tom Christianson suggested to stat and compare the dev and ino fields. This will work in UNIX, but, as @cjm pointed out, not in Windows.

If you can guarantee that no other program will write to the file, you could do the following both in Windows and UNIX:

  1. check the position the file descriptors for STDOUT and STDERR are at, if they are not equal, you redirected one of them with >> to a nonempty file.
  2. Otherwise, write 42 bytes to file descriptor 2
  3. Seek to the end of file descriptor 1. If it is 42 more than before, chances are high that both are redirected to the same file. If it is unchanged, files are different. If it is changed, but not by 42, someone else is writing there, all bets are off (but then, you're not in Windows, so the stat method will work).

[Mar 17, 2019] Translating Perl to Python

Mar 17, 2019 | stackoverflow.com

21


John Kugelman ,Jul 1, 2009 at 3:29

I found this Perl script while migrating my SQLite database to mysql

I was wondering (since I don't know Perl) how could one rewrite this in Python?

Bonus points for the shortest (code) answer :)

edit : sorry I meant shortest code, not strictly shortest answer

#! /usr/bin/perl

while ($line = <>){
    if (($line !~  /BEGIN TRANSACTION/) && ($line !~ /COMMIT/) && ($line !~ /sqlite_sequence/) && ($line !~ /CREATE UNIQUE INDEX/)){

        if ($line =~ /CREATE TABLE \"([a-z_]*)\"(.*)/){
                $name = $1;
                $sub = $2;
                $sub =~ s/\"//g; #"
                $line = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";
        }
        elsif ($line =~ /INSERT INTO \"([a-z_]*)\"(.*)/){
                $line = "INSERT INTO $1$2\n";
                $line =~ s/\"/\\\"/g; #"
                $line =~ s/\"/\'/g; #"
        }else{
                $line =~ s/\'\'/\\\'/g; #'
        }
        $line =~ s/([^\\'])\'t\'(.)/$1THIS_IS_TRUE$2/g; #'
        $line =~ s/THIS_IS_TRUE/1/g;
        $line =~ s/([^\\'])\'f\'(.)/$1THIS_IS_FALSE$2/g; #'
        $line =~ s/THIS_IS_FALSE/0/g;
        $line =~ s/AUTOINCREMENT/AUTO_INCREMENT/g;
        print $line;
    }
}

Some additional code was necessary to successfully migrate the sqlite database (handles one line Create table statements, foreign keys, fixes a bug in the original program that converted empty fields '' to \' .

I posted the code on the migrating my SQLite database to mysql Question

Jiaaro ,Jul 2, 2009 at 10:15

Here's a pretty literal translation with just the minimum of obvious style changes (putting all code into a function, using string rather than re operations where possible).
import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", r"\1THIS_IS_TRUE\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    print line,

main()

dr jimbob ,May 20, 2018 at 0:54

Alex Martelli's solution above works good, but needs some fixes and additions:

In the lines using regular expression substitution, the insertion of the matched groups must be double-escaped OR the replacement string must be prefixed with r to mark is as regular expression:

line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)

or

line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)

Also, this line should be added before print:

line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')

Last, the column names in create statements should be backticks in MySQL. Add this in line 15:

  sub = sub.replace('"','`')

Here's the complete script with modifications:

import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      sub = sub.replace('"','`')
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", "\\1THIS_IS_FALSE\\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    if re.search('^CREATE INDEX', line):
        line = line.replace('"','`')
    print line,

main()

Brad Gilbert ,Jul 1, 2009 at 18:43

Here is a slightly better version of the original.
#! /usr/bin/perl
use strict;
use warnings;
use 5.010; # for s/\K//;

while( <> ){
  next if m'
    BEGIN TRANSACTION   |
    COMMIT              |
    sqlite_sequence     |
    CREATE UNIQUE INDEX
  'x;

  if( my($name,$sub) = m'CREATE TABLE \"([a-z_]*)\"(.*)' ){
    # remove "
    $sub =~ s/\"//g; #"
    $_ = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";

  }elsif( /INSERT INTO \"([a-z_]*)\"(.*)/ ){
    $_ = "INSERT INTO $1$2\n";

    # " => \"
    s/\"/\\\"/g; #"
    # " => '
    s/\"/\'/g; #"

  }else{
    # '' => \'
    s/\'\'/\\\'/g; #'
  }

  # 't' => 1
  s/[^\\']\K\'t\'/1/g; #'

  # 'f' => 0
  s/[^\\']\K\'f\'/0/g; #'

  s/AUTOINCREMENT/AUTO_INCREMENT/g;
  print;
}

Mickey Mouse ,Jun 14, 2011 at 15:48

all of scripts on this page can't deal with simple sqlite3:
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Filename (
  FilenameId INTEGER,
  Name TEXT DEFAULT '',
  PRIMARY KEY(FilenameId) 
  );
INSERT INTO "Filename" VALUES(1,'');
INSERT INTO "Filename" VALUES(2,'bigfile1');
INSERT INTO "Filename" VALUES(3,'%gconf-tree.xml');

None were able to reformat "table_name" into proper mysql's `table_name` . Some messed up empty string value.

Sinan Ünür ,Jul 1, 2009 at 3:24

I am not sure what is so hard to understand about this that it requires a snide remark as in your comment above. Note that <> is called the diamond operator. s/// is the substitution operator and // is the match operator m// .

Ken_g6 ,Jul 1, 2009 at 3:22

Based on http://docs.python.org/dev/howto/regex.html ...
  1. Replace $line =~ /.*/ with re.search(r".*", line) .
  2. $line !~ /.*/ is just !($line =~ /.*/) .
  3. Replace $line =~ s/.*/x/g with line=re.sub(r".*", "x", line) .
  4. Replace $1 through $9 inside re.sub with \1 through \9 respectively.
  5. Outside a sub, save the return value, i.e. m=re.search() , and replace $1 with the return value of m.group(1) .
  6. For "INSERT INTO $1$2\n" specifically, you can do "INSERT INTO %s%s\n" % (m.group(1), m.group(2)) .

hpavc ,Jul 1, 2009 at 12:33

Real issue is do you know actually how to migrate the database? What is presented is merely a search and replace loop.

> ,

Shortest? The tilde signifies a regex in perl. "import re" and go from there. The only key differences are that you'll be using \1 and \2 instead of $1 and $2 when you assign values, and you'll be using %s for when you're replacing regexp matches inside strings.

[Mar 16, 2019] Translating Perl to Python - Stack Overflow

Mar 16, 2019 | stackoverflow.com

Translating Perl to Python Ask Question 21


John Kugelman ,Jul 1, 2009 at 3:29

I found this Perl script while migrating my SQLite database to mysql

I was wondering (since I don't know Perl) how could one rewrite this in Python?

Bonus points for the shortest (code) answer :)

edit : sorry I meant shortest code, not strictly shortest answer

#! /usr/bin/perl

while ($line = <>){
    if (($line !~  /BEGIN TRANSACTION/) && ($line !~ /COMMIT/) && ($line !~ /sqlite_sequence/) && ($line !~ /CREATE UNIQUE INDEX/)){

        if ($line =~ /CREATE TABLE \"([a-z_]*)\"(.*)/){
                $name = $1;
                $sub = $2;
                $sub =~ s/\"//g; #"
                $line = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";
        }
        elsif ($line =~ /INSERT INTO \"([a-z_]*)\"(.*)/){
                $line = "INSERT INTO $1$2\n";
                $line =~ s/\"/\\\"/g; #"
                $line =~ s/\"/\'/g; #"
        }else{
                $line =~ s/\'\'/\\\'/g; #'
        }
        $line =~ s/([^\\'])\'t\'(.)/$1THIS_IS_TRUE$2/g; #'
        $line =~ s/THIS_IS_TRUE/1/g;
        $line =~ s/([^\\'])\'f\'(.)/$1THIS_IS_FALSE$2/g; #'
        $line =~ s/THIS_IS_FALSE/0/g;
        $line =~ s/AUTOINCREMENT/AUTO_INCREMENT/g;
        print $line;
    }
}

Some additional code was necessary to successfully migrate the sqlite database (handles one line Create table statements, foreign keys, fixes a bug in the original program that converted empty fields '' to \' .

I posted the code on the migrating my SQLite database to mysql Question

Jiaaro ,Jul 2, 2009 at 10:15

Here's a pretty literal translation with just the minimum of obvious style changes (putting all code into a function, using string rather than re operations where possible).
import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", r"\1THIS_IS_TRUE\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    print line,

main()

dr jimbob ,May 20, 2018 at 0:54

Alex Martelli's solution above works good, but needs some fixes and additions:

In the lines using regular expression substitution, the insertion of the matched groups must be double-escaped OR the replacement string must be prefixed with r to mark is as regular expression:

line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)

or

line = re.sub(r"([^'])'f'(.)", r"\1THIS_IS_FALSE\2", line)

Also, this line should be added before print:

line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')

Last, the column names in create statements should be backticks in MySQL. Add this in line 15:

  sub = sub.replace('"','`')

Here's the complete script with modifications:

import re, fileinput

def main():
  for line in fileinput.input():
    process = False
    for nope in ('BEGIN TRANSACTION','COMMIT',
                 'sqlite_sequence','CREATE UNIQUE INDEX'):
      if nope in line: break
    else:
      process = True
    if not process: continue
    m = re.search('CREATE TABLE "([a-z_]*)"(.*)', line)
    if m:
      name, sub = m.groups()
      sub = sub.replace('"','`')
      line = '''DROP TABLE IF EXISTS %(name)s;
CREATE TABLE IF NOT EXISTS %(name)s%(sub)s
'''
      line = line % dict(name=name, sub=sub)
    else:
      m = re.search('INSERT INTO "([a-z_]*)"(.*)', line)
      if m:
        line = 'INSERT INTO %s%s\n' % m.groups()
        line = line.replace('"', r'\"')
        line = line.replace('"', "'")
    line = re.sub(r"([^'])'t'(.)", "\\1THIS_IS_TRUE\\2", line)
    line = line.replace('THIS_IS_TRUE', '1')
    line = re.sub(r"([^'])'f'(.)", "\\1THIS_IS_FALSE\\2", line)
    line = line.replace('THIS_IS_FALSE', '0')
    line = line.replace('AUTOINCREMENT', 'AUTO_INCREMENT')
    if re.search('^CREATE INDEX', line):
        line = line.replace('"','`')
    print line,

main()

Brad Gilbert ,Jul 1, 2009 at 18:43

Here is a slightly better version of the original.
#! /usr/bin/perl
use strict;
use warnings;
use 5.010; # for s/\K//;

while( <> ){
  next if m'
    BEGIN TRANSACTION   |
    COMMIT              |
    sqlite_sequence     |
    CREATE UNIQUE INDEX
  'x;

  if( my($name,$sub) = m'CREATE TABLE \"([a-z_]*)\"(.*)' ){
    # remove "
    $sub =~ s/\"//g; #"
    $_ = "DROP TABLE IF EXISTS $name;\nCREATE TABLE IF NOT EXISTS $name$sub\n";

  }elsif( /INSERT INTO \"([a-z_]*)\"(.*)/ ){
    $_ = "INSERT INTO $1$2\n";

    # " => \"
    s/\"/\\\"/g; #"
    # " => '
    s/\"/\'/g; #"

  }else{
    # '' => \'
    s/\'\'/\\\'/g; #'
  }

  # 't' => 1
  s/[^\\']\K\'t\'/1/g; #'

  # 'f' => 0
  s/[^\\']\K\'f\'/0/g; #'

  s/AUTOINCREMENT/AUTO_INCREMENT/g;
  print;
}

Mickey Mouse ,Jun 14, 2011 at 15:48

all of scripts on this page can't deal with simple sqlite3:
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE Filename (
  FilenameId INTEGER,
  Name TEXT DEFAULT '',
  PRIMARY KEY(FilenameId) 
  );
INSERT INTO "Filename" VALUES(1,'');
INSERT INTO "Filename" VALUES(2,'bigfile1');
INSERT INTO "Filename" VALUES(3,'%gconf-tree.xml');

None were able to reformat "table_name" into proper mysql's `table_name` . Some messed up empty string value.

Sinan Ünür ,Jul 1, 2009 at 3:24

I am not sure what is so hard to understand about this that it requires a snide remark as in your comment above. Note that <> is called the diamond operator. s/// is the substitution operator and // is the match operator m// .

Ken_g6 ,Jul 1, 2009 at 3:22

Based on http://docs.python.org/dev/howto/regex.html ...
  1. Replace $line =~ /.*/ with re.search(r".*", line) .
  2. $line !~ /.*/ is just !($line =~ /.*/) .
  3. Replace $line =~ s/.*/x/g with line=re.sub(r".*", "x", line) .
  4. Replace $1 through $9 inside re.sub with \1 through \9 respectively.
  5. Outside a sub, save the return value, i.e. m=re.search() , and replace $1 with the return value of m.group(1) .
  6. For "INSERT INTO $1$2\n" specifically, you can do "INSERT INTO %s%s\n" % (m.group(1), m.group(2)) .

hpavc ,Jul 1, 2009 at 12:33

Real issue is do you know actually how to migrate the database? What is presented is merely a search and replace loop.

> ,

Shortest? The tilde signifies a regex in perl. "import re" and go from there. The only key differences are that you'll be using \1 and \2 instead of $1 and $2 when you assign values, and you'll be using %s for when you're replacing regexp matches inside strings.

[Mar 16, 2019] Regex translation from Perl to Python - Stack Overflow

Mar 16, 2019 | stackoverflow.com

Regex translation from Perl to Python Ask Question 1


royskatt ,Jan 30, 2014 at 14:45

I would like to rewrite a small Perl programm to Python. I am processing text files with it as follows:

Input:

00000001;Root;;
00000002;  Documents;;
00000003;    oracle-advanced_plsql.zip;file;
00000004;  Public;;
00000005;  backup;;
00000006;    20110323-JM-F.7z.001;file;
00000007;    20110426-JM-F.7z.001;file;
00000008;    20110603-JM-F.7z.001;file;
00000009;    20110701-JM-F-via-summer_school;;
00000010;      20110701-JM-F-yyy.7z.001;file;

Desired output:

00000001;;Root;;
00000002;  ;Documents;;
00000003;    ;oracle-advanced_plsql.zip;file;
00000004;  ;Public;;
00000005;  ;backup;;
00000006;    ;20110323-JM-F.7z.001;file;
00000007;    ;20110426-JM-F.7z.001;file;
00000008;    ;20110603-JM-F.7z.001;file;
00000009;    ;20110701-JM-F-via-summer_school;;
00000010;      ;20110701-JM-F-yyy.7z.001;file;

Here is the working Perl code:

#filename: perl_regex.pl
#/usr/bin/perl -w
while(<>) {                                                           
  s/^(.*?;.*?)(\w)/$1;$2/;                                            
  print $_;                                                           
}

It call it from the command line: perl_regex.pl input.txt

Explanation of the Perl-style regex:

s/        # start search-and-replace regexp
  ^       # start at the beginning of this line
  (       # save the matched characters until ')' in $1
    .*?;  # go forward until finding the first semicolon
    .*?   # go forward until finding... (to be continued below)
  )
  (       # save the matched characters until ')' in $2
    \w    # ... the next alphanumeric character.
  )
/         # continue with the replace part
  $1;$2   # write all characters found above, but insert a ; before $2
/         # finish the search-and-replace regexp.

Could anyone tell me, how to get the same result in Python? Especially for the $1 and $2 variables I couldn't find something alike.

royskatt ,Jan 31, 2014 at 6:18

Python regular expression is very similar to Perl's, except:

Use re.sub to replace.

import re
import sys

for line in sys.stdin: # Explicitly iterate standard input line by line
    # `line` contains trailing newline!
    line = re.sub(r'^(.*?;.*?)(\w)', r'\1;\2', line)
    #print(line) # This print trailing newline
    sys.stdout.write(line) # Print the replaced string back.

royskatt ,Jan 31, 2014 at 16:36

The replace instruction for s/pattern/replace/ in python regexes is the re.sub(pattern, replace, string) function, or re.compile(pattern).sub(replace, string). In your case, you will do it so:
_re_pattern = re.compile(r"^(.*?;.*?)(\w)")
result = _re_pattern.sub(r"\1;\2", line)

Note that $1 becomes \1 . As for perl, you need to iterate over your lines the way you want to do it (open, inputfile, splitlines, ...).

[Mar 13, 2019] amp html - Convert img to amp-img - Stack Overflow

Mar 13, 2019 | stackoverflow.com

> ,

Which is the default way to convert an <img> to a <amp-img> ?

I explain myself: In the site that I'm converting to AMP I have lot of images without widht and height e.g.:

<img src="/img/image.png" alt="My image">

If I not specify the layout, the layout="container" is set by default and the most of the images throw the following error:

amp-img error: Layout not supported for: container

In the other hand, the most of the images don't fit with the responsive layout, which is recommended by Google for most of the cases

I have been checking the types of layout on the documentation:

But any of them seems to fit with an image that have to be shown as its real size, not specifying width or height.

So, in that case, which is the equivalent in AMP?

,

As you are saying you have multiple images, it's better you use the ' layout="responsive" ', with that, you will make your images responsive atleast.

Now regarding the Width and Height . They are must.

If you read the purpose of AMP, one of them is to make the pages ' Jumping/Flickering Free Content ', which happens if there is no width mentioned for Images.

By Specifying the Width, the Browser (mobile browser), can calculate the precise space and keep it for that Image and show the Content after that. In that way, there wont' be any flickering of the content, as the page and images are loaded.

Regarding the re-writing of your HTML, one tip I can provide is, you can write some small utility with PHP, Python or Node JavaScript, which can actually read the source image, calculate their dimensions and replace your IMG tags.

Hope this helps and wish you good luck for your AMP powered site :-)

[Mar 10, 2019] How do I detach a process from Terminal, entirely?

Mar 10, 2019 | superuser.com

stackoverflow.com, Aug 25, 2016 at 17:24

I use Tilda (drop-down terminal) on Ubuntu as my "command central" - pretty much the way others might use GNOME Do, Quicksilver or Launchy.

However, I'm struggling with how to completely detach a process (e.g. Firefox) from the terminal it's been launched from - i.e. prevent that such a (non-)child process

For example, in order to start Vim in a "proper" terminal window, I have tried a simple script like the following:

exec gnome-terminal -e "vim $@" &> /dev/null &

However, that still causes pollution (also, passing a file name doesn't seem to work).

lhunath, Sep 23, 2016 at 19:08

First of all; once you've started a process, you can background it by first stopping it (hit Ctrl - Z ) and then typing bg to let it resume in the background. It's now a "job", and its stdout / stderr / stdin are still connected to your terminal.

You can start a process as backgrounded immediately by appending a "&" to the end of it:

firefox &

To run it in the background silenced, use this:

firefox </dev/null &>/dev/null &

Some additional info:

nohup is a program you can use to run your application with such that its stdout/stderr can be sent to a file instead and such that closing the parent script won't SIGHUP the child. However, you need to have had the foresight to have used it before you started the application. Because of the way nohup works, you can't just apply it to a running process .

disown is a bash builtin that removes a shell job from the shell's job list. What this basically means is that you can't use fg , bg on it anymore, but more importantly, when you close your shell it won't hang or send a SIGHUP to that child anymore. Unlike nohup , disown is used after the process has been launched and backgrounded.

What you can't do, is change the stdout/stderr/stdin of a process after having launched it. At least not from the shell. If you launch your process and tell it that its stdout is your terminal (which is what you do by default), then that process is configured to output to your terminal. Your shell has no business with the processes' FD setup, that's purely something the process itself manages. The process itself can decide whether to close its stdout/stderr/stdin or not, but you can't use your shell to force it to do so.

To manage a background process' output, you have plenty of options from scripts, "nohup" probably being the first to come to mind. But for interactive processes you start but forgot to silence ( firefox < /dev/null &>/dev/null & ) you can't do much, really.

I recommend you get GNU screen . With screen you can just close your running shell when the process' output becomes a bother and open a new one ( ^Ac ).


Oh, and by the way, don't use " $@ " where you're using it.

$@ means, $1 , $2 , $3 ..., which would turn your command into:

gnome-terminal -e "vim $1" "$2" "$3" ...

That's probably not what you want because -e only takes one argument. Use $1 to show that your script can only handle one argument.

It's really difficult to get multiple arguments working properly in the scenario that you gave (with the gnome-terminal -e ) because -e takes only one argument, which is a shell command string. You'd have to encode your arguments into one. The best and most robust, but rather cludgy, way is like so:

gnome-terminal -e "vim $(printf "%q " "$@")"

Limited Atonement ,Aug 25, 2016 at 17:22

nohup cmd &

nohup detaches the process completely (daemonizes it)

Randy Proctor ,Sep 13, 2016 at 23:00

If you are using bash , try disown [ jobspec ] ; see bash(1) .

Another approach you can try is at now . If you're not superuser, your permission to use at may be restricted.

Stephen Rosen ,Jan 22, 2014 at 17:08

Reading these answers, I was under the initial impression that issuing nohup <command> & would be sufficient. Running zsh in gnome-terminal, I found that nohup <command> & did not prevent my shell from killing child processes on exit. Although nohup is useful, especially with non-interactive shells, it only guarantees this behavior if the child process does not reset its handler for the SIGHUP signal.

In my case, nohup should have prevented hangup signals from reaching the application, but the child application (VMWare Player in this case) was resetting its SIGHUP handler. As a result when the terminal emulator exits, it could still kill your subprocesses. This can only be resolved, to my knowledge, by ensuring that the process is removed from the shell's jobs table. If nohup is overridden with a shell builtin, as is sometimes the case, this may be sufficient, however, in the event that it is not...


disown is a shell builtin in bash , zsh , and ksh93 ,

<command> &
disown

or

<command> &; disown

if you prefer one-liners. This has the generally desirable effect of removing the subprocess from the jobs table. This allows you to exit the terminal emulator without accidentally signaling the child process at all. No matter what the SIGHUP handler looks like, this should not kill your child process.

After the disown, the process is still a child of your terminal emulator (play with pstree if you want to watch this in action), but after the terminal emulator exits, you should see it attached to the init process. In other words, everything is as it should be, and as you presumably want it to be.

What to do if your shell does not support disown ? I'd strongly advocate switching to one that does, but in the absence of that option, you have a few choices.

  1. screen and tmux can solve this problem, but they are much heavier weight solutions, and I dislike having to run them for such a simple task. They are much more suitable for situations in which you want to maintain a tty, typically on a remote machine.
  2. For many users, it may be desirable to see if your shell supports a capability like zsh's setopt nohup . This can be used to specify that SIGHUP should not be sent to the jobs in the jobs table when the shell exits. You can either apply this just before exiting the shell, or add it to shell configuration like ~/.zshrc if you always want it on.
  3. Find a way to edit the jobs table. I couldn't find a way to do this in tcsh or csh , which is somewhat disturbing.
  4. Write a small C program to fork off and exec() . This is a very poor solution, but the source should only consist of a couple dozen lines. You can then pass commands as commandline arguments to the C program, and thus avoid a process specific entry in the jobs table.

Sheljohn ,Jan 10 at 10:20

  1. nohup $COMMAND &
  2. $COMMAND & disown
  3. setsid command

I've been using number 2 for a very long time, but number 3 works just as well. Also, disown has a 'nohup' flag of '-h', can disown all processes with '-a', and can disown all running processes with '-ar'.

Silencing is accomplished by '$COMMAND &>/dev/null'.

Hope this helps!

dunkyp

add a comment ,Mar 25, 2009 at 1:51
I think screen might solve your problem

Nathan Fellman ,Mar 23, 2009 at 14:55

in tcsh (and maybe in other shells as well), you can use parentheses to detach the process.

Compare this:

> jobs # shows nothing
> firefox &
> jobs
[1]  + Running                       firefox

To this:

> jobs # shows nothing
> (firefox &)
> jobs # still shows nothing
>

This removes firefox from the jobs listing, but it is still tied to the terminal; if you logged in to this node via 'ssh', trying to log out will still hang the ssh process.

,

To disassociate tty shell run command through sub-shell for e.g.

(command)&

When exit used terminal closed but process is still alive.

check -

(sleep 100) & exit

Open other terminal

ps aux | grep sleep

Process is still alive.

[Mar 10, 2019] How to run tmux/screen with systemd 230 ?

Aug 02, 2018 | askubuntu.com

MvanGeest ,May 10, 2017 at 20:59

I run 16.04 and systemd now kills tmux when the user disconnects ( summary of the change ).

Is there a way to run tmux or screen (or any similar program) with systemd 230? I read all the heated disussion about pros and cons of the behavious but no solution was suggested.

(I see the behaviour in 229 as well)

WoJ ,Aug 2, 2016 at 20:30

RemainAfterExit=

Takes a boolean value that specifies whether the service shall be considered active even when all its processes exited. Defaults to no.

jpath ,Feb 13 at 12:29

The proper solution is to disable the offending systemd behavior system-wide.

Edit /etc/systemd/logind.conf ( you must sudo , of course) and set

KillUserProcesses=no

You can also put this setting in a separate file, e.g. /etc/systemd/logind.conf.d/99-dont-kill-user-processes.conf .

Then restart systemd-logind.service .

sudo systemctl restart systemd-logind

sarnold ,Dec 9, 2016 at 11:59

Based on @Rinzwind's answer and inspired by a unit description the best I could find is to use TaaS (Tmux as a Service) - a generic detached instance of tmux one reattaches to.
# cat /etc/systemd/system/tmux@.service

[Unit]
Description=tmux default session (detached)
Documentation=man:tmux(1)

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/tmux new-session -d -s %I
ExecStop=/usr/bin/tmux kill-server
KillMode=none

[Install]
WantedBy=multiplexer.target

# systemctl start tmux@instanceone.service
# systemctl start tmux@instancetwo.service
# tmux list-sessions

instanceone: 1 windows (created Sun Jul 24 00:52:15 2016) [193x49]
instancetwo: 1 windows (created Sun Jul 24 00:52:19 2016) [193x49]

# tmux attach-session -t instanceone

(instanceone)#

Robin Hartmann ,Aug 2, 2018 at 20:23

You need to set the Type of the service to forking , as explained here .

Let's assume the service you want to run in screen is called minecraft . Then you would open minecraft.service in a text editor and add or edit the entry Type=forking under the section [Service] .

> ,

According to https://unix.stackexchange.com/a/287282/117599 invoking tmux using
systemd-run --user --scope tmux

should also do the trick.

[Mar 10, 2019] linux - How to attach terminal to detached process

Mar 10, 2019 | unix.stackexchange.com

Ask Question 86


Gilles ,Feb 16, 2012 at 21:39

I have detached a process from my terminal, like this:
$ process &

That terminal is now long closed, but process is still running and I want to send some commands to that process's stdin. Is that possible?

Samuel Edwin Ward ,Dec 22, 2018 at 13:34

Yes, it is. First, create a pipe: mkfifo /tmp/fifo . Use gdb to attach to the process: gdb -p PID

Then close stdin: call close (0) ; and open it again: call open ("/tmp/fifo", 0600)

Finally, write away (from a different terminal, as gdb will probably hang):

echo blah > /tmp/fifo

NiKiZe ,Jan 6, 2017 at 22:52

When original terminal is no longer accessible...

reptyr might be what you want, see https://serverfault.com/a/284795/187998

Quote from there:

Have a look at reptyr , which does exactly that. The github page has all the information.
reptyr - A tool for "re-ptying" programs.

reptyr is a utility for taking an existing running program and attaching it to a new terminal. Started a long-running process over ssh, but have to leave and don't want to interrupt it? Just start a screen, use reptyr to grab it, and then kill the ssh session and head on home.

USAGE

reptyr PID

"reptyr PID" will grab the process with id PID and attach it to your current terminal.

After attaching, the process will take input from and write output to the new terminal, including ^C and ^Z. (Unfortunately, if you background it, you will still have to run "bg" or "fg" in the old terminal. This is likely impossible to fix in a reasonable way without patching your shell.)

manatwork ,Nov 20, 2014 at 22:59

I am quite sure you can not.

Check using ps x . If a process has a ? as controlling tty , you can not send input to it any more.

9942 ?        S      0:00 tail -F /var/log/messages
9947 pts/1    S      0:00 tail -F /var/log/messages

In this example, you can send input to 9947 doing something like echo "test" > /dev/pts/1 . The other process ( 9942 ) is not reachable.

Next time, you could use screen or tmux to avoid this situation.

Stéphane Gimenez ,Feb 16, 2012 at 16:16

EDIT : As Stephane Gimenez said, it's not that simple. It's only allowing you to print to a different terminal.

You can try to write to this process using /proc . It should be located in /proc/ pid /fd/0 , so a simple :

echo "hello" > /proc/PID/fd/0

should do it. I have not tried it, but it should work, as long as this process still has a valid stdin file descriptor. You can check it with ls -l on /proc/ pid /fd/ .

See nohup for more details about how to keep processes running.

Stéphane Gimenez ,Nov 20, 2015 at 5:08

Just ending the command line with & will not completely detach the process, it will just run it in the background. (With zsh you can use &! to actually detach it, otherwise you have do disown it later).

When a process runs in the background, it won't receive input from its controlling terminal anymore. But you can send it back into the foreground with fg and then it will read input again.

Otherwise, it's not possible to externally change its filedescriptors (including stdin) or to reattach a lost controlling terminal unless you use debugging tools (see Ansgar's answer , or have a look at the retty command).

[Mar 10, 2019] linux - Preventing tmux session created by systemd from automatically terminating on Ctrl+C - Stack Overflow

Mar 10, 2019 | stackoverflow.com

Preventing tmux session created by systemd from automatically terminating on Ctrl+C Ask Question -1


Jim Stewart ,Nov 10, 2018 at 12:55

Since a few days I'm successfully running the new Minecraft Bedrock Edition dedicated server on my Ubuntu 18.04 LTS home server. Because it should be available 24/7 and automatically startup after boot I created a systemd service for a detached tmux session:

tmux.minecraftserver.service

[Unit]
Description=tmux minecraft_server detached

[Service]
Type=forking
WorkingDirectory=/home/mine/minecraftserver
ExecStart=/usr/bin/tmux new -s minecraftserver -d "LD_LIBRARY_PATH=. /home/mine/minecraftser$
User=mine

[Install]
WantedBy=multi-user.target

Everything works as expected but there's one tiny thing that keeps bugging me:

How can I prevent tmux from terminating it's whole session when I press Ctrl+C ? I just want to terminate the Minecraft server process itself instead of the whole tmux session. When starting the server from the command line in a manually created tmux session this does work (session stays alive) but not when the session was brought up by systemd .

FlKo ,Nov 12, 2018 at 6:21

When starting the server from the command line in a manually created tmux session this does work (session stays alive) but not when the session was brought up by systemd .

The difference between these situations is actually unrelated to systemd. In one case, you're starting the server from a shell within the tmux session, and when the server terminates, control returns to the shell. In the other case, you're starting the server directly within the tmux session, and when it terminates there's no shell to return to, so the tmux session also dies.

tmux has an option to keep the session alive after the process inside it dies (look for remain-on-exit in the manpage), but that's probably not what you want: you want to be able to return to an interactive shell, to restart the server, investigate why it died, or perform maintenance tasks, for example. So it's probably better to change your command to this:

'LD_LIBRARY_PATH=. /home/mine/minecraftserver/ ; exec bash'