Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Best Python books for system administrators

News

Selected Computer Books

Python -- Scripting language with generators (semi-corportines)

 Best C language textbooks

Pycharm IDE

Best books about Regular Expressions
Unix Administration Best Books for Learning JavaScript Best Compiler Construction Books LAMP Classic Computer Books Best Algorithms and Data Structures Books

Nikolai Bezroukov. Introduction to Perl for Unix System Administrators

Best Red Hat Books

Best Pascal Programming Books

Python history

Humor Etc

Python is one of the few languages that survived "great programming language extinction period" that we currently experience and which corresponds with "downsizing" of IT in general. During this period several previously popular languages lost large part of their popularity and became niche languages.

Python is one of the few programming languages for which many books are still printed.  Like it or not, more and more system administrators use Python instead of Perl.  But out of this avalanche of books very few books are of high quality or are "enhanced" books (books with elaborate web site or video lectures). So far I have found only two such books

  1. Python for Everybody- Exploring Data in Python 3 - Severance, Dr. Charles Russell, Blumenberg, Sue, Hauser, Elliott, Andrion, Aimee, 245 pages
  2. Automate the Boring Stuff with Python, 2nd Edition- Practical Programming for Total Beginner  by Al Sweigart 592 pages  (November 12, 2019)

While Python presents itself as a simple language in no way it is simple. It is a very convoluted language with ridiculously complex type system. That's probably why Python has surprisingly few good general above beginner level Python programming books. That might be a difficultly of describing in concise form complex features of a large scripting language. In this sense Python shares its problems with Perl (but in many areas Perl paradoxically looks much simpler then Python). Perl also has very few really good "intermediate" and "advanced" books published.

As of specialized one devoted to modules Modular Programming with Python Erik Westra is really good. It takes the key concepts of Python (which in some sense was derivative of Modula 3 language) and explain how to use it in nine chapters, covering all bases. 

Some books at least try to be better by trying to help the reader and providing along with Web access to examples and electronic copy of the book in ePub format. One example of this trend is The Quick Python Book, Second Edition by Vern Ceder. (2nd edition, January 15, 2010). This 400 pages paperback from Manning cost you $0.73 (for a used copy).  Purchase of the print book comes with an offer of a free PDF, ePub, and Kindle eBook from Manning. I think most Manning books are distributed on those terms. Also available is all code from the book.  Manning also has a free Python book Manning - Exploring Python Basics January 2019 ISBN 9781617296581 181 pages

No Starch press offer free ebook edition with every print book purchased from nostarch.com.

As eBook can be converted to HTML and edited, you can instantly have your notes, which you can enhance and expand as you learn the language. that also allow you to correct errors and provide you own comments to some portions of the text. Try examples that are relevant to you and comment of them is  IMHO the best way to learn the language for people who already know similar language (and most Linux  sysadmin know Bash and Perl), some (usually older hands ;-) also know C and C++.  

This how those days any decent author of a programming language book should behave.

In this sense O'Reilly is now far below the curve and they actually moved back and no longer offer one free chapter for their new books. Here is one review of Vern Ceder book:

 Alexandros Gezerlis "Alex Gezerlis" (Seattle, WA)(REAL NAME)   - See all my reviews

Probably the best book on Python 3 currently available, July 10, 2010

"The Quick Python Book, Second Edition" is Vernon Ceder's reworking of the well-received volume "The Quick Python Book" by Daryl Harms and Kenneth McDonald. Ceder has removed a number of specialized chapters on COM, C & C++ extensions, JPython, HTMLgen & Zope and, more important, he has brought the text completely up to date, covering Python 3.1.

Most Python texts out there describe Python 2.x, so this book's main competition is: a) Mark Summerfield's "Programming in Python 3: A complete introduction to the Python Language, Second Edition", and b) Mark Pilgrim's "Dive into Python 3", while two other major books have incorporated material on Python 3, namely c) James Payne's "Beginning Python: Using Python 2.6 and Python 3.1" and d) Mark Lutz's "Learning Python: Powerful Object-Oriented Programming, 4th Edition".

The Good: this book is nice and short. It assumes a certain level of competence/background, so it does not waste space introducing the language-independent basics of flow control, object orientation, exception handling, and so on. It is example-based, and unlike in Pilgrim's volume the first few examples are short and thus readable. Chapter 3 ("The Quick Python overview") can be used as a compact reference when you're done reading the book, and various tables throughout the book help it function as a reference. Unlike its competition, it doesn't spend chapter upon chapter on databases, networking, or web applications. Instead, such topics are covered in only one (short) chapter at the end of the book. Ceder offers useful advice on the interrelation between older and newer Python features, whether discussing how to be more idiomatic (e.g. in chapter 6 on the format method vs % formatting, and in chapter 14 when introducing the "with" statement) or how to migrate from Python 2 to Python 3 (he devotes chapter 22 to this topic). On the publisher's website you can find a list of errata as well as the complete source code for the book. There you will see a link to an "Author online" forum in which you can interact with Ceder; perhaps more important, everyone who buys a paper copy of the book may also download a free PDF version. It is to be hoped that other publishers will follow Manning's example.

The Bad: the author is very clear that the book is aimed at those with experience in another programming language.

Even so, in a few cases the assumptions are Python-specific (and hence unwarranted): one example is in chapter 5, where he lets us know that if x is a list then y=x[:] makes a copy of x, though this does not really explain why we cannot simply say y=x to accomplish the same goal.

Another example: in chapter 12 Ceder uses character ranges expressed with [], though these are introduced much later (in chapter 17). Similarly, chapter 3 is quite good if you've already come into contact with Python before (even fleetingly). If you haven't, it may be obfuscating (though you could always just skip it on the first read).

On a different note, this book does not contain exercises, though Summerfield's, Payne's, and Lutz's volumes do (along with answers). As mentioned in the previous paragraph, Ceder does not include too much extraneous stuff something which in my opinion is definitely a plus.

However, he does not say absolutely anything on threading while Summerfield has a chapter on the subject and Payne a section. Similarly, Ceder does not mention function annotations at all, while Summerfield and Lutz each have a section on them. Finally, Ceder keeps referring the reader to the Python documentation for more details, and this can get frustrating. On the other hand, I suppose it would have been impossible for the book to stay at its current 320 pages otherwise.

Ceder's writing is concise, but this does not imply that he covers only the bare minimum of material. To pick a relatively advanced topic as an example, Ceder spends 2 pages on metaclasses, Summerfield 4.5 pages, Pilgrim and Payne devote half a page each only in the context of the changes from Python 2 to 3, while Lutz, in keeping with the mammoth size of his book, spends more than 30 pages on the topic. This (arbitrarily chosen) example is in some ways indicative of the wider approaches taken by the various Python 3 book authors.

In a nutshell, the fact that this book is considerably shorter than its competitors does not mean that it is shallow. The compactness is due partly to the author's succinct style of writing (which is not opaque, however) and partly to the fact that it does not contain too much on database programming, web services, and so on. All in all, if you're looking for a solid book on Python 3 that you stand a reasonable chance of reading cover-to-cover, then this is the volume you should buy. Four and a half stars.

Alex Gezerlis

Two other relatively recent  intro Python books that also cover basics and version 3 are

Intermediate books

One book that might  deserve your attention is Fluent Python published in 2015.  for example this book  explains slices really well.  And they are more powerful than most people realize. this book is a good start from those who alreasy know similar scripting language, for example Perl, because for them inro books like above are rather boring and just create allergy to Python as in many cases it looks like Perl is iether more concise, or more powerful, or both.  Ramalho book shows the power of Python  constructs and as such really help to understand the language.  It also teaches your how to inspect Python bytecode. Inspecting Python bytecode is not too difficult, and is extremely helpful to see what is going on under the hood. Access to bytecode from interactive prompt  is actually one of the most interesting features of Python implementation.  Each chapter of the book contains Further Reading section which taken together alone worth the proce of the  books

www.amazon.com

Paperback: 770 pages
Publisher: O'Reilly Media; 1 edition (August 20, 2015)
Language: English
ISBN-10: 1491946008
ISBN-13: 978-1491946008
Product Dimensions: 9.2 x 7 x 1.4 inches

Jascha Casadio on October 30, 2015

An excellent text covering very advanced Python features.

Among the books that are currently populating my Goodread's wishlist are no less than 20 titles dedicated to the Python language. They range from Django up to pandas, passing through Twisted and Test-Driver Development. Time is limited, so they often end up waiting in queue for months. But when I've seen Fluent Python on that shelf I had to make it mine immediately and put it in front of that queue. Getting through this book took me several months, not only because we are talking about some 700 hundreds good pages, but mostly due to the fact that it covers advanced topics that most of the Pythonists currently living on planet Earth never heard of in their life. Fluent Python is one of those books that you must taste little by little or you get devoured by those fierce topics and examples.

Released late this summer, Fluent Python is the latest work of Ramalho, a name that should sound familiar to those that have been already diving deeply into, allow me the term, Python's high-end features, powerful things, such as coroutines, that most developers never heard of in their life. Those that did probably hope never being tested on them during a job interview. And that's pretty much what the book is all about. Neither style nor the the basics of the language, but very advanced features. Quite a rare book indeed, since almost all of the Python books available introduce the readers to the language and don't get past Object Oriented Programming.

An excellent text overall, no doubts. Not for the faint of heart. Still, I am a bit puzzled by the fact that some chapters look extremely simple, others cover quirks and intricacies that you can probably live without, unless you dare touching the very core of the language ,and that get you to reach the end of a chapter with that what the hell expression on your face. The chapter covering abstract classes is an example of the former. Don't get me wrong, it's interesting and the examples well laid out. Still, it looks a basic concept that doesn't fit this kind of book.

A couple of words on the examples: they are throughout the whole book well done. The author often presents the same concepts in different flavors or does work on the same example and improves it as concepts are taken into the discussion. The code is intense but easy to follow. Key lines are extensively explained later on, so that the reader won't miss that specific features that makes it all possible. There are so many gems that you will probably end up writing most of that code down to make it yours. This is actually the best thing the reader can do. Try it, modify it, assimilate it, master it.

Among the many topics covered there are two that are worth mentioning: the first is chapter four, which covers strings, Unicode and bytes. Marvelous, simply marvelous. The examples, the explanations. So clear and to the point. You definitely get away from it with a deep understanding of how strings work in Python 2.7 and 3.

The second is that dedicated to futures. Actually it's the whole topic, which spans several chapters at the very end of the book. The authors shows how working with threads and subprocesses improve the efficiency of an application, and how easy it is to exploit them through the futures that are now available in the language. He does gives us a very interesting example in many different flavors, showing us how the code and performance change. Great.

Decorators and closures are also well described, even if not as good as the aforementioned topics. In that sense, the author does complement what we find about the subject in Effective Python: 59 Specific Ways to Write Better Python, another must have for any serious Pythonist.

Overall, a great Python book. A must have for any Python developer interested in getting the most out of the language.

As usual, you can find more reviews on my personal blog: http://books.lostinmalloc.com . Feel free to pass by and share your thoughts!

Python Cookbook

Python Cookbook, Third edition  by David Beazley, Brian K. Jones  is a really good  intermediate book which  acrutally teaches "by example" you how to program in Python.  It was published in 2013 and has over around 700 pages.

Here are some Amazon review

renaissance geek on June 23, 2013

The Python Domestic Science Textbook?

A few years ago now I was working in a job that required me to code in PERL. My PERL is passable but no better than that so when I found a copy of the PERL cookbook it was something of a life saver and constant companion. The PERL cookbook is deeply pragmatic and addresses real world problems with the language almost as an afterthought. (Which now I think about is actually a pretty good description of PERL anyway!) The Python cookbook is a very different beast and is much more an exercise in learning the intricacies and nuances of the language. I'm not sure cookbook is the right title - if the PERL Cookbook is a cookbook then the Python Cookbook is more of a domestic science textbook. A bit deeper, a bit dryer and not so focused on immediate problems. This is no way meant to imply that it's a bad book, on the contrary it's a very good book just not entirely what I was expecting.

The book itself is divided into fifteen large sections covering the likes of data structures and algorithms; functions; metaprogramming and concurrency with each section consisting of a number of problems. The problems are structured as a definition of the problem, a solution and a discussion of the solution and how it can be extended. Due to the nature of the Python language a large part of solving the problems lies in knowing which module(s) to include in your code so each problem is generally only a couple of pages, but that is certainly enough to give the solution and reasonably detailed discussion.

As with all books of this type there is going to be some complaints of why is X included and not Y and to be honest if you tried to cover all the possible problems a practicing python programmer is likely to run across the book would end up so large as to be unusable. That being said there was, for me at least, one glaring omission.

I do a lot of data processing with reasonably large data sets, and with the buzz around big data I'm sure I'm not the only one, and frequently find that I have to break down the data sets or I simply consume all the system resources and the program exits. I would have expected at least some treatment of working with very large data sets which seems to be entirely missing.

However this is an issue based on what I use Python for and may very well not matter to you. Even though there may not be exactly the solution you are looking for, there are 260 problems and solutions in the Python cookbook so if you don't learn something new you are probably a certified Python genius and beyond manuals anyway.

There are a couple of quick final points to make about the Python cookbook. Firstly it uses Python 3 and as many very useful third party modules haven't been ported from Python 2.X over to Python 3 yet Python 2.X is still probably still more widely used.

Secondly although this is a language learning book it's not aimed at the novice programmer, think of it more as language nuances and inflections for the experienced Pythonista rather than a how to learn Python book and you won't go far wrong.

Bluegeek on June 13, 2013
Review: "Python Cookbook" by David Beazley and Brian K. Jones; O'Reilly Media

The "Python Cookbook" is a book that brings the Python scripting language to O'Reilly's popular "Cookbook" format. Each Cookbook provides a series of "Recipes" that teach users common techniques that can be used to become productive quickly and as a reference to those who might've forgotten how to do something.

I reviewed this book in the Mobi e-book format. Reading it on Kindle for PC, the Table of Contents only shows the major sections rather than the individual recipes and this made it harder to find what I was looking for. This is apparently a limitation of Kindle for PC, since my Kindle 3 and Kindle for Android had no such issue.

When I use an O'Reilly "Cookbook", I judge it according to its' usefulness: Can I become productive quickly? Is it easy to find what I need? Does it provide helpful tips? Does it teach me where to find the answers to my questions?

This book is not targeted at new Python programmers, but that's where I'm at. The best way for me to learn a new scripting language is to dive right in and try to write something useful, and that was my goal for the "Python Cookbook". I also had "Learning Python" handy to cover any of the basics.

My first Python script was written to read in lists of subnets from two separate files and check that every subnet in list B was also in list A.

I used Recipe 13.3 to parse the command line options. Recipe 5.1 showed me how to read and write files. Recipe 2.11 taught me how to strip carriage returns out of my lines. Recipe 1.10, "Removing Duplicates from a Sequence while Maintaining Order", was very helpful and I was able to reuse the code in my own script. Recipe 2.14, "Combining and Concatenating Strings", helped me with my print statements. Considering this was the first Python script I ever wrote and that it ran, I consider both it and the "Python Cookbook" a success.

I had a bit more trouble with my second script. I was trying to write a script to find the subnet address given an interface address in CIDR notation. Recipe 11.4 introduced the ipaddress module, but this module refused to accept a string variable containing the interface in CIDR notation. I ended up installing another module (netaddr) I found via Google and things went better after that. I suspect the problem was that I was using ActivePython 2.7.2.5 [64 bit] and this book was written for Python 3.

As a DNS professional I was disappointed that there were no DNS-related recipes in the Network and Web Programming section, but Web-related topics were well-represented in the book.

The "Python Cookbook" doesn't seem to have quite the depth and organization of the "Perl Cookbook" but I'm sure I will rely on it heavily as I learn to use Python. It did allow me to be productive very quickly and it passes the "Cookbook" standard with flying colors. Any book that can get me to the point of writing a working, useful script in less than a day is worth using. I recommend this book to anyone who has a basic understanding of Python and wants to get past "Hello, World" and "Eat Spam" as fast as possible.

Reviewer's Note: I received a free copy of the "Python Cookbook" which was used to write this review.

William P Ross Enthusiast: Architecture on May 6, 2016
Treasure Trove of Python Recipes

Python Cookbook goes in depth on a variety of different Python topics. Each section is similar to a question that might be asked on Stack Overflow. The recipes range in difficulty from easy to advanced metaprogramming.

One particular recipe that I liked was 9.1 on how to time a function. When I am using Python I often need to time the code, and usually I need to look up how to do it. This example created a decorator function for timing. It makes it so that you can just put @timethis on top of a function and see how long it takes to execute. I appreciated how elegant this solution was as opposed to the way I was implementing it.

Most examples are self contained and all the code examples that I tried worked. Additionally, there is a GitHub that the authors created which provides all the code for the examples if you do not want type it yourself. The examples themselves were applied to real world problems; I could see how the recipe was used clearly. When the authors felt they could not provide an entire solution in the text, they point the correct place to visit online.

The range in topics was impressive. I found the most challenging chapters to be 9, 12, and 15 which were on metaprogramming, concurrency, and C Extensions. At the beginning of the book the recipes cover topics you would expect like data structures and algorithms, strings, and generators. I found myself surprised that I had not seen a lot of the techniques and solutions before. They were well crafted solutions, and I appreciated how much time and detail the authors must have spent to gather the information.

This is a great reference to have by your side when programming in Python.

Groundhog Day on June 30, 2015
Programming Pearls... Reloaded

Having read some humdrum works in the Cookbook series, my expectations were not very high. However, I soon discovered that this book is in a different league.

When he discusses a problem, Beazley gives you his favorite solution. He also presents alternatives, discusses pros and cons, and calls your attention to subtle details in the solution --- leaving you with a feeling of having learned something of value.

But that's only the beginning. It's hard to describe the pleasure of reading some of the solutions in the Iterators and Generators section, for instance. Actually, I take that back. The pleasure is the same kind as what you may have felt when you first came upon ideas in books such as Bentley's Programming Pearls, way back when.

I hadn't felt that excited about a programming book in a long time. This is one you can take along with you on a weekend just for the pleasure of sipping from it. Sad to say, but there are many O'Reilly books I feel like passing on soon after acquiring them. This one will have a special place on the shelves.

Devendra on September 1, 2013
Extensive tome of recipes for the Python 3 programmer

Python Cookbook is an extensive tome of recipes for the Python 3 programmer. It is a perfect companion book for those migrating Python 2 code to Python 3. If you are stuck with Python 2, you may still find the second edition of the book for sale, but the recipes may be dated as they cover Python 2.4. It is not a beginners book. If you are looking for a beginners book, I recommend Learning Python by Mark Lutz.

A quick chapter summary follows.

I've added this book to my list of references to look into, before heading to Google. Source code listings use syntax highlighting, a nice touch that makes the code easier, and less boring, to read.

I thank O'Reilly media for providing the book for review.

Dan on July 23, 2013
Wisdom - not just examples. Best viewed on a larger screen

The Active State repository of Python recipes includes many gems, but as the Authors observe in their preference: "most of these recipes are steeped in history and the past".

I'd add that the signal to noise ratio seems to be decreasing. The most prolific contributors (with the exception of Raymond Hettinger) have posted trivial examples rather than recipes. This book includes some simple examples too, but it's always in the context of a larger message. Excellent content and advice without the chaff.

I just bought this today. Unlike some early technical Kindle books I've purchased, the formatting is excellent. Kudos to the authors and publisher.

... ... ...

A. Zubarev on September 17, 2013
A book to read and come back again and again

I am tempted to state right away that this book is one of these rare "gems"! Absolutely worth every penny spent and perhaps even more in a way of getting more done in less time or even just can be used to advance professionally. So big thank you to Alex Martelli and David Ascher! I can't imagine how much time, energy, insight and effort the authors put into this book, but it is sure one of the longest professional books I have ever read.

Like I said, this book is very comprehensive at 608 pages long and touches most, if not all, aspects a typical IT pro would deal with in his or her professional life. It may appear though very dry, and in my opinion it should be, but it is the book to come back to again and again, time after time, year after year, so if you need a single specific recipe, you will not feel the book is very short thanks to the way it is structured.

I happen to actually use this book to cope with several assignments at work involving some medium to high complexity data processing for reporting purposes, thus more than a few recipes were used.

Namely, these were "Strings and Text" Ch. 2, "Numbers, Dates and Times" Ch. 3, "Files and I/O" Ch. 4, then hopped to "Functions" Ch. 7, which followed by "Parsing, Modifying and Rewriting XML" Ch. 6.6 and finally landed on "Integrating with a Relational Database" Ch. 6.8. I wish though chapter 7 "Functions" would precede most others because I think it belongs right after "Iterators and generators" which I needed to use as I expanded my program.

I must tell each did its magic, after all Python excels on processing text!

... ... ...

Books for Perl programmers

There is another intermediate book about Python specifically written for Perl programmers who wish to learn a new language.  Highly recommended.

Perl to Python Migration by Martin Brown

Nov 02, 2001 | Amazon.com

Paperback: 400 pages

Publisher: Addison-Wesley Professional; 1st edition (November 2, 2001)

Language: English
ISBN-10: 0201734885
ISBN-13: 978-0201734881
Product Dimensions: 9.2 x 7.4 x 0.7 inches
Shipping Weight: 1.3 pounds

My conclusion

Python is an excellent language for my intended use. It is a good language for many of the applications that one would use Lisp as a rapid prototyping environment for. The three main drawbacks are (1) execution time is slow, (2) there is very little compile-time error analysis, even less than Lisp, and (3) Python isn't called "Java", which is a requirement in its own right for some of my audience. I need to determine if JPython is close enough for them.

Python can be seen as either a practical (better libraries) version of Scheme, or as a cleaned-up (no $@&%) version of Perl. While Perl's philosophy is TIMTOWTDI (there's more than one way to do it), Python tries to provide a minimal subset that people will tend to use in the same way. One of Python's controversial features, using indentation level rather than begin/end or {/}, was driven by this philosophy: since there are no braces, there are no style wars over where to put the braces. Interestingly, Lisp has exactly the same philosophy on this point: everyone uses emacs to indent their code. If you deleted the parens on control structure special forms, Lisp and Python programs would look quite similar.

Python has the philosophy of making sensible compromises that make the easy things very easy, and don't preclude too many hard things. In my opinion it does a very good job. The easy things are easy, the harder things are progressively harder, and you tend not to notice the inconsistencies.

Lisp has the philosophy of making fewer compromises: of providing a very powerful and totally consistent core. This can make Lisp harder to learn because you operate at a higher level of abstraction right from the start and because you need to understand what you're doing, rather than just relying on what feels or looks nice. But it also means that in Lisp it is easier to add levels of abstraction and complexity; Lisp makes the very hard things not too hard.

Table of Contents

CONTENTS

Introduction
Python Overview
Python Execution
Components of a Python Program
Functions
Exceptions
Modules, Classes, and Object Orientation
Processing Information
Interfacing to the OS
File Manipulation
Data Management
Communicationing over a Network
Using Python for Web Development
Interface Building with Tk
Resource Guide - (in conjunction with material on the web) Preface

As a long time Perl and Python programmer, I'm constantly finding myself using Perl constructs in Python or Python constructs in Perl. Worse, I can suddenly find myself trying to use a feature or system in a Python program that just doesn't exist, or does and works differently. What I needed was some sort of cross reference.

After talking to other programmers and developers it became apparent that a number of people were "jumping ship" from Perl to Python. Some with varying levels of success. The reasons for using Python in preference to Perl were different for each person, but there were some common themes:

Python is object-oriented. Although Perl has object orientation facilities they are not integrated into the system. In Python everything is an object or class -- even the base data types are objects and have their own methods and attributes. This makes Python a great language to use in situations where you are working on applications that use object orientation as a core technology.

Python is easy to use. Python has a very clean and structured layout, and it's very easy to follow what's going on. Perl can frequently look like line noise, and particularly for new programmers this becomes a problem as they try to understand why their program works.

Python does what you tell it to. Perl suffers from a severe case of semi-intelligence. Statements and expressions in Perl have all sorts of rules, exceptions to those rules, and other artefacts that can make it difficult to follow what is going on. Just think about what happens when you call a function, was it called in list or scalar context?, what arguments did it have?, how do I get them back? These are regular questions when programming with Perl.

Python is cross-platform compatible. Perl is also cross-platform compatible, for certain values of compatible. Because of Perl's Unix roots -- it is very much a Unix language -- use Perl under Windows or MacOS and a large proportion of the built-in functions become obsolete. Python's functionality is supported by external modules, many of which use the correct version according to their platform whilst retaining the same API.

These aren't really hard and fast reasons for why you should make the jump to Python, but they are compelling reasons for why Python is at least worth a look.

In this book that's exactly what we try to do -- show you how you can make the move from Perl to Python. Many of the basics are the same, and despite a change of language, you're unlikely to need to change the algorithms you are already using.

For the more advanced programmer, or, like me, the programmer who has to use both languages most days, the book should also work as a handy cross reference to allow you to look up Perl and Python statements and functions, and find out what you should use in both languages.

Who is the book for?

Anybody who has used Perl and who wants or needs to learn Python will find this book useful. The book gives a full guide on the basics of programming in Python from the perspective of a Perl programmer. Right from the basics of line and program structure all the way up to object-orientation and using external modules.

You don't need to be an expert Perl programmer , you just need to be able to understand the basics. I've tried to make the text as easy to read and follow as possible. However, on occasion things get technical, and to get the best out of the book you should at least know how to use and create your own functions, extension modules and classes and objects.

Without trying to give you the plot before you've read the book, if you know how to work with references, classes and objects in Perl, then your migration to Python is already 50% there!

How to use this book

The book is split into three main sections. The first section gives you an introduction to Python, including information on the basic mechanics of the language and information on who and where it is used.

Section 2 concentrates on the fundamentals of the language from the basics of expressions and statements through to comparing Python's object types with Perl variables. We'll also look at creating functions, error handling, and extending Python with modules and classes.

The last section concentrates on applying Python in real situations compared to the equivalent Perl process, function or module. For example, Chapter 8 looks at Python's built-in functions and their Perl equivalents, whilst Chapter 10 looks at manipulating data and regular expressions.

There are also two appendices at the back of the book. Appendix A provides a handy guide to getting more information about Python, including other books, websites, newsgroups, mailing lists and details on how to get the Python documentation to help you with your programming. Appendix B is a quick reference -- it lists every Perl token, variable or function and provides you with the Python alternative. Where possible, it will also direct you straight to the page elsewhere in the book where you can find more information on the Python alternative.

If you are completely new to Python but know Perl then read Sections 1 and 2 first. If you've played with Python but never used it in anger, then start with Section 2. Finally if you know Python and just want quick reference information then use Appendix B to look up the Perl fragment you are looking for, or go straight to the suitable chapters in sections 2 and 3.

If you don't know Perl, then this book will not be a lot of help. Try reading Perl: The Complete Reference 2nd Edition (Osborne/McGraw-Hill, 2001) or Learning Perl, 2nd Edition (O'Reilly, 1997).

Chapter breakdown

There are fourteen chapters and two appendices in the book. Throughout the book examples of a Perl expression or statement are given, along with the equivalent Python version and information on why the changes and modifications are required. The rough content of each chapter is as follows:

Conventions Used in This Book

All Perl keywords are highlighted in bold, but functions are listed without parentheses. This is because the C functions on which the Perl versions may be based are shown like this().

Examples and code are displayed using a fixed-width font.

Function descriptions are formatted using the same fixed-width font.

Note: notes are formatted like this and include additional information about a particular topic. You'll also find similarly formatted "Warnings," which highlight possible dangerous tools or tricks to watch out for when programming.

Contacting the Author

I always welcome comments and suggestions on my work. The best way to contact me is via e-mail. You can use either books@mcwords.com. Alternatively, visit my website, http://www.mcwords.com, which contains resources and updated information about the scripts and contents of this book. You can find the homepage for this book at http://www.mcwords.com/projects/books/p2py/. A Customer on January 5, 2002

Very good language cross reference

I'm an avid Perl programmer who's wanted to do more with Python, but never seem to have the time to actually _do_ something in Python because I run out of time and end up doing it in Perl. This book has helped me get over that hurdle by providing a useful cross reference, so if you want to do X in Perl (with a perl code example that looks very typical), do Y in Python (with a plausible code fragment).

I would have rated this book 5 stars, as it very clearly achieves its stated goal. Unfortunately it is absolutely riddled with minor typo's. Perhaps that will be fixed in a future edition, but for now it makes life a bit more difficult since you have to pay very close attention to what is in the book vs. what you might already know and correct the book where it is wrong.

A Customer on April 26, 2002
Useful book, despite its errors

If you happen to be a Perl Hacker who wants to get into Python, this book is just what you're looking for. It's written from the perspective of a Perl programmer who has made the switch to Python. It introduces Python while explaining the relevant differences to Perl as it goes. There are also lots of code snippets showing Perl code along with its Python equivalents.

This book does have some problems however. It has more typos in it than any other programming book I've ever read. That applies to both the text itself as well as some of the code samples. There are also some places in which it explains things assuming knowledge of something else that isn't described until further on in.

That said, I still found the book useful, and for me it was probably worth the purchase price. I would just recommend that readers be very wary of the typos as you go along.

 

Python books lists

PythonBooks - Python Wiki

External sources of information about (Python) books:

 

Free Python books

There are several sites that list free Python books. Most of such books are outdated. Still some preserve their  value:


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Sep 19, 2020] Using global variables in a function

Jan 01, 2009 | stackoverflow.com

user46646 , 2009-01-08 05:45:02

3203 685

How can I create or use a global variable in a function?

If I create a global variable in one function, how can I use that global variable in another function? Do I need to store the global variable in a local variable of the function which needs its access? python global-variables scope share improve this question follow edited May 20 '18 at 10:47 Aran-Fey 27.4k 5 5 gold badges 66 66 silver badges 107 107 bronze badges asked Jan 8 '09 at 5:45 user46646 125k 41 41 gold badges 72 72 silver badges 82 82 bronze badges

> ,

add a comment 20 Answers Active Oldest Votes

Paul Stephenson , 2009-01-08 08:39:44

4338

You can use a global variable within other functions by declaring it as global within each function that assigns a value to it:

globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print(globvar)     # No need for global declaration to read value of globvar

set_globvar_to_one()
print_globvar()       # Prints 1

I imagine the reason for it is that, since global variables are so dangerous, Python wants to make sure that you really know that's what you're playing with by explicitly requiring the global keyword.

See other answers if you want to share a global variable across modules. share improve this answer follow edited Aug 11 at 21:32 Rayner Da Cruz 163 2 2 gold badges 4 4 silver badges 11 11 bronze badges answered Jan 8 '09 at 8:39 Paul Stephenson 57.3k 8 8 gold badges 44 44 silver badges 49 49 bronze badges

avgvstvs ,

It's extreme exaggeration to refer to globals as "so dangerous." Globals are perfectly fine in every language that has ever existed and ever will exist. They have their place. What you should have said is they can cause issues if you have no clue how to program. – Anthony Dec 22 '12 at 23:22

Jeff Shannon , 2009-01-08 09:19:55

785

If I'm understanding your situation correctly, what you're seeing is the result of how Python handles local (function) and global (module) namespaces.

Say you've got a module like this:

# sample.py
myGlobal = 5

def func1():
    myGlobal = 42

def func2():
    print myGlobal

func1()
func2()

You might expecting this to print 42, but instead it prints 5. As has already been mentioned, if you add a ' global ' declaration to func1() , then func2() will print 42.

def func1():
    global myGlobal
    myGlobal = 42

What's going on here is that Python assumes that any name that is assigned to , anywhere within a function, is local to that function unless explicitly told otherwise. If it is only reading from a name, and the name doesn't exist locally, it will try to look up the name in any containing scopes (e.g. the module's global scope).

When you assign 42 to the name myGlobal , therefore, Python creates a local variable that shadows the global variable of the same name. That local goes out of scope and is garbage-collected when func1() returns; meanwhile, func2() can never see anything other than the (unmodified) global name. Note that this namespace decision happens at compile time, not at runtime -- if you were to read the value of myGlobal inside func1() before you assign to it, you'd get an UnboundLocalError , because Python has already decided that it must be a local variable but it has not had any value associated with it yet. But by using the ' global ' statement, you tell Python that it should look elsewhere for the name instead of assigning to it locally.

(I believe that this behavior originated largely through an optimization of local namespaces -- without this behavior, Python's VM would need to perform at least three name lookups each time a new name is assigned to inside a function (to ensure that the name didn't already exist at module/builtin level), which would significantly slow down a very common operation.) share improve this answer follow edited Mar 16 '15 at 18:01 Michael 5,138 3 3 gold badges 48 48 silver badges 69 69 bronze badges answered Jan 8 '09 at 9:19 Jeff Shannon 8,673 1 1 gold badge 13 13 silver badges 7 7 bronze badges

BlackJack ,

You mentioned that the namespace decision happens at compile time , I don't think it is true. from what I learn python's compilation only checks for syntax error, not name error try this example def A(): x+=1 , if you don't run it, it will not give UnboundLocalError , please verify thank you – watashiSHUN Oct 12 '15 at 22:36

gimel , 2009-01-08 05:59:04

230

You may want to explore the notion of namespaces . In Python, the module is the natural place for global data:

Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user's global variables. On the other hand, if you know what you are doing you can touch a module's global variables with the same notation used to refer to its functions, modname.itemname .

A specific use of global-in-a-module is described here - How do I share global variables across modules? , and for completeness the contents are shared here:

The canonical way to share information across modules within a single program is to create a special configuration module (often called config or cfg ). Just import the configuration module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example:

File: config.py

x = 0   # Default value of the 'x' configuration setting

File: mod.py

import config
config.x = 1

File: main.py

import config
import mod
print config.x
share improve this answer follow edited May 21 '19 at 10:33 Georgy 4,728 5 5 gold badges 37 37 silver badges 46 46 bronze badges answered Jan 8 '09 at 5:59 gimel 69.2k 10 10 gold badges 68 68 silver badges 103 103 bronze badges

Psychzander ,

for a reason I don't like the config.x can I get rid of it? I came with x = lambda: config.x and then I have the new value in x() . for some reason, having a = config.x does not do the trick for me. – vladosaurus Jan 2 '18 at 14:56

SingleNegationElimination ,

96

Python uses a simple heuristic to decide which scope it should load a variable from, between local and global. If a variable name appears on the left hand side of an assignment, but is not declared global, it is assumed to be local. If it does not appear on the left hand side of an assignment, it is assumed to be global.

>>> import dis
>>> def foo():
...     global bar
...     baz = 5
...     print bar
...     print baz
...     print quux
... 
>>> dis.disassemble(foo.func_code)
  3           0 LOAD_CONST               1 (5)
              3 STORE_FAST               0 (baz)

  4           6 LOAD_GLOBAL              0 (bar)
              9 PRINT_ITEM          
             10 PRINT_NEWLINE       

  5          11 LOAD_FAST                0 (baz)
             14 PRINT_ITEM          
             15 PRINT_NEWLINE       

  6          16 LOAD_GLOBAL              1 (quux)
             19 PRINT_ITEM          
             20 PRINT_NEWLINE       
             21 LOAD_CONST               0 (None)
             24 RETURN_VALUE        
>>>

See how baz, which appears on the left side of an assignment in foo() , is the only LOAD_FAST variable. share improve this answer follow answered Jul 12 '11 at 12:35 SingleNegationElimination 131k 24 24 gold badges 238 238 silver badges 280 280 bronze badges

,

The heuristic looks for binding operations . Assignment is one such operation, importing another. But the target of a for loop and the name after as in with and except statements also are bound to. – Martijn Pieters ♦ Aug 8 '15 at 23:56

J S , 2009-01-08 09:03:33

64

If you want to refer to a global variable in a function, you can use the global keyword to declare which variables are global. You don't have to use it in all cases (as someone here incorrectly claims) - if the name referenced in an expression cannot be found in local scope or scopes in the functions in which this function is defined, it is looked up among global variables.

However, if you assign to a new variable not declared as global in the function, it is implicitly declared as local, and it can overshadow any existing global variable with the same name.

Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill. share improve this answer follow edited Mar 4 '17 at 22:00 Peter Mortensen 26.5k 21 21 gold badges 92 92 silver badges 122 122 bronze badges answered Jan 8 '09 at 9:03 J S 957 6 6 silver badges 6 6 bronze badges

Paul Uszak ,

Absolutely re. zealots. Most Python users use it for scripting and create little functions to separate out small bits of code. – Paul Uszak Sep 22 '19 at 22:57

Aaron Hall , 2016-01-01 19:55:14

54
If I create a global variable in one function, how can I use that variable in another function?

We can create a global with the following function:

def create_global_variable():
    global global_variable # must declare it to be a global first
    # modifications are thus reflected on the module's global scope
    global_variable = 'Foo'

Writing a function does not actually run its code. So we call the create_global_variable function:

>>> create_global_variable()
Using globals without modification

You can just use it, so long as you don't expect to change which object it points to:

For example,

def use_global_variable():
    return global_variable + '!!!'

and now we can use the global variable:

>>> use_global_variable()
'Foo!!!'
Modification of the global variable from inside a function

To point the global variable at a different object, you are required to use the global keyword again:

def change_global_variable():
    global global_variable
    global_variable = 'Bar'

Note that after writing this function, the code actually changing it has still not run:

>>> use_global_variable()
'Foo!!!'

So after calling the function:

>>> change_global_variable()

we can see that the global variable has been changed. The global_variable name now points to 'Bar' :

>>> use_global_variable()
'Bar!!!'

Note that "global" in Python is not truly global - it's only global to the module level. So it is only available to functions written in the modules in which it is global. Functions remember the module in which they are written, so when they are exported into other modules, they still look in the module in which they were created to find global variables.

Local variables with the same name

If you create a local variable with the same name, it will overshadow a global variable:

def use_local_with_same_name_as_global():
    # bad name for a local variable, though.
    global_variable = 'Baz' 
    return global_variable + '!!!'

>>> use_local_with_same_name_as_global()
'Baz!!!'

But using that misnamed local variable does not change the global variable:

>>> use_global_variable()
'Bar!!!'

Note that you should avoid using the local variables with the same names as globals unless you know precisely what you are doing and have a very good reason to do so. I have not yet encountered such a reason.

We get the same behavior in classes

A follow on comment asks:

what to do if I want to create a global variable inside a function inside a class and want to use that variable inside another function inside another class?

Here I demonstrate we get the same behavior in methods as we do in regular functions:

class Foo:
    def foo(self):
        global global_variable
        global_variable = 'Foo'

class Bar:
    def bar(self):
        return global_variable + '!!!'

Foo().foo()

And now:

>>> Bar().bar()
'Foo!!!'

But I would suggest instead of using global variables you use class attributes, to avoid cluttering the module namespace. Also note we don't use self arguments here - these could be class methods (handy if mutating the class attribute from the usual cls argument) or static methods (no self or cls ). share improve this answer follow edited Jan 19 at 14:41 answered Jan 1 '16 at 19:55 Aaron Hall ♦ 258k 69 69 gold badges 352 352 silver badges 302 302 bronze badges

anonmanx ,

Cool, but what to do if I want to create a global variable inside a function inside a class and want to use that variable inside another function inside another class? Kinda stuck here – anonmanx Jan 19 at 14:29

Rauni Lillemets ,

50

In addition to already existing answers and to make this more confusing:

In Python, variables that are only referenced inside a function are implicitly global . If a variable is assigned a new value anywhere within the function's body, it's assumed to be a local . If a variable is ever assigned a new value inside the function, the variable is implicitly local, and you need to explicitly declare it as 'global'.

Though a bit surprising at first, a moment's consideration explains this. On one hand, requiring global for assigned variables provides a bar against unintended side-effects. On the other hand, if global was required for all global references, you'd be using global all the time. You'd have to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the global declaration for identifying side-effects.

Source: What are the rules for local and global variables in Python? . share improve this answer follow edited Jul 20 '14 at 10:36 Peter Mortensen 26.5k 21 21 gold badges 92 92 silver badges 122 122 bronze badges answered Jul 4 '14 at 10:23 Rauni Lillemets 1,924 1 1 gold badge 22 22 silver badges 32 32 bronze badges

> ,

add a comment

Bohdan , 2013-10-03 05:41:16

36

With parallel execution, global variables can cause unexpected results if you don't understand what is happening. Here is an example of using a global variable within multiprocessing. We can clearly see that each process works with its own copy of the variable:

import multiprocessing
import os
import random
import sys
import time

def worker(new_value):
    old_value = get_value()
    set_value(random.randint(1, 99))
    print('pid=[{pid}] '
          'old_value=[{old_value:2}] '
          'new_value=[{new_value:2}] '
          'get_value=[{get_value:2}]'.format(
          pid=str(os.getpid()),
          old_value=old_value,
          new_value=new_value,
          get_value=get_value()))

def get_value():
    global global_variable
    return global_variable

def set_value(new_value):
    global global_variable
    global_variable = new_value

global_variable = -1

print('before set_value(), get_value() = [%s]' % get_value())
set_value(new_value=-2)
print('after  set_value(), get_value() = [%s]' % get_value())

processPool = multiprocessing.Pool(processes=5)
processPool.map(func=worker, iterable=range(15))

Output:

before set_value(), get_value() = [-1]
after  set_value(), get_value() = [-2]
pid=[53970] old_value=[-2] new_value=[ 0] get_value=[23]
pid=[53971] old_value=[-2] new_value=[ 1] get_value=[42]
pid=[53970] old_value=[23] new_value=[ 4] get_value=[50]
pid=[53970] old_value=[50] new_value=[ 6] get_value=[14]
pid=[53971] old_value=[42] new_value=[ 5] get_value=[31]
pid=[53972] old_value=[-2] new_value=[ 2] get_value=[44]
pid=[53973] old_value=[-2] new_value=[ 3] get_value=[94]
pid=[53970] old_value=[14] new_value=[ 7] get_value=[21]
pid=[53971] old_value=[31] new_value=[ 8] get_value=[34]
pid=[53972] old_value=[44] new_value=[ 9] get_value=[59]
pid=[53973] old_value=[94] new_value=[10] get_value=[87]
pid=[53970] old_value=[21] new_value=[11] get_value=[21]
pid=[53971] old_value=[34] new_value=[12] get_value=[82]
pid=[53972] old_value=[59] new_value=[13] get_value=[ 4]
pid=[53973] old_value=[87] new_value=[14] get_value=[70]
share improve this answer follow edited Jan 3 '17 at 2:34 Rob Bednark 17.9k 15 15 gold badges 65 65 silver badges 95 95 bronze badges answered Oct 3 '13 at 5:41 Bohdan 12.8k 11 11 gold badges 66 66 silver badges 65 65 bronze badges

> ,

add a comment

gxyd , 2014-12-04 06:27:43

27

What you are saying is to use the method like this:

globvar = 5

def f():
    var = globvar
    print(var)

f()  # Prints 5

But the better way is to use the global variable like this:

globavar = 5
def f():
    global globvar
    print(globvar)
f()   #prints 5

Both give the same output. share improve this answer follow edited Apr 24 '18 at 6:25 answered Dec 4 '14 at 6:27 gxyd 533 5 5 silver badges 11 11 bronze badges

> ,

add a comment

user2876408 ,

27

As it turns out the answer is always simple.

Here is a small sample module with a simple way to show it in a main definition:

def five(enterAnumber,sumation):
    global helper
    helper  = enterAnumber + sumation

def isTheNumber():
    return helper

Here is how to show it in a main definition:

import TestPy

def main():
    atest  = TestPy
    atest.five(5,8)
    print(atest.isTheNumber())

if __name__ == '__main__':
    main()

This simple code works just like that, and it will execute. I hope it helps. share improve this answer follow edited Sep 28 '18 at 11:34 AEF 3,419 1 1 gold badge 10 10 silver badges 22 22 bronze badges answered Oct 13 '13 at 16:07 user2876408 271 3 3 silver badges 2 2 bronze badges

jtlz2 ,

thanks, i'm new to python, but know a bit of java. what you said worked for me. and writing global a<ENTER> within the class.. seems to make more sense to me than within a function writing 'global a'.. I notice you can't say global a=4 – barlop Oct 19 '13 at 18:55

Mohamed El-Saka , 2014-12-20 12:45:26

23

You need to reference the global variable in every function you want to use.

As follows:

var = "test"

def printGlobalText():
    global var #wWe are telling to explicitly use the global version
    var = "global from printGlobalText fun."
    print "var from printGlobalText: " + var

def printLocalText():
    #We are NOT telling to explicitly use the global version, so we are creating a local variable
    var = "local version from printLocalText fun"
    print "var from printLocalText: " + var

printGlobalText()
printLocalText()
"""
Output Result:
var from printGlobalText: global from printGlobalText fun.
var from printLocalText: local version from printLocalText
[Finished in 0.1s]
"""
share improve this answer follow edited Feb 4 '15 at 18:45 Peter Mortensen 26.5k 21 21 gold badges 92 92 silver badges 122 122 bronze badges answered Dec 20 '14 at 12:45 Mohamed El-Saka 522 4 4 silver badges 11 11 bronze badges

spazm ,

'in every function you want to use' is subtly incorrect, should be closer to: 'in every function where you want to update ' – spazm Mar 19 '15 at 23:43

Kylotan , 2009-01-09 11:56:19

22

You're not actually storing the global in a local variable, just creating a local reference to the same object that your original global reference refers to. Remember that pretty much everything in Python is a name referring to an object, and nothing gets copied in usual operation.

If you didn't have to explicitly specify when an identifier was to refer to a predefined global, then you'd presumably have to explicitly specify when an identifier is a new local variable instead (for example, with something like the 'var' command seen in JavaScript). Since local variables are more common than global variables in any serious and non-trivial system, Python's system makes more sense in most cases.

You could have a language which attempted to guess, using a global variable if it existed or creating a local variable if it didn't. However, that would be very error-prone. For example, importing another module could inadvertently introduce a global variable by that name, changing the behaviour of your program. share improve this answer follow edited May 30 '11 at 21:09 Peter Mortensen 26.5k 21 21 gold badges 92 92 silver badges 122 122 bronze badges answered Jan 9 '09 at 11:56 Kylotan 17.7k 7 7 gold badges 39 39 silver badges 73 73 bronze badges

> ,

add a comment

Sagar Mehta ,

18

Try this:

def x1():
    global x
    x = 6

def x2():
    global x
    x = x+1
    print x

x = 5
x1()
x2()  # output --> 7
share improve this answer follow edited Jun 20 '18 at 21:55 Vicrobot 3,180 1 1 gold badge 10 10 silver badges 27 27 bronze badges answered Feb 4 '15 at 19:19 Sagar Mehta 242 2 2 silver badges 5 5 bronze badges

> ,

add a comment

Martin Thoma , 2017-04-07 18:52:13

16

In case you have a local variable with the same name, you might want to use the globals() function .

globals()['your_global_var'] = 42
share improve this answer follow edited Apr 7 '17 at 19:15 answered Apr 7 '17 at 18:52 Martin Thoma 80.7k 101 101 gold badges 453 453 silver badges 699 699 bronze badges

> ,

add a comment

, 2015-10-24 15:46:18

14

Following on and as an add on, use a file to contain all global variables all declared locally and then import as :

File initval.py :

Stocksin = 300
Prices = []

File getstocks.py :

import initval as iv

def getmystocks(): 
    iv.Stocksin = getstockcount()


def getmycharts():
    for ic in range(iv.Stocksin):
share improve this answer follow edited May 21 '19 at 10:59 Georgy 4,728 5 5 gold badges 37 37 silver badges 46 46 bronze badges answered Oct 24 '15 at 15:46 user5473311

olibre ,

What is the advantage to move the global variables to another file? Is it just to group together the global variables in a tiny file? And why using the statement import ... as ... ? Why not just import ... ? – olibre May 23 '17 at 20:21

Mike Lampton , 2016-01-07 20:41:19

13

Writing to explicit elements of a global array does not apparently need the global declaration, though writing to it "wholesale" does have that requirement:

import numpy as np

hostValue = 3.14159
hostArray = np.array([2., 3.])
hostMatrix = np.array([[1.0, 0.0],[ 0.0, 1.0]])

def func1():
    global hostValue    # mandatory, else local.
    hostValue = 2.0

def func2():
    global hostValue    # mandatory, else UnboundLocalError.
    hostValue += 1.0

def func3():
    global hostArray    # mandatory, else local.
    hostArray = np.array([14., 15.])

def func4():            # no need for globals
    hostArray[0] = 123.4

def func5():            # no need for globals
    hostArray[1] += 1.0

def func6():            # no need for globals
    hostMatrix[1][1] = 12.

def func7():            # no need for globals
    hostMatrix[0][0] += 0.33

func1()
print "After func1(), hostValue = ", hostValue
func2()
print "After func2(), hostValue = ", hostValue
func3()
print "After func3(), hostArray = ", hostArray
func4()
print "After func4(), hostArray = ", hostArray
func5()
print "After func5(), hostArray = ", hostArray
func6()
print "After func6(), hostMatrix = \n", hostMatrix
func7()
print "After func7(), hostMatrix = \n", hostMatrix
share improve this answer follow edited Jan 8 '16 at 22:35 answered Jan 7 '16 at 20:41 Mike Lampton 151 1 1 silver badge 5 5 bronze badges

> ,

add a comment

llewellyn falco , 2017-08-19 08:48:27

7

Reference the class namespace where you want the change to show up.

In this example, runner is using max from the file config. I want my test to change the value of max when runner is using it.

main/config.py

max = 15000

main/runner.py

from main import config
def check_threads():
    return max < thread_count

tests/runner_test.py

from main import runner                # <----- 1. add file
from main.runner import check_threads
class RunnerTest(unittest):
   def test_threads(self):
       runner.max = 0                  # <----- 2. set global 
       check_threads()
share improve this answer follow answered Aug 19 '17 at 8:48 llewellyn falco 2,061 12 12 silver badges 12 12 bronze badges

> ,

add a comment

Rafaël Dera ,

7

I'm adding this as I haven't seen it in any of the other answers and it might be useful for someone struggling with something similar. The globals() function returns a mutable global symbol dictionary where you can "magically" make data available for the rest of your code. For example:

from pickle import load
def loaditem(name):
    with open(r"C:\pickle\file\location"+"\{}.dat".format(name), "rb") as openfile:
        globals()[name] = load(openfile)
    return True

and

from pickle import dump
def dumpfile(name):
    with open(name+".dat", "wb") as outfile:
        dump(globals()[name], outfile)
    return True

Will just let you dump/load variables out of and into the global namespace. Super convenient, no muss, no fuss. Pretty sure it's Python 3 only. share improve this answer follow edited May 21 '19 at 12:08 Georgy 4,728 5 5 gold badges 37 37 silver badges 46 46 bronze badges answered Sep 5 '17 at 14:59 Rafaël Dera 377 2 2 silver badges 7 7 bronze badges

Kiran Jonnalagadda ,

globals() always returns globals available in the local context, so a mutation here may not reflect in another module. – Kiran Jonnalagadda May 8 '19 at 4:53

> ,

3 Globals are fine - Except with Multiprocessing

Globals in connection with multiprocessing on different platforms/envrionments as Windows/Mac OS on the one side and Linux on the other are troublesome.

I will show you this with a simple example pointing out a problem which I run into some time ago.

If you want to understand, why things are different on Windows/MacOs and Linux you need to know that, the default mechanism to start a new process on ...

They are different in Memory allocation an initialisation ... (but I don't go into this here).

Let's have a look at the problem/example ...

import multiprocessing

counter = 0

def do(task_id):
    global counter
    counter +=1
    print(f'task {task_id}: counter = {counter}')

if __name__ == '__main__':

    pool = multiprocessing.Pool(processes=4)
    task_ids = list(range(4))
    pool.map(do, task_ids)
Windows

If you run this on Windows (And I suppose on MacOS too), you get the following output ...

task 0: counter = 1
task 1: counter = 2
task 2: counter = 3
task 3: counter = 4
Linux

If you run this on Linux, you get the following instead.

task 0: counter = 1
task 1: counter = 1
task 2: counter = 1
task 3: counter = 1
share improve this answer follow

[Sep 19, 2020] 7. Simple statements -- Python 3.8.6rc1 documentation

Sep 19, 2020 | docs.python.org

7.12. The global statement

global_stmt ::=  "global" identifier ("," identifier)*

The global statement is a declaration which holds for the entire current code block. It means that the listed identifiers are to be interpreted as globals. It would be impossible to assign to a global variable without global , although free variables may refer to globals without being declared global.

Names listed in a global statement must not be used in the same code block textually preceding that global statement.

Names listed in a global statement must not be defined as formal parameters or in a for loop control target, class definition, function definition, import statement, or variable annotation.

CPython implementation detail: The current implementation does not enforce some of these restrictions, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program.

Programmer's note: global is a directive to the parser. It applies only to code parsed at the same time as the global statement. In particular, a global statement contained in a string or code object supplied to the built-in exec() function does not affect the code block containing the function call, and code contained in such a string is unaffected by global statements in the code containing the function call. The same applies to the eval() and compile() functions. 7.13. The nonlocal statement

nonlocal_stmt ::=  "nonlocal" identifier ("," identifier)*

The nonlocal statement causes the listed identifiers to refer to previously bound variables in the nearest enclosing scope excluding globals. This is important because the default behavior for binding is to search the local namespace first. The statement allows encapsulated code to rebind variables outside of the local scope besides the global (module) scope.

Names listed in a nonlocal statement, unlike those listed in a global statement, must refer to pre-existing bindings in an enclosing scope (the scope in which a new binding should be created cannot be determined unambiguously).

Names listed in a nonlocal statement must not collide with pre-existing bindings in the local scope.

See also

PEP 3104 - Access to Names in Outer Scopes

The specification for the nonlocal statement.

[Sep 18, 2020] Global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill.

Notable quotes:
"... Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill. ..."
Nov 22, 2019 | stackoverflow.com

Peter Mortensen, Mar 4 '17 at 22:00

If you want to refer to a global variable in a function, you can use the global keyword to declare which variables are global. You don't have to use it in all cases (as someone here incorrectly claims) - if the name referenced in an expression cannot be found in local scope or scopes in the functions in which this function is defined, it is looked up among global variables.

However, if you assign to a new variable not declared as global in the function, it is implicitly declared as local, and it can overshadow any existing global variable with the same name.

Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill.

J S, Jan 8 '09

Absolutely re. zealots. Most Python users use it for scripting and create little functions to separate out small bits of code. – Paul Uszak Sep 22 at 22:57

[Sep 18, 2020] How Python saves memory when storing strings

Sep 18, 2020 | rushter.com

Last updated on August 10, 2018, in python

Since Python 3, the str type uses Unicode representation. Unicode strings can take up to 4 bytes per character depending on the encoding, which sometimes can be expensive from a memory perspective.

To reduce memory consumption and improve performance, Python uses three kinds of internal representations for Unicode strings:

When programming in Python all strings behave the same, and most of the time we don't notice any difference. However, the difference can be very remarkable and sometimes unexpected when working with large amounts of text.

To see the difference in internal representations, we can use the sys.getsizeof function, which returns the size of an object in bytes:

>>> import sys
>>> string = 'hello'
>>> sys.getsizeof(string)
54
>>> # 1-byte encoding
>>> sys.getsizeof(string+'!')-sys.getsizeof(string)
1
>>> # 2-byte encoding
>>> string2  = '你'
>>> sys.getsizeof(string2+'好')-sys.getsizeof(string2)
2
>>> sys.getsizeof(string2)
76
>>> # 4-byte encoding
>>> string3 = '🐍'
>>> sys.getsizeof(string3+'💻')-sys.getsizeof(string3)
4
>>> sys.getsizeof(string3)
80

As you can see, depending on the content of a string, Python uses different encodings. Note that every string in Python takes additional 49-80 bytes of memory, where it stores supplementary information, such as hash, length, length in bytes, encoding type and string flags. That's why an empty string takes 49 bytes of memory.

We can retrieve encoding directly from an object using ctypes :

import ctypes

class PyUnicodeObject(ctypes.Structure):
    # internal fields of the string object
    _fields_ = [("ob_refcnt", ctypes.c_long),
                ("ob_type", ctypes.c_void_p),
                ("length", ctypes.c_ssize_t),
                ("hash", ctypes.c_ssize_t),
                ("interned", ctypes.c_uint, 2),
                ("kind", ctypes.c_uint, 3),
                ("compact", ctypes.c_uint, 1),
                ("ascii", ctypes.c_uint, 1),
                ("ready", ctypes.c_uint, 1),
                # ...
                # ...
                ]


def get_string_kind(string):
    return PyUnicodeObject.from_address(id(string)).kind
>>> get_string_kind('Hello')
1
>>> get_string_kind('你好')
2
>>> get_string_kind('🐍')
4

If all characters in a string can fit in ASCII range, then they are encoded using 1-byte Latin-1 encoding. Basically, Latin-1 represents the first 256 Unicode characters. It supports many Latin languages, such as English, Swedish, Italian, Norwegian and so on. However, it cannot store non-Latin languages, such as Chinese, Japanese, Hebrew, Cyrillic. That is because their codepoints (numerical indexes) defined outside of 1-byte (0-255) range.

>>> ord('a')
97
>>> ord('你')
20320
>>> ord('!')
33

Most of the popular natural languages can fit in 2-byte (UCS-2) encoding. The 4-byte (UCS-4) encoding is used when a string contains special symbols, emojis or rare languages. There are almost 300 blocks (ranges) in the Unicode standard. You can find the 4-byte blocks after the 0xFFFF block.

Let's suppose we have a 10GB of ASCII text and we want to load it in memory. If you insert a single emoji in our text the size of a string will increase by the factor of 4! This is a huge difference that you may encounter in practice when working with NLP problems.

Why Python doesn't use UTF-8 encoding internally

The most well-known and popular Unicode encoding is UTF-8, but Python doesn't use it internally.

When a string is stored in the UTF-8 encoding, each character is encoded using 1-4 bytes depending on the character it is representing. It's a storage efficient encoding, but it has one significant disadvantage. Since each character can vary in length of bytes, there is no way to randomly access an individual character by index without scanning the string. So, to perform a simple operation such as string[5] with UTF-8 Python would need to scan a string until it finds a required character. Fixed length encodings don't have such problem, to locate a character by index Python just multiplies an index number by the length of one character (1, 2 or 4 bytes).

String interning

When working with empty strings or ASCII strings of one character Python uses string interning . Interned strings act as singletons, that is, if you have two identical strings that are interned, there is only one copy of them in the memory.

>>> a = 'hello'
>>> b = 'world'
>>> a[4],b[1]
('o', 'o')
>>> id(a[4]), id(b[1]), a[4] is b[1]
(4567926352, 4567926352, True)
>>> id('')
4545673904
>>> id('')
4545673904

As you can see, both string slices point to the same address in the memory. It's possible because Python strings are immutable.

In Python, string interning is not limed to characters or empty strings. Strings that are created during code compilation can also be interned if their length does not exceed 20 characters.

This includes:

When you hit enter in Python REPL, your statement gets compiled down to the bytecode. That's why all short strings in REPL are also interned.

>>> a = 'teststring'
>>> b = 'teststring'
>>> id(a), id(b), a is b
(4569487216, 4569487216, True)
>>> a = 'test'*5
>>> b = 'test'*5
>>> len(a), id(a), id(b), a is b
(20, 4569499232, 4569499232, True)
>>> a = 'test'*6
>>> b = 'test'*6
>>> len(a), id(a), id(b), a is b
(24, 4569479328, 4569479168, False)

This example will not work, because such strings are not constants:

>>> open('test.txt','w').write('hello')
5
>>> open('test.txt','r').read()
'hello'
>>> a = open('test.txt','r').read()
>>> b = open('test.txt','r').read()
>>> id(a), id(b), a is b
(4384934576, 4384934688, False)
>>> len(a), id(a), id(b), a is b
(5, 4384934576, 4384934688, False)

String interning technique saves tens of thousands of duplicate string allocations. Internally, string interning is maintained by a global dictionary where strings are used as keys. To check if there is already an identical string in the memory Python performs dictionary membership operation.

The unicode object is almost 16 000 lines of C code, so there are a lot of small optimizations which are not mentioned in this article. If you want to learn more about Unicode in Python, I would recommend you to read PEPs about strings and check the code of the unicode object.

[Sep 04, 2020] Python Scope the LEGB Rule- Resolving Names in Your Code by Leodanis Pozo Ramos

Mar 18, 2020 | realpython.com

Now that you know how to initialize a variable. Let's talk about the scope of these variables. Not all variables can be accessed from anywhere in a program. The part of a program where a variable is accessible is called its scope. There are four major types of variable scope and is the basis for the LEGB rule . LEGB stands for Local -> Enclosing -> Global -> Built-in .

Let's learn more about scopes...

Local Scope

Whenever you define a variable within a function, its scope lies ONLY within the function. It is accessible from the point at which it is defined until the end of the function and exists for as long as the function is executing ( Source ). Which means its value cannot be changed or even accessed from outside the function. Let's take a simple example:

def print_number():
    first_num = 1
    # Print statement 1
    print("The first number defined is: ", first_num)

print_number()
# Print statement 2
print("The first number defined is: ", first_num)
The first number defined is:  1



---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

<ipython-input-10-17d580e01222> in <module>
      6 print_number()
      7 # Print statement 2
----> 8 print("The first number defined is: ", first_num)


NameError: name 'first_num' is not defined

We were able to print the first_num variable by calling the function print_number() (# Print statement 1). But when trying to access and then print the same variable from outside the function (# Print statement 2), it raised a NameError . This is because first_num is "local" to the function - thus, it cannot be reached from outside the function body.

Enclosing Scope

What if we have a nested function (function defined inside another function)? How does the scope change? Let's see with the help of an example.

def outer():
    first_num = 1
    def inner():
        second_num = 2
        # Print statement 1 - Scope: Inner
        print("first_num from outer: ", first_num)
        # Print statement 2 - Scope: Inner
        print("second_num from inner: ", second_num)
    inner()
    # Print statement 3 - Scope: Outer
    print("second_num from inner: ", second_num)

outer()
first_num from outer:  1
second_num from inner:  2



---------------------------------------------------------------------------

NameError                                 Traceback (most recent call last)

<ipython-input-4-13943a1eb01e> in <module>
     11     print("second_num from inner: ", second_num)
     12
---> 13 outer()


<ipython-input-4-13943a1eb01e> in outer()
      9     inner()
     10     # Print statement 3 - Scope: Outer
---> 11     print("second_num from inner: ", second_num)
     12
     13 outer()


NameError: name 'second_num' is not defined

Got an error? This is because you cannot access second_num from outer() (# Print statement 3). It is not defined within that function. However, you can access first_num from inner() (# Print statement 1), because the scope of first_num is larger, it is within outer() .
This is an enclosing scope . Outer 's variables have a larger scope and can be accessed from the enclosed function inner() .

Global Scope

This is perhaps the easiest scope to understand. Whenever a variable is defined outside any function, it becomes a global variable, and its scope is anywhere within the program. Which means it can be used by any function.

greeting = "Hello"

def greeting_world():
    world = "World"
    print(greeting, world)

def greeting_name(name):
    print(greeting, name)

greeting_world()
greeting_name("Samuel")
Hello World
Hello Samuel
Built-in Scope

This is the widest scope that exists! All the special reserved keywords fall under this scope. We can call the keywords anywhere within our program without having to define them before use.

Keywords are simply special reserved words. They are kept for specific purposes and cannot be used for any other purpose in the program.
These are the keywords in Python:

LEGB Rule

LEGB (Local -> Enclosing -> Global -> Built-in) is the logic followed by a Python interpreter when it is executing your program.

Let's say you're calling print(x) within inner() , which is a function nested in outer() . Then Python will first look if "x" was defined locally within inner() . If not, the variable defined in outer() will be used. This is the enclosing function. If it also wasn't defined there, the Python interpreter will go up another level - to the global scope. Above that, you will only find the built-in scope, which contains special variables reserved for Python itself.

So far, so good!

Next, let's revisit some examples from before to see if they can create problems when the use-case becomes slightly more complex.

Scenario 1: Global Scope

Remember the greeting_world() function from earlier? Let's say you wanted to be able to change the global variable greeting ("Hello") to set a new value ("Hi") for the greeting, so that greeting_world() prints "Hi World"

greeting = "Hello"

def change_greeting(new_greeting):
    greeting = new_greeting

def greeting_world():
    world = "World"
    print(greeting, world)

change_greeting("Hi")
greeting_world()
Hello World

Well...that did not go as wanted! Why?

This is because when we set the value of greeting to "Hi", it created a new local variable greeting in the scope of change_greeting() . It did not change anything for the global greeting . This is where the global keyword comes in handy.

Global Keyword

With global , you're telling Python to use the globally defined variable instead of locally creating one. To use the keyword, simply type 'global', followed by the variable name. Let's see this in action on Scenario 1.

greeting = "Hello"

def change_greeting(new_greeting):
    global greeting
    greeting = new_greeting

def greeting_world():
    world = "World"
    print(greeting, world)

change_greeting("Hi")
greeting_world()
Hi World

That worked! Let's move to the next scenario...

Scenario 2: Enclosing Scope

Here, we have a look at the outer() and inner() nested functions from the Enclosing Scope example. Let's try to change the value of first_num from 1 to 0 from within inner() .

def outer():
    first_num = 1
    def inner():
        first_num = 0
        second_num = 1
        print("inner - second_num is: ", second_num)
    inner()
    print("outer - first_num is: ", first_num)

outer()
inner - second_num is:  1
outer - first_num is:  1

It isn't always that simple, is it! For such purposes, we make use of nonlocal keyword within Python.

Nonlocal Keyword

This is another handy keyword that allows us to work more flexibly and tidily with variable scopes. The nonlocal keyword is useful in nested functions. It causes the variable to refer to the previously bound variable in the closest enclosing scope. In other words, it will prevent the variable from trying to bind locally first, and force it to go a level 'higher up'. The syntax is similar to the global keyword.

def outer():
    first_num = 1
    def inner():
        nonlocal first_num
        first_num = 0
        second_num = 1
        print("inner - second_num is: ", second_num)
    inner()
    print("outer - first_num is: ", first_num)

outer()
inner - second_num is:  1
outer - first_num is:  0
Conclusion

You now know what Python's scope of variables is, the LEGB rule, and how you should use the global and nonlocal keywords. You'll be able to easily manipulate variables in nested functions, without any problem. To learn more about programming in Python, you should definitely take a look at DataCamp's Intro to Python for Data Science course . It is an interactive course that covers all the basics: from variables and calculations to lists, functions, and packages.

[Sep 03, 2020] How to Format Strings in Python by John Paul Mueller

Sep 03, 2020 | www.dummies.com

RELATED BOOK

Beginning Programming with Python For Dummies

You can format strings in a number of ways using Python. The main emphasis of formatting is to present the string in a form that is both pleasing to the user and easy to understand. Formatting doesn't mean adding effects in this case, but refers merely to the presentation of the data. For example, the user might want a fixed-point number rather than a decimal number as output.

You have quite a few ways to format strings. However, the focus of most formatting is the format() function. You create a formatting specification as part of the string and then use the format() function to add data to that string. A format specification may be as simple as two curly brackets {} that specify a placeholder for data.

You can number the placeholder to create special effects. For example, {0} would contain the first data element in a string. When the data elements are numbered, you can even repeat them so that the same data appears more than once in the string.

The formatting specification follows a colon. When you want to create just a formatting specification, the curly brackets contain just the colon and whatever formatting you want to use.

For example, {:f} would create a fixed-point number as output. If you want to number the entries, the number that precedes the colon: {0:f} creates a fixed-point number output for data element one. The formatting specification follows this form, with the italicized elements serving as placeholders here:

[[fill]align][sign][#][0][width][,][.precision][type]

This specification provides you with the in-depth details, but here's an overview of what the various entries mean:

The formatting specification elements must appear in the correct order or Python won't know what to do with them. If you specify the alignment before the fill character, Python displays an error message rather than performing the required formatting. The following steps help you see how the formatting specification works and demonstrate the order you need to follow in using the various formatting specification criteria.

  1. Open a Python File window.

    You see an editor in which you can type the example code.

  2. Type the following code into the window -- pressing Enter after each line:

    Formatted = "{:d}"
    print(Formatted.format(7000))
    Formatted = "{:,d}"
    print(Formatted.format(7000))
    Formatted = "{:^15,d}"
    print(Formatted.format(7000))
    Formatted = "{:*^15,d}"
    print(Formatted.format(7000))
    Formatted = "{:*^15.2f}"
    print(Formatted.format(7000))
    Formatted = "{:*>15X}"
    print(Formatted.format(7000))
    Formatted = "{:*<#15x}"
    print(Formatted.format(7000))
    Formatted = "A {0} {1} and a {0} {2}."
    print(Formatted.format("blue", "car", "truck"))
    

    The example starts simply with a field formatted as a decimal value. It then adds a thousands separator to the output. The next step is to make the field wider than needed to hold the data and to center the data within the field. Finally, the field has an asterisk added to pad the output.

    Of course, there are other data types in the example. The next step is to display the same data in fixed-point format. The example also shows the output in both uppercase and lowercase hexadecimal format. The uppercase output is right aligned and the lowercase output is left aligned.

    Finally, the example shows how you can use numbered fields to your advantage. In this case, it creates an interesting string output that repeats one of the input values.

[Sep 02, 2020] 5 Advanced String Methods in Python

Sep 02, 2020 | towardsdatascience.com

Format beautiful strings using f-string notation in python3.8 Yaakov Bressler Yaakov Bressler Follow Aug 24 · 2 min read Introduction to F-String Notation

Formatting strings are a basic yet essential part of programming. F-Strings (introduced in Python3.6) improve the simplicity and intuitiveness of formatting rich strings.

F-String notation (formatted strings) require an f at the start of a string (ex. f"hello" ). There are many advanced methods available for formatting beautiful strings, 5 such methods are discussed below.

1. Variables in String

Insert/replace a variable in place.

name = "Alexander Hamilton"
print(f"hello {name}")
# hello Alexander Hamilton
2. Variable Formatting in String

Format a variable such as an integer in place.

2.1 Print a large integer with commas:
x = 2*10**5
print(f"You have ${x:,} in your bank account")
# You have $200,000 in your bank account
2.2 Print a float with a specified number of digits:
pi = 3.1415926535897932384 
print(f"The first 5 digits of pi are {pi:.5f}")
# The first 5 digits of pi are 3.14159
3. In Place Boolean Logic:

Perhaps you want to format a string based on several conditions. F-string notation allows for in place boolean arguments, just as you would expect from lambdas.

new_user = False
user_name = "Alexander Hamilton"
print(f"{'Congrats on making your account' if new_user else 'Welcome back'} {user_name}!")
# Welcome back Alexander Hamilton!
4. Printing Variable Names & Values

As of python 3.8, f-string notation allows printing of variable names with their values -- an especially useful tool for debugging:

max_price = 20000
min_price = 4000print(f"{max_price=:,}  |  {min_price=:,})
# max_price=20,000  |  min_price=4,000
5. Formatting Variables in Pre-populated Strings (MailMerge)

String formatting allows replacement of variables in pre-populated strings.

This is especially useful for allowing external users to format an email message which the program will fill in.

5.1 Str.Format Method (Preferred)
my_message = "Welcome to our platform {FNAME}! Your favorite ice cream flavor is {FLAVOR}."FNAME = "Alexander"
FLAVOR = "Mint Chocolate Chip"my_message_formatted = my_message.format(**{"FNAME":FNAME, "FLAVOR":FLAVOR})print(my_message_formatted)
# Welcome to our platform Alexander! Your favorite ice cream flavor is Mint Chocolate Chip.
5.2 F-String Method (with Eval)

Here's one instance where f string notation falls short (simply because it's intended to be used in code )

my_message = "Welcome to our platform {FNAME}! Your favorite ice cream flavor is {FLAVOR}."FNAME = "Alexander"
FLAVOR = "Mint Chocolate Chip"my_message_formatted = eval(f"f'{my_message}'")print(my_message_formatted)
# Welcome to our platform Alexander! Your favorite ice cream flavor is Mint Chocolate Chip.

Conclusion

F-string notation is a simple way to format beautiful strings. Demonstrated above are 5 advanced methods for formatting strings in python3.8.

Consider implementing these methods into your python code.

If you have additional noteworthy string methods, please share them in the comments!

[Sep 01, 2020] String Slicing in Python - GeeksforGeeks

Sep 01, 2020 | www.geeksforgeeks.org

String Slicing in Python Last Updated: 31-12-2019

Python slicing is about obtaining a sub-string from the given string by slicing it respectively from start to end.
Python slicing can be done in two ways.

slice() Constructor

The slice() constructor creates a slice object representing the set of indices specified by range(start, stop, step).

Syntax:

Parameters:
start: Starting index where the slicing of object starts.
stop: Ending index where the slicing of object stops.
step: It is an optional argument that determines the increment between each index for slicing.

Return Type: Returns a sliced object containing elements in the given range only.

Index tracker for positive and negative index:
Negative comes into considers when tracking the string in reverse.

python-string-slice

Example

filter_none

edit

play_arrow

brightness_4

# Python program to demonstrate # string slicing # String slicing String = 'ASTRING' # Using slice constructor s1 = slice ( 3 ) s2 = slice ( 1 , 5 , 2 ) s3 = slice ( - 1 , - 12 , - 2 ) print ( "String slicing" ) print (String[s1]) print (String[s2]) print (String[s3])
Output:
String slicing
AST
SR
GITA
Extending indexing

In Python, indexing syntax can be used as a substitute for the slice object. This is an easy and convenient way to slice a string both syntax wise and execution wise.

Syntax

string[start:end:step]

start, end and step have the same mechanism as slice() constructor.

Example

filter_none

edit

play_arrow

brightness_4

# Python program to demonstrate # string slicing # String slicing String = 'ASTRING' # Using indexing sequence print (String[: 3 ]) print (String[ 1 : 5 : 2 ]) print (String[ - 1 : - 12 : - 2 ]) # Prints string in reverse print ( "\nReverse String" ) print (String[:: - 1 ])
Output:
AST
SR
GITA

Reverse String
GNIRTSA

Note: To know more about strings click here .

[Aug 27, 2020] The chapter on debugging from Think Python book

Aug 27, 2020 | www.reddit.com

4 days ago

I'd also highly recommend this chapter on debugging from Think Python book: https://greenteapress.com/thinkpython2/html/thinkpython2021.html

and this article: https://jvns.ca/blog/2019/06/23/a-few-debugging-resources/

TIMEBACK

Ends 08/31/2020. Save an extra $10 on all departments, including groceries, personal care, health & nutrition, home essentials, beauty, kitchenware, and electronics by using this Walmart coupon. Minimum spend of $50 required.

$10 Coupon - Walmart Promo Codes 2020 - The Wall Street Journal

[Aug 19, 2020] python - subprocess.check_output return code

Jan 01, 2014 | stackoverflow.com

Ask Question Asked 6 years, 3 months ago Active 7 months ago Viewed 77k times


Juicy ,

I am using:

grepOut = subprocess.check_output("grep " + search + " tmp", shell=True)

To run a terminal command, I know that I can use a try/except to catch the error but how can I get the value of the error code?

I found this on the official documentation:

 exception subprocess.CalledProcessError

    Exception raised when a process run by check_call() or check_output() returns a non-zero exit status.

    returncode

        Exit status of the child process.

But there are no examples given and Google was of no help.

jfs ,

"Google was of no help" : the first link (almost there it shows e.output ), the second link is the exact match (it shows e.returncode ) the search term: CalledProcessError . – jfs May 2 '14 at 15:06

DanGar , 2014-05-02 05:07:05

You can get the error code and results from the exception that is raised.

This can be done through the fields returncode and output .

For example:

import subprocess

try:
   grepOut = subprocess.check_output("grep " + "test" + " tmp", shell=True)                       
except subprocess.CalledProcessError as grepexc:                                                                                                   
    print "error code", grepexc.returncode, grepexc.output

DanGar ,

Thank you exactly what I wanted. But now I am wondering, is there a way to get a return code without a try/except? IE just get the return code of the check_output, whether it is 0 or 1 or other is not important to me and I don't actually need to save the output. – Juicy May 2 '14 at 5:12

jfs , 2014-05-02 16:09:20

is there a way to get a return code without a try/except?

check_output raises an exception if it receives non-zero exit status because it frequently means that a command failed. grep may return non-zero exit status even if there is no error -- you could use .communicate() in this case:

from subprocess import Popen, PIPE

pattern, filename = 'test', 'tmp'
p = Popen(['grep', pattern, filename], stdin=PIPE, stdout=PIPE, stderr=PIPE,
          bufsize=-1)
output, error = p.communicate()
if p.returncode == 0:
   print('%r is found in %s: %r' % (pattern, filename, output))
elif p.returncode == 1:
   print('%r is NOT found in %s: %r' % (pattern, filename, output))
else:
   assert p.returncode > 1
   print('error occurred: %r' % (error,))

You don't need to call an external command to filter lines, you could do it in pure Python:

with open('tmp') as file:
    for line in file:
        if 'test' in line:
            print line,

If you don't need the output; you could use subprocess.call() :

import os
from subprocess import call
try:
    from subprocess import DEVNULL # Python 3
except ImportError: # Python 2
    DEVNULL = open(os.devnull, 'r+b', 0)

returncode = call(['grep', 'test', 'tmp'], 
                  stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)

mkobit , 2017-09-15 14:52:56

Python 3.5 introduced the subprocess.run() method. The signature looks like:

subprocess.run(
  args, 
  *, 
  stdin=None, 
  input=None, 
  stdout=None, 
  stderr=None, 
  shell=False, 
  timeout=None, 
  check=False
)

The returned result is a subprocess.CompletedProcess . In 3.5, you can access the args , returncode , stdout , and stderr from the executed process.

Example:

>>> result = subprocess.run(['ls', '/tmp'], stdout=subprocess.DEVNULL)
>>> result.returncode
0

>>> result = subprocess.run(['ls', '/nonexistent'], stderr=subprocess.DEVNULL)
>>> result.returncode
2

Dean Kayton ,

I reckon this is the most up-to-date approach. The syntax is much more simple and intuitive and was probably added for just that reason. – Dean Kayton Jul 22 '19 at 11:46

Noam Manos ,

In Python 2 - use commands module:

import command
rc, out = commands.getstatusoutput("ls missing-file")
if rc != 0: print "Error occurred: %s" % out

In Python 3 - use subprocess module:

import subprocess
rc, out = subprocess.getstatusoutput("ls missing-file")
if rc != 0: print ("Error occurred:", out)

Error occurred: ls: cannot access missing-file: No such file or directory

[Aug 19, 2020] python - subprocess.check_output return code - Stack Overflow

Jan 01, 2014 | stackoverflow.com

subprocess.check_output return code Ask Question Asked 6 years, 3 months ago Active 7 months ago Viewed 77k times


Juicy ,

I am using:

grepOut = subprocess.check_output("grep " + search + " tmp", shell=True)

To run a terminal command, I know that I can use a try/except to catch the error but how can I get the value of the error code?

I found this on the official documentation:

 exception subprocess.CalledProcessError

    Exception raised when a process run by check_call() or check_output() returns a non-zero exit status.

    returncode

        Exit status of the child process.

But there are no examples given and Google was of no help.

jfs ,

"Google was of no help" : the first link (almost there it shows e.output ), the second link is the exact match (it shows e.returncode ) the search term: CalledProcessError . – jfs May 2 '14 at 15:06

DanGar , 2014-05-02 05:07:05

You can get the error code and results from the exception that is raised.

This can be done through the fields returncode and output .

For example:

import subprocess

try:
   grepOut = subprocess.check_output("grep " + "test" + " tmp", shell=True)                       
except subprocess.CalledProcessError as grepexc:                                                                                                   
    print "error code", grepexc.returncode, grepexc.output

DanGar ,

Thank you exactly what I wanted. But now I am wondering, is there a way to get a return code without a try/except? IE just get the return code of the check_output, whether it is 0 or 1 or other is not important to me and I don't actually need to save the output. – Juicy May 2 '14 at 5:12

jfs , 2014-05-02 16:09:20

is there a way to get a return code without a try/except?

check_output raises an exception if it receives non-zero exit status because it frequently means that a command failed. grep may return non-zero exit status even if there is no error -- you could use .communicate() in this case:

from subprocess import Popen, PIPE

pattern, filename = 'test', 'tmp'
p = Popen(['grep', pattern, filename], stdin=PIPE, stdout=PIPE, stderr=PIPE,
          bufsize=-1)
output, error = p.communicate()
if p.returncode == 0:
   print('%r is found in %s: %r' % (pattern, filename, output))
elif p.returncode == 1:
   print('%r is NOT found in %s: %r' % (pattern, filename, output))
else:
   assert p.returncode > 1
   print('error occurred: %r' % (error,))

You don't need to call an external command to filter lines, you could do it in pure Python:

with open('tmp') as file:
    for line in file:
        if 'test' in line:
            print line,

If you don't need the output; you could use subprocess.call() :

import os
from subprocess import call
try:
    from subprocess import DEVNULL # Python 3
except ImportError: # Python 2
    DEVNULL = open(os.devnull, 'r+b', 0)

returncode = call(['grep', 'test', 'tmp'], 
                  stdin=DEVNULL, stdout=DEVNULL, stderr=DEVNULL)

> ,

add a comment

mkobit , 2017-09-15 14:52:56

Python 3.5 introduced the subprocess.run() method. The signature looks like:

subprocess.run(
  args, 
  *, 
  stdin=None, 
  input=None, 
  stdout=None, 
  stderr=None, 
  shell=False, 
  timeout=None, 
  check=False
)

The returned result is a subprocess.CompletedProcess . In 3.5, you can access the args , returncode , stdout , and stderr from the executed process.

Example:

>>> result = subprocess.run(['ls', '/tmp'], stdout=subprocess.DEVNULL)
>>> result.returncode
0

>>> result = subprocess.run(['ls', '/nonexistent'], stderr=subprocess.DEVNULL)
>>> result.returncode
2

Dean Kayton ,

I reckon this is the most up-to-date approach. The syntax is much more simple and intuitive and was probably added for just that reason. – Dean Kayton Jul 22 '19 at 11:46

simfinite , 2019-07-01 14:36:06

To get both output and return code (without try/except) simply use subprocess.getstatusoutput (Python 3 required)

electrovir ,

please read stackoverflow.com/help/how-to-answer on how to write a good answer. – DjSh Jul 1 '19 at 14:45

> ,

In Python 2 - use commands module:

import command
rc, out = commands.getstatusoutput("ls missing-file")
if rc != 0: print "Error occurred: %s" % out

In Python 3 - use subprocess module:

import subprocess
rc, out = subprocess.getstatusoutput("ls missing-file")
if rc != 0: print ("Error occurred:", out)

Error occurred: ls: cannot access missing-file: No such file or directory

[Aug 19, 2020] How to get both return code and output from subprocess in Python

Aug 19, 2020 | stackoverflow.com

n This question already has answers here : subprocess.check_output return code (5 answers) Closed 5 years ago .

While developing python wrapper library for Android Debug Bridge (ADB), I'm using subprocess to execute adb commands in shell. Here is the simplified example:

import subprocess

...

def exec_adb_command(adb_command):
    return = subprocess.call(adb_command)

If command executed propery exec_adb_command returns 0 which is OK.

But some adb commands return not only "0" or "1" but also generate some output which I want to catch also. adb devices for example:

D:\git\adb-lib\test>adb devices
List of devices attached
07eeb4bb        device

I've already tried subprocess.check_output() for that purpose, and it does return output but not the return code ("0" or "1").

Ideally I would want to get a tuple where t[0] is return code and t[1] is actual output.

Am I missing something in subprocess module which already allows to get such kind of results?

Thanks! python subprocess adb share improve this question follow asked Jun 19 '15 at 12:10 Viktor Malyi 1,761 2 2 gold badges 18 18 silver badges 34 34 bronze badges


> ,

add a comment 1 Answer Active Oldest Votes

Padraic Cunningham ,

Popen and communicate will allow you to get the output and the return code.

from subprocess import Popen,PIPE,STDOUT

out = Popen(["adb", "devices"],stderr=STDOUT,stdout=PIPE)

t = out.communicate()[0],out.returncode
print(t)
('List of devices attached \n\n', 0)

check_output may also be suitable, a non-zero exit status will raise a CalledProcessError:

from subprocess import check_output, CalledProcessError

try:
    out = check_output(["adb", "devices"])
    t = 0, out
except CalledProcessError as e:
    t = e.returncode, e.message

You also need to redirect stderr to store the error output:

from subprocess import check_output, CalledProcessError

from tempfile import TemporaryFile

def get_out(*args):
    with TemporaryFile() as t:
        try:
            out = check_output(args, stderr=t)
            return  0, out
        except CalledProcessError as e:
            t.seek(0)
            return e.returncode, t.read()

Just pass your commands:

In [5]: get_out("adb","devices")
Out[5]: (0, 'List of devices attached \n\n')

In [6]: get_out("adb","devices","foo")
Out[6]: (1, 'Usage: adb devices [-l]\n')

Noam Manos ,

Thank you for the broad answer! – Viktor Malyi Jun 19 '15 at 12:48

[Aug 18, 2020] In Python, how can I access the namespace of the main module from an imported module

Jan 01, 2010 | stackoverflow.com

Damon ,

Specifically, I need to get at some objects and globals from the main module in an imported module. I know how to find those things when the parent module wants some particular thing from a child module, but I can't figure out how to go in the other direction.

kenorb ,

Well, okay, I assume that must be the case since I can't find any answers about this anywhere. The bottom line is I'm not concerned about reusing any of this code, I'm just trying to break this project apart into multiple files so it's more manageable to find what I'm looking for. I don't really know what I'm doing, honestly, so this has been a learning experience, but really I just want it to be easier to find the code I want to edit when I want to edit it. – Damon Sep 6 '10 at 2:15

Ignacio Vazquez-Abrams , 2010-09-06 00:17:37

import __main__

But don't do this.

Charlie Parker ,

@Alcott: Because it can very easily turn your program into a pretzel. – Ignacio Vazquez-Abrams Jul 4 '14 at 5:06

tzot ,

The answer you're looking for is:

import __main__

main_global1= __main__.global1

However, whenever a module module1 needs stuff from the __main__ module, then:

> ,

add a comment

Pieter , 2010-09-06 07:51:55

I think this would work:

import sys    
main_mod = sys.modules['__main__']

Ignacio Vazquez-Abrams ,

This is equivalent to import __main__ as main_mod . – Ignacio Vazquez-Abrams Sep 6 '10 at 8:51

> ,

Not sure if it is a good practice but maybe you could pass the objects and variables you need as parameters to the methods or classes you call in the imported module.

[Aug 17, 2020] Calling an external command from Python

Jan 01, 2008 | stackoverflow.com

Ask Question Asked 11 years, 11 months ago Active 3 months ago Viewed 3.5m times


> ,

How do you call an external command (as if I'd typed it at the Unix shell or Windows command prompt) from within a Python script?

freshWoWer , 2008-09-18 01:35:30

edited Nov 27 '19 at 21:26 Chris 80.2k 21 21 gold badges 175 175 silver badges 162 162 bronze badges asked Sep 18 '08 at 1:35 freshWoWer 51.3k 10 10 gold badges 31 31 silver badges 33 33 bronze badges

> ,

add a comment 61 Answers Active Oldest Votes 1 2 3 Next

> ,

Look at the subprocess module in the standard library:
import subprocess
subprocess.run(["ls", "-l"])

The advantage of subprocess vs. system is that it is more flexible (you can get the stdout , stderr , the "real" status code, better error handling, etc...).

The official documentation recommends the subprocess module over the alternative os.system() :

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function [ os.system() ].

The Replacing Older Functions with the subprocess Module section in the subprocess documentation may have some helpful recipes.

For versions of Python before 3.5, use call :

import subprocess
subprocess.call(["ls", "-l"])

David Cournapeau , 2008-09-18 01:39:35

edited Aug 31 '19 at 3:17 Corey Goldberg 50.5k 23 23 gold badges 114 114 silver badges 133 133 bronze badges answered Sep 18 '08 at 1:39 David Cournapeau 67.7k 7 7 gold badges 58 58 silver badges 67 67 bronze badges

Daniel F ,

Is there a way to use variable substitution? IE I tried to do echo $PATH by using call(["echo", "$PATH"]) , but it just echoed the literal string $PATH instead of doing any substitution. I know I could get the PATH environment variable, but I'm wondering if there is an easy way to have the command behave exactly as if I had executed it in bash. – Kevin Wheeler Sep 1 '15 at 23:17

Eli Courtwright ,

Here's a summary of the ways to call external programs and the advantages and disadvantages of each:
  1. os.system("some_command with args") passes the command and arguments to your system's shell. This is nice because you can actually run multiple commands at once in this manner and set up pipes and input/output redirection. For example:
    os.system("some_command < input_file | another_command > output_file")
    

However, while this is convenient, you have to manually handle the escaping of shell characters such as spaces, etc. On the other hand, this also lets you run commands which are simply shell commands and not actually external programs. See the documentation .

  1. stream = os.popen("some_command with args") will do the same thing as os.system except that it gives you a file-like object that you can use to access standard input/output for that process. There are 3 other variants of popen that all handle the i/o slightly differently. If you pass everything as a string, then your command is passed to the shell; if you pass them as a list then you don't need to worry about escaping anything. See the documentation .
  2. The Popen class of the subprocess module. This is intended as a replacement for os.popen but has the downside of being slightly more complicated by virtue of being so comprehensive. For example, you'd say:
    print subprocess.Popen("echo Hello World", shell=True, stdout=subprocess.PIPE).stdout.read()
    

    instead of:

    print os.popen("echo Hello World").read()
    

    but it is nice to have all of the options there in one unified class instead of 4 different popen functions. See the documentation .

  3. The call function from the subprocess module. This is basically just like the Popen class and takes all of the same arguments, but it simply waits until the command completes and gives you the return code. For example:
    return_code = subprocess.call("echo Hello World", shell=True)
    

    See the documentation .

  4. If you're on Python 3.5 or later, you can use the new subprocess.run function, which is a lot like the above but even more flexible and returns a CompletedProcess object when the command finishes executing.
  5. The os module also has all of the fork/exec/spawn functions that you'd have in a C program, but I don't recommend using them directly.

The subprocess module should probably be what you use.

Finally please be aware that for all methods where you pass the final command to be executed by the shell as a string and you are responsible for escaping it. There are serious security implications if any part of the string that you pass can not be fully trusted. For example, if a user is entering some/any part of the string. If you are unsure, only use these methods with constants. To give you a hint of the implications consider this code:

print subprocess.Popen("echo %s " % user_input, stdout=PIPE).stdout.read()

and imagine that the user enters something "my mama didnt love me && rm -rf /" which could erase the whole filesystem.

tripleee ,

Nice answer/explanation. How is this answer justifying Python's motto as described in this article ? fastcompany.com/3026446/ "Stylistically, Perl and Python have different philosophies. Perl's best known mottos is " There's More Than One Way to Do It". Python is designed to have one obvious way to do it" Seem like it should be the other way! In Perl I know only two ways to execute a command - using back-tick or open . – Jean May 26 '15 at 21:16

> ,

> ,

show 1 more comment

> ,

Typical implementation:
import subprocess

p = subprocess.Popen('ls', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
    print line,
retval = p.wait()

You are free to do what you want with the stdout data in the pipe. In fact, you can simply omit those parameters ( stdout= and stderr= ) and it'll behave like os.system() .

EmmEff ,

edited Oct 4 '19 at 2:33 Trenton McKinney 13.5k 13 13 gold badges 27 27 silver badges 43 43 bronze badges answered Sep 18 '08 at 18:20 EmmEff 6,517 2 2 gold badges 14 14 silver badges 18 18 bronze badges

tripleee ,

.readlines() reads all lines at once i.e., it blocks until the subprocess exits (closes its end of the pipe). To read in real time (if there is no buffering issues) you could: for line in iter(p.stdout.readline, ''): print line,jfs Nov 16 '12 at 14:12

newtover , 2010-02-12 10:15:34

Some hints on detaching the child process from the calling one (starting the child process in background).

Suppose you want to start a long task from a CGI script. That is, the child process should live longer than the CGI script execution process.

The classical example from the subprocess module documentation is:

import subprocess
import sys

# Some code here

pid = subprocess.Popen([sys.executable, "longtask.py"]) # Call subprocess

# Some more code here

The idea here is that you do not want to wait in the line 'call subprocess' until the longtask.py is finished. But it is not clear what happens after the line 'some more code here' from the example.

My target platform was FreeBSD, but the development was on Windows, so I faced the problem on Windows first.

On Windows (Windows XP), the parent process will not finish until the longtask.py has finished its work. It is not what you want in a CGI script. The problem is not specific to Python; in the PHP community the problems are the same.

The solution is to pass DETACHED_PROCESS Process Creation Flag to the underlying CreateProcess function in Windows API. If you happen to have installed pywin32, you can import the flag from the win32process module, otherwise you should define it yourself:

DETACHED_PROCESS = 0x00000008

pid = subprocess.Popen([sys.executable, "longtask.py"],
                       creationflags=DETACHED_PROCESS).pid

/* UPD 2015.10.27 @eryksun in a comment below notes, that the semantically correct flag is CREATE_NEW_CONSOLE (0x00000010) */

On FreeBSD we have another problem: when the parent process is finished, it finishes the child processes as well. And that is not what you want in a CGI script either. Some experiments showed that the problem seemed to be in sharing sys.stdout. And the working solution was the following:

pid = subprocess.Popen([sys.executable, "longtask.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)

I have not checked the code on other platforms and do not know the reasons of the behaviour on FreeBSD. If anyone knows, please share your ideas. Googling on starting background processes in Python does not shed any light yet.

maranas ,

i noticed a possible "quirk" with developing py2exe apps in pydev+eclipse. i was able to tell that the main script was not detached because eclipse's output window was not terminating; even if the script executes to completion it is still waiting for returns. but, when i tried compiling to a py2exe executable, the expected behavior occurs (runs the processes as detached, then quits). i am not sure, but the executable name is not in the process list anymore. this works for all approaches (os.system("start *"), os.spawnl with os.P_DETACH, subprocs, etc.) – maranas Apr 9 '10 at 8:09

> ,

Charlie Parker ,

you might also need CREATE_NEW_PROCESS_GROUP flag. See Popen waiting for child process even when the immediate child has terminatedjfs Nov 16 '12 at 14:16

nimish , 2008-09-18 01:37:24

import os
os.system("your command")

Note that this is dangerous, since the command isn't cleaned. I leave it up to you to google for the relevant documentation on the 'os' and 'sys' modules. There are a bunch of functions (exec* and spawn*) that will do similar things.

tripleee ,

No idea what I meant nearly a decade ago (check the date!), but if I had to guess, it would be that there's no validation done. – nimish Jun 6 '18 at 16:01

> ,

Nikolay Shindarov ,

Note the timestamp on this guy: the "correct" answer has 40x the votes and is answer #1. – nimish Dec 3 '18 at 18:41

sirwart , 2008-09-18 01:42:30

I'd recommend using the subprocess module instead of os.system because it does shell escaping for you and is therefore much safer.
subprocess.call(['ping', 'localhost'])

Lie Ryan ,

If you want to create a list out of a command with parameters , a list which can be used with subprocess when shell=False , then use shlex.split for an easy way to do this docs.python.org/2/library/shlex.html#shlex.split (it's the recommended way according to the docs docs.python.org/2/library/subprocess.html#popen-constructor ) – Daniel F Sep 20 '18 at 18:07

> ,

> ,

add a comment

> ,

import os
cmd = 'ls -al'
os.system(cmd)

If you want to return the results of the command, you can use os.popen . However, this is deprecated since version 2.6 in favor of the subprocess module , which other answers have covered well.

Alexandra Franks ,

edited Jan 26 '16 at 16:53 Patrick M 8,966 8 8 gold badges 54 54 silver badges 93 93 bronze badges answered Sep 18 '08 at 1:37 Alexandra Franks 2,726 1 1 gold badge 16 16 silver badges 22 22 bronze badges

Stefan Gruenwald ,

popen is deprecated in favor of subprocess . – Fox Wilson Aug 8 '14 at 0:22

> ,

There are lots of different libraries which allow you to call external commands with Python. For each library I've given a description and shown an example of calling an external command. The command I used as the example is ls -l (list all files). If you want to find out more about any of the libraries I've listed and linked the documentation for each of them.

Sources:

These are all the libraries:

Hopefully this will help you make a decision on which library to use :)

subprocess

Subprocess allows you to call external commands and connect them to their input/output/error pipes (stdin, stdout, and stderr). Subprocess is the default choice for running commands, but sometimes other modules are better.

subprocess.run(["ls", "-l"]) # Run command
subprocess.run(["ls", "-l"], stdout=subprocess.PIPE) # This will run the command and return any output
subprocess.run(shlex.split("ls -l")) # You can also use the shlex library to split the command

os

os is used for "operating system dependent functionality". It can also be used to call external commands with os.system and os.popen (Note: There is also a subprocess.popen). os will always run the shell and is a simple alternative for people who don't need to, or don't know how to use subprocess.run .

os.system("ls -l") # run command
os.popen("ls -l").read() # This will run the command and return any output

sh

sh is a subprocess interface which lets you call programs as if they were functions. This is useful if you want to run a command multiple times.

sh.ls("-l") # Run command normally
ls_cmd = sh.Command("ls") # Save command as a variable
ls_cmd() # Run command as if it were a function

plumbum

plumbum is a library for "script-like" Python programs. You can call programs like functions as in sh . Plumbum is useful if you want to run a pipeline without the shell.

ls_cmd = plumbum.local("ls -l") # get command
ls_cmd() # run command

pexpect

pexpect lets you spawn child applications, control them and find patterns in their output. This is a better alternative to subprocess for commands that expect a tty on Unix.

pexpect.run("ls -l") # Run command as normal
child = pexpect.spawn('scp foo user@example.com:.') # Spawns child application
child.expect('Password:') # When this is the output
child.sendline('mypassword')

fabric

fabric is a Python 2.5 and 2.7 library. It allows you to execute local and remote shell commands. Fabric is simple alternative for running commands in a secure shell (SSH)

fabric.operations.local('ls -l') # Run command as normal
fabric.operations.local('ls -l', capture = True) # Run command and receive output

envoy

envoy is known as "subprocess for humans". It is used as a convenience wrapper around the subprocess module.

r = envoy.run("ls -l") # Run command
r.std_out # get output

commands

commands contains wrapper functions for os.popen , but it has been removed from Python 3 since subprocess is a better alternative.

The edit was based on J.F. Sebastian's comment.

Tom Fuller ,

edited May 28 '17 at 23:14 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Oct 29 '16 at 14:02 Tom Fuller 4,280 6 6 gold badges 29 29 silver badges 38 38 bronze badges

> ,

add a comment

> ,

I always use fabric for this things like:
from fabric.operations import local
result = local('ls', capture=True)
print "Content:/n%s" % (result, )

But this seem to be a good tool: sh (Python subprocess interface) .

Look at an example:

from sh import vgdisplay
print vgdisplay()
print vgdisplay('-v')
print vgdisplay(v=True)

Jorge E. Cardona ,

edited Nov 29 '19 at 21:47 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Mar 13 '12 at 0:12 Jorge E. Cardona 81.8k 3 3 gold badges 29 29 silver badges 39 39 bronze badges

> ,

add a comment

> ,

Check the "pexpect" Python library, too.

It allows for interactive controlling of external programs/commands, even ssh, ftp, telnet, etc. You can just type something like:

child = pexpect.spawn('ftp 192.168.0.24')

child.expect('(?i)name .*: ')

child.sendline('anonymous')

child.expect('(?i)password')

athanassis , 2010-10-07 07:09:04

edited May 28 '17 at 23:02 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Oct 7 '10 at 7:09 athanassis 909 6 6 silver badges 4 4 bronze badges

> ,

add a comment

> ,

With the standard library

Use the subprocess module (Python 3):

import subprocess
subprocess.run(['ls', '-l'])

It is the recommended standard way. However, more complicated tasks (pipes, output, input, etc.) can be tedious to construct and write.

Note on Python version: If you are still using Python 2, subprocess.call works in a similar way.

ProTip: shlex.split can help you to parse the command for run , call , and other subprocess functions in case you don't want (or you can't!) provide them in form of lists:

import shlex
import subprocess
subprocess.run(shlex.split('ls -l'))
With external dependencies

If you do not mind external dependencies, use plumbum :

from plumbum.cmd import ifconfig
print(ifconfig['wlan0']())

It is the best subprocess wrapper. It's cross-platform, i.e. it works on both Windows and Unix-like systems. Install by pip install plumbum .

Another popular library is sh :

from sh import ifconfig
print(ifconfig('wlan0'))

However, sh dropped Windows support, so it's not as awesome as it used to be. Install by pip install sh .

6 revs, 2 users 79%
, 2019-11-29 21:54:25

edited Nov 29 '19 at 21:54 community wiki
6 revs, 2 users 79%
Honza Javorek

> ,

add a comment

> ,

If you need the output from the command you are calling, then you can use subprocess.check_output (Python 2.7+).
>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

Also note the shell parameter.

If shell is True , the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user's home directory. However, note that Python itself offers implementations of many shell-like features (in particular, glob , fnmatch , os.walk() , os.path.expandvars() , os.path.expanduser() , and shutil ).

Facundo Casco , 2011-04-28 20:29:29

edited Jun 3 '18 at 20:18 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Apr 28 '11 at 20:29 Facundo Casco 8,229 5 5 gold badges 38 38 silver badges 61 61 bronze badges

Bruno Bronosky ,

Note that check_output requires a list rather than a string. If you don't rely on quoted spaces to make your call valid, the simplest, most readable way to do this is subprocess.check_output("ls -l /dev/null".split()) . – Bruno Bronosky Jan 30 '18 at 18:18

> ,

This is how I run my commands. This code has everything you need pretty much
from subprocess import Popen, PIPE
cmd = "ls -l ~/"
p = Popen(cmd , shell=True, stdout=PIPE, stderr=PIPE)
out, err = p.communicate()
print "Return code: ", p.returncode
print out.rstrip(), err.rstrip()

Usman Khan , 2012-10-28 05:14:01

edited Oct 28 '12 at 5:44 answered Oct 28 '12 at 5:14 Usman Khan 621 5 5 silver badges 3 3 bronze badges

Adam Matan ,

I think it's acceptable for hard-coded commands, if it increases readability. – Adam Matan Apr 2 '14 at 13:07

> ,

Update:

subprocess.run is the recommended approach as of Python 3.5 if your code does not need to maintain compatibility with earlier Python versions. It's more consistent and offers similar ease-of-use as Envoy. (Piping isn't as straightforward though. See this question for how .)

Here's some examples from the documentation .

Run a process:

>>> subprocess.run(["ls", "-l"])  # Doesn't capture output
CompletedProcess(args=['ls', '-l'], returncode=0)

Raise on failed run:

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1

Capture output:

>>> subprocess.run(["ls", "-l", "/dev/null"], stdout=subprocess.PIPE)
CompletedProcess(args=['ls', '-l', '/dev/null'], returncode=0,
stdout=b'crw-rw-rw- 1 root root 1, 3 Jan 23 16:23 /dev/null\n')
Original answer:

I recommend trying Envoy . It's a wrapper for subprocess, which in turn aims to replace the older modules and functions. Envoy is subprocess for humans.

Example usage from the README :

>>> r = envoy.run('git config', data='data to pipe in', timeout=2)

>>> r.status_code
129
>>> r.std_out
'usage: git config [options]'
>>> r.std_err
''

Pipe stuff around too:

>>> r = envoy.run('uptime | pbcopy')

>>> r.command
'pbcopy'
>>> r.status_code
0

>>> r.history
[<Response 'uptime'>]

Joe , 2012-11-15 17:13:22

edited Nov 29 '19 at 21:52 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Nov 15 '12 at 17:13 Joe 14.1k 9 9 gold badges 52 52 silver badges 69 69 bronze badges

> ,

add a comment

> ,

Use subprocess .

...or for a very simple command:

import os
os.system('cat testfile')

Ben Hoffstein , 2008-09-18 01:43:30

edited Nov 29 '19 at 21:39 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Sep 18 '08 at 1:43 Ben Hoffstein 96.2k 8 8 gold badges 97 97 silver badges 117 117 bronze badges

> ,

add a comment

> ,

Calling an external command in Python

Simple, use subprocess.run , which returns a CompletedProcess object:

>>> import subprocess
>>> completed_process = subprocess.run('python --version')
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
>>> completed_process
CompletedProcess(args='python --version', returncode=0)
Why?

As of Python 3.5, the documentation recommends subprocess.run :

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

Here's an example of the simplest possible usage - and it does exactly as asked:

>>> import subprocess
>>> completed_process = subprocess.run('python --version')
Python 3.6.1 :: Anaconda 4.4.0 (64-bit)
>>> completed_process
CompletedProcess(args='python --version', returncode=0)

run waits for the command to successfully finish, then returns a CompletedProcess object. It may instead raise TimeoutExpired (if you give it a timeout= argument) or CalledProcessError (if it fails and you pass check=True ).

As you might infer from the above example, stdout and stderr both get piped to your own stdout and stderr by default.

We can inspect the returned object and see the command that was given and the returncode:

>>> completed_process.args
'python --version'
>>> completed_process.returncode
0
Capturing output

If you want to capture the output, you can pass subprocess.PIPE to the appropriate stderr or stdout :

>>> cp = subprocess.run('python --version', 
                        stderr=subprocess.PIPE, 
                        stdout=subprocess.PIPE)
>>> cp.stderr
b'Python 3.6.1 :: Anaconda 4.4.0 (64-bit)\r\n'
>>> cp.stdout
b''

(I find it interesting and slightly counterintuitive that the version info gets put to stderr instead of stdout.)

Pass a command list

One might easily move from manually providing a command string (like the question suggests) to providing a string built programmatically. Don't build strings programmatically. This is a potential security issue. It's better to assume you don't trust the input.

>>> import textwrap
>>> args = ['python', textwrap.__file__]
>>> cp = subprocess.run(args, stdout=subprocess.PIPE)
>>> cp.stdout
b'Hello there.\r\n  This is indented.\r\n'

Note, only args should be passed positionally.

Full Signature

Here's the actual signature in the source and as shown by help(run) :

def run(*popenargs, input=None, timeout=None, check=False, **kwargs):

The popenargs and kwargs are given to the Popen constructor. input can be a string of bytes (or unicode, if specify encoding or universal_newlines=True ) that will be piped to the subprocess's stdin.

The documentation describes timeout= and check=True better than I could:

The timeout argument is passed to Popen.communicate(). If the timeout expires, the child process will be killed and waited for. The TimeoutExpired exception will be re-raised after the child process has terminated.

If check is true, and the process exits with a non-zero exit code, a CalledProcessError exception will be raised. Attributes of that exception hold the arguments, the exit code, and stdout and stderr if they were captured.

and this example for check=True is better than one I could come up with:

>>> subprocess.run("exit 1", shell=True, check=True)
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command 'exit 1' returned non-zero exit status 1
Expanded Signature

Here's an expanded signature, as given in the documentation:

subprocess.run(args, *, stdin=None, input=None, stdout=None, stderr=None, 
shell=False, cwd=None, timeout=None, check=False, encoding=None, 
errors=None)

Note that this indicates that only the args list should be passed positionally. So pass the remaining arguments as keyword arguments.

Popen

When use Popen instead? I would struggle to find use-case based on the arguments alone. Direct usage of Popen would, however, give you access to its methods, including poll , 'send_signal', 'terminate', and 'wait'.

Here's the Popen signature as given in the source . I think this is the most precise encapsulation of the information (as opposed to help(Popen) ):

def __init__(self, args, bufsize=-1, executable=None,
             stdin=None, stdout=None, stderr=None,
             preexec_fn=None, close_fds=_PLATFORM_DEFAULT_CLOSE_FDS,
             shell=False, cwd=None, env=None, universal_newlines=False,
             startupinfo=None, creationflags=0,
             restore_signals=True, start_new_session=False,
             pass_fds=(), *, encoding=None, errors=None):

But more informative is the Popen documentation :

subprocess.Popen(args, bufsize=-1, executable=None, stdin=None,
                 stdout=None, stderr=None, preexec_fn=None, close_fds=True,
                 shell=False, cwd=None, env=None, universal_newlines=False,
                 startupinfo=None, creationflags=0, restore_signals=True,
                 start_new_session=False, pass_fds=(), *, encoding=None, errors=None)

Execute a child program in a new process. On POSIX, the class uses os.execvp()-like behavior to execute the child program. On Windows, the class uses the Windows CreateProcess() function. The arguments to Popen are as follows.

Understanding the remaining documentation on Popen will be left as an exercise for the reader.

Aaron Hall , 2017-10-18 16:37:52

answered Oct 18 '17 at 16:37 Aaron Hall ♦ 254k 68 68 gold badges 349 349 silver badges 300 300 bronze badges

tripleee ,

A simple example of two-way communication between a primary process and a subprocess can be found here: stackoverflow.com/a/52841475/1349673James Hirschorn Oct 16 '18 at 18:05

> ,

os.system is OK, but kind of dated. It's also not very secure. Instead, try subprocess . subprocess does not call sh directly and is therefore more secure than os.system .

Get more information here .

Martin W , 2008-09-18 01:53:27

edited Dec 10 '16 at 13:25 Dimitris Fasarakis Hilliard 106k 24 24 gold badges 206 206 silver badges 210 210 bronze badges answered Sep 18 '08 at 1:53 Martin W 1,269 7 7 silver badges 12 12 bronze badges

tripleee ,

While I agree with the overall recommendation, subprocess does not remove all of the security problems, and has some pesky issues of its own. – tripleee Dec 3 '18 at 5:36

> ,

There is also Plumbum
>>> from plumbum import local
>>> ls = local["ls"]
>>> ls
LocalCommand(<LocalPath /bin/ls>)
>>> ls()
u'build.py\ndist\ndocs\nLICENSE\nplumbum\nREADME.rst\nsetup.py\ntests\ntodo.txt\n'
>>> notepad = local["c:\\windows\\notepad.exe"]
>>> notepad()                                   # Notepad window pops up
u''                                             # Notepad window is closed by user, command returns

stuckintheshuck ,

answered Oct 10 '14 at 17:41 stuckintheshuck 2,123 2 2 gold badges 22 22 silver badges 31 31 bronze badges

> ,

add a comment

> ,

It can be this simple:
import os
cmd = "your command"
os.system(cmd)

Samadi Salahedine , 2018-04-30 13:47:17

edited Jun 8 '18 at 12:06 answered Apr 30 '18 at 13:47 Samadi Salahedine 478 4 4 silver badges 12 12 bronze badges

tripleee ,

This fails to point out the drawbacks, which are explained in much more detail in PEP-324 . The documentation for os.system explicitly recommends avoiding it in favor of subprocess . – tripleee Dec 3 '18 at 5:02

> ,

Use:
import os

cmd = 'ls -al'

os.system(cmd)

os - This module provides a portable way of using operating system-dependent functionality.

For the more os functions, here is the documentation.

Priyankara ,

edited May 28 '17 at 23:05 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jun 29 '15 at 11:34 Priyankara 710 11 11 silver badges 20 20 bronze badges

Corey Goldberg ,

it's also deprecated. use subprocess – Corey Goldberg Dec 9 '15 at 18:13

> ,

I quite like shell_command for its simplicity. It's built on top of the subprocess module.

Here's an example from the documentation:

>>> from shell_command import shell_call
>>> shell_call("ls *.py")
setup.py  shell_command.py  test_shell_command.py
0
>>> shell_call("ls -l *.py")
-rw-r--r-- 1 ncoghlan ncoghlan  391 2011-12-11 12:07 setup.py
-rw-r--r-- 1 ncoghlan ncoghlan 7855 2011-12-11 16:16 shell_command.py
-rwxr-xr-x 1 ncoghlan ncoghlan 8463 2011-12-11 16:17 test_shell_command.py
0

mdwhatcott ,

edited Nov 29 '19 at 21:49 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Aug 13 '12 at 18:36 mdwhatcott 4,547 2 2 gold badges 29 29 silver badges 45 45 bronze badges

> ,

add a comment

> ,

There is another difference here which is not mentioned previously.

subprocess.Popen executes the <command> as a subprocess. In my case, I need to execute file <a> which needs to communicate with another program, <b>.

I tried subprocess, and execution was successful. However <b> could not communicate with <a>. Everything is normal when I run both from the terminal.

One more: (NOTE: kwrite behaves different from other applications. If you try the below with Firefox, the results will not be the same.)

If you try os.system("kwrite") , program flow freezes until the user closes kwrite. To overcome that I tried instead os.system(konsole -e kwrite) . This time program continued to flow, but kwrite became the subprocess of the console.

Anyone runs the kwrite not being a subprocess (i.e. in the system monitor it must appear at the leftmost edge of the tree).

Atinc Delican ,

edited Jun 3 '18 at 20:14 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jan 8 '10 at 21:11 Atinc Delican 265 2 2 silver badges 2 2 bronze badges

Peter Mortensen ,

What do you mean by "Anyone runs the kwrite not being a subprocess" ? – Peter Mortensen Jun 3 '18 at 20:14

> ,

os.system does not allow you to store results, so if you want to store results in some list or something, a subprocess.call works.

Saurabh Bangad ,

edited Nov 29 '19 at 21:48 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jun 11 '12 at 22:28 Saurabh Bangad 357 2 2 silver badges 2 2 bronze badges

> ,

add a comment

> ,

subprocess.check_call is convenient if you don't want to test return values. It throws an exception on any error.

cdunn2001 , 2011-01-18 19:21:44

answered Jan 18 '11 at 19:21 cdunn2001 14.9k 7 7 gold badges 49 49 silver badges 42 42 bronze badges

> ,

add a comment

> ,

I tend to use subprocess together with shlex (to handle escaping of quoted strings):
>>> import subprocess, shlex
>>> command = 'ls -l "/your/path/with spaces/"'
>>> call_params = shlex.split(command)
>>> print call_params
["ls", "-l", "/your/path/with spaces/"]
>>> subprocess.call(call_params)

Emil Stenström , 2014-04-30 14:37:04

answered Apr 30 '14 at 14:37 Emil Stenström 8,952 8 8 gold badges 43 43 silver badges 68 68 bronze badges

> ,

add a comment

> ,

Shameless plug, I wrote a library for this :P https://github.com/houqp/shell.py

It's basically a wrapper for popen and shlex for now. It also supports piping commands so you can chain commands easier in Python. So you can do things like:

ex('echo hello shell.py') | "awk '{print $2}'"

houqp , 2014-05-01 20:49:01

answered May 1 '14 at 20:49 houqp 571 7 7 silver badges 11 11 bronze badges

> ,

add a comment

> ,

You can use Popen, and then you can check the procedure's status:
from subprocess import Popen

proc = Popen(['ls', '-l'])
if proc.poll() is None:
    proc.kill()

Check out subprocess.Popen .

admire ,

edited May 28 '17 at 23:01 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jul 16 '12 at 15:16 admire 320 2 2 silver badges 6 6 bronze badges

> ,

add a comment

> ,

In Windows you can just import the subprocess module and run external commands by calling subprocess.Popen() , subprocess.Popen().communicate() and subprocess.Popen().wait() as below:
# Python script to run a command line
import subprocess

def execute(cmd):
    """
        Purpose  : To execute a command and return exit status
        Argument : cmd - command to execute
        Return   : exit_code
    """
    process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    (result, error) = process.communicate()

    rc = process.wait()

    if rc != 0:
        print "Error: failed to execute command:", cmd
        print error
    return result
# def

command = "tasklist | grep python"
print "This process detail: \n", execute(command)

Output:

This process detail:
python.exe                     604 RDP-Tcp#0                  4      5,660 K

Swadhikar C ,

edited May 28 '17 at 23:08 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jun 17 '16 at 9:14 Swadhikar C 1,578 1 1 gold badge 15 15 silver badges 27 27 bronze badges

> ,

add a comment

> ,

To fetch the network id from the OpenStack Neutron :
#!/usr/bin/python
import os
netid = "nova net-list | awk '/ External / { print $2 }'"
temp = os.popen(netid).read()  /* Here temp also contains new line (\n) */
networkId = temp.rstrip()
print(networkId)

Output of nova net-list

+--------------------------------------+------------+------+
| ID                                   | Label      | CIDR |
+--------------------------------------+------------+------+
| 431c9014-5b5d-4b51-a357-66020ffbb123 | test1      | None |
| 27a74fcd-37c0-4789-9414-9531b7e3f126 | External   | None |
| 5a2712e9-70dc-4b0e-9281-17e02f4684c9 | management | None |
| 7aa697f5-0e60-4c15-b4cc-9cb659698512 | Internal   | None |
+--------------------------------------+------------+------+

Output of print(networkId)

27a74fcd-37c0-4789-9414-9531b7e3f126

IRSHAD ,

edited Nov 29 '19 at 22:05 Peter Mortensen 26.3k 21 21 gold badges 91 91 silver badges 121 121 bronze badges answered Jul 20 '16 at 9:50 IRSHAD 1,995 22 22 silver badges 34 34 bronze badges

tripleee ,

You should not recommend os.popen() in 2016. The Awk script could easily be replaced with native Python code. – tripleee Dec 3 '18 at 5:49

Yuval Atzmon ,

Under Linux, in case you would like to call an external command that will execute independently (will keep running after the python script terminates), you can use a simple queue as task spooler or the at command

An example with task spooler:

import os
os.system('ts <your-command>')

Notes about task spooler ( ts ):

  1. You could set the number of concurrent processes to be run ("slots") with:

    ts -S <number-of-slots>

  2. Installing ts doesn't requires admin privileges. You can download and compile it from source with a simple make , add it to your path and you're done.

tripleee ,

ts is not standard on any distro I know of, though the pointer to at is mildly useful. You should probably also mention batch . As elsewhere, the os.system() recommendation should probably at least mention that subprocess is its recommended replacement. – tripleee Dec 3 '18 at 5:43

[Aug 15, 2020] Facebook open-sources a static analyzer for Python code - Help Net Security

Aug 15, 2020 | www.helpnetsecurity.com

Facebook has open-sourced Pysa (Python Static Analyzer), a tool that looks at how data flows through the code and helps developers prevent data flowing into places it shouldn't.

... "Pysa performs iterative rounds of analysis to build summaries to determine which functions return data from a source and which functions have parameters that eventually reach a sink. If Pysa finds that a source eventually connects to a sink, it reports an issue."

You can get Pysa from here and you can use a number of already developed definitions to help it find security issues.

[Aug 12, 2020] regex - Python positive-lookbehind split variable-width

Jan 01, 2014 | stackoverflow.com

Ask Question Asked 6 years, 4 months ago Active 6 years, 4 months ago Viewed 624 times

https://tpc.googlesyndication.com/safeframe/1-0-37/html/container.html Report this ad


Mr. Polywhirl , 2014-03-30 18:37:42

I though that I have set up the expression appropriately, but the split is not working as intended.

c = re.compile(r'(?<=^\d\.\d{1,2})\s+');
for header in ['1.1 Introduction', '1.42 Appendix']:
    print re.split(c, header)

Expected result:

['1.1', 'Introduction']
['1.42',  'Appendix']

I am getting the following stacktrace:

Traceback (most recent call last):
File "foo.py", line 1, in
c = re.compile(r'(?<=^\d.\d{1,2})\s+');
File "C:\Python27\lib\re.py", line 190, in compile
return _compile(pattern, flags)
File "C:\Python27\lib\re.py", line 242, in _compile
raise error, v # invalid expression
sre_constants.error: look-behind requires fixed-width pattern
<<< Process finished. (Exit code 1)

BrenBarn ,

the error message says it - you can't have variable-length lookaround in python's regex engine. – roippi Mar 30 '14 at 18:40

Jerry , 2014-03-30 18:43:00

Lookbehinds in python cannot be of variable width, so your lookbehind is not valid.

You can use a capture group as a workaround:

c = re.compile(r'(^\d\.\d{1,2})\s+');
for header in ['1.1 Introduction', '1.42 Appendix']:
    print re.split(c, header)[1:] # Remove the first element because it's empty

Output:

['1.1', 'Introduction']
['1.42', 'Appendix']

Jerry ,

Do as long as the expression is not global, splitting in the first space is your approach. Sounds like it will work in my case. – Mr. Polywhirl Mar 30 '14 at 18:57

Ammar , 2014-03-30 18:41:57

your error in the regex is in the part {1,2} because Lookbehinds need to be fixed-width, thus quantifiers are not allowed.

try this website to test your regex before you put it in code.

BUT in your case you don't need to use regex at all:

simply try this:

for header in ['1.1 Introduction', '1.42 Appendix']:
    print header.split(' ')

result:

['1.1', 'Introduction']
['1.42', 'Appendix']

hope this helps.

Ammar ,

So; * , + , {n} , etc. are all not allowed? – Mr. Polywhirl Mar 30 '14 at 18:55

> ,

My solution may look lame. But you are checking only two digits after dot. So, you can use two lookbehind.

c = re.compile(r'(?:(?<=^\d\.\d\d)|(?<=^\d\.\d))\s+');

[Aug 12, 2020] Finding Patterns of Text with Regular Expressions

Aug 12, 2020 | automatetheboringstuff.com

Finding Patterns of Text with Regular Expressions

The previous phone number–finding program works, but it uses a lot of code to do something limited: the isPhoneNumber() function is 17 lines but can find only one pattern of phone numbers. What about a phone number formatted like 415.555.4242 or (415) 555-4242? What if the phone number had an extension, like 415-555-4242 x99? The isPhoneNumber() function would fail to validate them. You could add yet more code for these additional patterns, but there is an easier way.

Regular expressions, called regexes for short, are descriptions for a pattern of text. For example, a \d in a regex stands for a digit character -- that is, any single numeral from 0 to 9. The regex \d\d\d-\d\d\d-\d\d\d\d is used by Python to match the same text pattern the previous isPhoneNumber() function did: a string of three numbers, a hyphen, three more numbers, another hyphen, and four numbers. Any other string would not match the \d\d\d-\d\d\d-\d\d\d\d regex.

But regular expressions can be much more sophisticated. For example, adding a 3 in braces ( {3} ) after a pattern is like saying, "Match this pattern three times." So the slightly shorter regex \d{3}-\d{3}-\d{4} also matches the correct phone number format.

Creating Regex Objects

All the regex functions in Python are in the re module. Enter the following into the interactive shell to import this module:

>>> import re

NOTE

Most of the examples in this chapter will require the re module, so remember to import it at the beginning of any script you write or any time you restart Mu. Otherwise, you'll get a NameError: name 're' is not defined error message.

Passing a string value representing your regular expression to re.compile() returns a Regex pattern object (or simply, a Regex object).

To create a Regex object that matches the phone number pattern, enter the following into the interactive shell. (Remember that \d means "a digit character" and \d\d\d-\d\d\d-\d\d\d\d is the regular expression for a phone number pattern.)

>>> phoneNumRegex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d')

Now the phoneNumRegex variable contains a Regex object.

Matching Regex Objects

A Regex object's search() method searches the string it is passed for any matches to the regex. The search() method will return None if the regex pattern is not found in the string. If the pattern is found, the search() method returns a Match object, which have a group() method that will return the actual matched text from the searched string. (I'll explain groups shortly.) For example, enter the following into the interactive shell:

>>> phoneNumRegex = re.compile(r'\d\d\d-\d\d\d-\d\d\d\d')
>>> mo = phoneNumRegex.search('My number is 415-555-4242.')
>>> print('Phone number found: ' + mo.group())
Phone number found: 415-555-4242

The mo variable name is just a generic name to use for Match objects. This example might seem complicated at first, but it is much shorter than the earlier isPhoneNumber.py program and does the same thing.

Here, we pass our desired pattern to re.compile() and store the resulting Regex object in phoneNumRegex . Then we call search() on phoneNumRegex and pass search() the string we want to match for during the search. The result of the search gets stored in the variable mo . In this example, we know that our pattern will be found in the string, so we know that a Match object will be returned. Knowing that mo contains a Match object and not the null value None , we can call group() on mo to return the match. Writing mo.group() inside our print() function call displays the whole match, 415-555-4242 .

Review of Regular Expression Matching

While there are several steps to using regular expressions in Python, each step is fairly simple.

  1. Import the regex module with import re .
  2. Create a Regex object with the re.compile() function. (Remember to use a raw string.)
  3. Pass the string you want to search into the Regex object's search() method. This returns a Match object.
  4. Call the Match object's group() method to return a string of the actual matched text.

NOTE

While I encourage you to enter the example code into the interactive shell, you should also make use of web-based regular expression testers, which can show you exactly how a regex matches a piece of text that you enter. I recommend the tester at https://pythex.org/ . More Pattern Matching with Regular Expressions

Now that you know the basic steps for creating and finding regular expression objects using Python, you're ready to try some of their more powerful pattern-matching capabilities.

Grouping with Parentheses

Say you want to separate the area code from the rest of the phone number. Adding parentheses will create groups in the regex: (\d\d\d)-(\d\d\d-\d\d\d\d) . Then you can use the group() match object method to grab the matching text from just one group.

The first set of parentheses in a regex string will be group 1 . The second set will be group 2 . By passing the integer 1 or 2 to the group() match object method, you can grab different parts of the matched text. Passing 0 or nothing to the group() method will return the entire matched text. Enter the following into the interactive shell:

>>> phoneNumRegex = re.compile(r'(\d\d\d)-(\d\d\d-\d\d\d\d)')
>>> mo = phoneNumRegex.search('My number is 415-555-4242.')
>>> mo.group(1)
'415'
>>> mo.group(2)
'555-4242'
>>> mo.group(0)
'415-555-4242'
>>> mo.group()
'415-555-4242'

If you would like to retrieve all the groups at once, use the groups() method -- note the plural form for the name.

>>> mo.groups()
('415', '555-4242')
>>> areaCode, mainNumber = mo.groups()
>>> print(areaCode)
415
>>> print(mainNumber)
555-4242

Since mo.groups() returns a tuple of multiple values, you can use the multiple-assignment trick to assign each value to a separate variable, as in the previous areaCode, mainNumber = mo.groups() line.

Parentheses have a special meaning in regular expressions, but what do you do if you need to match a parenthesis in your text? For instance, maybe the phone numbers you are trying to match have the area code set in parentheses. In this case, you need to escape the ( and ) characters with a backslash. Enter the following into the interactive shell:

>>> phoneNumRegex = re.compile(r'(\(\d\d\d\)) (\d\d\d-\d\d\d\d)')
>>> mo = phoneNumRegex.search('My phone number is (415) 555-4242.')
>>> mo.group(1)
'(415)'
>>> mo.group(2)
'555-4242'

The \( and \) escape characters in the raw string passed to re.compile() will match actual parenthesis characters. In regular expressions, the following characters have special meanings:

. ^ $ * + ? { } [ ] \ | ( )

If you want to detect these characters as part of your text pattern, you need to escape them with a backslash:

\. \^ \$ \* \+ \? \{ \} \[ \] \\ \| \( \)

Make sure to double-check that you haven't mistaken escaped parentheses \( and \) for parentheses ( and ) in a regular expression. If you receive an error message about "missing )" or "unbalanced parenthesis," you may have forgotten to include the closing unescaped parenthesis for a group, like in this example:

>>> re.compile(r'(\(Parentheses\)')
Traceback (most recent call last):
-- snip --
re.error: missing ), unterminated subpattern at position 0

The error message tells you that there is an opening parenthesis at index 0 of the r'(\(Parentheses\)' string that is missing its corresponding closing parenthesis.

Matching Multiple Groups with the Pipe

The | character is called a pipe . You can use it anywhere you want to match one of many expressions. For example, the regular expression r'Batman|Tina Fey' will match either 'Batman' or 'Tina Fey' .

When both Batman and Tina Fey occur in the searched string, the first occurrence of matching text will be returned as the Match object. Enter the following into the interactive shell:

>>> heroRegex = re.compile (r'Batman|Tina Fey')
>>> mo1 = heroRegex.search('Batman and Tina Fey')
>>> mo1.group()
'Batman'

>>> mo2 = heroRegex.search('Tina Fey and Batman')
>>> mo2.group()
'Tina Fey'

NOTE

You can find all matching occurrences with the findall() method that's discussed in " The findall() Method " on page 171 .

You can also use the pipe to match one of several patterns as part of your regex. For example, say you wanted to match any of the strings 'Batman' , 'Batmobile' , 'Batcopter' , and 'Batbat' . Since all these strings start with Bat , it would be nice if you could specify that prefix only once. This can be done with parentheses. Enter the following into the interactive shell:

>>> batRegex = re.compile(r'Bat(man|mobile|copter|bat)')
>>> mo = batRegex.search('Batmobile lost a wheel')
>>> mo.group()
'Batmobile'
>>> mo.group(1)
'mobile'

The method call mo.group() returns the full matched text 'Batmobile' , while mo.group(1) returns just the part of the matched text inside the first parentheses group, 'mobile' . By using the pipe character and grouping parentheses, you can specify several alternative patterns you would like your regex to match.

If you need to match an actual pipe character, escape it with a backslash, like \| .

Optional Matching with the Question Mark

Sometimes there is a pattern that you want to match only optionally. That is, the regex should find a match regardless of whether that bit of text is there. The ? character flags the group that precedes it as an optional part of the pattern. For example, enter the following into the interactive shell:

>>> batRegex = re.compile(r'Bat(wo)?man')
>>> mo1 = batRegex.search('The Adventures of Batman')
>>> mo1.group()
'Batman'

>>> mo2 = batRegex.search('The Adventures of Batwoman')
>>> mo2.group()
'Batwoman'

The (wo)? part of the regular expression means that the pattern wo is an optional group. The regex will match text that has zero instances or one instance of wo in it. This is why the regex matches both 'Batwoman' and 'Batman' .

Using the earlier phone number example, you can make the regex look for phone numbers that do or do not have an area code. Enter the following into the interactive shell:

>>> phoneRegex = re.compile(r'(\d\d\d-)?\d\d\d-\d\d\d\d')
>>> mo1 = phoneRegex.search('My number is 415-555-4242')
>>> mo1.group()
'415-555-4242'

>>> mo2 = phoneRegex.search('My number is 555-4242')
>>> mo2.group()
'555-4242'

You can think of the ? as saying, "Match zero or one of the group preceding this question mark."

If you need to match an actual question mark character, escape it with \? .

[Aug 11, 2020] Python debugger: Stepping into a function that you have called interactively

Jan 01, 2008 | stackoverflow.com

Ask Question Asked 11 years, 9 months ago Active 5 years, 4 months ago Viewed 28k times

https://tpc.googlesyndication.com/safeframe/1-0-37/html/container.html Report this ad


, 2008-10-23 05:35:32

Python is quite cool, but unfortunately, its debugger is not as good as perl -d.

One thing that I do very commonly when experimenting with code is to call a function from within the debugger, and step into that function, like so:

# NOTE THAT THIS PROGRAM EXITS IMMEDIATELY WITHOUT CALLING FOO()
~> cat -n /tmp/show_perl.pl
1  #!/usr/local/bin/perl
2
3  sub foo {
4      print "hi\n";
5      print "bye\n";
6  }
7
8  exit 0;

~> perl -d /tmp/show_perl.pl

Loading DB routines from perl5db.pl version 1.28
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(/tmp/show_perl.pl:8):    exit 0;

# MAGIC HAPPENS HERE -- I AM STEPPING INTO A FUNCTION THAT I AM CALLING INTERACTIVELY
  DB<1> s foo() 
main::((eval 6)[/usr/local/lib/perl5/5.8.6/perl5db.pl:628]:3):
3:      foo();


  DB<<2>> s
main::foo(/tmp/show_perl.pl:4):     print "hi\n";


  DB<<2>> n
hi
main::foo(/tmp/show_perl.pl:5):     print "bye\n";


  DB<<2>> n
bye


  DB<2> n
Debugged program terminated.  Use q to quit or R to restart,
  use O inhibit_exit to avoid stopping after program termination,
  h q, h R or h O to get additional info.  

  DB<2> q

This is incredibly useful when trying to step through a function's handling of various different inputs to figure out why it fails. However, it does not seem to work in either pdb or pydb (I'd show an equivalent python example to the one above but it results in a large exception stack dump).

So my question is twofold:

  1. Am I missing something?
  2. Is there a python debugger that would indeed let me do this?

Obviously I could put the calls in the code myself, but I love working interactively, eg. not having to start from scratch when I want to try calling with a slightly different set of arguments.

Dunatotatos ,

can you please accept the last answer? – sureshvv Apr 15 '15 at 6:30

, 2008-10-23 05:42:00

And I've answered my own question! It's the "debug" command in pydb:

~> cat -n /tmp/test_python.py
     1  #!/usr/local/bin/python
     2
     3  def foo():
     4      print "hi"
     5      print "bye"
     6
     7  exit(0)
     8

~> pydb /tmp/test_python.py
(/tmp/test_python.py:7):  <module>
7 exit(0)


(Pydb) debug foo()
ENTERING RECURSIVE DEBUGGER
------------------------Call level 11
(/tmp/test_python.py:3):  foo
3 def foo():

((Pydb)) s
(/tmp/test_python.py:4):  foo
4     print "hi"

((Pydb)) s
hi
(/tmp/test_python.py:5):  foo
5     print "bye"


((Pydb)) s
bye
------------------------Return from level 11 (<type 'NoneType'>)
----------------------Return from level 10 (<type 'NoneType'>)
LEAVING RECURSIVE DEBUGGER
(/tmp/test_python.py:7):  <module>

,

add a comment

https://tpc.googlesyndication.com/safeframe/1-0-37/html/container.html Report this ad

Simon ,

You can interactively debug a function with pdb as well, provided the script you want to debug does not exit() at the end:

$ cat test.py
#!/usr/bin/python

def foo(f, g):
        h = f+g
        print h
        return 2*f

To debug, start an interactive python session and import pdb:

$ python
Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) 
[GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pdb
>>> import test
>>> pdb.runcall(test.foo, 1, 2)
> /Users/simon/Desktop/test.py(4)foo()
-> h = f+g
(Pdb) n
> /Users/simon/Desktop/test.py(5)foo()
-> print h
(Pdb)

The pdb module comes with python and is documented in the modules docs at http://docs.python.org/modindex.html

> ,

add a comment

Jerub , 2008-10-23 05:46:25

There is a python debugger that is part of the core distribution of python called 'pdb'. I rarely use it myself, but find it useful sometimes.

Given this program:

def foo():
    a = 0
    print "hi"

    a += 1

    print "bye"

foo()

Here is a session debugging it:

$ python /usr/lib/python2.5/pdb.py /var/tmp/pdbtest.py         ~
> /var/tmp/pdbtest.py(2)<module>()
-> def foo():
(Pdb) s
> /var/tmp/pdbtest.py(10)<module>()
-> foo()
(Pdb) s
--Call--
> /var/tmp/pdbtest.py(2)foo()
-> def foo():
(Pdb) s
> /var/tmp/pdbtest.py(3)foo()
-> a = 0
(Pdb) s
> /var/tmp/pdbtest.py(4)foo()
-> print "hi"
(Pdb) print a
0
(Pdb) s
hi
> /var/tmp/pdbtest.py(6)foo()
-> a += 1
(Pdb) s
> /var/tmp/pdbtest.py(8)foo()
-> print "bye"
(Pdb) print a
1
(Pdb) s
bye
--Return--
> /var/tmp/pdbtest.py(8)foo()->None
-> print "bye"
(Pdb) s
--Return--
> /var/tmp/pdbtest.py(10)<module>()->None
-> foo()
(Pdb) s

> ,

add a comment

https://tpc.googlesyndication.com/safeframe/1-0-37/html/container.html Report this ad

Alex Coventry , 2008-10-23 10:55:53

For interactive work on code I'm developing, I usually find it more efficient to set a programmatic "break point" in the code itself with pdb.set_trace . This makes it easir to break on the program's state deep in a a loop, too: if <state>: pdb.set_trace()

Daryl Spitzer ,

See also stackoverflow.com/questions/150375/Daryl Spitzer Oct 23 '08 at 16:23

Jeremy Cantrell , 2008-10-23 17:07:50

If you're more familiar with a GUI debugger, there's winpdb ('win' in this case does not refer to Windows). I actually use it on Linux.

On debian/ubuntu:

sudo aptitude install winpdb

Then just put this in your code where you want it to break:

import rpdb2; rpdb2.start_embedded_debugger_interactive_password()

Then start winpdb and attach to your running script.

Adam Ryczkowski ,

Thanks. I wasn't aware of that! – Ben Oct 23 '13 at 15:02

[Aug 09, 2020] Does Python have a defined or operator like Perl- - Stack Overflow

Aug 09, 2020 | stackoverflow.com

Franz Kafka ,

This question already has answers here :

juanpa.arrivillaga ,

the issue isn't or it's that foo['doesntexist'] is raising an error. – juanpa.arrivillaga Mar 4 '17 at 1:08

Francisco Couzo ,

You don't need the or at all:

bar = foo.get('doesntexist', foo['key1'])

> ,

Perfect, thank you! That is exactly what I was looking for. – Franz Kafka Mar 4 '17 at 1:11

> ,

The reason your method is not working is because Python raises an error before your boolean expression is able to be evaluated. Instead of simply indexing the dictionary, you need to use .get() instead. Rather than raising an error, .get() returns None which allows your expression to be evaluated correctly:

bar = foo.get('doesntexist') or foo.get('key1')

Note that if the value of the first key is 0 or any other value that evaluates to false, your method will fail. If this is fine with you than use the method above.

However, if you'd like not to have the behavior described above, you must test if the first value is None . At this point however, it would be better to simply extract this logic to a separate function:

def get_key(d, test_key, default_key):
    """
    Given a dictionary(d), return the value of test_key.
    If test_key does not exists, return the default_key 
    instead.
    """
    if d.get(test_key) is not None:
        return d.get(test_key)
    return d.get(default_key)

Which can be used like so:

foo = {'key1':'val1','key2':'val2'}
bar = get_key(foo, 'doesntexist', 'key1')

[Aug 09, 2020] Python2 Tutorial Passing Arguments

Notable quotes:
"... This means that Python initially behaves like call-by-reference, but as soon as we are changing the value of such a variable, Python "switches" to call-by-value. ..."
Nov 23, 2019 | python-course.eu
Parameter Passing "call by value" and "call by name" The most common evaluation strategy when passing arguments to a function has been call by value and call by reference: In ALGOL 60 and COBOL has been known a different concept, which was called call-by-name, which isn't used anymore in modern languages. and what about Python? There are books which call the strategy of Python call-by-value and others call it call-by-reference. You may ask yourself, what is right. Humpty Dumpty supplies the explanation: Humpty Dumpty supplies the explanation: Humpty Dumpty supplies the explanation: --- "When I use a word," Humpty Dumpty said, in a rather a scornful tone, "it means just what I choose it to mean - neither more nor less." --- "The question is," said Alice, "whether you can make words mean so many different things." --- "The question is," said Humpty Dumpty, "which is to be master - that's all." --- "The question is," said Alice, "whether you can make words mean so many different things." --- "The question is," said Humpty Dumpty, "which is to be master - that's all." --- "The question is," said Humpty Dumpty, "which is to be master - that's all." Lewis Carroll, Through the Looking-Glass To come back to our initial question what is used in Python: The authors who call the mechanism call-by-value and those who call it call-by-reference are stretching the definitions until they fit. Correctly speaking, Python uses a mechanism, which is known as "Call-by-Object", sometimes also called "Call by Object Reference" or "Call by Sharing". To come back to our initial question what is used in Python: The authors who call the mechanism call-by-value and those who call it call-by-reference are stretching the definitions until they fit. Correctly speaking, Python uses a mechanism, which is known as "Call-by-Object", sometimes also called "Call by Object Reference" or "Call by Sharing". To come back to our initial question what is used in Python: The authors who call the mechanism call-by-value and those who call it call-by-reference are stretching the definitions until they fit. Correctly speaking, Python uses a mechanism, which is known as "Call-by-Object", sometimes also called "Call by Object Reference" or "Call by Sharing". Correctly speaking, Python uses a mechanism, which is known as "Call-by-Object", sometimes also called "Call by Object Reference" or "Call by Sharing". Paramterübergabe

If you pass immutable arguments like integers, strings or tuples to a function, the passing acts like call-by-value. The object reference is passed to the function parameters. They can't be changed within the function, because they can't be changed at all, i.e. they are immutable. It's different, if we pass mutable arguments. They are also passed by object reference, but they can be changed in place in the function. If we pass a list to a function, we have to consider two cases: Elements of a list can be changed in place, i.e. the list will be changed even in the caller's scope. If a new list is assigned to the name, the old list will not be affected, i.e. the list in the caller's scope will remain untouched. First, let's have a look at the integer variables. The parameter inside of the function remains a reference to the arguments variable, as long as the parameter is not changed. As soon as a new value will be assigned to it, Python creates a separate local variable. The caller's variable will not be changed this way: First, let's have a look at the integer variables. The parameter inside of the function remains a reference to the arguments variable, as long as the parameter is not changed. As soon as a new value will be assigned to it, Python creates a separate local variable. The caller's variable will not be changed this way: First, let's have a look at the integer variables. The parameter inside of the function remains a reference to the arguments variable, as long as the parameter is not changed. As soon as a new value will be assigned to it, Python creates a separate local variable. The caller's variable will not be changed this way:

def ref_demo(x):
    print "x=",x," id=",id(x)
    x=42
    print "x=",x," id=",id(x)
Paramterübergabe In the example above, we used the id() function, which takes an object as a parameter. id(obj) returns the "identity" of the object "obj". This identity, the return value of the function, is an integer which is unique and constant for this object during its lifetime. Two different objects with non-overlapping lifetimes may have the same id() value. If you call the function ref_demo() - like we do in the green block further down - we can check with the id() function what happens to x. We can see that in the main scope, x has the identity 41902552. In the first print statement of the ref_demo() function, the x from the main scope is used, because we can see that we get the same identity. After we have assigned the value 42 to x, x gets a new identity 41903752, i.e. a separate memory location from the global x. So, when we are back in the main scope x has still the original value 9. This means that Python initially behaves like call-by-reference, but as soon as we are changing the value of such a variable, Python "switches" to call-by-value. If you call the function ref_demo() - like we do in the green block further down - we can check with the id() function what happens to x. We can see that in the main scope, x has the identity 41902552. In the first print statement of the ref_demo() function, the x from the main scope is used, because we can see that we get the same identity. After we have assigned the value 42 to x, x gets a new identity 41903752, i.e. a separate memory location from the global x. So, when we are back in the main scope x has still the original value 9.

This means that Python initially behaves like call-by-reference, but as soon as we are changing the value of such a variable, Python "switches" to call-by-value. If you call the function ref_demo() - like we do in the green block further down - we can check with the id() function what happens to x. We can see that in the main scope, x has the identity 41902552.

In the first print statement of the ref_demo() function, the x from the main scope is used, because we can see that we get the same identity. After we have assigned the value 42 to x, x gets a new identity 41903752, i.e. a separate memory location from the global x. So, when we are back in the main scope x has still the original value 9.

This means that Python initially behaves like call-by-reference, but as soon as we are changing the value of such a variable, Python "switches" to call-by-value.

>>> x = 9
>>> id(x)
41902552
>>> ref_demo(x)
x= 9  id= 41902552
x= 42  id= 41903752
>>> id(x)
41902552
>>>
Side effects A function is said to have a side effect if, in addition to producing a value, it modifies the caller's environment in other ways. For example, a function might modify a global or static variable, modify one of its arguments, raise an exception, write data to a display or file and so on. In many cases these side effects are wanted, i.e. they are part of the functions specification.

But in other cases, they are not wanted , they are hidden side effects. In this chapter we are only interested in the side effects, which change global variables, which have been passed as arguments to a function. Let's assume, we are passing a list to a function. We expect that the function is not changing this list. First let's have a look at a function which has no side effects.

As a new list is assigned to the parameter list in func1(), a new memory location is created for list and list becomes a local variable. In many cases these side effects are wanted, i.e. they are part of the functions specification. But in other cases, they are not wanted , they are hidden side effects. In this chapter we are only interested in the side effects, which change global variables, which have been passed as arguments to a function. Let's assume, we are passing a list to a function. We expect that the function is not changing this list. First let's have a look at a function which has no side effects. As a new list is assigned to the parameter list in func1(), a new memory location is created for list and list becomes a local variable. In many cases these side effects are wanted, i.e. they are part of the functions specification. But in other cases, they are not wanted , they are hidden side effects. In this chapter we are only interested in the side effects, which change global variables, which have been passed as arguments to a function. Let's assume, we are passing a list to a function. We expect that the function is not changing this list. First let's have a look at a function which has no side effects. As a new list is assigned to the parameter list in func1(), a new memory location is created for list and list becomes a local variable. Let's assume, we are passing a list to a function. We expect that the function is not changing this list. First let's have a look at a function which has no side effects. As a new list is assigned to the parameter list in func1(), a new memory location is created for list and list becomes a local variable.

>>> def func1(list):
...     print list
...     list = [47,11]
...     print list
... 
>>> fib = [0,1,1,2,3,5,8]
>>> func1(fib)
[0, 1, 1, 2, 3, 5, 8]
[47, 11]
>>> print fib
[0, 1, 1, 2, 3, 5, 8]
>>>
This changes drastically, if we include something in the list by using +=. To show this, we have a different function func2() in the following example:
>>> def func2(list):
...     print list
...     list += [47,11]
...     print list
... 
>>> fib = [0,1,1,2,3,5,8]
>>> func2(fib)
[0, 1, 1, 2, 3, 5, 8]
[0, 1, 1, 2, 3, 5, 8, 47, 11]
>>> print fib
[0, 1, 1, 2, 3, 5, 8, 47, 11]
>>>
The user of the function can prevent this by passing a copy to the function. In this case a shallow copy is sufficient:
>>> def func2(list):
...     print list
...     list += [47,11]
...     print list
... 
>>> fib = [0,1,1,2,3,5,8]
>>> func2(fib[:])
[0, 1, 1, 2, 3, 5, 8]
[0, 1, 1, 2, 3, 5, 8, 47, 11]
>>> print fib
[0, 1, 1, 2, 3, 5, 8]
>>>
Command Line Arguments It's possible to write Python scripts using command line arguments. If you call a Python script from a shell, the arguments are placed after the script name. The arguments are separated by spaces. Inside of the script these arguments are accessible through the list variable sys.argv. The name of the script is included in this list sys.argv[0]. sys.argv[1] contains the first parameter, sys.argv[2] the second and so on. The following script (arguments.py) prints all arguments: The following script (arguments.py) prints all arguments:
 # Module sys has to be imported:
import sys                

# Iteration over all arguments:
for eachArg in sys.argv:   
        print eachArg
Example call to this script:
python argumente.py python course for beginners
This call creates the following output:
argumente.py
python
course
for
beginners
Variable Length of Parameters We will introduce now functions, which can take an arbitrary number of arguments. Those who have some programming background in C or C++ know this from the varargs feature of these languages. The asterisk "*" is used in Python to define a variable number of arguments. The asterisk character has to precede a variable identifier in the parameter list.
>>> def varpafu(*x): print(x)
... 
>>> varpafu()
()
>>> varpafu(34,"Do you like Python?", "Of course")
(34, 'Do you like Python?', 'Of course')
>>>
We learn from the previous example that the arguments passed to the function call of varpafu() are collected in a tuple, which can be accessed as a "normal" variable x within the body of the function. If the function is called without any arguments, the value of x is an empty tuple. Sometimes, it's necessary to use positional parameters followed by an arbitrary number of parameters in a function definition. This is possible, but the positional parameters always have to precede the arbitrary parameters. In the following example, we have a positional parameter "city", - the main location, - which always have to be given, followed by an arbitrary number of other locations: Sometimes, it's necessary to use positional parameters followed by an arbitrary number of parameters in a function definition. This is possible, but the positional parameters always have to precede the arbitrary parameters. In the following example, we have a positional parameter "city", - the main location, - which always have to be given, followed by an arbitrary number of other locations: Sometimes, it's necessary to use positional parameters followed by an arbitrary number of parameters in a function definition. This is possible, but the positional parameters always have to precede the arbitrary parameters. In the following example, we have a positional parameter "city", - the main location, - which always have to be given, followed by an arbitrary number of other locations:
>>> def locations(city, *other_cities): print(city, other_cities)
... 
>>> locations("Paris")
('Paris', ())
>>> locations("Paris", "Strasbourg", "Lyon", "Dijon", "Bordeaux", "Marseille")
('Paris', ('Strasbourg', 'Lyon', 'Dijon', 'Bordeaux', 'Marseille'))
>>>
Exercise Write a function which calculates the arithmetic mean of a variable number of values.
Solution
Nov 23, 2019 | www.python-course.eu
def arithmetic_mean(x, *l):
    """ The function calculates the arithmetic mean of a non-empty
        arbitrary number of numbers """
    sum = x
    for i in l:
        sum += i

    return sum / (1.0 + len(l))
You might ask yourself, why we used both a positional parameter "x" and the variable parameter "*l" in our function definition. We could have only used *l to contain all our numbers. The idea is that we wanted to enforce that we always have a non-empty list of numbers. This is necessary to prevent a division by zero error, because the average of a an empty list of numbers is not defined.

In the following interactive Python session, we can learn how to use this function. We assume that the function arithmetic_mean is saved in a file called statistics.py.

>>> from statistics import arithmetic_mean
>>> arithmetic_mean(4,7,9)
6.666666666666667
>>> arithmetic_mean(4,7,9,45,-3.7,99)
26.71666666666667
This works fine, but there is a catch. What if somebody wants to call the function with a list, instead of a variable number of numbers, as we have shown above? We can see in the following that we raise an error, as most hopefully, you might expect:
>>> l = [4,7,9,45,-3.7,99]
>>> arithmetic_mean(l)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "statistics.py", line 8, in arithmetic_mean
    return sum / (1.0 + len(l))
TypeError: unsupported operand type(s) for /: 'list' and 'float'
The rescue is using another asterisk:
>>> arithmetic_mean(*l)
26.71666666666667
>>>

* in Function Calls A * can appear in function calls as well, as we have just seen in the previous exercise: The semantics is in this case "inverse" to a star in a function definition. An argument will be unpacked and not packed. In other words, the elements of the list or tuple are singularized:

>>> def f(x,y,z):
...     print(x,y,z)
... 
>>> p = (47,11,12)
>>> f(*p)
(47, 11, 12)
There is hardly any need to mention that this way of calling our function is more comfortable the following one:
>>> f(p[0],p[1],p[2])
(47, 11, 12)
>>>
Additionally to being less comfortable, the previous call (f(p[0],p[1],p[2])) doesn't work in the general case, i.e. lists of unknown lengths. "Unknown" mean, that the length is only known at runtime and not when we are writing the script. Arbitrary Keyword Parameters There is also a mechanism for an arbitrary number of keyword parameters. To do this, we use the double asterisk "**" notation:
>>> def f(**args):
...     print(args)
... 
>>> f()
{}
>>> f(de="German",en="English",fr="French")
{'fr': 'French', 'de': 'German', 'en': 'English'}
>>>
Double Asterisk in Function Calls The following example demonstrates the usage of ** in a function call:
>>> def f(a,b,x,y):
...     print(a,b,x,y)
...
>>> d = {'a':'append', 'b':'block','x':'extract','y':'yes'}
>>> f(**d)
('append', 'block', 'extract', 'yes')
and now in combination with *:
>>> t = (47,11)
>>> d = {'x':'extract','y':'yes'}
>>> f(*t, **d)
(47, 11, 'extract', 'yes')
>>>

[Aug 09, 2020] What is the Python equivalent of Perl's 'if exists' for hashes

Aug 09, 2020 | stackoverflow.com

flybonzai ,

I'm writing a script for work, and need to be able to create a hash of arrays that will check to see if a key exists in the hash (or dictionary), and if it does I will roll up some values from the new line into the existing hash values. Here is my code in Perl, what would be the translation in Python?

if (exists($rollUpHash{$hashKey}))
        {
          say("Same key found, summing up!")
          $rollUpHash{$hashKey}[14] += $lineFields[14];
          $rollUpHash{$hashKey}[15] += $lineFields[15];
          $rollUpHash{$hashKey}[16] += $lineFields[16];
          $rollUpHash{$hashKey}[17] += $lineFields[17];
          $rollUpHash{$hashKey}[24] += $lineFields[24];
          push @{$rollUpHash{$hashKey}}, $sumDeduct_NonDeduct_ytd;
          # print %rollUpHash;
        }
      else
        {
          $rollUpHash{$hashKey} = \@lineFields;
        }

ikegami ,

You might also want to look at a collections.defaultdict docs.python.org/2/library/Padraic Cunningham Jul 22 '15 at 20:35

> ,

If you're just checking if the key exists, you can do if "key" in your_dictionary

Edit:

To handle the unintended second part of your question, about adding the new value to the array, you can do something like this

# -1 will give you the last item in the list every time
for key, value in nums.iteritems():
    nums[key].append(value[-1]+value[-1])

[Aug 06, 2020] The Pros and Cons of Using Jupyter Notebooks as Your Editor for Data Science Work - by Steffen Sjursen - Better Programming - Medium

Notable quotes:
"... a lot. ..."
"... that one place. ..."
Aug 06, 2020 | medium.com

The Pros and Cons of Using Jupyter Notebooks as Your Editor for Data Science Work TL;DR: PyCharm's probably better Steffen Sjursen Steffen Sjursen Follow Mar 1 · 5 min read

Image for post Image for post
Image by Gerd Altmann from Pixabay

Jupyter notebooks have three particularly strong benefits:

When prototyping, the cell-based approach of Jupyter notebooks is great. But you quickly end up programming several steps -- instead of looking at object-oriented programming.

Downsides of Jupyter notebooks

When we're writing code in cells instead of functions/classes/objects, you quickly end up with duplicate code that does the same thing, which is very hard to maintain.

Don't get the support from a powerful IDE.

Consequences of duplicate code:

There's also a tricky problem related to plotting. How are you sharing plots outside of the data science team? At first, Jupyter Notebook is a great way of sharing plots -- just share the notebook! But how do you ensure the data there's fresh? Easy, just have them run the notebook.

But in large organizations, you might run into a lot of issues as you don't want too many users having direct access to the underlying data (for GDPR issues or otherwise). In practice, in a workplace, we've noticed plots from Jupyter typically get shared by copy/pasting into PowerPoint. It's highly ineffective to have your data scientists do copy/paste cycles whenever the data changes.

An example

Let's look at an example notebook from Kaggle.

Image for post

This was very easy to get started on. I just copied some cells from the introductory examples and then explored on my own. But here we also see the downside -- access-credentials management is now duplicated across all of the notebooks. What if they change? Then, every notebook needs to be changed as well.

The dataset is interesting. But there's no canonical one -- so if you want to reuse it, you're copying the SQL statement.

With poor discipline, you can also move into weird versioning issues where you start accumulating multiple Jupyter notebooks that no one remembers.

Benefits from an IDE

As alternatives to Jupyter Notebook, you could checkout Spyder and PyCharm.

Spyder has an interesting feature where it's very good at reloading code while keeping the state of your current session. Thus, Spyder is great if you're trying to do most of your transformations in pandas or some other tool on your local machine. It also comes with Anaconda, if that's your Python distribution of choice.

PyCharm is really good at building proper Python code and getting replicable results. There's a lot of functionality for greater productivity. Both Spyder and PyCharm enable cell-based processing with #%% comments in your code, so we can still prototype code in the same way as in Jupyter Notebook.

Image for post

For one of our previous clients, we wanted to improve code quality but were not allowed to access data on any local machines. So we spent the effort to spin up VMs with PyCharm to get access to data in a secure manner. It paid off quickly -- development speed and code quality increased by a factor of a lot. And code made it into production a lot faster as well.

Getting machine learning into production

Something to think about is where computations are run. For code that's easy to put into Docker, deploying that to any cloud solution is easy. For notebooks, there are also good options, though you're more locked into specific solutions.

If you do want to look into Jupyter notebooks, it's definitely worth looking into Amazon SageMaker and/or Kubeflow . These solutions enable easier deployment to production for code in notebooks.

We've had a lot of success with the following approach:

  • Use PyCharm (which has improved code quality by a far bit)
  • Every transformation of data needs to exist in exactly one place in our repository. (So any issues with that transformation needs to be fixed in that one place. )
  • Every transformation needs to be in production (so available as a table/file/output), so other data scientists can reuse it in their models. And if that transformation improves, all subsequent pipelines are automatically improved as well.

Conclusion

Should you remove notebooks completely? Notebooks have a lot of pros. It depends on where you are and what your main concerns are

  • If all of the machine learning is already on the cloud and only needs some light scripting -- notebooks are probably the easiest way there
  • Be careful with a heavy reliance on notebooks when the data-engineering team is short staffed or when the data scientist team is immature -- as this is when you can build up an unmanageable amount of bad habits and technical debt in a short amount of time
  • If your problems are fairly unique and require a lot of self-developed code, Jupyter notebooks will grow in size and complexity in ways that'll be hard to maintain
  • The larger the team, the more concerns we have about collaborative coding and the reuse of results between team members, and we should be moving away from Jupyter
  • If you have cross-functional teams, with both software engineers and data scientists, getting the most out of version control and object-oriented programming is easier. If you have a cross-functional team, you'll probably get more benefits by moving away from Jupyter.

[Aug 05, 2020] The Pros and Cons of Using Jupyter Notebooks as Your Editor for Data Science Work - by Steffen Sjursen - Better Programming - Medium

Aug 05, 2020 | medium.com

The Pros and Cons of Using Jupyter Notebooks as Your Editor for Data Science Work TL;DR: PyCharm's probably better Steffen Sjursen Steffen Sjursen Follow Mar 1 · 5 min read

Image for post Image for post
Image by Gerd Altmann from Pixabay

Jupyter notebooks have three particularly strong benefits:

When prototyping, the cell-based approach of Jupyter notebooks is great. But you quickly end up programming several steps -- instead of looking at object-oriented programming.

Downsides of Jupyter notebooks

When we're writing code in cells instead of functions/classes/objects, you quickly end up with duplicate code that does the same thing, which is very hard to maintain.

Don't get the support from a powerful IDE.

Consequences of duplicate code:

There's also a tricky problem related to plotting. How are you sharing plots outside of the data science team? At first, Jupyter Notebook is a great way of sharing plots -- just share the notebook! But how do you ensure the data there's fresh? Easy, just have them run the notebook.

But in large organizations, you might run into a lot of issues as you don't want too many users having direct access to the underlying data (for GDPR issues or otherwise). In practice, in a workplace, we've noticed plots from Jupyter typically get shared by copy/pasting into PowerPoint. It's highly ineffective to have your data scientists do copy/paste cycles whenever the data changes.

An example

Let's look at an example notebook from Kaggle.

Image for post

This was very easy to get started on. I just copied some cells from the introductory examples and then explored on my own. But here we also see the downside -- access-credentials management is now duplicated across all of the notebooks. What if they change? Then, every notebook needs to be changed as well.

The dataset is interesting. But there's no canonical one -- so if you want to reuse it, you're copying the SQL statement.

With poor discipline, you can also move into weird versioning issues where you start accumulating multiple Jupyter notebooks that no one remembers.

Benefits from an IDE

As alternatives to Jupyter Notebook, you could checkout Spyder and PyCharm.

Spyder has an interesting feature where it's very good at reloading code while keeping the state of your current session. Thus, Spyder is great if you're trying to do most of your transformations in pandas or some other tool on your local machine. It also comes with Anaconda, if that's your Python distribution of choice.

PyCharm is really good at building proper Python code and getting replicable results. There's a lot of functionality for greater productivity. Both Spyder and PyCharm enable cell-based processing with #%% comments in your code, so we can still prototype code in the same way as in Jupyter Notebook.

Image for post

For one of our previous clients, we wanted to improve code quality but were not allowed to access data on any local machines. So we spent the effort to spin up VMs with PyCharm to get access to data in a secure manner. It paid off quickly -- development speed and code quality increased by a factor of a lot. And code made it into production a lot faster as well.

Getting machine learning into production

Something to think about is where computations are run. For code that's easy to put into Docker, deploying that to any cloud solution is easy. For notebooks, there are also good options, though you're more locked into specific solutions.

If you do want to look into Jupyter notebooks, it's definitely worth looking into Amazon SageMaker and/or Kubeflow . These solutions enable easier deployment to production for code in notebooks.

We've had a lot of success with the following approach:

  • Use PyCharm (which has improved code quality by a far bit)
  • Every transformation of data needs to exist in exactly one place in our repository. (So any issues with that transformation needs to be fixed in that one place. )
  • Every transformation needs to be in production (so available as a table/file/output), so other data scientists can reuse it in their models. And if that transformation improves, all subsequent pipelines are automatically improved as well.

Conclusion

Should you remove notebooks completely? Notebooks have a lot of pros. It depends on where you are and what your main concerns are

  • If all of the machine learning is already on the cloud and only needs some light scripting -- notebooks are probably the easiest way there
  • Be careful with a heavy reliance on notebooks when the data-engineering team is short staffed or when the data scientist team is immature -- as this is when you can build up an unmanageable amount of bad habits and technical debt in a short amount of time
  • If your problems are fairly unique and require a lot of self-developed code, Jupyter notebooks will grow in size and complexity in ways that'll be hard to maintain
  • The larger the team, the more concerns we have about collaborative coding and the reuse of results between team members, and we should be moving away from Jupyter
  • If you have cross-functional teams, with both software engineers and data scientists, getting the most out of version control and object-oriented programming is easier. If you have a cross-functional team, you'll probably get more benefits by moving away from Jupyter.

[Aug 05, 2020] How to create Jupyter Notebook in PyCharm - Softhints

Aug 05, 2020 | blog.softhints.com

Go to the profile of John D K Published 2 years ago 2 min read By John D K

PyCharm support working with Jupyter Notebooks local and remote connection. You can create new Jupyter Notebook by right click of the mouse and selecting:

Selection_052

Then you can add new cells and enter code or Markdown. Once the cell is created and fill with code you can executed:

Selection_049-1

Table of Contents Server started from PyCharm

Starting PyCharm Jupyter Notebook server:

Selection_056

If you want to check details:

Selection_053

Connect to local server from PyCharm

Starting PyCharm Jupyter Notebook server:

Currently running servers:
http://localhost:8888/?token=56b65986e4771f5fcc4345ba39417c4f0677b3fa5dbdvd46 :: /home/user/test

Note : If you don't have access to terminal or you don't want to use it. Then you can get the address and the token by:

Selection_055 Jupyter Notebook Python PyCharm

[Jul 29, 2020] statistics -- Mathematical statistics functions -- Python 3.8.5 documentation

Jul 29, 2020 | docs.python.org

statistics -- Mathematical statistics functions

New in version 3.4.

Source code: Lib/statistics.py


This module provides functions for calculating mathematical statistics of numeric ( Real -valued) data.

The module is not intended to be a competitor to third-party libraries such as NumPy , SciPy , or proprietary full-featured statistics packages aimed at professional statisticians such as Minitab, SAS and Matlab. It is aimed at the level of graphing and scientific calculators.

Unless explicitly noted, these functions support int , float , Decimal and Fraction . Behaviour with other types (whether in the numeric tower or not) is currently unsupported. Collections with a mix of types are also undefined and implementation-dependent. If your input data consists of mixed types, you may be able to use map() to ensure a consistent result, for example: map(float, input_data) .

Averages and measures of central location

These functions calculate an average or typical value from a population or sample.

mean()

Arithmetic mean ("average") of data.

fmean()

Fast, floating point arithmetic mean.

geometric_mean()

Geometric mean of data.

harmonic_mean()

Harmonic mean of data.

median()

Median (middle value) of data.

median_low()

Low median of data.

median_high()

High median of data.

median_grouped()

Median, or 50th percentile, of grouped data.

mode()

Single mode (most common value) of discrete or nominal data.

multimode()

List of modes (most common values) of discrete or nomimal data.

quantiles()

Divide data into intervals with equal probability.

Measures of spread

These functions calculate a measure of how much the population or sample tends to deviate from the typical or average values.

pstdev()

Population standard deviation of data.

pvariance()

Population variance of data.

stdev()

Sample standard deviation of data.

variance()

Sample variance of data.

Function details

Note: The functions do not require the data given to them to be sorted. However, for reading convenience, most of the examples show sorted sequences.

statistics. mean ( data )

Return the sample arithmetic mean of data which can be a sequence or iterable.

The arithmetic mean is the sum of the data divided by the number of data points. It is commonly called "the average", although it is only one of many different mathematical averages. It is a measure of the central location of the data.

If data is empty, StatisticsError will be raised.

Some examples of use:

>>> mean([1, 2, 3, 4, 4])
2.8
>>> mean([-1.0, 2.5, 3.25, 5.75])
2.625

>>> from fractions import Fraction as F
>>> mean([F(3, 7), F(1, 21), F(5, 3), F(1, 3)])
Fraction(13, 21)

>>> from decimal import Decimal as D
>>> mean([D("0.5"), D("0.75"), D("0.625"), D("0.375")])
Decimal('0.5625')

Note

The mean is strongly affected by outliers and is not a robust estimator for central location: the mean is not necessarily a typical example of the data points. For more robust measures of central location, see median() and mode() .

The sample mean gives an unbiased estimate of the true population mean, so that when taken on average over all the possible samples, mean(sample) converges on the true mean of the entire population. If data represents the entire population rather than a sample, then mean(data) is equivalent to calculating the true population mean μ.

statistics. fmean ( data )

Convert data to floats and compute the arithmetic mean.

This runs faster than the mean() function and it always returns a float . The data may be a sequence or iterable. If the input dataset is empty, raises a StatisticsError .

>>> fmean([3.5, 4.0, 5.25])
4.25

New in version 3.8.

statistics. geometric_mean ( data )

Convert data to floats and compute the geometric mean.

The geometric mean indicates the central tendency or typical value of the data using the product of the values (as opposed to the arithmetic mean which uses their sum).

Raises a StatisticsError if the input dataset is empty, if it contains a zero, or if it contains a negative value. The data may be a sequence or iterable.

No special efforts are made to achieve exact results. (However, this may change in the future.)

>>> round(geometric_mean([54, 24, 36]), 1)
36.0

New in version 3.8.

statistics. harmonic_mean ( data )

Return the harmonic mean of data , a sequence or iterable of real-valued numbers.

The harmonic mean, sometimes called the subcontrary mean, is the reciprocal of the arithmetic mean() of the reciprocals of the data. For example, the harmonic mean of three values a , b and c will be equivalent to 3/(1/a + 1/b + 1/c) . If one of the values is zero, the result will be zero.

The harmonic mean is a type of average, a measure of the central location of the data. It is often appropriate when averaging rates or ratios, for example speeds.

Suppose a car travels 10 km at 40 km/hr, then another 10 km at 60 km/hr. What is the average speed?

>>> harmonic_mean([40, 60])
48.0

Suppose an investor purchases an equal value of shares in each of three companies, with P/E (price/earning) ratios of 2.5, 3 and 10. What is the average P/E ratio for the investor's portfolio?

>>> harmonic_mean([2.5, 3, 10])  # For an equal investment portfolio.
3.6

StatisticsError is raised if data is empty, or any element is less than zero.

The current algorithm has an early-out when it encounters a zero in the input. This means that the subsequent inputs are not tested for validity. (This behavior may change in the future.)

New in version 3.6.

statistics. median ( data )

Return the median (middle value) of numeric data, using the common "mean of middle two" method. If data is empty, StatisticsError is raised. data can be a sequence or iterable.

The median is a robust measure of central location and is less affected by the presence of outliers. When the number of data points is odd, the middle data point is returned:

>>> median([1, 3, 5])
3

When the number of data points is even, the median is interpolated by taking the average of the two middle values:

>>> median([1, 3, 5, 7])
4.0

This is suited for when your data is discrete, and you don't mind that the median may not be an actual data point.

If the data is ordinal (supports order operations) but not numeric (doesn't support addition), consider using median_low() or median_high() instead.

statistics. median_low ( data )

Return the low median of numeric data. If data is empty, StatisticsError is raised. data can be a sequence or iterable.

The low median is always a member of the data set. When the number of data points is odd, the middle value is returned. When it is even, the smaller of the two middle values is returned.

>>> median_low([1, 3, 5])
3
>>> median_low([1, 3, 5, 7])
3

Use the low median when your data are discrete and you prefer the median to be an actual data point rather than interpolated.

statistics. median_high ( data )

Return the high median of data. If data is empty, StatisticsError is raised. data can be a sequence or iterable.

The high median is always a member of the data set. When the number of data points is odd, the middle value is returned. When it is even, the larger of the two middle values is returned.

>>> median_high([1, 3, 5])
3
>>> median_high([1, 3, 5, 7])
5

Use the high median when your data are discrete and you prefer the median to be an actual data point rather than interpolated.

statistics. median_grouped ( data , interval=1 )

Return the median of grouped continuous data, calculated as the 50th percentile, using interpolation. If data is empty, StatisticsError is raised. data can be a sequence or iterable.

>>> median_grouped([52, 52, 53, 54])
52.5

In the following example, the data are rounded, so that each value represents the midpoint of data classes, e.g. 1 is the midpoint of the class 0.5–1.5, 2 is the midpoint of 1.5–2.5, 3 is the midpoint of 2.5–3.5, etc. With the data given, the middle value falls somewhere in the class 3.5–4.5, and interpolation is used to estimate it:

>>> median_grouped([1, 2, 2, 3, 4, 4, 4, 4, 4, 5])
3.7

Optional argument interval represents the class interval, and defaults to 1. Changing the class interval naturally will change the interpolation:

>>> median_grouped([1, 3, 3, 5, 7], interval=1)
3.25
>>> median_grouped([1, 3, 3, 5, 7], interval=2)
3.5

This function does not check whether the data points are at least interval apart.

CPython implementation detail: Under some circumstances, median_grouped() may coerce data points to floats. This behaviour is likely to change in the future.

See also

  • "Statistics for the Behavioral Sciences", Frederick J Gravetter and Larry B Wallnau (8th Edition).

  • The SSMEDIAN function in the Gnome Gnumeric spreadsheet, including this discussion .

statistics. mode ( data )

Return the single most common data point from discrete or nominal data . The mode (when it exists) is the most typical value and serves as a measure of central location.

If there are multiple modes with the same frequency, returns the first one encountered in the data . If the smallest or largest of those is desired instead, use min(multimode(data)) or max(multimode(data)) . If the input data is empty, StatisticsError is raised.

mode assumes discrete data and returns a single value. This is the standard treatment of the mode as commonly taught in schools:

>>> mode([1, 1, 2, 3, 3, 3, 3, 4])
3

The mode is unique in that it is the only statistic in this package that also applies to nominal (non-numeric) data:

>>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
'red'

Changed in version 3.8: Now handles multimodal datasets by returning the first mode encountered. Formerly, it raised StatisticsError when more than one mode was found.

statistics. multimode ( data )

Return a list of the most frequently occurring values in the order they were first encountered in the data . Will return more than one result if there are multiple modes or an empty list if the data is empty:

>>> multimode('aabbbbccddddeeffffgg')
['b', 'd', 'f']
>>> multimode('')
[]

New in version 3.8.

statistics. pstdev ( data , mu=None )

Return the population standard deviation (the square root of the population variance). See pvariance() for arguments and other details.

>>> pstdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
0.986893273527251
statistics. pvariance ( data , mu=None )

Return the population variance of data , a non-empty sequence or iterable of real-valued numbers. Variance, or second moment about the mean, is a measure of the variability (spread or dispersion) of data. A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.

If the optional second argument mu is given, it is typically the mean of the data . It can also be used to compute the second moment around a point that is not the mean. If it is missing or None (the default), the arithmetic mean is automatically calculated.

Use this function to calculate the variance from the entire population. To estimate the variance from a sample, the variance() function is usually a better choice.

Raises StatisticsError if data is empty.

Examples:

>>> data = [0.0, 0.25, 0.25, 1.25, 1.5, 1.75, 2.75, 3.25]
>>> pvariance(data)
1.25

If you have already calculated the mean of your data, you can pass it as the optional second argument mu to avoid recalculation:

>>> mu = mean(data)
>>> pvariance(data, mu)
1.25

Decimals and Fractions are supported:

>>> from decimal import Decimal as D
>>> pvariance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
Decimal('24.815')

>>> from fractions import Fraction as F
>>> pvariance([F(1, 4), F(5, 4), F(1, 2)])
Fraction(13, 72)

Note

When called with the entire population, this gives the population variance σ². When called on a sample instead, this is the biased sample variance s², also known as variance with N degrees of freedom.

If you somehow know the true population mean μ, you may use this function to calculate the variance of a sample, giving the known population mean as the second argument. Provided the data points are a random sample of the population, the result will be an unbiased estimate of the population variance.

statistics. stdev ( data , xbar=None )

Return the sample standard deviation (the square root of the sample variance). See variance() for arguments and other details.

>>> stdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
1.0810874155219827
statistics. variance ( data , xbar=None )

Return the sample variance of data , an iterable of at least two real-valued numbers. Variance, or second moment about the mean, is a measure of the variability (spread or dispersion) of data. A large variance indicates that the data is spread out; a small variance indicates it is clustered closely around the mean.

If the optional second argument xbar is given, it should be the mean of data . If it is missing or None (the default), the mean is automatically calculated.

Use this function when your data is a sample from a population. To calculate the variance from the entire population, see pvariance() .

Raises StatisticsError if data has fewer than two values.

Examples:

>>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]
>>> variance(data)
1.3720238095238095

If you have already calculated the mean of your data, you can pass it as the optional second argument xbar to avoid recalculation:

>>> m = mean(data)
>>> variance(data, m)
1.3720238095238095

This function does not attempt to verify that you have passed the actual mean as xbar . Using arbitrary values for xbar can lead to invalid or impossible results.

Decimal and Fraction values are supported:

>>> from decimal import Decimal as D
>>> variance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
Decimal('31.01875')

>>> from fractions import Fraction as F
>>> variance([F(1, 6), F(1, 2), F(5, 3)])
Fraction(67, 108)

Note

This is the sample variance s² with Bessel's correction, also known as variance with N-1 degrees of freedom. Provided that the data points are representative (e.g. independent and identically distributed), the result should be an unbiased estimate of the true population variance.

If you somehow know the actual population mean μ you should pass it to the pvariance() function as the mu parameter to get the variance of a sample.

statistics. quantiles ( data , * , n=4 , method='exclusive' )

Divide data into n continuous intervals with equal probability. Returns a list of n - 1 cut points separating the intervals.

Set n to 4 for quartiles (the default). Set n to 10 for deciles. Set n to 100 for percentiles which gives the 99 cuts points that separate data into 100 equal sized groups. Raises StatisticsError if n is not least 1.

The data can be any iterable containing sample data. For meaningful results, the number of data points in data should be larger than n . Raises StatisticsError if there are not at least two data points.

The cut points are linearly interpolated from the two nearest data points. For example, if a cut point falls one-third of the distance between two sample values, 100 and 112 , the cut-point will evaluate to 104 .

The method for computing quantiles can be varied depending on whether the data includes or excludes the lowest and highest possible values from the population.

The default method is "exclusive" and is used for data sampled from a population that can have more extreme values than found in the samples. The portion of the population falling below the i-th of m sorted data points is computed as i / (m + 1) . Given nine sample values, the method sorts them and assigns the following percentiles: 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.

Setting the method to "inclusive" is used for describing population data or for samples that are known to include the most extreme values from the population. The minimum value in data is treated as the 0th percentile and the maximum value is treated as the 100th percentile. The portion of the population falling below the i-th of m sorted data points is computed as (i - 1) / (m - 1) . Given 11 sample values, the method sorts them and assigns the following percentiles: 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.

# Decile cut points for empirically sampled data
>>> data = [105, 129, 87, 86, 111, 111, 89, 81, 108, 92, 110,
...         100, 75, 105, 103, 109, 76, 119, 99, 91, 103, 129,
...         106, 101, 84, 111, 74, 87, 86, 103, 103, 106, 86,
...         111, 75, 87, 102, 121, 111, 88, 89, 101, 106, 95,
...         103, 107, 101, 81, 109, 104]
>>> [round(q, 1) for q in quantiles(data, n=10)]
[81.0, 86.2, 89.0, 99.4, 102.5, 103.6, 106.0, 109.8, 111.0]

New in version 3.8.

Exceptions

A single exception is defined:

exception statistics. StatisticsError

Subclass of ValueError for statistics-related exceptions.

NormalDist objects

NormalDist is a tool for creating and manipulating normal distributions of a random variable . It is a class that treats the mean and standard deviation of data measurements as a single entity.

Normal distributions arise from the Central Limit Theorem and have a wide range of applications in statistics.

class statistics. NormalDist ( mu=0.0 , sigma=1.0 )

Returns a new NormalDist object where mu represents the arithmetic mean and sigma represents the standard deviation .

If sigma is negative, raises StatisticsError .

mean

A read-only property for the arithmetic mean of a normal distribution.

median

A read-only property for the median of a normal distribution.

mode

A read-only property for the mode of a normal distribution.

stdev

A read-only property for the standard deviation of a normal distribution.

variance

A read-only property for the variance of a normal distribution. Equal to the square of the standard deviation.

classmethod from_samples ( data )

Makes a normal distribution instance with mu and sigma parameters estimated from the data using fmean() and stdev() .

The data can be any iterable and should consist of values that can be converted to type float . If data does not contain at least two elements, raises StatisticsError because it takes at least one point to estimate a central value and at least two points to estimate dispersion.

samples ( n , * , seed=None )

Generates n random samples for a given mean and standard deviation. Returns a list of float values.

If seed is given, creates a new instance of the underlying random number generator. This is useful for creating reproducible results, even in a multi-threading context.

pdf ( x )

Using a probability density function (pdf) , compute the relative likelihood that a random variable X will be near the given value x . Mathematically, it is the limit of the ratio P(x <= X < x+dx) / dx as dx approaches zero.

The relative likelihood is computed as the probability of a sample occurring in a narrow range divided by the width of the range (hence the word "density"). Since the likelihood is relative to other points, its value can be greater than 1.0 .

cdf ( x )

Using a cumulative distribution function (cdf) , compute the probability that a random variable X will be less than or equal to x . Mathematically, it is written P(X <= x) .

inv_cdf ( p )

Compute the inverse cumulative distribution function, also known as the quantile function or the percent-point function. Mathematically, it is written x : P(X <= x) = p .

Finds the value x of the random variable X such that the probability of the variable being less than or equal to that value equals the given probability p .

overlap ( other )

Measures the agreement between two normal probability distributions. Returns a value between 0.0 and 1.0 giving the overlapping area for the two probability density functions .

quantiles ( n=4 )

Divide the normal distribution into n continuous intervals with equal probability. Returns a list of (n - 1) cut points separating the intervals.

Set n to 4 for quartiles (the default). Set n to 10 for deciles. Set n to 100 for percentiles which gives the 99 cuts points that separate the normal distribution into 100 equal sized groups.

Instances of NormalDist support addition, subtraction, multiplication and division by a constant. These operations are used for translation and scaling. For example:

>>> temperature_february = NormalDist(5, 2.5)             # Celsius
>>> temperature_february * (9/5) + 32                     # Fahrenheit
NormalDist(mu=41.0, sigma=4.5)

Dividing a constant by an instance of NormalDist is not supported because the result wouldn't be normally distributed.

Since normal distributions arise from additive effects of independent variables, it is possible to add and subtract two independent normally distributed random variables represented as instances of NormalDist . For example:

>>> birth_weights = NormalDist.from_samples([2.5, 3.1, 2.1, 2.4, 2.7, 3.5])
>>> drug_effects = NormalDist(0.4, 0.15)
>>> combined = birth_weights + drug_effects
>>> round(combined.mean, 1)
3.1
>>> round(combined.stdev, 1)
0.5

New in version 3.8.

NormalDist Examples and Recipes

NormalDist readily solves classic probability problems.

For example, given historical data for SAT exams showing that scores are normally distributed with a mean of 1060 and a standard deviation of 195, determine the percentage of students with test scores between 1100 and 1200, after rounding to the nearest whole number:

>>> sat = NormalDist(1060, 195)
>>> fraction = sat.cdf(1200 + 0.5) - sat.cdf(1100 - 0.5)
>>> round(fraction * 100.0, 1)
18.4

Find the quartiles and deciles for the SAT scores:

>>> list(map(round, sat.quantiles()))
[928, 1060, 1192]
>>> list(map(round, sat.quantiles(n=10)))
[810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310]

To estimate the distribution for a model than isn't easy to solve analytically, NormalDist can generate input samples for a Monte Carlo simulation :

>>> def model(x, y, z):
...     return (3*x + 7*x*y - 5*y) / (11 * z)
...
>>> n = 100_000
>>> X = NormalDist(10, 2.5).samples(n, seed=3652260728)
>>> Y = NormalDist(15, 1.75).samples(n, seed=4582495471)
>>> Z = NormalDist(50, 1.25).samples(n, seed=6582483453)
>>> quantiles(map(model, X, Y, Z))       
[1.4591308524824727, 1.8035946855390597, 2.175091447274739]

Normal distributions can be used to approximate Binomial distributions when the sample size is large and when the probability of a successful trial is near 50%.

For example, an open source conference has 750 attendees and two rooms with a 500 person capacity. There is a talk about Python and another about Ruby. In previous conferences, 65% of the attendees preferred to listen to Python talks. Assuming the population preferences haven't changed, what is the probability that the Python room will stay within its capacity limits?

>>> n = 750             # Sample size
>>> p = 0.65            # Preference for Python
>>> q = 1.0 - p         # Preference for Ruby
>>> k = 500             # Room capacity

>>> # Approximation using the cumulative normal distribution
>>> from math import sqrt
>>> round(NormalDist(mu=n*p, sigma=sqrt(n*p*q)).cdf(k + 0.5), 4)
0.8402

>>> # Solution using the cumulative binomial distribution
>>> from math import comb, fsum
>>> round(fsum(comb(n, r) * p**r * q**(n-r) for r in range(k+1)), 4)
0.8402

>>> # Approximation using a simulation
>>> from random import seed, choices
>>> seed(8675309)
>>> def trial():
...     return choices(('Python', 'Ruby'), (p, q), k=n).count('Python')
>>> mean(trial() <= k for i in range(10_000))
0.8398

Normal distributions commonly arise in machine learning problems.

Wikipedia has a nice example of a Naive Bayesian Classifier . The challenge is to predict a person's gender from measurements of normally distributed features including height, weight, and foot size.

We're given a training dataset with measurements for eight people. The measurements are assumed to be normally distributed, so we summarize the data with NormalDist :

>>> height_male = NormalDist.from_samples([6, 5.92, 5.58, 5.92])
>>> height_female = NormalDist.from_samples([5, 5.5, 5.42, 5.75])
>>> weight_male = NormalDist.from_samples([180, 190, 170, 165])
>>> weight_female = NormalDist.from_samples([100, 150, 130, 150])
>>> foot_size_male = NormalDist.from_samples([12, 11, 12, 10])
>>> foot_size_female = NormalDist.from_samples([6, 8, 7, 9])

Next, we encounter a new person whose feature measurements are known but whose gender is unknown:

>>> ht = 6.0        # height
>>> wt = 130        # weight
>>> fs = 8          # foot size

Starting with a 50% prior probability of being male or female, we compute the posterior as the prior times the product of likelihoods for the feature measurements given the gender:

>>> prior_male = 0.5
>>> prior_female = 0.5
>>> posterior_male = (prior_male * height_male.pdf(ht) *
...                   weight_male.pdf(wt) * foot_size_male.pdf(fs))

>>> posterior_female = (prior_female * height_female.pdf(ht) *
...                     weight_female.pdf(wt) * foot_size_female.pdf(fs))

The final prediction goes to the largest posterior. This is known as the maximum a posteriori or MAP:

>>> 'male' if posterior_male > posterior_female else 'female'
'f

[Jul 28, 2020] Intro into Python functions

Jul 28, 2020 | my.safaribooksonline.com

FUNCTIONS

You're already familiar with the print() , input() , and len() functions from the previous chapters. Python provides several built-in functions like these, but you can also write your own functions. A function is like a miniprogram within a program.

To better understand how functions work, let's create one. Enter this program into the file editor and save it as helloFunc.py :

➊ def hello():
➋ print('Howdy!')
print('Howdy!!!')
print('Hello there.')

➌ hello()
hello()
hello()

You can view the execution of this program at https://autbor.com/hellofunc/ . The first line is a def statement ➊ , which defines a function named hello() . The code in the block that follows the def statement ➋ is the body of the function. This code is executed when the function is called, not when the function is first defined.

The hello() lines after the function ➌ are function calls. In code, a function call is just the function's name followed by parentheses, possibly with some number of arguments in between the parentheses. When the program execution reaches these calls, it will jump to the top line in the function and begin executing the code there. When it reaches the end of the function, the execution returns to the line that called the function and continues moving through the code as before.

Since this program calls hello() three times, the code in the hello() function is executed three times. When you run this program, the output looks like this:

Howdy!
Howdy!!!
Hello there.
Howdy!
Howdy!!!
Hello there.
Howdy!
Howdy!!!
Hello there.

A major purpose of functions is to group code that gets executed multiple times. Without a function defined, you would have to copy and paste this code each time, and the program would look like this:

print('Howdy!')
print('Howdy!!!')
print('Hello there.')
print('Howdy!')
print('Howdy!!!')
print('Hello there.')
print('Howdy!')
print('Howdy!!!')
print('Hello there.')

In general, you always want to avoid duplicating code because if you ever decide to update the code -- if, for example, you find a bug you need to fix -- you'll have to remember to change the code everywhere you copied it.

As you get more programming experience, you'll often find yourself deduplicating code, which means getting rid of duplicated or copy-and-pasted code. Deduplication makes your programs shorter, easier to read, and easier to update.

def Statements with Parameters

When you call the print() or len() function, you pass them values, called arguments , by typing them between the parentheses. You can also define your own functions that accept arguments. Type this example into the file editor and save it as helloFunc2.py :


def hello(name):
print('Hello, ' + name)➋


hello('Alice')
hello('Bob')

When you run this program, the output looks like this:

Hello, Alice
Hello, Bob

You can view the execution of this program at https://autbor.com/hellofunc2/ . The definition of the hello() function in this program has a parameter called name ➊ . Parameters are variables that contain arguments. When a function is called with arguments, the arguments are stored in the parameters. The first time the hello() function is called, it is passed the argument 'Alice' ➌ . The program execution enters the function, and the parameter name is automatically set to 'Alice' , which is what gets printed by the print() statement ➋ .

One special thing to note about parameters is that the value stored in a parameter is forgotten when the function returns. For example, if you added print(name) after hello('Bob') in the previous program, the program would give you a NameError because there is no variable named name . This variable is destroyed after the function call hello('Bob') returns, so print(name) would refer to a name variable that does not exist.

This is similar to how a program's variables are forgotten when the program terminates. I'll talk more about why that happens later in the chapter, when I discuss what a function's local scope is.

Define, Call, Pass, Argument, Parameter

The terms define , call , pass , argument , and parameter can be confusing. Let's look at a code example to review these terms:


def sayHello(name):
print('Hello, ' + name)

sayHello('Al')

To define a function is to create it, just like an assignment statement like spam = 42 creates the spam variable. The def statement defines the sayHello() function ➊ . The sayHello('Al') line ➋ calls the now-created function, sending the execution to the top of the function's code. This function call is also known as passing the string value 'Al' to the function. A value being passed to a function in a function call is an argument . The argument 'Al' is assigned to a local variable named name . Variables that have arguments assigned to them are parameters .

It's easy to mix up these terms, but keeping them straight will ensure that you know precisely what the text in this chapter means.

Return Values and return Statements

When you call the len() function and pass it an argument such as 'Hello' , the function call evaluates to the integer value 5 , which is the length of the string you passed it. In general, the value that a function call evaluates to is called the return value of the function.

When creating a function using the def statement, you can specify what the return value should be with a return statement. A return statement consists of the following:

When an expression is used with a return statement, the return value is what this expression evaluates to. For example, the following program defines a function that returns a different string depending on what number it is passed as an argument. Enter this code into the file editor and save it as magic8Ball.py :

➊ import random

➋ def getAnswer(answerNumber):
➌ if answerNumber == 1:
return 'Gold will go up'
elif answerNumber == 2:
return 'Gol will go down'
elif answerNumber == 3:
return 'Silver will go up'
elif answerNumber == 4:
return 'Silver will go down'
elif answerNumber == 5:
return 'Platinum will go upr'
elif answerNumber == 6:
return 'Platinum will go down'
elif answerNumber == 7:
return 'Palladium will go up'
elif answerNumber == 8:
return 'Palladium will go down '
elif answerNumber == 9:
return 'All precious metals will go up'

➍ r = random.randint(1, 9)
➎ fortune = getAnswer(r)
➏ print(fortune)

You can view the execution of this program at https://autbor.com/magic8ball/ . When this program starts, Python first imports the random module ➊ . Then the getAnswer() function is defined ➋ . Because the function is being defined (and not called), the execution skips over the code in it. Next, the random.randint() function is called with two arguments: 1 and 9 ➍ . It evaluates to a random integer between 1 and 9 (including 1 and 9 themselves), and this value is stored in a variable named r .

The getAnswer() function is called with r as the argument ➎ . The program execution moves to the top of the getAnswer() function ➌ , and the value r is stored in a parameter named answerNumber . Then, depending on the value in answerNumber , the function returns one of many possible string values. The program execution returns to the line at the bottom of the program that originally called getAnswer() ➎ . The returned string is assigned to a variable named fortune , which then gets passed to a print() call ➏ and is printed to the screen.

Note that since you can pass return values as an argument to another function call, you could shorten these three lines:

r = random.randint(1, 9)
fortune = getAnswer(r)
print(fortune)

to this single equivalent line:

print(getAnswer(random.randint(1, 9)))

Remember, expressions are composed of values and operators. A function call can be used in an expression because the call evaluates to its return value.

The None Value

In Python, there is a value called None , which represents the absence of a value. The None value is the only value of the NoneType data type. (Other programming languages might call this value null , nil , or undefined .) Just like the Boolean True and False values, None must be typed with a capital N .

This value-without-a-value can be helpful when you need to store something that won't be confused for a real value in a variable. One place where None is used is as the return value of print() . The print() function displays text on the screen, but it doesn't need to return anything in the same way len() or input() does. But since all function calls need to evaluate to a return value, print() returns None . To see this in action, enter the following into the interactive shell:

>>> spam = print('Hello!')
Hello!
>>> None == spam
True

Behind the scenes, Python adds return None to the end of any function definition with no return statement. This is similar to how a while or for loop implicitly ends with a continue statement. Also, if you use a return statement without a value (that is, just the return keyword by itself), then None is returned.

Keyword Arguments and the print() Function

Most arguments are identified by their position in the function call. For example, random.randint(1, 10) is different from random.randint(10, 1) . The function call random.randint(1, 10) will return a random integer between 1 and 10 because the first argument is the low end of the range and the second argument is the high end (while random.randint(10, 1) causes an error).

However, rather than through their position, keyword arguments are identified by the keyword put before them in the function call. Keyword arguments are often used for optional parameters . For example, the print() function has the optional parameters end and sep to specify what should be printed at the end of its arguments and between its arguments (separating them), respectively.

If you ran a program with the following code:

print('Hello')
print('World')

the output would look like this:

Hello
World

The two outputted strings appear on separate lines because the print() function automatically adds a newline character to the end of the string it is passed. However, you can set the end keyword argument to change the newline character to a different string. For example, if the code were this:

print('Hello', end='')
print('World')

the output would look like this:

HelloWorld

The output is printed on a single line because there is no longer a newline printed after 'Hello' . Instead, the blank string is printed. This is useful if you need to disable the newline that gets added to the end of every print() function call.

Similarly, when you pass multiple string values to print() , the function will automatically separate them with a single space. Enter the following into the interactive shell:

>>> print('cats', 'dogs', 'mice')
cats dogs mice

But you could replace the default separating string by passing the sep keyword argument a different string. Enter the following into the interactive shell:

>>> print('cats', 'dogs', 'mice', sep=',')
cats,dogs,mice

You can add keyword arguments to the functions you write as well, but first you'll have to learn about the list and dictionary data types in the next two chapters. For now, just know that some functions have optional keyword arguments that can be specified when the function is called.

The Call Stack

Imagine that you have a meandering conversation with someone. You talk about your friend Alice, which then reminds you of a story about your coworker Bob, but first you have to explain something about your cousin Carol. You finish you story about Carol and go back to talking about Bob, and when you finish your story about Bob, you go back to talking about Alice. But then you are reminded about your brother David, so you tell a story about him, and then get back to finishing your original story about Alice. Your conversation followed a stack -like structure, like in Figure 3-1 . The conversation is stack-like because the current topic is always at the top of the stack.

Figure 3-1: Your meandering conversation stack

Similar to our meandering conversation, calling a function doesn't send the execution on a one-way trip to the top of a function. Python will remember which line of code called the function so that the execution can return there when it encounters a return statement. If that original function called other functions, the execution would return to those function calls first, before returning from the original function call.

Open a file editor window and enter the following code, saving it as abcdCallStack.py :

def a():
print('a() starts')
➊ b()
➋ d()
print('a() returns')

def b():
print('b() starts')
➌ c()
print('b() returns')

def c():
➍ print('c() starts')
print('c() returns')

def d():
print('d() starts')
print('d() returns')

➎ a()

If you run this program, the output will look like this:

a() starts
b() starts
c() starts
c() returns
b() returns
d() starts
d() returns
a() returns

You can view the execution of this program at https://autbor.com/abcdcallstack/ . When a() is called ➎ , it calls b() ➊ , which in turn calls c() ➌ . The c() function doesn't call anything; it just displays c() starts ➍ and c() returns before returning to the line in b() that called it ➌ . Once execution returns to the code in b() that called c() , it returns to the line in a() that called b() ➊ . The execution continues to the next line in the b() function ➋ , which is a call to d() . Like the c() function, the d() function also doesn't call anything. It just displays d() starts and d() returns before returning to the line in b() that called it. Since b() contains no other code, the execution returns to the line in a() that called b() ➋ . The last line in a() displays a() returns before returning to the original a() call at the end of the program ➎ .

The call stack is how Python remembers where to return the execution after each function call. The call stack isn't stored in a variable in your program; rather, Python handles it behind the scenes. When your program calls a function, Python creates a frame object on the top of the call stack. Frame objects store the line number of the original function call so that Python can remember where to return. If another function call is made, Python puts another frame object on the call stack above the other one.

When a function call returns, Python removes a frame object from the top of the stack and moves the execution to the line number stored in it. Note that frame objects are always added and removed from the top of the stack and not from any other place. Figure 3-2 illustrates the state of the call stack in abcdCallStack.py as each function is called and returns.

image

Figure 3-2: The frame objects of the call stack as abcdCallStack.py calls and returns from functions

The top of the call stack is which function the execution is currently in. When the call stack is empty, the execution is on a line outside of all functions.

The call stack is a technical detail that you don't strictly need to know about to write programs. It's enough to understand that function calls return to the line number they were called from. However, understanding call stacks makes it easier to understand local and global scopes, described in the next section.

Local and Global Scope

Parameters and variables that are assigned in a called function are said to exist in that function's local scope . Variables that are assigned outside all functions are said to exist in the global scope . A variable that exists in a local scope is called a local variable , while a variable that exists in the global scope is called a global variable . A variable must be one or the other; it cannot be both local and global.

Think of a scope as a container for variables. When a scope is destroyed, all the values stored in the scope's variables are forgotten. There is only one global scope, and it is created when your program begins. When your program terminates, the global scope is destroyed, and all its variables are forgotten. Otherwise, the next time you ran a program, the variables would remember their values from the last time you ran it.

A local scope is created whenever a function is called. Any variables assigned in the function exist within the function's local scope. When the function returns, the local scope is destroyed, and these variables are forgotten. The next time you call the function, the local variables will not remember the values stored in them from the last time the function was called. Local variables are also stored in frame objects on the call stack.

Scopes matter for several reasons:

The reason Python has different scopes instead of just making everything a global variable is so that when variables are modified by the code in a particular call to a function, the function interacts with the rest of the program only through its parameters and the return value. This narrows down the number of lines of code that may be causing a bug. If your program contained nothing but global variables and had a bug because of a variable being set to a bad value, then it would be hard to track down where this bad value was set. It could have been set from anywhere in the program, and your program could be hundreds or thousands of lines long! But if the bug is caused by a local variable with a bad value, you know that only the code in that one function could have set it incorrectly.

While using global variables in small programs is fine, it is a bad habit to rely on global variables as your programs get larger and larger.

Local Variables Cannot Be Used in the Global Scope

Consider this program, which will cause an error when you run it:

def spam():
➊ eggs = 31337
spam()
print(eggs)

If you run this program, the output will look like this:

Traceback (most recent call last):
File "C:/test1.py", line 4, in <module>
print(eggs)
NameError: name 'eggs' is not defined

The error happens because the eggs variable exists only in the local scope created when spam() is called ➊ . Once the program execution returns from spam , that local scope is destroyed, and there is no longer a variable named eggs . So when your program tries to run print(eggs) , Python gives you an error saying that eggs is not defined. This makes sense if you think about it; when the program execution is in the global scope, no local scopes exist, so there can't be any local variables. This is why only global variables can be used in the global scope.

Local Scopes Cannot Use Variables in Other Local Scopes

A new local scope is created whenever a function is called, including when a function is called from another function. Consider this program:

def spam():
➊ eggs = 99
➋ bacon()
➌ print(eggs)

def bacon():
ham = 101
➍ eggs = 0

➎ spam()

You can view the execution of this program at https://autbor.com/otherlocalscopes/ . When the program starts, the spam() function is called ➎ , and a local scope is created. The local variable eggs ➊ is set to 99 . Then the bacon() function is called ➋ , and a second local scope is created. Multiple local scopes can exist at the same time. In this new local scope, the local variable ham is set to 101 , and a local variable eggs -- which is different from the one in spam() 's local scope -- is also created ➍ and set to 0 .

When bacon() returns, the local scope for that call is destroyed, including its eggs variable. The program execution continues in the spam() function to print the value of eggs ➌ . Since the local scope for the call to spam() still exists, the only eggs variable is the spam() function's eggs variable, which was set to 99 . This is what the program prints.

The upshot is that local variables in one function are completely separate from the local variables in another function.

Global Variables Can Be Read from a Local Scope

Consider the following program:

def spam():
print(eggs)
eggs = 42
spam()
print(eggs)

You can view the execution of this program at https://autbor.com/readglobal/ . Since there is no parameter named eggs or any code that assigns eggs a value in the spam() function, when eggs is used in spam() , Python considers it a reference to the global variable eggs . This is why 42 is printed when the previous program is run.

Local and Global Variables with the Same Name

Technically, it's perfectly acceptable to use the same variable name for a global variable and local variables in different scopes in Python. But, to simplify your life, avoid doing this. To see what happens, enter the following code into the file editor and save it as localGlobalSameName.py :

def spam():
➊ eggs = 'spam local'
print(eggs) # prints 'spam local'

def bacon():
➋ eggs = 'bacon local'
print(eggs) # prints 'bacon local'
spam()
print(eggs) # prints 'bacon local'

➌ eggs = 'global'
bacon()
print(eggs) # prints 'global'

When you run this program, it outputs the following:

bacon local
spam local
bacon local
global

You can view the execution of this program at https://autbor.com/localglobalsamename/ . There are actually three different variables in this program, but confusingly they are all named eggs . The variables are as follows:

➊ A variable named eggs that exists in a local scope when spam() is called.

➋ A variable named eggs that exists in a local scope when bacon() is called.

➌ A variable named eggs that exists in the global scope.

Since these three separate variables all have the same name, it can be confusing to keep track of which one is being used at any given time. This is why you should avoid using the same variable name in different scopes.

The global Statement

If you need to modify a global variable from within a function, use the global statement. If you have a line such as global eggs at the top of a function, it tells Python, "In this function, eggs refers to the global variable, so don't create a local variable with this name." For example, enter the following code into the file editor and save it as globalStatement.py :

def spam():
➊ global eggs
➋ eggs = 'spam'

eggs = 'global'
spam()
print(eggs)

When you run this program, the final print() call will output this:

spam

You can view the execution of this program at https://autbor.com/globalstatement/ . Because eggs is declared global at the top of spam() ➊ , when eggs is set to 'spam' ➋ , this assignment is done to the globally scoped eggs . No local eggs variable is created.

There are four rules to tell whether a variable is in a local scope or global scope:

To get a better feel for these rules, here's an example program. Enter the following code into the file editor and save it as sameNameLocalGlobal.py :

def spam():
➊ global eggs
eggs = 'spam' # this is the global

def bacon():
➋ eggs = 'bacon' # this is a local

def ham():
➌ print(eggs) # this is the global

eggs = 42 # this is the global
spam()
print(eggs)

In the spam() function, eggs is the global eggs variable because there's a global statement for eggs at the beginning of the function ➊ . In bacon() , eggs is a local variable because there's an assignment statement for it in that function ➋ . In ham() ➌ , eggs is the global variable because there is no assignment statement or global statement for it in that function. If you run sameNameLocalGlobal.py , the output will look like this:

spam

You can view the execution of this program at https://autbor.com/sameNameLocalGlobal/ . In a function, a variable will either always be global or always be local. The code in a function can't use a local variable named eggs and then use the global eggs variable later in that same function.

NOTE

If you ever want to modify the value stored in a global variable from in a function, you must use a global statement on that variable.

If you try to use a local variable in a function before you assign a value to it, as in the following program, Python will give you an error. To see this, enter the following into the file editor and save it as sameNameError.py :

def spam():
print(eggs) # ERROR!
➊ eggs = 'spam local'

➋ eggs = 'global'
spam()

If you run the previous program, it produces an error message.

Traceback (most recent call last):
File "C:/sameNameError.py", line 6, in <module>
spam()
File "C:/sameNameError.py", line 2, in spam
print(eggs) # ERROR!
UnboundLocalError: local variable 'eggs' referenced before assignment

You can view the execution of this program at https://autbor.com/sameNameError/ . This error happens because Python sees that there is an assignment statement for eggs in the spam() function ➊ and, therefore, considers eggs to be local. But because print(eggs) is executed before eggs is assigned anything, the local variable eggs doesn't exist. Python will not fall back to using the global eggs variable ➋ .

FUNCTIONS AS "BLACK BOXES"

Often, all you need to know about a function are its inputs (the parameters) and output value; you don't always have to burden yourself with how the function's code actually works. When you think about functions in this high-level way, it's common to say that you're treating a function as a "black box."

This idea is fundamental to modern programming. Later chapters in this book will show you several modules with functions that were written by other people. While you can take a peek at the source code if you're curious, you don't need to know how these functions work in order to use them. And because writing functions without global variables is encouraged, you usually don't have to worry about the function's code interacting with the rest of your program. Exception Handling

Right now, getting an error, or exception , in your Python program means the entire program will crash. You don't want this to happen in real-world programs. Instead, you want the program to detect errors, handle them, and then continue to run.

For example, consider the following program, which has a divide-by-zero error. Open a file editor window and enter the following code, saving it as zeroDivide.py :

def spam(divideBy):
return 42 / divideBy

print(spam(2))
print(spam(12))
print(spam(0))
print(spam(1))

We've defined a function called spam , given it a parameter, and then printed the value of that function with various parameters to see what happens. This is the output you get when you run the previous code:

21.0
3.5
Traceback (most recent call last):
File "C:/zeroDivide.py", line 6, in <module>
print(spam(0))
File "C:/zeroDivide.py", line 2, in spam
return 42 / divideBy
ZeroDivisionError: division by zero

You can view the execution of this program at https://autbor.com/zerodivide/ . A ZeroDivisionError happens whenever you try to divide a number by zero. From the line number given in the error message, you know that the return statement in spam() is causing an error.

Errors can be handled with try and except statements. The code that could potentially have an error is put in a try clause. The program execution moves to the start of a following except clause if an error happens.

You can put the previous divide-by-zero code in a try clause and have an except clause contain code to handle what happens when this error occurs.

def spam(divideBy):
try:
return 42 / divideBy
except ZeroDivisionError:
print('Error: Invalid argument.')

print(spam(2))
print(spam(12))
print(spam(0))
print(spam(1))

When code in a try clause causes an error, the program execution immediately moves to the code in the except clause. After running that code, the execution continues as normal. The output of the previous program is as follows:

21.0
3.5
Error: Invalid argument.
None
42.0

You can view the execution of this program at https://autbor.com/tryexceptzerodivide/ . Note that any errors that occur in function calls in a try block will also be caught. Consider the following program, which instead has the spam() calls in the try block:

def spam(divideBy):
return 42 / divideBy

try:
print(spam(2))
print(spam(12))
print(spam(0))
print(spam(1))
except ZeroDivisionError:
print('Error: Invalid argument.')

When this program is run, the output looks like this:

21.0
3.5
Error: Invalid argument.

You can view the execution of this program at https://autbor.com/spamintry/ . The reason print(spam(1)) is never executed is because once the execution jumps to the code in the except clause, it does not return to the try clause. Instead, it just continues moving down the program as normal.

A Short Program: Zigzag

Let's use the programming concepts you've learned so far to create a small animation program. This program will create a back-and-forth, zigzag pattern until the user stops it by pressing the Mu editor's Stop button or by pressing CTRL-C . When you run this program, the output will look something like this:

********
********
********
********
********
********
********
********
********

Type the following source code into the file editor, and save the file as zigzag.py :

import time, sys
indent = 0 # How many spaces to indent.
indentIncreasing = True # Whether the indentation is increasing or not.

try:
while True: # The main program loop.
print(' ' * indent, end='')
print('********')
time.sleep(0.1) # Pause for 1/10 of a second.

if indentIncreasing:
# Increase the number of spaces:
indent = indent + 1
if indent == 20:
# Change direction:
indentIncreasing = False

else:
# Decrease the number of spaces:
indent = indent - 1
if indent == 0:
# Change direction:
indentIncreasing = True
except KeyboardInterrupt:
sys.exit()

Let's look at this code line by line, starting at the top.

import time, sys
indent = 0 # How many spaces to indent.
indentIncreasing = True # Whether the indentation is increasing or not.

First, we'll import the time and sys modules. Our program uses two variables: the indent variable keeps track of how many spaces of indentation are before the band of eight asterisks and indentIncreasing contains a Boolean value to determine if the amount of indentation is increasing or decreasing.

try:
while True: # The main program loop.
print(' ' * indent, end='')
print('********')
time.sleep(0.1) # Pause for 1/10 of a second.

Next, we place the rest of the program inside a try statement. When the user presses CTRL-C while a Python program is running, Python raises the KeyboardInterrupt exception. If there is no try - except statement to catch this exception, the program crashes with an ugly error message. However, for our program, we want it to cleanly handle the KeyboardInterrupt exception by calling sys.exit() . (The code for this is in the except statement at the end of the program.)

The while True: infinite loop will repeat the instructions in our program forever. This involves using ' ' * indent to print the correct amount of spaces of indentation. We don't want to automatically print a newline after these spaces, so we also pass end='' to the first print() call. A second print() call prints the band of asterisks. The time.sleep() function hasn't been covered yet, but suffice it to say that it introduces a one-tenth-second pause in our program at this point.

if indentIncreasing:
# Increase the number of spaces:
indent = indent + 1
if indent == 20:
indentIncreasing = False # Change direction.

Next, we want to adjust the amount of indentation for the next time we print asterisks. If indentIncreasing is True , then we want to add one to indent . But once indent reaches 20 , we want the indentation to decrease.

else:
# Decrease the number of spaces:
indent = indent - 1
if indent == 0:
indentIncreasing = True # Change direction.

Meanwhile, if indentIncreasing was False , we want to subtract one from indent . Once indent reaches 0 , we want the indentation to increase once again. Either way, the program execution will jump back to the start of the main program loop to print the asterisks again.

except KeyboardInterrupt:
sys.exit()

If the user presses CTRL-C at any point that the program execution is in the try block, the KeyboardInterrrupt exception is raised and handled by this except statement. The program execution moves inside the except block, which runs sys.exit() and quits the program. This way, even though the main program loop is an infinite loop, the user has a way to shut down the program.

Summary

Functions are the primary way to compartmentalize your code into logical groups. Since the variables in functions exist in their own local scopes, the code in one function cannot directly affect the values of variables in other functions. This limits what code could be changing the values of your variables, which can be helpful when it comes to debugging your code.

Functions are a great tool to help you organize your code. You can think of them as black boxes: they have inputs in the form of parameters and outputs in the form of return values, and the code in them doesn't affect variables in other functions.

In previous chapters, a single error could cause your programs to crash. In this chapter, you learned about try and except statements, which can run code when an error has been detected. This can make your programs more resilient to common error cases.

Practice Questions

1 . Why are functions advantageous to have in your programs?

2 . When does the code in a function execute: when the function is defined or when the function is called?

3 . What statement creates a function?

4 . What is the difference between a function and a function call?

5 . How many global scopes are there in a Python program? How many local scopes?

6 . What happens to variables in a local scope when the function call returns?

7 . What is a return value? Can a return value be part of an expression?

8 . If a function does not have a return statement, what is the return value of a call to that function?

9 . How can you force a variable in a function to refer to the global variable?

10 . What is the data type of None ?

11 . What does the import areallyourpetsnamederic statement do?

12 . If you had a function named bacon() in a module named spam , how would you call it after importing spam ?

13 . How can you prevent a program from crashing when it gets an error?

14 . What goes in the try clause? What goes in the except clause?

[Dec 26, 2019] What is the Python equivalent of Perl's 'if exists' for hashes?

Jan 01, 2015 | stackoverflow.com

Ask Question Asked 4 years, 5 months ago Active 4 years, 5 months ago Viewed 583 times The Streaming SQL Materialized View Engine powered by Timely Dataflow. View all job openings!


flybonzai ,

I'm writing a script for work, and need to be able to create a hash of arrays that will check to see if a key exists in the hash (or dictionary), and if it does I will roll up some values from the new line into the existing hash values. Here is my code in Perl, what would be the translation in Python?
if (exists($rollUpHash{$hashKey}))
        {
          say("Same key found, summing up!")
          $rollUpHash{$hashKey}[14] += $lineFields[14];
          $rollUpHash{$hashKey}[15] += $lineFields[15];
          $rollUpHash{$hashKey}[16] += $lineFields[16];
          $rollUpHash{$hashKey}[17] += $lineFields[17];
          $rollUpHash{$hashKey}[24] += $lineFields[24];
          push @{$rollUpHash{$hashKey}}, $sumDeduct_NonDeduct_ytd;
          # print %rollUpHash;
        }
      else
        {
          $rollUpHash{$hashKey} = \@lineFields;
        }

blasko , 2015-07-22 20:15:35

If you're just checking if the key exists, you can do if "key" in your_dictionary

Edit:

To handle the unintended second part of your question, about adding the new value to the array, you can do something like this

# -1 will give you the last item in the list every time
for key, value in nums.iteritems():
    nums[key].append(value[-1]+value[-1])

omri_saadon ,

You can use this as well
rollUpHash.get(key, None)

If the key exists then the function return the value of this key, else the function will return whatever you assigned as the default value (second parameter)

if rollUpHash.has_key(hashkey):
    print "Same key found, summing up!"
    rollUpHash[hashkey][14] += lineFields[14]
    ...
    ...
    rollUpHash[hashkey].append(sumDeduct_NonDeduct_ytd)
else:
    rollUpHash[hashkey] = lineFields

omri_saadon , 2015-07-23 16:31:30

So if I want to add the rest of the line as a list to my dictionary using key as the key, how would that look? – flybonzai Jul 22 '15 at 22:05

[Dec 26, 2019] Python Get Today's Current Date and Time - nixCraft

Dec 26, 2019 | www.cyberciti.biz

How to get current date and time in Python?

The syntax is:

time.strftime(format)
## 24 hour format ##
time.strftime("%H:%M:%S")
## 12 hour format ##
time.strftime("%I:%M:%S")

time.strftime(format) ## 24 hour format ## time.strftime("%H:%M:%S") ## 12 hour format ## time.strftime("%I:%M:%S")

Here is another example done using datetime module:

#!/usr/bin/python3
from datetime import date
 
 
today = date.today()
fdate = date.today().strftime('%d/%m/%Y')
 
print("Today's current date is -", today)
print("Date in dd/mm/YYYY format -", fdate)

#!/usr/bin/python3 from datetime import date today = date.today() fdate = date.today().strftime('%d/%m/%Y') print("Today's current date is -", today) print("Date in dd/mm/YYYY format -", fdate) Getting current time in python using time module

Let us see some examples of time module:

Examples

A simple program in python to get today's date and time:

#!/usr/bin/python
 
import time
print (time.strftime("%H:%M:%S"))
 
## 12 hour format ##
print (time.strftime("%I:%M:%S"))

#!/usr/bin/python import time print (time.strftime("%H:%M:%S")) ## 12 hour format ## print (time.strftime("%I:%M:%S"))

Sample outputs:

23:46:08
11:46:08

To print current date use:

#!/usr/bin/python
 
import time
## dd/mm/yyyy format
print (time.strftime("%d/%m/%Y"))

#!/usr/bin/python import time ## dd/mm/yyyy format print (time.strftime("%d/%m/%Y"))

Sample outputs:

04/10/2013
Getting locals date and time in Python
#!/usr/bin/python
# Purpose: Python Current Date Time Demo
# Author: nixCraft 
# --------------------------------------------
import time
 
now = time.strftime("%c")
## date and time representation
print "Current date & time " + time.strftime("%c")
 
## Only date representation
print "Current date "  + time.strftime("%x")
 
## Only time representation
print "Current time " + time.strftime("%X")
 
## Display current date and time from now variable 
print ("Current time %s"  % now )

#!/usr/bin/python # Purpose: Python Current Date Time Demo # Author: nixCraft # -------------------------------------------- import time now = time.strftime("%c") ## date and time representation print "Current date & time " + time.strftime("%c") ## Only date representation print "Current date " + time.strftime("%x") ## Only time representation print "Current time " + time.strftime("%X") ## Display current date and time from now variable print ("Current time %s" % now )

Sample outputs:

Current date & time Sat Oct  5 00:04:59 2013
Current date 10/05/13
Current time 00:04:59
Current time Sat Oct  5 00:04:59 2013
Format strings to get the current date and time in Python

The following directives can be embedded in the format string:

Directive Meaning
%a Weekday name.
%A Full weekday name.
%b Abbreviated month name.
%B Full month name.
%c Appropriate date and time representation.
%d Day of the month as a decimal number [01,31].
%H Hour (24-hour clock) as a decimal number [00,23].
%I Hour (12-hour clock) as a decimal number [01,12].
%j Day of the year as a decimal number [001,366].
%m Month as a decimal number [01,12].
%M Minute as a decimal number [00,59].
%p Equivalent of either AM or PM.
%S Second as a decimal number [00,61].
%U Week number of the year (Sunday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Sunday are considered to be in week 0.
%w Weekday as a decimal number [0(Sunday),6].
%W Week number of the year (Monday as the first day of the week) as a decimal number [00,53]. All days in a new year preceding the first Monday are considered to be in week 0.
%x Appropriate date representation.
%X Apropriate time representation.
%y Year without century as a decimal number [00,99].
%Y Year with century as a decimal number.
%Z Time zone name (no characters if no time zone exists).
%% A literal '%' character.
Get the current date and time in Python using datetime module

The syntax is:

now = datetime.datetime.now()
now.hour
now.mintue
now.year
now.day
now.month

now = datetime.datetime.now() now.hour now.mintue now.year now.day now.month Examples
#!/usr/bin/python
import datetime
i = datetime.datetime.now()
 
print ("Current date & time = %s" % i)
 
print ("Date and time in ISO format = %s" % i.isoformat() )
 
print ("Current year = %s" %i.year)
 
print ("Current month = %s" %i.month)
 
print ("Current date (day) =  %s" %i.day)
 
print ("dd/mm/yyyy format =  %s/%s/%s" % (i.day, i.month, i.year) )
 
print ("Current hour = %s" %i.hour)
 
print ("Current minute = %s" %i.minute)
 
print ("Current second =  %s" %i.second)
 
print ("hh:mm:ss format = %s:%s:%s" % (i.hour, i.month, i.second) )

#!/usr/bin/python import datetime i = datetime.datetime.now() print ("Current date & time = %s" % i) print ("Date and time in ISO format = %s" % i.isoformat() ) print ("Current year = %s" %i.year) print ("Current month = %s" %i.month) print ("Current date (day) = %s" %i.day) print ("dd/mm/yyyy format = %s/%s/%s" % (i.day, i.month, i.year) ) print ("Current hour = %s" %i.hour) print ("Current minute = %s" %i.minute) print ("Current second = %s" %i.second) print ("hh:mm:ss format = %s:%s:%s" % (i.hour, i.month, i.second) )

Sample outputs:

Current date & time = 2013-10-05 00:15:31.769049
Date and time in ISO format = 2013-10-05T00:15:31.769049
Current year = 2013
Current month = 10
Current date (day) =  5
dd/mm/yyyy format =  5/10/2013
Current hour = 0
Current minute = 15
Current second =  31
hh:mm:ss format = 0:10:31

I tested above programes with both Python 2 and Python 3:

Python Get Today's Current Date and Time
How to get current date and time in Python?

You can use date.strftime(format) to get a string representing the date, controlled by an explicit format string (see above table):

#!/usr/bin/python
 
from datetime import datetime
 
i = datetime.now()
 
print str(i)
 
print i.strftime('%Y/%m/%d %H:%M:%S')

#!/usr/bin/python from datetime import datetime i = datetime.now() print str(i) print i.strftime('%Y/%m/%d %H:%M:%S')

Sample outputs:

2013-10-05 00:20:30.495228
2013/10/05 00:20:30
Conclusion

You learned how to to get today's date and current date and time in Python using various methods. See python time and datetime module for more info.

[Nov 26, 2019] PythonWarts

Nov 21, 2019 | wiki.python.org

The Type/Class Dichotomy

(Fixed in Python 2.2.)

Down in the bowels of the C implementation of Python, types and classes are subtly different. A type is a C structure that contains several tables of pointers to C functions. For example, one table points to the functions that implement numeric operators such as + and * . Types which don't implement a function simply set the corresponding pointer to NULL; otherwise, there's a pointer to a C function that implements the corresponding operation. A class, on the other hand, is a Python object; methods with special names, such as __getitem__ and __add__ are used to add operator semantics, such as dictionary access or numeric behaviour.

The problem is that types and classes are similar in one respect, but different in others. Both types and classes provide attributes and methods, and operators can be overridden for them, but you couldn't subclass a type because types and classes are implemented differently. This meant that you couldn't subclass Python's built-in lists, dictionaries, file objects to add a new method or different behaviour. It was possible to simulate a built-in type by providing all the required special methods -- for example, the UserList class in the standard library simulates lists by implementing __len__ , __getitem__ , __setitem__ , __setslice__ , and so forth -- but this is tedious, and the wrapper class can become out of date if future Python versions add new methods or attributes. If code did an explicit type check, as in:

def f(list_arg):
    if not isinstance(list_arg, list):
        raise TypeError, "list_arg should be a list"

Instances of your list lookalike subclass would never be acceptable as arguments to f() . (There are very good arguments that explicit isinstance() checks are un-Pythonic and should be avoided for just this reason; explicit checks prevent you from passing in arguments that will behave just like the desired type.)

Jython doesn't have this type/class dichotomy because it can subclass an arbitrary Java class. Jim Fulton's ExtensionClass could alleviate the problem for CPython, but it's not a standard component of Python, requires compiling a C extension, and can't simulate a class completely so it ends up introducing various new problems that occur less often but much more difficult to debug or work around when you encounter them. For example, with ExtensionClass you can't define a list-like class and override methods to define comparison between instances of the class and regular Python lists.

This problem can be fixed, but the flip side is that such generality will probably demand a speed penalty. The C code that makes up Python can take shortcuts if it's known that a Python object will belong to a given C type. For example, classes, instances, and modules are all implemented as namespaces using the dictionary type. If it's possible to subclass dictionaries, then it should also be possible to use a dictionary subclass to implement a class, instance, or module. (This could be used for all sorts of clever tricks by changing the semantics of the dictionary class used; for example, a read-only dictionary could prevent you from modifying instances or modules.) But this means that retrieving a dictionary key can't use special-case code that assumes the default C implementation, but instead has to call more general code. I'd like to see this problem fixed, because it would make some tasks easier and cleaner, but how much loss of speed am I willing to pay for it? I don't really know...

Python 2.2 made it possible to subclass built-in types such as lists and dictionaries, and the distinction between classes and types is greatly reduced. The changes are complex to describe, requiring the introduction of a new set of inheritance semantics and relationships, called "new-style classes" as a shorthand.

Backward compatibility was of great concern, however, so it wasn't possible to jettison the old rules (or "classic classes", as they're called). Instead the two sets of rules will coexist. If you subclass a built-in type or the new-in-2.2 object type, they'll obey the rules for new-style classes; otherwise they'll follow the rules for classic classes. In some distant future version -- perhaps Python 3.0 -- the old-style rules may be discarded; how this change will be gradually introduced isn't clear.

The best way to learn about new-style classes is to read Guido van Rossum's essay . For a highly detailed dissection of the new rules, you can read the three PEPS describing them: PEP 252: "Making Types Look More Like Classes" , PEP 253: "Subtyping Built-in Types" , and PEP 254: "Making Classes Look More Like Types" . (You'll need to be a serious Python wizard to find the PEPs very helpful; they are deep magic, indeed.)

[Nov 26, 2019] Indentation Error in Python

Feb 20, 2013 | stackoverflow.com
I can't compile because of this part in my code:
    if command == 'HOWMANY':
        opcodegroupr = "A0"
        opcoder = "85"
    elif command == 'IDENTIFY':
        opcodegroupr = "A0"
        opcoder = "81"

I have this error:

Sorry: IndentationError: ('unindent does not match any outer indentation level', ('wsn.py', 1016, 30, "\t\telif command == 'IDENTIFY':\n"))

But I don't see any indentation error. What can be the problem?

Martijn Pieters ,Feb 20, 2013 at 11:54

You are mixing tabs and spaces.

Find the exact location with:

python -tt yourscript.py

and replace all tabs with spaces. You really want to configure your text editor to only insert spaces for tabs as well.

poke ,Feb 20, 2013 at 11:55

Or the other way around (depends on your personal preference) – poke Feb 20 '13 at 11:55

poke ,Feb 20, 2013 at 12:02

@MartijnPieters If you use tabs, you have tabs, so you do not need to care about its visual presentation. You should never mix tabs and spaces, but apart from that, just choose one and stick to it . You are right, it's a never-ending debate; it totally depends on your personal preference -- hence my comment. – poke Feb 20 '13 at 12:02

neil ,Feb 20, 2013 at 12:02

I have never understood why you would want to use spaces instead of tabs - 1 tab is 1 level of indent and then the size of that is a display preference - but it seems the world disagrees with me. – neil Feb 20 '13 at 12:02

Martijn Pieters ♦ ,Feb 20, 2013 at 12:13

@poke: That's very nice, but in any decent-sized project you will not be the only developer. As soon as you have two people together, there is a large chance you'll disagree about tab size. And pretending that noone will ever make the mistake of mixing tabs and spaces is sticking your head in the sand, frankly. There is a reason that every major style guide for OSS (python or otherwise) states you need to use spaces only . :-) – Martijn Pieters ♦ Feb 20 '13 at 12:13

geoffspear ,Feb 20, 2013 at 12:22

There should be one, and preferably only one, obvious way to do it. Following the style of the python codebase itself is obvious. – geoffspear Feb 20 '13 at 12:22

[Nov 26, 2019] Python Myths about Indentation

Nov 06, 2017 | www.secnetix.de

Python: Myths about Indentation

Note: Lines beginning with " >>> " and " ... " indicate input to Python (these are the default prompts of the interactive interpreter). Everything else is output from Python.

There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.


"Whitespace is significant in Python source code."

No, not in general. Only the indentation level of your statements is significant (i.e. the whitespace at the very left of your statements). Everywhere else, whitespace is not significant and can be used as you like, just like in any other language. You can also insert empty lines that contain nothing (or only arbitrary whitespace) anywhere.

Also, the exact amount of indentation doesn't matter at all, but only the relative indentation of nested blocks (relative to each other).

Furthermore, the indentation level is ignored when you use explicit or implicit continuation lines. For example, you can split a list across multiple lines, and the indentation is completely insignificant. So, if you want, you can do things like this:

>>> foo = [
... 'some string',
... 'another string',
... 'short string'
... ]
>>> print foo
['some string', 'another string', 'short string']

>>> bar = 'this is ' \
... 'one long string ' \
... 'that is split ' \
... 'across multiple lines'
>>> print bar
this is one long string that is split across multiple lines

"Python forces me to use a certain indentation style."

Yes and no. First of all, you can write the inner block all on one line if you like, therefore not having to care about intendation at all. The following three versions of an "if" statement are all valid and do exactly the same thing (output omitted for brevity):

>>> if 1 + 1 == 2:
... print "foo"
... print "bar"
... x = 42

>>> if 1 + 1 == 2:
... print "foo"; print "bar"; x = 42

>>> if 1 + 1 == 2: print "foo"; print "bar"; x = 42

Of course, most of the time you will want to write the blocks in separate lines (like the first version above), but sometimes you have a bunch of similar "if" statements which can be conveniently written on one line each.

If you decide to write the block on separate lines, then yes, Python forces you to obey its indentation rules, which simply means: The enclosed block (that's two "print" statements and one assignment in the above example) have to be indented more than the "if" statement itself. That's it. And frankly, would you really want to indent it in any other way? I don't think so.

So the conclusion is: Python forces you to use indentation that you would have used anyway, unless you wanted to obfuscate the structure of the program. In other words: Python does not allow to obfuscate the structure of a program by using bogus indentations. In my opinion, that's a very good thing.

Have you ever seen code like this in C or C++?

/* Warning: bogus C code! */

if (some condition)
if (another condition)
do_something(fancy);
else
this_sucks(badluck);

Either the indentation is wrong, or the program is buggy, because an "else" always applies to the nearest "if", unless you use braces. This is an essential problem in C and C++. Of course, you could resort to always use braces, no matter what, but that's tiresome and bloats the source code, and it doesn't prevent you from accidentally obfuscating the code by still having the wrong indentation. (And that's just a very simple example. In practice, C code can be much more complex.)

In Python, the above problems can never occur, because indentation levels and logical block structure are always consistent. The program always does what you expect when you look at the indentation.

Quoting the famous book writer Bruce Eckel:

Because blocks are denoted by indentation in Python, indentation is uniform in Python programs. And indentation is meaningful to us as readers. So because we have consistent code formatting, I can read somebody else's code and I'm not constantly tripping over, "Oh, I see. They're putting their curly braces here or there." I don't have to think about that.


"You cannot safely mix tabs and spaces in Python."

That's right, and you don't want that. To be exact, you cannot safely mix tabs and spaces in C either: While it doesn't make a difference to the compiler, it can make a big difference to humans looking at the code. If you move a piece of C source to an editor with different tabstops, it will all look wrong (and possibly behave differently than it looks at first sight). You can easily introduce well-hidden bugs in code that has been mangled that way. That's why mixing tabs and spaces in C isn't really "safe" either. Also see the "bogus C code" example above.

Therefore, it is generally a good idea not to mix tabs and spaces for indentation. If you use tabs only or spaces only, you're fine.

Furthermore, it can be a good idea to avoid tabs alltogether, because the semantics of tabs are not very well-defined in the computer world, and they can be displayed completely differently on different types of systems and editors. Also, tabs often get destroyed or wrongly converted during copy&paste operations, or when a piece of source code is inserted into a web page or other kind of markup code.

Most good editors support transparent translation of tabs, automatic indent and dedent. That is, when you press the tab key, the editor will insert enough spaces (not actual tab characters!) to get you to the next position which is a multiple of eight (or four, or whatever you prefer), and some other key (usually Backspace) will get you back to the previous indentation level.

In other words, it's behaving like you would expect a tab key to do, but still maintaining portability by using spaces in the file only. This is convenient and safe.

Having said that -- If you know what you're doing, you can of course use tabs and spaces to your liking, and then use tools like "expand" (on UNIX machines, for example) before giving the source to others. If you use tab characters, Python assumes that tab stops are eight positions apart.


"I just don't like it."

That's perfectly OK; you're free to dislike it (and you're probably not alone). Granted, the fact that indentation is used to indicate the block structure might be regarded as uncommon and requiring to get used to it, but it does have a lot of advantages, and you get used to it very quickly when you seriously start programming in Python.

Having said that, you can use keywords to indicate the end of a block (instead of indentation), such as " endif ". These are not really Python keywords, but there is a tool that comes with Python which converts code using "end" keywords to correct indentation and removes those keywords. It can be used as a pre-processor to the Python compiler. However, no real Python programmer uses it, of course.
[Update] It seems this tool has been removed from recent versions of Python. Probably because nobody really used it.


"How does the compiler parse the indentation?"

The parsing is well-defined and quite simple. Basically, changes to the indentation level are inserted as tokens into the token stream.

The lexical analyzer (tokenizer) uses a stack to store indentation levels. At the beginning, the stack contains just the value 0, which is the leftmost position. Whenever a nested block begins, the new indentation level is pushed on the stack, and an "INDENT" token is inserted into the token stream which is passed to the parser. There can never be more than one "INDENT" token in a row.

When a line is encountered with a smaller indentation level, values are popped from the stack until a value is on top which is equal to the new indentation level (if none is found, a syntax error occurs). For each value popped, a "DEDENT" token is generated. Obviously, there can be multiple "DEDENT" tokens in a row.

At the end of the source code, "DEDENT" tokens are generated for each indentation level left on the stack, until just the 0 is left.

Look at the following piece of sample code:

>>> if foo:
... if bar:
... x = 42
... else:
... print foo
...
In the following table, you can see the tokens produced on the left, and the indentation stack on the right.
<if> <foo> <:> [0]
<INDENT> <if> <bar> <:> [0, 4]
<INDENT> <x> <=> <42> [0, 4, 8]
<DEDENT> <DEDENT> <else> <:> [0]
<INDENT> <print> <foo> [0, 2]
<DEDENT> [0]
Note that after the lexical analysis (before parsing starts), there is no whitespace left in the list of tokens (except possibly within string literals, of course). In other words, the indentation is handled by the lexer, not by the parser.

The parser then simply handles the "INDENT" and "DEDENT" tokens as block delimiters -- exactly like curly braces are handled by a C compiler.

The above example is intentionally simple. There are more things to it, such as continuation lines. They are well-defined, too, and you can read about them in the Python Language Reference if you're interested, which includes a complete formal grammar of the language.

[Nov 23, 2019] What is the meaning of a single and a double underscore before an object name?

Notable quotes:
"... does not import objects whose name starts with an underscore. ..."
Aug 19, 2009 | stackoverflow.com
Can someone please explain the exact meaning of having leading underscores before an object's name in Python? Also, explain the difference between a single and a double leading underscore. Also, does that meaning stay the same whether the object in question is a variable, a function, a method, etc.?

Andrew Keeton , 2009-08-19 17:15:53

Single Underscore

Names, in a class, with a leading underscore are simply to indicate to other programmers that the attribute or method is intended to be private. However, nothing special is done with the name itself.

To quote PEP-8 :

_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.

Double Underscore (Name Mangling)

From the Python docs :

Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam , where classname is the current class name with leading underscore(s) stripped. This mangling is done without regard to the syntactic position of the identifier, so it can be used to define class-private instance and class variables, methods, variables stored in globals, and even variables stored in instances. private to this class on instances of other classes.

And a warning from the same page:

Name mangling is intended to give classes an easy way to define "private" instance variables and methods, without having to worry about instance variables defined by derived classes, or mucking with instance variables by code outside the class. Note that the mangling rules are designed mostly to avoid accidents; it still is possible for a determined soul to access or modify a variable that is considered private.

Example
>>> class MyClass():
...     def __init__(self):
...             self.__superprivate = "Hello"
...             self._semiprivate = ", world!"
...
>>> mc = MyClass()
>>> print mc.__superprivate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: myClass instance has no attribute '__superprivate'
>>> print mc._semiprivate
, world!
>>> print mc.__dict__
{'_MyClass__superprivate': 'Hello', '_semiprivate': ', world!'}

Alex Martelli , 2009-08-19 17:52:36

Excellent answers so far but some tidbits are missing. A single leading underscore isn't exactly just a convention: if you use from foobar import * , and module foobar does not define an __all__ list, the names imported from the module do not include those with a leading underscore. Let's say it's mostly a convention, since this case is a pretty obscure corner;-).

The leading-underscore convention is widely used not just for private names, but also for what C++ would call protected ones -- for example, names of methods that are fully intended to be overridden by subclasses (even ones that have to be overridden since in the base class they raise NotImplementedError !-) are often single-leading-underscore names to indicate to code using instances of that class (or subclasses) that said methods are not meant to be called directly.

For example, to make a thread-safe queue with a different queueing discipline than FIFO, one imports Queue, subclasses Queue.Queue, and overrides such methods as _get and _put ; "client code" never calls those ("hook") methods, but rather the ("organizing") public methods such as put and get (this is known as the Template Method design pattern -- see e.g. here for an interesting presentation based on a video of a talk of mine on the subject, with the addition of synopses of the transcript).

Ned Batchelder , 2009-08-19 17:21:29

__foo__ : this is just a convention, a way for the Python system to use names that won't conflict with user names.

_foo : this is just a convention, a way for the programmer to indicate that the variable is private (whatever that means in Python).

__foo : this has real meaning: the interpreter replaces this name with _classname__foo as a way to ensure that the name will not overlap with a similar name in another class.

No other form of underscores have meaning in the Python world.

There's no difference between class, variable, global, etc in these conventions.

2 revs, 2 users 93%
, 2016-05-17 10:09:08

._variable is semiprivate and meant just for convention

.__variable is often incorrectly considered superprivate, while it's actual meaning is just to namemangle to prevent accidental access [1]

.__variable__ is typically reserved for builtin methods or variables

You can still access .__mangled variables if you desperately want to. The double underscores just namemangles, or renames, the variable to something like instance._className__mangled

Example:

class Test(object):
    def __init__(self):
        self.__a = 'a'
        self._b = 'b'

>>> t = Test()
>>> t._b
'b'

t._b is accessible because it is only hidden by convention

>>> t.__a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Test' object has no attribute '__a'

t.__a isn't found because it no longer exists due to namemangling

>>> t._Test__a
'a'

By accessing instance._className__variable instead of just the double underscore name, you can access the hidden value

9 revs, 8 users 82%
, 2018-08-21 19:42:09

Single underscore at the beginning:

Python doesn't have real private methods. Instead, one underscore at the start of a method or attribute name means you shouldn't access this method, because it's not part of the API.

class BaseForm(StrAndUnicode):

    def _get_errors(self):
        "Returns an ErrorDict for the data provided for the form"
        if self._errors is None:
            self.full_clean()
        return self._errors

    errors = property(_get_errors)

(This code snippet was taken from django source code: django/forms/forms.py). In this code, errors is a public property, but the method this property calls, _get_errors, is "private", so you shouldn't access it.

Two underscores at the beginning:

This causes a lot of confusion. It should not be used to create a private method. It should be used to avoid your method being overridden by a subclass or accessed accidentally. Let's see an example:

class A(object):
    def __test(self):
        print "I'm a test method in class A"

    def test(self):
        self.__test()

a = A()
a.test()
# a.__test() # This fails with an AttributeError
a._A__test() # Works! We can access the mangled name directly!

Output:

$ python test.py
I'm test method in class A
I'm test method in class A

Now create a subclass B and do customization for __test method

class B(A):
    def __test(self):
        print "I'm test method in class B"

b = B()
b.test()

Output will be....

$ python test.py
I'm test method in class A

As we have seen, A.test() didn't call B.__test() methods, as we might expect. But in fact, this is the correct behavior for __. The two methods called __test() are automatically renamed (mangled) to _A__test() and _B__test(), so they do not accidentally override. When you create a method starting with __ it means that you don't want to anyone to be able to override it, and you only intend to access it from inside its own class.

Two underscores at the beginning and at the end:

When we see a method like __this__ , don't call it. This is a method which python is meant to call, not you. Let's take a look:

>>> name = "test string"
>>> name.__len__()
11
>>> len(name)
11

>>> number = 10
>>> number.__add__(40)
50
>>> number + 50
60

There is always an operator or native function which calls these magic methods. Sometimes it's just a hook python calls in specific situations. For example __init__() is called when the object is created after __new__() is called to build the instance...

Let's take an example...

class FalseCalculator(object):

    def __init__(self, number):
        self.number = number

    def __add__(self, number):
        return self.number - number

    def __sub__(self, number):
        return self.number + number

number = FalseCalculator(20)
print number + 10      # 10
print number - 20      # 40

For more details, see the PEP-8 guide . For more magic methods, see this PDF .

Tim D , 2012-01-11 16:28:22

Sometimes you have what appears to be a tuple with a leading underscore as in
def foo(bar):
    return _('my_' + bar)

In this case, what's going on is that _() is an alias for a localization function that operates on text to put it into the proper language, etc. based on the locale. For example, Sphinx does this, and you'll find among the imports

from sphinx.locale import l_, _

and in sphinx.locale, _() is assigned as an alias of some localization function.

Dev Maha , 2013-04-15 01:58:14

If one really wants to make a variable read-only, IMHO the best way would be to use property() with only getter passed to it. With property() we can have complete control over the data.
class PrivateVarC(object):

    def get_x(self):
        pass

    def set_x(self, val):
        pass

    rwvar = property(get_p, set_p)  

    ronly = property(get_p)

I understand that OP asked a little different question but since I found another question asking for 'how to set private variables' marked duplicate with this one, I thought of adding this additional info here.

SilentGhost ,

Single leading underscores is a convention. there is no difference from the interpreter's point of view if whether names starts with a single underscore or not.

Double leading and trailing underscores are used for built-in methods, such as __init__ , __bool__ , etc.

Double leading underscores w/o trailing counterparts are a convention too, however, the class methods will be mangled by the interpreter. For variables or basic function names no difference exists.

3 revs
, 2018-12-16 11:41:34

Since so many people are referring to Raymond's talk , I'll just make it a little easier by writing down what he said:

The intention of the double underscores was not about privacy. The intention was to use it exactly like this

class Circle(object):

    def __init__(self, radius):
        self.radius = radius

    def area(self):
        p = self.__perimeter()
        r = p / math.pi / 2.0
        return math.pi * r ** 2.0

    def perimeter(self):
        return 2.0 * math.pi * self.radius

    __perimeter = perimeter  # local reference


class Tire(Circle):

    def perimeter(self):
        return Circle.perimeter(self) * 1.25

It's actually the opposite of privacy, it's all about freedom. It makes your subclasses free to override any one method without breaking the others .

Say you don't keep a local reference of perimeter in Circle . Now, a derived class Tire overrides the implementation of perimeter , without touching area . When you call Tire(5).area() , in theory it should still be using Circle.perimeter for computation, but in reality it's using Tire.perimeter , which is not the intended behavior. That's why we need a local reference in Circle.

But why __perimeter instead of _perimeter ? Because _perimeter still gives derived class the chance to override:

class Tire(Circle):

    def perimeter(self):
        return Circle.perimeter(self) * 1.25

    _perimeter = perimeter

Double underscores has name mangling, so there's a very little chance that the local reference in parent class get override in derived class. thus " makes your subclasses free to override any one method without breaking the others ".

If your class won't be inherited, or method overriding does not break anything, then you simply don't need __double_leading_underscore .

u0b34a0f6ae , 2009-08-19 17:31:04

Your question is good, it is not only about methods. Functions and objects in modules are commonly prefixed with one underscore as well, and can be prefixed by two.

But __double_underscore names are not name-mangled in modules, for example. What happens is that names beginning with one (or more) underscores are not imported if you import all from a module (from module import *), nor are the names shown in help(module).

Marc , 2014-08-22 19:15:48

Here is a simple illustrative example on how double underscore properties can affect an inherited class. So with the following setup:
class parent(object):
    __default = "parent"
    def __init__(self, name=None):
        self.default = name or self.__default

    @property
    def default(self):
        return self.__default

    @default.setter
    def default(self, value):
        self.__default = value


class child(parent):
    __default = "child"

if you then create a child instance in the python REPL, you will see the below

child_a = child()
child_a.default            # 'parent'
child_a._child__default    # 'child'
child_a._parent__default   # 'parent'

child_b = child("orphan")
## this will show 
child_b.default            # 'orphan'
child_a._child__default    # 'child'
child_a._parent__default   # 'orphan'

This may be obvious to some, but it caught me off guard in a much more complex environment

aptro , 2015-02-07 17:57:10

"Private" instance variables that cannot be accessed except from inside an object don't exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice.

reference https://docs.python.org/2/tutorial/classes.html#private-variables-and-class-local-references

grepit , 2019-01-21 22:23:39

Great answers and all are correct.I have provided simple example along with simple definition/meaning.

Meaning:

some_variable --► it's public anyone can see this.

_some_variable --► it's public anyone can see this but it's a convention to indicate private... warning no enforcement is done by Python.

__some_varaible --► Python replaces the variable name with _classname__some_varaible (AKA name mangling) and it reduces/hides it's visibility and be more like private variable.

Just to be honest here According to Python documentation

""Private" instance variables that cannot be accessed except from inside an object don't exist in Python"

The example:

class A():
    here="abc"
    _here="_abc"
    __here="__abc"


aObject=A()
print(aObject.here) 
print(aObject._here)
# now if we try to print __here then it will fail because it's not public variable 
#print(aObject.__here)

2 revs
, 2017-11-04 17:51:49

Getting the facts of _ and __ is pretty easy; the other answers express them pretty well. The usage is much harder to determine.

This is how I see it:

_

Should be used to indicate that a function is not for public use as for example an API. This and the import restriction make it behave much like internal in c#.

__

Should be used to avoid name collision in the inheritace hirarchy and to avoid latebinding. Much like private in c#.

==>

If you want to indicate that something is not for public use, but it should act like protected use _ . If you want to indicate that something is not for public use, but it should act like private use __ .

This is also a quote that I like very much:

The problem is that the author of a class may legitimately think "this attribute/method name should be private, only accessible from within this class definition" and use the __private convention. But later on, a user of that class may make a subclass that legitimately needs access to that name. So either the superclass has to be modified (which may be difficult or impossible), or the subclass code has to use manually mangled names (which is ugly and fragile at best).

But the problem with that is in my opinion that if there's no IDE that warns you when you override methods, finding the error might take you a while if you have accidentially overriden a method from a base-class.

[Nov 23, 2019] How can I remove a trailing newline?

Jan 01, 2008 | stackoverflow.com

Ask Question Asked 11 years ago Active 16 days ago Viewed 1.7m times


, 2008-11-08 18:25:24

What is the Python equivalent of Perl's chomp function, which removes the last character of a string if it is a newline?

9 revs, 7 users 37%
, 2017-05-11 19:54:59

Try the method rstrip() (see doc Python 2 and Python 3 )
>>> 'test string\n'.rstrip()
'test string'

Python's rstrip() method strips all kinds of trailing whitespace by default, not just one newline as Perl does with chomp .

>>> 'test string \n \r\n\n\r \n\n'.rstrip()
'test string'

To strip only newlines:

>>> 'test string \n \r\n\n\r \n\n'.rstrip('\n')
'test string \n \r\n\n\r '

There are also the methods lstrip() and strip() :

>>> s = "   \n\r\n  \n  abc   def \n\r\n  \n  "
>>> s.strip()
'abc   def'
>>> s.lstrip()
'abc   def \n\r\n  \n  '
>>> s.rstrip()
'   \n\r\n  \n  abc   def'

Ryan Ginstrom , 2008-11-09 05:52:43

And I would say the "pythonic" way to get lines without trailing newline characters is splitlines().
>>> text = "line 1\nline 2\r\nline 3\nline 4"
>>> text.splitlines()
['line 1', 'line 2', 'line 3', 'line 4']

Mike ,

The canonical way to strip end-of-line (EOL) characters is to use the string rstrip() method removing any trailing \r or \n. Here are examples for Mac, Windows, and Unix EOL characters.
>>> 'Mac EOL\r'.rstrip('\r\n')
'Mac EOL'
>>> 'Windows EOL\r\n'.rstrip('\r\n')
'Windows EOL'
>>> 'Unix EOL\n'.rstrip('\r\n')
'Unix EOL'

Using '\r\n' as the parameter to rstrip means that it will strip out any trailing combination of '\r' or '\n'. That's why it works in all three cases above.

This nuance matters in rare cases. For example, I once had to process a text file which contained an HL7 message. The HL7 standard requires a trailing '\r' as its EOL character. The Windows machine on which I was using this message had appended its own '\r\n' EOL character. Therefore, the end of each line looked like '\r\r\n'. Using rstrip('\r\n') would have taken off the entire '\r\r\n' which is not what I wanted. In that case, I simply sliced off the last two characters instead.

Note that unlike Perl's chomp function, this will strip all specified characters at the end of the string, not just one:

>>> "Hello\n\n\n".rstrip("\n")
"Hello"

, 2008-11-28 17:31:34

Note that rstrip doesn't act exactly like Perl's chomp() because it doesn't modify the string. That is, in Perl:
$x="a\n";

chomp $x

results in $x being "a" .

but in Python:

x="a\n"

x.rstrip()

will mean that the value of x is still "a\n" . Even x=x.rstrip() doesn't always give the same result, as it strips all whitespace from the end of the string, not just one newline at most.

Jamie ,

I might use something like this:
import os
s = s.rstrip(os.linesep)

I think the problem with rstrip("\n") is that you'll probably want to make sure the line separator is portable. (some antiquated systems are rumored to use "\r\n" ). The other gotcha is that rstrip will strip out repeated whitespace. Hopefully os.linesep will contain the right characters. the above works for me.

kiriloff , 2013-05-13 16:41:22

You may use line = line.rstrip('\n') . This will strip all newlines from the end of the string, not just one.

slec , 2015-03-09 08:02:55

s = s.rstrip()

will remove all newlines at the end of the string s . The assignment is needed because rstrip returns a new string instead of modifying the original string.

Alien Life Form ,

This would replicate exactly perl's chomp (minus behavior on arrays) for "\n" line terminator:
def chomp(x):
    if x.endswith("\r\n"): return x[:-2]
    if x.endswith("\n") or x.endswith("\r"): return x[:-1]
    return x

(Note: it does not modify string 'in place'; it does not strip extra trailing whitespace; takes \r\n in account)

Hackaholic ,

you can use strip:
line = line.strip()

demo:

>>> "\n\n hello world \n\n".strip()
'hello world'

mihaicc ,

"line 1\nline 2\r\n...".replace('\n', '').replace('\r', '')
>>> 'line 1line 2...'

or you could always get geekier with regexps :)

have fun!

Carlos Valiente , 2011-04-27 11:43:20

Careful with "foo".rstrip(os.linesep) : That will only chomp the newline characters for the platform where your Python is being executed. Imagine you're chimping the lines of a Windows file under Linux, for instance:
$ python
Python 2.7.1 (r271:86832, Mar 18 2011, 09:09:48) 
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.platform
'linux2'
>>> "foo\r\n".rstrip(os.linesep)
'foo\r'
>>>

Use "foo".rstrip("\r\n") instead, as Mike says above.

minopret , 2013-10-23 01:32:11

An example in Python's documentation simply uses line.strip() .

Perl's chomp function removes one linebreak sequence from the end of a string only if it's actually there.

Here is how I plan to do that in Python, if process is conceptually the function that I need in order to do something useful to each line from this file:

import os
sep_pos = -len(os.linesep)
with open("file.txt") as f:
    for line in f:
        if line[sep_pos:] == os.linesep:
            line = line[:sep_pos]
        process(line)

ingydotnet ,

rstrip doesn't do the same thing as chomp, on so many levels. Read http://perldoc.perl.org/functions/chomp.html and see that chomp is very complex indeed.

However, my main point is that chomp removes at most 1 line ending, whereas rstrip will remove as many as it can.

Here you can see rstrip removing all the newlines:

>>> 'foo\n\n'.rstrip(os.linesep)
'foo'

A much closer approximation of typical Perl chomp usage can be accomplished with re.sub, like this:

>>> re.sub(os.linesep + r'\Z','','foo\n\n')
'foo\n'

Andrew Grimm ,

I don't program in Python, but I came across an FAQ at python.org advocating S.rstrip("\r\n") for python 2.2 or later.

, 2014-01-20 19:07:03

import re

r_unwanted = re.compile("[\n\t\r]")
r_unwanted.sub("", your_text)

Leozj ,

If your question is to clean up all the line breaks in a multiple line str object (oldstr), you can split it into a list according to the delimiter '\n' and then join this list into a new str(newstr).

newstr = "".join(oldstr.split('\n'))

kuzzooroo ,

I find it convenient to have be able to get the chomped lines via in iterator, parallel to the way you can get the un-chomped lines from a file object. You can do so with the following code:
def chomped_lines(it):
    return map(operator.methodcaller('rstrip', '\r\n'), it)

Sample usage:

with open("file.txt") as infile:
    for line in chomped_lines(infile):
        process(line)

Chij , 2011-11-30 14:04:19

workaround solution for special case:

if the newline character is the last character (as is the case with most file inputs), then for any element in the collection you can index as follows:

foobar= foobar[:-1]

to slice out your newline character.

user3780389 , 2017-04-26 17:58:16

It looks like there is not a perfect analog for perl's chomp . In particular, rstrip cannot handle multi-character newline delimiters like \r\n . However, splitlines does as pointed out here . Following my answer on a different question, you can combine join and splitlines to remove/replace all newlines from a string s :
''.join(s.splitlines())

The following removes exactly one trailing newline (as chomp would, I believe). Passing True as the keepends argument to splitlines retain the delimiters. Then, splitlines is called again to remove the delimiters on just the last "line":

def chomp(s):
    if len(s):
        lines = s.splitlines(True)
        last = lines.pop()
        return ''.join(lines + last.splitlines())
    else:
        return ''

Taylor Edmiston ,

I'm bubbling up my regular expression based answer from one I posted earlier in the comments of another answer. I think using re is a clearer more explicit solution to this problem than str.rstrip .
>>> import re

If you want to remove one or more trailing newline chars:

>>> re.sub(r'[\n\r]+$', '', '\nx\r\n')
'\nx'

If you want to remove newline chars everywhere (not just trailing):

>>> re.sub(r'[\n\r]+', '', '\nx\r\n')
'x'

If you want to remove only 1-2 trailing newline chars (i.e., \r , \n , \r\n , \n\r , \r\r , \n\n )

>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n\r\n')
'\nx\r'
>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n\r')
'\nx\r'
>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n')
'\nx'

I have a feeling what most people really want here, is to remove just one occurrence of a trailing newline character, either \r\n or \n and nothing more.

>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\n\n', count=1)
'\nx\n'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\r\n\r\n', count=1)
'\nx\r\n'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\r\n', count=1)
'\nx'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\n', count=1)
'\nx'

(The ?: is to create a non-capturing group.)

(By the way this is not what '...'.rstrip('\n', '').rstrip('\r', '') does which may not be clear to others stumbling upon this thread. str.rstrip strips as many of the trailing characters as possible, so a string like foo\n\n\n would result in a false positive of foo whereas you may have wanted to preserve the other newlines after stripping a single trailing one.)

Help me , 2016-05-20 12:29:21

Just use :
line = line.rstrip("\n")

or

line = line.strip("\n")

You don't need any of this complicated stuff

, 2016-11-22 18:30:37

>>> '   spacious   '.rstrip()
'   spacious'
>>> "AABAA".rstrip("A")
  'AAB'
>>> "ABBA".rstrip("AB") # both AB and BA are stripped
   ''
>>> "ABCABBA".rstrip("AB")
   'ABC'

internetional , 2016-11-22 20:17:58

There are three types of line endings that we normally encounter: \n , \r and \r\n . A rather simple regular expression in re.sub , namely r"\r?\n?$" , is able to catch them all.

(And we gotta catch 'em all , am I right?)

import re

re.sub(r"\r?\n?$", "", the_text, 1)

With the last argument, we limit the number of occurences replaced to one, mimicking chomp to some extent. Example:

import re

text_1 = "hellothere\n\n\n"
text_2 = "hellothere\n\n\r"
text_3 = "hellothere\n\n\r\n"

a = re.sub(r"\r?\n?$", "", text_1, 1)
b = re.sub(r"\r?\n?$", "", text_2, 1)
c = re.sub(r"\r?\n?$", "", text_3, 1)

... where a == b == c is True .

Venfah Nazir , 2018-06-15 07:24:21


This will work both for windows and linux (bit expensive with re sub if you are looking for only re solution)

import re 
if re.search("(\\r|)\\n$", line):
    line = re.sub("(\\r|)\\n$", "", line)

Stephen Miller ,

If you are concerned about speed (say you have a looong list of strings) and you know the nature of the newline char, string slicing is actually faster than rstrip. A little test to illustrate this:
import time

loops = 50000000

def method1(loops=loops):
    test_string = 'num\n'
    t0 = time.time()
    for num in xrange(loops):
        out_sting = test_string[:-1]
    t1 = time.time()
    print('Method 1: ' + str(t1 - t0))

def method2(loops=loops):
    test_string = 'num\n'
    t0 = time.time()
    for num in xrange(loops):
        out_sting = test_string.rstrip()
    t1 = time.time()
    print('Method 2: ' + str(t1 - t0))

method1()
method2()

Output:

Method 1: 3.92700004578
Method 2: 6.73000001907

sim , 2019-10-22 07:43:27

s = '''Hello  World \t\n\r\tHi There'''
# import the module string   
import string
# use the method translate to convert 
s.translate({ord(c): None for c in string.whitespace}
>>'HelloWorldHiThere'

With regex

s = '''  Hello  World 
\t\n\r\tHi '''
print(re.sub(r"\s+", "", s), sep='')  # \s matches all white spaces
>HelloWorldHi

Replace \n,\t,\r

s.replace('\n', '').replace('\t','').replace('\r','')
>'  Hello  World Hi '

With regex

s = '''Hello  World \t\n\r\tHi There'''
regex = re.compile(r'[\n\r\t]')
regex.sub("", s)
>'Hello  World Hi There'

with Join

s = '''Hello  World \t\n\r\tHi There'''
' '.join(s.split())
>'Hello  World Hi There'

DeepBlue , 2019-11-06 20:50:30

First split lines then join them by any separator you like.
  x = ' '.join(x.splitlines())

should work like a charm.

user4178860 , 2014-10-24 18:34:12

A catch all:
line = line.rstrip('\r|\n')

Flimm , 2016-06-30 16:20:15

rstrip does not take regular expression. "hi|||\n\n".rstrip("\r|\n") returns "hi"Flimm Jun 30 '16 at 16:20

[Nov 22, 2019] Using global variables in a function

Jan 01, 2009 | stackoverflow.com

user46646 , 2009-01-08 05:45:02

How can I create or use a global variable in a function?

If I create a global variable in one function, how can I use that global variable in another function? Do I need to store the global variable in a local variable of the function which needs its access?

Paul Stephenson , 2009-01-08 08:39:44

You can use a global variable in other functions by declaring it as global in each function that assigns to it:
globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print(globvar)     # No need for global declaration to read value of globvar

set_globvar_to_one()
print_globvar()       # Prints 1

I imagine the reason for it is that, since global variables are so dangerous, Python wants to make sure that you really know that's what you're playing with by explicitly requiring the global keyword.

See other answers if you want to share a global variable across modules.

Jeff Shannon , 2009-01-08 09:19:55

If I'm understanding your situation correctly, what you're seeing is the result of how Python handles local (function) and global (module) namespaces.

Say you've got a module like this:

# sample.py
myGlobal = 5

def func1():
    myGlobal = 42

def func2():
    print myGlobal

func1()
func2()

You might expecting this to print 42, but instead it prints 5. As has already been mentioned, if you add a ' global ' declaration to func1() , then func2() will print 42.

def func1():
    global myGlobal
    myGlobal = 42

What's going on here is that Python assumes that any name that is assigned to , anywhere within a function, is local to that function unless explicitly told otherwise. If it is only reading from a name, and the name doesn't exist locally, it will try to look up the name in any containing scopes (e.g. the module's global scope).

When you assign 42 to the name myGlobal , therefore, Python creates a local variable that shadows the global variable of the same name. That local goes out of scope and is garbage-collected when func1() returns; meanwhile, func2() can never see anything other than the (unmodified) global name. Note that this namespace decision happens at compile time, not at runtime -- if you were to read the value of myGlobal inside func1() before you assign to it, you'd get an UnboundLocalError , because Python has already decided that it must be a local variable but it has not had any value associated with it yet. But by using the ' global ' statement, you tell Python that it should look elsewhere for the name instead of assigning to it locally.

(I believe that this behavior originated largely through an optimization of local namespaces -- without this behavior, Python's VM would need to perform at least three name lookups each time a new name is assigned to inside a function (to ensure that the name didn't already exist at module/builtin level), which would significantly slow down a very common operation.)

gimel , 2009-01-08 05:59:04

You may want to explore the notion of namespaces . In Python, the module is the natural place for global data:

Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user's global variables. On the other hand, if you know what you are doing you can touch a module's global variables with the same notation used to refer to its functions, modname.itemname .

A specific use of global-in-a-module is described here - How do I share global variables across modules? , and for completeness the contents are shared here:

The canonical way to share information across modules within a single program is to create a special configuration module (often called config or cfg ). Just import the configuration module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example:

File: config.py

x = 0   # Default value of the 'x' configuration setting

File: mod.py

import config
config.x = 1

File: main.py

import config
import mod
print config.x

SingleNegationElimination ,

Python uses a simple heuristic to decide which scope it should load a variable from, between local and global. If a variable name appears on the left hand side of an assignment, but is not declared global, it is assumed to be local. If it does not appear on the left hand side of an assignment, it is assumed to be global.
>>> import dis
>>> def foo():
...     global bar
...     baz = 5
...     print bar
...     print baz
...     print quux
... 
>>> dis.disassemble(foo.func_code)
  3           0 LOAD_CONST               1 (5)
              3 STORE_FAST               0 (baz)

  4           6 LOAD_GLOBAL              0 (bar)
              9 PRINT_ITEM          
             10 PRINT_NEWLINE       

  5          11 LOAD_FAST                0 (baz)
             14 PRINT_ITEM          
             15 PRINT_NEWLINE       

  6          16 LOAD_GLOBAL              1 (quux)
             19 PRINT_ITEM          
             20 PRINT_NEWLINE       
             21 LOAD_CONST               0 (None)
             24 RETURN_VALUE        
>>>

See how baz, which appears on the left side of an assignment in foo() , is the only LOAD_FAST variable.

J S , 2009-01-08 09:03:33

If you want to refer to a global variable in a function, you can use the global keyword to declare which variables are global. You don't have to use it in all cases (as someone here incorrectly claims) - if the name referenced in an expression cannot be found in local scope or scopes in the functions in which this function is defined, it is looked up among global variables.

However, if you assign to a new variable not declared as global in the function, it is implicitly declared as local, and it can overshadow any existing global variable with the same name.

Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill.

Rauni Lillemets ,

In addition to already existing answers and to make this more confusing:

In Python, variables that are only referenced inside a function are implicitly global . If a variable is assigned a new value anywhere within the function's body, it's assumed to be a local . If a variable is ever assigned a new value inside the function, the variable is implicitly local, and you need to explicitly declare it as 'global'.

Though a bit surprising at first, a moment's consideration explains this. On one hand, requiring global for assigned variables provides a bar against unintended side-effects. On the other hand, if global was required for all global references, you'd be using global all the time. You'd have to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the global declaration for identifying side-effects.

Source: What are the rules for local and global variables in Python? .

Aaron Hall ,

If I create a global variable in one function, how can I use that variable in another function?

We can create a global with the following function:

def create_global_variable():
    global global_variable # must declare it to be a global first
    # modifications are thus reflected on the module's global scope
    global_variable = 'Foo'

Writing a function does not actually run its code. So we call the create_global_variable function:

>>> create_global_variable()
Using globals without modification

You can just use it, so long as you don't expect to change which object it points to:

For example,

def use_global_variable():
    return global_variable + '!!!'

and now we can use the global variable:

>>> use_global_variable()
'Foo!!!'
Modification of the global variable from inside a function

To point the global variable at a different object, you are required to use the global keyword again:

def change_global_variable():
    global global_variable
    global_variable = 'Bar'

Note that after writing this function, the code actually changing it has still not run:

>>> use_global_variable()
'Foo!!!'

So after calling the function:

>>> change_global_variable()

we can see that the global variable has been changed. The global_variable name now points to 'Bar' :

>>> use_global_variable()
'Bar!!!'

Note that "global" in Python is not truly global - it's only global to the module level. So it is only available to functions written in the modules in which it is global. Functions remember the module in which they are written, so when they are exported into other modules, they still look in the module in which they were created to find global variables.

Local variables with the same name

If you create a local variable with the same name, it will overshadow a global variable:

def use_local_with_same_name_as_global():
    # bad name for a local variable, though.
    global_variable = 'Baz' 
    return global_variable + '!!!'

>>> use_local_with_same_name_as_global()
'Baz!!!'

But using that misnamed local variable does not change the global variable:

>>> use_global_variable()
'Bar!!!'

Note that you should avoid using the local variables with the same names as globals unless you know precisely what you are doing and have a very good reason to do so. I have not yet encountered such a reason.

Bohdan , 2013-10-03 05:41:16

With parallel execution, global variables can cause unexpected results if you don't understand what is happening. Here is an example of using a global variable within multiprocessing. We can clearly see that each process works with its own copy of the variable:
import multiprocessing
import os
import random
import sys
import time

def worker(new_value):
    old_value = get_value()
    set_value(random.randint(1, 99))
    print('pid=[{pid}] '
          'old_value=[{old_value:2}] '
          'new_value=[{new_value:2}] '
          'get_value=[{get_value:2}]'.format(
          pid=str(os.getpid()),
          old_value=old_value,
          new_value=new_value,
          get_value=get_value()))

def get_value():
    global global_variable
    return global_variable

def set_value(new_value):
    global global_variable
    global_variable = new_value

global_variable = -1

print('before set_value(), get_value() = [%s]' % get_value())
set_value(new_value=-2)
print('after  set_value(), get_value() = [%s]' % get_value())

processPool = multiprocessing.Pool(processes=5)
processPool.map(func=worker, iterable=range(15))

Output:

before set_value(), get_value() = [-1]
after  set_value(), get_value() = [-2]
pid=[53970] old_value=[-2] new_value=[ 0] get_value=[23]
pid=[53971] old_value=[-2] new_value=[ 1] get_value=[42]
pid=[53970] old_value=[23] new_value=[ 4] get_value=[50]
pid=[53970] old_value=[50] new_value=[ 6] get_value=[14]
pid=[53971] old_value=[42] new_value=[ 5] get_value=[31]
pid=[53972] old_value=[-2] new_value=[ 2] get_value=[44]
pid=[53973] old_value=[-2] new_value=[ 3] get_value=[94]
pid=[53970] old_value=[14] new_value=[ 7] get_value=[21]
pid=[53971] old_value=[31] new_value=[ 8] get_value=[34]
pid=[53972] old_value=[44] new_value=[ 9] get_value=[59]
pid=[53973] old_value=[94] new_value=[10] get_value=[87]
pid=[53970] old_value=[21] new_value=[11] get_value=[21]
pid=[53971] old_value=[34] new_value=[12] get_value=[82]
pid=[53972] old_value=[59] new_value=[13] get_value=[ 4]
pid=[53973] old_value=[87] new_value=[14] get_value=[70]

user2876408 ,

As it turns out the answer is always simple.

Here is a small sample module with a simple way to show it in a main definition:

def five(enterAnumber,sumation):
    global helper
    helper  = enterAnumber + sumation

def isTheNumber():
    return helper

Here is how to show it in a main definition:

import TestPy

def main():
    atest  = TestPy
    atest.five(5,8)
    print(atest.isTheNumber())

if __name__ == '__main__':
    main()

This simple code works just like that, and it will execute. I hope it helps.

gxyd , 2014-12-04 06:27:43

What you are saying is to use the method like this:
globvar = 5

def f():
    var = globvar
    print(var)

f()  # Prints 5

But the better way is to use the global variable like this:

globavar = 5
def f():
    global globvar
    print(globvar)
f()   #prints 5

Both give the same output.

Mohamed El-Saka , 2014-12-20 12:45:26

You need to reference the global variable in every function you want to use.

As follows:

var = "test"

def printGlobalText():
    global var #wWe are telling to explicitly use the global version
    var = "global from printGlobalText fun."
    print "var from printGlobalText: " + var

def printLocalText():
    #We are NOT telling to explicitly use the global version, so we are creating a local variable
    var = "local version from printLocalText fun"
    print "var from printLocalText: " + var

printGlobalText()
printLocalText()
"""
Output Result:
var from printGlobalText: global from printGlobalText fun.
var from printLocalText: local version from printLocalText
[Finished in 0.1s]
"""

Kylotan , 2009-01-09 11:56:19

You're not actually storing the global in a local variable, just creating a local reference to the same object that your original global reference refers to. Remember that pretty much everything in Python is a name referring to an object, and nothing gets copied in usual operation.

If you didn't have to explicitly specify when an identifier was to refer to a predefined global, then you'd presumably have to explicitly specify when an identifier is a new local variable instead (for example, with something like the 'var' command seen in JavaScript). Since local variables are more common than global variables in any serious and non-trivial system, Python's system makes more sense in most cases.

You could have a language which attempted to guess, using a global variable if it existed or creating a local variable if it didn't. However, that would be very error-prone. For example, importing another module could inadvertently introduce a global variable by that name, changing the behaviour of your program.

Sagar Mehta ,

Try this:
def x1():
    global x
    x = 6

def x2():
    global x
    x = x+1
    print x

x = 5
x1()
x2()  # output --> 7

Martin Thoma , 2017-04-07 18:52:13

In case you have a local variable with the same name, you might want to use the globals() function .
globals()['your_global_var'] = 42

, 2015-10-24 15:46:18

Following on and as an add on, use a file to contain all global variables all declared locally and then import as :

File initval.py :

Stocksin = 300
Prices = []

File getstocks.py :

import initval as iv

def getmystocks(): 
    iv.Stocksin = getstockcount()


def getmycharts():
    for ic in range(iv.Stocksin):

Mike Lampton , 2016-01-07 20:41:19

Writing to explicit elements of a global array does not apparently need the global declaration, though writing to it "wholesale" does have that requirement:
import numpy as np

hostValue = 3.14159
hostArray = np.array([2., 3.])
hostMatrix = np.array([[1.0, 0.0],[ 0.0, 1.0]])

def func1():
    global hostValue    # mandatory, else local.
    hostValue = 2.0

def func2():
    global hostValue    # mandatory, else UnboundLocalError.
    hostValue += 1.0

def func3():
    global hostArray    # mandatory, else local.
    hostArray = np.array([14., 15.])

def func4():            # no need for globals
    hostArray[0] = 123.4

def func5():            # no need for globals
    hostArray[1] += 1.0

def func6():            # no need for globals
    hostMatrix[1][1] = 12.

def func7():            # no need for globals
    hostMatrix[0][0] += 0.33

func1()
print "After func1(), hostValue = ", hostValue
func2()
print "After func2(), hostValue = ", hostValue
func3()
print "After func3(), hostArray = ", hostArray
func4()
print "After func4(), hostArray = ", hostArray
func5()
print "After func5(), hostArray = ", hostArray
func6()
print "After func6(), hostMatrix = \n", hostMatrix
func7()
print "After func7(), hostMatrix = \n", hostMatrix

Rafaël Dera ,

I'm adding this as I haven't seen it in any of the other answers and it might be useful for someone struggling with something similar. The globals() function returns a mutable global symbol dictionary where you can "magically" make data available for the rest of your code. For example:
from pickle import load
def loaditem(name):
    with open(r"C:\pickle\file\location"+"\{}.dat".format(name), "rb") as openfile:
        globals()[name] = load(openfile)
    return True

and

from pickle import dump
def dumpfile(name):
    with open(name+".dat", "wb") as outfile:
        dump(globals()[name], outfile)
    return True

Will just let you dump/load variables out of and into the global namespace. Super convenient, no muss, no fuss. Pretty sure it's Python 3 only.

llewellyn falco , 2017-08-19 08:48:27

Reference the class namespace where you want the change to show up.

In this example, runner is using max from the file config. I want my test to change the value of max when runner is using it.

main/config.py

max = 15000

main/runner.py

from main import config
def check_threads():
    return max < thread_count

tests/runner_test.py

from main import runner                # <----- 1. add file
from main.runner import check_threads
class RunnerTest(unittest):
   def test_threads(self):
       runner.max = 0                  # <----- 2. set global 
       check_threads()

[Nov 21, 2019] Python Gotchas

Nov 21, 2019 | web.archive.org
What is a "gotcha"?

The word "gotcha" started out as the expression "Got you!" This is something that someone who speaks idiomatic American English might say when he succeeds in playing a trick or prank on someone else. "I really got you with that trick!"

The expression "Got you!" is pronounced "Got ya!" or "Got cha!".

Among computer programmers, a "gotcha" has become a term for a feature of a programming language that is likely to play tricks on you to display behavior that is different than what you expect.

Just as a fly or a mosquito can "bite" you, we say that a gotcha can "bite" you.

This is a page devoted to Python "gotchas". Python is a very clean and intuitive language, so it hasn't got many gotchas, but it still has a few that often bite beginning Python programmers. My hope is that if you are warned in advance about these gotchas, you won't be bit quite so hard!

Note that a gotcha isn't necessarily a problem in the language itself. Rather, it is a situation in which there is a mismatch between the programmer's expections of how the language will work, and the way the language actually does work. Often, the source of a gotcha lies not in the language, but in the programmer. Part of what creates a programmer's expectations is his own personal background. A programmer with a Windows or mainframe background, or a background in COBOL or the Algol-based family of languages (PL/1, Pascal, etc.), is especially prone to experiencing gotchas in Python, a language that evolved in a Unix environment and incorporates a number of conventions of the C family of programming languages (C, C++, Java).

If you're such a programmer, don't worry. There aren't many Python gotchas. Keep learning Python. It is a great language, and you'll soon come to love it.

Other Lists of Python Gotchas

Table of Contents
1 Backslashes are escape characters
2 "raw" strings and backslashes (when dealing with Windows filenames)
3 Integer division
4 A comma at the end of a print statement prevents a newline from being written... but appears to write a trailing space.
5 Omitting parentheses when invoking a method
6 Mutable defaults for function/method arguments
6.1 The Solution
6.2 Never use a mutable object -- that is: a list, a dictionary, or a class instance -- as the default value of an argument.
6.3 Ignore rule 1 only if you really, really , REALLY know what you're doing.
1 Backslashes are escape characters

This is a language feature that is so common on Unix that Unix programmers never think twice about it. Certainly, a Unix programmer would never consider it to be a gotcha. But for someone coming from a Windows background, it may very well be unfamiliar.

The gotcha may occur when you try to code a Windows filename like this:

myFilename = "c:\newproject\typenames.txt"
myFile = open(myFilename, "r")

and -- even though the input file exists -- when you run your program, you get the error message

IOError: [Errno 2] No such file or directory: 'c:\newproject\typenames.txt'

To find out what's going on, you put in some debugging code:

myFilename = "c:\newproject\typenames.txt"
print "(" + myFilename + ")"

And what you see printed on the console is:

(c:
ewproject       ypenames.txt)

What has happened is that you forgot that in Python (as in most languages that evolved in a Unix environment) in quoted string literals the backslash has the magical power of an escape character . This means that a backslash isn't interpreted as a backslash, but as a signal that the next character is to be given a special interpretation. So when you coded

myFilename = "c:\newproject\typenames.txt"

the "\n" that begins "\newproject" was interpreted as the newline character, and the "\t" that begins "\typenames.txt" was interpreted as the tab character. That's why, when you printed the filename, you got the result that you did. And it is why Python couldn't find your file -- because no file with the name c: (newline) ewproject (tab) ypenames.txt could be found.

To put a backslash into a string, you need to code two backslashes -- that is, the escape character followed by a backslash. So to get the filename that you wanted, you needed to code

myFilename = "c:\\newproject\\typenames.txt"

And under some circumstances, if Python prints information to the console, you will see the two backslashes rather than one. For example, this is part of the difference between the repr() function and the str() function.

myFilename = "c:\\newproject\\typenames.txt"
print repr(myFilename), str(myFilename)

produces

'c:\\newproject\\typenames.txt' c:\newproject\typenames.txt

Escaped characters are documented in the Python Language Reference Manual. If they are new to you, you will find them disconcerting for a while, but you will gradually grow to appreciate their power.

2 "raw" strings and backslashes (when dealing with Windows filenames)

Once upon a time there was a beatiful Windows programmer named Red Ridinghood.

One day, Red's supervisor told her that they were going to start building a new application called GrandmasHouse. The feature list for the application was so long that they would never have attempted to get to GrandmasHouse if they hadn't learned about a shortcut through Python Woods that would make the journey much shorter.

So Red started working her way through Python, and indeed found the going quick and easy. She loved the woods, and was happy to be travelling in them.

There was only one problem. Her programs did a lot of file manipulation, and so she had to do a lot of coding of filenames. Windows filenames used a backslash as a separator, but within Python the backslash had the magic power of an escape character , so every time she wanted a backslash in a filename she had to code two backslashes, like this:

myServer = "\\\\aServer" # ==> \\aServer
myFilename = myServer + "\\aSharename\\aDirName\\aFilename"

This feature of Python got very old very quickly. Red started calling it The Wolf, and it was the one part of Python that she hated.

One day as she was walking through the forest, she came to a clearing. In the clearing was a charming little pub, and inside the pub she met a tall, dark, and handsome stranger named Rawstrings.

Rawstrings said he could save her from The Wolf. All she had to do, he said, was to put an "r" in front of her quoted string literals. This would change them from escaped strings into raw strings. The backslash would lose its powers as an escape character, and become just an ordinary character. For example, with raw strings, you could code r"\t" and you wouldn't get a string contining a single tab character -- you would get a string containing the backslash character followed by "t".

So Red, instead of coding

myServer = "\\\\aServer"

could just code

myServer = r"\\aServer"

Red was seduced by the things that Rawstrings was telling her, and she began to spend a lot of time in his company.

Then one day, she coded

myDirname = r"c:\aDirname\"

and her program blew up with the following message:

myDirname = r"c:\aDirname\"
                         ^
SyntaxError: invalid token

After some experimenting, she discovered that -- contrary to what Rawstrings had told her -- the backslash seemingly hadn't lost all of its magic powers after all. For example, she could code:

aString = r"abc\"xyz"; print aString

When she did this, it seemed perfectly legal. The double-quote just before "xyz" did not close the raw string at all. Somehow the backslash seemed to protect it -- it wasn't recognized as the closing delimiter of the raw string, but was included in the string. When she coded print aString , she got

abc\"xyz

It was this protective character that the backslash had acquired that made

myDirname = r"c:\aDirname\"

blow up. The final backslash was protecting the closing double-quote, so it was not being recognized as a closing quote. And since there was nothing after the double-quote, the raw string was not closed, and she got an error. She tried coding the raw string with two backslashes at the end -- as if the backslash was an escape character --

myDirname = r"c:\aDirname\\"

but that didn't do it either. Instead of getting the single closing backslash that she wanted, she got two backslashes at the end:

c:\aDirname\\

She was in despair. She couldn't figure out any way to use raw strings to put a single backslash at the end of a string, and she didn't want to have to go back to fighting The Wolf.

Fortunately, at this point she confided her troubles to Paul Woodman, a co-worker who had started exploring Python a few months earlier. Here is what he told her.

In raw strings, backslashes do not have the magical power of escape characters that they do in regular strings. But they don't lose all of their magical powers.

In raw strings -- as you discovered -- backslashes have the magical power of protection characters . Basically, this means that a backslash protects any character that follows it from being recognized as the closing delimiter of the raw string.

Coming from a Windows programming background, you assumed that support for raw strings was a feature whose purpose was to make the work of coding Windows filenames easier by removing the magical escape character powers from the backslash. And you were surprised to discover that raw strings aren't truly raw in the way that you expected -- raw in the sense that the backslash had no magical powers.

The reason for the special powers of backslashes in raw strings is that -- contrary to what you assumed -- raw strings were not developed to make it easier for Windows programmers to code filenames containing backslash characters. In fact, raw strings were originally developed to make the work of coding regular expressions easier. In raw strings, the backslash has the magical power of a protection character because that is just the kind of behavior it needs to have in order to make it easier to code regular expressions. The feature that you can't end a raw string with a single backslash is not a bug. It is a feature, because it is not legal to end a regular expression with a single backslash (or an odd number of backslashes).

Unfortunately for you, this power makes it impossible to create a raw string that ends in a single backslash, or in an odd number of backslashes. So raw expressions won't do what you want them to, namely save you from The Wolf.

But don't give despair! There is a way...

In Python, there are a number of functions in the os.path module that change forward slashes in a string to the appropriate filename separator for the platform that you are on. One of these function is os.path.normcase() . The trick is to enter all of your filename strings using forward slashes, and then let os.path.normcase() change them to backslashes for you, this way.

myDirname = os.path.normcase("c:/aDirname/")

It takes a bit of practice to get into the habit of specifying filenames this way, but you'll find that you adapt to it surprisingly easily, and you'll find it a lot easier than struggling with The Wolf.

Red was super happy to hear this. She transferred to Woodman's project team, and they all coded happily ever after!

3 Integer division

This is the feature of Python that (along with case-sensitivity) most frequently bites students who are learning Python as their first programming language.

The gotcha is this -- if you divide an integer by an integer, you get an integer. If the division produces a remainder, the result is truncated downward. (Note that this is true truncation, not rounding toward zero.)

print 5/3     # produces 1
print (-5)/3  # produces -2
print 5/3.0   # produces 1.66666666667, because 3.0 is a float, not an integer

This behavior is called "classic" division, and is something that Python inherited from the C programming language. What one would like, of course, is for Python division to behave more intuitively, like "true" division:

print 5/3      # produces  1.66666666667
print (-5)/3   # produces -1.66666666667

Starting in Python 2.2, the process of slowly changing the behavior of the division operator began. First, a new floor operator // was introduced to provide division with downward truncation of the results. Second, a technique was introduced for a programmer to make the division operator behave in the more intuitive "true division" fashion. In order to get true division, you need to put this statement at the top of your module.

from __future__ import division

In Python version 2.3, Python will continue to use old-style classic division by default, but will also issue a warning whenever division is applied to two integers. You can use this to find code that's affected by the change and fix it. The fix -- depending on what you want your program to be doing -- will mean either changing the / division operator to the // floor operator, or adding the from __future__ import division statement to your module.

Eventually (in Python version 3) the new "true division" behavior for the division operator will become standard, and you won't need the from __future__ import division statement.

The bottom line is that you should not be running a version of Python older than 2.2, and should upgrade to 2.3 as soon as possible, and you should put from __future__ import division at the beginning of all of your modules.

For more information on this subject, see section 6 of What's New in Python 2.2 .

4 A comma at the end of a print statement prevents a newline from being written... but appears to write a trailing space.

It doesn't really write a trailing space -- it just causes an immediately subsequent print statement to write a leading space!

The Python Language Reference Manual says, about the print statement,

A "\n" character is written at the end, unless the print statement ends with a comma.

But it also says that if two print statements in succession write to stdout, and the first one ends with a comma (and so doesn't write a trailing newline), then the second one prepends a leading space to its output. (See section "6.6 The print statement" in the Python Reference Manual. Thanks to Marcus Rubenstein and Hans Meine for pointing this out to me.)

So

for i in range(10): print "*",
print

produces

* * * * * * * * * *

If you want to print a string without any trailing characters at all, your best bet is to use sys.stdout.write()

import sys
for i in range(10): sys.stdout.write("*")
sys.stdout.write("\n")

produces

**********
5 Omitting parentheses when invoking a method

In Python, omitting the trailing parentheses from the end of a method call (one that takes no arguments) is not a syntax error.

The place where this most frequently bites me is with the "close" method on file objects. Suppose you have an output file called "foo" and you want to close it. The correct way to do this is:

foo.close()

However, if you accidentally omit the trailing parentheses, and code this:

foo.close

Python will not report a syntax error, because this is not an error in Python. In Python, this is a perfectly legitimate statement that returns the method object "close". (Remember that methods are first-class objects in Python.) If you do this in the Python interpreter, you will get a message like this:

<built-in method close of file object at 0x007E6AE0>

The nastiness about this gotcha is that if you fail to code the trailing parentheses on a "close" method for an output file, the output file will not be closed properly. The file's output buffer will not be flushed out to disk, and the part of the output stream that was still left in the output buffer will be lost. After your program finishes, part of your output file will be missing, and you won't know why.

The best way of dealing with this gotcha is just to be aware that it can be a problem, and to be alert. Be careful to code the parenthese on your method calls, and especially careful to code them on calls to the "close" method of file objects.

And if you find yourself with an output file that seems to be inexplicably truncated, your first thought should be to check for missing parentheses in the file.close() statement that closes the file.

Programs like PyChecker and PyLint may be able to detect this kind of error, which is one good reason to use them.

6 Mutable defaults for function/method arguments

There's a Python gotcha that bites everybody as they learn Python. In fact, I think it was Tim Peters who suggested that every programmer get caught by it exactly two times. It is call the mutable defaults trap. Programmers are usually bit by the mutable defaults trap when coding class methods, but I'd like to begin with explaining it in functions, and then move on to talk about class methods.

The gotcha occurs when you are coding default values for the arguments to a function or a method. Here is an example for a function named functionF :

def functionF(argString = "abc", argList = []):

Here's what most beginning Python programmers believe will happen when functionF is called without any arguments:

A new string object containing "abc" will be created and bound to the "argString" variable name. A new, empty list object will be created and bound to the "argList" variable name. In short, if the arguments are omitted by the caller, the functionF will always get "abc" and [] in its arguments.

This, however, is not what will happen. Here's why.

The objects that provide the default values are not created at the time that functionF is called. They are created at the time that the statement that defines the function is executed . If functionF , for example, is contained in a module named moduleM , then the statement that defines functionF will probably be executed at the time when moduleM is imported.

When the def statement that creates functionF is executed:

After that, whenever functionF is called without arguments, argString will be bound to the default string object, and argList will be bound to the default list object. In such a case, argString will always be "abc", but argList may or may not be an empty list. Here's why.

There is a crucial difference between a string object and a list object. A string object is immutable, whereas a list object is mutable. That means that the default for argString can never be changed, but the default for argList can be changed.

Let's see how the default for argList can be changed. Here is a program. It invokes functionF four times. Each time that functionF is invoked it displays the values of the arguments that it receives, then adds something to each of the arguments.

def functionF(argString="abc", argList = []):
        print argString, argList
        argString = argString + "xyz"
        argList.append("F")

for i in range(4): functionF()

The output of this program is:

abc []
abc ['F']
abc ['F', 'F']
abc ['F', 'F', 'F']

As you can see, the first time through, the argument have exactly the default that we expect. On the second and all subsequent passes, the argString value remains unchanged -- just what we would expect from an immutable object. The line

argString = argString + "xyz"

creates a new object -- the string "abcxyz" -- and binds the name "argString" to that new object, but it doesn't change the default object for the argString argument.

But the case is quite different with argList, whose value is a list -- a mutable object. On each pass, we append a member to the list, and the list grows. On the fourth invocation of functionF -- that is, after three earlier invocations -- argList contains three members.

6.1 The Solution

This behavior is not a wart in the Python language. It really is a feature, not a bug. There are times when you really do want to use mutable default arguments. One thing they can do (for example) is retain a list of results from previous invocations, something that might be very handy.

But for most programmers -- especially beginning Pythonistas -- this behavior is a gotcha. So for most cases we adopt the following rules.

  1. 6.2 Never use a mutable object -- that is: a list, a dictionary, or a class instance -- as the default value of an argument.
  2. 6.3 Ignore rule 1 only if you really, really , REALLY know what you're doing.

So... we plan always to follow rule #1. Now, the question is how to do it... how to code functionF in order to get the behavior that we want.

Fortunately, the solution is straightforward. The mutable objects used as defaults are replaced by None, and then the arguments are tested for None.

def functionF(argString="abc", argList = None):
        if argList is None: argList = []
        ...

Another solution that you will sometimes see is this:

def functionF(argString="abc", argList=None):
        argList = argList or []
        ...

This solution, however, is not equivalent to the first, and should be avoided. See Learning Python p. 123 for a discussion of the differences. Thanks to Lloyd Kvam for pointing this out to me.

And of course, in some situations the best solution is simply not to supply a default for the argument.

Now let's look at how the mutable arguments gotcha presents itself when a class method is given a mutable default for one of its arguments. Here is a complete program.

# define a class for company employees
class Employee:

        def __init__ (self, argName, argDependents=[]):
                # an employee has two attributes: a name, and a list of his dependents
                self.name = argName
                self.Dependents = argDependents

        def addDependent(self, argName):
                # an employee can add a dependent by getting married or having a baby
                self.Dependents.append(argName)

        def show(self):
                print
                print "My name is.......: ", self.name
                print "My dependents are: ", str(self.Dependents)


#---------------------------------------------------
#   main routine -- hire employees for the company
#---------------------------------------------------

# hire a married employee, with dependents
joe = Employee("Joe Smith", ["Sarah Smith", "Suzy Smith"])

# hire a couple of unmarried employess, without dependents
mike = Employee("Michael Nesmith")
barb = Employee("Barbara Bush")

# mike gets married and acquires a dependent
mike.addDependent("Nancy Nesmith")

# now have our employees tell us about themselves
joe.show()
mike.show()
barb.show()

Let's look at what happens when this program is run. First, the code that defines the Employee class is run. Then we hire Joe. Joe has two dependents, so that fact is recorded at the time that the joe object is created. Next we hire Mike and Barb. Then Mike acquires a dependent. Finally, the last three statements of the program ask each employee to tell us about himself. Here is the result.

My name is.......:  Joe Smith
My dependents are:  ['Sarah Smith', 'Suzy Smith']

My name is.......:  Michael Nesmith
My dependents are:  ['Nancy Nesmith']

My name is.......:  Barbara Bush
My dependents are:  ['Nancy Nesmith']

Joe is just fine. But somehow, when Mike acquired Nancy as his dependent, Barb also acquired Nancy as a dependent. This of course is wrong. And we're now in a position to understand what is causing the program to behave this way.

When the code that defines the Employee class is run, objects for the class definition, the method definitions, and the default values for each argument are created. The constructor has an argument argDependents whose default value is an empty list, so an empty list object is created and attached to the __init__ method as the default value for argDependents .

When we hire Joe, he already has a list of dependents, which is passed in to the Employee constructor -- so the argDependents attribute does not use the default empty list object.

Next we hire Mike and Barb. Since they have no dependents, the default value for argDependents is used. Remember -- this is the empty list object that was created when the code that defined the Employee class was run. So in both cases, the empty list is bound to the argDependents argument, and then -- again in both cases -- it is bound to the self.Dependents attribute. The result is that after Mike and Barb are hired, the self.Dependents attribute of both Mike and Barb point to the same object -- the default empty list object.

When Michael gets married, and Nancy Nesmith is added to his self.dependents list, Barb also acquires Nancy as a dependent, because Barb's self.dependents variable name is bound to the same list object as Mike's self.dependents variable name.

So this is what happens when mutuable objects are used as defaults for arguments in class methods. If the defaults are used when the method is called, different class instances end up sharing references to the same object.

And that is why you should never, never , NEVER use a list or a dictionary as a default value for an argument to a class method. Unless, of course, you really, really , REALLY know what you're doing.

[Nov 21, 2019] Mutable default arguments

Nov 21, 2019 | archive.is

This one bites beginners over and over again. It's really a variant of #2, combined with unexpected behavior of default arguments. Consider this function:

>>> def popo(x=[]):
...     x.append(666)
...     print x
...     
>>> popo([1, 2, 3])
[1, 2, 3, 666]
>>> x = [1, 2]
>>> popo(x)
[1, 2, 666]
>>> x
[1, 2, 666]

This was expected. But now:

>>> popo()
[666]
>>> popo()
[666, 666]
>>> popo()
[666, 666, 666]

Maybe you expected that the output would be [666] in all cases... after all, when popo() is called without arguments, it takes [] as the default argument for x, right? Wrong. The default argument is bound *once*, when the function is *created*, not when it's called. (In other words, for a function f(x=[]) , x is *not* bound whenever the function is called. x got bound to [] when we defined f , and that's it.) So if it's a mutable object, and it has changed, then the next function call will take this same list (which has different contents now) as its default argument.

Solution: This behavior can occasionally be useful. In general, just watch out for unwanted side effects.

[Nov 21, 2019] The += operator

Nov 21, 2019 | archive.is

In languages like C, augmented assignment operators like += are a shorthand for a longer expression. For example,

x += 42;

is syntactic sugar for

x = x + 42;

So, you might think that it's the same in Python. Sure enough, it seems that way at first:

a = 1
a = a + 42
# a is 43
a = 1
a += 42
# a is 43

However, for mutable objects, x += y is not necessarily the same as x = x + y. Consider lists:

>>> z = [1, 2, 3]
>>> id(z)
24213240
>>> z += [4]
>>> id(z)
24213240
>>> z = z + [5]
>>> id(z)
24226184

x += y changes the list in-place, having the same effect as the extend method. x = x + y creates a new list and rebinds it to x, which is something else. A subtle difference that can lead to subtle and hard-to-catch bugs.

Not only that, it also leads to surprising behavior when mixing mutable and immutable containers:

>>> t = ([],)
>>> t[0] += [2, 3]
Traceback (most recent call last):
  File "<input>", line 1, in ?
TypeError: object doesn't support item assignment
>>> t
([2, 3],)

Sure enough, tuples don't support item assignment -- but after applying the +=, the list inside the tuple *did* change! The reason is again that += changes in-place. The item assignment doesn't work, but when the exception occurs, the item has already been changed in place.

This is one pitfall that I personally consider a wart. emoticon:smile

Solution: depending on your stance on this, you can: avoid += altogether; use it for integers only; or just live with it. :-)

[Nov 21, 2019] https://www.reddit.com/r/changemyview/comments/5xvwx4/cmv_python_is_a_terrible_programming_language_for/

Nov 21, 2019 | www.reddit.com

level 1


FJ_van_Fleppensteijn 8 points · 2 years ago

I disagree with OP because

So I shall first argue against your points and then say why python is even more awful than you thin.:

  1. Really, privateness via underscores, are you really talking about syntax? What does that matter? It means you consume one less keyword. I wish more languages just automatically made any binding that started with an underscore private but unlike Python actually enforced it.

  2. Dynamic typing has real advantages over static typing. At least over static pessimistic typing. We've all been in the situation with statically typed languages where you know it's safe, you can prove it's safe but the compiler can't verify your proof and know it's safe so it will reject it. Pessimistic static typing which is used by almost all statically typed languages does not reject when it can prove it is unsafe, but when it can't prove it to be safe. While a lot of modern languages have greatly more expressive static type systems like say Idris they still cannot do the same as dynamic typing in terms of flexibility.

  3. Switch: I agree

  4. Scopes aren't determined by indentation, python has no block scoping, grouping is determined by intendation but this does not create a new scope in python at all. The only way to create a new scope in Python is a function. And again, this is purely syntax and largely irrelevant.

Now onto why Python sucks more than you think and why it is clear that Guido just bashed together a bunch of features from many languages without fully understanding most of them:

  1. As said, python has no actual block scoping. In python3 they had to hack around this by introducing the nonlocal keyword, this means that for people who expect block scoping it becomes surprising such as that for x in ... in Python does not intorduce a new scope which you might think. x actually is assigned to any variable in the outer scope and continues to exist after the loop unless you del it afterwards, it does not shadow any existing variable at all. The only way to crudely simulate scope is to del variables which is obviously crude. Shadowing does not exist in Python as well requiring that Python variables be unecessarily long to avoid conflict

  2. Assignment in python is the same as initialization, you initialize a variable by assigning to a nonexistent one. Together with long variable names this means that a simple typo in Python is not an error the implementation can diagnose, it just creates a new variable. Reading from an unitialized variable is diagnosed by the implementation as an exception.

  3. Python does not believe in enforcing privacy, purely treating it as a convention. Because underscored variables are visible many optimization techniques are now removed. Privacy isn't just a code organization thing, it means the implementation can apply farmore aggressive optimizations in the real of inlining and just removing th eoriginal definition as it's not exported

  4. This ties into python's ridiculously poor module system. If you do import string in a module foo.py then foo.string technically becomes part of your modules public interface. Probably not something you want, you should do import string as _string to avoid this. But no one does this and how can you blame people when official examples in the documentation don't do this either.

  5. Furthermore, since Python's search path includes the directory the script is located in. If bar.py is located in /usr/bin and you run /usr/bin/python-script which imports bar then it tries to import bar.py first. So if you put common-module-name.py inside of /usr/bin you can watch every Python script in there break. Yap, Python basically decides for you what other names you can put in your PATH.

  6. Exceptions for flow-control, oh my god. This is the "pythonic" way and the cause of numerous bugs because exceptions-for-flow-control are basically gotos. Guido in general does not seem to know or care about what the difference between an error an exceptional situation and a logical branch is and gleefully uses the same mechanism for all three not realizing how different ones are appropriate in different cases. I really like how Rust basically has two different forms of exceptions, calling one "panics" where the convention is that a panic is caused by an error of the programmer and that a panic in a well behaving program should never occur. A panic is a bug, no excuse.

  7. No tail call elimination, apparently by design because Guido doesn't want people to use it. He also at first misunderstood the concept thinking it would only eliminate calls in direct recursion until people pointed out that TCE eliminates every tail call allowing even complicated mutually recursive definitions to run in constant stack space.

  8. No character data type, not making a distinction between a string of length 1 and a character is stupid.

  9. Too many falsy values leading to easy bugs. Most dynamically typed languages have only one or two falsy values. Really only False and None should be falsy, the rest truthy. This eliminates a goddamn amount of bugs and makes code clearer. if foo: tells me little about what you test for. if not foo.is_empty(): makes it clear we are testing for the emptiness of some collection.

  10. Python functions go so far to even expose the names of formal paramatres as part of their interface, no one underscores those meaning you can't even change the name of paramatres in your module without changing the interface. Python really does not believe in keeping implementation seperate from interface. Python really has no positional paramatres, only keyword arguments that act like they are positional.

  11. Python is slow, far slower than other dynamically typed programming languages.

level 2
jzpenny 42∆ 1 point · 2 years ago
· edited 2 years ago

Scopes aren't determined by indentation, python has no block scoping, grouping is determined by intendation but this does not create a new scope in python at all. The only way to create a new scope in Python is a function. And again, this is purely syntax and largely irrelevant.

Huh? You might be right in a semantic sense: the person was really talking about grouping rather than scoping, but it's super relevant . Python is a language that, in the age of the ubiquitous HTML-driven web, cannot be cut and pasted from essentially any browser into an editor and executed successfully, if there is any indention/code grouping at all . That's... uhh. Certainly a "bold" design decision.

To add a few beefs that I haven't seen mentioned:

(All that said, it's still a powerful language and it has its pluses, too) level 3

FJ_van_Fleppensteijn 1 point · 2 years ago

Huh? You might be right in a semantic sense: the person was really talking about grouping rather than scoping, but it's super relevant. Python is a language that, in the age of the ubiquitous HTML-driven web, cannot be cut and pasted from essentially any browser into an editor and executed successfully, if there is any indention/code grouping at all. That's... uhh. Certainly a "bold" design decision.

THat's true, but if you want to integrate it you're going to have to do some formatting anyway.

I think it's pretty easy to cut and paste, replace tabs with spaces or in reverseand proeprly do the indenting with a couple of quick hotkeys to make it work. Takes 10 seconds.

Referential ambiguity leading to non-obvious behavior in even very common tasks like for loops iterating over lists of objects

How is that ambiguous? It invokes the iteration protocol. THe data structure itself in its documentation defines over what it iterates and usually this is quite obvious anyway.

Novel syntactic constructions like list comprehensions are not easy to learn or obvious to apply, and in many cases appear to amount to little than band-aids around core deficiencies in the language

That these comprehensions are unusual to you almost makes me think you never used anything but C and Java. Comprehensions are found in quite a few languages.

The problem with Python is that they work poorly since Python is not an expression-oriented language. They had to add the x if y else z syntax which is bizarre to give it a modicum of workability and even that doesn't really solve it.

When we have this many core data types floating around and so many subtle typing issues between them, why not just have sigils? Is this really an innovation? I realize it's more C-like, but what's good for C may not be that great for a language that's ultimately so very unlike it. Coding OPP (other peoples' python) is so often a frustrating experience when you get down into the guts because it's so much teasing out of non-obviousness here.

Because there are far many data types than there are possible sigils.

Data types in Python like in most modern languages are an open class. You can define a new data type which transparently works the same as if it was built in. So the list is infinite. level 4

jzpenny 42∆ 1 point · 2 years ago

THat's true, but if you want to integrate it you're going to have to do some formatting anyway.

Oh come on. I can cut and paste perl from a perl poetry website and it'll run just fine. I can modify it without changing any formatting, will run just fine. It's a major oversight and pain with python that, as a developer, one eventually develops somewhat second-nature workarounds for, but the fact that one has to is still a serious weakness. How many people begin their journey into programming in this very natural way, cutting and pasting code, only to stop right there because of crap like this?

I think it's pretty easy to cut and paste, replace tabs with spaces or in reverseand proeprly do the indenting with a couple of quick hotkeys to make it work.

In general, it is, especially if you have the correctly formatted source as a guide. But turning whitespace into a significant part of the code makes maintaining the formatting essential to correct code execution, and again, that's just a needless imposition. There are no meaningful advantages to forcing everyone to do it in this way. C doesn't do this.

How is that ambiguous? It invokes the iteration protocol. THe data structure itself in its documentation defines over what it iterates and usually this is quite obvious anyway.

I can provide you some examples, but for example, try creating a list of objects, then looping over that list and selectively deleting from it. It plain doesn't work correctly, if you're using the list delete features. Now I realize that this is not the "pythonic" way to do things: that'd be a list comprehension. But for a coder coming from other languages, Python's behavior here is, even if noted in some of the obscure and not always exactly readable documentation, subtle and tricky behavior.

That these comprehensions are unusual to you almost makes me think you never used anything but C and Java. Comprehensions are found in quite a few languages.

Uhm, aside from boutique languages (and C#), we're basically into lisp and erlang territory, there. And as you say, it's a much more natural syntax in those expression-based languages where the entry points and structure are much more clear.

Because there are far many data types than there are possible sigils

Basic data types? No there aren't, and anything that isn't one of those basic datatype is just a complex data structure made of basic datatype legos. Take some generic code. What is the variable "calendar.day"? Is it a class method returning an object, a method returning a string, or an int, or is there in fact a variable called 'day' somewhere else? You can make some really confusing Python, this way, incidentally. level 5

FJ_van_Fleppensteijn 1 point · 2 years ago

Oh come on. I can cut and paste perl from a perl poetry website and it'll run just fine. I can modify it without changing any formatting, will run just fine. It's a major oversight and pain with python that, as a developer, one eventually develops somewhat second-nature workarounds for, but the fact that one has to is still a serious weakness. How many people begin their journey into programming in this very natural way, cutting and pasting code, only to stop right there because of crap like this?

Yeah it will run, but come on, you don't want inconsistent formatting and variable names and all that throughout your code. In general you refactor stuff to make it adhere to the house styleguides of the project you are working on anyway.

Others have to deal with that code later. "that it works" is not enough, code is read more often than written.

I can provide you some examples, but for example, try creating a list of objects, then looping over that list and selectively deleting from it. It plain doesn't work correctly, if you're using the list delete features. Now I realize that this is not the "pythonic" way to do things: that'd be a list comprehension. But for a coder coming from other languages, Python's behavior here is, even if noted in some of the obscure and not always exactly readable documentation, subtle and tricky behavior.

Indeed, you can obviously not modify the contents of the thing you iterate over (Rust's type system enforces this statically).

In order to modify it you have to loop the old fashioned way by just incrementing an index which is still there.

Uhm, aside from boutique languages (and C#), we're basically into lisp and erlang territory, there. And as you say, it's a much more natural syntax in those expression-based languages where the entry points and structure are much more clear.

Yes, like I said in my original post. Guido clearly doesn't understand all the fancy features he imports from allover the place. List comprehensions really need an expression-oriented syntax and Guido doesn't like that for some reason.

Basic data types? No there aren't, and anything that isn't one of those basic datatype is just a complex data structure made of basic datatype legos. Take some generic code. What is the variable "calendar.day"? Is it a class method returning an object, a method returning a string, or an int, or is there in fact a variable called 'day' somewhere else? You can make some really confusing Python, this way, incidentally.

The point is that Python does not distinguish between primitive and user defined types and it may change between version, some things which used to be library things defined in python itself became implementation-intrinsic in later versions for superior performance. The difference is transparent for the most part.

[Nov 21, 2019] How to fix Python indentation

Nov 21, 2019 | stackoverflow.com

ephemient ,Feb 28, 2010 at 0:35

Use the reindent.py script that you find in the Tools/scripts/ directory of your Python installation:

Change Python (.py) files to use 4-space indents and no hard tab characters. Also trim excess spaces and tabs from ends of lines, and remove empty lines at the end of files. Also ensure the last line ends with a newline.

Have a look at that script for detailed usage instructions.

[Nov 21, 2019] Python Formatter Tool

Jul 3, 2014 | stackoverflow.com

tricasse ,Jul 3, 2014 at 19:15

I was wondering if there exists a sort of Python beautifier like the gnu-indent command line tool for C code. Of course indentation is not the point in Python since it is programmer's responsibility but I wish to get my code written in a perfectly homogenous way, taking care particularly of having always identical blank space between operands or after and before separators and between blocks.

Mike A ,Mar 1, 2010 at 17:49

I am the one who asks the question. In fact, the tool the closest to my needs seems to be PythonTidy (it's a Python program of course : Python is best served by himself ;) ).

tom ,Sep 29, 2014 at 18:26

autopep8 attempts to automate making your code conform to pep8 coding standards

https://pypi.python.org/pypi/autopep8

Eyal Levin ,Oct 31, 2017 at 9:31

You can also try yapf :

A formatter for Python files

https://github.com/google/yapf/

Vinko Vrsalovic ,Jun 23, 2009 at 12:57

PyLint has some formatting checks.

xxx ,Jun 23, 2009 at 12:57

Have you looked at pindent ?

[Nov 21, 2019] Advanced Python or Understanding Python - YouTube

This guy is a good example of a typical language complexity junkie. He tried to make simple things complex and complex incomprehancible.
Nov 21, 2019 | www.youtube.com

Rajendra Divecha , 6 years ago

Hr Gwea , 1 year ago

1:07:00 seven or eight bytes per character? That is non-sense. The farthest character in the Unicode table (0x10FFFF) can be encoded with just 4 bytes.

nya_nya_boo , 8 years ago div class="co

mment-renderer-text-content expanded"> @ThunderAppeal My complaint deals with what I have to work around and fix every day, all day long. Not with how wonderful and detailed the structural constructs I can produce with this system will be. I would rather read a badly documented but clearly coded hack that takes up ten pages and gets the job done in a slow and plodding manor, then a ten line haiku that does things with wonder and panache....but also takes me 2 hours to dig through to find the misplaced '

Christian Beasley , 8 years ago

@Testerer004 Well, you see, most people don't wait till there 20 an in college to learn programming. I know most of my friends started learning C and creating games in middle school. So, I would say a lot of us are most likely extremely young.

ThunderAppeal , 8 years ago

v> @addmoreice Python is a nice little language when everything already has some kind of shape or form, however in order to message that data with python to prepare it to be processed python is a nightmare. Every little thing a module has to be imported, there has to be a regexp module imported a strings module imported.While its very nice that scope is enforced automatically in order to manipulate truly manipulate data you have to get comfortable with list comprehensions.

ThunderAppeal , 8 years ago

@addmoreice Whilst reading python code you happen across a function (ie func()) and if it isnt predefined in the code somewhere you have to hunt down its definition somewhere in the python module directory. It may be as easy as help(func) at the python prompt but you have to do is so often. Even then the python docs are crufty and crusty, too much is taken for granted. Something that python presumably was supposed to move away from when they copied elements of Perl.

ThunderAppeal , 8 years ago

@addmoreice Even Mark Lutz admits that list comprehensions are ugly, frankly I'm quite fond of them. But again they only work when the data is nice and pretty and neat. Otherwise you have to labor under how python interprets the data and work with it in a very structured way. Additionally python does not have types, it has containers, types are an after thought. Containers take precedence over everything. Perl on the other hand was designed specifically with text processing in mind and

ThunderAppeal , 8 years ago

@addmoreice Again, the problem is not language the problem is *you*. You are mixing up programming languages and documentation. I was trained by my boss to document everything I did, I had to take responsibility for all of my work. Today, 'programmers' like you snivel and whine and bitch and moan like a bunch of little babies. Programming languages dont play nearly as much of a role in 'making a programmers job easy.' as much as you imagine it does. Either someone knows the subject or they

[Nov 21, 2019] You must add parentheses after a function name to call it

Nov 21, 2019 | archive.is

Always Use Parentheses to Call a Function You must add parentheses after a function name to call it, whether it takes arguments or not. That is, use function() , not function . Python functions are simply objects that have a special operation, a call, that you trigger with the parentheses. Like all objects, they can also be assigned to variables, and used indirectly: x = function; x() . In Python training, this seems to occur most often with files. It's common to see beginners type file.close to close a file, rather than file.close() ; because it's legal to reference a function without calling it, the first version without parenthesis succeeds silently, but does not close the file!

[Nov 21, 2019] Passing an Array/List into Python

Dec 17, 2017 | stackoverflow.com

JAL ,Dec 17, 2017 at 8:12

I've been looking at passing arrays, or lists, as Python tends to call them, into a function.

I read something about using *args, such as:

def someFunc(*args)
    for x in args
        print x

But not sure if this is right/wrong. Nothing seems to work as I want. I'm used to be able to pass arrays into PHP function with ease and this is confusing me. It also seems I can't do this:

def someFunc(*args, someString)

As it throws up an error.

I think I've just got myself completely confused and looking for someone to clear it up for me.

Rafał Rawicki ,Feb 13 at 15:08

When you define your function using this syntax:
def someFunc(*args)
    for x in args
        print x

You're telling it that you expect a variable number of arguments. If you want to pass in a List (Array from other languages) you'd do something like this:

def someFunc(myList = [], *args)
    for x in myList:
        print x

Then you can call it with this:

items = [1,2,3,4,5]

someFunc(items)

You need to define named arguments before variable arguments, and variable arguments before keyword arguments. You can also have this:

def someFunc(arg1, arg2, arg3, *args, **kwargs)
    for x in args
        print x

Which requires at least three arguments, and supports variable numbers of other arguments and keyword arguments.

JoshD ,Oct 18, 2010 at 20:28

You can pass lists just like other types:
l = [1,2,3]

def stuff(a):
   for x in a:
      print a


stuff(l)

This prints the list l. Keep in mind lists are passed as references not as a deep copy.

Gintautas Miliauskas ,Oct 18, 2010 at 16:14

Python lists (which are not just arrays because their size can be changed on the fly) are normal Python objects and can be passed in to functions as any variable. The * syntax is used for unpacking lists, which is probably not something you want to do now.

,

You don't need to use the asterisk to accept a list.

Simply give the argument a name in the definition, and pass in a list like

def takes_list(a_list):
    for item in a_list:
         print item

[Nov 21, 2019] http://web.archive.org/web/20031002184114/www.amk.ca/python/writing/warts.html

Nov 21, 2019 | web.archive.org

Python has 3 built-in functions, map() , filter() , and reduce() , that provide some support for the functional style of programming. In versions of Python before 2.2, using these functions sometimes required a little hack with default arguments.

Let's choose one particular function for our example: filter(func, L) loops over the list L , returning a new list containing all the elements of L for which func(element) returned true. Take the situation where you have a list of numbers L , and want to find all the objects that are even numbers (multiples of 2). You could write this as:

L2 = filter(lambda N: (N % 2) == 0, L)

In case you're not familiar with the lambda statement: it's defining a little function without a name that takes a single parameter N and returns the value of the comparison expression.

L2 is then set to a list containing all of the even numbers in L :

>>> L = [0, 5, 4, 32.5, 17, -6]
>>> L2 = filter(lambda N: (N % 2) == 0, L)
>>> print L2
[0, 4, -6]
>>>

Now, what if we want to extend this so you can find all the multiples of a given number m ? In versions of Python up to 2.1, the rules for variable scoping would prevent the obvious code from working inside a function:

def f(L):
    m = 3
    L2 = filter(lambda N: (N % m) == 0, L)
    return L2

f([0, 2, 3, 17, -15])

On running this in Python 2.1, you would get a NameError for m in the second line of f() . Inside the anonymous function defined by the lambda statement, Python's scoping rules dictated that first the function's local variables and then the module-level variables were searched; f() 's local variables were never searched at all. The solution was to specify V as a default argument:

def f(L):
    m = 3
    L2 = filter(lambda N, m=m: (N % m) == 0, L)
    return L2

f([0, 2, 3, 17, -15])

If the filtering function required several of f() 's local variables, they'd all have to be passed in as default arguments. This results in the self=self, v=v that can be seen cluttering many uses of map() and filter() . Fixing this required modifying Python's scoping rules.

Python 2.1 added static scoping to the language to fix this problem. In 2.1, it had to be deliberately enabled in a module by including the directive from __future__ import nested_scopes at the top of a module, and the new scoping behaviour became the default only in Python 2.2. This was done to avoid breaking existing code whose behaviour would have changed as a result of the new scoping rules; Python 2.1 printed warnings about programs that were going to produce different results in Python 2.2 so that everyone had lots of time to fix their code. (See PEP 236 for a description of how incompatible changes are gradually introduced into Python.)

The first effect of static scoping was that the m=m default argument became unnecessary in the above example; the first version of the function now works as written. Under the new rules, when a given variable name is not assigned a value within a function (by an assignment, or the def , class , or import statements), references to the variable will be looked up in the local namespace of the enclosing scope. A more detailed explanation of the rules, and a dissection of the implementation, can be found in PEP 227: "Nested Scopes" . Scoping Rules

(Fixed in Python 2.1/2.2.)

This is another demonstration of the same problem. Recursive functions, as in following example, work fine when they're placed at the top-level of a module.:

def fact(n):
    if n == 1: return 1
    else: return n*fact(n-1)

However, if this definition is nested inside another function, it breaks as in the following example:

def g():
    def fact(n):
        if n == 1: return 1
        else: return n*fact(n-1)
    print fact(5)

When the nested version of fact() was called, it can't access its own name because the name fact was neither in the locals or in the module-level globals.:

[amk@mira amk]$ python2.1 /tmp/t.py
Traceback (innermost last):
  File "/tmp/t.py", line 8, in ?
    g()
  File "/tmp/t.py", line 6, in g
    print fact(5)
  File "/tmp/t.py", line 5, in fact
    else: return n*fact(n-1)
NameError: fact

This problem went away because of the static scoping introduced in Python 2.1 and described in the previous section.

[Nov 21, 2019] Python Functions

Nov 21, 2019 | www.w3schools.com

Passing a List as a Parameter

You can send any data types of parameter to a function (string, number, list, dictionary etc.), and it will be treated as the same data type inside the function.

E.g. if you send a List as a parameter, it will still be a List when it reaches the function:

Example def my_function(food):
for x in food:
print (x)

fruits = [ "apple" , "banana" , "cherry" ]

my_function(fruits)

[Nov 21, 2019] No do statement

Nov 21, 2019 | web.archive.org

http://web.archive.org/web/20031002184114/www.amk.ca/python/writing/warts.html

Python has a while statement which has the loop test at the beginning of the loop, but there's no variant which has the test at the end. As a result, you commonly see code like this:

# Read lines until a blank line is found
while 1:
    line = sys.stdin.readline()
    if line == "\n": 
        break

The while 1: ... if (condition): break idiom is often seen in Python code. The code might be clearer if you could write:

# Read lines until a blank line is found
do:
    line = sys.stdin.readline()
while line != "\n"

Adding this new control structure to Python would be pretty straightforward, though any existing code that used "do" as a variable name would be broken by the introduction of a new do keyword. PEP 315: "Enhanced While Loop" is a detailed proposal for adding a do construct, but at this point no ruling on it has been made.

The addition of iterators in Python 2.2 has made it possible to write many such loops by using the for statement instead of while , an idiom that you might find preferable.:

for line in sys.stdin:
    if line == '\n':
        break

[Nov 21, 2019] zephyrfalcon.org labs 10 Python pitfalls

Nov 21, 2019 | archive.is

String concatenation

var S : String;
for I := 1 to 10000 do begin
    S := S + Something(I);
end;

(This piece of code assumes a string type of more than 255 characters, which was the maximum in Turbo Pascal, aside... ;-)

Similar code in Python is likely to be highly inefficient. Since Python strings are immutable (as opposed to Pascal strings), a new string is created for every iteration (and old ones are thrown away). This may result in unexpected performance hits. Using string concatenation with + or += is OK for small changes, but it's usually not recommended in a loop.

Solution: If at all possible, create a list of values, then use string.join (or the join() method) to glue them together as one long string. Sometimes this can result in dramatic speedups.

To illustrate this, a simple benchmark. ( timeit is a simple function that runs another function and returns how long it took to complete, in seconds.)

>>> def f():
...     s = ""
...     for i in range(100000):
...         s = s + "abcdefg"[i % 7]
...     
>>> timeit(f)
23.7819999456

>>> def g():
...     z = []
...     for i in range(100000):
...         z.append("abcdefg"[i % 7])
...     return ''.join(z)
...     
>>> timeit(g)
0.343000054359

Update: This was fixed in CPython 2.4. According to the What's New in Python 2.4 page: "String concatenations in statements of the form s = s + "abc" and s += "abc" are now performed more efficiently in certain circumstances. This optimization won't be present in other Python implementations such as Jython, so you shouldn't rely on it; using the join() method of strings is still recommended when you want to efficiently glue a large number of strings together."

[Nov 21, 2019] Python 3.3 Access string internal representation - Stack Overflow

Nov 21, 2019 | stackoverflow.com

> ,

The internal switch to a more space-efficient storage for unicode values introduced by PEP-393 were made for performance reasons only.

As such they have zero impact on how encoding from and decoding to unicode str values works in Python code. There is absolutely no point in accessing the internal representation from Python. The character A is either stored as 41 , 4100 or 41000000 , depending on how much space the highest codepoint in the string demands, but it'll still be encoded to 41 in ASCII, Latin-1 or UTF-8.

Unless you are writing a C extension that has to deal with this internal representation, there is absolutely no need to worry about how Python actually stored the data.

To debug encoding or decoding issues, I'd use the ascii() function to represent a string using only ASCII codepoints and Python string literal escapes, or you can use the ord() function for turning individual characters into an integer for each codepoint.

For bytes values, the binascii.hexlify() function also comes in handy to quickly turn a series of bytes into their hex representations.

[Nov 21, 2019] What is internal representation of string in Python 3.x - Stack Overflow

Nov 21, 2019 | stackoverflow.com

What is internal representation of string in Python 3.x Ask Question Asked 9 years, 11 months ago Active 8 months ago Viewed 13k times 24 3


thebat ,Dec 3, 2009 at 6:57

In Python 3.x, a string consists of items of Unicode ordinal. (See the quotation from the language reference below.) What is the internal representation of Unicode string? Is it UTF-16?

The items of a string object are Unicode code units. A Unicode code unit is represented by a string object of one item and can hold either a 16-bit or 32-bit value representing a Unicode ordinal (the maximum value for the ordinal is given in sys.maxunicode, and depends on how Python is configured at compile time). Surrogate pairs may be present in the Unicode object, and will be reported as two separate items.

Joachim Sauer ,May 6, 2015 at 10:08

There has been NO CHANGE in Unicode internal representation between Python 2.X and 3.X.

It's definitely NOT UTF-16. UTF-anything is a byte-oriented EXTERNAL representation.

Each code unit (character, surrogate, etc) has been assigned a number from range(0, 2 ** 21). This is called its "ordinal".

Really, the documentation you quoted says it all. Most Python binaries use 16-bit ordinals which restricts you to the Basic Multilingual Plane ("BMP") unless you want to muck about with surrogates (handy if you can't find your hair shirt and your bed of nails is off being de-rusted). For working with the full Unicode repertoire, you'd prefer a "wide build" (32 bits wide).

Briefly, the internal representation in a unicode object is an array of 16-bit unsigned integers, or an array of 32-bit unsigned integers (using only 21 bits).

gwideman ,Oct 24, 2017 at 3:42

The internal representation will change in Python 3.3 which implements PEP 393 . The new representation will pick one or several of ascii, latin-1, utf-8, utf-16, utf-32, generally trying to get a compact representation.

Implicit conversions into surrogate pairs will only be done when talking to legacy APIs (those only exist on windows, where wchar_t is two bytes); the Python string will be preserved. Here are the release notes .

Matthew Brett ,Feb 28, 2015 at 20:32

In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393 .

For previous Pythons, the internal representation depends on the build flags of Python. Python can be built with flag values --enable-unicode=ucs2 or --enable-unicode=ucs4 . ucs2 builds do in fact use UTF-16 as their internal representation , and ucs4 builds use UCS-4 / UTF-32.

codeape ,Dec 3, 2009 at 9:25

Looking at the source code for CPython 3.1.5, in Include/unicodeobject.h :
/* --- Unicode Type ------------------------------------------------------- */

typedef struct {
    PyObject_HEAD
    Py_ssize_t length;          /* Length of raw Unicode data in buffer */
    Py_UNICODE *str;            /* Raw Unicode buffer */
    long hash;                  /* Hash value; -1 if not set */
    int state;                  /* != 0 if interned. In this case the two
                                 * references from the dictionary to this object
                                 * are *not* counted in ob_refcnt. */
    PyObject *defenc;           /* (Default) Encoded version as Python
                                   string, or NULL; this is used for
                                   implementing the buffer protocol */
} PyUnicodeObject;

The characters are stored as an array of Py_UNICODE . On most platforms, I believe Py_UNICODE is #define d as wchar_t .

Ned Deily ,Dec 3, 2009 at 7:12

It depends: see here . This is still true for Python 3 as far as internal representation goes.

YOU ,Dec 3, 2009 at 7:18

I think, Its hard to judge difference between UTF-16, which is just a sequences of 16 bit words, to Python's string object.

And If python is compiled with Unicode=UCS4 option, it will be comparing between UTF-32 and Python string.

So, better consider, they are in different category, although you can transform each others.

,

>>> import array; s = 'Привет мир!'; b = array.array('u', s).tobytes(); print(b); print(len(s) * 4 == len(b))
b'\x1f\x04\x00\x00@\x04\x00\x008\x04\x00\x002\x04\x00\x005\x04\x00\x00B\x04\x00\x00 \x00\x00\x00<\x04\x00\x008\x04\x00\x00@\x04\x00\x00!\x00\x00\x00'
True
>>> import array; s = 'test'; b = array.array('u', s).tobytes(); print(b); print(len(s) * 4 == len(b))
b't\x00\x00\x00e\x00\x00\x00s\x00\x00\x00t\x00\x00\x00'
True
>>>

[Nov 18, 2019] The Definitive Guide to Python import Statements Chris Yeh

Nov 18, 2019 | chrisyeh96.github.io

The Definitive Guide to Python import Statements I've almost never been able to write correct Python import statements on the first go. Behavior is inconsistent between Python 2.7 and Python 3.6 (the two versions that I test here), and there is no single method for guaranteeing that imports will always work. This post is my dive into how to resolve common importing problems. Unless otherwise stated, all examples here work with both Python 2.7 and 3.6. Table of Contents

Summary / Key Points Basic Definitions Example Directory Structure
test/                      # root folder
    packA/                 # package packA
        subA/              # subpackage subA
            __init__.py
            sa1.py
            sa2.py
        __init__.py
        a1.py
        a2.py
    packB/                 # package packB (implicit namespace package)
        b1.py
        b2.py
    math.py
    random.py
    other.py
    start.py

Note that we do not place a __init__.py file in our root test/ folder.

What is an import ?

When a module is imported, Python runs all of the code in the module file. When a package is imported, Python runs all of the code in the package's __init__.py file, if such a file exists. All of the objects defined in the module or the package's __init__.py file are made available to the importer.

Basics of the Python import and sys.path

According to Python documentation, here is how an import statement searches for the correct module or package to import:

When a module named spam is imported, the interpreter first searches for a built-in module with that name. If not found, it then searches for a file named spam.py in a list of directories given by the variable sys.path . sys.path is initialized from these locations:

After initialization, Python programs can modify sys.path . The directory containing the script being run is placed at the beginning of the search path, ahead of the standard library path. This means that scripts in that directory will be loaded instead of modules of the same name in the library directory. Source: Python 2 and 3

Technically, Python's documentation is incomplete. The interpreter will not only look for a file (i.e. module) named spam.py , it will also look for a folder (i.e. package) named spam .

Note that the Python interpreter first searches through the list of built-in modules , modules that are compiled directly into the Python interpreter. This list of built-in modules is installation-dependent and can be found in sys.builtin_module_names (Python 2 and 3 ). Some built-in modules that are commonly included are sys (always included), math , itertools , and time , among others.

Unlike built-in modules which are first in the search path, the rest of the modules in Python's standard library (not built-ins) come after the directory of the current script. This leads to confusing behavior: it is possible to "replace" some but not all modules in Python's standard library. For example, on my computer (Windows 10, Python 3.6), the math module is a built-in module, whereas the random module is not. Thus, import math in start.py will import the math module from the standard library, NOT my own math.py file in the same directory. However, import random in start.py will import my random.py file, NOT the random module from the standard library.

Also, Python imports are case-sensitive. import Spam is not the same as import spam .

The function pkgutil.iter_modules (Python 2 and 3 ) can be used to get a list of all importable modules from a given path:

import pkgutil
search_path = ['.'] # set to None to see all modules importable from sys.path
all_modules = [x[1] for x in pkgutil.iter_modules(path=search_path)]
print(all_modules)

Sources

More on sys.path

To see what is in sys.path , run the following in the interpreter or as a script:

import sys
print(sys.path)

Python's documentation for sys.path describes it as

A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH , plus an installation-dependent default.

As initialized upon program startup, the first item of this list, path[0] , is the directory containing the script that was used to invoke the Python interpreter. If the script directory is not available (e.g. if the interpreter is invoked interactively or if the script is read from standard input), path[0] is the empty string, which directs Python to search modules in the current directory first. Notice that the script directory is inserted before the entries inserted as a result of PYTHONPATH .

Source: Python 2 and 3

The documentation for Python's command line interface adds the following about running scripts from the command line. Specifically, when running python <script>.py , then

If the script name refers directly to a Python file, the directory containing that file is added to the start of sys.path, and the file is executed as the main module.

Source: Python 2 and 3

Let's recap the order in which Python searches for modules to import:

  1. modules in the Python Standard Library (e.g. math , os )
  2. modules or packages in a directory specified by sys.path :
    1. If the Python interpreter is run interactively:
      • sys.path[0] is the empty string '' . This tells Python to search the current working directory from which you launched the interpreter, i.e. the output of pwd on Unix systems.

      If we run a script with python <script>.py :

      • sys.path[0] is the path to <script>.py
    2. directories in the PYTHONPATH environment variable
    3. default sys.path locations

Note that when running a Python script, sys.path doesn't care what your current "working directory" is. It only cares about the path to the script . For example, if my shell is currently at the test/ folder and I run python ./packA/subA/subA1.py , then sys.path includes test/packA/subA/ but NOT test/ .

Additionally, sys.path is shared across all imported modules. For example, suppose we call python start.py . Let start.py import packA.a1 , and let a1.py print out sys.path . Then sys.path will include test/ (the path to start.py ), but NOT test/packA/ (the path to a1.py ). What this means is that a1.py can call import other since other.py is a file in test/ .

All about __init__.py

An __init__.py file has 2 functions.

  1. convert a folder of scripts into an importable package of modules (before Python 3.3)
  2. run package initialization code
Converting a folder of scripts into an importable package of modules

In order to import a module or package from a directory that is not in the same directory as the script we are writing (or the directory from which we run the Python interactive interpreter), that module needs to be in a package.

As defined above, any directory with a file named __init__.py is a Python package. This file can be empty. For example, when running Python 2.7, start.py can import the package packA but not packB because there is no __init__.py file in the test/packB/ directory.

This does NOT apply to Python 3.3 and above, thanks to the adoption of implicit namespace packages. Basically, Python 3.3+ treats all folders as packages, so empty __init__.py files are no longer necessary and can be omitted.

For example, packB is a namespace package because it doesn't have a __init__.py file in the folder. If we start a Python 3.6 interactive interpreter in the test/ directory, then we get the following output:

>>> import packB
>>> packB
<module 'packB' (namespace)>

Sources

  1. What is init .py for?
  2. PEP 420: Implicit Namespace Packages
Running package initialization code

The first time that a package or one of its modules is imported, Python will execute the __init__.py file in the root folder of the package if the file exists. All objects and functions defined in __init__.py are considered part of the package namespace.

Consider the following example.

test/packA/a1.py

def a1_func():
    print("running a1_func()")

test/packA/__init__.py

## this import makes a1_func directly accessible from packA.a1_func
from packA.a1 import a1_func

def packA_func():
    print("running packA_func()")

test/start.py

import packA  # "import packA.a1" will work just the same

packA.packA_func()
packA.a1_func()
packA.a1.a1_func()

output of running python start.py :

running packA_func()
running a1_func()
running a1_func()

*Note: if a1.py calls import a2 and we run python a1.py , then test/packA/__init__.py will NOT be called, even though it seems like a2 is part of the packA package. This is because when Python runs a script (in this case a1.py ), its containing folder is not considered a package.

Using Objects from the Imported Module or Package

There are 4 different syntaxes for writing import statements.

  1. import <package>
  2. import <module>
  3. from <package> import <module or subpackage or object>
  4. from <module> import <object>

Let X be whatever name comes after import .

Optionally, as Y can be added after any import X statement: import X as Y . This renames X to Y within the script. Note that the name X itself is no longer valid. A common example is import numpy as np .

The argument to the import function can be a single name, or a list of multiple names. Each of these names can be optionally renamed via as . For example, this would be a valid import statement in start.py : import packA as pA, packA.a1, packA.subA.sa1 as sa1

Example: start.py needs to import the helloWorld() function in sa1.py

Use dir() to examine the contents of an imported module

After importing a module, use the dir() function to get a list of accessible names from the module. For example, suppose I import sa1 . If sa1.py defines a helloWorld() function, then dir(sa1) would include helloWorld .

>>> from packA.subA import sa1
>>> dir(sa1)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'helloWorld']
Importing Packages

Importing a package is conceptually equivalent to importing the package's __init__.py file as a module. Indeed, this is what Python treats the package as:

>>> import packA
>>> packA
<module 'packA' from 'packA\__init__.py'>

Only objects declared in the imported package's __init__.py are accessible to the importer. For example, since packB lacks a __init__.py file, calling import packB (in Python 3.3+) has very little use because no objects in the packB package are made available. A subsequent call to packB.b1 would fail because it has not been imported yet.

Absolute vs. Relative Import

An absolute import uses the full path (starting from the project's root folder) to the desired module to import.

A relative import uses the relative path (starting from the path of the current module) to the desired module to import. There are two types of relative imports:

The Python documentation says the following about Python 3's handling of relative imports:

The only acceptable syntax for relative imports is from .[module] import name. All import forms not starting with . are interpreted as absolute imports.

Source: What's New in Python 3.0

For example, suppose we are running start.py which imports a1 which in turn imports other , a2 , and sa1 . Then the import statements in a1.py would look as follows:

Note that for relative imports, the dots . can go up only up to (but not including) the directory containing the script run from the command line. Thus, from .. import other is invalid in a1.py . Doing so results in the error ValueError: attempted relative import beyond top-level package .

In general, absolute imports are preferred over relative imports. They avoid the confusion between explicit vs. implicit relative imports. In addition, any script that uses explicit relative imports cannot be run directly:

Note that relative imports are based on the name of the current module. Since the name of the main module is always " main ", modules intended for use as the main module of a Python application must always use absolute imports.

Source: Python 2 and 3

Sources

Case Examples Case 1: sys.path is known ahead of time

If you only ever call python start.py or python other.py , then it is very easy to set up the imports for all of the modules. In this case, sys.path will always include test/ in its search path. Therefore, all of the import statements can be written relative to the test/ folder.

Ex: a file in the test project needs to import the helloWorld() function in sa1.py

Case 2: sys.path could change

Often, we want to be flexible in how we use a Python script, whether run directly on the command line or imported as a module into another script. As shown below, this is where we run into problems, especially on Python 3.

Example : Suppose start.py needs to import a2 which needs to import sa2 . Assume that start.py is always run directly, never imported. We also want to be able to run a2 on its own.

Seems easy enough, right? After all, we just need 2 import statements total: 1 in start.py and another in a2.py .

Problem : This is clearly a case where sys.path changes. When we run start.py , sys.path contains test/ . When we run a2.py , sys.path contains test/packA/ .

The import statement in start.py is easy. Since start.py it is always run directly and never imported, we know that test/ will always be in sys.path when it is run. Then importing a2 is simply import packA.a2 .

The import statement in a2.py is trickier. When we run start.py directly, sys.path contains test/ , so a2.py should call from packA.subA import sa2 . However, if we instead run a2.py directly, then sys.path contains test/packA/ . Now the import would fail because packA is not a folder inside test/packA/ .

Instead, we could try from subA import sa2 . This corrects the problem when we run a2.py directly. But now we have a problem when we run start.py directly. Under Python 3, this fails because subA is not in sys.path . (This is OK in Python 2, thanks to its support for implicit relative imports.)

Let's summarize our findings about the import statement in a2.py :

Run from packA.subA import sa2 from subA import sa2
start.py OK Py2 OK, Py3 fail ( subA not in test/ )
a2.py fail ( packA not in test/packA/ ) OK

For completeness sake, I also tried using relative imports: from .subA import sa2 . This matches the result of from packA.subA import sa2 .

Solutions (Workarounds) : I am unaware of a clean solution to this problem. Here are some workarounds:

  1. Use absolute imports rooted at the test/ directory (i.e. middle column in the table above). This guarantees that running start.py directly will always work. In order to run a2.py directly, run it as an imported module instead of as a script:
    1. change directories to test/ in the console
    2. python -m packA.a2
  2. Use absolute imports rooted at the test/ directory (i.e. middle column in the table above). This guarantees that running start.py directly will always work. In order to run a2.py directly, we can modify sys.path in a2.py to include test/packA/ , before sa2 is imported.
     import os, sys
     sys.path.append(os.path.dirname(os.path.dirname(os.path.realpath(__file__))))
    
     # now this works, even when a2.py is run directly
     from packA.subA import sa2
    

    NOTE: This method usually works. However, under some Python installations, the __file__ variable might not be correct. In this case, we would need to use the Python built-in inspect package. See this StackOverflow answer for instructions.

  3. Only use Python 2, and use implicit relative imports (i.e. the right column in the table above)
  4. Use absolute imports rooted at the test/ directory, and add test/ to the PYTHONPATH environment variable.
Case 3: sys.path could change (take 2)

A harder problem to deal with is the following example. Suppose a2.py never needs to be run directly, but it is imported by both start.py and a1.py which are run directly.

In this case, using Solution 1 described above won't work. However, the other solutions are still valid.

Case 4: Importing from Parent Directory

If we do not modify PYTHONPATH and avoid modifying sys.path programmatically, then the following is a major limitation of Python imports:

When running a script directly, it is impossible to import anything from its parent directory.

For example, if I were to run python sa1.py , then it is impossible for sa1.py to import anything from a1.py without resorting to a PYTHONPATH or sys.path workaround.

At first, it may seem that relative imports (e.g. from .. import a1 ) could work around this limitation. However, the script that is being run (in this case sa1.py ) is considered the "top-level module." Attempting to import anything from a folder above this script results in this error: ValueError: attempted relative import beyond top-level package .

My approach is to avoid writing scripts that have to import from the parent directory. In cases where this must happen, the preferred workaround is to modify sys.path .

Python 2 vs. Python 3

The most important differences between how Python 2 and Python 3 treat import statements have been documented above. They are re-stated again here, along with some other less important differences.

  1. Python 2 supports implicit relative imports. Python 3 does not.
  2. Python 2 requires __init__.py files inside a folder in order for the folder to be considered a package and made importable. In Python 3.3 and above, thanks to its support of implicit namespace packages, all folders are packages regardless of the presence of a __init__.py file.
  3. In Python 2, one could write from <module> import * within a function. In Python 3, the from <module> import * syntax is only allowed at the module level, no longer inside functions.

Sources:

Miscellaneous topics and readings not covered here, but worth exploring

[Nov 18, 2019] How to convert list into string with quotes in python - Stack Overflow

Nov 18, 2019 | stackoverflow.com

How to convert list into string with quotes in python Ask Question Asked 6 years, 7 months ago Active 1 year, 2 months ago Viewed 9k times 4 0


alexis ,Apr 17, 2013 at 8:21

i have a list and want it as a string with quotes mylist = [1,2,3]

require O/P as myString = "'1','2','3'"

i tried mystring = '\',\''.join(mylist)

it gave me result as

mystring = "1','2','3"

first and last quotes (') are missing

Matthias ,Apr 17, 2013 at 8:53

This seems to be the only solution so far that isn't a hack...
>>> mylist = [1,2,3]
>>> ','.join("'{0}'".format(x) for x in mylist)
"'1','2','3'"

This can also be written more compactly as:

>>> ','.join(map("'{0}'".format, mylist))
"'1','2','3'"

jamylak ,Apr 17, 2013 at 8:36

as a simple hack, why don't you..
mystring = "'%s'" %"','".join(mylist)

wrap the result of your commands in quotes?

ashokadhikari ,Apr 17, 2013 at 8:24

you can do this as well
mylist = [1, 2, 3]
mystring = str(map(str, mylist)).strip("[]")

atomh33ls ,Apr 17, 2013 at 8:46

>>> mylist = [1,2,3]
>>> str([str(x) for x in mylist]).strip("[]")
"'1','2','3'"

> ,

OR regular repr :
>>> l=[1,2,3]
>>> ','.join(repr(str(i)) for i in l)
"'1','2','3'"
>>>

[Nov 18, 2019] Python enclose each word of a space separated string in quotes - Stack Overflow

Nov 18, 2019 | stackoverflow.com

Python: enclose each word of a space separated string in quotes Ask Question Asked 4 years, 1 month ago Active 1 year, 9 months ago Viewed 3k times 2


vaultah ,Sep 24, 2015 at 15:53

I have a string eg:
line="a sentence with a few words"

I want to convert the above in a string with each of the words in double quotes, eg:

 "a" "sentence" "with" "a" "few" "words"

Any suggestions?

mkc ,Sep 24, 2015 at 18:26

Split the line into words, wrap each word in quotes, then re-join:
' '.join('"{}"'.format(word) for word in line.split(' '))

Anand S Kumar ,Sep 24, 2015 at 15:54

Since you say -

I want to convert the above in a string with each of the words in double quotes

You can use the following regex -

>>> line="a sentence with a few words"
>>> import re
>>> re.sub(r'(\w+)',r'"\1"',line)
'"a" "sentence" "with" "a" "few" "words"'

This would take into consideration punctuations, etc as well (if that is really what you wanted) -

>>> line="a sentence with a few words. And, lots of punctuations!"
>>> re.sub(r'(\w+)',r'"\1"',line)
'"a" "sentence" "with" "a" "few" "words". "And", "lots" "of" "punctuations"!'

user4568737 user4568737

add a comment ,Aug 10, 2017 at 18:42
Or you can something simpler (more implementation but easier for beginners) by searching for each space in the quote then slice whatever between the spaces, add " before and after it then print it.
quote = "they stumble who run fast"
first_space = 0
last_space = quote.find(" ")
while last_space != -1:
    print("\"" + quote[first_space:last_space] + "\"")
    first_space = last_space + 1
    last_space = quote.find(" ",last_space + 1)

Above code will output for you the following:

"they"
"stumble"
"who"
"run"

Stephen Rauch ,Jan 28, 2018 at 3:29

The first answer missed an instance of the original quote. The last string/word "fast" was not printed. This solution will print the last string:
quote = "they stumble who run fast"

start = 0
location = quote.find(" ")

while location >=0:
    index_word = quote[start:location]
    print(index_word)

    start = location + 1
    location = quote.find(" ", location + 1)

#this runs outside the While Loop, will print the final word
index_word = quote[start:]
print(index_word)

This is the result:

they
stumble
who
run
fast

[Nov 15, 2019] Go versus Python 3 -- fastest programs

Nov 15, 2019 | pages.debian.net
Back in April 2010, Russ Cox charitably suggested that only fannkuch-redux, fasta, k-nucleotide, mandlebrot, nbody, reverse-complement and spectral-norm were close to fair comparisons. As someone who implemented programming languages, his interest was "measuring the quality of the generated code when both compilers are presented with what amounts to the same program."

Differences in approach - to memory management, parallel programming, regex, arbitrary precision arithmetic, implementation technique - don't fit in that kind-of fair comparison -- but we still have to deal with them.

These are only the fastest programs. There may be additional measurements for programs which seem more-like a fair comparison to you. Always look at the source code.

mandelbrot
source secs mem gz busy cpu load
Go 5.47 31,088 905 21.77 100% 99% 99% 99%
Python 3 259.50 48,192 688 1,036.70 100% 100% 100% 100%
spectral-norm
source secs mem gz busy cpu load
Go 3.94 2,740 548 15.74 100% 100% 100% 100%
Python 3 169.87 49,188 417 675.02 100% 99% 99% 99%
n-body
source secs mem gz busy cpu load
Go 21.25 1,588 1310 21.48 0% 1% 100% 0%
Python 3 865.18 8,176 1196 874.96 2% 20% 79% 0%
fasta
source secs mem gz busy cpu load
Go 2.08 3,560 1358 5.61 80% 37% 76% 78%
Python 3 63.55 844,180 1947 129.71 40% 71% 33% 61%
fannkuch-redux
source secs mem gz busy cpu load
Go 17.56 1,524 900 70.00 100% 100% 100% 100%
Python 3 534.40 47,236 950 2,104.05 99% 97% 99% 99%
k-nucleotide
source secs mem gz busy cpu load
Go 11.77 160,184 1607 44.52 94% 98% 94% 92%
Python 3 72.24 199,856 1967 275.38 94% 94% 96% 96%
reverse-complement
source secs mem gz busy cpu load
Go 3.83 1,782,940 1338 6.67 21% 74% 16% 62%
Python 3 16.93 1,777,852 434 17.58 78% 21% 4% 0%
binary-trees
source secs mem gz busy cpu load
Go 25.17 361,152 1005 86.86 87% 89% 86% 84%
Python 3 80.30 448,004 589 286.50 95% 87% 87% 88%
pidigits
source secs mem gz busy cpu load
Go 2.10 8,448 603 2.17 1% 48% 55% 0%
Python 3 3.47 10,356 386 3.53 1% 1% 0% 100%
regex-redux
source secs mem gz busy cpu load
Go 29.82 428,224 802 62.65 48% 45% 52% 65%
Python 3 18.45 457,340 512 37.52 69% 37% 50% 48%
Go go version go1.13 linux/amd64
Python 3 Python 3.8.0

[Nov 15, 2019] Python Displaces C++ In TIOBE Index Top 3 - Slashdot

Nov 15, 2019 | developers.slashdot.org

raymorris ( 2726007 ) , Saturday September 08, 2018 @03:58PM ( #57276974 ) Journal

Losing backwards compatibility with point releases ( Score: 5 , Interesting)

Never mind Python 2 vs 3; one major reason I shy away from Python is the incompatibility in point releases. I'd see "requires Python 2.6 and see that I have Python 2.7 so it should be fine, right? Nope, code written for 2.6 won't run under Python 2.7. It needs to be EXACTLY 2.6.

It's at this point that some Python fanboi gets really upset and starts screaming about how that's now problem, with Python you set up separate virtual environments for each script, so that each one can have exactly the version of Python it is written for, with exactly the version of each library. When there is some bug or security issue you then hope that there is a patch for each, and deal with all that. (As opposed to every other peice of software in the world, which you simply upgrade to the latest version to get all the latest fixes). Yes, you CAN deal with that problem, it's possible, in most cases. You shouldn't have to. Every other language does some simple things to maintain backward compatibility in point releases (and mostly in major releases too).

Also the fact that most languages use every day and have used for decades use braces for blocks means my eyes and mind are very much trained for that. Braces aren't necessarily BETTER than just using indentation, but it's kinda like building a car which uses the pedals to steer and a hand stick for the gas.

It's not necessarily inherently better or worse, but it would be almost undriveable for an experienced driver with decades of muscle memory in normal cars.

Python's seemingly random choices on these things make it feel like using your feet to steer a car. There should be compelling reasons before you break long-established conventions and Python seems to prefer to break conventions just to be different. It seems the Python team is a bit like Berstein in that way. It's really annoying.

slack_justyb ( 862874 ) , Saturday September 08, 2018 @05:00PM ( #57277244 )
Re:Losing backwards compatibility with point relea ( Score: 5 , Informative)
Python 2.7. It needs to be EXACTLY 2.6

Yeah, just FYI Python 2.7 is in a way its own thing. Different from the 2.x and different from the 3.x series. 2.6 is a no holds barred pure 2.x whereas 2.7 is a mixture of 2.x and 3.x features. So if you want to compare point releases, best to try that with the 3.x series. Also, if you're using something that requires the 2.x series, you shouldn't use that unless it is absolutely critical with zero replacements.

You shouldn't have to. Every other language does some simple things to maintain backward compatibility in point releases (and mostly in major releases too).

Again see argument about 3.x, but yeah not every language does this. Java 8/9 transition breaks things. ASP.Net to ASP.Net core breaks things along the way. I'm interested in what languages you have in mind, because I know quite a few languages that do maintain backwards compatibility (ish). For example, C++ pre and post namespace breaks fstreams in programs, but compilers provide flags to override that, so it depends on what you mean by breaking. Does it count if the compiler by default breaks, but providing flags fixes it? Because if your definition means including flags break compatibility, then oooh boy are there a a shit ton of broken languages.

Also the fact that most languages use every day and have used for decades use braces for blocks means my eyes and mind are very much trained for that

Yeah, it's clear that you've never used a positional programming language. I guess it'll be a sign of my age, but buddy, program COBOL or RPG on punch cards and let me know about that curly brace issue you're having. Positional and indentation has been used way, way, way, longer than curly braces. That's not me knocking on the curly braces, I love my C/C++ folks out there! But I hate to tell you C and C-style is pretty recent in the timeline of all things computer.

[Nov 14, 2019] How to implement a switch-case statement in Python by Sreeram Sceenivasan

Oct 24, 2017 | jaxenter.com

...You can use it to execute different blocks of code, depending on the variable value during runtime. Here's an example of a switch statement in Java.

public static void switch_demo(String[] args) {
 
        int month = 8;
        String monthString;
        switch (month) {
            case 1:  monthString = "January";
                     break;
            case 2:  monthString = "February";
                     break;
            case 3:  monthString = "March";
                     break;
            case 4:  monthString = "April";
                     break;
            case 5:  monthString = "May";
                     break;
            case 6:  monthString = "June";
                     break;
            case 7:  monthString = "July";
                     break;
            case 8:  monthString = "August";
                     break;
            case 9:  monthString = "September";
                     break;
            case 10: monthString = "October";
                     break;
            case 11: monthString = "November";
                     break;
            case 12: monthString = "December";
                     break;
            default: monthString = "Invalid month";
                     break;
        }
        System.out.println(monthString);
}

Here's how it works:

  1. Compiler generates a jump table for switch case statement
  2. The switch variable/expression is evaluated once
  3. Switch statement looks up the evaluated variable/expression in the jump table and directly decides which code block to execute.
  4. If no match is found, then the code under default case is executed

In the above example, depending on the value of variable month , a different message will be displayed in the standard output. In this case, since the month=8, 'August' will be printed in standard output.

Read also: An introduction to the Python programming language

When Guido van Rossum developed Python, he wanted to create a "simple" programming language that bypassed the vulnerabilities of other systems. Due to the simple syntax and sophisticated syntactic phrases, the language has become the standard for various scientific applications such as machine learning. Switch statements

Although popular languages like Java and PHP have in-built switch statement, you may be surprised to know that Python language doesn't have one. As such, you may be tempted to use a series of if-else-if blocks, using an if condition for each case of your switch statement.

However, because of the jump table, a switch statement is much faster than an if-else-if ladder. Instead of evaluating each condition sequentially, it only has to look up the evaluated variable/expression once and directly jump to the appropriate branch of code to execute it.

SEE MORE: Python jumps past Java, Javascript is still most popular language for GitHubbers

How to implement switch statement in Python

The Pythonic way to implement switch statement is to use the powerful dictionary mappings, also known as associative arrays, that provide simple one-to-one key-value mappings.

def switch_demo(argument):
switcher = {
  1: "January",
  2: "February",
  3: "March",
  4: "April",
  5: "May",
  6: "June",
  7: "July",
  8: "August",
  9: "September",
  10: "October",
  11: "November",
  12: "December"
}
print switcher.get(argument, "Invalid month")

Here's the Python implementation of the above switch statement. In the following example, we create a dictionary named switcher to store all the switch-like cases.

In the above example, when you pass an argument to the switch_demo function, it is looked up against the switcher dictionary mapping. If a match is found, the associated value is printed, else a default string ('Invalid Month') is printed. The default string helps implement the 'default case' of a switch statement.

Dictionary mapping for functions

Here's where it gets more interesting. The values of a Python dictionary can be of any data type. So you don't have to confine yourself to using constants (integers, strings), you can also use function names and lambdas as values.

For example, you can also implement the above switch statement by creating a dictionary of function names as values. In this case, switcher is a dictionary of function names, and not strings.

def one():
    return "January"
 
def two():
    return "February"
 
def three():
    return "March"
 
def four():
    return "April"
 
def five():
    return "May"
 
def six():
    return "June"
 
def seven():
    return "July"
 
def eight():
    return "August"
 
def nine():
    return "September"
 
def ten():
    return "October"
 
def eleven():
    return "November"
 
def twelve():
    return "December"
 
 
def numbers_to_months(argument):
    switcher = {
        1: one,
        2: two,
        3: three,
        4: four,
        5: five,
        6: six,
        7: seven,
        8: eight,
        9: nine,
        10: ten,
        11: eleven,
        12: twelve
    }
    # Get the function from switcher dictionary
    func = switcher.get(argument, lambda: "Invalid month")
    # Execute the function
    print func()

Although the above functions are quite simple and only return strings, you can use this approach to execute elaborate blocks of code within each function.

... ... ...

[Nov 13, 2019] python - Running shell command and capturing the output

Nov 22, 2012 | stackoverflow.com

Vartec's answer doesn't read all lines, so I made a version that did:

def run_command(command):
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

Usage is the same as the accepted answer:

command = 'mysqladmin create test -uroot -pmysqladmin12'.split()
for line in run_command(command):
    print(line)
share python - Running shell command and capturing the output - Stack Overflow Share a link to this answer Copy link | improve this answer edited May 23 '17 at 11:33 Community ♦ 1 1 1 silver badge answered Oct 30 '12 at 9:24 Max Ekman Max Ekman 769 5 5 silver badges 5 5 bronze badges

[Nov 13, 2019] Execute shell commands in Python

Nov 13, 2019 | unix.stackexchange.com

Execute shell commands in Python Ask Question Asked 4 years ago Active 2 months ago Viewed 557k times 67 32


fooot ,Nov 8, 2017 at 21:39

I'm currently studying penetration testing and Python programming. I just want to know how I would go about executing a Linux command in Python. The commands I want to execute are:
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080

If I just use print in Python and run it in the terminal will it do the same as executing it as if you was typing it yourself and pressing Enter ?

binarysubstrate ,Feb 28 at 19:58

You can use os.system() , like this:
import os
os.system('ls')

Or in your case:

os.system('echo 1 > /proc/sys/net/ipv4/ip_forward')
os.system('iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080')

Better yet, you can use subprocess's call, it is safer, more powerful and likely faster:

from subprocess import call
call('echo "I like potatos"', shell=True)

Or, without invoking shell:

call(['echo', 'I like potatos'])

If you want to capture the output, one way of doing it is like this:

import subprocess
cmd = ['echo', 'I like potatos']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

o, e = proc.communicate()

print('Output: ' + o.decode('ascii'))
print('Error: '  + e.decode('ascii'))
print('code: ' + str(proc.returncode))

I highly recommend setting a timeout in communicate , and also to capture the exceptions you can get when calling it. This is a very error-prone code, so you should expect errors to happen and handle them accordingly.

https://docs.python.org/3/library/subprocess.html

jordanm ,Oct 23, 2015 at 15:43

The first command simply writes to a file. You wouldn't execute that as a shell command because python can read and write to files without the help of a shell:
with open('/proc/sys/net/ipv4/ip_forward', 'w') as f:
    f.write("1")

The iptables command is something you may want to execute externally. The best way to do this is to use the subprocess module .

import subprocess
subprocess.check_call(['iptables', '-t', 'nat', '-A',
                       'PREROUTING', '-p', 'tcp', 
                       '--destination-port', '80',
                       '-j', 'REDIRECT', '--to-port', '8080'])

Note that this method also does not use a shell, which is unnecessary overhead.

Tom Hunt ,Oct 23, 2015 at 15:41

The quickest way:
import os
os.system("your command here")

This isn't the most flexible approach; if you need any more control over your process than "run it once, to completion, and block until it exits", then you should use the subprocess module instead.

jordanm ,Apr 5, 2018 at 9:23

As a general rule, you'd better use python bindings whenever possible (better Exception catching, among other advantages.)

For the echo command, it's obviously better to use python to write in the file as suggested in @jordanm's answer.

For the iptables command, maybe python-iptables ( PyPi page , GitHub page with description and doc ) would provide what you need (I didn't check your specific command).

This would make you depend on an external lib, so you have to weight the benefits. Using subprocess works, but if you want to use the output, you'll have to parse it yourself, and deal with output changes in future iptables versions.

,

A python version of your shell. Be careful, I haven't tested it.
from subprocess import run

def bash(command):
        run(command.split())

>>> bash('find / -name null')
/dev/null
/sys/fs/selinux/null
/sys/devices/virtual/mem/null
/sys/class/mem/null
/usr/lib/kbd/consoletrans/null

[Nov 13, 2019] Python for Loops (Definite Iteration) by John Sturtz

Jan 30, 2019 | realpython.com

9 Comments basics python
Tweet Share Email

Table of Contents

Watch Now This tutorial has a related video course created by the Real Python team. Watch it together with the written tutorial to deepen your understanding: For Loops in Python (Definite Iteration)

This tutorial will show you how to perform definite iteration with a Python for loop.

In the previous tutorial in this introductory series, you learned the following:

Here's what you'll cover in this tutorial:

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code. Remove ads

A Survey of Definite Iteration in Programming

Definite iteration loops are frequently referred to as for loops because for is the keyword that is used to introduce them in nearly all programming languages, including Python.

Historically, programming languages have offered a few assorted flavors of for loop. These are briefly described in the following sections.

Numeric Range Loop

The most basic for loop is a simple numeric range statement with start and end values. The exact format varies depending on the language but typically looks something like this:

for i = 1 to 10
    <loop body>

Here, the body of the loop is executed ten times. The variable i assumes the value 1 on the first iteration, 2 on the second, and so on. This sort of for loop is used in the languages BASIC, Algol, and Pascal.

Three-Expression Loop

Another form of for loop popularized by the C programming language contains three parts:

This type of has the following form:

for (i = 1; i <= 10; i++)
    <loop body>
Technical Note: In the C programming language, i++ increments the variable i . It is roughly equivalent to i += 1 in Python.

This loop is interpreted as follows:

Three-expression for loops are popular because the expressions specified for the three parts can be nearly anything, so this has quite a bit more flexibility than the simpler numeric range form shown above. These for loops are also featured in the C++, Java, PHP, and Perl languages.

Collection-Based or Iterator-Based Loop

This type of loop iterates over a collection of objects, rather than specifying numeric values or conditions:

for i in <collection>
    <loop body>

Each time through the loop, the variable i takes on the value of the next object in <collection> . This type of for loop is arguably the most generalized and abstract. Perl and PHP also support this type of loop, but it is introduced by the keyword foreach instead of for .

Further Reading: See the For loop Wikipedia page for an in-depth look at the implementation of definite iteration across programming languages. Remove ads The Python for Loop

Of the loop types listed above, Python only implements the last: collection-based iteration. At first blush, that may seem like a raw deal, but rest assured that Python's implementation of definite iteration is so versatile that you won't end up feeling cheated!

Shortly, you'll dig into the guts of Python's for loop in detail. But for now, let's start with a quick prototype and example, just to get acquainted.

Python's for loop looks like this:

for <var> in <iterable>:
    <statement(s)>

<iterable> is a collection of objects -- for example, a list or tuple. The <statement(s)> in the loop body are denoted by indentation, as with all Python control structures, and are executed once for each item in <iterable> . The loop variable <var> takes on the value of the next element in <iterable> each time through the loop.

Here is a representative example:

>>>
>>> a = ['foo', 'bar', 'baz']
>>> for i in a:
...     print(i)
...
foo
bar
baz

In this example, <iterable> is the list a , and <var> is the variable i . Each time through the loop, i takes on a successive item in a , so print() displays the values 'foo' , 'bar' , and 'baz' , respectively. A for loop like this is the Pythonic way to process the items in an iterable.

But what exactly is an iterable? Before examining for loops further, it will be beneficial to delve more deeply into what iterables are in Python.

Iterables

In Python, iterable means an object can be used in iteration. The term is used as:

If an object is iterable, it can be passed to the built-in Python function iter() , which returns something called an iterator . Yes, the terminology gets a bit repetitive. Hang in there. It all works out in the end.

Each of the objects in the following example is an iterable and returns some type of iterator when passed to iter() :

>>>
>>> iter('foobar')                             # String
<str_iterator object at 0x036E2750>

>>> iter(['foo', 'bar', 'baz'])                # List
<list_iterator object at 0x036E27D0>

>>> iter(('foo', 'bar', 'baz'))                # Tuple
<tuple_iterator object at 0x036E27F0>

>>> iter({'foo', 'bar', 'baz'})                # Set
<set_iterator object at 0x036DEA08>

>>> iter({'foo': 1, 'bar': 2, 'baz': 3})       # Dict
<dict_keyiterator object at 0x036DD990>

These object types, on the other hand, aren't iterable:

>>>
>>> iter(42)                                   # Integer
Traceback (most recent call last):
  File "<pyshell#26>", line 1, in <module>
    iter(42)
TypeError: 'int' object is not iterable

>>> iter(3.1)                                  # Float
Traceback (most recent call last):
  File "<pyshell#27>", line 1, in <module>
    iter(3.1)
TypeError: 'float' object is not iterable

>>> iter(len)                                  # Built-in function
Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    iter(len)
TypeError: 'builtin_function_or_method' object is not iterable

All the data types you have encountered so far that are collection or container types are iterable. These include the string , list , tuple , dict , set , and frozenset types.

But these are by no means the only types that you can iterate over. Many objects that are built into Python or defined in modules are designed to be iterable. For example, open files in Python are iterable. As you will see soon in the tutorial on file I/O, iterating over an open file object reads data from the file.

In fact, almost any object in Python can be made iterable. Even user-defined objects can be designed in such a way that they can be iterated over. (You will find out how that is done in the upcoming article on object-oriented programming.)

Remove ads Iterators

Okay, now you know what it means for an object to be iterable, and you know how to use iter() to obtain an iterator from it. Once you've got an iterator, what can you do with it?

An iterator is essentially a value producer that yields successive values from its associated iterable object. The built-in function next() is used to obtain the next value from in iterator.

Here is an example using the same list as above:

>>>
>>> a = ['foo', 'bar', 'baz']

>>> itr = iter(a)
>>> itr
<list_iterator object at 0x031EFD10>

>>> next(itr)
'foo'
>>> next(itr)
'bar'
>>> next(itr)
'baz'

In this example, a is an iterable list and itr is the associated iterator, obtained with iter() . Each next(itr) call obtains the next value from itr .

Notice how an iterator retains its state internally. It knows which values have been obtained already, so when you call next() , it knows what value to return next.

What happens when the iterator runs out of values? Let's make one more next() call on the iterator above:

>>>
>>> next(itr)
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    next(itr)
StopIteration

If all the values from an iterator have been returned already, a subsequent next() call raises a StopIteration exception. Any further attempts to obtain values from the iterator will fail.

You can only obtain values from an iterator in one direction. You can't go backward. There is no prev() function. But you can define two independent iterators on the same iterable object:

>>>
>>> a
['foo', 'bar', 'baz']

>>> itr1 = iter(a)
>>> itr2 = iter(a)

>>> next(itr1)
'foo'
>>> next(itr1)
'bar'
>>> next(itr1)
'baz'

>>> next(itr2)
'foo'

Even when iterator itr1 is already at the end of the list, itr2 is still at the beginning. Each iterator maintains its own internal state, independent of the other.

If you want to grab all the values from an iterator at once, you can use the built-in list() function. Among other possible uses, list() takes an iterator as its argument, and returns a list consisting of all the values that the iterator yielded:

>>>
>>> a = ['foo', 'bar', 'baz']
>>> itr = iter(a)
>>> list(itr)
['foo', 'bar', 'baz']

Similarly, the built-in tuple() and set() functions return a tuple and a set, respectively, from all the values an iterator yields:

>>>
>>> a = ['foo', 'bar', 'baz']

>>> itr = iter(a)
>>> tuple(itr)
('foo', 'bar', 'baz')

>>> itr = iter(a)
>>> set(itr)
{'baz', 'foo', 'bar'}

It isn't necessarily advised to make a habit of this. Part of the elegance of iterators is that they are "lazy." That means that when you create an iterator, it doesn't generate all the items it can yield just then. It waits until you ask for them with next() . Items are not created until they are requested.

When you use list() , tuple() , or the like, you are forcing the iterator to generate all its values at once, so they can all be returned. If the total number of objects the iterator returns is very large, that may take a long time.

In fact, it is possible to create an iterator in Python that returns an endless series of objects. (You will learn how to do this in upcoming tutorials on generator functions and itertools .) If you try to grab all the values at once from an endless iterator, the program will hang .

Remove ads The Guts of the Python for Loop

You now have been introduced to all the concepts you need to fully understand how Python's for loop works. Before proceeding, let's review the relevant terms:

Term Meaning
Iteration The process of looping through the objects or items in a collection
Iterable An object (or the adjective used to describe an object) that can be iterated over
Iterator The object that produces successive items or values from its associated iterable
iter() The built-in function used to obtain an iterator from an iterable

Now, consider again the simple for loop presented at the start of this tutorial:

>>>
>>> a = ['foo', 'bar', 'baz']
>>> for i in a:
...     print(i)
...
foo
bar
baz

This loop can be described entirely in terms of the concepts you have just learned about. To carry out the iteration this for loop describes, Python does the following:

The loop body is executed once for each item next() returns, with loop variable i set to the given item for each iteration.

This sequence of events is summarized in the following diagram:

Schematic Diagram of a Python for Loop

Perhaps this seems like a lot of unnecessary monkey business, but the benefit is substantial. Python treats looping over all iterables in exactly this way, and in Python, iterables and iterators abound:

You will discover more about all the above throughout this series. They can all be the target of a for loop, and the syntax is the same across the board. It's elegant in its simplicity and eminently versatile.

Iterating Through a Dictionary

You saw earlier that an iterator can be obtained from a dictionary with iter() , so you know dictionaries must be iterable. What happens when you loop through a dictionary? Let's see:

>>>
>>> d = {'foo': 1, 'bar': 2, 'baz': 3}
>>> for k in d:
...     print(k)
...
foo
bar
baz

As you can see, when a for loop iterates through a dictionary, the loop variable is assigned to the dictionary's keys.

To access the dictionary values within the loop, you can make a dictionary reference using the key as usual:

>>>
>>> for k in d:
...     print(d[k])
...
1
2
3

You can also iterate through a dictionary's values directly by using .values() :

>>>
>>> for v in d.values():
...     print(v)
...
1
2
3

In fact, you can iterate through both the keys and values of a dictionary simultaneously. That is because the loop variable of a for loop isn't limited to just a single variable. It can also be a tuple, in which case the assignments are made from the items in the iterable using packing and unpacking, just as with an assignment statement:

>>>
>>> i, j = (1, 2)
>>> print(i, j)
1 2

>>> for i, j in [(1, 2), (3, 4), (5, 6)]:
...     print(i, j)
...
1 2
3 4
5 6

As noted in the tutorial on Python dictionaries , the dictionary method .items() effectively returns a list of key/value pairs as tuples:

>>>
>>> d = {'foo': 1, 'bar': 2, 'baz': 3}

>>> d.items()
dict_items([('foo', 1), ('bar', 2), ('baz', 3)])

Thus, the Pythonic way to iterate through a dictionary accessing both the keys and values looks like this:

>>>
>>> d = {'foo': 1, 'bar': 2, 'baz': 3}
>>> for k, v in d.items():
...     print('k =', k, ', v =', v)
...
k = foo , v = 1
k = bar , v = 2
k = baz , v = 3
Remove ads The range() Function

In the first section of this tutorial, you saw a type of for loop called a numeric range loop , in which starting and ending numeric values are specified. Although this form of for loop isn't directly built into Python, it is easily arrived at.

For example, if you wanted to iterate through the values from 0 to 4 , you could simply do this:

>>>
>>> for n in (0, 1, 2, 3, 4):
...     print(n)
...
0
1
2
3
4

This solution isn't too bad when there are just a few numbers. But if the number range were much larger, it would become tedious pretty quickly.

Happily, Python provides a better option -- the built-in range() function, which returns an iterable that yields a sequence of integers.

range(<end>) returns an iterable that yields integers starting with 0 , up to but not including <end> :

>>>
>>> x = range(5)
>>> x
range(0, 5)
>>> type(x)
<class 'range'>

Note that range() returns an object of class range , not a list or tuple of the values. Because a range object is an iterable, you can obtain the values by iterating over them with a for loop:

>>>
>>> for n in x:
...     print(n)
...
0
1
2
3
4

You could also snag all the values at once with list() or tuple() . In a REPL session, that can be a convenient way to quickly display what the values are:

>>>
>>> list(x)
[0, 1, 2, 3, 4]

>>> tuple(x)
(0, 1, 2, 3, 4)

However, when range() is used in code that is part of a larger application, it is typically considered poor practice to use list() or tuple() in this way. Like iterators, range objects are lazy -- the values in the specified range are not generated until they are requested. Using list() or tuple() on a range object forces all the values to be returned at once. This is rarely necessary, and if the list is long, it can waste time and memory.

range(<begin>, <end>, <stride>) returns an iterable that yields integers starting with <begin> , up to but not including <end> . If specified, <stride> indicates an amount to skip between values (analogous to the stride value used for string and list slicing):

>>>
>>> list(range(5, 20, 3))
[5, 8, 11, 14, 17]

If <stride> is omitted, it defaults to 1 :

>>>
>>> list(range(5, 10, 1))
[5, 6, 7, 8, 9]
>>> list(range(5, 10))
[5, 6, 7, 8, 9]

All the parameters specified to range() must be integers, but any of them can be negative. Naturally, if <begin> is greater than <end> , <stride> must be negative (if you want any results):

>>>
>>> list(range(-5, 5))
[-5, -4, -3, -2, -1, 0, 1, 2, 3, 4]

>>> list(range(5, -5))
[]
>>> list(range(5, -5, -1))
[5, 4, 3, 2, 1, 0, -1, -2, -3, -4]
Technical Note: Strictly speaking, range() isn't exactly a built-in function. It is implemented as a callable class that creates an immutable sequence type. But for practical purposes, it behaves like a built-in function.

For more information on range() , see the Real Python article Python's range() Function (Guide) . Remove ads Altering for Loop Behavior

You saw in the previous tutorial in this introductory series how execution of a while loop can be interrupted with break and continue statements and modified with an else clause . These capabilities are available with the for loop as well.

The break and continue Statements

break and continue work the same way with for loops as with while loops. break terminates the loop completely and proceeds to the first statement following the loop:

>>>
>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if 'b' in i:
...         break
...     print(i)
...
foo

continue terminates the current iteration and proceeds to the next iteration:

>>>
>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if 'b' in i:
...         continue
...     print(i)
...
foo
qux
The else Clause

A for loop can have an else clause as well. The interpretation is analogous to that of a while loop. The else clause will be executed if the loop terminates through exhaustion of the iterable:

>>>
>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     print(i)
... else:
...     print('Done.')  # Will execute
...
foo
bar
baz
qux
Done.

The else clause won't be executed if the list is broken out of with a break statement:

>>>
>>> for i in ['foo', 'bar', 'baz', 'qux']:
...     if i == 'bar':
...         break
...     print(i)
... else:
...     print('Done.')  # Will not execute
...
foo
Conclusion

This tutorial presented the for loop, the workhorse of definite iteration in Python.

You also learned about the inner workings of iterables and iterators , two important object types that underlie definite iteration, but also figure prominently in a wide variety of other Python code.

In the next two tutorials in this introductory series, you will shift gears a little and explore how Python programs can interact with the user via input from the keyboard and output to the console.

[Nov 13, 2019] I m annoyed with Python s ternary operator by Brian

Notable quotes:
"... THIS IS LAME!!! ..."
Jul 02, 2013 | pythontesting.net

46 Comments

The ternary operator is a way to concisely say:
"If test , then a , else b ",

with the value of the statement being the value of a or b .

language how to say it
C test ? a : b
C++ test ? a : b
javaScript test ? a : b
Perl (not perl 6) test ? a : b
PHP test ? a : b
Ruby test ? a : b
Julia test ? a : b
Did I forget some language? probably
Python a if test else b
Why??

Why??

Ok. Now that I've written this post, I'll remember it.

However, I just want to say on behalf of all of the other multiple-language programmers in the world, THIS IS LAME!!!

Joe , July 2, 2013 at 4:11 pm

All those languages essentially copied it from the granddaddy (i.e., C). Also, the ternary operator (properly called "conditional expressions") are a relatively recent addition to Python (2.5) and from what I heard Guido resisted adding it.

And the BDFL also didn't copy other C features, e.g., increment/decrement operators, || and && for Boolean operations, switch/case statements, and not least: BRACES!!! (for scope, of course).

Craig , July 2, 2013 at 7:16 pm

To be fair, I find the inconsistency to be a little jarring too. While a lot of python flows nicely this jars; and working across multiple languages is one of those oddities you need to remember for apparently no good reason.

[Nov 12, 2019] c - python (conditional-ternary) operator for assignments

Jul 30, 2014 | stackoverflow.com
This question already has an answer here:

karadoc ,May 14, 2013 at 13:01

Python has such an operator:
variable = something if condition else something_else

Alternatively, although not recommended (see @karadoc's comment):

variable = (condition and something) or something_else

[Nov 11, 2019] Python Advanced Python and the Shell

Nov 11, 2019 | www.python-course.eu

Python and the Shell Shell Python and Shell Shell is a term, which is often used and often misunderstood. Like the shell of an egg, either hen or Python snake, or a mussel, the shell in computer science is generally seen as a piece of software that provides an interface for a user to some other software or the operating system. So the shell can be an interface between the operating system and the services of the kernel of this operating system. But a web browser or a program functioning as an email client can be seen as shell as well.

Understanding this, it's obvious that a shell can be either

But in most cases the term shell is used as a synonym for a command line interface (CLI). The best known and most often used shells under Linux and Unix are the Bourne-Shell, C-Shell or Bash shell. The Bourne shell (sh) was modelled after the Multics shell, and is the first Unix shell.
Most operating system shells can be used in both interactive and batch mode. System programming System programming (also known as systems programming) stands for the activity of programming system components or system software. System programming provides software or services to the computer hardware, while application programming produces software which provides tools or services for the user.

"System focused programming" as it is made possible with the aid of the sys and the os module, serves as an abstraction layer between the application, i.e. the Python script or program, and the operating system, e.g. Linux or Microsoft Windows. By means of the abstraction layer it is possible to implement platform independent applications in Python, even if they access operating specific functionalities.

Therefore Python is well suited for system programming, or even platform independent system programming. The general advantages of Python are valid in system focused programming as well:

The os Module The os module is the most important module for interacting with the operating system. The os module allows platform independent programming by providing abstract methods. Nevertheless it is also possible by using the system() and the exec*() function families to include system independent program parts. (Remark: The exec*()-Functions are introduced in detail in our chapter " Forks and Forking in Python ")
The os module provides various methods, e.g. the access to the file system. Executing Shell scripts with os.system() It's not possible in Python to read a character without having to type the return key as well. On the other hand this is very easy on the Bash shell. The Bash command " read -n 1 waits for a key (any key) to be typed. If you import os, it's easy to write a script providing getch() by using os.system() and the Bash shell. getch() waits just for one character to be typed without a return:
import os
def getch():
     os.system("bash -c \"read -n 1\"")
 
getch()
The script above works only under Linux. Under Windows you will have to import the module msvcrt. Pricipially we only have to import getch() from this module.
So this is the Windows solution of the problem:
from msvcrt import getch
The following script implements a platform independent solution to the problem:
import os, platform
if platform.system() == "Windows":
    import msvcrt
def getch():
    if platform.system() == "Linux":
        os.system("bash -c \"read -n 1\"")
    else:
        msvcrt.getch()

print("Type a key!")
getch()
print("Okay")
The previous script harbours a problem. You can't use the getch() function, if you are interested in the key which has been typed, because os.system() doesn't return the result of the called shell commands.
We show in the following script, how we can execute shell scripts and return the output of these scripts into python by using os.popen():
>>> import os
>>> dir = os.popen("ls").readlines()
>>> print dir
['curses.py\n', 'curses.pyc\n', 'errors.txt\n', 'getch.py\n', 'getch.pyc\n', 'more.py\n',
'numbers.txt\n', 'output.txt\n', 'redirecting_output.py\n', 'redirecting_stderr2.py\n', 
'redirecting_stderr.py\n', 'streams.py\n',  'test.txt\n']
>>>
The output of the shell script can be read line by line, as can be seen in the following example:
import os

command = " "
while (command != "exit"):
    command = raw_input("Command: ")
    handle = os.popen(command)
    line = " "
    while line:
        line = handle.read()
        print line
    handle.close()

print "Ciao that's it!"
subprocess Module The subprocess module is available since Python 2.4.
It's possible to create spawn processes with the module subprocess, connect to their input, output, and error pipes, and obtain their return codes.
The module subprocess was created to replace various other modules: Working with the subprocess Module Instead of using the system method of the os-Module
os.system('touch xyz')
we can use the Popen() command of the subprocess Module. By using Popen() we are capable to get the output of the script:
>>> x = subprocess.Popen(['touch', 'xyz'])
>>> print x

>>> x.poll()
0
>>> x.returncode
0
The shell command cp -r xyz abc can be send to the shell from Python by using the Popen() method of the subprocess-Module in the following way:
p = subprocess.Popen(['cp','-r', "xyz", "abc"])
There is no need to escape the Shell metacharacters like $, > usw..
If you want to emulate the behaviour of os.system, the optional parameter shell has to be set to true, i.e.
shell=True
and we have to use a string instead of a list:
p=subprocess.Popen("cp -r xyz abc", shell=True)

As we have said above, it is also possible to catch the output from the shell command or shell script into Python. To do this, we have to set the optional parameter stdout of Popen() to subprocess.PIPE:
>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> print process.stdout.read()
total 132
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:03 abc
-rw-r--r-- 1 bernd bernd   0 2010-10-06 10:04 abcd
-rw-r--r-- 1 bernd bernd 660 2010-09-30 21:34 curses.py

If a shell command or shell script has been started with Popen(), the Python script doesn't wait until the shell command or shell script is finished. To wait until it is finished, you have to use the wait() method:

>>> process = subprocess.Popen(['ls','-l'], stdout=subprocess.PIPE)
>>> process.wait()
0
Functions to manipulate paths, files and directories
Function Description
getcwd() returns a string with the path of the current working directory
chdir(path) Change the current working directory to path.
Example under Windows:
>>> os.chdir("c:\Windows")
>>> os.getcwd()
'c:\\Windows'
An similiar example under Linux:
>>> import os
>>> os.getcwd()
'/home/homer'
>>> os.chdir("/home/lisa")
>>> os.getcwd()
'/home/lisa'
>>>
getcwdu() like getcwd() but unicode as output
listdir(path) A list with the content of the directory defined by "path", i.e. subdirectories and file names.
>>> os.listdir("/home/homer")
['.gnome2', '.pulse', '.gconf', '.gconfd', '.beagle', '.gnome2_private', '.gksu.lock', 'Public', '.ICEauthority', '.bash_history', '.compiz', '.gvfs', '.update-notifier', '.cache', 'Desktop', 'Videos', '.profile', '.config', '.esd_auth', '.viminfo', '.sudo_as_admin_successful', 'mbox', '.xsession-errors', '.bashrc', 'Music', '.dbus', '.local', '.gstreamer-0.10', 'Documents', '.gtk-bookmarks', 'Downloads', 'Pictures', '.pulse-cookie', '.nautilus', 'examples.desktop', 'Templates', '.bash_logout']
>>>
mkdir(path[, mode=0755]) Create a directory named path with numeric mode "mode", if it doesn't already exist. The default mode is 0777 (octal). On some systems, mode is ignored. If it is used, the current umask value is first masked out. If the directory already exists, OSError is raised. Parent directories will not be created, if they don't exist.
makedirs(name[, mode=511]) Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Raises an error exception if the leaf directory already exists or cannot be created.
rename(old, new) The file or directory "old" is renamed to "new" If "new" is a directory, an error will be raised. On Unix and Linux, if "new" exists and is a file, it will be replaced silently if the user has permission to do so.
renames(old, new) Works like rename(), except that it creates recursively any intermediate directories needed to make the "new" pathname.
rmdir(path) remove (delete) the directory "path". rmdir() works only, if the direcotry "path" is empty, otherwise an error is raised. To remove whole directory trees, shutil.rmdtree() can be used.

Further function and methods working on files and directories can be found in the module shutil. Amongst other possibilities it provides the possibility to copy files and directories with shutil.copyfile(src,dst).

Previous Chapter: Introduction into the sys module
Next Chapter: Forks and Forking in Python

[Nov 11, 2019] Python system command

Nov 11, 2019 | www.journaldev.com

In this tutorial we will learn about Python System Command. Previously we learned about Python Random Number .

Table of Contents [ hide ]

Python System Command

While making a program in python, you may need to exeucte some shell commands for your program. For example, if you use Pycharm IDE, you may notice that there is option to share your project on github. And you probably know that file transferring is done by git , which is operated using command line. So, Pycharm executes some shell commands in background to do it.

However, In this tutorial we will learn some basics about executing shell commands from your python code.

Python os.system() function

We can execute system command by using os.system() function. According to the official document, it has been said that

This is implemented by calling the Standard C function system(), and has the same limitations.

However, if command generates any output, it is sent to the interpreter standard output stream. Using this command is not recommended. In the following code we will try to know the version of git using the system command git --version .

import os cmd = "git --version" returned_value = os.system(cmd) # returns the exit code in unix print('returned value:', returned_value)

The following output found in ubuntu 16.04 where git is installed already.

git version 2 .14.2 returned value : 0

Notice that we are not printing the git version command output to console, it's being printed because console is the standard output stream here.

Python subprocess.call() Function

In the previous section, we saw that os.system() function works fine. But it's not recommended way to execute shell commands. We will use Python subprocess module to execute system commands.

We can run shell commands by using subprocess.call() function. See the following code which is equivalent to the previous code.

import subprocess cmd = "git --version" returned_value = subprocess.call(cmd, shell=True) # returns the exit code in unix print('returned value:', returned_value)

And the output will be same also.
Python System Command

Python subprocess.check_output() function

So far, we executed the system commands with the help of python. But we could not manipulate the output produced by those commands. Using subprocess.check_output() function we can store the output in a variable.

import subprocess cmd = "date" # returns output as byte string returned_output = subprocess.check_output(cmd) # using decode() function to convert byte string to string print('Current date is:', returned_output.decode( "utf-8" ))

It will produce output like the following

Current date is : Thu Oct 5 16 :31:41 IST 2017

So, in the above sections we have discussed about basic ideas about executing python system command. But there is no limit in learning. If you wish, you can learn more about Python System command using subprocess module from official documentation .

[Nov 11, 2019] python - How to access environment variable values

Nov 11, 2019 | stackoverflow.com
Environment variables are accessed through os.environ
import os
print(os.environ['HOME'])

Or you can see a list of all the environment variables using:

os.environ

As sometimes you might need to see a complete list!

# using get will return `None` if a key is not present rather than raise a `KeyError`
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))

# os.getenv is equivalent, and can also give a default value instead of `None`
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))

Python default installation on Windows is C:\Python . If you want to find out while running python you can do:

import sys
print(sys.prefix)

,

import sys
print sys.argv[0]

This will print foo.py for python foo.py , dir/foo.py for python dir/foo.py , etc. It's the first argument to python . (Note that after py2exe it would be foo.exe .)

[Nov 11, 2019] How can I find the current OS in Python

Nov 11, 2019 | stackoverflow.com

> ,May 29, 2012 at 21:57

Possible Duplicate:
Python: What OS am I running on?

As the title says, how can I find the current operating system in python?

Shital Shah ,Sep 23 at 23:34

I usually use sys.platform ( docs ) to get the platform. sys.platform will distinguish between linux, other unixes, and OS X, while os.name is " posix " for all of them.

For much more detailed information, use the platform module . This has cross-platform functions that will give you information on the machine architecture, OS and OS version, version of Python, etc. Also it has os-specific functions to get things like the particular linux distribution.

xssChauhan ,Sep 9 at 7:34

If you want user readable data but still detailed, you can use platform.platform()
>>> import platform
>>> platform.platform()
'Linux-3.3.0-8.fc16.x86_64-x86_64-with-fedora-16-Verne'

platform also has some other useful methods:

>>> platform.system()
'Windows'
>>> platform.release()
'XP'
>>> platform.version()
'5.1.2600'

Here's a few different possible calls you can make to identify where you are

import platform
import sys

def linux_distribution():
  try:
    return platform.linux_distribution()
  except:
    return "N/A"

print("""Python version: %s
dist: %s
linux_distribution: %s
system: %s
machine: %s
platform: %s
uname: %s
version: %s
mac_ver: %s
""" % (
sys.version.split('\n'),
str(platform.dist()),
linux_distribution(),
platform.system(),
platform.machine(),
platform.platform(),
platform.uname(),
platform.version(),
platform.mac_ver(),))

The outputs of this script ran on a few different systems (Linux, Windows, Solaris, MacOS) and architectures (x86, x64, Itanium, power pc, sparc) is available here: https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version

Steg ,Mar 31, 2015 at 15:13

import os
print os.name

This gives you the essential information you will usually need. To distinguish between, say, different editions of Windows, you will have to use a platform-specific method.

UnkwnTech ,Sep 21, 2008 at 6:17

https://docs.python.org/library/os.html

To complement Greg's post, if you're on a posix system, which includes MacOS, Linux, Unix, etc. you can use os.uname() to get a better feel for what kind of system it is.

> ,

Something along the lines:
import os
if (os.name == "posix"):
    print os.system("uname -a")
# insert other possible OSes here
# ...
else:
    print "unknown OS"

[Nov 08, 2019] Is losing BDFL a death sentence for open source projects such as Python? by Jason Baker

Jul 16, 2018 | opensource.com
What happens when a Benevolent Dictator For Life moves on from an open source project? up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 08, 2019] The Python community is special

Oct 29, 2019 | opensource.com

Moshe Zadka (Community Moderator) Feed 26 up 2 comments The Python community is amazing. It was one of the first to adopt a code of conduct, first for the Python Software Foundation and then for PyCon . There is a real commitment to diversity and inclusion: blog posts and conference talks on this theme are frequent, thoughtful, and well-read by Python community members.

While the community is global, there is a lot of great activity in the local community as well. Local Python meet-ups are a great place to meet wonderful people who are smart, experienced, and eager to help. A lot of meet-ups will explicitly have time set aside for experienced people to help newcomers who want to learn a new concept or to get past an issue with their code. My local community took the time to support me as I began my Python journey, and I am privileged to continue to give back to new developers.

Whether you can attend a local community meet-up or you spend time with the online Python community across IRC, Slack, and Twitter, I am sure you will meet lovely people who want to help you succeed as a developer.

[Nov 08, 2019] Selected Python-related articles from opensource.org

There are lots of ways to save time as a sysadmin. Here's one way to build a web app using open source tools to automate a chunk of your daily routine away
Nov 08, 2019 | opensource.com
  1. A dozen ways to learn Python These resources will get you started and well on your way to proficiency with Python. Don Watkins (Community Moderator) 27 Aug 2019 51
  2. Building a non-breaking breakpoint for Python debugging Have you ever wondered how to speed up a debugger? Here are some lessons learned while building one for Python. Liran Haimovitch 13 Aug 2019 39
  3. Sending custom emails with Python Customize your group emails with Mailmerge, a command-line program that can handle simple and complex emails. Brian "bex" Exelbierd (Red Hat) 08 Aug 2019 42
  4. Save and load Python data with JSON The JSON format saves you from creating your own data formats, and is particularly easy to learn if you already know Python. Here's how to use it with Python. Seth Kenlon (Red Hat) 16 Jul 2019 36
  5. Parse arguments with Python Parse Python like a pro with the argparse module. Seth Kenlon (Red Hat) 03 Jul 2019 59
  6. Learn object-oriented programming with Python Make your code more modular with Python classes. Seth Kenlon (Red Hat) 05 Jul 2019 62
  7. Get modular with Python functions Minimize your coding workload by using Python functions for repeating tasks. Seth Kenlon (Red Hat) 01 Jul 2019 62
  8. Learn Python with these awesome resources Expand your Python knowledge by adding these resources to your personal learning network. Don Watkins (Community Moderator) 31 May 2019 71
  9. Run your blog on GitHub Pages with Python Create a blog with Pelican, a Python-based blogging platform that works well with GitHub. Erik O'Shaughnessy 23 May 2019 76
  10. 100 ways to learn Python and R for data science Want to learn data science? Find recommended courses in the Data Science Repo, a community-sourced directory of Python and R learning resources. Chris Engelhardt Dorris Scott Annu Singh 15 May 2019 59
  11. How to analyze log data with Python and Apache Spark Case study with NASA logs to show how Spark can be leveraged for analyzing data at scale. Dipanjan (DJ) Sarkar (Red Hat) 14 May 2019 76
  12. Learn Python by teaching in your community A free and fun way to learn by teaching others. Don Watkins (Community Moderator) 13 May 2019 47
  13. Format Python however you like with Black Learn more about solving common Python problems in our series covering seven PyPI libraries. Moshe Zadka (Community Moderator) 02 May 2019 80
  14. Write faster C extensions for Python with Cython Learn more about solving common Python problems in our series covering seven PyPI libraries. Moshe Zadka (Community Moderator) 01 May 2019 64
  15. Managing Python packages the right way Don't fall victim to the perils of Python package management. László Kiss Kollár 04 Apr 2019 75
  16. 7 steps for hunting down Python code bugs Learn some tricks to minimize the time you spend tracking down the reasons your code fails. Maria Mckinley 08 Feb 2019 108
  17. Getting started with Pelican: A Python-based static site generator Pelican is a great choice for Python users who want to self-host a simple website or blog. Craig Sebenik 07 Jan 2019 119
  18. Python 3.7 beginner's cheat sheet Get acquainted with Python's built-in pieces. Nicholas Hunt-Walker 20 Sep 2018 177
  19. 18 Python programming books for beginners and veterans Get started with this popular language or buff up on your coding skills with this curated book list. Jen Wike Huger (Red Hat) 10 Sep 2018 157
  20. Why I love Xonsh Ever wondered if Python can be your shell? Moshe Zadka (Community Moderator) 04 Sep 2018 123

[Nov 08, 2019] Pylint: Making your Python code consistent

Oct 21, 2019 | opensource.com

Pylint is your friend when you want to avoid arguing about code complexity.

Moshe Zadka (Community Moderator) Feed 30 up 1 comment

Get the highlights in your inbox every week.

flake8 and black will take care of "local" style: where the newlines occur, how comments are formatted, or find issues like commented out code or bad practices in log formatting.

Pylint is extremely aggressive by default. It will offer strong opinions on everything from checking if declared interfaces are actually implemented to opportunities to refactor duplicate code, which can be a lot to a new user. One way of introducing it gently to a project, or a team, is to start by turning all checkers off, and then enabling checkers one by one. This is especially useful if you already use flake8, black, and mypy : Pylint has quite a few checkers that overlap in functionality.

More Python Resources

However, one of the things unique to Pylint is the ability to enforce higher-level issues: for example, number of lines in a function, or number of methods in a class.

These numbers might be different from project to project and can depend on the development team's preferences. However, once the team comes to an agreement about the parameters, it is useful to enforce those parameters using an automated tool. This is where Pylint shines.

Configuring Pylint

In order to start with an empty configuration, start your .pylintrc with

[MESSAGES CONTROL]

disable=all

This disables all Pylint messages. Since many of them are redundant, this makes sense. In Pylint, a message is a specific kind of warning.

You can check that all messages have been turned off by running pylint :

$ pylint <my package>

In general, it is not a great idea to add parameters to the pylint command-line: the best place to configure your pylint is the .pylintrc . In order to have it do something useful, we need to enable some messages.

In order to enable messages, add to your .pylintrc , under the [MESSAGES CONTROL] .

enable = <message>,

...

For the "messages" (what Pylint calls different kinds of warnings) that look useful. Some of my favorites include too-many-lines , too-many-arguments , and too-many-branches . All of those limit complexity of modules or functions, and serve as an objective check, without a human nitpicker needed, for code complexity measurement.

A checker is a source of messages : every message belongs to exactly one checker. Many of the most useful messages are under the design checker . The default numbers are usually good, but tweaking the maximums is straightfoward: we can add a section called DESIGN in the .pylintrc .

[ DESIGN ]

max-args=7

max-locals=15

Another good source of useful messages is the refactoring checker. Some of my favorite messages to enable there are consider-using-dict-comprehension , stop-iteration-return (which looks for generators which use raise StopIteration when return is the correct way to stop the iteration). and chained-comparison , which will suggest using syntax like 1 <= x < 5 rather than the less obvious 1 <= x && x > 5

Finally, an expensive checker, in terms of performance, but highly useful, is similarities . It is designed to enforce "Don't Repeat Yourself" (the DRY principle) by explicitly looking for copy-paste between different parts of the code. It only has one message to enable: duplicate-code . The default "minimum similarity lines" is set to 4 . It is possible to set it to a different value using the .pylintrc .

[ SIMILARITIES ]

min-similarity-lines=3

Pylint makes code reviews easy

If you are sick of code reviews where you point out that a class is too complicated, or that two different functions are basically the same, add Pylint to your Continuous Integration configuration, and only have the arguments about complexity guidelines for your project once .

[Nov 07, 2019] Is BDFL a death sentence Opensource.com

Nov 07, 2019 | opensource.com

What happens when a Benevolent Dictator For Life moves on from an open source project? 16 Jul 2018 Jason Baker (Red Hat) Feed 131 up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 02, 2019] William J. Turkel and Adam Crymble, Manipulating Strings in Python,

Oct 16, 2019 | programminghistorian.org

Lesson Goals This lesson is a brief introduction to This lesson is a brief introduction to string manipulation techniques in Python. Knowing how to manipulate strings plays a crucial role in most text processing tasks. If you’d like to experiment with the following lessons, you can write and execute short programs as we’ve been doing in previous lessons in the series, or you can open up a Python shell / Terminal to try them out on the command line. Manipulating Python Strings If you have been exposed to another programming language before, you might have learned that you need to If you have been exposed to another programming language before, you might have learned that you need to declare or type variables before you can store anything in them. This is not necessary when working with strings in Python. We can create a string simply by putting content wrapped with quotation marks into it with an equal sign (=):

message = "Hello World"
String Operators: Adding and Multiplying A string is a type of object, one that consists of a series of characters. Python already knows how to deal with a number of general-purpose and powerful representations, including strings. One way to manipulate strings is by using A string is a type of object, one that consists of a series of characters. Python already knows how to deal with a number of general-purpose and powerful representations, including strings. One way to manipulate strings is by using string operators . These operators are represented by symbols that you likely associate with mathematics, such as +, -, *, /, and =. When used with strings, they perform actions that are similar to, but not the same as, their mathematical counterparts. Concatenate This term means to join strings together. The process is known as This term means to join strings together. The process is known as concatenating strings and it is done using the plus (+) operator. Note that you must be explicit about where you want blank spaces to occur by placing them between single quotation marks also. In this example, the string “message1” is given the content “hello world”. In this example, the string “message1” is given the content “hello world”. In this example, the string “message1” is given the content “hello world”.
message1 = 'hello' + ' ' + 'world'
print(message1)
-> hello world
Multiply If you want multiple copies of a string, use the multiplication (*) operator. In this example, string If you want multiple copies of a string, use the multiplication (*) operator. In this example, string message2a is given the content “hello” times three; string message 2b is given content “world”; then we print both strings.
message2a = 'hello ' * 3
message2b = 'world'
print(message2a + message2b)
-> hello hello hello world
Append What if you want to add material to the end of a string successively? There is a special operator for that (+=). What if you want to add material to the end of a string successively? There is a special operator for that (+=).
message3 = 'howdy'
message3 += ' '
message3 += 'world'
print(message3)
-> howdy world
String Methods: Finding, Changing In addition to operators, Python comes pre-installed with dozens of string methods that allow you to do things to strings. Used alone or in combination, these methods can do just about anything you can imagine to strings. The good news is that you can reference a list of String Methods on the In addition to operators, Python comes pre-installed with dozens of string methods that allow you to do things to strings. Used alone or in combination, these methods can do just about anything you can imagine to strings. The good news is that you can reference a list of String Methods on the Python website , including information on how to use each properly. To make sure that you’ve got a basic grasp of string methods, what follows is a brief overview of some of the more commonly used ones: Length You can determine the number of characters in a string using You can determine the number of characters in a string using len . Note that the blank space counts as a separate character.
message4 = 'hello' + ' ' + 'world'
print(len(message4))
-> 11
Find You can search a string for a You can search a string for a substring and your program will return the starting index position of that substring. This is helpful for further processing. Note that indexes are numbered from left to right and that the count starts with position 0, not 1.
message5 = "hello world"
message5a = message5.find("worl")
print(message5a)
-> 6
If the substring is not present, the program will return a value of -1. If the substring is not present, the program will return a value of -1.
message6 = "Hello World"
message6b = message6.find("squirrel")
print(message6b)
-> -1
Lower Case Sometimes it is useful to convert a string to lower case. For example, if we standardize case it makes it easier for the computer to recognize that “Sometimes” and “sometimes” are the same word. Sometimes it is useful to convert a string to lower case. For example, if we standardize case it makes it easier for the computer to recognize that “Sometimes” and “sometimes” are the same word.
message7 = "HELLO WORLD"
message7a = message7.lower()
print(message7a)
-> hello world
The opposite effect, raising characters to upper case, can be achieved by changing The opposite effect, raising characters to upper case, can be achieved by changing .lower() to .upper() . Replace If you need to replace a substring throughout a string you can do so with the If you need to replace a substring throughout a string you can do so with the replace method.
message8 = "HELLO WORLD"
message8a = message8.replace("L", "pizza")
print(message8a)
-> HEpizzapizzaO WORpizzaD
Slice If you want to If you want to slice off unwanted parts of a string from the beginning or end you can do so by creating a substring. The same kind of technique also allows you to break a long string into more manageable components.
message9 = "Hello World"
message9a = message9[1:8]
print(message9a)
-> ello Wo
You can substitute variables for the integers used in this example. You can substitute variables for the integers used in this example.
startLoc = 2
endLoc = 8
message9b = message9[startLoc: endLoc]
print(message9b)
-> llo Wo
This makes it much easier to use this method in conjunction with the This makes it much easier to use this method in conjunction with the find method as in the next example, which checks for the letter “d” in the first six characters of “Hello World” and correctly tells us it is not there (-1). This technique is much more useful in longer strings â€" entire documents for example. Note that the absence of an integer before the colon signifies we want to start at the beginning of the string. We could use the same technique to tell the program to go all the way to the end by putting no integer after the colon. And remember, index positions start counting from 0 rather than 1.
message9 = "Hello World"
print(message9[:5].find("d"))
-> -1
There are lots more, but the string methods above are a good start. Note that in this last example, we are using square brackets instead of parentheses. This difference in There are lots more, but the string methods above are a good start. Note that in this last example, we are using square brackets instead of parentheses. This difference in syntax signals an important distinction. In Python, parentheses are usually used to pass an argument to a function. So when we see something like
print(len(message7))
it means pass the string it means pass the string message7 to the function len then send the returned value of that function to the print statement to be printed. If a function can be called without an argument, you often have to include a pair of empty parentheses after the function name anyway. We saw an example of that, too:
message7 = "HELLO WORLD"
message7a = message7.lower()
print(message7a)
-> hello world
This statement tells Python to apply the This statement tells Python to apply the lower function to the string message7 and store the returned value in the string message7a . The square brackets serve a different purpose. If you think of a string as a sequence of characters, and you want to be able to access the contents of the string by their location within the sequence, then you need some way of giving Python a location within a sequence. That is what the square brackets do: indicate a beginning and ending location within a sequence as we saw when using the The square brackets serve a different purpose. If you think of a string as a sequence of characters, and you want to be able to access the contents of the string by their location within the sequence, then you need some way of giving Python a location within a sequence. That is what the square brackets do: indicate a beginning and ending location within a sequence as we saw when using the The square brackets serve a different purpose. If you think of a string as a sequence of characters, and you want to be able to access the contents of the string by their location within the sequence, then you need some way of giving Python a location within a sequence. That is what the square brackets do: indicate a beginning and ending location within a sequence as we saw when using the slice method. Escape Sequences What do you do when you need to include quotation marks within a string? You don’t want the Python interpreter to get the wrong idea and end the string when it comes across one of these characters. In Python, you can put a backslash (\) in front of a quotation mark so that it doesn’t terminate the string. These are known as escape sequences. What do you do when you need to include quotation marks within a string? You don’t want the Python interpreter to get the wrong idea and end the string when it comes across one of these characters. In Python, you can put a backslash (\) in front of a quotation mark so that it doesn’t terminate the string. These are known as escape sequences.
print('\"')
-> "
print('The program printed \"hello world\"')
-> The program printed "hello world"
Two other escape sequences allow you to print tabs and newlines: Two other escape sequences allow you to print tabs and newlines:
print('hello\thello\thello\nworld')
->hello hello hello
world
Suggested Reading Code Syncing To follow along with future lessons it is important that you have the right files and programs in your programming-historian directory. At the end of each chapter you can download the programming-historian zip file to make sure you have the correct code. Note we have removed unneeded files from earlier lessons. Your directory may contain more files and that’s ok! To follow along with future lessons it is important that you have the right files and programs in your programming-historian directory. At the end of each chapter you can download the programming-historian zip file to make sure you have the correct code. Note we have removed unneeded files from earlier lessons. Your directory may contain more files and that’s ok! Great! Now you're ready to move on to the next lesson .
About the authors
William J. Turkel is Professor of History at the University of Western Ontario. Adam Crymble is a senior lecturer of digital history at the University of Hertfordshire. William J. Turkel is Professor of History at the University of Western Ontario. Adam Crymble is a senior lecturer of digital history at the University of Hertfordshire. Adam Crymble is a senior lecturer of digital history at the University of Hertfordshire. Adam Crymble is a senior lecturer of digital history at the University of Hertfordshire.

[Nov 02, 2019] Copied variable changes the original

Nov 14, 2011 | stackoverflow.com

Copied variable changes the original? Ask Question Asked 7 years, 11 months ago Active 2 years, 9 months ago Viewed 61k times 46 17


André Freitas ,Nov 14, 2011 at 13:56

I have a simple problem in Python that is very very strange.
def estExt(matriz,erro):
    # (1) Determinar o vector X das soluções
    print ("Matrix after:");
    print(matriz);

    aux=matriz;
    x=solucoes(aux); # IF aux is a copy of matrix, why the matrix is changed??

    print ("Matrix before: ");
    print(matriz)

...

As you see below, the matrix matriz is changed in spite of the fact that aux is the one being changed by the function solucoes() .

Matrix before:
[[7, 8, 9, 24], [8, 9, 10, 27], [9, 10, 8, 27]]

Matrix after:
[[7, 8, 9, 24], [0.0, -0.14285714285714235, -0.2857142857142847, -0.42857142857142705], [0.0, 0.0, -3.0, -3.0000000000000018]]

André Freitas ,Nov 14, 2011 at 17:16

The line
aux=matriz;

Does not make a copy of matriz , it merely creates a new reference to matriz named aux . You probably want

aux=matriz[:]

Which will make a copy, assuming matriz is a simple data structure. If it is more complex, you should probably use copy.deepcopy

aux = copy.deepcopy(matriz)

As an aside, you don't need semi-colons after each statement, python doesn't use them as EOL markers.

André Freitas ,Nov 15, 2011 at 8:49

Use copy module
aux = copy.deepcopy(matriz) # there is copy.copy too for shallow copying

Minor one: semicolons are not needed.

aux is not a copy of matrix , it's just a different name that refers to the same object.

[Oct 22, 2019] Perl vs Python log processing performance

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 6 years, 11 months ago Active 6 years, 11 months ago Viewed 2k times 0 0

texasbruce ,Nov 11, 2012 at 2:05

I am working on a web-based log management system that will be built on the Grails framework and I am going to use one of the text processing languages like Python or Perl. I have created Python and Perl scripts that load log files and parse each line to save them to a MySQL database (the file contains about 40,000 lines, about 7MB). It took 1 min 2 secs using Perl and only 17 secs using Python .

I had supposed that Perl would be faster than Python, as Perl is the original text processing language (my suspicions also coming from different blogs where I was reading about Perl text processing performance).

Also I was not expecting a 47 second difference between Perl and Python. Why is Perl taking more time than Python to process my log file? Is it because I am using some wrong db module or my code and regular expression for Perl can be improved?

Note: I am a Java and Groovy developer and I have no experience with Perl (I am using Strawberry Perl v5.16). Also I have made this test with Java (1 min 5 secs) and Groovy (1 min 7 secs) but more than 1 min to process the log file is too much, so both languages are out and now I want to choose between Perl and Python.

PERL Code

use DBI;
use DBD::mysql;
# make connection to database
$connection = DBI->connect("dbi:mysql:logs:localhost:3306","root","") || die      "Cannot connect: $DBI::errstr";

# set the value of your SQL query
$query = "insert into logs (line_number, dated, time_stamp, thread, level, logger, user, message)
        values (?, ?, ?, ?, ?, ?, ?, ?) ";

# prepare your statement for connecting to the database
$statement = $connection->prepare($query); 

$runningTime = time;

# open text file
open (LOG,'catalina2.txt') || die "Cannot read logfile!\n";;

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}  

# close the open text file
close(LOG);

# close database connection
$connection->disconnect;

$runningTime = time - $runningTime;
printf("\n\nTotal running time: %02d:%02d:%02d\n\n", int($runningTime / 3600),   int(($runningTime % 3600) / 60), int($runningTime % 60));

# exit the script
exit;

PYTHON Code

import re
import mysql.connector
import time

file = open("D:\catalina2.txt","r")
rexp = re.compile('^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$')
conn = mysql.connector.connect(user='root',host='localhost',database='logs')
cursor = conn.cursor()

tic = time.clock()

increment  = 1
for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

conn.commit()
cursor.close()
conn.close()

toc = time.clock()
print "Total time: %s" % (toc - tic)

David-SkyMesh ,Nov 11, 2012 at 1:35

It is not a fair comparison:

You are only calling cursor.execute once in Python:

for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

But you are calling $statement->execute many times in Perl:

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}

By the way, for the Python version, calling cursor.execute once for every row will be slow. You can make it faster by using cursor.executemany :

sql = 'insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)'
args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))

cursor.executemany(sql, args)

If there are too many lines in the log file, you may need to break this up into blocks:

args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))
    if increment % 1000 == 0:
        cursor.executemany(sql, args)
        args = []
if args:
    cursor.executemany(sql, args)

(Also, don't use file.readlines() because this creates a list (which may be huge). file is an iterator which spits out one line at a time, so for text in file suffices.)

[Oct 22, 2019] Difference in regex behavior between Perl and Python?

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 10 years, 6 months ago Active 10 years, 6 months ago Viewed 2k times 3 1


Gumbo ,Apr 16, 2009 at 18:42

I have a couple email addresses, 'support@company.com' and '1234567@tickets.company.com' .

In perl, I could take the To: line of a raw email and find either of the above addresses with

/\w+@(tickets\.)?company\.com/i

In python, I simply wrote the above regex as '\w+@(tickets\.)?company\.com' expecting the same result. However, support@company.com isn't found at all and a findall on the second returns a list containing only 'tickets.' . So clearly the '(tickets\.)?' is the problem area, but what exactly is the difference in regular expression rules between Perl and Python that I'm missing?

Axeman ,Apr 16, 2009 at 21:10

The documentation for re.findall :
findall(pattern, string, flags=0)
    Return a list of all non-overlapping matches in the string.

    If one or more groups are present in the pattern, return a
    list of groups; this will be a list of tuples if the pattern
    has more than one group.

    Empty matches are included in the result.

Since (tickets\.) is a group, findall returns that instead of the whole match. If you want the whole match, put a group around the whole pattern and/or use non-grouping matches, i.e.

r'(\w+@(tickets\.)?company\.com)'
r'\w+@(?:tickets\.)?company\.com'

Note that you'll have to pick out the first element of each tuple returned by findall in the first case.

chaos ,Apr 16, 2009 at 18:45

I think the problem is in your expectations of extracted values. Try using this in your current Python code:
'(\w+@(?:tickets\.)?company\.com)'

Jason Coon ,Apr 16, 2009 at 18:46

Two problems jump out at me:
  1. You need to use a raw string to avoid having to escape " \ "
  2. You need to escape " . "

So try:

r'\w+@(tickets\.)?company\.com'

EDIT

Sample output:

>>> import re
>>> exp = re.compile(r'\w+@(tickets\.)?company\.com')
>>> bool(exp.match("s@company.com"))
True
>>> bool(exp.match("1234567@tickets.company.com"))
True

,

There isn't a difference in the regexes, but there is a difference in what you are looking for. Your regex is capturing only "tickets." if it exists in both regexes. You probably want something like this
#!/usr/bin/python

import re

regex = re.compile("(\w+@(?:tickets\.)?company\.com)");

a = [
    "foo@company.com", 
    "foo@tickets.company.com", 
    "foo@ticketsacompany.com",
    "foo@compant.org"
];

for string in a:
    print regex.findall(string)

[Oct 14, 2019] Python Strings, Functions and Examples by Meenakshi Agarwal

Oct 14, 2019 | www.techbeamers.com
How to Create Strings in Python? Creating strings is easy as you only need to enclose the characters either in single or double-quotes. In the following example, we are providing different ways to initialize strings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings. Creating strings is easy as you only need to enclose the characters either in single or double-quotes. In the following example, we are providing different ways to initialize strings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings. In the following example, we are providing different ways to initialize strings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings. In the following example, we are providing different ways to initialize strings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings. To share an important note that you can also use triple quotes to create strings. However, programmers use them to mark multi-line strings and docstrings.
# Python string examples - all assignments are identical.
String_var = 'Python'
String_var = "Python"
String_var = """Python"""

# with Triple quotes Strings can extend to multiple lines
String_var = """ This document will help you to
explore all the concepts
of Python Strings!!! """

# Replace "document" with "tutorial" and store in another variable
substr_var = String_var.replace("document", "tutorial")
print (substr_var)

Index and Slice Strings in Python

Access Individual Characters of a String You need to know the index of a character to retrieve it from the String. Like the most programming languages, Python allows to index from the zeroth position in Strings. But it also supports negative indexes. Index of '-1' represents the last character of the String. Similarly using '-2', we can access the penultimate element of the string and so on. You need to know the index of a character to retrieve it from the String. Like the most programming languages, Python allows to index from the zeroth position in Strings. But it also supports negative indexes. Index of '-1' represents the last character of the String. Similarly using '-2', we can access the penultimate element of the string and so on. Like the most programming languages, Python allows to index from the zeroth position in Strings. But it also supports negative indexes. Index of '-1' represents the last character of the String. Similarly using '-2', we can access the penultimate element of the string and so on. Like the most programming languages, Python allows to index from the zeroth position in Strings. But it also supports negative indexes. Index of '-1' represents the last character of the String. Similarly using '-2', we can access the penultimate element of the string and so on.

sample_str = 'Python String'

print (sample_str[0])       # return 1st character
# output: P

print (sample_str[-1])      # return last character
# output: g

print (sample_str[-2])      # return last second character
# output: n
Slice a String in Python To retrieve a range of characters in a String, we use 'slicing operator,' the colon ':' sign. With the slicing operator, we define the range as [a:b]. It'll let us print all the characters of the String starting from index 'a' up to char at index 'b-1'. So the char at index 'b' is not a part of the output. To retrieve a range of characters in a String, we use 'slicing operator,' the colon ':' sign. With the slicing operator, we define the range as [a:b]. It'll let us print all the characters of the String starting from index 'a' up to char at index 'b-1'. So the char at index 'b' is not a part of the output.
sample_str = 'Python String'
print (sample_str[3:5])     #return a range of character
# ho
print (sample_str[7:])      # return all characters from index 7
# String
print (sample_str[:6])      # return all characters before index 6
# Python
print (sample_str[7:-4])
# St
Next, we have a no. of Python tutorials/quizzes/interview questions on this blog. If you like to try them, then refer any of the posts listed below. Suggested Reading: ☛ Next, we have a no. of Python tutorials/quizzes/interview questions on this blog. If you like to try them, then refer any of the posts listed below. Suggested Reading: ☛ Suggested Reading: ☛ Suggested Reading: ☛ ☛ 100+ Python Interview Questions TOC Python Strings – Common Error Codes 1- If we try to retrieve characters at out of range index, then 'IndexError' exception will be raised. 1- If we try to retrieve characters at out of range index, then 'IndexError' exception will be raised.
sample_str = "Python Supports Machine Learning."
print (sample_str[1024])      #index must be in range

# IndexError: string index out of range
2- String index must be of the integer data type. You should not use a float or any other data type for this purpose. Otherwise, the Python subsystem will flag a TypeError exception as it detects a data type violation for the string index. 2- String index must be of the integer data type. You should not use a float or any other data type for this purpose. Otherwise, the Python subsystem will flag a TypeError exception as it detects a data type violation for the string index.
sample_str = "Welcome post"
print (sample_str[1.25])      #index must be an integer

# TypeError: string indices must be integers
Modify/Delete a String in Python Python Strings are by design immutable. It suggests that once a String binds to a variable; it can't be modified. If you want to update the String, then re-assign a new String value to the same variable. Python Strings are by design immutable. It suggests that once a String binds to a variable; it can't be modified. If you want to update the String, then re-assign a new String value to the same variable. If you want to update the String, then re-assign a new String value to the same variable. If you want to update the String, then re-assign a new String value to the same variable.
sample_str = 'Python String'
sample_str[2] = 'a'

# TypeError: 'str' object does not support item assignment

sample_str = 'Programming String'
print (sample_str)

# Output=> Programming String
Similarly, we cannot modify the Strings by deleting some characters from it. Instead, we can remove the Strings altogether by using the 'del' command. Similarly, we cannot modify the Strings by deleting some characters from it. Instead, we can remove the Strings altogether by using the 'del' command.
sample_str = "Python is the best scripting language."
del sample_str[1]
# TypeError: 'str' object doesn't support item deletion

del sample_str
print (sample_str)
# NameError: name 'sample_str' is not defined
Suggested Reading: ☛ Suggested Reading: ☛ ☛ Python Programming Interview Questions TOC String Operators in Python Concatenation (+) It combines two strings into one. It combines two strings into one.
# example
var1 = 'Python'
var2 = 'String'
print (var1+var2)
# PythonString
Repetition (*) This operator creates a new string by repeating it a given number of times. This operator creates a new string by repeating it a given number of times.
# example
var1 = 'Python'
print (var1*3)
# PythonPythonPython
Slicing [ ] The slice operator prints the character at a given index. The slice operator prints the character at a given index.
# example
var1 = 'Python'
print (var1[2])
# t
Range Slicing [x:y] It prints the characters present in the given range. It prints the characters present in the given range.
# example
var1 = 'Python'
print (var1[2:5])
# tho
Membership (in) This operator returns 'True' value if the character is present in the given String. This operator returns 'True' value if the character is present in the given String.
# example
var1 = 'Python'
print ('n' in var1)
# True
Membership (not in) It returns 'True' value if the character is not present in the given String. It returns 'True' value if the character is not present in the given String.
# example
var1 = 'Python'
print ('N' not in var1)
# True
Iterating (for) With this operator, we can iterate through all the characters of a string. With this operator, we can iterate through all the characters of a string.
# example
for var in var1: print (var, end ="")
# Python
Raw String (r/R) We can use it to ignore the actual meaning of Escape characters inside a string. For this, we add 'r' or 'R' in front of the String. We can use it to ignore the actual meaning of Escape characters inside a string. For this, we add 'r' or 'R' in front of the String.
# example
print (r'\n')
# \n
print (R'\n')
# \n
TOC

String Formatting Operators in Python

Python Escape Characters

An Escape sequence starts with a backslash (\) which signals the compiler to treat it differently. Python subsystem automatically interprets an escape sequence irrespective of it is in a single-quoted or double-quoted Strings.

We need a way to tell Python that the double-quotes inside the string are not the string markup quotes. Instead, they are the part of the String and should appear in the output. To resolve this issue, we can escape the double-quotes and single-quotes as: We need a way to tell Python that the double-quotes inside the string are not the string markup quotes. Instead, they are the part of the String and should appear in the output. To resolve this issue, we can escape the double-quotes and single-quotes as: To resolve this issue, we can escape the double-quotes and single-quotes as: To resolve this issue, we can escape the double-quotes and single-quotes as:

print ("Python is a "widely" used language")

# SyntaxError: invalid syntax

# After escaping with double-quotes

print ("Python is a \"widely\" used language")

# Output: Python is a "widely" used language
List of Escape Characters
Here is the complete list of escape characters that are represented using backslash notation.

\\ Backslash (\)

\" Double-quote (")

\a ASCII bell (BEL)

\b ASCII backspace (BS)

\cx or \Cx Control-x

\f ASCII Form feed (FF)

\n ASCII linefeed (LF)

\N{name} Character named name in the Unicode database (Unicode only)

\r Carriage Return (CR)

\t Horizontal Tab (TAB)

\uxxxx A character with 16-bit hex value xxxx (Unicode only)

\Uxxxxxxxx A character with 32-bit hex value xxxxxxxx (Unicode only)

\v ASCII vertical tab (VT)

\ooo Characters with octal value ooo

\xnn A character with hex value nn where n can be anything from the range 0-9, a-f or A-F.

... ... ...

Here's a simple example. Here's a simple example. Here's a simple example.

print ("Employee Name: %s,\nEmployee Age:%d" % ('Ashish',25))
# Employee Name: Ashish, 
# Employee Age: 25
List of Format Symbols

Following is the table containing the complete list of symbols that you can use with the '%' operator.

Unicode String support in Python

Regular Strings stores as the 8-bit ASCII value whereas Unicode String follows the 16-bit ASCII standard. This extension allows the strings to include characters from the different languages of the world. In Python, the letter 'u' works as a prefix to distinguish between Unicode and usual strings.

print (u' Hello Python!!')
#Hello Python

Suggested Reading:

Python Online Practice Test

TOC

Built-in String Functions in Python Conversion Functions

1. capitalize() – Returns the string with the first character capitalized and rest of the characters in lower case.

var = 'PYTHON'
print (var.capitalize())
# Python

2. lower() – Converts all the characters of the String to lowercase

var = 'TechBeamers'
print (var.lower())
# techbeamers

3. upper() – Converts all the characters of the String to uppercase

var = 'TechBeamers'
print (var.upper())
# TECHBEAMERS

4. swapcase() – Swaps the case of every character in the String means that lowercase characters got converted to uppercase and vice-versa.

var = 'TechBeamers'
print (var.swapcase())
# tECHbEAMERS

5. title() – Returns the 'titlecased' version of String, which means that all words start with uppercase and the rest of the characters in words are in lowercase.

var = 'welcome to Python programming'
print (var.title())
# Welcome To Python Programming

6. count( str[, beg [, end]]) – Returns the number of times substring 'str' occurs in the range [beg, end] if beg and end index are given else the search continues in full String Search is case-sensitive.

var='TechBeamers'
str='e'
print (var.count(str))
# 3
var1='Eagle Eyes'
print (var1.count('e'))
# 2
var2='Eagle Eyes'
print (var2.count('E',0,5))
# 1

TOC

Comparison Functions – Part1

1. islower() – Returns 'True' if all the characters in the String are in lowercase. If any of the char is in uppercase, it will return False.

var='Python'
print (var.islower())
# False

var='python'
print (var.islower())
# True

2. isupper() – Returns 'True' if all the characters in the String are in uppercase. If any of the char is in lowercase, it will return False.

var='Python'
print (var.isupper())
# False

var='PYTHON'
print (var.isupper())
# True

3. isdecimal() – Returns 'True' if all the characters in String are decimal. If any character in the String is of other data-type, it will return False.

Decimal characters are those from the Unicode category Nd.

num=u'2016'
print (num.isdecimal())
# True

4. isdigit() – Returns 'True' for any char for which isdecimal() would return 'True and some characters in the 'No' category. If there are any characters other than these, it will return False'.

Precisely, digits are the characters for which Unicode property includes: Numeric_Type=Digit or Numeric_Type=Decimal.

For example, superscripts are digits but fractions not.

print ('2'.isdigit())
# True

print ('²'.isdigit())
# True
Comparison Functions – Part2

1. isnumeric() – Returns 'True' if all the characters of the Unicode String lie in any one of the categories Nd', No, and NI.

If there are any characters other than these, it will return False.

Precisely, Numeric characters are those for which Unicode property includes: Numeric_Type=Digit, Numeric_Type=Decimal or Numeric_Type=Numeric.

num=u'2016'
print (num.isnumeric())
# True

num=u'year2016'
print (num.isnumeric())
# False

2. isalpha() – Returns 'True' if String contains at least one character (non-empty String) and all the characters are alphabetic, 'False' otherwise.

print ('python'.isalpha())
# True

print ('python3'.isalpha())
# False

3. isalnum() – Returns 'True' if String contains at least one character (non-empty String) and all the characters are either alphabetic or decimal digits, 'False' otherwise.

print ('python'.isalnum())
# True
print ('python3'.isalnum())
# True

TOC

Padding Functions

1. rjust(width[,fillchar]) – Returns string filled with input char while pushing the original content on the right side.

By default, the padding uses a space. Otherwise 'fillchar' specifies the filler character.

var='Python'
print (var.rjust(10))
# Python

print (var.rjust(10,'-'))
# ----Python

2. ljust(width[,fillchar]) – Returns a padded version of String with the original String left-justified to a total of width columns

By default, the padding uses a space. Otherwise 'fillchar' specifies the filler character.

var='Python'
print (var.ljust(10))
# Python

print (var.ljust(10,'-'))
# Python----

3. center(width[,fillchar]) – Returns string filled with the input char while pushing the original content into the center.

By default, the padding uses a space. Otherwise 'fillchar' specifies the filler character.

var='Python'
print (var.center(20))
# Python

print (var.center(20,'*'))
# *******Python*******

4. zfill(width) – Returns string filled with the original content padded on the left with zeros so that total length of String becomes equal to the input size.

If there is a leading sign (+/-) present in the String, then with this function padding starts after the symbol, not before it.

var='Python'
print (var.zfill(10))
# 0000Python

var='+Python'
print (var.zfill(10))
# +000Python

TOC

Search Functions

1. find(str [,i [,j]]) – Searches for 'str' in complete String (if i and j not defined) or in a sub-string of String (if i and j are defined).This function returns the index if 'str' is found else returns '-1'.

Here, i=search starts from this index, j=search ends at this index.

var="Tech Beamers"
str="Beam"
print (var.find(str))
# 5

var="Tech Beamers"
str="Beam"
print (var.find(str,4))
# 5

var="Tech Beamers"
str="Beam"
print (var.find(str,7))
# -1

2. index(str[,i [,j]]) – This is same as 'find' method. The only difference is that it raises 'ValueError' exception if 'str' doesn't exist.

var='Tech Beamers'
str='Beam'
print (var.index(str))
# 5

var='Tech Beamers'
str='Beam'
print (var.index(str,4))
# 5

var='Tech Beamers'
str='Beam'
print (var.index(str,7))
# ValueError: substring not found

3. rfind(str[,i [,j]]) – This is same as find() just that this function returns the last index where 'str' is found. If 'str' is not found, it returns '-1'.

var='This is a good example'
str='is'
print (var.rfind(str,0,10))
# 5

print (var.rfind(str,10))
# -1

4. count(str[,i [,j]]) – Returns the number of occurrences of substring 'str' in the String. Searches for 'str' in the complete String (if i and j not defined) or in a sub-string of String (if i and j are defined).

Where: i=search starts from this index, j=search ends at this index.

var='This is a good example'
str='is'
print (var.count(str))
# 2

print (var.count(str,4,10))
# 1

TOC

String Substitution Functions

1. replace(old,new[,count]) – Replaces all the occurrences of substring 'old' with 'new' in the String.

If the count is available, then only 'count' number of occurrences of 'old' will be replaced with the 'new' var.

Where old =substring to replace, new =substring

var='This is a good example'
str='was'
print (var.replace('is',str))
# Thwas was a good exampleprint (var.replace('is',str,1))
# Thwas is a good example

2. split([sep[,maxsplit]]) – Returns a list of substring obtained after splitting the String with 'sep' as a delimiter.

Where, sep= delimiter, the default is space, maxsplit= number of splits to be done

var = "This is a good example"
print (var.split())
# ['This', 'is', 'a', 'good', 'example']print (var.split(' ', 3))
# ['This', 'is', 'a', 'good example']

3. splitlines(num) – Splits the String at line breaks and returns the list after removing the line breaks.

Where, num = if this is a positive value. It indicates that line breaks will appear in the returned list.

var='Print new line\nNextline\n\nMove again to new line'
print (var.splitlines())
# ['Print new line', 'Nextline', '', 'Move again to new line']print (var.splitlines(1))
# ['Print new line\n', 'Nextline\n', '\n', 'Move again to new line']

4. join(seq) – Returns a String obtained after concatenating the sequence 'seq' with a delimiter string.

Where: the seq= sequence of elements to join

seq=('ab','bc','cd')
str='='
print (str.join(seq))
# ab=bc=cd

TOC

Misc String Functions

1. lstrip([chars]) – Returns a string after removing the characters from the beginning of the String.

Where: Chars=this is the character to be trimmed from the String.

The default is whitespace character.

var=' This is a good example '
print (var.lstrip())
# This is a good example
var='*****This is a good example*****'
print (var.lstrip('*'))
# This is a good example**********

2. rstrip() – Returns a string after removing the characters from the End of the String.

Where: Chars=this is the character to be trimmed from the String. The default is whitespace character.

var=' This is a good example '
print (var.rstrip())
# This is a good example
var='*****This is a good example*****'
print (var.lstrip('*'))
# *****This is a good example

3. rindex(str[,i [,j]]) – Searches for 'str' in the complete String (if i and j not defined) or in a sub-string of String (if i and j are defined). This function returns the last index where 'str' is available.

If 'str' is not there, then it raises a ValueError exception.

Where: i=search starts from this index, j=search ends at this index.

var='This is a good example'
str='is'
print (var.rindex(str,0,10))
# 5
print (var.rindex(str,10))
# ValueError: substring not found

4. len(string) – Returns the length of given String

var='This is a good example'
print (len(var))
# 22

TOC

In this post, we tried to cover most of the string functionality available in Python. Hope that you would now have a better understanding of the Python strings.

If you have any question regarding Python strings, please let us know. We'll try to solve it at the earliest possible time.

In Python 3.6, a new style known as f-strings got introduced, do go through it.

All the Best,

TechBeamers

[Oct 14, 2019] Coroutines and Tasks -- Python 3.7.5rc1 documentation

Oct 14, 2019 | docs.python.org

Coroutines declared with async/await syntax is the preferred way of writing asyncio applications. For example, the following snippet of code (requires Python 3.7+) prints "hello", waits 1 second, and then prints "world":

>>>

>>> import asyncio

>>> async def main():
...     print('hello')
...     await asyncio.sleep(1)
...     print('world')

>>> asyncio.run(main())
hello
world

Note that simply calling a coroutine will not schedule it to be executed:

>>>
>>> main()
<coroutine object main at 0x1053bb7c8>

To actually run a coroutine, asyncio provides three main mechanisms:

Awaitables ¶

We say that an object is an awaitable object if it can be used in an await expression. Many asyncio APIs are designed to accept awaitables.

There are three main types of awaitable objects: coroutines , Tasks , and Futures .

Coroutines

Python coroutines are awaitables and therefore can be awaited from other coroutines:

import asyncio

async def nested():
    return 42

async def main():
    # Nothing happens if we just call "nested()".
    # A coroutine object is created but not awaited,
    # so it *won't run at all*.
    nested()

    # Let's do it differently now and await it:
    print(await nested())  # will print "42".

asyncio.run(main())

Important

In this documentation the term "coroutine" can be used for two closely related concepts:

asyncio also supports legacy generator-based coroutines.

[Oct 14, 2019] Coroutine in Python - GeeksforGeeks

Oct 14, 2019 | www.geeksforgeeks.org

In Python 2.5, a slight modification to the yield statement was introduced, now yield can also be used as expression . For example on the right side of the assignment

line = (yield)

whatever value we send to coroutine is captured and returned by (yield) expression. A value can be send to the coroutine by send() method. For example, consider this coroutine which print out name having prefix "Dear" in it. We will send names to coroutine using send() method.

# Python3 program for demonstrating

# coroutine execution

def print_name(prefix):

print ( "Searching prefix:{}" . format (prefix))

while True :

name = ( yield )

if prefix in name:

print (name) # calling coroutine, nothing will happen

corou = print_name( "Dear" )

# This will start execution of coroutine and

# Prints first line "Searchig prefix..."

# and advance execution to the first yield expression

corou.__next__() # sending inputs

corou.send( "Atul" )

corou.send( "Dear Atul" )

Output:

Searching prefix:Dear
Dear Atul

Execution of Coroutine

Execution of coroutine is similar to the generator. When we call coroutine nothing happens, it runs only in response to the next() and send() method. This can be seen clearly in above example, as only after calling __next__() method, out coroutine starts executing. After this call, execution advances to the first yield expression, now execution pauses and wait for value to be sent to corou object. When first value is sent to it, it checks for prefix and print name if prefix present. After printing name it goes through loop until it encounters name = (yield) expression again.

Closing a Coroutine

Coroutine might run indefinitely, to close coroutine close() method is used. When coroutine is closed it generates GeneratorExit exception which can be catched in usual way. After closing coroutine, if we try to send values, it will raise StopIteration exception. Following is a simple example :

# Python3 program for demonstrating

# closing a coroutine

def print_name(prefix):

print ( "Searching prefix:{}" . format (prefix))

try :

while True :

name = ( yield )

if prefix in name:

print (name)

except GeneratorExit:

print ( "Closing coroutine!!" )

corou = print_name( "Dear" )

corou.__next__()

corou.send( "Atul" )

corou.send( "Dear Atul" )

corou.close()

Output:

Searching prefix:Dear
Dear Atul
Closing coroutine!!

Chaining coroutines for creating pipeline

Coroutines can be used to set pipes. We can chain together coroutines and push data through pipe using send() method. A pipe needs :

pipeline
Following is a simple example of chaining –

# Python3 program for demonstrating

# coroutine chaining

def producer(sentence, next_coroutine):

'''

Producer which just split strings and

feed it to pattern_filter coroutine

'''

tokens = sentence.split( " " )

for token in tokens:

next_coroutine.send(token)

next_coroutine.close()

def pattern_filter(pattern = "ing" , next_coroutine = None ):

'''

Search for pattern in received token

and if pattern got matched, send it to

print_token() coroutine for printing

'''

print ( "Searching for {}" . format (pattern))

try :

while True :

token = ( yield )

if pattern in token:

next_coroutine.send(token)

except GeneratorExit:

print ( "Done with filtering!!" ) def print_token():

'''

Act as a sink, simply print the

received tokens

'''

print ( "I'm sink, i'll print tokens" )

try :

while True :

token = ( yield )

print (token)

except GeneratorExit:

print ( "Done with printing!" )

pt = print_token()

pt.__next__()

pf = pattern_filter(next_coroutine = pt)

pf.__next__() sentence = "Bob is running behind a fast moving car"

producer(sentence, pf)

Output:

I'm sink, i'll print tokens
Searching for ing
running
moving
Done with filtering!!
Done with printing!

References

This article is contributed by Atul Kumar . If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.

[Oct 13, 2019] What is the system function in python

Oct 13, 2019 | stackoverflow.com

what is the system function in python Ask Question Asked 9 years, 3 months ago Active 9 years, 3 months ago Viewed 6k times 1


Eva Feldman ,Jul 6, 2010 at 15:55

I want to play with system command in python . for example we have this function in perl : system("ls -la"); and its run ls -la what is the system function in python ? Thanks in Advance .

Felix Kling ,Jul 6, 2010 at 15:58

It is os.system :
import os
os.system('ls -la')

But this won't give you any output. So subprocess.check_output is probably more what you want:

>>> import subprocess
>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

KLee1 ,Jul 6, 2010 at 16:00

import os
os.system("")

From here

> ,

In the os module there is os.system() .

But if you want to do more advanced things with subprocesses the subprocess module provides a higher level interface with more possibilities that is usually preferable.

[Oct 13, 2019] Python generators and coroutines

The new syntax is 'yield from' ( PEP 380 ) and it allows true coroutines in Python >3.3
Nov 16, 2017 | stackoverflow.com

Python generators and coroutines Ask Question up vote down vote favorite 6

Giuseppe Maggiore ,May 10, 2011 at 10:25

I am studying coroutines and generators in various programming languages.

I was wondering if there is a cleaner way to combine together two coroutines implemented via generators than yielding back at the caller whatever the callee yields?

Let's say that we are using the following convention: all yields apart from the last one return null, while the last one returns the result of the coroutine. So, for example, we could have a coroutine that invokes another:

def A():
  # yield until a certain condition is met
  yield result

def B():
  # do something that may or may not yield
  x = bind(A())
  # ...
  return result

in this case I wish that through bind (which may or may not be implementable, that's the question) the coroutine B yields whenever A yields until A returns its final result, which is then assigned to x allowing B to continue.

I suspect that the actual code should explicitly iterate A so:

def B():
  # do something that may or may not yield
  for x in A(): ()
  # ...
  return result

which is a tad ugly and error prone...

PS: it's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better!

S.Lott ,May 10, 2011 at 10:38

This is all covered on the Python web site. python.org/dev/peps/pep-0342 , python.org/dev/peps/pep-0334 and numerous blogs cover this. eecho.info/Echo/python/coroutines-python . Please Google, read, and then ask specific questions based on what you've read. – S.Lott May 10 '11 at 10:38

S.Lott ,May 10, 2011 at 13:04

I thought the examples clearly demonstrated idiomatic. Since I'm unable to understand what's wrong with the examples, could you state which examples you found to be unclear? Which examples were confusing? Can you be more specific on how all those examples where not able to show idiomatic Python? – S.Lott May 10 '11 at 13:04

Giuseppe Maggiore ,May 10, 2011 at 13:09

I've read precisely those articles, and the PEP-342 leaves me somewhat confused: is it some actual extension that is currently working in Python? Is the Trampoline class shown there part of the standard libraries of the language? BTW, my question was very precise, and it was about the IDIOMATIC way to pass control around coroutines. The fact that I can read about a ton of ways to do so really does not help. Neither does your snarkiness... – Giuseppe Maggiore May 10 '11 at 13:09

Giuseppe Maggiore ,May 10, 2011 at 13:11

Idiomatic is about the "standard" way to perform some function; there is absolutely nothing wrong with iterating the results of a nested coroutine, but there are examples in the literature of programming languages where yielding automatically climbs down the call stack and so you do not need to re-yield at each caller, hence my curiosity if this pattern is covered by sintactic sugar in Python or not! – Giuseppe Maggiore May 10 '11 at 13:11

S.Lott ,May 10, 2011 at 13:19

@Giuseppe Maggiore: "programming languages where yielding automatically climbs down the call stack" That doesn't sound like the same question. Are you asking for idiomatic Python -- as shown by numerous examples -- or are you asking for some other feature that's not shown in the Python examples but is shown in other languages? I'm afraid that I can't understand your question at all. Can you please clarify what you're really looking for? – S.Lott May 10 '11 at 13:19

blubb ,May 10, 2011 at 10:37

Are you looking for something like this?
def B():
   for x in A():
     if x is None:
       yield
     else:
       break

   # continue, x contains value A yielded

Giuseppe Maggiore ,May 10, 2011 at 12:59

Yes, that is what I am doing. My question is if this is the idiomatic way or if there is some syntactic construct that is capable of hiding this pattern which recurs very often in my application. – Giuseppe Maggiore May 10 '11 at 12:59

blubb ,May 10, 2011 at 13:31

@Guiseppe Maggiore: I'm not aware of any such constructs. However, it seems strange that you need this pattern often... I can't think of many valid used cases off the top of my head. If you give more context information, maybe we can propose an alternative solution which is more elegant over all? – blubb May 10 '11 at 13:31

Giuseppe Maggiore ,May 10, 2011 at 15:17

It's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better! – Giuseppe Maggiore May 10 '11 at 15:17

blubb ,May 10, 2011 at 15:57

@Guiseppe Maggiore: I'd propose you add that last comment to the question so that other get a chance of answering it, too... – blubb May 10 '11 at 15:57

Simon Radford ,Nov 11, 2011 at 0:24

Edit: I recommend using Greenlet . But if you're interested in a pure Python approach, read on.

This is addressed in PEP 342 , but it's somewhat tough to understand at first. I'll try to explain simply how it works.

First, let me sum up what I think is the problem you're really trying to solve.

Problem

You have a callstack of generator functions calling other generator functions. What you really want is to be able to yield from the generator at the top, and have the yield propagate all the way down the stack.

The problem is that Python does not ( at a language level ) support real coroutines, only generators. (But, they can be implemented.) Real coroutines allow you to halt an entire stack of function calls and switch to a different stack. Generators only allow you to halt a single function. If a generator f() wants to yield, the yield statement has to be in f(), not in another function that f() calls.

The solution that I think you're using now, is to do something like in Simon Stelling's answer (i.e. have f() call g() by yielding all of g()'s results). This is very verbose and ugly, and you're looking for syntax sugar to wrap up that pattern. Note that this essentially unwinds the stack every time you yield, and then winds it back up again afterwards.

Solution

There is a better way to solve this problem. You basically implement coroutines by running your generators on top of a "trampoline" system.

To make this work, you need to follow a couple patterns: 1. When you want to call another coroutine, yield it. 2. Instead of returning a value, yield it.

so

def f():
    result = g()
    #  
    return return_value

becomes

def f():
    result = yield g()
    #  
    yield return_value

Say you're in f(). The trampoline system called f(). When you yield a generator (say g()), the trampoline system calls g() on your behalf. Then when g() has finished yielding values, the trampoline system restarts f(). This means that you're not actually using the Python stack; the trampoline system manages a callstack instead.

When you yield something other than a generator, the trampoline system treats it as a return value. It passes that value back to the caller generator through the yield statement (using .send() method of generators).

Comments

This kind of system is extremely important and useful in asynchronous applications, like those using Tornado or Twisted. You can halt an entire callstack when it's blocked, go do something else, and then come back and continue execution of the first callstack where it left off.

The drawback of the above solution is that it requires you to write essentially all your functions as generators. It may be better to use an implementation of true coroutines for Python - see below.

Alternatives

There are several implementations of coroutines for Python, see: http://en.wikipedia.org/wiki/Coroutine#Implementations_for_Python

Greenlet is an excellent choice. It is a Python module that modifies the CPython interpreter to allow true coroutines by swapping out the callstack.

Python 3.3 should provide syntax for delegating to a subgenerator, see PEP 380 .

gaborous ,Nov 9, 2012 at 10:04

Very useful and clear answer, thank's! However, when you say that standard Python coroutines essentially require to write all functions as generators, did you mean only first level functions or really all functions? As you said above, when yielding something other than a generator, the trampoline system still works, so theoretically we can just yield at the first-layer functions any other functions that may or may not be generators themselves. Am I right? – gaborous Nov 9 '12 at 10:04

Simon Radford ,Nov 21, 2012 at 21:37

All "functions" between the trampoline system and a yield must be written as generators. You can call regular functions normally, but then you can't effectively "yield" from that function or any functions it calls. Does that make sense / answer your question? – Simon Radford Nov 21 '12 at 21:37

Simon Radford ,Nov 21, 2012 at 21:39

I highly recommend using Greenlet - it's a true implementation of coroutines for Python, and you don't have to use any of these patterns I've described. The trampoline stuff is for people who are interested in how you can do it in pure Python. – Simon Radford Nov 21 '12 at 21:39

Nick Sweeting ,Jun 7, 2015 at 22:12

To anyone reading this in 2015 or later, the new syntax is 'yield from' ( PEP 380 ) and it allows true coroutines in Python >3.3 . – Nick Sweeting Jun 7 '15 at 22:12

[Oct 13, 2019] Effective Python Item 40 Consider Coroutines to Run Many Functions Concurrently

Nov 16, 2017 | www.informit.com

Threads give Python programmers a way to run multiple functions seemingly at the same time (see Item 37: "Use Threads for Blocking I/O, Avoid for Parallelism"). But there are three big problems with threads:

Python can work around all these issues with coroutines . Coroutines let you have many seemingly simultaneous functions in your Python programs. They're implemented as an extension to generators. The cost of starting a generator coroutine is a function call. Once active, they each use less than 1 KB of memory until they're exhausted.

Coroutines work by enabling the code consuming a generator to send a value back into the generator function after each yield expression. The generator function receives the value passed to the send function as the result of the corresponding yield expression.

def my_coroutine():
    while True:
        received = yield
        print('Received:', received)

it = my_coroutine()
next(it)             # Prime the coroutine
it.send('First')
it.send('Second')

>>>
Received: First
Received: Second

The initial call to next is required to prepare the generator for receiving the first send by advancing it to the first yield expression. Together, yield and send provide generators with a standard way to vary their next yielded value in response to external input.

For example, say you want to implement a generator coroutine that yields the minimum value it's been sent so far. Here, the bare yield prepares the coroutine with the initial minimum value sent in from the outside. Then the generator repeatedly yields the new minimum in exchange for the next value to consider.

def minimize():
    current = yield
    while True:
        value = yield current
        current = min(value, current)

The code consuming the generator can run one step at a time and will output the minimum value seen after each input.

it = minimize()
next(it)            # Prime the generator
print(it.send(10))
print(it.send(4))
print(it.send(22))
print(it.send(-1))

>>>
10
4
4
-1

The generator function will seemingly run forever, making forward progress with each new call to send . Like threads, coroutines are independent functions that can consume inputs from their environment and produce resulting outputs. The difference is that coroutines pause at each yield expression in the generator function and resume after each call to send from the outside. This is the magical mechanism of coroutines.

This behavior allows the code consuming the generator to take action after each yield expression in the coroutine. The consuming code can use the generator's output values to call other functions and update data structures. Most importantly, it can advance other generator functions until their next yield expressions. By advancing many separate generators in lockstep, they will all seem to be running simultaneously, mimicking the concurrent behavior of Python threads.

The Game of Life

Let me demonstrate the simultaneous behavior of coroutines with an example. Say you want to use coroutines to implement Conway's Game of Life. The rules of the game are simple. You have a two-dimensional grid of an arbitrary size. Each cell in the grid can either be alive or empty.

ALIVE = '*'
EMPTY = '-'

The game progresses one tick of the clock at a time. At each tick, each cell counts how many of its neighboring eight cells are still alive. Based on its neighbor count, each cell decides if it will keep living, die, or regenerate. Here's an example of a 5×5 Game of Life grid after four generations with time going to the right. I'll explain the specific rules further below.

  0   |   1   |   2   |   3   |   4
----- | ----- | ----- | ----- | -----
-*--- | --*-- | --**- | --*-- | -----
--**- | --**- | -*--- | -*--- | -**--
---*- | --**- | --**- | --*-- | -----
----- | ----- | ----- | ----- | -----

I can model this game by representing each cell as a generator coroutine running in lockstep with all the others.

To implement this, first I need a way to retrieve the status of neighboring cells. I can do this with a coroutine named count_neighbors that works by yielding Query objects. The Query class I define myself. Its purpose is to provide the generator coroutine with a way to ask its surrounding environment for information.

Query = namedtuple('Query', ('y', 'x'))

The coroutine yields a Query for each neighbor. The result of each yield expression will be the value ALIVE or EMPTY . That's the interface contract I've defined between the coroutine and its consuming code. The count_neighbors generator sees the neighbors' states and returns the count of living neighbors.

def count_neighbors(y, x):
    n_ = yield Query(y + 1, x + 0)  # North
    ne = yield Query(y + 1, x + 1)  # Northeast
    # Define e_, se, s_, sw, w_, nw ...
    # ...
    neighbor_states = [n_, ne, e_, se, s_, sw, w_, nw]
    count = 0
    for state in neighbor_states:
        if state == ALIVE:
            count += 1
    return count

I can drive the count_neighbors coroutine with fake data to test it. Here, I show how Query objects will be yielded for each neighbor. count_neighbors expects to receive cell states corresponding to each Query through the coroutine's send method. The final count is returned in the StopIteration exception that is raised when the generator is exhausted by the return statement.

it = count_neighbors(10, 5)
q1 = next(it)                  # Get the first query
print('First yield: ', q1)
q2 = it.send(ALIVE)            # Send q1 state, get q2
print('Second yield:', q2)
q3 = it.send(ALIVE)            # Send q2 state, get q3
# ...
try:
    count = it.send(EMPTY)     # Send q8 state, retrieve count
except StopIteration as e:
    print('Count: ', e.value)  # Value from return statement
>>>
First yield:  Query(y=11, x=5)
Second yield: Query(y=11, x=6)
...
Count:  2

Now I need the ability to indicate that a cell will transition to a new state in response to the neighbor count that it found from count_neighbors . To do this, I define another coroutine called step_cell . This generator will indicate transitions in a cell's state by yielding Transition objects. This is another class that I define, just like the Query class.

Transition = namedtuple('Transition', ('y', 'x', 'state'))

The step_cell coroutine receives its coordinates in the grid as arguments. It yields a Query to get the initial state of those coordinates. It runs count_neighbors to inspect the cells around it. It runs the game logic to determine what state the cell should have for the next clock tick. Finally, it yields a Transition object to tell the environment the cell's next state.

def game_logic(state, neighbors):
    # ...

def step_cell(y, x):
    state = yield Query(y, x)
    neighbors = yield from count_neighbors(y, x)
    next_state = game_logic(state, neighbors)
    yield Transition(y, x, next_state)

Importantly, the call to count_neighbors uses the yield from expression. This expression allows Python to compose generator coroutines together, making it easy to reuse smaller pieces of functionality and build complex coroutines from simpler ones. When count_neighbors is exhausted, the final value it returns (with the return statement) will be passed to step_cell as the result of the yield from expression.

Now, I can finally define the simple game logic for Conway's Game of Life. There are only three rules.

def game_logic(state, neighbors):
    if state == ALIVE:
        if neighbors < 2:
            return EMPTY     # Die: Too few
        elif neighbors > 3:
            return EMPTY     # Die: Too many
    else:
        if neighbors == 3:
            return ALIVE     # Regenerate
    return state

I can drive the step_cell coroutine with fake data to test it.

it = step_cell(10, 5)
q0 = next(it)           # Initial location query
print('Me:      ', q0)
q1 = it.send(ALIVE)     # Send my status, get neighbor query
print('Q1:      ', q1)
# ...
t1 = it.send(EMPTY)     # Send for q8, get game decision
print('Outcome: ', t1)

>>>
Me:       Query(y=10, x=5)
Q1:       Query(y=11, x=5)
...
Outcome:  Transition(y=10, x=5, state='-')

The goal of the game is to run this logic for a whole grid of cells in lockstep. To do this, I can further compose the step_cell coroutine into a simulate coroutine. This coroutine progresses the grid of cells forward by yielding from step_cell many times. After progressing every coordinate, it yields a TICK object to indicate that the current generation of cells have all transitioned.

TICK = object()

def simulate(height, width):
    while True:
        for y in range(height):
            for x in range(width):
                yield from step_cell(y, x)
        yield TICK

What's impressive about simulate is that it's completely disconnected from the surrounding environment. I still haven't defined how the grid is represented in Python objects, how Query , Transition , and TICK values are handled on the outside, nor how the game gets its initial state. But the logic is clear. Each cell will transition by running step_cell . Then the game clock will tick. This will continue forever, as long as the simulate coroutine is advanced.

This is the beauty of coroutines. They help you focus on the logic of what you're trying to accomplish. They decouple your code's instructions for the environment from the implementation that carries out your wishes. This enables you to run coroutines seemingly in parallel. This also allows you to improve the implementation of following those instructions over time without changing the coroutines.

Now, I want to run simulate in a real environment. To do that, I need to represent the state of each cell in the grid. Here, I define a class to contain the grid:

class Grid(object):
    def __init__(self, height, width):
        self.height = height
        self.width = width
        self.rows = []
        for _ in range(self.height):
            self.rows.append([EMPTY] * self.width)

    def __str__(self):
        # ...

The grid allows you to get and set the value of any coordinate. Coordinates that are out of bounds will wrap around, making the grid act like infinite looping space.

    def query(self, y, x):
        return self.rows[y % self.height][x % self.width]

    def assign(self, y, x, state):
        self.rows[y % self.height][x % self.width] = state

At last, I can define the function that interprets the values yielded from simulate and all of its interior coroutines. This function turns the instructions from the coroutines into interactions with the surrounding environment. It progresses the whole grid of cells forward a single step and then returns a new grid containing the next state.

def live_a_generation(grid, sim):
    progeny = Grid(grid.height, grid.width)
    item = next(sim)
    while item is not TICK:
        if isinstance(item, Query):
            state = grid.query(item.y, item.x)
            item = sim.send(state)
        else:  # Must be a Transition
            progeny.assign(item.y, item.x, item.state)
            item = next(sim)
    return progeny

To see this function in action, I need to create a grid and set its initial state. Here, I make a classic shape called a glider.

grid = Grid(5, 9)
grid.assign(0, 3, ALIVE)
# ...
print(grid)

>>>
---*-----
----*----
--***----
---------
---------

Now I can progress this grid forward one generation at a time. You can see how the glider moves down and to the right on the grid based on the simple rules from the game_logic function.

class ColumnPrinter(object):
    # ...

columns = ColumnPrinter()
sim = simulate(grid.height, grid.width)
for i in range(5):
    columns.append(str(grid))
    grid = live_a_generation(grid, sim)

print(columns)

>>>
    0     |     1     |     2     |     3     |     4
---*----- | --------- | --------- | --------- | ---------
----*---- | --*-*---- | ----*---- | ---*----- | ----*----
--***---- | ---**---- | --*-*---- | ----**--- | -----*---
--------- | ---*----- | ---**---- | ---**---- | ---***---
--------- | --------- | --------- | --------- | ---------

The best part about this approach is that I can change the game_logic function without having to update the code that surrounds it. I can change the rules or add larger spheres of influence with the existing mechanics of Query , Transition , and TICK . This demonstrates how coroutines enable the separation of concerns, which is an important design principle.

Coroutines in Python 2

Unfortunately, Python 2 is missing some of the syntactical sugar that makes coroutines so elegant in Python 3. There are two limitations. First, there is no yield from expression. That means that when you want to compose generator coroutines in Python 2, you need to include an additional loop at the delegation point.

# Python 2
def delegated():
    yield 1
    yield 2

def composed():
    yield 'A'
    for value in delegated():  # yield from in Python 3
        yield value
    yield 'B'

print list(composed())

>>>
['A', 1, 2, 'B']

The second limitation is that there is no support for the return statement in Python 2 generators. To get the same behavior that interacts correctly with try / except / finally blocks, you need to define your own exception type and raise it when you want to return a value.

# Python 2
class MyReturn(Exception):
    def __init__(self, value):
        self.value = value

def delegated():
    yield 1
    raise MyReturn(2)  # return 2 in Python 3
    yield 'Not reached'

def composed():
    try:
        for value in delegated():
            yield value
    except MyReturn as e:
        output = e.value
    yield output * 4

print list(composed())

>>>
[1, 8]
Things to Remember

[Oct 13, 2019] https://www.quora.com/If-Donald-Knuth-were-25-years-old-today-which-programming-language-would-he-choose

Notable quotes:
"... He mostly writes in C today. ..."
Oct 13, 2019 | www.quora.com

Eugene Miya , A friend/colleague. Sometimes driver. Other shared experiences. Updated Mar 22 2017 · Author has 11.2k answers and 7.9m answer views

He mostly writes in C today.

I can assure you he at least knows about Python. Guido's office at Dropbox is 1 -- 2 blocks by a backdoor gate from Don's house.

I would tend to doubt that he would use R (I've used S before as one of my stat packages). Don would probably write something for himself.

Don is not big on functional languages, so I would doubt either Haskell (sorry Paul) or LISP (but McCarthy lived just around the corner from Don; I used to drive him to meetings; actually, I've driven all 3 of us to meetings, and he got his wife an electric version of my car based on riding in my car (score one for friend's choices)). He does use emacs and he does write MLISP macros, but he believes in being closer to the hardware which is why he sticks with MMIX (and MIX) in his books.

Don't discount him learning the machine language of a given architecture.

I'm having dinner with Don and Jill and a dozen other mutual friends in 3 weeks or so (our quarterly dinner). I can ask him then, if I remember (either a calendar entry or at job). I try not to bother him with things like this. Don is well connected to the hacker community

Don's name was brought up at an undergrad architecture seminar today, but Don was not in the audience (an amazing audience; I took a photo for the collection of architects and other computer scientists in the audience (Hennessey and Patterson were talking)). I came close to biking by his house on my way back home.

We do have a mutual friend (actually, I introduced Don to my biology friend at Don's request) who arrives next week, and Don is my wine drinking proxy. So there is a chance I may see him sooner.

Steven de Rooij , Theoretical computer scientist Answered Mar 9, 2017 · Author has 4.6k answers and 7.7m answer views

Nice question :-)

Don Knuth would want to use something that’s low level, because details matter . So no Haskell; LISP is borderline. Perhaps if the Lisp machine ever had become a thing.

He’d want something with well-defined and simple semantics, so definitely no R. Python also contains quite a few strange ad hoc rules, especially in its OO and lambda features. Yes Python is easy to learn and it looks pretty, but Don doesn’t care about superficialities like that. He’d want a language whose version number is converging to a mathematical constant, which is also not in favor of R or Python.

What remains is C. Out of the five languages listed, my guess is Don would pick that one. But actually, his own old choice of Pascal suits him even better. I don’t think any languages have been invented since was written that score higher on the Knuthometer than Knuth’s own original pick.

And yes, I feel that this is actually a conclusion that bears some thinking about. 24.1k views ·

Dan Allen , I've been programming for 34 years now. Still not finished. Answered Mar 9, 2017 · Author has 4.5k answers and 1.8m answer views

In The Art of Computer Programming I think he'd do exactly what he did. He'd invent his own architecture and implement programs in an assembly language targeting that theoretical machine.

He did that for a reason because he wanted to reveal the detail of algorithms at the lowest level of detail which is machine level.

He didn't use any available languages at the time and I don't see why that would suit his purpose now. All the languages above are too high-level for his purposes.

[Sep 30, 2019] Get command line arguments as string

Sep 30, 2019 | stackoverflow.com

KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

[Sep 28, 2019] Python quoting conventions

Sep 28, 2019 | stackoverflow.com

Share a link to this answer Copy link edited Jul 7 '11 at 14:11 answered Sep 11 '08 at 10:06 Will Harris Will Harris 20.4k 11 11 gold badges 60 60 silver badges 63 63 bronze badges

| show 2 more comments 96 votes

mlissner ,Aug 6, 2011 at 6:19

Quoting the official docs at https://docs.python.org/2.0/ref/strings.html :

In plain English: String literals can be enclosed in matching single quotes (') or double quotes (").

So there is no difference. Instead, people will tell you to choose whichever style that matches the context, and to be consistent . And I would agree - adding that it is pointless to try to come up with "conventions" for this sort of thing because you'll only end up confusing any newcomers.

eksortso ,Jun 10, 2013 at 23:36

I used to prefer ' , especially for '''docstrings''' , as I find """this creates some fluff""" . Also, ' can be typed without the Shift key on my Swiss German keyboard.

I have since changed to using triple quotes for """docstrings""" , to conform to PEP 257 .

Garth Kidd ,Sep 11, 2008 at 10:21

I'm with Will:

I'll stick with that even if it means a lot of escaping.

I get the most value out of single quoted identifiers standing out because of the quotes. The rest of the practices are there just to give those single quoted identifiers some standing room.

Tony Meyer ,Sep 11, 2008 at 8:33

If the string you have contains one, then you should use the other. For example, "You're able to do this" , or 'He said "Hi!"' . Other than that, you should simply be as consistent as you can (within a module, within a package, within a project, within an organisation).

If your code is going to be read by people who work with C/C++ (or if you switch between those languages and Python), then using '' for single-character strings, and "" for longer strings might help ease the transition. (Likewise for following other languages where they are not interchangeable).

The Python code I've seen in the wild tends to favour " over ' , but only slightly. The one exception is that """these""" are much more common than '''these''' , from what I have seen.

jblocksom ,Jul 30, 2009 at 20:35

Triple quoted comments are an interesting subtopic of this question. PEP 257 specifies triple quotes for doc strings . I did a quick check using Google Code Search and found that triple double quotes in Python are about 10x as popular as triple single quotes -- 1.3M vs 131K occurrences in the code Google indexes. So in the multi line case your code is probably going to be more familiar to people if it uses triple double quotes.

Paolo ,Sep 30, 2013 at 15:39

"If you're going to use apostrophes, 
       ^

you'll definitely want to use double quotes".
   ^

For that simple reason, I always use double quotes on the outside. Always

Speaking of fluff, what good is streamlining your string literals with ' if you're going to have to use escape characters to represent apostrophes? Does it offend coders to read novels? I can't imagine how painful high school English class was for you!

dolma33 ,Mar 17, 2014 at 23:13

Python uses quotes something like this:
mystringliteral1="this is a string with 'quotes'"
mystringliteral2='this is a string with "quotes"'
mystringliteral3="""this is a string with "quotes" and more 'quotes'"""
mystringliteral4='''this is a string with 'quotes' and more "quotes"'''
mystringliteral5='this is a string with \"quotes\"'
mystringliteral6='this is a string with \042quotes\042'
mystringliteral6='this is a string with \047quotes\047'

print mystringliteral1
print mystringliteral2
print mystringliteral3
print mystringliteral4
print mystringliteral5
print mystringliteral6

Which gives the following output:

this is a string with 'quotes'
this is a string with "quotes"
this is a string with "quotes" and more 'quotes'
this is a string with 'quotes' and more "quotes"
this is a string with "quotes"
this is a string with 'quotes'

Matt Sheppard ,Sep 11, 2008 at 8:40

I use double quotes in general, but not for any specific reason - Probably just out of habit from Java.

I guess you're also more likely to want apostrophes in an inline literal string than you are to want double quotes.

schwa ,Sep 28, 2008 at 3:35

Personally I stick with one or the other. It doesn't matter. And providing your own meaning to either quote is just to confuse other people when you collaborate.

maxpolk ,Sep 21, 2013 at 17:45

It's probably a stylistic preference more than anything. I just checked PEP 8 and didn't see any mention of single versus double quotes.

I prefer single quotes because its only one keystroke instead of two. That is, I don't have to mash the shift key to make single quote.

stivlo ,Mar 11, 2012 at 4:25

In Perl you want to use single quotes when you have a string which doesn't need to interpolate variables or escaped characters like \n, \t, \r, etc.

PHP makes the same distinction as Perl: content in single quotes will not be interpreted (not even \n will be converted), as opposed to double quotes which can contain variables to have their value printed out.

Python does not, I'm afraid. Technically seen, there is no $ token (or the like) to separate a name/text from a variable in Python. Both features make Python more readable, less confusing, after all. Single and double quotes can be used interchangeably in Python.

Andrew Dalke ,Mar 10, 2009 at 6:25

I chose to use double quotes because they are easier to see.

Alphy

add a comment ,Jul 22, 2009 at 19:50
I just use whatever strikes my fancy at the time; it's convenient to be able to switch between the two at a whim!

Of course, when quoting quote characetrs, switching between the two might not be so whimsical after all...

Vinko Vrsalovic ,Sep 11, 2008 at 8:25

Your team's taste or your project's coding guidelines.

If you are in a multilanguage environment, you might wish to encourage the use of the same type of quotes for strings that the other language uses, for instance. Else, I personally like best the look of '

Mario F ,Sep 11, 2008 at 8:28

None as far as I know. Although if you look at some code, " " is commonly used for strings of text (I guess ' is more common inside text than "), and ' ' appears in hashkeys and things like that.

Acumenus ,Apr 16, 2013 at 22:32

I aim to minimize both pixels and surprise. I typically prefer ' in order to minimize pixels, but " instead if the string has an apostrophe, again to minimize pixels. For a docstring, however, I prefer """ over ''' because the latter is non-standard, uncommon, and therefore surprising. If now I have a bunch of strings where I used " per the above logic, but also one that can get away with a ' , I may still use " in it to preserve consistency, only to minimize surprise.

Perhaps it helps to think of the pixel minimization philosophy in the following way. Would you rather that English characters looked like A B C or AA BB CC ? The latter choice wastes 50% of the non-empty pixels.

Philipp ,Jul 5, 2010 at 13:06

I use double quotes because I have been doing so for years in most languages (C++, Java, VB ) except Bash, because I also use double quotes in normal text and because I'm using a (modified) non-English keyboard where both characters require the shift key.

Adam Smith ,Dec 16, 2013 at 23:38

' = "

/ = \ = \\

example :

f = open('c:\word.txt', 'r')
f = open("c:\word.txt", "r")
f = open("c:/word.txt", "r")
f = open("c:\\\word.txt", "r")

Results are the same

=>> no, they're not the same. A single backslash will escape characters. You just happen to luck out in that example because \k and \w aren't valid escapes like \t or \n or \\ or \"

If you want to use single backslashes (and have them interpreted as such), then you need to use a "raw" string. You can do this by putting an ' r ' in front of the string

im_raw = r'c:\temp.txt'
non_raw = 'c:\\temp.txt'
another_way = 'c:/temp.txt'

As far as paths in Windows are concerned, forward slashes are interpreted the same way. Clearly the string itself is different though. I wouldn't guarantee that they're handled this way on an external device though.

[Sep 28, 2019] The Power of Python's String Templates

Sep 28, 2019 | www.thoughtco.com

Science, Tech, Math › Computer Science Python's String Templates

Print
python icon done in the Tango! style

The people from the Tango! project/Wikimedia Commons

Computer Science Computer Science View More by Al Lukaszewski Albert Lukaszewski, Ph.D., is a veteran computer programmer, software engineer, and author, who specializes in the Python language. Updated December 31, 2018 Python is an interpreted, object-oriented, high-level programming language . It is easy to learn because its syntax emphasizes readability, which reduces the expense of program maintenance. Many programmers love working with Python because -- without the compilation step -- testing and debugging go quickly.​ Python Web Templating Templating, especially web templating, represents data in forms usually intended to be readable by a viewer. The simplest form of a templating engine substitutes values into the template to produce the output. Aside from the string constants and the deprecated string functions, which moved to string methods, Python's string module also includes string templates. The template itself is a class that receives a string as its argument. The object instantiated from that class is called a template string object. Template strings were first introduced in Python 2.4. Where string formatting operators used the percentage sign for substitutions, the template object uses dollar signs. Outside of these uses of the dollar sign, any appearance of $ causes a ValueError to be raised. The methods available through template strings are as follows: Template objects also have one publicly available attribute: The sample shell session below serves to illustrate template string objects.

>>> from string import Template

>>> s = Template('$when, $who $action $what.')

>>> s.substitute(when='In the summer', who='John', action='drinks', what='iced tea') 'In the summer, John drinks iced tea.'

>>> s.substitute(when='At night', who='Jean', action='eats', what='popcorn') 'At night, Jean eats popcorn.'

>>> s.template '$when, $who $action $what.'

>>> d = dict(when='in the summer')

>>> Template('$who $action $what $when').safe_substitute(d) '$who $action $what in the summer'

[Sep 28, 2019] python - Get command line arguments as string - Stack Overflow

Sep 28, 2019 | stackoverflow.com

Get command line arguments as string Ask Question Asked 3 years, 3 months ago Active 2 months ago Viewed 21k times 7 1


KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

> ,

Or can do format , and multiply the string by length of sys.argv :
(' {}'*len(sys.argv)).lstrip().format(*sys.argv)

Or can do % :

(' %s'*len(sys.argv)).lstrip()%sys.argv

[Sep 27, 2019] String Manipulation in Python

Sep 27, 2019 | www.pythonforbeginners.com

String Manipulation in Python Overview

A string is a list of characters in order. 

A character is anything you can type on the keyboard in one keystroke,
like a letter, a number, or a backslash. 

Strings can have spaces: "hello world". 

An empty string is a string that has 0 characters.

Python strings are immutable

Python recognize as strings everything that is delimited by quotation marks
(" " or ' ').
String Manipulation
To manipulate strings, we can use some of Pythons built-in methods.
Creation
word = "Hello World"

>>> print word
Hello World
Accessing

Use [ ] to access characters in a string
word = "Hello World"
letter=word[0]

>>> print letter
H
Length
word = "Hello World"

>>> len(word)
11
Finding
word = "Hello World"

>>> print word.count('l')      # count how many times l is in the string
3

>>> print word.find("H")     # find the word H in the string
0

>>> print word.index("World")        # find the letters World in the string
6
Count
s =  "Count, the number     of spaces"

>>> print s.count(' ')
8
Slicing
Use [ # : # ] to get set of letter

Keep in mind that python, as many other languages, starts to count from 0!!
word = "Hello World"

print word[0]          #get one char of the word
print word[0:1]        #get one char of the word (same as above)
print word[0:3]        #get the first three char
print word[:3]         #get the first three char
print word[-3:]        #get the last three char
print word[3:]         #get all but the three first char
print word[:-3]        #get all but the three last character
word = "Hello World"
 
word[start:end]         # items start through end-1
word[start:]            # items start through the rest of the list
word[:end]              # items from the beginning through end-1
word[:]                 # a copy of the whole list
Split Strings
word = "Hello World"

>>> word.split(' ')  # Split on whitespace
['Hello', 'World']
Startswith / Endswith
word = "hello world"

>>> word.startswith("H")
True

>>> word.endswith("d")
True

>>> word.endswith("w")
False
Repeat Strings
print "."* 10    # prints ten dots

>>> print "." * 10
..........
Replacing
word = "Hello World"

>>> word.replace("Hello", "Goodbye")
'Goodbye World'
Changing Upper and Lower Case Strings
string = "Hello World"

>>> print string.upper()
HELLO WORLD

>>> print string.lower()
hello world

>>> print string.title()
Hello World

>>> print string.capitalize()
Hello world

>>> print string.swapcase()
hELLO wORLD
Reversing
string = "Hello World"

>>> print ' '.join(reversed(string))
d l r o W   o l l e H
Strip

Python strings have the strip(), lstrip(), rstrip() methods for removing 
any character from both ends of a string. 

If the characters to be removed are not specified then white-space will be removed
word = "Hello World"
Strip off newline characters from end of the string
>>> print word.strip('
')
Hello World
strip()     #removes from both ends
lstrip()    #removes leading characters (Left-strip)
rstrip()    #removes trailing characters (Right-strip)

>>> word = "    xyz    "

>>> print word
    xyz    

>>> print word.strip()
xyz

>>> print word.lstrip()
xyz    

>>> print word.rstrip()
    xyz
Concatenation

To concatenate strings in Python use the "+" operator.
"Hello " + "World" # = "Hello World"
"Hello " + "World" + "!"# = "Hello World!"
Join
>>> print ":".join(word)  # #add a : between every char
H:e:l:l:o: :W:o:r:l:d

>>> print " ".join(word)  # add a whitespace between every char
H e l l o   W o r l d
Testing

A string in Python can be tested for truth value. 

The return type will be in Boolean value (True or False)
word = "Hello World"
 
word.isalnum()         #check if all char are alphanumeric 
word.isalpha()         #check if all char in the string are alphabetic
word.isdigit()         #test if string contains digits
word.istitle()         #test if string contains title words
word.isupper()         #test if string contains upper case
word.islower()         #test if string contains lower case
word.isspace()         #test if string contains spaces
word.endswith('d')     #test if string endswith a d
word.startswith('H')   #test if string startswith H
If you liked this article, please share it with your friends.

[Sep 27, 2019] 6.1. string -- Common string operations -- Python 3.4.10 documentation

Sep 27, 2019 | docs.python.org

Source code: Lib/string.py


See also

Text Sequence Type -- str

String Methods 6.1.1. String constants

The constants defined in this module are:

string. ascii_letters
The concatenation of the ascii_lowercase and ascii_uppercase constants described below. This value is not locale-dependent.
string. ascii_lowercase
The lowercase letters 'abcdefghijklmnopqrstuvwxyz' . This value is not locale-dependent and will not change.
string. ascii_uppercase
The uppercase letters 'ABCDEFGHIJKLMNOPQRSTUVWXYZ' . This value is not locale-dependent and will not change.
string. digits
The string '0123456789' .
string. hexdigits
The string '0123456789abcdefABCDEF' .
string. octdigits
The string '01234567' .
string. punctuation
String of ASCII characters which are considered punctuation characters in the C locale.
string. printable
String of ASCII characters which are considered printable. This is a combination of digits , ascii_letters , punctuation , and whitespace .
string. whitespace
A string containing all ASCII characters that are considered whitespace. This includes the characters space, tab, linefeed, return, formfeed, and vertical tab.
6.1.2. String Formatting

The built-in string class provides the ability to do complex variable substitutions and value formatting via the format() method described in PEP 3101 . The Formatter class in the string module allows you to create and customize your own string formatting behaviors using the same implementation as the built-in format() method.

class string. Formatter
The Formatter class has the following public methods:
format ( format_string , *args , **kwargs )
format() is the primary API method. It takes a format string and an arbitrary set of positional and keyword arguments. format() is just a wrapper that calls vformat() .
vformat ( format_string , args , kwargs )
This function does the actual work of formatting. It is exposed as a separate function for cases where you want to pass in a predefined dictionary of arguments, rather than unpacking and repacking the dictionary as individual arguments using the *args and **kwargs syntax. vformat() does the work of breaking up the format string into character data and replacement fields. It calls the various methods described below.

In addition, the Formatter defines a number of methods that are intended to be replaced by subclasses:

parse ( format_string )
Loop over the format_string and return an iterable of tuples ( literal_text , field_name , format_spec , conversion ). This is used by vformat() to break the string into either literal text, or replacement fields.

The values in the tuple conceptually represent a span of literal text followed by a single replacement field. If there is no literal text (which can happen if two replacement fields occur consecutively), then literal_text will be a zero-length string. If there is no replacement field, then the values of field_name , format_spec and conversion will be None .

get_field ( field_name , args , kwargs )
Given field_name as returned by parse() (see above), convert it to an object to be formatted. Returns a tuple (obj, used_key). The default version takes strings of the form defined in PEP 3101 , such as "0[name]" or "label.title". args and kwargs are as passed in to vformat() . The return value used_key has the same meaning as the key parameter to get_value() .
get_value ( key , args , kwargs )
Retrieve a given field value. The key argument will be either an integer or a string. If it is an integer, it represents the index of the positional argument in args ; if it is a string, then it represents a named argument in kwargs .

The args parameter is set to the list of positional arguments to vformat() , and the kwargs parameter is set to the dictionary of keyword arguments.

For compound field names, these functions are only called for the first component of the field name; Subsequent components are handled through normal attribute and indexing operations.

So for example, the field expression '0.name' would cause get_value() to be called with a key argument of 0. The name attribute will be looked up after get_value() returns by calling the built-in getattr() function.

If the index or keyword refers to an item that does not exist, then an IndexError or KeyError should be raised.

check_unused_args ( used_args , args , kwargs )
Implement checking for unused arguments if desired. The arguments to this function is the set of all argument keys that were actually referred to in the format string (integers for positional arguments, and strings for named arguments), and a reference to the args and kwargs that was passed to vformat. The set of unused args can be calculated from these parameters. check_unused_args() is assumed to raise an exception if the check fails.
format_field ( value , format_spec )
format_field() simply calls the global format() built-in. The method is provided so that subclasses can override it.
convert_field ( value , conversion )
Converts the value (returned by get_field() ) given a conversion type (as in the tuple returned by the parse() method). The default version understands 's' (str), 'r' (repr) and 'a' (ascii) conversion types.
6.1.3. Format String Syntax

The str.format() method and the Formatter class share the same syntax for format strings (although in the case of Formatter , subclasses can define their own format string syntax).

Format strings contain "replacement fields" surrounded by curly braces {} . Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: {{ and }} .

The grammar for a replacement field is as follows:

replacement_field ::=  "{" [field_name] ["!" conversion] [":" format_spec] "}"
field_name        ::=  arg_name ("." attribute_name | "[" element_index "]")*
arg_name          ::=  [identifier | integer]
attribute_name    ::=  identifier
element_index     ::=  integer | index_string
index_string      ::=  <any source character except "]"> +
conversion        ::=  "r" | "s" | "a"
format_spec       ::=  <described in the next section>

In less formal terms, the replacement field can start with a field_name that specifies the object whose value is to be formatted and inserted into the output instead of the replacement field. The field_name is optionally followed by a conversion field, which is preceded by an exclamation point '!' , and a format_spec , which is preceded by a colon ':' . These specify a non-default format for the replacement value.

See also the Format Specification Mini-Language section.

The field_name itself begins with an arg_name that is either a number or a keyword. If it's a number, it refers to a positional argument, and if it's a keyword, it refers to a named keyword argument. If the numerical arg_names in a format string are 0, 1, 2, ... in sequence, they can all be omitted (not just some) and the numbers 0, 1, 2, ... will be automatically inserted in that order. Because arg_name is not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings '10' or