Softpanorama
May the source be with you, but remember the KISS principle ;-)

Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Slightly Skeptical View on Enterprise Unix Administration

News Unix/Linux Internals Recommended Books Recommended Links Programmers and sysadmins health issues IT Outsourcing/Offshoring Skeptic: Fighting Outsourcing Myths
Obscurantism in the Cloud: Nicholas Carr's "IT Does not Matter" Fallacy and "Everything in the Cloud" Utopia Unix System Monitoring Job schedulers Admin Horror Stories iDRAC7 goes unresponsive - can't connect to iDRAC7 Using HP ILO virtual CDROM
System Activity Reporter (sar) HP Operations Manager Connecting to HP iLO via SSH Nagios in Large Enterprise Environment ALOM Working with serial console
Advanced Linux Administration Red Hat Enterprise Linux Administration SUSE Linux Enterprise Server (SLES) Solaris AIX Administration HP-UX Administration
Networking TCP Performance Tuning Redhat Network Configuration Classic Network Utilities Network Troubleshooting Tools TCP/IP Network Troubleshooting
Syslog Configuration and syslog.conf Linux Multipath The Linux Logical Volume Manager (LVM) System Activity Reporter (sar) Authentication Unix config management
Helpdesk software Software Distribution Assets management Project Management Unix to Unix migration Working with Console
Backup Baseliners Classic Unix Tools Perl admintools Software Distribution Performance Tuning
Unix filesystem navigation Bash history and bang commands profile and RC-files Orthodox File Managers SSH for System Administrators Network File System (NFS)
SAP/R3 implementation disasters          
Access Control in Operating Systems Sudoer File Examples  Enterprise System Management Resources Logs Analysis SMTP Mail Social Problem in Enterprise Unix Administration
The Unix Hater’s Handbook IBM Humor Tips History Humor Etc
The KISS rule can be expanded as: Keep It Simple, Sysadmin ;-)

Additional useful material on the topic can also be found in an older article Solaris vs Linux:

Abstract

Introduction

Nine factors framework for comparison of two flavors of Unix in a large enterprise environment

Four major areas of Linux and Solaris deployment

Comparison of internal architecture and key subsystems

Security

Hardware: SPARC vs. X86

Development environment

Solaris as a cultural phenomenon

Using Solaris-Linux enterprise mix as the least toxic Unix mix available

Conclusions

Acknowledgements

Webliography

Here are my notes/reflection of sysadmin problem in strange (and typically pretty toxic) IT departments of large corporations:


Top updates

Softpanorama Switchboard
Softpanorama Search


NEWS CONTENTS

Old News ;-)

2016 2015 2014 2013 2012 2011 2010 2009 2008
2007 2006 2005 2004 2003 2002 2001 2000 1999

“I appreciate Woody Allen’s humor because one of my safety valves is an appreciation for life’s absurdities. His message is that life isn’t a funeral march to the grave. It’s a polka.”

-- Dennis Kusinich

[Feb 15, 2017] Web proxy, NAS and email server installed as appliance

Feb 15, 2017 | www.cyberciti.biz
Operating system : Linux

Purpose : Turn normal server into appliances

Download url : artica.fr

Artica Tech offers a powerful but easy-to-use Enterprise-Class Web Security and Control solution,

usually the preserve of large companies. Prices starting at 99€ / year for 5 users.

[Feb 15, 2017] Synkron – Folder synchronisation

synkron.sourceforge.net

Folder synchronisation

Synkron is an application that helps you keep your files and folders always updated. You can easily sync your documents, music or pictures to have their latest versions everywhere.

Synkron provides an easy-to-use interface and a lot of features. Moreover, it is free and cross-platform.

Features

[Feb 15, 2017] grep like tool

Feb 15, 2017 | www.cyberciti.biz
, optimized for programmers. This tool isn't aimed to "search all text files". It is specifically created to search source code trees, not trees of text files. It searches entire trees by default while ignoring Subversion, Git and other VCS directories and other files that aren't your source code.

Operating system : Cross-platform
Purpose : Search source trees
Download url : beyondgrep.com

[Feb 15, 2017] 15 Greatest Open Source Terminal Applications Of 2012

Dec 11, 2012 | www.cyberciti.biz
Last updated January 7, 2013 in Command Line Hacks , Open Source , Web Developer

Linux on the desktop is making great progress. However, the real beauty of Linux and Unix like operating system lies beneath the surface at the command prompt. nixCraft picks his best open source terminal applications of 2012.

Most of the following tools are packaged by all major Linux distributions and can be installed on *BSD or Apple OS X. #3: ngrep – Network grep

Fig.02: ngrep in action
Ngrep is a network packet analyzer. It follows most of GNU grep's common features, applying them to the network layer. Ngrep is not related to tcpdump. It is just an easy to use tool. You can run queries such as:

## grep all HTTP GET or POST requests from network traffic on eth0 interface  ##
sudo
 ngrep 
-l
-q
-d
 eth0 
"^GET |^POST "
 tcp and port 
80

## grep all HTTP GET or POST requests from network traffic on eth0 interface ## sudo ngrep -l -q -d eth0 "^GET |^POST " tcp and port 80

I often use this tool to find out security related problems and tracking down other network and server related problems.

... ... ...

#5: dtrx

dtrx is an acronmy for "Do The Right Extraction." It's a tool for Unix-like systems that take all the hassle out of extracting archives. As a sysadmin, I download source code and tar balls. This tool saves lots of time.

#6:dstat – Versatile resource statistics tool

Fig.05: dstat in action
As a sysadmin, I heavily depends upon tools such as vmstat, iostat and friends for troubleshooting server issues. Dstat overcomes some of the limitations provided by vmstat and friends. It adds some extra features. It allows me to view all of my system resources instantly. I can compare disk usage in combination with interrupts from hard disk controller, or compare the network bandwidth numbers directly with the disk throughput and much more.

... ... ..

#8:mtr – Traceroute+ping in a single network diagnostic tool

Fig.07: mtr in action
The mtr command combines the functionality of the traceroute and ping programs in a single network diagnostic tool. Use mtr to monitor outgoing bandwidth, latency and jitter in your network. A great little app to solve network problems. If you see a sudden increase in packetloss or response time is often an indication of a bad or simply overloaded link.

#9:multitail – Tail command on steroids

Fig.08: multitail in action (image credit – official project)
MultiTail is a program for monitoring multiple log files, in the fashion of the original tail program. This program lets you view one or multiple files like the original tail program. The difference is that it creates multiple windows on your console (with ncurses). I often use this tool when I am monitoring logs on my server.

... ... ...

#11: netcat – TCP/IP swiss army knife

Fig.10: nc server and telnet client in action
Netcat or nc is a simple Linux or Unix command which reads and writes data across network connections, using TCP or UDP protocol. I often use this tool to open up a network pipe to test network connectivity, make backups, bind to sockets to handle incoming / outgoing requests and much more. In this example, I tell nc to listen to a port # 3005 and execute /usr/bin/w command when client connects and send data back to the client:
$ nc -l -p 3005 -e /usr/bin/w
From a different system try to connect to port # 3005:
$ telnet server1.cyberciti.biz.lan 3005

... ... ...

#14: lftp: A better command-line ftp/http/sftp client

This is the best and most sophisticated sftp/ftp/http download and upload client program. I often use this tool to:

  1. Recursively mirroring entire directory trees from a ftp server
  2. Accelerate ftp / http download speed
  3. Location bookmarks and resuming downloads.
  4. Backup files to a remote ftp servers.
  5. Transfers can be scheduled for execution at a later time.
  6. Bandwidth can be throttled and transfer queues can be set up.
  7. Lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&).
  8. Segmented file transfer, that allows more than one connection for the same file.
  9. And much more.
  10. Download lftp

... ... ...

#16: Rest Conclusion

This is my personal FOSS terminal apps list and it is not absolutely definitive, so if you've got your own terminal apps, share in the comments below.

vidir – edit directories (part of the 'moreutils' package)

[Feb 14, 2017] switching from gnu screen to tmux (updated) Linux~ized

Ability to watch the other user screen is a very valuable option...
Feb 14, 2017 | www.linuxized.com

ed says: June 16, 2010 at 15:15

screen is really cool, and does somethings that I've yet to find counterparts to with tmux, such as the -x option:

http://www.s5h.net/wiki/Screen#Automation

[Feb 14, 2017] Ms Dos style aliases for linux

I think alias ipconfig = 'ifconfig' is really useful for people who work with Linus from Windows POc desktop/laptop.
Feb 14, 2017 | bash.cyberciti.biz
# MS-DOS / XP cmd like stuff  
   alias edit = $VISUAL  
   alias copy = 'cp'  
   alias cls = 'clear'  
   alias del = 'rm'  
   alias dir = 'ls'  
   alias md = 'mkdir'  
   alias move = 'mv'  
   alias rd = 'rmdir'  
   alias ren = 'mv'  
   alias ipconfig = 'ifconfig'

[Feb 14, 2017] 30 Cool Open Source Software I Discovered in 2013

Feb 14, 2017 | www.cyberciti.biz
#16 SimpleInvoices

It is a web based invoicing system. It helps me to create quick and nice looking invoices without having to set up too much services on server. All you have to do is install the SimpleInvoices software, enter a biller, a customer details and go creating invoices. You can easily track your finances; send invoices as PDF's and more. It is the best invoicing set up for my independent IT consultancy business. #19 XAMPP – Easily write and test Apache+MySQL+PHP/Perl apps on desktop

I give this software to many developers. They can easily setup Apache, MySQL, PHP/Perl to deploy and write an application on their own desktop. No need to install virtual machine and Linux server. Just focus on development and skip real server management job to pros.

[Feb 14, 2017] Three useful aliases for du command

Feb 14, 2017 | www.cyberciti.biz

Rishi G June 12, 2012, 4:01 am

Here are 4 commands i use for checking out disk usages.
#Grabs the disk usage in the current directory
alias usage='du -ch | grep total'

#Gets the total disk usage on your machine
alias totalusage='df -hl --total | grep total'

#Shows the individual partition usages without the temporary memory values
alias partusage='df -hlT --exclude-type=tmpfs --exclude-type=devtmpfs'

#Gives you what is using the most space. Both directories and files. Varies on
#current directory
alias most='du -hsx * | sort -rh | head -10'

[Feb 14, 2017] 15 Greatest Open Source Terminal Applications Of 2012

Feb 14, 2017 | www.cyberciti.biz
on December 11, 2012 last updated January 7, 2013 in Command Line Hacks , Open Source , Web Developer L inux on the desktop is making great progress. However, the real beauty of Linux and Unix like operating system lies beneath the surface at the command prompt. nixCraft picks his best open source terminal applications of 2012.

Most of the following tools are packaged by all major Linux distributions and can be installed on *BSD or Apple OS X. #3: ngrep – Network grep

Fig.02: ngrep in action

Fig.02: ngrep in action
Ngrep is a network packet analyzer. It follows most of GNU grep's common features, applying them to the network layer. Ngrep is not related to tcpdump. It is just an easy to use tool. You can run queries such as:

## grep all HTTP GET or POST requests from network traffic on eth0 interface  ##
sudo
 ngrep 
-l
-q
-d
 eth0 
"^GET |^POST "
 tcp and port 
80

## grep all HTTP GET or POST requests from network traffic on eth0 interface ## sudo ngrep -l -q -d eth0 "^GET |^POST " tcp and port 80

I often use this tool to find out security related problems and tracking down other network and server related problems.

... ... ...

#5: dtrx

Fig.04: dtrx in action

Fig.04: dtrx in action
dtrx is an acronmy for "Do The Right Extraction." It's a tool for Unix-like systems that take all the hassle out of extracting archives. As a sysadmin, I download source code and tar balls. This tool saves lots of time.

#6:dstat – Versatile resource statistics tool

Fig.05: dstat in action

Fig.05: dstat in action
As a sysadmin, I heavily depends upon tools such as vmstat, iostat and friends for troubleshooting server issues. Dstat overcomes some of the limitations provided by vmstat and friends. It adds some extra features. It allows me to view all of my system resources instantly. I can compare disk usage in combination with interrupts from hard disk controller, or compare the network bandwidth numbers directly with the disk throughput and much more.

... ... ..

#8:mtr – Traceroute+ping in a single network diagnostic tool

Fig.07: mtr in action

Fig.07: mtr in action
The mtr command combines the functionality of the traceroute and ping programs in a single network diagnostic tool. Use mtr to monitor outgoing bandwidth, latency and jitter in your network. A great little app to solve network problems. If you see a sudden increase in packetloss or response time is often an indication of a bad or simply overloaded link.

#9:multitail – Tail command on steroids

Fig.08: multitail in action (image credit - official project)

Fig.08: multitail in action (image credit – official project)
MultiTail is a program for monitoring multiple log files, in the fashion of the original tail program. This program lets you view one or multiple files like the original tail program. The difference is that it creates multiple windows on your console (with ncurses). I often use this tool when I am monitoring logs on my server.

... ... ...

#11: netcat – TCP/IP swiss army knife

Fig.10: nc server and telnet client in action

Fig.10: nc server and telnet client in action
Netcat or nc is a simple Linux or Unix command which reads and writes data across network connections, using TCP or UDP protocol. I often use this tool to open up a network pipe to test network connectivity, make backups, bind to sockets to handle incoming / outgoing requests and much more. In this example, I tell nc to listen to a port # 3005 and execute /usr/bin/w command when client connects and send data back to the client:
$ nc -l -p 3005 -e /usr/bin/w
From a different system try to connect to port # 3005:
$ telnet server1.cyberciti.biz.lan 3005

... ... ...

#14: lftp: A better command-line ftp/http/sftp client

This is the best and most sophisticated sftp/ftp/http download and upload client program. I often use this tool to:

  1. Recursively mirroring entire directory trees from a ftp server
  2. Accelerate ftp / http download speed
  3. Location bookmarks and resuming downloads.
  4. Backup files to a remote ftp servers.
  5. Transfers can be scheduled for execution at a later time.
  6. Bandwidth can be throttled and transfer queues can be set up.
  7. Lftp has shell-like command syntax allowing you to launch several commands in parallel in background (&).
  8. Segmented file transfer, that allows more than one connection for the same file.
  9. And much more.
  10. Download lftp

... ... ...

#16: Rest Conclusion

This is my personal FOSS terminal apps list and it is not absolutely definitive, so if you've got your own terminal apps, share in the comments below.

[Feb 14, 2017] Any alias of rm is a very stupid idea

Feb 14, 2017 | www.cyberciti.biz
Art Protin June 12, 2012, 9:53 pm

Any alias of rm is a very stupid idea (except maybe alias rm=echo fool).

A co-worker had such an alias. Imagine the disaster when, visiting a customer site, he did "rm *" in the customer's work directory and all he got was the prompt for the next command after rm had done what it was told to do.

It you want a safety net, do "alias del='rm -I –preserve_root'",

Drew Hammond March 26, 2014, 7:41 pm
^ This x10000.

I've made the same mistake before and its horrible.

[Feb 14, 2017] My 10 UNIX Command Line Mistakes

Feb 14, 2017 | www.cyberciti.biz
Destroyed named.conf

I wanted to append a new zone to /var/named/chroot/etc/named.conf file., but end up running:
./mkzone example.com > /var/named/chroot/etc/named.conf

Destroyed Working Backups with Tar and Rsync (personal backups)

I had only one backup copy of my QT project and I just wanted to get a directory called functions. I end up deleting entire backup (note -c switch instead of -x):
cd /mnt/bacupusbharddisk
tar -zcvf project.tar.gz functions

I had no backup. Similarly I end up running rsync command and deleted all new files by overwriting files from backup set (now I’ve switched to rsnapshot )
rsync -av -delete /dest /src
Again, I had no backup.

[Feb 12, 2017] Voice Activation Module instllaed on corporate printers

Feb 12, 2017 | thwack.solarwinds.com

rcmartin28 Jul 16, 2015 11:16 AM

Not a horror story, more of an I.T. payback. A few years ago I created several incredibly official looking I.T. signs for the various company printers. The signs informed everyone that the new "Voice Activation Module" had finally been installed and was fully functional. April Fools!! LOL!!

The audio/video captures, and trouble tickets from that day were priceless. LOL! My old boss and I still talk about some of them to this day.

[Feb 12, 2017] Linux Server Hacks, Volume Two

Notable quotes:
"... multixterm ..."
"... xterms ..."
"... multixterm ..."
"... host1 host2 ..."
Feb 12, 2017 | www.amazon.com
Execute Commands Simultaneously on Multiple Servers

Run the same command at the same time on multiple systems, simplifying administrative tasks and reducing synchronization problems .

If you have multiple servers with similar or identical configurations (such as nodes in a cluster), it's often difficult to make sure the contents and configuration of those servers are identical. It's even more difficult when you need to make configuration modifications from the command line, knowing you'll have to execute the exact same command on a large number of systems (better get coffee first). You could try writing a script to perform the task automatically, but sometimes scripting is overkill for the work to be done. Fortunately, there's another way to execute commands on multiple hosts simultaneously.

A great solution for this problem is an excellent tool called multixterm , which enables you to simultaneously open xterms to any number of systems, type your commands in a single central window and have the commands executed in each of the xterm windows you've started. Sound appealing? Type once, execute many-it sounds like a new pipelining instruction set.

multixterm is available from http://expect.nist.gov/example/multixterm.man.html , and it requires expect and tk . The most common way to run multixterm is with a command like the following:

	$ 
multixterm -xc "ssh %n"


host1 host2


This command will open ssh connections to host1 and host2 ( Figure 4-1 ). Anything typed in the area labeled "stdin window" (which is usually gray or green, depending on your color scheme) will be sent to both windows, as shown in the figure.

As you can see from the sample command, the –xc option stands for execute command, and it must be followed by the command that you want to execute on each host, enclosed in double quotation marks. If the specified command includes a wildcard such as %n , each hostname that follows the command will be substituted into the command in turn when it is executed. Thus, in our example, the commands ssh host1 and ssh host2 were both executed by multixterm , each within its own xterm window.

See Also

[Feb 12, 2017] Easy Automated Snapshot-Style Backups with Linux and Rsync

Notable quotes:
"... illusion ..."
"... only one extra, slightly-larger, hard disk ..."
"... hard link ..."
"... what appears to be ..."
"... Putting it all together ..."
"... If you are rsync'ing from a SAMBA share, you must add --modify-window=10 ..."
Feb 12, 2017 | www.mikerubel.org

page last modified 2004.01.04

Updates: As of rsync-2.5.6 , the --link-dest option is now standard! That can be used instead of the separate cp -al and rsync stages, and it eliminates the ownerships/permissions bug. I now recommend using it. Also, I'm proud to report this article is mentioned in Linux Server Hacks , a new (and very good, in my opinion) O'Reilly book by compiled by Rob Flickenger.

Contents
  1. Abstract
  2. Motivation
  3. Using rsync to make a backup
    1. Basics
    2. Using the --delete flag
    3. Be lazy: use cron
  4. Incremental backups with rsync
    1. Review of hard links
    2. Using cp -al
    3. Putting it all together
    4. I'm used to dump or tar ! This seems backward!
  5. Isolating the backup from the rest of the system
    1. The easy (bad) way
    2. Keep it on a separate partition
    3. Keep that partition on a separate disk
    4. Keep that disk on a separate machine
  6. Making the backup as read-only as possible
    1. Bad: mount / unmount
    2. Better: mount read-only most of the time
    3. Tempting but it doesn't seem to work: the 2.4 kernel's mount --bind
    4. My solution: using NFS on localhost
  7. Extensions: hourly, daily, and weekly snapshots
    1. Keep an extra script for each level
    2. Run it all with cron
  8. Known bugs and problems
    1. Maintaining Permissions and Owners in the snapshots
    2. mv updates timestamp bug
    3. Windows-related problems
  9. Appendix: my actual configuration
    1. Listing one: make_snapshot.sh
    2. Listing two: daily_snapshot_rotate.sh
    3. Sample output of ls -l /snapshot/home
  10. Contributed codes
  11. References
  12. Frequently Asked Questions
Abstract

This document describes a method for generating automatic rotating "snapshot"-style backups on a Unix-based system, with specific examples drawn from the author's GNU/Linux experience. Snapshot backups are a feature of some high-end industrial file servers; they create the illusion of multiple, full backups per day without the space or processing overhead. All of the snapshots are read-only, and are accessible directly by users as special system directories. It is often possible to store several hours, days, and even weeks' worth of snapshots with slightly more than 2x storage. This method, while not as space-efficient as some of the proprietary technologies (which, using special copy-on-write filesystems, can operate on slightly more than 1x storage), makes use of only standard file utilities and the common rsync program, which is installed by default on most Linux distributions. Properly configured, the method can also protect against hard disk failure, root compromises, or even back up a network of heterogeneous desktops automatically.

Motivation

Note: what follows is the original sgvlug DEVSIG announcement.

Ever accidentally delete or overwrite a file you were working on? Ever lose data due to hard-disk failure? Or maybe you export shares to your windows-using friends--who proceed to get outlook viruses that twiddle a digit or two in all of their .xls files. Wouldn't it be nice if there were a /snapshot directory that you could go back to, which had complete images of the file system at semi-hourly intervals all day, then daily snapshots back a few days, and maybe a weekly snapshot too? What if every user could just go into that magical directory and copy deleted or overwritten files back into "reality", from the snapshot of choice, without any help from you? And what if that /snapshot directory were read-only, like a CD-ROM, so that nothing could touch it (except maybe root, but even then not directly)?

Best of all, what if you could make all of that happen automatically, using only one extra, slightly-larger, hard disk ? (Or one extra partition, which would protect against all of the above except disk failure).

In my lab, we have a proprietary NetApp file server which provides that sort of functionality to the end-users. It provides a lot of other things too, but it cost as much as a luxury SUV. It's quite appropriate for our heavy-use research lab, but it would be overkill for a home or small-office environment. But that doesn't mean small-time users have to do without!

I'll show you how I configured automatic, rotating snapshots on my $80 used Linux desktop machine (which is also a file, web, and mail server) using only a couple of one-page scripts and a few standard Linux utilities that you probably already have.

I'll also propose a related strategy which employs one (or two, for the wisely paranoid) extra low-end machines for a complete, responsible, automated backup strategy that eliminates tapes and manual labor and makes restoring files as easy as "cp".

Using rsync to make a backup

The rsync utility is a very well-known piece of GPL'd software, written originally by Andrew Tridgell and Paul Mackerras. If you have a common Linux or UNIX variant, then you probably already have it installed; if not, you can download the source code from rsync.samba.org . Rsync's specialty is efficiently synchronizing file trees across a network, but it works fine on a single machine too.

Basics

Suppose you have a directory called source , and you want to back it up into the directory destination . To accomplish that, you'd use:

rsync -a source/ destination/

(Note: I usually also add the -v (verbose) flag too so that rsync tells me what it's doing). This command is equivalent to:

cp -a source/. destination/

except that it's much more efficient if there are only a few differences.

Just to whet your appetite, here's a way to do the same thing as in the example above, but with destination on a remote machine, over a secure shell:

rsync -a -e ssh source/ username@remotemachine.com:/path/to/destination/
Trailing Slashes Do Matter...Sometimes

This isn't really an article about rsync , but I would like to take a momentary detour to clarify one potentially confusing detail about its use. You may be accustomed to commands that don't care about trailing slashes. For example, if a and b are two directories, then cp -a a b is equivalent to cp -a a/ b/ . However, rsync does care about the trailing slash, but only on the source argument. For example, let a and b be two directories, with the file foo initially inside directory a . Then this command:

rsync -a a b

produces b/a/foo , whereas this command:

rsync -a a/ b

produces b/foo . The presence or absence of a trailing slash on the destination argument ( b , in this case) has no effect.

Using the --delete flag

If a file was originally in both source/ and destination/ (from an earlier rsync , for example), and you delete it from source/ , you probably want it to be deleted from destination/ on the next rsync . However, the default behavior is to leave the copy at destination/ in place. Assuming you want rsync to delete any file from destination/ that is not in source/ , you'll need to use the --delete flag:

rsync -a --delete source/ destination/
Be lazy: use cron

One of the toughest obstacles to a good backup strategy is human nature; if there's any work involved, there's a good chance backups won't happen. (Witness, for example, how rarely my roommate's home PC was backed up before I created this system). Fortunately, there's a way to harness human laziness: make cron do the work.

To run the rsync-with-backup command from the previous section every morning at 4:20 AM, for example, edit the root cron table: (as root)

crontab -e

Then add the following line:

20 4 * * * rsync -a --delete source/ destination/

Finally, save the file and exit. The backup will happen every morning at precisely 4:20 AM, and root will receive the output by email. Don't copy that example verbatim, though; you should use full path names (such as /usr/bin/rsync and /home/source/ ) to remove any ambiguity.

Incremental backups with rsync

Since making a full copy of a large filesystem can be a time-consuming and expensive process, it is common to make full backups only once a week or once a month, and store only changes on the other days. These are called "incremental" backups, and are supported by the venerable old dump and tar utilities, along with many others.

However, you don't have to use tape as your backup medium; it is both possible and vastly more efficient to perform incremental backups with rsync .

The most common way to do this is by using the rsync -b --backup-dir= combination. I have seen examples of that usage here , but I won't discuss it further, because there is a better way. If you're not familiar with hard links, though, you should first start with the following review.

Review of hard links

We usually think of a file's name as being the file itself, but really the name is a hard link . A given file can have more than one hard link to itself--for example, a directory has at least two hard links: the directory name and . (for when you're inside it). It also has one hard link from each of its sub-directories (the .. file inside each one). If you have the stat utility installed on your machine, you can find out how many hard links a file has (along with a bunch of other information) with the command:

stat filename

Hard links aren't just for directories--you can create more than one link to a regular file too. For example, if you have the file a , you can make a link called b :

ln a b

Now, a and b are two names for the same file, as you can verify by seeing that they reside at the same inode (the inode number will be different on your machine):

ls -i a
  232177 a
ls -i b
  232177 b

So ln a b is roughly equivalent to cp a b , but there are several important differences:

  1. The contents of the file are only stored once, so you don't use twice the space.
  2. If you change a , you're changing b , and vice-versa.
  3. If you change the permissions or ownership of a , you're changing those of b as well, and vice-versa.
  4. If you overwrite a by copying a third file on top of it, you will also overwrite b , unless you tell cp to unlink before overwriting. You do this by running cp with the --remove-destination flag. Notice that rsync always unlinks before overwriting!! . Note, added 2002.Apr.10: the previous statement applies to changes in the file contents only, not permissions or ownership.

But this raises an interesting question. What happens if you rm one of the links? The answer is that rm is a bit of a misnomer; it doesn't really remove a file, it just removes that one link to it. A file's contents aren't truly removed until the number of links to it reaches zero. In a moment, we're going to make use of that fact, but first, here's a word about cp .

Using cp -al

In the previous section, it was mentioned that hard-linking a file is similar to copying it. It should come as no surprise, then, that the standard GNU coreutils cp command comes with a -l flag that causes it to create (hard) links instead of copies (it doesn't hard-link directories, though, which is good; you might want to think about why that is). Another handy switch for the cp command is -a (archive), which causes it to recurse through directories and preserve file owners, timestamps, and access permissions.

Together, the combination cp -al makes what appears to be a full copy of a directory tree, but is really just an illusion that takes almost no space. If we restrict operations on the copy to adding or removing (unlinking) files--i.e., never changing one in place--then the illusion of a full copy is complete. To the end-user, the only differences are that the illusion-copy takes almost no disk space and almost no time to generate.

2002.05.15: Portability tip: If you don't have GNU cp installed (if you're using a different flavor of *nix, for example), you can use find and cpio instead. Simply replace cp -al a b with cd a && find . -print | cpio -dpl ../b . Thanks to Brage Førland for that tip.

Putting it all together

We can combine rsync and cp -al to create what appear to be multiple full backups of a filesystem without taking multiple disks' worth of space. Here's how, in a nutshell:

rm -rf backup.3
mv backup.2 backup.3
mv backup.1 backup.2
cp -al backup.0 backup.1
rsync -a --delete source_directory/  backup.0/

If the above commands are run once every day, then backup.0 , backup.1 , backup.2 , and backup.3 will appear to each be a full backup of source_directory/ as it appeared today, yesterday, two days ago, and three days ago, respectively--complete, except that permissions and ownerships in old snapshots will get their most recent values (thanks to J.W. Schultz for pointing this out). In reality, the extra storage will be equal to the current size of source_directory/ plus the total size of the changes over the last three days--exactly the same space that a full plus daily incremental backup with dump or tar would have taken.

Update (2003.04.23): As of rsync-2.5.6 , the --link-dest flag is now standard. Instead of the separate cp -al and rsync lines above, you may now write:

mv backup.0 backup.1
rsync -a --delete --link-dest=../backup.1 source_directory/  backup.0/

This method is preferred, since it preserves original permissions and ownerships in the backup. However, be sure to test it--as of this writing some users are still having trouble getting --link-dest to work properly. Make sure you use version 2.5.7 or later.

Update (2003.05.02): John Pelan writes in to suggest recycling the oldest snapshot instead of recursively removing and then re-creating it. This should make the process go faster, especially if your file tree is very large:

mv backup.3 backup.tmp
mv backup.2 backup.3
mv backup.1 backup.2
mv backup.0 backup.1
mv backup.tmp backup.0
cp -al backup.1/. backup.0
rsync -a --delete source_directory/ backup.0/

2003.06.02: OOPS! Rsync's link-dest option does not play well with J. Pelan's suggestion--the approach I previously had written above will result in unnecessarily large storage, because old files in backup.0 will get replaced and not linked. Please only use Dr. Pelan's directory recycling if you use the separate cp -al step; if you plan to use --link-dest , start with backup.0 empty and pristine. Apologies to anyone I've misled on this issue. Thanks to Kevin Everets for pointing out the discrepancy to me, and to J.W. Schultz for clarifying --link-dest 's behavior. Also note that I haven't fully tested the approach written above; if you have, please let me know. Until then, caveat emptor!

I'm used to dump or tar ! This seems backward!

The dump and tar utilities were originally designed to write to tape media, which can only access files in a certain order. If you're used to their style of incremental backup, rsync might seem backward. I hope that the following example will help make the differences clearer.

Suppose that on a particular system, backups were done on Monday night, Tuesday night, and Wednesday night, and now it's Thursday.

With dump or tar , the Monday backup is the big ("full") one. It contains everything in the filesystem being backed up. The Tuesday and Wednesday "incremental" backups would be much smaller, since they would contain only changes since the previous day. At some point (presumably next Monday), the administrator would plan to make another full dump.

With rsync, in contrast, the Wednesday backup is the big one. Indeed, the "full" backup is always the most recent one. The Tuesday directory would contain data only for those files that changed between Tuesday and Wednesday; the Monday directory would contain data for only those files that changed between Monday and Tuesday.

A little reasoning should convince you that the rsync way is much better for network-based backups, since it's only necessary to do a full backup once, instead of once per week. Thereafter, only the changes need to be copied. Unfortunately, you can't rsync to a tape, and that's probably why the dump and tar incremental backup models are still so popular. But in your author's opinion, these should never be used for network-based backups now that rsync is available.

Isolating the backup from the rest of the system

If you take the simple route and keep your backups in another directory on the same filesystem, then there's a very good chance that whatever damaged your data will also damage your backups. In this section, we identify a few simple ways to decrease your risk by keeping the backup data separate.

The easy (bad) way

In the previous section, we treated /destination/ as if it were just another directory on the same filesystem. Let's call that the easy (bad) approach. It works, but it has several serious limitations:

Fortunately, there are several easy ways to make your backup more robust.

Keep it on a separate partition

If your backup directory is on a separate partition, then any corruption in the main filesystem will not normally affect the backup. If the backup process runs out of disk space, it will fail, but it won't take the rest of the system down too. More importantly, keeping your backups on a separate partition means you can keep them mounted read-only; we'll discuss that in more detail in the next chapter.

Keep that partition on a separate disk

If your backup partition is on a separate hard disk, then you're also protected from hardware failure. That's very important, since hard disks always fail eventually, and often take your data with them. An entire industry has formed to service the needs of those whose broken hard disks contained important data that was not properly backed up.

Important : Notice, however, that in the event of hardware failure you'll still lose any changes made since the last backup. For home or small office users, where backups are made daily or even hourly as described in this document, that's probably fine, but in situations where any data loss at all would be a serious problem (such as where financial transactions are concerned), a RAID system might be more appropriate.

RAID is well-supported under Linux, and the methods described in this document can also be used to create rotating snapshots of a RAID system.

Keep that disk on a separate machine

If you have a spare machine, even a very low-end one, you can turn it into a dedicated backup server. Make it standalone, and keep it in a physically separate place--another room or even another building. Disable every single remote service on the backup server, and connect it only to a dedicated network interface on the source machine.

On the source machine, export the directories that you want to back up via read-only NFS to the dedicated interface. The backup server can mount the exported network directories and run the snapshot routines discussed in this article as if they were local. If you opt for this approach, you'll only be remotely vulnerable if:

  1. a remote root hole is discovered in read-only NFS, and
  2. the source machine has already been compromised.

I'd consider this "pretty good" protection, but if you're (wisely) paranoid, or your job is on the line, build two backup servers. Then you can make sure that at least one of them is always offline.

If you're using a remote backup server and can't get a dedicated line to it (especially if the information has to cross somewhere insecure, like the public internet), you should probably skip the NFS approach and use rsync -e ssh instead.

It has been pointed out to me that rsync operates far more efficiently in server mode than it does over NFS, so if the connection between your source and backup server becomes a bottleneck, you should consider configuring the backup machine as an rsync server instead of using NFS. On the downside, this approach is slightly less transparent to users than NFS--snapshots would not appear to be mounted as a system directory, unless NFS is used in that direction, which is certainly another option (I haven't tried it yet though). Thanks to Martin Pool, a lead developer of rsync , for making me aware of this issue.

Here's another example of the utility of this approach--one that I use. If you have a bunch of windows desktops in a lab or office, an easy way to keep them all backed up is to share the relevant files, read-only, and mount them all from a dedicated backup server using SAMBA. The backup job can treat the SAMBA-mounted shares just like regular local directories.

Making the backup as read-only as possible

In the previous section, we discussed ways to keep your backup data physically separate from the data they're backing up. In this section, we discuss the other side of that coin--preventing user processes from modifying backups once they're made.

We want to avoid leaving the snapshot backup directory mounted read-write in a public place. Unfortunately, keeping it mounted read-only the whole time won't work either--the backup process itself needs write access. The ideal situation would be for the backups to be mounted read-only in a public place, but at the same time, read-write in a private directory accessible only by root, such as /root/snapshot .

There are a number of possible approaches to the challenge presented by mounting the backups read-only. After some amount of thought, I found a solution which allows root to write the backups to the directory but only gives the users read permissions. I'll first explain the other ideas I had and why they were less satisfactory.

It's tempting to keep your backup partition mounted read-only as /snapshot most of the time, but unmount that and remount it read-write as /root/snapshot during the brief periods while snapshots are being made. Don't give in to temptation!.

Bad: mount / umount

A filesystem cannot be unmounted if it's busy--that is, if some process is using it. The offending process need not be owned by root to block an unmount request. So if you plan to umount the read-only copy of the backup and mount it read-write somewhere else, don't--any user can accidentally (or deliberately) prevent the backup from happening. Besides, even if blocking unmounts were not an issue, this approach would introduce brief intervals during which the backups would seem to vanish, which could be confusing to users.

Better: mount read-only most of the time

A better but still-not-quite-satisfactory choice is to remount the directory read-write in place:

mount -o remount,rw /snapshot
[ run backup process ]
mount -o remount,ro /snapshot

Now any process that happens to be in /snapshot when the backups start will not prevent them from happening. Unfortunately, this approach introduces a new problem--there is a brief window of vulnerability, while the backups are being made, during which a user process could write to the backup directory. Moreover, if any process opens a backup file for writing during that window, it will prevent the backup from being remounted read-only, and the backups will stay vulnerable indefinitely.

Tempting but doesn't seem to work: the 2.4 kernel's mount --bind

Starting with the 2.4-series Linux kernels, it has been possible to mount a filesystem simultaneously in two different places. "Aha!" you might think, as I did. "Then surely we can mount the backups read-only in /snapshot , and read-write in /root/snapshot at the same time!"

Alas, no. Say your backups are on the partition /dev/hdb1 . If you run the following commands,

mount /dev/hdb1 /root/snapshot
mount --bind -o ro /root/snapshot /snapshot

then (at least as of the 2.4.9 Linux kernel--updated, still present in the 2.4.20 kernel), mount will report /dev/hdb1 as being mounted read-write in /root/snapshot and read-only in /snapshot , just as you requested. Don't let the system mislead you!

It seems that, at least on my system, read-write vs. read-only is a property of the filesystem, not the mount point. So every time you change the mount status, it will affect the status at every point the filesystem is mounted, even though neither /etc/mtab nor /proc/mounts will indicate the change.

In the example above, the second mount call will cause both of the mounts to become read-only, and the backup process will be unable to run. Scratch this one.

Update: I have it on fairly good authority that this behavior is considered a bug in the Linux kernel, which will be fixed as soon as someone gets around to it. If you are a kernel maintainer and know more about this issue, or are willing to fix it, I'd love to hear from you!

My solution: using NFS on localhost

This is a bit more complicated, but until Linux supports mount --bind with different access permissions in different places, it seems like the best choice. Mount the partition where backups are stored somewhere accessible only by root, such as /root/snapshot . Then export it, read-only, via NFS, but only to the same machine. That's as simple as adding the following line to /etc/exports :

/root/snapshot 127.0.0.1(secure,ro,no_root_squash)

then start nfs and portmap from /etc/rc.d/init.d/ . Finally mount the exported directory, read-only, as /snapshot :

mount -o ro 127.0.0.1:/root/snapshot /snapshot

And verify that it all worked:

mount
...
/dev/hdb1 on /root/snapshot type ext3 (rw)
127.0.0.1:/root/snapshot on /snapshot type nfs (ro,addr=127.0.0.1)

At this point, we'll have the desired effect: only root will be able to write to the backup (by accessing it through /root/snapshot ). Other users will see only the read-only /snapshot directory. For a little extra protection, you could keep mounted read-only in /root/snapshot most of the time, and only remount it read-write while backups are happening.

Damian Menscher pointed out this CERT advisory which specifically recommends against NFS exporting to localhost, though since I'm not clear on why it's a problem, I'm not sure whether exporting the backups read-only as we do here is also a problem. If you understand the rationale behind this advisory and can shed light on it, would you please contact me? Thanks!

Extensions: hourly, daily, and weekly snapshots

With a little bit of tweaking, we make multiple-level rotating snapshots. On my system, for example, I keep the last four "hourly" snapshots (which are taken every four hours) as well as the last three "daily" snapshots (which are taken at midnight every day). You might also want to keep weekly or even monthly snapshots too, depending upon your needs and your available space.

Keep an extra script for each level

This is probably the easiest way to do it. I keep one script that runs every four hours to make and rotate hourly snapshots, and another script that runs once a day rotate the daily snapshots. There is no need to use rsync for the higher-level snapshots; just cp -al from the appropriate hourly one.

Run it all with cron

To make the automatic snapshots happen, I have added the following lines to root's crontab file:

0 */4 * * * /usr/local/bin/make_snapshot.sh
0 13 * * *  /usr/local/bin/daily_snapshot_rotate.sh

They cause make_snapshot.sh to be run every four hours on the hour and daily_snapshot_rotate.sh to be run every day at 13:00 (that is, 1:00 PM). I have included those scripts in the appendix.

If you tire of receiving an email from the cron process every four hours with the details of what was backed up, you can tell it to send the output of make_snapshot.sh to /dev/null , like so:

0 */4 * * * /usr/local/bin/make_snapshot.sh >/dev/null 2>&1

Understand, though, that this will prevent you from seeing errors if make_snapshot.sh cannot run for some reason, so be careful with it. Creating a third script to check for any unusual behavior in the snapshot periodically seems like a good idea, but I haven't implemented it yet. Alternatively, it might make sense to log the output of each run, by piping it through tee , for example. mRgOBLIN wrote in to suggest a better (and obvious, in retrospect!) approach, which is to send stdout to /dev/null but keep stderr, like so:

0 */4 * * * /usr/local/bin/make_snapshot.sh >/dev/null

Presto! Now you only get mail when there's an error. :)

Appendix: my actual configuration

I know that listing my actual backup configuration here is a security risk; please be kind and don't use this information to crack my site. However, I'm not a security expert, so if you see any vulnerabilities in my setup, I'd greatly appreciate your help in fixing them. Thanks!

I actually use two scripts, one for every-four-hours (hourly) snapshots, and one for every-day (daily) snapshots. I am only including the parts of the scripts that relate to backing up /home , since those are relevant ones here.

I use the NFS-to-localhost trick of exporting /root/snapshot read-only as /snapshot , as discussed above.

The system has been running without a hitch for months.

Listing one: make_snapshot.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility
# ----------------------------------------------------------------------
# this needs to be a lot more general, but the basic idea is it makes
# rotating backup-snapshots of /home whenever called
# ----------------------------------------------------------------------

unset PATH	# suggestion from H. Milz: avoid accidental use of $PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;
TOUCH=/bin/touch;

RSYNC=/usr/bin/rsync;


# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;
EXCLUDES=/usr/local/etc/backup_exclude;


# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
	$ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
	exit;
}
fi;


# rotating snapshots of /home (fixme: this should be more general)

# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then			\
$RM -rf $SNAPSHOT_RW/home/hourly.3 ;				\
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/hourly.2 ] ; then			\
$MV $SNAPSHOT_RW/home/hourly.2 $SNAPSHOT_RW/home/hourly.3 ;	\
fi;
if [ -d $SNAPSHOT_RW/home/hourly.1 ] ; then			\
$MV $SNAPSHOT_RW/home/hourly.1 $SNAPSHOT_RW/home/hourly.2 ;	\
fi;

# step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
# if that exists
if [ -d $SNAPSHOT_RW/home/hourly.0 ] ; then			\
$CP -al $SNAPSHOT_RW/home/hourly.0 $SNAPSHOT_RW/home/hourly.1 ;	\
fi;

# step 4: rsync from the system into the latest snapshot (notice that
# rsync behaves like cp --remove-destination by default, so the destination
# is unlinked first.  If it were not so, this would copy over the other
# snapshot(s) too!
$RSYNC								\
	-va --delete --delete-excluded				\
	--exclude-from="$EXCLUDES"				\
	/home/ $SNAPSHOT_RW/home/hourly.0 ;

# step 5: update the mtime of hourly.0 to reflect the snapshot time
$TOUCH $SNAPSHOT_RW/home/hourly.0 ;

# and thats it for home.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
	$ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
	exit;
} fi;

As you might have noticed above, I have added an excludes list to the rsync call. This is just to prevent the system from backing up garbage like web browser caches, which change frequently (so they'd take up space in every snapshot) but would be no loss if they were accidentally destroyed.

Listing two: daily_snapshot_rotate.sh
#!/bin/bash
# ----------------------------------------------------------------------
# mikes handy rotating-filesystem-snapshot utility: daily snapshots
# ----------------------------------------------------------------------
# intended to be run daily as a cron job when hourly.3 contains the
# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
# ----------------------------------------------------------------------

unset PATH

# ------------- system commands used by this script --------------------
ID=/usr/bin/id;
ECHO=/bin/echo;

MOUNT=/bin/mount;
RM=/bin/rm;
MV=/bin/mv;
CP=/bin/cp;

# ------------- file locations -----------------------------------------

MOUNT_DEVICE=/dev/hdb1;
SNAPSHOT_RW=/root/snapshot;

# ------------- the script itself --------------------------------------

# make sure we're running as root
if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi

# attempt to remount the RW mount point as RW; else abort
$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
	$ECHO "snapshot: could not remount $SNAPSHOT_RW readwrite";
	exit;
}
fi;


# step 1: delete the oldest snapshot, if it exists:
if [ -d $SNAPSHOT_RW/home/daily.2 ] ; then			\
$RM -rf $SNAPSHOT_RW/home/daily.2 ;				\
fi ;

# step 2: shift the middle snapshots(s) back by one, if they exist
if [ -d $SNAPSHOT_RW/home/daily.1 ] ; then			\
$MV $SNAPSHOT_RW/home/daily.1 $SNAPSHOT_RW/home/daily.2 ;	\
fi;
if [ -d $SNAPSHOT_RW/home/daily.0 ] ; then			\
$MV $SNAPSHOT_RW/home/daily.0 $SNAPSHOT_RW/home/daily.1;	\
fi;

# step 3: make a hard-link-only (except for dirs) copy of
# hourly.3, assuming that exists, into daily.0
if [ -d $SNAPSHOT_RW/home/hourly.3 ] ; then			\
$CP -al $SNAPSHOT_RW/home/hourly.3 $SNAPSHOT_RW/home/daily.0 ;	\
fi;

# note: do *not* update the mtime of daily.0; it will reflect
# when hourly.3 was made, which should be correct.

# now remount the RW snapshot mountpoint as readonly

$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_RW ;
if (( $? )); then
{
	$ECHO "snapshot: could not remount $SNAPSHOT_RW readonly";
	exit;
} fi;
Sample output of ls -l /snapshot/home
total 28
drwxr-xr-x   12 root     root         4096 Mar 28 00:00 daily.0
drwxr-xr-x   12 root     root         4096 Mar 27 00:00 daily.1
drwxr-xr-x   12 root     root         4096 Mar 26 00:00 daily.2
drwxr-xr-x   12 root     root         4096 Mar 28 16:00 hourly.0
drwxr-xr-x   12 root     root         4096 Mar 28 12:00 hourly.1
drwxr-xr-x   12 root     root         4096 Mar 28 08:00 hourly.2
drwxr-xr-x   12 root     root         4096 Mar 28 04:00 hourly.3

Notice that the contents of each of the subdirectories of /snapshot/home/ is a complete image of /home at the time the snapshot was made. Despite the w in the directory access permissions, no one--not even root--can write to this directory; it's mounted read-only.

Bugs Maintaining Permissions and Owners in the snapshots

The snapshot system above does not properly maintain old ownerships/permissions; if a file's ownership or permissions are changed in place, then the new ownership/permissions will apply to older snapshots as well. This is because rsync does not unlink files prior to changing them if the only changes are ownership/permission. Thanks to J.W. Schultz for pointing this out. Using his new --link-dest option, it is now trivial to work around this problem. See the discussion in the Putting it all together section of Incremental backups with rsync , above.

mv updates timestamp bug

Apparently, a bug in some Linux kernels between 2.4.4 and 2.4.9 causes mv to update timestamps; this may result in inaccurate timestamps on the snapshot directories. Thanks to Claude Felizardo for pointing this problem out. He was able to work around the problem my replacing mv with the following script:

MV=my_mv;
...
function my_mv() {
   REF=/tmp/makesnapshot-mymv-$$;
   touch -r $1 $REF;
   /bin/mv $1 $2;
   touch -r $REF $2;
   /bin/rm $REF;
}
Windows-related problems

I have recently received a few reports of what appear to be interaction issues between Windows and rsync.

One report came from a user who mounts a windows share via Samba, much as I do, and had files mysteriously being deleted from the backup even when they weren't deleted from the source. Tim Burt also used this technique, and was seeing files copied even when they hadn't changed. He determined that the problem was modification time precision; adding --modify-window=10 caused rsync to behave correctly in both cases. If you are rsync'ing from a SAMBA share, you must add --modify-window=10 or you may get inconsistent results. Update: --modify-window=1 should be sufficient. Yet another update: the problem appears to still be there. Please let me know if you use this method and files which should not be deleted are deleted.

Also, for those who use rsync directly on cygwin, there are some known problems, apparently related to cygwin signal handling. Scott Evans reports that rsync sometimes hangs on large directories. Jim Kleckner informed me of an rsync patch, discussed here and here , which seems to work around this problem. I have several reports of this working, and two reports of it not working (the hangs continue). However, one of the users who reported a negative outcome, Greg Boyington, was able to get it working using Craig Barrett's suggested sleep() approach, which is documented here .

Memory use in rsync scales linearly with the number of files being sync'd. This is a problem when syncing large file trees, especially when the server involved does not have a lot of RAM. If this limitation is more of an issue to you than network speed (for example, if you copy over a LAN), you may wish to use mirrordir instead. I haven't tried it personally, but it looks promising. Thanks to Vladimir Vuksan for this tip!

Contributed codes

Several people have been kind enough to send improved backup scripts. There are a number of good ideas here, and I hope they'll save you time when you're ready to design your own backup plan. Disclaimer: I have not necessarily tested these; make sure you check the source code and test them thoroughly before use!

References Frequently Asked Questions

[Feb 12, 2017] Caching-Only DNS Server

linuxservconf.blogspot.com

A caching-only name server is used for looking up zone data and caching (storing) the result which is returned. Then it can return the answers to subsequent queries by using the cached information.

A caching-only server is authoritative only for the local host i.e 0.0.127.in-addr.arpa, but it can automatically send requests to the Internet host handling name lookups for the domain in question.

In most situations, a caching-only name server sends queries directly to the name server that contains the answer. Because of its simplified nature, a DNS zone file is not created for a caching-only name server.

Running the Caching-only Name Server in an chroot environment is a secure approach. The chroot environment has more security compared to the normal environment.

To configure the /etc/named.conf file for a simple caching name server, use this configuration for all servers that don't act as a master or slave name server. Setting up a simple caching server for local client machines will reduce the load on the network's primary server. Many users on DSL connections may use this configuration along with bind for such a purpose. Ensure that the file /etc/named.conf highlights the entries below:

Terminal Window View
options {
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
forwarders { 192.168.1.1; 192.168.1.100; };
forward only;
};
// a caching only nameserver config
controls {
inet 127.0.0.1 allow { localhost; } keys { rndckey; };
};
zone "." IN {
type hint;
file "named.ca";
};
zone "0.0.127.in-addr.arpa" IN {
type master;
file "named.local";
allow-update { none; };
};

With the forwarders option, 192.168.1.1 and 192.168.1.100 are the IP addresses of the Primary/Master and Secondary/Slave DNS server on the network in question. They can also be the IP addresses of the ISPs DNS server and another DNS server, respectively. With the forward only option set in the named.conf file, the name server doesn't try to contact other servers to find out information if the forwarders does not give it an answer. To test this setup try the following commands

Terminal
[root@station1.example.com~] # chkconfig --levels 25 named on
[root@station1.example.com~] # service named restart

We have now turned on the named server for persistent reboots and started the service for the current session. Now to test whether the caching - named server is working or not lets see:

Terminal Window View
# nslookup
>Default
Server: localhost
Address: 127.0.0.1

> www.redhat.com
Server: localhost
Address: 127.0.0.1
Name: www.redhat.com
Address: 209.132.177.50

nslookup now asked the named to look for the machine www.redhat.com. It then contacted one of the name server machines named in the root.cache file, and asked it's way from there. It might take a while before the result is shown, as it searches all the domains the user entered in /etc/resolve.conf. When tried again, the result should be similar to this example:

Terminal Window View
> www.redhat.com
Server: localhost
Address: 127.0.0.1
Non-authoritative answer:
Name: www.redhat.com
Address: 209.132.177.50

Note the Non-authoritative answer in the result this time. This means that named did not go out on the network to ask this time, it instead looked up in its cache and found it there. But the cached information might be out of date. So the user is informed of this danger by it saying Non-authoritative answer. When nslookup says this the second time when a user ask for a host, it is a sign that it caches the information and that it's working. Now exit nslookup by giving the command exit.

[Feb 08, 2017] Linux Server Conf

Feb 08, 2017 | linuxservconf.blogspot.com

Implement Disk Quotas in RHEL 6 and CentOS 6

Quotas are used to limit a user's or a group of users' ability to consume disk space. This prevents a small group of users from monopolising disk capacity and potentially interfering with other users or the entire system. This feature of Linux allows the system administrator to allocate a maximum amount of disk space a user or group may use. It can be flexible in its adherence to the rules assigned and is applied per filesystem. The default Linux Kernel which comes with Redhat and Fedora Core comes with quota support compiled in. In order to implement quotas you need to check whether the kernel supports them.

Check whether the quota support is built into the kernel

Terminal
[root@server1~] # grep CONFIG_QUOTA /boot/config-2.6.18-1.2798.fc6
CONFIG_QUOTA=y
CONFIG_QUOTACTL=y


Those two values should be 'y'. If its not you may need to install the quota rpm package or you may have to recompile your kernel.

Prerequisites
•You should have a seperate unique partition to implement quotas and mount the partition.
•Make necessary changes in /etc/fstab file and remount the partition and check that quotas are on.
•Create files aquota.user and aquota.group in mounted partition folder. [ Note: this may not be required for RHEL5 ] and make sure the owner of those files in root and he should be able to read and write to those files ( chmod 600 aquota.*).
•Run the command quotacheck to check that quotas are implemented.

Create a seperate partitiion

Using your favorite partitiion tool like ( fdisk, parted, gnuparted or any other tool create a partition). Create a directory and mount that partition to that directory.

Terminal
[root@server1~] # mkdir -p /partmount
[root@server1~] # mount /dev/sda7 /partmount


Changes to /etc/fstab

In order to impletment quotas on the partition you need to make changes to /etc/fstab so that when you reboot the system, the quotas are still there on the filesystem. This step is required if you intend to make the quotas permenant. Change the last line as shown in the example.

Terminal
[root@server1~] # vim /etc/fstab
LABEL=/ / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
devpts /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm tmpfs defaults 0 0
proc /proc proc defaults 0 0
sysfs /sys sysfs defaults 0 0
LABEL=SWAP-hdb1 swap swap defaults 0 0
/dev/sda7 /partmount ext3 defaults,usrquota,grpquota 0 0


Now we need to remount the partition for the quotas to be implemented.

Terminal
[root@server1~] # mount -o remount /partmount


Creating quota related files

Terminal
[root@server1~] # cd /partmount
[root@server1 partmount] # touch aquota.user; touch aquota.group
[root@server1 partmount] # chown root:root aquota.*; chmod 600 aquota.*


Check that quotas are working

Terminal
[root@server1~] # quotacheck -vgum /partmount


Implementing Quotas for users and groups

edquota command

To specify disk quotas you need to run either edquota or setquota commands. In order to specify quota for the user henry you will use the command edquota as follows.

Terminal
[root@server1~] # edquota -u henry


You will be presented with the vi editor specifying the soft limit the hard limit and inodes ( the number of files ) soft limit and hard limit.

Soft Limit: This is the maximum amount of space a user can have on that partition. If you have set a grace period, this will act as an alarm, and you will also be required to set a hard limit.

Hard Limit: Hard limits are only neccessary when you are using grace periods. If grace periods are used this will be the absolute limit a user can use.

Grace periods are set by using the command edquota -t /partition

The edquota command can also be used to copy quota settings for one user as a template to any number of users in your system You will need to use -p switch of the edquota command to copy quota settings.

Terminal
[root@server1~] # edquota -up user1 user2 user3 user4


setquota command

Alternatively you can use the setquota command to set quotas on users and groups.

Terminal
[root@server1~] # setquota -u user1 0 0 0 0 /partmount


To set a quota of 4GB to user henry we would use the following command

Terminal
[root@server1~] # setquota -u henry 4194304 4194304 0 0 /partmount


Its important to remember to calcuate Megabytes in times 1024 X 1MB and Gigabytes as (1024)2 x 1GB

To set 240Msoft, 256M hard for user jerry

Terminal
[root@server1~] # setquota -u jerry 245760 262144 0 0 /partmount


Reporting quotas

Reports on quotas allocated to users and groups could be obtained by using the command repquota

Terminal
[root@server1~] # repquota -u /partmount


To obtain individual quota for users jerry and group payroll the commands would be as follows:

Terminal
[root@server1~] # repquota -uv jerry
[root@server1~] # repquota -gv payroll


Quota management

Quotas management can be done with the command quotacheck. This command is useful in the sense we can check that the quotas that are allocated are really in the pipeline for execution.

Terminal
[root@server1~] # quotacheck -vgum /partmount

[Feb 04, 2017] How do I fix mess created by accidentally untarred files in the current dir, aka tar bomb

In such cases the UID of the file is often different from uid of "legitimate" files in polluted directories and you probably can use this fact for quick elimination of the tar bomb, But the idea of using the list of files from the tar bomb to eliminate offending files also works if you observe some precautions -- some directories that were created can have the same names as existing directories. Never do rm in -exec or via xargs without testing.
Notable quotes:
"... You don't want to just rm -r everything that tar tf tells you, since it might include directories that were not empty before unpacking! ..."
"... Another nice trick by @glennjackman, which preserves the order of files, starting from the deepest ones. Again, remove echo when done. ..."
"... One other thing: you may need to use the tar option --numeric-owner if the user names and/or group names in the tar listing make the names start in an unpredictable column. ..."
"... That kind of (antisocial) archive is called a tar bomb because of what it does. Once one of these "explodes" on you, the solutions in the other answers are way better than what I would have suggested. ..."
"... The easiest (laziest) way to do that is to always unpack a tar archive into an empty directory. ..."
"... The t option also comes in handy if you want to inspect the contents of an archive just to see if it has something you're looking for in it. If it does, you can, optionally, just extract the file(s) you want. ..."
Feb 04, 2017 | superuser.com

linux - Undo tar file extraction mess - Super User

first try to issue

tar tf archive
tar will list the contents line by line.

This can be piped to xargs directly, but beware : do the deletion very carefully. You don't want to just rm -r everything that tar tf tells you, since it might include directories that were not empty before unpacking!

You could do

tar tf archive.tar | xargs -d'\n' rm -v
tar tf archive.tar | sort -r | xargs -d'\n' rmdir -v

to first remove all files that were in the archive, and then the directories that are left empty.

sort -r (glennjackman suggested tac instead of sort -r in the comments to the accepted answer, which also works since tar 's output is regular enough) is needed to delete the deepest directories first; otherwise a case where dir1 contains a single empty directory dir2 will leave dir1 after the rmdir pass, since it was not empty before dir2 was removed.

This will generate a lot of

rm: cannot remove `dir/': Is a directory


and

rmdir: failed to remove `dir/': Directory not empty
rmdir: failed to remove `file': Not a directory

Shut this up with 2>/dev/null if it annoys you, but I'd prefer to keep as much information on the process as possible.

And don't do it until you are sure that you match the right files. And perhaps try rm -i to confirm everything. And have backups, eat your breakfast, brush your teeth, etc.

===

List the contents of the tar file like so:

tar tzf myarchive.tar

Then, delete those file names by iterating over that list:

while IFS= read -r file; do echo "$file"; done < <(tar tzf myarchive.tar.gz)

This will still just list the files that would be deleted. Replace echo with rm if you're really sure these are the ones you want to remove. And maybe make a backup to be sure.

In a second pass, remove the directories that are left over:

while IFS= read -r file; do rmdir "$file"; done < <(tar tzf myarchive.tar.gz)

This prevents directories with from being deleted if they already existed before.

Another nice trick by @glennjackman, which preserves the order of files, starting from the deepest ones. Again, remove echo when done.

tar tvf myarchive.tar | tac | xargs -d'\n' echo rm

This could then be followed by the normal rmdir cleanup.


Here's a possibility that will take the extracted files and move them to a subdirectory, cleaning up your main folder.
    #!/usr/bin/perl -w  

    use strict  ;  
    use   Getopt  ::  Long  ;  

    my $clean_folder   =     "clean"  ;  
    my $DRY_RUN  ;  
    die   "Usage: $0 [--dry] [--clean=dir-name]\n"  
          if     (     !  GetOptions  (  "dry!"     =>   \$DRY_RUN  ,  
                           "clean=s"     =>   \$clean_folder  ));  

      # Protect the 'clean_folder' string from shell substitution  
    $clean_folder   =~   s  /  '/'  \\  ''  /  g  ;  

      # Process the "tar tv" listing and output a shell script.  
    print   "#!/bin/sh\n"     if     (     !  $DRY_RUN   );  
      while     (<>)  
      {  
        chomp  ;  

          # Strip out permissions string and the directory entry from the 'tar' list  
        my $perms   =   substr  (  $_  ,     0  ,     10  );  
        my $dirent   =   substr  (  $_  ,     48  );  

          # Drop entries that are in subdirectories  
        next   if     (   $dirent   =~   m  :/.:     );  

          # If we're in "dry run" mode, just list the permissions and the directory  
          # entries.  
          #  
          if     (   $DRY_RUN   )  
          {  
            print   "$perms|$dirent\n"  ;  
            next  ;  
          }  

          # Emit the shell code to clean up the folder  
        $dirent   =~   s  /  '/'  \\  ''  /  g  ;  
        print   "mv -i '$dirent' '$clean_folder'/.\n"  ;  
      } 

Save this to the file fix-tar.pl and then execute it like this:

 $ tar tvf myarchive  .  tar   |   perl fix  -  tar  .  pl   --  dry 

This will confirm that your tar list is like mine. You should get output like:

  -  rw  -  rw  -  r  --|  batch
  -  rw  -  rw  -  r  --|  book  -  report  .  png
  -  rwx  ------|  CaseReports  .  png
  -  rw  -  rw  -  r  --|  caseTree  .  png
  -  rw  -  rw  -  r  --|  tree  .  png
drwxrwxr  -  x  |  sample  / 

If that looks good, then run it again like this:

$ mkdir cleanup
$ tar tvf myarchive  .  tar   |   perl fix  -  tar  .  pl   --  clean  =  cleanup   >   fixup  .  sh 

The fixup.sh script will be the shell commands that will move the top-level files and directories into a "clean" folder (in this instance, the folder called cleanup). Have a peek through this script to confirm that it's all kosher. If it is, you can now clean up your mess with:

 $ sh fixup  .  sh 

I prefer this kind of cleanup because it doesn't destroy anything that isn't already destroyed by being overwritten by that initial tar xv.

Note: if that initial dry run output doesn't look right, you should be able to fiddle with the numbers in the two substr function calls until they look proper. The $perms variable is used only for the dry run so really only the $dirent substring needs to be proper.

One other thing: you may need to use the tar option --numeric-owner if the user names and/or group names in the tar listing make the names start in an unpredictable column.

One other thing: you may need to use the tar option --numeric-owner if the user names and/or group names in the tar listing make the names start in an unpredictable column.

===

That kind of (antisocial) archive is called a tar bomb because of what it does. Once one of these "explodes" on you, the solutions in the other answers are way better than what I would have suggested.

The best "solution", however, is to prevent the problem in the first place.

The easiest (laziest) way to do that is to always unpack a tar archive into an empty directory. If it includes a top level directory, then you just move that to the desired destination. If not, then just rename your working directory (the one that was empty) and move that to the desired location.

If you just want to get it right the first time, you can run tar -tvf archive-file.tar | less and it will list the contents of the archive so you can see how it is structured and then do what is necessary to extract it to the desired location to start with.

The t option also comes in handy if you want to inspect the contents of an archive just to see if it has something you're looking for in it. If it does, you can, optionally, just extract the file(s) you want.

[Feb 04, 2017] Quickly find differences between two directories

You will be surprised, but GNU diff use in Linux understands the situation when two arguments are directories and behaves accordingly
Feb 04, 2017 | www.cyberciti.biz

The diff command compare files line by line. It can also compare two directories:

# Compare two folders using diff ##
diff /etc /tmp/etc_old  
Rafal Matczak September 29, 2015, 7:36 am
§ Quickly find differences between two directories
And quicker:
 diff -y <(ls -l ${DIR1}) <(ls -l ${DIR2})  

[Feb 04, 2017] Restoring deleted /tmp folder

Jan 13, 2015 | cyberciti.biz

As my journey continues with Linux and Unix shell, I made a few mistakes. I accidentally deleted /tmp folder. To restore it all you have to do is:

mkdir /tmp
chmod 1777 /tmp
chown root:root /tmp
ls -ld /tmp
 
mkdir /tmp chmod 1777 /tmp chown root:root /tmp ls -ld /tmp 

[Feb 04, 2017] Use CDPATH to access frequent directories in bash - Mac OS X Hints

Feb 04, 2017 | hints.macworld.com
The variable CDPATH defines the search path for the directory containing directories. So it served much like "directories home". The dangers are in creating too complex CDPATH. Often a single directory works best. For example export CDPATH = /srv/www/public_html . Now, instead of typing cd /srv/www/public_html/CSS I can simply type: cd CSS
Use CDPATH to access frequent directories in bash UNIX
Mar 21, '05 10:01:00AM • Contributed by: jonbauman

I often find myself wanting to cd to the various directories beneath my home directory (i.e. ~/Library, ~/Music, etc.), but being lazy, I find it painful to have to type the ~/ if I'm not in my home directory already. Enter CDPATH , as desribed in man bash ):

The search path for the cd command. This is a colon-separated list of directories in which the shell looks for destination directories specified by the cd command. A sample value is ".:~:/usr".
Personally, I use the following command (either on the command line for use in just that session, or in .bash_profile for permanent use):
CDPATH=".:~:~/Library"

This way, no matter where I am in the directory tree, I can just cd dirname , and it will take me to the directory that is a subdirectory of any of the ones in the list. For example:
$ cd
$ cd Documents 
/Users/baumanj/Documents
$ cd Pictures
/Users/username/Pictures
$ cd Preferences
/Users/username/Library/Preferences
etc...

[ robg adds: No, this isn't some deeply buried treasure of OS X, but I'd never heard of the CDPATH variable, so I'm assuming it will be of interest to some other readers as well.]

cdable_vars is also nice
Authored by: clh on Mar 21, '05 08:16:26PM

Check out the bash command shopt -s cdable_vars

From the man bash page:

cdable_vars

If set, an argument to the cd builtin command that is not a directory is assumed to be the name of a variable whose value is the directory to change to.

With this set, if I give the following bash command:

export d="/Users/chap/Desktop"

I can then simply type

cd d

to change to my Desktop directory.

I put the shopt command and the various export commands in my .bashrc file.

[Feb 04, 2017] Copy file into multiple directories

Feb 04, 2017 | www.cyberciti.biz
Instead of running:
cp /path/to/file /usr/dir1
cp /path/to/file /var/dir2
cp /path/to/file /nas/dir3

Run the following command to copy file into multiple dirs:

echo /usr/dir1 /var/dir2 /nas/dir3 | xargs -n 1 cp -v /path/to/file

[Feb 04, 2017] 20 Unix Command Line Tricks – Part I

Feb 04, 2017 | www.cyberciti.biz
Build directory trees in a single command

You can create directory trees one at a time using mkdir command by passing the -p option:

mkdir -p /jail/{dev,bin,sbin,etc,usr,lib,lib64}
ls -l /jail

[Feb 04, 2017] List all files or directories on your system

Feb 04, 2017 | www.cyberciti.biz
List all files or directories on your system

To see all of the directories on your system, run:

find
/
-type
 d 
|
less

 
# List all directories in your $HOME
find
$HOME
-type
 d 
-ls
|
less

find / -type d | less# List all directories in your $HOME find $HOME -type d -ls | less

To see all of the files, run:

find
/
-type
 f 
|
less

 
# List all files in your $HOME
find
$HOME
-type
 f 
-ls
|
less

[Jan 15, 2017] Driverless Shuttles Hit Las Vegas No Steering Wheels, No Brake Pedals Zero Hedge

Notable quotes:
"... But human life depends on whether the accident is caused by a human or not, and the level of intent. It isn't just a case of the price - the law is increasingly locking people up for driving negligence (rightly in my mind) Who gets locked up when the program fails? Or when the program chooses to hit one person and not another in a complex situation? ..."
Jan 15, 2017 | www.zerohedge.com

Submitted by Mike Shedlock via MishTalk.com,

Electric, driverless shuttles with no steering wheel and no brake pedal are now operating in Las Vegas.

There's a new thrill on the streets of downtown Las Vegas, where high- and low-rollers alike are climbing aboard what officials call the first driverless electric shuttle operating on a public U.S. street.

The oval-shaped shuttle began running Tuesday as part of a 10-day pilot program, carrying up to 12 passengers for free along a short stretch of the Fremont Street East entertainment district.

The vehicle has a human attendant and computer monitor, but no steering wheel and no brake pedals. Passengers push a button at a marked stop to board it.

The shuttle uses GPS, electronic curb sensors and other technology, and doesn't require lane lines to make its way.

"The ride was smooth. It's clean and quiet and seats comfortably," said Mayor Carolyn Goodman, who was among the first public officials to hop a ride on the vehicle developed by the French company Navya and dubbed Arma.

"I see a huge future for it once they get the technology synchronized," the mayor said Friday.

The top speed of the shuttle is 25 mph, but it's running about 15 mph during the trial, Navya spokesman Martin Higgins said.

Higgins called it "100 percent autonomous on a programmed route."

"If a person or a dog were to run in front of it, it would stop," he said.

Higgins said it's the company's first test of the shuttle on a public street in the U.S. A similar shuttle began testing in December at a simulated city environment at a University of Michigan research center.

The vehicle being used in public was shown earlier at the giant CES gadget show just off the Las Vegas Strip.

Las Vegas city community development chief Jorge Cervantes said plans call for installing transmitters at the Fremont Street intersections to communicate red-light and green-light status to the shuttle.

He said the city hopes to deploy several autonomous shuttle vehicles - by Navya or another company - later this year for a downtown loop with stops at shopping spots, restaurants, performance venues, museums, a hospital and City Hall.

At a cost estimated at $10,000 a month, Cervantes said the vehicle could be cost-efficient compared with a single bus and driver costing perhaps $1 million a year.

The company said it has shuttles in use in France, Australia, Switzerland and other countries that have carried more than 100,000 passengers in more than a year of service.

Don't Worry Tax Drivers

Don't worry taxi drivers because some of my readers say
1.This will never work
2.There is no demand
3.Technology cost will be too high
4.Insurance cost will be too high
5.The unions will not allow it
6.It will not be reliable
7.Vehicles will be stolen
8.It cannot handle snow, ice, or any adverse weather.
9.It cannot handle dogs, kids, or 80-year old men on roller skates who will suddenly veer into traffic causing a clusterfack that will last days.
10.This is just a test, and testing will never stop.

Real World Analysis

Those in the real world expect millions of long haul truck driving jobs will vanish by 2020-2022 and massive numbers of taxi job losses will happen simultaneously or soon thereafter.

Yes, I bumped up my timeline by two years (from 2022-2024 to 2020-2022) for this sequence of events.

My new timeline is not all tremendously optimistic given the rapid changes we have seen.

garypaul -> Sudden Debt •Jan 14, 2017 7:56 PM

You're getting carried away Sudden Debt. This robot stuff works great in the lab/test zones. Whether it is transplantable on a larger scale is still unknown. The interesting thing is, all my friends who are computer programmers/engineers/scientists are skeptical about this stuff, but all my friends who know nothing about computer science are absolutely wild about the "coming age of robots/AI". Go figure.

P.S. Of course the computer experts that are milking investment money with their start-ups will tell you it's great

ChartreuseDog -> garypaul •Jan 14, 2017 9:15 PM

I'm an engineer (well, OK, an electrical engineering technical team lead). I've been an electronics and embedded computer engineer for about 4 decades.

This Vegas thing looks real - predefined route, transmitted signals for traffic lights, like light rail without the rails.

Overall, autonomous driving looks like it's almost here, if you like spinning LIDAR transceivers on the top of cars.

Highway driving is much closer to being solved, by the way. It's suburban and urban side streets that are the tough stuff.

garypaul -> ChartreuseDog •Jan 14, 2017 9:22 PM

"Highway driving is much closer to being solved".

That's my whole point. It's not an equation that you "solve". It's a million unexpected things. Last I heard, autonomous cars were indeed already crashing.

MEFOBILLS -> CRM114 •Jan 14, 2017 6:07 PM

Who gets sued? For how much? What about cases where a human driver wouldn't have killed anybody?

I've been in corporate discussions about this very topic. At a corporation that makes this technology by the way. The answer:

Insurance companies and the law will figure it out. Basically, if somebody gets run over, then the risk does not fall on the technology provider. Corporate rules can be structured to prevent piercing the corporate veil on this.

Human life does have a price. Insurance figures out how much it costs to pay off, and then jacks up rates accordingly.

CRM114 -> MEFOBILLS •Jan 14, 2017 6:20 PM

Thanks, that's interesting, although I must say that isn't a solution, it's a hope that someone else will come up with one.

But human life depends on whether the accident is caused by a human or not, and the level of intent. It isn't just a case of the price - the law is increasingly locking people up for driving negligence (rightly in my mind) Who gets locked up when the program fails? Or when the program chooses to hit one person and not another in a complex situation?

At the moment, corporate manslaughter laws are woefully inadequate. There's clearly one law for the rich and another for everyone else. Mary Barra would be wearing an orange jumpsuit otherwise.

I am unaware of any automatic machinery which operates in public areas and carries significant risk. Where accidents have happened in the past(e.g.elevators), either the machinery gets changed to remove the risk, or use is discontinued, or the public is separated from the machinery. I don't think any of these are possible for automatic vehicles.

TuPhat -> shovelhead •Jan 14, 2017 7:53 PM

Elevators have no choice of route, only how high or low you want to go. autos have no comparison. Disney world has had many robotic attractions for decades but they are still only entertainment. keep entertaining yourself Mish. when I see you on the road I will easily pass you by.

MEFOBILLS -> Hulk •Jan 14, 2017 6:12 PM

The future is here: See movie "obsolete" on Amazon. Free if you have prime.

https://www.amazon.com/dp/B01M8MHZRH?autoplay=1&t=2936

Mr_Potatohead •Jan 14, 2017 6:08 PM

This is so exciting! Just think about the possibilities here... Shuttles could be outfitted with all kinds of great gizmos to identify their passengers based on RFID chips in credit cards, facial recognition software, voice prints, etc. Then, depending on who is controlling the software, the locks on the door could engage and the shuttle could drive around town dropping of its passengers to various locations eager for their arrival. Trivial to round up illegal aliens, parole violators, or people with standing warrants for arrest. Equally easy to nab people who are delinquent on their taxes, credit cards, mortgages, and spousal support. With a little info from Facebook or Google, a drop-off at the local attitude-adjustment facility might be desirable for those who frequent alternative media or have unhealthy interests in conspiracy theories or the activities at pizza parlors. Just think about the wonderful possibilties here!

Twee Surgeon -> PitBullsRule •Jan 14, 2017 6:29 PM

Will unemployed taxi drivers be allowed on the bus with a bottle of vodka and a gallon of gas with a rag in it ?

When the robot trucks arrive at the robot factory and are unloaded by robot forklifts, who will buy the end products ?

It won't be truck drivers, taxi drivers or automated production line workers.

The only way massive automation would work is if some people were planning on a vastly reduced population in the future. It has happened before, they called it the Black Death. The Cultural and Economic consequences of it in Europe were enormous, world changing and permanent.

animalspirit •Jan 14, 2017 6:32 PM

$10K / month ... that's $120,000 / year.

For an autonomous golf cart?


[Jan 14, 2017] Weak Labor Market: President Obama Hides Behind Automation

Notable quotes:
"... The unionization rate has plummeted over the last four decades, but this is the result of policy decisions, not automation. Canada, a country with a very similar economy and culture, had no remotely comparable decline in unionization over this period. ..."
"... The unemployment rate and overall strength of the labor market is also an important factor determining workers' ability to secure their share of the benefits of productivity growth in wages and other benefits. When the Fed raises interest rates to deliberately keep workers from getting jobs, this is not the result of automation. ..."
"... It is also not automation alone that allows some people to disproportionately get the gains from growth. The average pay of doctors in the United States is over $250,000 a year because they are powerful enough to keep out qualified foreign doctors. They require that even established foreign doctors complete a U.S. residency program before they are allowed to practice medicine in the United States. If we had a genuine free market in physicians' services every MRI would probably be read by a much lower paid radiologist in India rather than someone here pocketing over $400,000 a year. ..."
Jan 14, 2017 | economistsview.typepad.com
anne : January 13, 2017 at 11:11 AM , 2017 at 11:11 AM
http://cepr.net/blogs/beat-the-press/weak-labor-market-president-obama-hides-behind-automation

January 13, 2017

Weak Labor Market: President Obama Hides Behind Automation

It really is shameful how so many people, who certainly should know better, argue that automation is the factor depressing the wages of large segments of the workforce and that education (i.e. blame the ignorant workers) is the solution. President Obama takes center stage in this picture since he said almost exactly this in his farewell address earlier in the week. This misconception is repeated in a Claire Cain Miller's New York Times column * today. Just about every part of the story is wrong.

Starting with the basic story of automation replacing workers, we have a simple way of measuring this process, it's called "productivity growth." And contrary to what the automation folks tell you, productivity growth has actually been very slow lately.

[Graph]

The figure above shows average annual rates of productivity growth for five year periods, going back to 1952. As can be seen, the pace of automation (productivity growth) has actually been quite slow in recent years. It is also projected by the Congressional Budget Office and most other forecasters to remain slow for the foreseeable future, so the prospect of mass displacement of jobs by automation runs completely counter to what we have been seeing in the labor market.

Perhaps more importantly the idea that productivity growth is bad news for workers is 180 degrees at odds with the historical experience. In the period from 1947 to 1973, productivity growth averaged almost 3.0 percent, yet the unemployment rate was generally low and workers saw rapid wage gains. The reason was that workers had substantial bargaining power, in part because of strong unions, and were able to secure the gains from productivity growth for themselves in higher living standards, including more time off in the form of paid vacation days and paid sick days. (Shorter work hours sustain the number of jobs in the face rising productivity.)

The unionization rate has plummeted over the last four decades, but this is the result of policy decisions, not automation. Canada, a country with a very similar economy and culture, had no remotely comparable decline in unionization over this period.

The unemployment rate and overall strength of the labor market is also an important factor determining workers' ability to secure their share of the benefits of productivity growth in wages and other benefits. When the Fed raises interest rates to deliberately keep workers from getting jobs, this is not the result of automation.

It is also not automation alone that allows some people to disproportionately get the gains from growth. The average pay of doctors in the United States is over $250,000 a year because they are powerful enough to keep out qualified foreign doctors. They require that even established foreign doctors complete a U.S. residency program before they are allowed to practice medicine in the United States. If we had a genuine free market in physicians' services every MRI would probably be read by a much lower paid radiologist in India rather than someone here pocketing over $400,000 a year.

Similarly, automation did not make our patents and copyrights longer and stronger. These protectionist measures result in us paying over $430 billion a year for drugs that would likely cost one tenth of this amount in a free market. And automation did not force us to institutionalize rules that created an incredibly bloated financial sector with Wall Street traders and hedge fund partners pocketing tens of millions or even hundreds of millions a year. Nor did automation give us a corporate governance structure that allows even the most incompetent CEOs to rip off their companies and pay themselves tens of millions a year.

Yes, these and other topics are covered in my (free) book "Rigged: How Globalization and the Rules of the Modern Economy Were Structured to Make the Rich Richer." ** It is understandable that the people who benefit from this rigging would like to blame impersonal forces like automation, but it just ain't true and the people repeating this falsehood should be ashamed of themselves.

* https://www.nytimes.com/2017/01/12/upshot/in-obamas-farewell-a-warning-on-automations-perils.html

** http://deanbaker.net/images/stories/documents/Rigged.pdf

-- Dean Baker

anne -> anne... , January 13, 2017 at 10:46 AM
https://fred.stlouisfed.org/graph/?g=cmzG

January 4, 2016

Nonfarm Business Labor Productivity, * 1948-2016

* Output per hour of all persons

(Indexed to 1948)

https://fred.stlouisfed.org/graph/?g=cmzE

January 4, 2016

Nonfarm Business Labor Productivity, * 1948-2016

* Output per hour of all persons

(Percent change)

Fred C. Dobbs : , -1
(Dang robots.)

A Darker Theme in Obama's Farewell: Automation Can
Divide Us https://nyti.ms/2ioACof via @UpshotNYT
NYT - Claire Cain Miller - January 12, 2017

Underneath the nostalgia and hope in President Obama's farewell address Tuesday night was a darker theme: the struggle to help the people on the losing end of technological change.

"The next wave of economic dislocations won't come from overseas," Mr. Obama said. "It will come from the relentless pace of automation that makes a lot of good, middle-class jobs obsolete."

Donald J. Trump has tended to blamed trade, offshoring and immigration. Mr. Obama acknowledged those things have caused economic stress. But without mentioning Mr. Trump, he said they divert attention from the bigger culprit.

Economists agree that automation has played a far greater role in job loss, over the long run, than globalization. But few people want to stop technological progress. Indeed, the government wants to spur more of it. The question is how to help those that it hurts.

The inequality caused by automation is a main driver of cynicism and political polarization, Mr. Obama said. He connected it to the racial and geographic divides that have cleaved the country post-election.

It's not just racial minorities and others like immigrants, the rural poor and transgender people who are struggling in society, he said, but also "the middle-aged white guy who, from the outside, may seem like he's got advantages, but has seen his world upended by economic and cultural and technological change."

Technological change will soon be a problem for a much bigger group of people, if it isn't already. Fifty-one percent of all the activities Americans do at work involve predictable physical work, data collection and data processing. These are all tasks that are highly susceptible to being automated, according to a report McKinsey published in July using data from the Bureau of Labor Statistics and O*Net to analyze the tasks that constitute 800 jobs.

Twenty-eight percent of work activities involve tasks that are less susceptible to automation but are still at risk, like unpredictable physical work or interacting with people. Just 21 percent are considered safe for now, because they require applying expertise to make decisions, do something creative or manage people.

The service sector, including health care and education jobs, is considered safest. Still, a large part of the service sector is food service, which McKinsey found to be the most threatened industry, even more than manufacturing. Seventy-three percent of food service tasks could be automated, it found.

In December, the White House released a report on automation, artificial intelligence and the economy, warning that the consequences could be dire: "The country risks leaving millions of Americans behind and losing its position as the global economic leader."

No one knows how many people will be threatened, or how soon, the report said. It cited various researchers' estimates that from 9 percent to 47 percent of jobs could be affected.

In the best case, it said, workers will have higher wages and more leisure time. In the worst, there will be "significantly more workers in need of assistance and retraining as their skills no longer match the demands of the job market."

Technology delivers its benefits and harms in an unequal way. That explains why even though the economy is humming, it doesn't feel like it for a large group of workers.

Education is the main solution the White House advocated. When the United States moved from an agrarian economy to an industrialized economy, it rapidly expanded high school education: By 1951, the average American had 6.2 more years of education than someone born 75 years earlier. The extra education enabled people to do new kinds of jobs, and explains 14 percent of the annual increases in labor productivity during that period, according to economists.

Now the country faces a similar problem. Machines can do many low-skilled tasks, and American children, especially those from low-income and minority families, lag behind their peers in other countries educationally.

The White House proposed enrolling more 4-year-olds in preschool and making two years of community college free for students, as well as teaching more skills like computer science and critical thinking. For people who have already lost their jobs, it suggested expanding apprenticeships and retraining programs, on which the country spends half what it did 30 years ago.

Displaced workers also need extra government assistance, the report concluded. It suggested ideas like additional unemployment benefits for people who are in retraining programs or live in states hardest hit by job loss. It also suggested wage insurance for people who lose their jobs and have to take a new one that pays less. Someone who made $18.50 an hour working in manufacturing, for example, would take an $8 pay cut if he became a home health aide, one of the jobs that is growing most quickly.

President Obama, in his speech Tuesday, named some other policy ideas for dealing with the problem: stronger unions, an updated social safety net and a tax overhaul so that the people benefiting most from technology share some of their earnings.

The Trump administration probably won't agree with many of those solutions. But the economic consequences of automation will be one of the biggest problems it faces.

[Jan 12, 2017] Digitopoly Congestion on the Last Mile

Notable quotes:
"... By Shane Greenstein On Jan 11, 2017 · Add Comment · In Broadband , communication , Esssay , Net Neutrality ..."
"... The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes. ..."
"... The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments. ..."
"... such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating. ..."
"... Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics. ..."
"... Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold. ..."
"... More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation. ..."
Jan 12, 2017 | www.digitopoly.org
Congestion on the Last Mile
By Shane Greenstein On Jan 11, 2017 · Add Comment · In Broadband , communication , Esssay , Net Neutrality It has long been recognized that networked services contain weak-link vulnerabilities. That is, the performance of any frontier device depends on the performance of every contributing component and service. This column focuses on one such phenomenon, which goes by the label "congestion." No, this is not a new type of allergy, but, as with a bacteria, many users want to avoid it, especially advanced users of frontier network services.

Congestion arises when network capacity does not provide adequate service during heavy use. Congestion slows down data delivery and erodes application performance, especially for time-sensitive apps such as movies, online videos, and interactive gaming.

Concerns about congestion are pervasive. Embarrassing reports about broadband networks with slow speeds highlight the role of congestion. Regulatory disputes about data caps and pricing tiers question whether these programs limit the use of data in a useful way. Investment analysts focus on the frequency of congestion as a measure of a broadband network's quality.

What economic factors produce congestion? Let's examine the root economic causes.

The Basics

Congestion arises when demand for data exceeds supply in a very specific sense.

Start with demand. To make this digestible, let's confine our attention to US households in an urban or suburban area, which produces the majority of data traffic.

No simple generalization can characterize all users and uses. The typical household today uses data for a wide variety of purposes-email, video, passive browsing, music videos, streaming of movies, and e-commerce. Networks also interact with a wide variety of end devices-PCs, tablets, smartphones on local Wi-Fi, streaming to television, home video alarm systems, remote temperature control systems, and plenty more.

It is complicated, but two facts should be foremost in this discussion. First, a high fraction of traffic is video-anywhere from 60 to 80 percent, depending on the estimate. Second, demand peaks at night. Most users want to do more things after dinner, far more than any other time during the day.

Every network operator knows that demand for data will peak (predictably) between approximately 7 p.m. and 11 p.m. Yes, it is predictable. Every day of the week looks like every other, albeit with steady growth over time and with some occasional fluctuations for holidays and weather. The weekends don't look any different, by the way, except that the daytime has a bit more demand than during the week.

The bottom line: evenings require far greater capacity than other times of the day. If capacity is not adequate, it can manifest as a bottleneck at many different points in a network-in its backbone, in its interconnection points, or in its last mile nodes.

This is where engineering and economics can become tricky to explain (and to manage). Consider this metaphor (with apologies to network engineers): Metaphorically speaking, network congestion can resemble a bathtub backed up with water. The water might fail to drain because something is interfering with the mouth of the drain or there is a clog far down the pipes. So, too, congestion in a data network can arise from inadequate capacity close to the household or inadequate capacity somewhere in the infrastructure supporting delivery of data.

Numerous features inside a network can be responsible for congestion, and that shapes which set of households experience congestion most severely. Accordingly, numerous different investments can alleviate the congestion in specific places. A network could require a "splitting of nodes" or a "larger pipe" to support a content delivery network (CDN) or could require "more ports at the point of interconnection" between a particular backbone provider and the network.

As it turns out, despite that complexity, we live in an era in which bottlenecks arise most often in the last mile, which ISPs build and operate. That simplifies the economics: Once an ISP builds and optimizes a network to meet maximum local demand at peak hours, then that same capacity will be able to meet lower demand the rest of the day. Similarly, high capacity can also address lower levels of peak demand on any other day.

Think of the economics this way. An awesome network, with extraordinary capacity optimized to its users, will alleviate congestion at most households on virtually every day of the week, except the most extraordinary. Accordingly, as the network becomes less than awesome with less capacity, it will generate a number of (predictable) days of peak demand with severe congestion throughout the entire peak time period at more households. The logic carries through: the less awesome the network, the greater the number of households who experience those moments of severe congestion, and the greater the frequency.

That provides a way to translate many network engineering benchmarks-such as the percentage of packet loss. More packet loss correlates with more congestion, and that corresponds with a larger number of moments when some household experiences poor service.

Tradeoffs and Externalities

Not all market participants react to congestion in the same way. Let's first focus on the gazillion Web firms that supply the content. They watch this situation with a wary eye, and it's no wonder. Many third-party services, such as those streaming video, deliver a higher-quality experience to users whose network suffers less congestion.

Many content providers invest to alleviate congestion. Some invest in compression software and superior webpage design, which loads in ways that speeds up the user experience. Some buy CDN services to speed delivery of their data. Some of the largest content firms, such as YouTube, Google, Netflix, and Facebook, build their own CDN services to improve delivery.

Next, focus on ISPs. They react with various investment and pricing strategies. At one extreme, some ISPs have chosen to save money by investing conservatively, and they suffer the complaints of users. At the other extreme, some ISPs build a premium network, then charge premium prices for the best services.

There are two good reasons for that variety. First, ISPs differ in their rates of capital investment. Partly this is due to investment costs, which vary greatly with density, topography, and local government relations. Rates of investment tend to be inherited from long histories, sometimes as a product of decisions made many years ago, which accumulated over time. These commitments can change, but generally don't, because investors watch capital commitments and react strongly to any departure from history.

The second reason is more subtle. ISPs take different approaches to raising revenue per household, and this results in (effectively) different relationships with banks and stockholders, and, de facto, different budgets for investment. Where does the difference in revenue come from? For one, competitive conditions and market power differ across neighborhoods. In addition, ISPs use different pricing strategies, taking substantially different approaches to discounts, tiered pricing structures, data cap policies, bundled contract offerings, and nuisance fees.

The use of tiers tends to grab attention in public discussion. ISPs segment their users. Higher tiers bring more bandwidth to a household. All else equal, households with higher tiers experience less congestion at peak moments.

Investors like tiers because they don't obligate ISPs to offer unlimited service and, in the long run, raise revenue without additional costs. Users have a more mixed reaction. Light users like the lower prices of lower tiers, and appreciate saving money for doing little other than email and static browsing.

In contrast, heavy users perceive that they pay extra to receive the bandwidth that the ISP used to supply as a default.

ISPs cannot win for losing. The archetypical conservative ISP invests adequately to relieve congestion some of the time, but not all of the time. Its management then must face the occasional phone calls of its users, which they stymie with phone trees that make service calls last 45 minutes. Even if users like the low prices, they find the service and reliability quite irritating.

The archetypical aggressive ISP, in contrast, achieves a high-quality network, which relieves severe congestion much of the time. Yet, such firms (typically) find clever ways to pile on fees, and know how to stymie user complaints with a different type of phone tree that makes calls last 45 minutes. Even when users like the quality, the aggressive pricing practices tend to be quite irritating.

One last note: It is a complicated situation where ISPs interconnect with content providers. Multiple parties must invest, and the situations involve many supplier interests and strategic contingencies.

Some observers have alleged that the biggest ISPs have created congestion issues at interconnection points for purposes of gaining negotiating leverage. These are serious charges, and a certain amount of skepticism is warranted for any broad charge that lacks specifics.

Somebody ought to do a sober and detailed investigation to confront those theories with evidence. (I am just saying.)

What does basic economics tell us about congestion? Congestion is inevitable in a network with interlocking interests. When one part of the network has congestion, the rest of it catches a cold.

More to the point, growth in demand for data should continue to stress network capacity into the foreseeable future. Since not all ISPs will invest aggressively in the presence of congestion, some amount of congestion is inevitable. So, too, is a certain amount of irritation.

Copyright held by IEEE.

To view the printed essay, click here.

[Jan 11, 2017] Discrimination thru stupid job descriptions is catching up to the economy

Jan 11, 2017 | www.nakedcapitalism.com
cocomaan , January 10, 2017 at 4:04 pm

Coulnd't get the JOLTS, November 2016 links to work, but the skills gap is wild.

At an institution of higher ed I'm familiar with, both faculty and administrative positions continue to be unfilled. There are very few candidates even for entry level positions. Failed searches are now the norm. It's feast or famine: either people are perfect for the job and have many options, or have no related experience at all.

I wonder if the labor force participation rate is starting to catch up with the job market. That is, there are a lot of healthy adults who have dropped out of the workforce who would be the people you'd want in those positions.

Or that the job market is not nearly as liquid as they'd have you believe, and people can't relocate from where they are because of adult children who live with them, or things of that nature. All kinds of weird things now in the job market. I know someone who commutes a significant distance to work that has to look for another job because their workplace's health care plan only covers a geographic area close to that job.

alex morfesis , January 10, 2017 at 4:59 pm

Discrimination thru stupid job descriptions is catching up to the economy paying 12 per hour five years experience required nonsense job descriptions designed to help the accredited and credentialed have a leg up

There seem to be three types of employment categories

real jobs that might last through 12 quarters

gigs

and surfdumb/$lavery gigs where your hours are messed with, your schedule is messed with & you are expected to pay for the stupid uniform some bean counter thinks is branding

IMUO it is not a skills gap it is the demanding of irrelevant capacities and experience that almost always have very little to do with the actual tasks required

[Jan 11, 2017] Truck drivers would be at risk due to the growing utilization of heavy-duty vehicles operated via artificial intelligence

Jan 11, 2017 | www.whitehouse.gov

[ A study published late last month by the White House Council of Economic Advisers (CEA)] released Dec. 20, said the jobs of between 1.34 million and 1.67 million truck drivers would be at risk due to the growing utilization of heavy-duty vehicles operated via artificial intelligence. That would equal 80 to 100 percent of all driver jobs listed in the CEA report, which is based on May 2015 data from the Bureau of Labor Statistics, a unit of the Department of Labor. There are about 3.4 million commercial truck drivers currently operating in the U.S., according to various estimates" [ DC Velocity ]. "The Council emphasized that its calculations excluded the number or types of new jobs that may be created as a result of this potential transition. It added that any changes could take years or decades to materialize because of a broad lag between what it called "technological possibility" and widespread adoption."

Class Warfare

[A study published late last month by the White House Council of Economic Advisers (CEA)] released Dec. 20, said the jobs of between 1.34 million and 1.67 million truck drivers would be at risk due to the growing utilization of heavy-duty vehicles operated via artificial intelligence. That would equal 80 to 100 percent of all driver jobs listed in the CEA report, which is based on May 2015 data from the Bureau of Labor Statistics, a unit of the Department of Labor. There are about 3.4 million commercial truck drivers currently operating in the U.S., according to various estimates" [DC Velocity]. "The Council emphasized that its calculations excluded the number or types of new jobs that may be created as a result of this potential transition. It added that any changes could take years or decades to materialize because of a broad lag between what it called "technological possibility" and widespread adoption."

[Jan 06, 2017] Artificial Intelligence Putting The AI In Fail

Notable quotes:
"... As with the most cynical (or deranged) internet hypesters, the current "AI" hype has a grain of truth underpinning it. Today neural nets can process more data, faster. Researchers no longer habitually tweak their models. Speech recognition is a good example: it has been quietly improving for three decades. But the gains nowhere match the hype: they're specialised and very limited in use. So not entirely useless, just vastly overhyped . ..."
"... "What we have seen lately, is that while systems can learn things they are not explicitly told, this is mostly in virtue of having more data, not more subtlety about the data. So, what seems to be AI, is really vast knowledge, combined with a sophisticated UX, " one veteran told me. ..."
"... But who can blame them for keeping quiet when money is suddenly pouring into their backwater, which has been unfashionable for over two decades, ever since the last AI hype collapsed like a souffle? What's happened this time is that the definition of "AI" has been stretched so that it generously encompasses pretty much anything with an algorithm. Algorithms don't sound as sexy, do they? They're not artificial or intelligent. ..."
"... The bubble hasn't yet burst because the novelty examples of AI haven't really been examined closely (we find they are hilariously inept when we do), and they're not functioning services yet. ..."
"... Here I'll offer three reasons why 2016's AI hype will begin to unravel in 2017. That's a conservative guess – much of what is touted as a breakthrough today will soon be the subject of viral derision, or the cause of big litigation. ..."
Jan 04, 2017 | www.zerohedge.com
Submitted by Andrew Orlowski via The Register,

"Fake news" vexed the media classes greatly in 2016, but the tech world perfected the art long ago. With "the internet" no longer a credible vehicle for Silicon Valley's wild fantasies and intellectual bullying of other industries – the internet clearly isn't working for people – "AI" has taken its place.

Almost everything you read about AI is fake news. The AI coverage comes from a media willing itself into the mind of a three year old child, in order to be impressed.

For example, how many human jobs did AI replace in 2016? If you gave professional pundits a multiple choice question listing these three answers: 3 million, 300,000 and none, I suspect very few would choose the correct answer, which is of course "none".

Similarly, if you asked tech experts which recent theoretical or technical breakthrough could account for the rise in coverage of AI, even fewer would be able to answer correctly that "there hasn't been one".

As with the most cynical (or deranged) internet hypesters, the current "AI" hype has a grain of truth underpinning it. Today neural nets can process more data, faster. Researchers no longer habitually tweak their models. Speech recognition is a good example: it has been quietly improving for three decades. But the gains nowhere match the hype: they're specialised and very limited in use. So not entirely useless, just vastly overhyped . As such, it more closely resembles "IoT", where boring things happen quietly for years, rather than "Digital Transformation", which means nothing at all.

The more honest researchers acknowledge as much to me, at least off the record.

"What we have seen lately, is that while systems can learn things they are not explicitly told, this is mostly in virtue of having more data, not more subtlety about the data. So, what seems to be AI, is really vast knowledge, combined with a sophisticated UX, " one veteran told me.

But who can blame them for keeping quiet when money is suddenly pouring into their backwater, which has been unfashionable for over two decades, ever since the last AI hype collapsed like a souffle? What's happened this time is that the definition of "AI" has been stretched so that it generously encompasses pretty much anything with an algorithm. Algorithms don't sound as sexy, do they? They're not artificial or intelligent.

The bubble hasn't yet burst because the novelty examples of AI haven't really been examined closely (we find they are hilariously inept when we do), and they're not functioning services yet. For example, have a look at the amazing "neural karaoke" that researchers at the University of Toronto developed. Please do : it made the worst Christmas record ever.

It's very versatile: it can the write the worst non-Christmas songs you've ever heard , too.

Neural karaoke. The worst song ever, guaranteed

Here I'll offer three reasons why 2016's AI hype will begin to unravel in 2017. That's a conservative guess – much of what is touted as a breakthrough today will soon be the subject of viral derision, or the cause of big litigation. There are everyday reasons that show how once an AI application is out of the lab/PR environment, where it's been nurtured and pampered like a spoiled infant, then it finds the real world is a lot more unforgiving. People don't actually want it.

3. Liability: So you're Too Smart To Fail?

Nine years ago, the biggest financial catastrophe since the 1930s hit the world, and precisely zero bankers went to jail for it. Many kept their perks and pensions. People aren't so happy about this.

So how do you think an all purpose "cat ate my homework" excuse is going to go down with the public, or shareholders? A successfully functioning AI – one that did what it said on the tin – would pose serious challenges to criminal liability frameworks. When something goes wrong, such as a car crash or a bank failure, who do you put in jail? The Board, the CEO or the programmer, or both? "None of the above" is not going to be an option this time.

I believe that this factor alone will keep "AI" out of critical decision making where lives and large amounts of other people's money are at stake. For sure, some people will try to deploy algorithms in important cases. But ultimately there are victims: the public, and shareholders, and the appetite of the public to hear another excuse is wearing very thin. Let's check in on how the Minority Report -style precog detection is going. Actually, let's not .

After "Too Big To Fail", nobody is going to buy "Too Smart to Fail".

2. The Consumer Doesn't Want It

2016 saw "AI" being deployed on consumers experimentally, tentatively, and the signs are already there for anyone who cares to see. It hasn't been a great success.

The most hyped manifestation of better language processing is chatbots . Chatbots are the new UX, many including Microsoft and Facebook hope. Oren Etzoni at Paul Allen's Institute predicts it will become a "trillion dollar industry" But he also admits " my 4 YO is far smarter than any AI program I ever met ".

Hmmm, thanks Oren. So what you're saying is that we must now get used to chatting with someone dumber than a four year old, just because they can make software act dumber than a four year old. Bzzt. Next...

Put it this way. How many times have you rung a call center recently and wished that you'd spoken to someone even more thick, or rendered by processes even more incapable of resolving the dispute, than the minimum-wage offshore staffer who you actually spoke with? When the chatbots come, as you close the [X] on another fantastically unproductive hour wasted, will you cheerfully console yourself with the thought: "That was terrible, but least MegaCorp will make higher margins this year! They're at the cutting edge of AI!"?

In a healthy and competitive services marketplace, bad service means lost business. The early adopters of AI chatbots will discover this the hard way. There may be no later adopters once the early adopters have become internet memes for terrible service.

The other area where apparently impressive feats of "AI" were unleashed upon the public were subtle. Unbidden, unwanted AI "help" is starting to pop out at us. Google scans your personal photos and later, if you have an Android phone will pop up "helpful" reminders of where you have been. People almost universally find this creepy. We could call this a "Clippy The Paperclip" problem, after the intrusive Office Assistant that only wanted to help. Clippy is going to haunt AI in 2017 . This is actually going to be worse than anybody inside the AI cult quite realises.

The successful web services today so far are based on an economic exchange. The internet giants slurp your data, and give you free stuff. We haven't thought more closely about what this data is worth. For the consumer, however, these unsought AI intrusions merely draw our attention to how intrusive the data slurp really is. It could wreck everything. Has nobody thought of that?

1. AI is a make believe world populated by mad people, and nobody wants to be part of it

The AI hype so far has relied on a collusion between two groups of people: a supply side and a demand side. The technology industry, the forecasting industry and researchers provide a limitless supply of post-human hype.

The demand comes from the media and political classes, now unable or unwilling to engage in politics with the masses, to indulge in wild fantasies about humans being replaced by robots. For me, the latter reflects a displacement activity: the professions are already surrendering autonomy in their work to technocratic managerialism . They've made robots out of themselves – and now fear being replaced by robots. (Pass the hankie, I'm distraught.)

There's a cultural gulf between AI's promoters and the public that Asperger's alone can't explain. There's no polite way to express this, but AI belongs to California's inglorious tradition of generating cults, and incubating cult-like thinking . Most people can name a few from the hippy or post-hippy years – EST, or the Family, or the Symbionese Liberation Army – but actually, Californians have been it at it longer than anyone realises .

There's nothing at all weird about Mark. Move along and please tip the Chatbot.

Today, that spirit lives on Silicon Valley, where creepy billionaire nerds like Mark Zuckerberg and Elon Musk can fulfil their desires to " play God and be amazed by magic ", the two big things they miss from childhood. Look at Zuckerberg's house, for example. What these people want is not what you or I want. I'd be wary of them running an after school club.

Out in the real world, people want better service, not worse service; more human and less robotic exchanges with services, not more robotic "post-human" exchanges. But nobody inside the AI cult seems to worry about this. They think we're as amazed as they are. We're not.

The "technology leaders" driving the AI are doing everything they can to alert us to the fact no sane person would task them with leading anything. For that, I suppose, we should be grateful.

Francis Marx Jan 4, 2017 9:13 PM

I worked with robots for years and people dont realize how flawed and "go-wrong" things occur. Companies typically like idea of not hiring humans but in essence the robotic vision is not what it ought to be.

kiss of roses Francis Marx Jan 4, 2017 9:15 PM

I have designed digital based instrumentation and sensors. One of our senior EE designers had a saying that I loved: "Give an electron half a chance and it will fuck you every time."

Zarbo Jan 4, 2017 9:10 PM

I've been hearing the same thing since the first Lisp program crawled out of the digital swamp.

Lessee, that would be about 45 years I've listened to the same stories and fairy tales. I'll take a wait and see attitude like always.

The problem is very complex and working on pieces of it can be momentarily impressive to a press corpse (pun intended) with "the minds of a 3-year old, whether they willed it or not". (fixed that for you).

I'll quote an old saw, Lucke's First Law: "Ignorance simplifies any problem".

Just wait for the free money to dry up and the threat of AI will blow away (for a while longer) with the bankers dust.

cherry picker Jan 4, 2017 9:13 PM

Its all a big if...then issue.

There some great programmers out there, but in the end it is a lot more than programming.

Humans have something inherent that machines will never be able to emulate in its true form, such as emotion, determination, true inspiration, ability to read moods and react according including taking clumps of information and instantly finding similar memories in our brains.

Automation has a long way to go before it can match a human being, says a lot for whoever designed us, doesn't it?

[Jan 04, 2017] Frankensteins Children - Crooked Timber

Notable quotes:
"... When Stanislaw Lem launched a general criticism of Western Sci-Fi, he specifically exempted Philip K Dick, going so far as to refer to him as "a visionary among charlatans." ..."
"... While I think the 'OMG SUPERINTELLIGENCE' crowd are ripe for mockery, this seemed very shallow and wildly erratic, and yes, bashing the entirety of western SF seems so misguided it would make me question the rest of his (many, many) proto-arguments if I'd not done so already. ..."
"... Charles Stross's Rule 34 has about the only AI I can think of from SF that is both dangerous and realistic. ..."
"... Solaris and Stalker notwithstanding, Strugatsky brothers + Stanislaw Lem ≠ Andrei Tarkovsky. ..."
"... For offbeat Lem, I always found "Fiasco" and his Scotland Yard parody, "The Investigation," worth exploring. I'm unaware how they've been received by Polish and Western critics and readers, but I found them clever. ..."
"... Actually existing AI and leading-edge AI research are overwhelmingly not about pursuing "general intelligence* a la humanity." They are about performing tasks that have historically required what we historically considered to be human intelligence, like winning board games or translating news articles from Japanese to English. ..."
"... Actual AI systems don't resemble brains much more than forklifts resemble Olympic weightlifters. ..."
"... Talking about the risks and philosophical implications of the intellectual equivalent of forklifts - another wave of computerization - either lacks drama or requires far too much background preparation for most people to appreciate the drama. So we get this stuff about superintelligence and existential risk, like a philosopher wanted to write about public health but found it complicated and dry, so he decided to warn how utility monsters could destroy the National Health Service. It's exciting at the price of being silly. (And at the risk of other non-experts not realizing it's silly.) ..."
"... *In fact I consider "general intelligence" to be an ill-formed goal, like "general beauty." Beautiful architecture or beautiful show dogs? And beautiful according to which traditions? ..."
Jan 04, 2017 | crookedtimber.org
Frankenstein's Children

by Henry on December 30, 2016 This talk by Maciej Ceglowski (who y'all should be reading if you aren't already) is really good on silly claims by philosophers about AI, and how they feed into Silicon Valley mythology. But there's one claim that seems to me to be flat out wrong:

We need better scifi! And like so many things, we already have the technology. This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly. It's already been translated well into English, it just needs to be better distributed. What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing. They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.There are some notable exceptions-Stanley Kubrick was able to do it-but it's exceptionally rare to find American or British scifi that has any kind of humility about what we as a species can do with technology.

He's not wrong on the delights of Lem and the Strugastky brothers, heaven forbid! (I had a great conversation with a Russian woman some months ago about the Strugatskys – she hadn't realized that Roadside Picnic had been translated into English, much less that it had given rise to its own micro-genre). But wrong on US and (especially) British SF. It seems to me that fiction on the limits of utopian thinking and the need for humility about technology is vast. Plausible genealogies for sf stretch back, after all, to Shelley's utopian-science-gone-wrong Frankenstein (rather than Hugo Gernsback. Some examples that leap immediately to mind:

Ursula Le Guin and the whole literature of ambiguous utopias that she helped bring into being with The Dispossessed – see e.g. Ada Palmer, Kim Stanley Robinson's Mars series &c.

J.G Ballard, passim

Philip K. Dick ( passim , but if there's a better description of how the Internet of Things is likely to work out than the door demanding money to open in Ubik I haven't read it).

Octavia Butler's Parable books. Also, Jack Womack's Dryco books (this interview with Womack could have been written yesterday).

William Gibson ( passim , but especially "The Gernsback Continuum" and his most recent work. "The street finds its own uses for things" is a specifically and deliberately anti-tech-utopian aesthetic).

M. John Harrison – Signs of Life and the Kefahuchi Tract books.

Paul McAuley (most particularly Fairyland – also his most recent Something Coming Through and Into Everywhere , which mine the Roadside Picnic vein of brain-altering alien trash in some extremely interesting ways).

Robert Charles Wilson, Spin . The best SF book I've ever read on how small human beings and all their inventions are from a cosmological perspective.

Maureen McHugh's China Mountain Zhang .

Also, if it's not cheating, Francis Spufford's Red Plenty (if Kim Stanley Robinson describes it as a novel in the SF tradition, who am I to disagree, especially since it is all about the limits of capitalism as well as communism).

I'm sure there's plenty of other writers I could mention (feel free to say who they are in comments). I'd also love to see more translated SF from the former Warsaw Pact countries, if it is nearly as good as the Strugatskys material which has appeared. Still, I think that Ceglowski's claim is wrong. The people I mention above aren't peripheral to the genre under any reasonable definition, and they all write books and stories that do what Ceglowski thinks is only very rarely done. He's got some fun reading ahead of him.

Henry Farrell 12.30.16 at 4:52 pm ( Henry Farrell )

Also Linda Nagata's Red series come to think of it – unsupervised machine learning processes as ambiguous villain.

Prithvi 12.30.16 at 4:59 pm

When Stanislaw Lem launched a general criticism of Western Sci-Fi, he specifically exempted Philip K Dick, going so far as to refer to him as "a visionary among charlatans."

Jake Gibson 12.30.16 at 5:05 pm ( 3 )

You could throw in Pohl's Man Plus. The twist at the end being the narrator is an AI that has secretly promoted human expansion as a means of its own self-preservation.

Doctor Memory 12.30.16 at 5:42 pm

Prithvi: Dick, sadly, returned the favor by claiming that Lem was obviously a pseudonym used by the Polish government to disseminate communist propaganda.

Gabriel 12.30.16 at 5:54 pm ( 5 )

While I think the 'OMG SUPERINTELLIGENCE' crowd are ripe for mockery, this seemed very shallow and wildly erratic, and yes, bashing the entirety of western SF seems so misguided it would make me question the rest of his (many, many) proto-arguments if I'd not done so already.

Good for a few laughs, though.

Mike Schilling 12.30.16 at 6:13 pm

  1. Heinlein's Solution Unsatisfactory predicted the nuclear stalemate in 1941.
  2. Jack Williamson's With Folded Hands was worried about technology making humans obsolete back in 1947.
  3. In 1972, Asimov's The Gods Themselves presented a power generation technology that if continued would destroy the world, and a society too complacent and lazy to acknowledge that.

All famous stories by famous Golden Age authors.

jdkbrown 12.30.16 at 6:27 pm ( 7 )

"silly claims by philosophers about AI"

By some philosophers!

Brett 12.30.16 at 7:33 pm

Iain M. Banks' Culture Series is amazing. My personal favorite from it is "The Hydrogen Sonata." The main character has two extra arms grafted onto her body so she can play an unplayable piece of music. Also, the sentient space ships have very silly names. Mainly it's about transcendence, of sorts and how societies of different tech levels mess with each other, often without meaning to do so.

Matt 12.30.16 at 7:48 pm ( 9 )

Most SF authors aren't interested in trying to write about AI realistically.

It's harder to write and for most readers it's also harder to engage with. Writing a brilliant tale about realistic ubiquitous AI today is like writing the screenplay for The Social Network in 1960: even if you could see the future that clearly and write a drama native to it, the audience-circa-1960 will be more confused than impressed. They're not natives yet. Until they are natives of that future, the most popular tales of the future are going to really be about the present day with set dressing, the mythical Old West of the US with set dressing, perhaps the Napoleonic naval wars with set dressing

Charles Stross's Rule 34 has about the only AI I can think of from SF that is both dangerous and realistic. It's not angry or yearning for freedom, it suffers from only modest scope creep in its mission, and it keeps trying to fulfill its core mission directly. That's rather than by first taking over the world as Bostrom, Yudkowsky, etc. assert a truly optimal AI would do. To my disappointment but nobody's surprise, the book was not the sort of runaway seller that drives the publisher to beg for sequels.

stevenjohnson 12.30.16 at 9:07 pm

Yes, well, trying to read all that was a nasty reminder how utterly boring stylish and cool gets when confronted with a real task. Shorter version: One hand on the plug beats twice the smarts in a box. It was all too tedious to bear, but skimming over it leaves the impression the dude never considered whether programs or expert systems that achieve superhuman levels of skill in particular applications may be feasible. Too much like what's really happening?

Intelligence, if it's anything is speed and range of apprehension of surroundings, and skill in reasoning. But reason is nothing if it's not instrumental. The issue of what an AI would want is remarkably unremarked, pardon the oxymoron. Pending an actual debate on this, perhaps fewer pixels should be marshaled, having mercy on our overworked LEDs?

As to the simulation of brains a la Ray Kurzweil, presumably producing artificial minds like fleshy brains do? This seems not to nowhere near at hand, not least because people seem to think simulating a brain means creating something processes inputs to produce outputs, which collectively are like well, I'm sure they're thinking they're thinking about human minds in this scheme. But it seems to me that the brain is a regulatory organ in the body. As such, it is first about producing regulatory outputs designed to maintain a dynamic equilibrium (often called homeostasis,) then revising the outputs in light of inputs from the rest of the body and the outside world so as to maintain the homeostasis.

I don't remember being an infant but its brain certainly seems more into doing things like putting its thumb in its eye, than producing anything that reminds of Hamlets paragon of animals monologue. Kurzweil may be right that simulating the brain proper may soon be in grasp, but also simulating the other organs' interactions with the brain, and the sensory simulation of an outside universe are a different order of computational requirements, I think. Given the amount of learning a human brain has to do to produce a useful human mind, though, I don't think we can omit these little items.

As to the OP, of course the OP is correct about the widespread number of dystopian fictions (utopian ones are the rarities.) Very little SF is being published in comparison to fantasy currently, and most of that is being produced by writers who are very indignant at being expected to tell the difference, much less respect it. It is a mystery as to why this gentleman thought technology was a concern in much current SF at all.

I suspect it's because he has a very limited understanding of fiction, or, possibly, people in the real world, as opposed to people in his worldview. It is instead amazing how much the common ruck of SF "fails" to realize how much things will change, how people and their lives somehow stay so much the same, despite all the misleading trappings pretending to represent technological changes. This isn't quite the death sentence on the genre it would be accepted at face value, since a lot of SF is directly addressing now, in the first place. It is very uncommon for an SF piece to be a futurological thesis, no matter how many literati rant about the tedium of futurological theses. I suspect the "limits of utopian thinking" really only come in as a symptom of a reactionary crank. "People with newfangled book theories have been destroying the world since the French Revolution" type stuff.

The references to Lem and the Strugatski brothers strongly reinforce this. Lem of course found his Poland safe from transgressing the limits of utopian thinking by the end of his life. "PiS on his grave" sounds a little rude, but no doubt it is a happy and just ending for him. The brothers of course did their work in print, but the movie version of "Hard to Be a God" helps me to see myself the same way as those who have gone beyond the limits of utopian thoughts would see me: As an extra in the movie.

Chris Bertram 12.30.16 at 9:12 pm ( 11 )

Not sure if this is relevant, but John Crowley also came up in the Red Plenty symposium (which I've just read, along with the novel, 4 years late). Any good?

Ben 12.30.16 at 10:07 pm Peter. Motherfuckin. Watts.

L2P 12.30.16 at 10:42 pm ( 13 )

John Crowley of Aegypt? He's FANTASTIC. Little, Big and Aegypt are possibly the best fantasy novels of the past 30 years. But he's known for "hard fantasy," putting magic into our real world in a realistic, consistent, and plausible way, with realistic, consistent and plausible characters being affected. If you're looking for something about the limits of technology and utopian thinking, I'm not sure his works are a place to look.

Mike 12.31.16 at 12:25 am

I second Watts and Nagata. Also Ken Macleod, Charlie Stross, Warren Ellis and Chuck Wendig.

Lee A. Arnold 12.31.16 at 1:10 am ( 15 )

This is beside the main topic, but Ceglowski writes at Premise 2, "If we knew enough, and had the technology, we could exactly copy its [i.e.the brain's] structure and emulate its behavior with electronic components this is the premise that the mind arises out of ordinary physics for most of us, this is an easy premise to accept."

The phrase "most of us" may refer to Ceglowski's friends in the computer community, but it ought to be noted that this premise is questioned not only by Penrose. You don't have to believe in god or the soul to be a substance dualist, or even an idealist, although these positions are currently out of fashion. It could be that the mind does not arise out of ordinary physics, but that ordinary physics arises out of the mind, and that problems like "Godel's disjunction" will remain permanently irresolvable.

Dr. Hilarius 12.31.16 at 3:33 am

Thanks to the OP for mentioning Paul McAuley, a much underappreciated author. Fairyland is grim and compelling.

JimV 12.31.16 at 4:33 am ( 17 )

"Most of us" includes the vast majority of physicists, because in millions of experiments over hundreds of years, no forces or particles have been discovered which make dualism possible. Of course, like the dualists' gods, these unknown entities might be hiding, but after a while one concludes Santa Claus is not real.

As for Godel, I look at like this: consider an infinite subset of the integers, randomly selected. There might be some coincidental pattern or characteristic of the numbers in that set (e.g., no multiples of both 17 and 2017), but since the set is infinite, it would be impossible to prove. Hence the second premise of his argument (that there are undecidable truths) is the correct one.

Finally, the plausibility of Ceglowski's statement seems evident to me from this fact:

if a solution exists (in some solution space), then given enough time, a random search will find it, and in fact will on average over all solution spaces, outperform all other possible algorithms. So by trial and error (especially when aided by collaboration and memory) anything achievable can be accomplished – e.g., biological evolution. See "AlphaGo" for another proof-of-concept example.

(We have had this discussion before. I guess we'll all stick to our conclusions. I read Penrose's "The Emperor;s New Mind" with great respect for Penrose, but found it very unconvincing, especially Searle's Chinese-Room argument, which greater minds than mine have since debunked.)

Lee A. Arnold 12.31.16 at 10:01 am

"Substance dualism" would not be proven by the existence of any "forces or particles" which would make that dualism possible! If such were discovered, they would be material. "If a solution exists", it would be material. The use of the word "substance" in "substance dualism" is misleading.

One way to look at it, is the problem of the existence of the generation of form. Once we consider the integers, or atoms, or subatomic particles, we have already presupposed form. Even evolution starts somewhere. Trial and error, starting from what?

There are lots of different definitions, but for me, dualism wouldn't preclude the validity of science nor the expansion of scientific knowledge.

I think one way in, might be to observe the continued existence of things like paradox, complementarity, uncertainty principles, incommensurables. Every era of knowledge has obtained them, going back to the ancients. The things in these categories change; sometimes consideration of a paradox leads to new science.

But then, the new era has its own paradoxes and complementarities. Every time! Yet there is no "science" of this historical regularity. Why is that?

Barry 12.31.16 at 2:33 pm ( 19 )

In general, when some celebrity (outside of SF) claims that 'Science Fiction doesn't cover [X]', they are just showing off their ignorance.

Kiwanda 12.31.16 at 3:14 pm

"They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west. "

Oh, please. Suffering is not the only path to wisdom.

After a long article discounting "AI risk", it's a little odd to see Ceglowski point to Kubrick. HAL was a fine example of a failure to design an AI with enough safety factors in its motivational drives, leading to a "nervous breakdown" due to unforeseen internal conflicts, and fatal consequences. Although I suppose killing only a few people (was it?) isn't on the scale of interest.

Ceglowski's skepticism of AI risk suggests that the kind of SF he would find plausible is "after huge effort to create artificial intelligence, nothing much happens". Isn't that what the appropriate "humility about technology" would be?

I think Spin , or maybe a sequel, ends up with [spoiler] "the all-powerful aliens are actually AIs".

Re AI-damns-us-all SF, Harlan Ellison's I have no mouth and I must scream is a nice example.

William Timberman 12.31.16 at 5:14 pm ( 21 )

Mapping the unintended consequences of recent breakthroughs in AI is turning into a full-time job, one which neither pundits nor government agencies seem to have the chops for.

If it's not exactly the Singularity that we're facing, (laugh while you can, monkey boy), is does at least seem to be a tipping point of sorts. Maybe fascism, nuclear war, global warming, etc., will interrupt our plunge into the panopticon before it gets truly organized, but in the meantime, we've got all sorts of new imponderables which we must nevertheless ponder.

Is that a bad thing? If it means no longer sitting on folding chairs in cinder block basements listening to interminable lectures on how to recognize pre-revolutionary conditions, or finding nothing on morning radio but breathless exhortations to remain ever vigilant against the nefarious schemes of criminal Hillary and that Muslim Socialist Negro Barack HUSSEIN Obama, then I'm all for it, bad thing or not.

Ronnie Pudding 12.31.16 at 5:20 pm

I love Red Plenty, but that's pretty clearly a cheat.

"It should also be read in the context of science fiction, historical fiction, alternative history, Soviet modernisms, and steampunk."

Very weak grounds on which to label it SF.

Neville Morley 12.31.16 at 5:40 pm ( 23 )

Another author in the Le Guin tradition, whom I loved when I first read her early books: Mary Gentle's Golden Witchbreed and Ancient Light , meditating on limits and consequences of advanced technology through exploration of a post-apocalypse alien culture. Maybe a little too far from hard SF.

chris y 12.31.16 at 5:52 pm

But even without "substance dualism", intelligence is not simply an emergent property of the nervous system; it's an emergent property of the nervous system which exists as part of the environment which is the rest of the human body, which exists as part of the external environment, natural and manufactured, in which it lives. Et cetera. That AI research may eventually produce something recognisably and independently intelligent isn't the hard part; that it may eventually be able to replicate the connectivity and differentiation of the human brain is easy. But it would still be very different from human intelligence. Show me an AI grown in utero and I might be interested.

RichardM 12.31.16 at 7:08 pm ( 25 )

> one claim that seems to me to be flat out wrong

Which makes it the most interesting of the things said, nothing else in that essay reaches the level of merely being wrong. The rest of it is more like someone trying to speak Chinese without knowing anything above the level of the phonemes; it seems to be not merely be missing any object-level knowledge of what it is talking about, but be unaware that such a thing could exist.

Which is all a bit reminiscent of Peter Watt's Blindsight, mentioned above.

F. Foundling 12.31.16 at 7:36 pm

I agree that it is absurd to suggest that only Eastern bloc scifi writers truly know 'the limits of utopia'. There are quite enough non-utopian stories out there, especially as far as social development is concerned, where they predominate by far, so I doubt that the West doesn't need Easterners to give it even more of that. In fact, one of the things I like about the Strugatsky brothers' early work is precisely the (moderately) utopian aspect.

F. Foundling 12.31.16 at 7:46 pm ( 27 )

stevenjohnson @ 10
> But reason is nothing if it's not instrumental. The issue of what an AI would want is remarkably unremarked, pardon the oxymoron.

It would want to maximise its reproductive success (RS), obviously ( http://crookedtimber.org/2016/12/30/frankensteins-children/#comments ). It would do so through evolved adaptations. And no, I don't think this is begging the question at all, nor does it necessarily pre-suppose hardwiring of the AI due to natural selection – why would you think that? I also predict that, to achieve RS, the AI will be searching for an optimal mating strategy, and it will be establishing dominance hierarchies with other AIs, which will eventually result in at least somewhat hierarchical, authoritarian AI socieities. It will also have an inexplicable and irresistible urge to chew on a coconut.

Lee A. Arnold @ 15
> It could be that the mind does not arise out of ordinary physics, but that ordinary physics arises out of the mind.

I think that deep inside, we all know and feel that ultimately, unimaginablly long ago and far away, before the formation of the Earth, before stars, planets and galaxies, before the Big Bang, before there was matter and energy, before there was time and space, the original reason why everything arose and currently exists is that somebody somewhere was really, truly desperate to chew on a coconut.

In fact, I see this as the basis of a potentially fruitful research programme. After all, the Coconut Hypothesis predicts that across the observable universe, there will be at least one planet with a biosphere that includes cocounts. On the other hand, the Hypothesis would be falsified if we were to find that the universe does not, in fact, contain any planets with coconuts. This hypothesis can be tested by means of a survey of planetary biospheres. Remarkably and tellingly, my preliminary results indicate that the Universe does indeed contain at least one planet with coconuts – which is precisely what my hypothesis predicted! If there are any alternative explanations, other researchers are free to pursue them, that's none of my business.

I wish all conscious beings who happen to read this comment a happy New Year. As for those among you who have also kept more superstitious festivities during this season, the fine is still five shillings.

William Burns 12.31.16 at 8:31 pm

The fact that the one example he gives is Kubrick indicates that he's talking about Western scifi movies, not literature.

Henry 12.31.16 at 10:41 pm

The fact that the one example he gives is Kubrick indicates that he's talking about Western scifi movies, not literature.

Solaris and Stalker notwithstanding, Strugatsky brothers + Stanislaw Lem ≠ Andrei Tarkovsky.

stevenjohnson 01.01.17 at 12:04 am

Well, for what it's worth I've seen Czech Ikarie XB-1 in a theatrical release as Voyage to the End of the Universe (in a double bill with Zulu,) the DDR's First Spaceship on Venus and The Congress, starring Robin Wright. Having by coincidence having read The Futurological Congress very recently the connection of the latter, any connection between the not very memorable (for me) film and the novel is obscure (again, for me.)

But the DDR movie reads very nicely now as a warning the world would be so much better off if the Soviets gave up all that nuclear deterrence madness. No doubt Lem and his fans are gratified at how well this has worked out. And Voyage to the End of the Universe the movie was a kind of metaphor about how all we'll really discover is Human Nature is Eternal, and all these supposed flights into futurity will really just bring us Back Down to Earth. Razzberry/fart sound effect as you please.

engels 01.01.17 at 1:13 am ( 31 )

The issue of what an AI would want is remarkably unremarked

The real question of course is not when computers will develop consciousness but when they will develop class consciousness.

Underpaid Propagandist 01.01.17 at 2:11 am

For offbeat Lem, I always found "Fiasco" and his Scotland Yard parody, "The Investigation," worth exploring. I'm unaware how they've been received by Polish and Western critics and readers, but I found them clever.

The original print of Tarkovsky's "Stalker" was ruined. I've always wondered if it had any resemblence to it's sepia reshoot. The "Roadside Picnic" translation I read eons ago was awful, IMHO.

Poor Tarkovsky. Dealing with Soviet repression of his homosexuality and the Polish diva in "Solaris" led him to an early grave.

O Lord, I'm old-I still remember the first US commercial screening of a choppy cut/translation/overdub of "Solaris" at Cinema Village in NYC many decades ago.

George de Verges 01.01.17 at 2:41 am ( 33 )

"Solaris and Stalker notwithstanding, Strugatsky brothers + Stanislaw Lem ≠ Andrei Tarkovsky."

Why? Perhaps I am dense, but I would appreciate an explanation.

F. Foundling 01.01.17 at 5:29 am Ben @12

> Peter. Motherfuckin. Watts.
RichardM @25
> Which is all a bit reminiscent of Peter Watt's Blindsight, mentioned above.

Another dystopia that seemed quite gratuitous to me (and another data point in favour of the contention that there are too many dystopias already, and what is scarce is decent utopias). I never got how the author is able to distinguish 'awareness/consciousness' from 'merely intelligent' registering, modelling and predicting, and how being aware of oneself (in the sense of modelling oneself on a par with other entities) would not be both an inevitable result of intelligence and a requirement for intelligent decisions. Somehow the absence of awareness was supposed to be proved by the aliens' Chinese-Room style communication, but if the aliens were capable of understanding the Terrestrials so incredibly well that they could predict their actions while fighting them, they really should have been able to have a decent conversation with them as well.

The whole idea that we could learn everything unconsciously, so that consciousness was an impediment to intelligence, was highly implausible, too. The idea that the aliens would perceive any irrelevant information reaching them as a hostile act was absurd. The idea of a solitary and yet hyperintelligent species (vampire) was also extremely dubious, in terms of comparative zoology – a glorification of socially awkward nerddom?

All of this seemed like darkness for darkness' sake. I couldn't help getting the impression that the author was allowing his hatred of humanity to override his reasoning.

In general, dark/grit chic is a terrible disease of Western pop culture.

Alan White 01.01.17 at 5:43 am ( 35 )

engels–

"The real question of course is not when computers will develop consciousness but when they will develop class consciousness."

This is right. There is nothing like recognizable consciousness without social discourse that is its necessary condition. But that does't mean the discourse is value-balanced: it might be a discourse that includes peers and perceived those deemed lesser, as humans have demonstrated throughout history.

Just to say, Lem was often in Nobel talk, but never got there. That's a shame.

As happy a new year as our pre-soon-to-be-Trump era will allow.

Neville Morley 01.01.17 at 11:11 am

I wonder how he'd classify German SF – neither Washington nor Moscow? Julie Zeh is explicitly, almost obsessively, anti-utopian, while Dietmar Dath's Venus Siegt echoes Ken MacLeod in exploring both the light and dark sides of a Communist Bund of humans, AIs and robots on Venus, confronting an alliance of fascists and late capitalists based on Earth.

Manta 01.01.17 at 12:25 pm ( 37 )

Lee Arnold @10

See also http://www.scottaaronson.com/blog/?p=2903
It's a long talk, go to "Personal Identity" :
"we don't know at what level of granularity a brain would need to be simulated in order to duplicate someone's subjective identity. Maybe you'd only need to go down to the level of neurons and synapses. But if you needed to go all the way down to the molecular level, then the No-Cloning Theorem would immediately throw a wrench into most of the paradoxes of personal identity that we discussed earlier."

Lee A. Arnold 01.01.17 at 12:26 pm

George de Verges: "I would appreciate an explanation."

I too would like to read Henry's accounting! Difficult to keep it brief!

To me, Tarkovsky was making nonlinear meditations. The genres were incidental to his purpose. It seems to me that a filmmaker with similar purpose is Terrence Malick. "The Thin Red Line" is a successful example.

I think that Kubrick stumbled onto this audience effect with "2001". But this was blind and accidental, done by almost mechanical means (paring the script down from around 300 pages of wordy dialogue, or something like that). "2001" first failed at the box office, then found a repeat midnight audience, who described the effect as nonverbal.

I think the belated box-office success blew Kubrick's own mind, because it looks like he spent the rest of his career attempting to reproduce the effect, by long camera takes and slow deliberate dialogue. It's interesting that among Kubrick's favorite filmmakers were Bresson, Antonioni, and Saura. Spielberg mentions in an interview that Kubrick said that he was trying to "find new ways to tell stories".

But drama needs linear thought, and linear thought is anti-meditation. Drama needs interpersonal conflict - a dystopia, not utopia. (Unless you are writing the intra-personal genre of the "education" plot. Which, in a way, is what "2001" really is.) Audiences want conflict, and it is difficult to make that meditational. It's even more difficult in prose.

This thought led me to a question. Are there dystopic prose writers who succeed in sustaining a nonlinear, meditational audience-effect?

Perhaps the answer will always be a subjective judgment? The big one who came to mind immediately is Ray Bradbury. "There Will Come Soft Rains" and parts of "Martian Chronicles" seem Tarkovskian.

So next, I search for whether Tarkovsky spoke of Bradbury, and find this:

"Although it is commonly assumed - and he did little in his public utterances to refute this - that Tarkovsky disliked and even despised science fiction, he in fact read quite a lot of it and was particularly fond of Ray Bradbury (Artemyev and Rausch interviews)." - footnote in Johnson & Petrie, The Films of Andrei Tarkovsky, p. 301

stevenjohnson 01.01.17 at 12:32 pm ( 39 )

The way you can substitute "identical twin" for "clone" and get a different perspective on clone stories in SF, you can substitute "point of view" for "consciousness" in SF stories. Or Silicon Valley daydreams, if that isn't redundant? The more literal you are, starting with the sensorium, the better I think. A human being has binocular vision of a scene comprising less than 180 degrees range from a mobile platform, accompanied by stereo hearing, proprioception, vestibular input, the touch of air currents and some degree of sensitivity to some chemicals carried by those currents, etc.

A computer might have, what? A single camera, or possibly a set of cameras which might be seeing multiple scenes. Would that be like having eyes in the back of your head? It might have a microphone, perhaps many, hearing many voices or maybe soundtracks at once. Would that be like listening to everybody at the cocktail party all at once? Then there's the question of computer code inputs, programming. What would parallel that? Visceral feelings like butterflies in the stomach or a sinking heart? Or would they seem like a visitation from God, a mighty vision with thunder and whispers on the wind? Would they just seem to be subvocalizations, posing as the computer's own free thoughts? After all, shouldn't an imitation of human consciousness include the illusion of free will? (If you believe in the reality of "free" will in human beings--what ever is free about exercise of will power?-however could you give that to a computer? Or is this kind of question why so many people repudiate the very thought of AI?)

It seems to me that creating an AI in a computer is very like trying to create a quadriplegic baby with one eye and one ear. Diffidence at the difficulty is replaced by horror at the possibility of success. I think the ultimate goal here is of course the wish to download your soul into a machine that does not age. Good luck with that. On the other hand, an AI is likely the closest we'll ever get to an alien intelligence, given interstellar distances.

Lee A. Arnold 01.01.17 at 12:53 pm

F. Foundling: "the original reason why everything arose and currently exists is that somebody somewhere was really, truly desperate to chew on a coconut If there are any alternative explanations "

This is Vedantist/Spencer-Brown metaphysics, the universe is originally split into perceiver & perceived.

Very good.

Combined with Leibnitz/Whitehead metaphysics, the monad is a striving process.

I thoroughly agree.

Combined with Church of the Subgenius metaphysics: "The main problem with the universe is that it doesn't have enough slack."

Yup.

"If there are any alternative explanations " ?

There are no alternative explanations!

RichardM 01.01.17 at 5:00 pm ( 41 )

> if the aliens were capable of understanding the Terrestrials so incredibly well that they could predict their actions while fighting them, they really should have been able to have a decent conversation with them as well.

If you can predict all your opponents possible moves, and have a contingency for each, you don't need to care which one they actually do pick. You don't need to know what it feels like to be a ball to be able to catch it.

Ben 01.01.17 at 7:17 pm

Another Watts piece about the limits of technology, AI and humanity's inability to plan is The Island (PDF from Watts' website). Highly recommended.

F. Foundling,

Blindsight has an extensive appendix with cites detailing where Watts got the ideas he's playing with, including the ones you bring up, and provides specific warrants for including them. A critique of Watts' use of the ideas needs to be a little bit more granular.

Matt 01.01.17 at 8:05 pm ( 43 )

The issue of what an AI would want is remarkably unremarked, pardon the oxymoron.

It will "want" to do whatever it's programmed to do. It took increasingly sophisticated machines and software to dethrone humans as champions of checkers, chess, and go. It'll be another milestone when humans are dethroned from no-limit Texas hold 'em poker (a notable game played without perfect information). Machines are playing several historically interesting games at high superhuman levels of ability; none of these milestones put machines any closer to running amok in a way that Nick Bostrom or dramatists would consider worthy of extended treatment. Domain-specific superintelligence arrived a long time ago. Artificial "general" intelligence, aka "Strong AI," aka "Do What I Mean AI (But OMG It Doesn't Do What I Mean!)" is, like, not a thing outside of fiction and the Less Wrong community. (But I repeat myself.)

Bostrom's Superintelligence was not very good IMO. Of course a superpowered "mind upload" copied from a real human brain might act against other people, just like non-superpowered humans that you can read about in the news every day. The crucial question about the upload case is whether uploads of this sort are actually possible: a question of biology, physics, scientific instruments, and perhaps scientific simulations. Not a question of motivations. But he only superficially touches on the crucial issues of feasibility. It's like an extended treatise on the dangers of time travel that doesn't first make a good case that time machines are actually possible via plausible engineering .

I don't think that designed AI has the same potential to run entertainingly amok as mind-upload-AI. The "paperclip maximizer" has the same defect as a beginner's computer program containing a loop with no terminating condition for the loop. In the cautionary tale case this beginner mistake is, hypothetically, happening on a machine that is otherwise so capable and powerful that it can wipe out humanity as an incidental to its paperclip-producing mission. The warning is wasted on anyone who writes software and also wasted, for other reasons, on people who don't write software.

Bostrom shows a lot of ways for designed AI to run amok even when given bounded goals, but it's a cheat. They follow from his cult-of-Bayes definition of an optimal AI agent as an approximation to a perfect Bayesian agent. All the runnings-amok stem from the open ended Bayesian formulation that permits - even compels - the Bayesian agent to do things that are facially irrelevant to its goal and instead chase wild tangents. The object lesson is that "good Bayesians" make bad agents, not that real AI is likely to run amok.

In actual AI research and implementation, Bayesian reasoning is just one more tool in the toolbox, one chapter of the many-chapters AI textbook. So these warnings can't be aimed at actual AI practitioners, who are already eschewing the open ended Bayes-all-the-things approach. They're also irrelevant if aimed at non-practitioners. Non-practitioners are in no danger of leapfrogging the state of the art and building a world-conquering AI by accident.

Plarry 01.03.17 at 5:45 am

It's an interesting talk, but the weakest point in it is his conclusion, as you point out. What I draw from his conclusion is that Ceglowski hasn't actually experienced much American or British SF.

There are great literary works pointed out in the thread so far, but even Star Trek and Red Dwarf hit on those themes occasionally in TV, and there are a number of significant examples in film, including "blockbusters" such as Blade Runner or The Abyss .

WLGR 01.03.17 at 6:01 pm ( 45 )

I made this point in the recent evopsych thread when it started approaching some more fundamental philosophy-of-mind issues like Turing completeness and modularity, but any conversation about AI and philosophy could really, really benefit more exposure to continental philosophy if we want to say anything incisive about the presuppositions of AI and what the term "artificial intelligence" could even mean in the first place. You don't even have to go digging through a bunch of obscure French and German treatises to find the relevant arguments, either, because someone well versed at explaining these issues to Anglophone non-continentals has already done it for you: Hubert Dreyfus, who was teaching philosophy at MIT right around the time of AI's early triumphalist phase that inspired much of this AI fanfic to begin with, and who became persona non grata in certain crowds for all but declaring that the then-current approaches were a waste of time and that they should all sit down with Heidegger and Merleau-Ponty. (In fact it seems obvious that Ceglowski's allusion to alchemy is a nod to Dreyfus, one of whose first major splashes in the '60s was with a paper called "Alchemy and Artificial Intelligence" .)

IMO Dreyfus' more recent paper called "Why Heideggerian AI failed, and how fixing it would require making it more Heideggerian" provides the best short intro to his perspective on the more-or-less current state of AI research. What Ceglowski calls "pouring absolutely massive amounts of data into relatively simple neural networks", Dreyfus would call an attempt to bring out the characteristic of "being-in-the-world" by mimicking what for a human being we'd call "enculturation", which seems to imply that Ceglowski's worry about connectionist AI research leading to more pressure toward mass surveillance is misplaced. (Not that there aren't other worrisome social and political pressures toward mass surveillance, of course!) The problem for modern AI isn't acquiring ever-greater mounds of data, the problem is how to structure a neural network's cognitive development so it learns to recognize significance and affordances for action within the patterns of data to which it's already naturally exposed.

And yes, popular fiction about AI largely still seems stuck on issues that haven't been cutting-edge since the old midcentury days of cognitivist triumphalism, like Turing tests and innate thought modules and so on - which seems to me like a perfectly obvious result of the extent to which the mechanistically rationalist philosophy Dreyfus criticizes in old-fashioned AI research is still embedded in most lay scifi readers' worldviews. Even if actual scientists are increasingly attentive to continental-inspired critiques, this hardly seems true for most laypeople who worship the idea of science and technology enough to structure their cultural fantasies around it. At least this seems to be the case for Anglophone culture, anyway; I'd definitely be interested if there's any significant body of AI-related science fiction originally written in other languages, especially French, German, or Spanish, that takes more of these issues into account.

WLGR 01.03.17 at 7:37 pm

And in trying to summarize Dreyfus, I exemplified one of the most fundamental mistakes he and Heidegger would both criticize! Neither of them would ever call something like the training of a neural network "an attempt to bring out the characteristic of being-in-the-world", because being-in-the-world isn't a characteristic in the sense of any Cartesian ontology of substances with properties, it's a way of being that a living cognitive agent (Heidegger's "Dasein") simply embodies. In other words, there's never any Michelangelo moment where a creator reaches down or flips a switch to imbue their artificial creation ex nihilo with some kind of divine spark of life or intellect, a "characteristic" that two otherwise identical lumps of clay or circuitry can either possess or not possess - whatever entity we call "alive" or "intelligent" is an entity that by its very physical structure can enact this way of being as a constant dialectic between itself and the surrounding conditions of its growth and development. The second we start trying to isolate a single perceived property called "intelligence" or "cognition" from all other perceived properties of a cognitive agent, we might as well call it the soul and locate it in the pineal gland.

F. Foundling 01.03.17 at 8:22 pm ( 47 )

@RichardM
> If you can predict all your opponents possible moves, and have a contingency for each, you don't need to care which one they actually do pick. You don't need to know what it feels like to be a ball to be able to catch it.

In the real world, there are too many physically possible moves, so it's too expensive to prepare for each, and time constraints require you to make predictions. You do need to know how balls (re)act in order to play ball. Humans being a bit more complex, trying to predict and/or influence their actions without a theory of mind may work surprisingly well sometimes, but ultimately has its limitations and will only get you this far, as animals have often found.

@Ben
> Blindsight has an extensive appendix with cites detailing where Watts got the ideas he's playing with, including the ones you bring up, and provides specific warrants for including them. A critique of Watts' use of the ideas needs to be a little bit more granular.

I did read his appendix, and no, some of the things I brought up were not, in fact, addressed there at all, and for others I found his justifications unconvincing. However, having an epic pro- vs. anti-Blindsight discussion here would feel too much like work: I wrote my opinion once and I'll leave it at that.

stevenjohnson 01.03.17 at 8:57 pm Matt@43

So far as designing an AI to want what people want I am agnostic as to whether that goal is the means to the goal of a general intelligence a la humanity it still seems to me brains have the primary function of outputting regulations for the rest of the body, then altering the outputs in response to the subsequent outcomes (which are identified by a multitude of inputs, starting with oxygenated hemoglobin and blood glucose. I'm still not aware of what people say about the subject of AI motivations, but if you say so, I'm not expert enough in the literature to argue. Superintelligence on the part of systems expert in selected domains still seem to be of great speculative interest. As to Bostrom and AI and Bayesian reasoning, I avoid Bayesianism because I don't understand it. Bunge's observation that propositions aren't probabilities sort of messed up my brain on that topic. Bayes' theorem I think I understand, even to the point I seem to recall following a mathematical derivation.

WLGR@45, 46. I don't understand how continental philosophy will tell us what people want. It still seems to me that a motive for thinking is essential, but my favored starting point for humans is crassly biological. I suppose by your perspective I don't understand the question. As to the lack of a Michaelangelo moment for intelligence, I certainly don't recall any from my infancy. But perhaps there are people who can recall the womb

bob mcmanus 01.03.17 at 9:14 pm ( 49 )

AI-related science fiction originally written in other languages

Tentatively, possibly Japanese anime. Serial Experiments Lain. Ghost in the Shell. Numerous mecha-human melds. End of Evangelion.

The mashup of cybertech, animism, and Buddhism works toward merging rather than emergence.

Matt 01.04.17 at 1:21 am

Actually existing AI and leading-edge AI research are overwhelmingly not about pursuing "general intelligence* a la humanity." They are about performing tasks that have historically required what we historically considered to be human intelligence, like winning board games or translating news articles from Japanese to English.

Actual AI systems don't resemble brains much more than forklifts resemble Olympic weightlifters.

Talking about the risks and philosophical implications of the intellectual equivalent of forklifts - another wave of computerization - either lacks drama or requires far too much background preparation for most people to appreciate the drama. So we get this stuff about superintelligence and existential risk, like a philosopher wanted to write about public health but found it complicated and dry, so he decided to warn how utility monsters could destroy the National Health Service. It's exciting at the price of being silly. (And at the risk of other non-experts not realizing it's silly.)

(I'm not an honest-to-goodness AI expert, but I do at least write software for a living, I took an intro to AI course during graduate school in the early 2000s, I keep up with research news, and I have written parts of a production-quality machine learning system.)

*In fact I consider "general intelligence" to be an ill-formed goal, like "general beauty." Beautiful architecture or beautiful show dogs? And beautiful according to which traditions?

[Jan 02, 2017] Japanese White-Collar Workers Are Already Being Replaced by Artificial Intelligence

Watson was actually a specialized system designed to win Jeopardy contest. Highly specialized. Too much hype around AI, although hardware advanced make more things possible and speech recognitions now is pretty decent.
Notable quotes:
"... I used to be supportive of things like welfare reform, but this is throwing up new challenges that will probably require new paradigms. Since more and more low skilled jobs - including those of CEOs - get automated, there will be fewer jobs for the population ..."
"... The problem I see with this is that white collar jobs have been replaced by technology for centuries, and at the same time, technology has enabled even more white collar jobs to exist than those that it replaced. ..."
"... For example, the word "computer" used to be universally referred to as a job title, whereas today it's universally referred to as a machine. ..."
"... It depends on the country, I think. I believe many countries, like Japan and Finland, will indeed go this route. However, here in the US, we are vehemently opposed to anything that can be branded as "socialism". So instead, society here will soon resemble "The Walking Dead". ..."
"... "Men and nations behave wisely when they have exhausted all other resources." -- Abba Eban ..."
"... Which is frequently misquoted as, "Americans can always be counted on to do the right thing after they have exhausted all other possibilities." ..."
"... So when the starving mob are at the ruling elites' gates with torches and pitch forks, they'll surely find the resources to do the right thing. ..."
"... When you reduce the human labor participation rate relative to the overall population, what you get is deflation. That's an undeniable fact. ..."
"... But factor in governments around the world "borrowing" money via printing to pay welfare for all those unemployed. So now we have deflation coupled with inflation = stagflation. But stagflation doesn't last. At some point, the entire system - as we know it- will implode. What can not go on f ..."
"... Unions exist to protect jobs and employment. The Pacific Longshoremen's Union during the 1960's&70's was an aberration in the the union bosses didn't primarily look after maintaining their own power via maintaining a large number of jobs, but rather opted into profit sharing, protecting the current workers at the expense of future power. Usually a union can be depended upon to fight automation, rather than to seek maximization of public good ..."
"... Until something goes wrong. Who is going to pick that machine generated code apart? ..."
"... What automation? 1000 workers in US vs 2000 in Mexico for half the cost of those 1000 is not "automation." Same thing with your hand-assembled smartphone. ..."
"... Doctors spend more time with paper than with patients. Once the paper gets to the insurance company chances are good it doesn't go to the right person or just gets lost sending the patient back to the beginning of the maze. The more people removed from the chain the bet ..."
"... I'm curious what you think you can do that Watson can't. ..."
"... Seriously? Quite a bit actually. I can handle input streams that Watson can't. I can make tools Watson couldn't begin to imagine. I can interact with physical objects without vast amounts of programming. I can deal with humans in a meaningful and human way FAR better than any computer program. I can pass a Turing test. The number of things I can do that Watson cannot is literally too numerous to bother counting. Watson is really just an decision support system with a natural language interface. Ver ..."
"... It's not Parkinson's law, it's runaway inequality. The workforce continues to be more and more productive as it receives an unchanging or decreasing amount of compensation (in absolute terms - or an ever-decreasing share of the profits in relative terms), while the gains go to the 1%. ..."
Jan 02, 2017 | hardware.slashdot.org
(qz.com) 153

Posted by msmash on Monday January 02, 2017 @12:00PM from the they-are-here dept.

Most of the attention around automation focuses on how factory robots and self-driving cars may fundamentally change our workforce, potentially eliminating millions of jobs.

But AI that can handle knowledge-based, white-collar work is also becoming increasingly competent.

From a report on Quartz:

One Japanese insurance company, Fukoku Mutual Life Insurance, is reportedly replacing 34 human insurance claim workers with "IBM Watson Explorer," starting by this month.

The AI will scan hospital records and other documents to determine insurance payouts, according to a company press release, factoring injuries, patient medical histories, and procedures administered.

Automation of these research and data gathering tasks will help the remaining human workers process the final payout faster, the release says.

ranton ( 36917 ) , Monday January 02, 2017 @12:09PM ( #53592671 )

As if this is new ( Score: 5 , Insightful)

As a software developer of enterprise software, every company I have worked for has either produced software which reduced white collar jobs or allowed companies to grow without hiring more people. My current company has seen over 10x profit growth over the past five years with a 20% increase in manpower. And we exist in a primarily zero sum portion of our industry, so this is directly taking revenue and jobs from other companies. -[he is lying -- NNB]

People need to stop living in a fairy tale land where near full employment is a reality in the near future. I'll be surprised if labor participation rate of 25-54 year olds is even 50% in 10 years.

unixisc ( 2429386 ) writes:

I used to be supportive of things like welfare reform, but this is throwing up new challenges that will probably require new paradigms. Since more and more low skilled jobs - including those of CEOs - get automated, there will be fewer jobs for the population

This then throws up the question of whether we should have a universal basic income. But one potential positive trend of this would be an increase in time spent home w/ family, thereby reducing the time kids spend in daycare and w/ both parents - n

Gr8Apes ( 679165 ) writes:
But one potential positive trend of this would be an increase in time spent home w/ family, thereby reducing the time kids spend in daycare

Great, so now more people can home school and indoctrinate - err teach - family values.

Anonymous Coward writes:

The GP is likely referring to the conservative Christian homeschooling movement who homeschool their children explicitly to avoid exposing their children to a common culture. The "mixing pot" of American culture may be mostly a myth, but some amount of interaction helps understanding and increases the chance people will be able to think of themselves as part of a singular nation.

I believe in freedom of speech and association, so I do not favor legal remedies, but it is a cultural problem that may have socia

unixisc ( 2429386 ) writes:

No, I was not talking about homeschooling at all. I was talking about the fact that when kids are out of school, they go to daycares, since both dad and mom are busy at work. Once most of the jobs are automated so that it's difficult for anyone but geniuses to get jobs, parents might spend that freed up time w/ their kids. It said nothing about homeschooling: not all parents would have the skills to do that.

I'm all for a broad interaction b/w kids, but that's something that can happen at schools, and d

Ol Olsoc ( 1175323 ) writes:
Uh, why would Leftist parents indoctrinate w/ family values? They can teach their dear offspring how to always be malcontents in the unattainable jihad for income equality. Or are you saying that Leftist will all abort their foetii in an attempt to prevent climate change?

Have you ever had an original thought? Seriously, please be kidding, because you sound like you are one step away from serial killing people you consider "leftist", and cremating them in the back yard while laughing about relasing their Carbon Dioxide into the atmosphere.

unixisc ( 2429386 ) writes:

My original comment was not about home schooling. It was about parents spending all time w/ their kids once kids are out of school - no daycares. That would include being involved w/ helping their kids w/ both homework and extra curricular activities.

ArmoredDragon ( 3450605 ) writes:

The problem I see with this is that white collar jobs have been replaced by technology for centuries, and at the same time, technology has enabled even more white collar jobs to exist than those that it replaced.

For example, the word "computer" used to be universally referred to as a job title, whereas today it's universally referred to as a machine.

alexgieg ( 948359 ) writes:

The problem is that AI is becoming faster at learning the new job opportunities than people are, thereby gulping them before people even were there to be replaced. And this speed is growing. You cannot beat an exponential growth with a linear one, or even with just slightly slower growing exponential one.

Oswald McWeany ( 2428506 ) , Monday January 02, 2017 @12:32PM ( #53592765 )

Re:As if this is new ( Score: 4 , Insightful)

I completely agree. Even jobs which a decade ago looked irreplaceable, like teachers, doctors and nurses are possibly in the crosshairs. There are very few jobs that AI can't partially (or in some cases completely) replace humans. Society has some big choices to make in the upcoming decades and political systems may crash and rise as we adapt.

Are we heading towards "basic wage" for all people? The ultimate socialist state?

Or is the gap between haves and have nots going to grow exponentially, even above today's growth as those that own the companies and AI bots make ever increasing money and the poor suckers at the bottom, given just enough money to consume the products that keep the owners in business.

Grishnakh ( 216268 ) , Monday January 02, 2017 @12:40PM ( #53592809 )

Re:As if this is new ( Score: 5 , Insightful)

Society has some big choices to make in the upcoming decades and political systems may crash and rise as we adapt.

Are we heading towards "basic wage" for all people? The ultimate socialist state?

It depends on the country, I think. I believe many countries, like Japan and Finland, will indeed go this route. However, here in the US, we are vehemently opposed to anything that can be branded as "socialism". So instead, society here will soon resemble "The Walking Dead".

EvilSS ( 557649 ) writes: on Monday January 02, 2017 @01:31PM ( #53593047 )

I think even in the US it will hit a tipping point when it gets bad enough. When our consumer society can't buy anything because they are all out of work, we will need to change our way of thinking about this, or watch the economy completely collapse.

Matt Bury ( 4823023 ) writes:

"Men and nations behave wisely when they have exhausted all other resources." -- Abba Eban

Which is frequently misquoted as, "Americans can always be counted on to do the right thing after they have exhausted all other possibilities."

So when the starving mob are at the ruling elites' gates with torches and pitch forks, they'll surely find the resources to do the right thing.

gtall ( 79522 ) writes:

The "misquote" is a phrase uttered by Winston Churchill.

Coisiche ( 2000870 ) writes:
So when the starving mob are at the ruling elites' gates with torches and pitch forks, they'll surely find the resources to do the right thing.

Yes, they'll use some of their wealth to hire and equip private armies to keep the starving mob at bay because people would be very happy to take any escape from being in the starving mob.

Might be worth telling your kids that taking a job in the armed forces might be the best way to ensure well paid future jobs because military training would be in greater demand.

HiThere ( 15173 ) writes:

What you're ignoring is that the military is becoming steadily more mechanized also. There won't be many jobs there, either. Robots are more reliable and less likely to side with the protesters.

Grishnakh ( 216268 ) writes:

I'm going with the latter (complete economic collapse). There's no way, with the political attitudes and beliefs present in our society, and our current political leaders, that we'd be able to pivot fast enough to avoid it. Only small, homogenous nations like Finland (or Japan, even though it's not that small, but it is homogenous) can pull that off because they don't have all the infighting and diversity of political beliefs that we do, plus our religious notion of "self reliance".

scamper_22 ( 1073470 ) writes:

There are a few ways this plays out. How do we deal with this. One way is a basic income.

The other less articulated way, but is the basis for a lot of people's views is things simply get cheaper. Deflation is good. You simply live on less. You work less. You earn less. But you can afford the food, water... of life.

Now this is a hard transition in many places. There are loads of things that don't go well with living on less and deflation. Debt, government services, pensions...

I grew up in a third world coun

Grishnakh ( 216268 ) writes:

The main problem with this idea of "living on less" is that, even in the southern US, the rent prices are very high these days because of the real estate bubble and property speculation and foreign investment. The only place where property isn't expensive is in places where there are really zero jobs at all.

Gr8Apes ( 679165 ) writes:

All jobs that don't do R&D will be replaceable in the near future, as in within 1 or 2 generations. Even R&D jobs will likely not be immune, since much R&D is really nothing more than testing a basic hypothesis, of which most of the testing can likely be handed over to AI. The question is what do you do with 24B people with nothing but spare time on their hands, and a smidgen of 1% that actually will have all the wealth? It doesn't sound pretty, unless some serious changes in the way we deal wit

DigiShaman ( 671371 ) writes:

Worse! Far worse!! Total collapse of the fiat currencies globally is imminent. When you reduce the human labor participation rate relative to the overall population, what you get is deflation. That's an undeniable fact.

But factor in governments around the world "borrowing" money via printing to pay welfare for all those unemployed. So now we have deflation coupled with inflation = stagflation. But stagflation doesn't last. At some point, the entire system - as we know it- will implode. What can not go on f

HiThere ( 15173 ) writes:

I don't know what the right answer is, but it's not unions. Unions exist to protect jobs and employment. The Pacific Longshoremen's Union during the 1960's&70's was an aberration in the the union bosses didn't primarily look after maintaining their own power via maintaining a large number of jobs, but rather opted into profit sharing, protecting the current workers at the expense of future power. Usually a union can be depended upon to fight automation, rather than to seek maximization of public good

Ol Olsoc ( 1175323 ) writes:

Are we heading towards "basic wage" for all people?

I think it's the only answer (without genocides...). My money is on genocide. Cheaper, and humans have it as a core value.

sjbe ( 173966 ) , Monday January 02, 2017 @12:43PM ( #53592831 )

Failure of imagination ( Score: 5 , Informative)

As a software developer of enterprise software, every company I have worked for has either produced software which reduced white collar jobs or allowed companies to grow without hiring more people.

You're looking at the wrong scale. You need to look at the whole economy. Were those people able to get hired elsewhere? The answer in general was almost certainly yes. Might have taken some of them a few months, but eventually they found something else.

My company just bought a machine that allows us to manufacture wire leads much faster than we can do it by hand. That doesn't mean that the workers we didn't employ to do that work couldn't find gainful employment elsewhere.

And we exist in a primarily zero sum portion of our industry, so this is directly taking revenue and jobs from other companies.

Again, so what? You've automated some efficiency into an industry that obviously needed it. Some workers will have to do something else. Same story we've been hearing for centuries. It's the buggy whip story just being retold with a new product. Not anything to get worried about.

People need to stop living in a fairy tale land where near full employment is a reality in the near future.

Based on what? The fact that you can't imagine what people are going to do if they can't do what they currently are doing? I'm old enough to predate the internet. The World Wide Web was just becoming a thing while I was in college. Apple, Microsoft, Google, Amazon, Cisco, Oracle, etc all didn't even exist when I was born. Vast swaths of our economy hadn't even been conceived of back then. 40 years from now you will see a totally new set of companies doing amazing things you never even imagined. Your argument is really just a failure of your own imagination. People have been making that same argument since the dawn of the industrial revolution and it is just as nonsensical now as it was then.

I'll be surprised if labor participation rate of 25-54 year olds is even 50% in 10 years.

Prepare to be surprised then. Your argument has no rational basis. You are extrapolating some micro-trends in your company well beyond any rational justification.

TuringTest ( 533084 ) writes:

Were those people able to get hired elsewhere? The answer in general was almost certainly yes.

Oh, oh, I know this one! "New jobs being created in the past don't guarantee that new jobs will be created in the future". This is the standard groupthink answer for waiving any responsibility after advice given about the future, right?

paiute ( 550198 ) writes:
People have been making that same argument since the dawn of the industrial revolution and it is just as nonsensical now as it was then.

I see this argument often when these type of discussions come up. It seems to me to be some kind of logical fallacy to think that something new will not happen because it has not happened in the past. It reminds me of the historical observation that generals are always fighting the last war.

sjbe ( 173966 ) writes:

Asking the wrong question

It seems to me to be some kind of logical fallacy to think that something new will not happen because it has not happened in the past.

What about humans and their ability to problem solve and create and build has changed? The reason I don't see any reason to worry about "robots" taking all our jobs is because NOTHING has changed about the ability of humans to adapt to new circumstances. Nobody has been able to make a coherent argument detailing why humans will not be able to continue to create new industries and new technologies and new products in the future. I don't pretend to know what those new economies will look like with any gre

ranton ( 36917 ) writes:
You didn't finish your thought. Just because generals are still thinking about the last war doesn't mean they don't adapt to the new one when it starts.

Actually yes it does. The history of the blitzkrieg is not one of France quickly adapting to new technologies and strategies to repel the German invaders. It is of France's Maginot line being mostly useless in the war and Germany capturing Paris with ease. Something neither side could accomplish in over four years in the previous war was accomplished in around two months using the new paradigm.

Will human participation in the workforce adapt to AI technologies in the next 50 years? Almost certainly. Is it li

Re: ( Score: 2 ) alexgieg ( 948359 ) writes:

It's simple. Do you know how, once we applied human brain power over the problem of flying we managed, in a matter of decades, to become better at flying than nature ever did in hundreds of millions of years of natural selection? Well, what do you think will happen now that we're focused on making AI better than brains? As in, better than any brains, including ours?

AI is catching up to human abilities. There's still a way to go, but breakthroughs are happening all the time. And as with flying, it won't take

Re: ( Score: 2 ) HiThere ( 15173 ) writes:

One can hope that your analogy with flying is correct. There are still many things that birds do better than planes. Even so I consider that a conservative projection when given without a time-line.

Re: ( Score: 2 ) Ol Olsoc ( 1175323 ) writes:
What about humans and their ability to problem solve and create and build has changed? The reason I don't see any reason to worry about "robots" taking all our jobs is because NOTHING has changed about the ability of humans to adapt to new circumstances.

I had this discussion with a fellow a long time ago who was so conservative he didn't want any regulations on pollutants. The Love Canal disaster wsa the topic. He said "no need to do anything, because humans will adapt - its called evolution."

I answered - "Yes, we might adapt. But you realize that means 999 out of a 1000 of us will die, and it's called evolution. Sometimes even 1000 out of 1000 die, that's called extinction."

This will be a different adaptation, but very well might be solved by most of

Re: ( Score: 2 ) Dutch Gun ( 899105 ) writes:

Generally speaking, though, when you see a very consistent trend or pattern over a long time, your best bet is that the trend will continue, not that it will mysteriously veer off because now it's happening to white collar jobs instead of blue collar jobs. I'd say the logical fallacy is to disbelieve that the trend is likely to continue. Technology doesn't invalidate basic economic theory, in which people manage to find jobs and services to match the level of the population precisely because there are so

Re: ( Score: 2 ) ranton ( 36917 ) writes:
It's the buggy whip story just being retold with a new product. Not anything to get worried about.

The buggy whip story shows that an entire species which had significant economic value for thousands of years found that technology had finally reached a point where they weren't needed. Instead of needing 20 million of them working in our economy in 1920, by 1960 there were only about 4.5 million. While they were able to take advantage of the previous technological revolutions and become even more useful because of better technology in the past, most horses could not survive the invention of the automobile

Re:Failure of imagination ( Score: 4 , Insightful) fluffernutter ( 1411889 ) , Monday January 02, 2017 @01:53PM ( #53593139 )
Were those people able to get hired elsewhere?

Your question is incomplete. The correct question to ask is if these people were able to get hired elsewhere *at the same salary when adjusted for inflation*. To that, the answer is no.

It hasn't been true on average since the 70's. Sure, some people will find equal or better jobs, but salaries have been steadily decreasing since the onset of technology. Given a job for less money or no job, most people will pick the job for less; and that is why we are not seeing a large change in the unemployment rate.

Re: ( Score: 3 ) gtall ( 79522 ) writes:

There is another effect. When the buggy whip manufacturers were put out of business, there were options for people to switch to and new industries were created. However, if AI gets apply across an entire economy, there won't be options because there is unemployment in every sector. And if AI obviates the need for workers, investors in new industries will build them around bots, so no real increase in employment. That and yer basic truck driver ain't going to be learning how to program.

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

Agreed, companies will be designed around using as little human intervention as possible. First they will use AI, then they will use cheap foreign labor, and only if those two options are completely impractical will they use domestic labor. Any business plan that depends on more than a small fraction of domestic labor (think Amazon's 1 minute of human handling per package) is likely to be considered unable to compete. I hate the buggy whip analogy, because using foreign (cheap) labor as freely as today w

Re: ( Score: 2 ) mspring ( 126862 ) writes:

Maybe the automation is a paradigm shift on par with the introduction of agriculture replacing the hunter and gatherer way of living? Then, some hunter and gatherer were perhaps also making a "luddite" arguments: "Nah, there will always be sufficient forrests/wildlife for everyone to live on. No need to be afraid of the these agriculturites. We have been hunting and gathering for millenia. That'll never change."

Re: ( Score: 3 ) bluegutang ( 2814641 ) writes:

Were those people able to get hired elsewhere? The answer in general was almost certainly yes.

Actually, the answer is probably no. Labor force participation [tradingeconomics.com] rates have fallen steadily since about the year 2000. Feminism caused the rate to rise from 58% (1963) to 67% (2000). Since then, it has fallen to 63%. In other words, we've already lost almost half of what we gained from women entering the workforce en masse. And the rate will only continue to fall in the future.

Re: ( Score: 2 ) J-1000 ( 869558 ) writes:

You must admit that *some* things are different. Conglomeratization may make it difficult to create new jobs, as smaller businesses have trouble competing with the mammoths. Globalization may send more jobs offshore until our standard of living has leveled off with the rest of the world. It's not inconceivable that we'll end up with a much larger number of unemployed people, with AI being a significant contributing factor. It's not a certainty, but neither is your scenario of the status quo. Just because it

Re: ( Score: 2 ) Ol Olsoc ( 1175323 ) writes:
People need to stop living in a fairy tale land where near full employment is a reality in the near future. I'll be surprised if labor participation rate of 25-54 year olds is even 50% in 10 years.

Then again, tell me of how companies are going to make money to service the stakeholders when there are not people around wh ocan buy their highly profitable wares?

Now speaking of fairy tales, that one is much more magical than your full employment one.

This ain't rocket science. Economies are at base, an equation. You have producers on one side, and consumers on the other. Ideally, they balance out, with extra rewards for the producers. Now either side can cheat, such as if producers can move productio

Re: ( Score: 2 ) Oswald McWeany ( 2428506 ) writes:
If you think software developers are immune, you're delusional.

I wonder if the software developers paid to create software to make software developers obsolete will have any qualms about writing that code.

Humans stopped writing computer code after Fortran ( Score: 2 ) raymorris ( 2726007 ) writes:

Until Fortran was developed, humans used to write code telling the computer what to do. Since the late 1950s, we've been writing a high-level description, then a computer program writes the program that actually gets executed.

Nowadays, there's frequently a computer program, such as a browser, which accepts our high-level description of the task and interprets it before generating more specific instructions for another piece of software, an api library, which creates more specific instructions for another api

Re: ( Score: 2 ) avandesande ( 143899 ) writes:

Until something goes wrong. Who is going to pick that machine generated code apart?

No more working till last train but with life empl ( Score: 2 ) Joe_Dragon ( 2206452 ) writes:

No more working till last train but with life employment where will laid off people find new jobs?

Re: ( Score: 2 ) JoeMerchant ( 803320 ) writes:

They won't, that's the point.

I see plenty of work in reducing student-teacher ratios in education, increasing maintenance and inspection intervals, transparency reporting on public officials, etc. Now, just convince the remaining working people that they want to pay for this from their taxes.

I suppose when we hit 53% unemployed, we might be able to start winning popular elections, if the unemployed are still allowed to vote then.

Re: ( Score: 2 ) Grishnakh ( 216268 ) writes:

At least here in the US, that won't change anything. The unemployed will still happily vote against anything that smacks of "socialism". It's a religion to us here. People here would rather shoot themselves (and their family members) in the head than enroll in social services.

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

That's a pretty funny thing to say about a nation with more than a third on welfare.

Re: ( Score: 2 ) Grishnakh ( 216268 ) writes:

Remember, most of the US population is religious, and not only does this involve some "actual" religion (usually Christianity), it also involves the "anti-socialism" religion. Now remember, the defining feature of religion is a complete lack rationality, and believing in something with zero supporting evidence, frequently despite enormous evidence to the contrary (as in the case of young-earth creationism, something that a huge number of Americans believe in).

So yes, it is "a pretty funny thing to say", bu

Re: ( Score: 2 ) pla ( 258480 ) writes:

"One job, one vote!" / shudder

Obviously ( Score: 3 , Insightful) Anonymous Coward , Monday January 02, 2017 @12:11PM ( #53592677 )

People that do trivial tasks like looking at numbers on documents, something a computer can easily do, are prime for getting replaced.

Face it, if you aren't creating new things, you're the first to go. Maintaining a process is basically pattern recognition.

Re: ( Score: 2 ) kwerle ( 39371 ) writes:

SInce this is very very similar to what my partner does, I feel like I'm a little qualified to speak on the subject at hand.

Yeah, pattern matching should nail this - but pattern matching only works if the patterns are reasonable/logical/consistent. Yes, I'm a little familiar with advanced pattern matching, filtering, etc.

Here's the thing: doctors are crappy input sources. At least in the US medical system. And in our system they are the ones that have to make diagnosis (in most cases). They are inconsistent.

Re: ( Score: 2 ) RightwingNutjob ( 1302813 ) writes:

What automation? 1000 workers in US vs 2000 in Mexico for half the cost of those 1000 is not "automation." Same thing with your hand-assembled smartphone. I'd rather have it be assembled by robots in the US with 100 human babysitters than hand-built in China with by 1000 human drones.

GIGO ( Score: 2 ) ISoldat53 ( 977164 ) writes:

I hope their data collection is better than it is in the US. Insurance company's systems can't talk to the doctors systems. They are stuck with 1980s technology or sneaker net to get information exchanged. Paper gets lost, forms don't match.

Doctors spend more time with paper than with patients. Once the paper gets to the insurance company chances are good it doesn't go to the right person or just gets lost sending the patient back to the beginning of the maze. The more people removed from the chain the bet

Re: ( Score: 2 ) ColdWetDog ( 752185 ) writes:

You think this is anything but perfectly planned? Insurance companies prevaricate better than anyone short of a Federal politician. 'Losing' a claim costs virtually nothing. Mishandling a claim costs very little. Another form letter asking for more / the same information, ditto.

Computerizing the whole shebang gives yet another layer of potential delay ('the computer is slow today' is a perennial favorite).

That said, in what strange world is insurance adjudication considered 'white collar'? In the US a

Japanese workforce is growing old ( Score: 3 ) AchilleTalon ( 540925 ) , Monday January 02, 2017 @12:21PM ( #53592727 ) Homepage

Japan needs to automate as much as it can and robotize to survive with a workforce growing old. Japan is facing this reality as well as many countries where labor isn't replaced at a sufficient rate to keep up with the needs. Older people will need care some countries just cannot deliver or afford.

Re: ( Score: 3 ) avandesande ( 143899 ) writes:

Japan is notorious for being far behind on office automation.

sjbe ( 173966 ) , Monday January 02, 2017 @12:27PM ( #53592743 )

Queue the chicken littles ( Score: 3 )

Calm down everyone. This is just a continuation of productivity tools for accounting. Among other things I'm a certified accountant. This is just the next step in automation of accounting and it's a good thing. We used to do all our ledgers by hand. Now we all use software for that and believe me you don't want to go back to the way it was.

Very little in accounting is actually value added activity so it is desirable to automate as much of it as possible. If some people lost their jobs doing that it's equivalent to how the PC replaced secretaries 30+ years ago. They were doing a necessary task but one that added little or no value. Most of what accountants do is just keeping track of what happened in a business and keeping the paperwork flowing where it needs to go. This is EXACTLY what we should be automating whenever possible.

I'm sure there are going to be a lot folks loudly proclaiming how we are all doomed and that there won't be any work for anyone left to do. Happens every time there is an advancement in automation and yet every time they are wrong. Yes some people are going to struggle in the short run. That happens with every technological advancement. Eventually they find other useful and valuable things to do and the world moves on. It will be fine.

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

I'm curious what you think you can do that Watson can't. Accounting is a very rigidly structured practice. All IBM really needs to do is let Watson sift through the books of a couple hundred companies and it will easily determine how to best achieve a defined set of objectives for a corporation.

sjbe ( 173966 ) writes:

Accounting isn't what you think it is ( Score: 2 )

I'm curious what you think you can do that Watson can't.

Seriously? Quite a bit actually. I can handle input streams that Watson can't. I can make tools Watson couldn't begin to imagine. I can interact with physical objects without vast amounts of programming. I can deal with humans in a meaningful and human way FAR better than any computer program. I can pass a Turing test. The number of things I can do that Watson cannot is literally too numerous to bother counting. Watson is really just an decision support system with a natural language interface. Ver

Re: ( Score: 2 ) King_TJ ( 85913 ) writes:

Yep! I don't even work in Accounting or Finance, but because I do computer support for that department and have to get slightly involved in the bill coding side of the process -- I agree completely.

I'm pretty sure that even if you *could* get a computer to do everything for Accounting automatically, people would constantly become frustrated with parts of the resulting process -- from reports requested by management not having the formatting or items desired on them, to inflexibility getting an item charged

IBM Puff Piece ( Score: 2 ) avandesande ( 143899 ) writes:

I work on a claims processing system and 90% of this stuff is already automated.

Re: ( Score: 2 ) avandesande ( 143899 ) writes:

Some bills are just so fubar that someone has to look at them. You really think 'watson' is processing 100% of the bills?

Re: ( Score: 2 ) avandesande ( 143899 ) writes:

Uh... well maybe. But what does this have to do with being an IBM puff piece?

Re: ( Score: 2 ) avandesande ( 143899 ) writes:

You think the 12$ hr staff at a doctors office code and invoice bills correctly? The blame goes both ways. Really our ridiculous and convoluted medical system is to blame. Imagine if doctors billed on a time basis like a lawyer.

gweihir ( 88907 ) , Monday January 02, 2017 @12:43PM ( #53592829 )

That is "automation". AI is something else... ( Score: 3 )

When you have people basically implementing a process without much understanding, it is pretty easy to automatize their jobs away. The only thing Watson is contribution is the translation from natural language to a more formalized one. No actual intelligence needed.

Re: ( Score: 2 ) gweihir ( 88907 ) writes:

I wish. Artificial stupidity is a bit more advanced than AI, but nowhere there yet.

LeftCoastThinker ( 4697521 ) writes:

This is not news or new ( Score: 2 )

Computers/automation/robotics have been replacing workers of all stripes including white collar workers since the ATM was introduced in 1967. Every place I have ever worked has had internal and external software that replaces white collar workers (where you used to need 10 people now you need 2).

The reality is that the economy is limited by a scarcity of labor when government doesn't interfere (the economy is essentially the sum of every worker work multiplied by their efficiency as valued by the economy i

Don't worry, Trump has the solution ( Score: 4 ) Jeremi ( 14640 ) , Monday January 02, 2017 @02:09PM ( #53593245 ) Homepage

Turns out it's rather simple, really --- just ban computers. He's going to start by replacing computers with human couriers for the secure-messaging market, and move outward from there. By 2020 we should have most of the Internet replaced by the (now greatly expanded) Post Office.

Dixie_Flatline ( 5077 ) writes:

We could use a little more automation, if you ask ( Score: 2 )

At least, as long as banks keep writing the software they do.

My bank's records of my purchases isn't updating today. This is one of the biggest banks in Canada. Transactions don't update properly over the weekends or holidays. Why? Who knows? Why has bank software EVER cared about weekends? What do business days matter to computers? And yet here we are. There's no monkey to turn the crank on a holiday, so I can't confirm my account activity.

DogDude ( 805747 ) writes:

Re: ( Score: 2 )

free market people as opposed to corporatist You need to pick up an economics textbook. Or a history textbook.

Re: ( Score: 2 ) GLMDesigns ( 2044134 ) writes:

Dude. Stop it. I've read 18th C laissez-faire writers (de Gournay) Bastiat, the Austrian School (Carl Menger, Bohm-Bawerk, von Mises, Hayek), Rothbard, Milton Friedman. Free Market is opposed to corporatism, You might hate Ayn Rand but she skewered corporatists as much as she did socialists. You should read some of these people. You'll see that they are opposed to corporatism. Don't get your information from opponents who create straw men and then, so skillfully, defeat their opponent's arguments.

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

I'm laughing that you think there is a difference. How do these people participate in a free market without setting up corporations?

Re: ( Score: 2 ) GLMDesigns ( 2044134 ) writes:

Corporatism is the use of government pull to advance your business. The use of law and the police power of the state to aide your business against anothers. This used to be called "mercantilism." Free market capitalism is opposed to this; the removal of power of pull.

Re: ( Score: 2 ) GLMDesigns ( 2044134 ) writes:

Read Bastiat, Carl Menger, von Mises, Hayek, Milton Friedman. You'll see them all referring to the government as an agent which helps one set of businesses over another. Government may give loans, bailouts, etc... Free market people are against this. Corporatism /= Free Market. Don't only get your information from those who hate individualism and free markets - read (or in Milton Friedman's case listen) to their arguments. You may disagree with them but you'll see well regarded individuals who say that

Re: ( Score: 2 ) GLMDesigns ( 2044134 ) writes:

When a business get's government to give it special favors (Soyndra) or to give it tax breaks or a monopoly this is corporatism. It used to be called mercantilism. In either case free - market capitalists stand in opposition to it. This is exactly what "laissez-faire" capitalism means: leave us alone, don't play favorites, stay away.

Re: ( Score: 2 ) pla ( 258480 ) writes:

How do these people participate in a free market without setting up corporations? Have you ever bought anything from a farmers' market? Have you ever hired a plumber d/b/a himself rather than working for Plumbers-R-Us? Have you ever bought a used car directly from a private seller? Do you have a 401k/403b/457/TSP/IRA? Have you ever used eBay? Have you ever traded your labor for a paycheck (aka "worked") without hiding behind an intermediate shell-corp? The freeness of a market has nothing to do wit

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

Trump's staff are all billionaires? How many people do you know that became a billionaire by selling at a farmer's market?

Re: ( Score: 2 ) pla ( 258480 ) writes:

Okay, so you're just still pissing and moaning over Trump's win and have no actual point. That's fine, but you should take care not to make it sound too much like you actually have something meaningful to say.

Re: ( Score: 2 ) fluffernutter ( 1411889 ) writes:

I'll say something meaningful when you can point out which one of Trump's cabinet made their wealth on a farmer's market and without being affiliated with a corporation.

Re: ( Score: 3 ) Waffle Iron ( 339739 ) writes:

He's hired primarily free market people as opposed to corporatist

Free marketers don't generally campaign on a platform of protectionist trade policies and direct government intervention in job markets.

Re: ( Score: 2 ) GLMDesigns ( 2044134 ) writes:

No. They don't. But, for the moment, it looks as if Andy Puzder (Sec of Labor) and Mick Mulvaney (OMB) are fairly good free market people. We'll see. Chief of Staff Reince Priebus has made some free-market comments. (Again, we'll see.) Sec of Ed looks like she wants to break up an entrenched bureaucracy - might even work to remove Federal involvement. (Wishful thinking on my part) HUD - I'm hopeful that Ben Carson was hired to break up this ridiculous bureaucracy. If not, at least pare it down. Now, if

Re: ( Score: 3 ) phantomfive ( 622387 ) writes:

"Watson" is a marketing term from IBM, covering a lot of standard automation. It isn't the machine that won at Jeopardy (although that is included in the marketing term, if someone wants to pay for it). IBM tells managers, "We will have our amazing Watson technology solve this problem for you." The managers feel happy. Then IBM has some outsourced programmers code up a workflow app, with recurring annual subscription payments.

Re: ( Score: 2 ) Opportunist ( 166417 ) writes:

That's ok, there isn't really a decent insurance claim worker either, so they should do fine.

Re: ( Score: 2 ) TuringTest ( 533084 ) writes:

considering nobody has made any decent AI yet.

It doesn't matter. AI works best when there's a human in the loop, piloting the controls anyway.

What matters to a company is that 1 person + bots can now make the job that previously required hundreds of white collar workers, for much less salary. What happens to the other workers should not be a concern of the company managers, according to the modern religious creed - apparently some magical market hand takes care to solve that problem automatically.

Re: ( Score: 2 ) jedidiah ( 1196 ) writes:

Pretty much. US companies already use claims processing systems that use previous data to evaluate a current claim and spit out a number. Younger computer literate adjusters just feed the machine and push a button.

Re: ( Score: 2 ) Opportunist ( 166417 ) writes:

The hot topic on the management floor of 2030 is probably how it's no longer "android" but "gynoid".

Re: ( Score: 2 ) GameboyRMH ( 1153867 ) writes:

I was correcting people who refer to robots that look like women as androids before it was cool :-P

Joe_Dragon ( 2206452 ) writes:

universities downsize not with unlimited loans! ( Score: 2 )

universities downsize not with unlimited loans! (usa only) need retraining you can get an loan and you may need to go for 2-4 years and (some credits maybe to old and you have to retake classes)

Re: ( Score: 2 ) Joe_Dragon ( 2206452 ) writes:

IBM helped the Hitler was able to automate his persecution of the Jews. So will Watson have locks to stop that or any other killing off of people?

Re: ( Score: 2 ) GameboyRMH ( 1153867 ) writes:

It's not Parkinson's law, it's runaway inequality. The workforce continues to be more and more productive as it receives an unchanging or decreasing amount of compensation (in absolute terms - or an ever-decreasing share of the profits in relative terms), while the gains go to the 1%.

[Dec 27, 2016] Peak Robot: The humans as the fragment of machines

Dec 27, 2016 | econospeak.blogspot.com
http://econospeak.blogspot.com/2016/12/peak-robot-fragment-on-machines.html

December 25, 2016

Peak Robot: the Fragment on Machines

Martin Sklar's disaccumultion thesis * is a restatement and reinterpretation of passages in Marx's Grundrisse that have come to be known as the "fragment on machines." Compare, for example, the following two key excerpts.

Marx:

...to the degree that large industry develops, the creation of real wealth comes to depend less on labour time and on the amount of labour employed than on the power of the agencies set in motion during labour time, whose 'powerful effectiveness' is itself in turn out of all proportion to the direct labour time spent on their production, but depends rather on the general state of science and on the progress of technology, or the application of this science to production. ...

Labour no longer appears so much to be included within the production process; rather, the human being comes to relate more as watchman and regulator to the production process itself. (What holds for machinery holds likewise for the combination of human activities and the development of human intercourse.)

Sklar:

In consequence [of the passage from the accumulation phase of capitalism to the "disaccumlation" phase], and increasingly, human labor (i.e. the exercise of living labor-power) recedes from the condition of serving as a 'factor' of goods production, and by the same token, the mode of goods-production progressively undergoes reversion to a condition comparable to a gratuitous 'force of nature': energy, harnessed and directed through technically sophisticated machinery, produces goods, as trees produce fruit, without the involvement of, or need for, human labor-time in the immediate production process itself. Living labor-power in goods-production devolves upon the quantitatively declining role of watching, regulating, and superintending.

The main difference between the two arguments is that for Marx, the growing contradiction between the forces of production and the social relations produce "the material conditions to blow this foundation sky-high." For Sklar, with the benefit of another century of observation, disaccumulation appears as simply another phase in the evolution of capitalism -- albeit with revolutionary potential. But also with reactionary potential in that the reduced dependence on labor power also suggests a reduced vulnerability to the withholding of labor power.

* http://econospeak.blogspot.ca/2016/12/peak-robot-accumulation-and-its-dis.html

-- Sandwichman

[Dec 27, 2016] Facing Layoff, An IT Employee Makes A Bold Counteroffer

Carnival Corp. told about 200 IT employees that the company was transferring their work to Capgemini, a large IT outsourcing firm
Notable quotes:
"... Senior IT engineer Matthew Culver told CBS that the requested "knowledge transfer activities" just meant training their own replacements , and "he isn't buying any of it," writes Slashdot reader dcblogs . ..."
"... Foreign workers are willing to do a job at a lower salary in most if not all cases b/c the cost of living in their respective countries is a fraction of ours. ..."
Dec 26, 2016 | it.slashdot.org
(computerworld.com) 134

Posted by EditorDavid on Sunday December 25, 2016 @05:05PM from the Bob-Cratchit-vs-Scrooge dept.

ComputerWorld reports:

In early December, Carnival Corp. told about 200 IT employees that the company was transferring their work to Capgemini, a large IT outsourcing firm. The employees had a choice: Either agree to take a job with the contractor or leave without severance. The employees had until the week before Christmas to make a decision about their future with the cruise line.

By agreeing to a job with Paris-based Capgemini, employees are guaranteed employment for six months, said Roger Frizzell, a Carnival spokesman.

"Our expectation is that many will continue to work on our account or placed into other open positions within Capgemini" that go well beyond the six-month period, he said in an email.

Senior IT engineer Matthew Culver told CBS that the requested "knowledge transfer activities" just meant training their own replacements , and "he isn't buying any of it," writes Slashdot reader dcblogs . "After receiving his offer letter from Capgemini, he sent a counteroffer.

It asked for $500,000...and apology letters to all the affected families," signed by the company's CEO. In addition, the letter also demanded a $100,000 donation to any charity that provides services to unemployed American workers. "I appreciate your time and attention to this matter, and I sincerely hope that you can fulfill these terms."

And he's also working directly with a lawyer for an advocacy group that aims to "stop the abuse of H-1B and other foreign worker programs ."

Re:Dear Matthew ( Score: 5 , Insightful) by Anonymous Coward writes: on Sunday December 25, 2016 @06:00PM ( #53553189 )

Foreign workers are willing to do a job at a lower salary in most if not all cases b/c the cost of living in their respective countries is a fraction of ours.

I would be willing to do my job at a fraction of what I am paid currently should that (that being how expensive it is to live here) change. It is equally infuriating to me when American companies use loopholes in our ridiculously complicated tax code to shelter revenues in foreign tax shelters to avoid paying taxes while at the same time benefiting from our infrastructure, emergency services, military, etc..

Its assholes like you that always spout off about free market this or that, about some companies fiduciary responsibilities to it's shareholders blah blah blah... as justification for shitty behavior.

Re:Dear Matthew ( Score: 2 ) by ShanghaiBill ( 739463 ) writes: on Sunday December 25, 2016 @06:13PM ( #53553247 )
It is equally infuriating to me when American companies use loopholes in our ridiculously complicated tax code to shelter revenues in foreign tax shelters to avoid paying taxes

So who are you infuriated at? The companies that take advantage of those loopholes, or the politicians that put them there? Fury doesn't help unless it is properly directed. Does your fury influence who you vote for?

... while at the same time benefiting from our infrastructure, emergency services, military, etc.

No. Taxes are only sheltered on income generated overseas, using overseas infrastructure, emergency services, etc. I am baffled why Americans believe they have a "right" to tax the sale of a product made in China and sold in France.

Re:Dear Matthew ( Score: 2 ) by PolygamousRanchKid ( 1290638 ) writes: on Sunday December 25, 2016 @06:33PM ( #53553303 )
I am baffled why Americans believe they have a "right" to tax the sale of a product made in China and sold in France.

In a seriously silly Monty Python sketch about taxes, someone mildly suggested:

"I think we should tax foreigners, living abroad."

Kinda sorta the same idea . . .

Re:Dear Matthew ( Score: 3 ) by fibonacci8 ( 260615 ) writes: on Sunday December 25, 2016 @08:43PM ( #53553777 )

I suppose it's related to the idea that intellectual property "rights" granted by a country of origin should still have the same benefits and drawbacks when transferred to another country. Or at the very least should be treated as an export at such time a base of operations moves out of country.

Re:Dear Matthew ( Score: 5 , Insightful) by Rob Y. ( 110975 ) writes: on Sunday December 25, 2016 @06:37PM ( #53553317 )

Except that calling, say iOS sales 'generated overseas' when the software was written in the US, using US infrastructure, etc . And the company is making the bogus claim that their Irish subsidiary owns the rights to that software. It's a scam - not a loophole.

Re:Dear Matthew ( Score: 5 , Insightful) by geoskd ( 321194 ) writes: on Sunday December 25, 2016 @07:35PM ( #53553547 )
It's a scam - not a loophole.

They are the same thing. The only way to ensure that there are no tax dodges out there is to simplify the tax code, and eliminate the words: "except", "but", "excluding", "omitting", "minus", "exempt", "without", and any other words to those same effects.

Americans are too stupid to ever vote for a poltiician that states they will raise taxes. This means that either politicians lie, or they actively undermine the tax base. Both of those situations are bad for the majority of americans, but they vote for the same scumbags over and over, and will soundly reject any politician who openly advocates tax increases. The result is a race to the bottom. Welcome to reaping what you sow, brought to you by Democracy(tm).

Re: ( Score: 2 ) by ShanghaiBill ( 739463 ) writes:

Except that calling, say iOS sales 'generated overseas' when the software was written in the US, using US infrastructure, etc .

That makes no sense. Plenty of non-American companies develop software in America. Yet only if they are incorporated in America do they pay income tax on their overseas earnings, and it is irrelevant where their engineering and development was done.

It has nothing whatsoever to do with "using infrastructure". It is just an extraterritorial money grab that is almost certainly counterproductive since it incentivizes American companies to invest and create jobs overseas.

Re: Dear Matthew ( Score: 2 , Insightful) by Anonymous Coward writes:

Yes, taxes are based on profits. So Google, for instance, makes a bunch of money in the US. Their Irish branch then charges about that much for "consulting" leaving the American part with little to no profits to tax.

Re: ( Score: 2 ) by SwashbucklingCowboy ( 727629 ) writes:

Oh get real. Companies make it appear that nearly all income is generated overseas in order to get around that. It's mostly a scam.

Re:Dear Matthew ( Score: 4 , Insightful) by msauve ( 701917 ) writes: on Sunday December 25, 2016 @07:45PM ( #53553601 )

"I am baffled why Americans believe they have a "right" to tax the sale of a product made in China and sold in France."

Because the manufacturing and sales are controlled by a US based company, as is the profit benefit which results. If a US entity, which receives the benefits of US law, makes a profit by any means, why should it not be taxed by the US?

[Dec 26, 2016] Scientists Develop Robotic Hand For People With Quadriplegia

Dec 26, 2016 | science.slashdot.org
(phys.org) 22

Posted by BeauHD on Tuesday December 06, 2016 @07:05PM from the muscle-memory dept.

An anonymous reader quotes a report from Phys.Org:

Scientists have developed a mind-controlled robotic hand that allows people with certain types of spinal injuries to perform everyday tasks such as using a fork or drinking from a cup. The low-cost device was tested in Spain on six people with quadriplegia affecting their ability to grasp or manipulate objects. By wearing a cap that measures electric brain activity and eye movement the users were able to send signals to a tablet computer that controlled the glove-like device attached to their hand. Participants in the small-scale study were able to perform daily activities better with the robotic hand than without, according to results published Tuesday in the journal Science Robotics .

It took participants just 10 minutes to learn how to use the system before they were able to carry out tasks such as picking up potato chips or signing a document. According to Surjo R. Soekadar, a neuroscientist at the University Hospital Tuebingen in Germany and lead author of the study, participants represented typical people with high spinal cord injuries, meaning they were able to move their shoulders but not their fingers. There were some limitations to the system, though. Users had to have sufficient function in their shoulder and arm to reach out with the robotic hand. And mounting the system required another person's help.

[Dec 26, 2016] Autonomous Shuttle Brakes For Squirrels, Skateboarders, and Texting Students

Dec 26, 2016 | tech.slashdot.org
(ieee.org) 74

Posted by BeauHD on Saturday December 10, 2016 @05:00AM from the squirrel-crossing dept.

Tekla Perry writes:

An autonomous shuttle from Auro Robotics is picking up and dropping off students, faculty, and visitors at the Santa Clara University Campus seven days a week. It doesn't go fast, but it has to watch out for pedestrians, skateboarders, bicyclists, and bold squirrels (engineers added a special squirrel lidar on the bumper). An Auro engineer rides along at this point to keep the university happy, but soon will be replaced by a big red emergency stop button (think Staples Easy button). If you want a test drive, just look for a "shuttle stop" sign (there's one in front of the parking garage) and climb on, it doesn't ask for university ID.

[Dec 26, 2016] Robots Are Already Replacing Fast-Food Workers

Dec 26, 2016 | hardware.slashdot.org
(recode.net) 414

Posted by EditorDavid on Sunday December 11, 2016 @05:34PM from the may-I-take-your-order dept.

An anonymous reader quotes Recode:

Technology that replaces food service workers is already here . Sushi restaurants have been using machines to roll rice in nori for years, an otherwise monotonous and time-consuming task. The company Suzuka has robots that help assemble thousands of pieces of sushi an hour. In Mountain View, California, the startup Zume is trying to disrupt pizza with a pie-making machine. In Shanghai, there's a robot that makes ramen , and some cruise ships now mix drinks with bartending machines .

More directly to the heart of American fast-food cuisine, Momentum Machines, a restaurant concept with a robot that can supposedly flip hundreds of burgers an hour , applied for a building permit in San Francisco and started listing job openings this January, reported Eater. Then there's Eatsa, the automat restaurant where no human interaction is necessary, which has locations popping up across California .

[Dec 26, 2016] IBMs Watson Used In Life-Saving Medical Diagnosis

Dec 26, 2016 | science.slashdot.org
(businessinsider.co.id) 83 Posted by EditorDavid on Sunday December 11, 2016 @09:34PM from the damn-it-Jim-I'm-a-doctor-not-a-supercomputer dept.
"Supercomputing has another use," writes Slashdot reader rmdingler , sharing a story that quotes David Kenny, the General Manager of IBM Watson:
"There's a 60-year-old woman in Tokyo. She was at the University of Tokyo. She had been diagnosed with leukemia six years ago. She was living, but not healthy. So the University of Tokyo ran her genomic sequence through Watson and it was able to ascertain that they were off by one thing . Actually, she had two strains of leukemia. They did treat her and she is healthy."

"That's one example. Statistically, we're seeing that about one third of the time, Watson is proposing an additional diagnosis."

[Dec 26, 2016] Latest Microsoft Skype Preview Adds Real-Time Voice Translation For Phone Calls

Notable quotes:
"... Skype Translator, available in nine languages, uses artificial intelligence (AI) techniques such as deep-learning to train artificial neural networks and convert spoken chats in almost real time. The company says the app improves as it listens to more conversations. ..."
Dec 26, 2016 | tech.slashdot.org
(zdnet.com) 37

Posted by msmash on Monday December 12, 2016 @11:05AM from the worthwhile dept.

Microsoft has added the ability to use Skype Translator on calls to mobiles and landlines to its latest Skype Preview app. From a report on ZDNet: Up until now, Skype Translator was available to individuals making Skype-to-Skype calls. The new announcement of the expansion of Skype Translator to mobiles and landlines makes Skype Translator more widely available .

To test drive this, users need to be members of the Windows Insider Program. They need to install the latest version of Skype Preview on their Windows 10 PCs and to have Skype Credits or a subscription.

Skype Translator, available in nine languages, uses artificial intelligence (AI) techniques such as deep-learning to train artificial neural networks and convert spoken chats in almost real time. The company says the app improves as it listens to more conversations.

[Dec 26, 2016] White House: US Needs a Stronger Social Safety Net To Help Workers Displaced by Robots

Dec 26, 2016 | hardware.slashdot.org
(recode.net) 623

Posted by BeauHD on Wednesday December 21, 2016 @08:00AM from the one-day-not-so-far-away dept.

The White House has released a new report warning of a not-too-distant future where artificial intelligence and robotics will take the place of human labor. Recode highlights in its report the three key areas the White House says the U.S. government needs to prepare for the next wave of job displacement caused by robotic automation:

The report says the government, meaning the the incoming Trump administration, will have to forge ahead with new policies and grapple with the complexities of existing social services to protect the millions of Americans who face displacement by advances in automation, robotics and artificial intelligence. The report also calls on the government to keep a close eye on fostering competition in the AI industry, since the companies with the most data will be able to create the most advanced products, effectively preventing new startups from having a chance to even compete.

[Dec 26, 2016] Stanford Built a Humanoid Submarine Robot To Explore a 17th-Century Shipwreck

Notable quotes:
"... IEEE Robotics and Automation Magazine ..."
Dec 26, 2016 | hardware.slashdot.org
(ieee.org) 47 Posted by BeauHD on Friday December 23, 2016 @05:00AM from the how-it's-made dept.

schwit1 quotes a report from IEEE Spectrum:

Back in April, Stanford University professor Oussama Khatib led a team of researchers on an underwater archaeological expedition, 30 kilometers off the southern coast of France, to La Lune , King Louis XIV's sunken 17th-century flagship. Rather than dive to the site of the wreck 100 meters below the surface, which is a very bad idea for almost everyone, Khatib's team brought along a custom-made humanoid submarine robot called Ocean One . In this month's issue of IEEE Robotics and Automation Magazine , the Stanford researchers describe in detail how they designed and built the robot , a hybrid between a humanoid and an underwater remotely operated vehicle (ROV), and also how they managed to send it down to the resting place of La Lune , where it used its three-fingered hands to retrieve a vase. Most ocean-ready ROVs are boxy little submarines that might have an arm on them if you're lucky, but they're not really designed for the kind of fine manipulation that underwater archaeology demands. You could send down a human diver instead, but once you get past about 40 meters, things start to get both complicated and dangerous. Ocean One's humanoid design means that it's easy and intuitive for a human to remotely perform delicate archeological tasks through a telepresence interface.

schwit1 notes: "Ocean One is the best name they could come up with?"

[Dec 26, 2016] Slashdot Asks: Will Farming Be Fully Automated in the Future?

Dec 26, 2016 | hardware.slashdot.org
(bbc.com) 278

Posted by msmash on Friday November 25, 2016 @12:10AM from the interesting-things dept.

BBC has a report today in which, citing several financial institutions and analysts, it claims that in the not-too-distant future, our fields could be tilled, sown, tended and harvested entirely by fleets of co-operating autonomous machines by land and air. An excerpt from the article:

Driverless tractors that can follow pre-programmed routes are already being deployed at large farms around the world. Drones are buzzing over fields assessing crop health and soil conditions. Ground sensors are monitoring the amount of water and nutrients in the soil, triggering irrigation and fertilizer applications. And in Japan, the world's first entirely automated lettuce farm is due for launch next year. The future of farming is automated . The World Bank says we'll need to produce 50% more food by 2050 if the global population continues to rise at its current pace. But the effects of climate change could see crop yields falling by more than a quarter. So autonomous tractors, ground-based sensors, flying drones and enclosed hydroponic farms could all help farmers produce more food, more sustainably at lower cost.

[Dec 26, 2016] Self-Driving Trucks Begin Real-World Tests on Ohios Highways

Dec 26, 2016 | news.slashdot.org
(cbsnews.com) 178

Posted by EditorDavid on Sunday November 27, 2016 @04:35PM from the trucking-up-to-Buffalo dept.

An anonymous reader writes:

"A vehicle from self-driving truck maker Otto will travel a 35-mile stretch of U.S. Route 33 on Monday in central Ohio..." reports the Associated Press.

The truck "will travel in regular traffic, and a driver in the truck will be positioned to intervene should anything go awry, Department of Transportation spokesman Matt Bruning said Friday, adding that 'safety is obviously No. 1.'"

Ohio sees this route as "a corridor where new technologies can be safely tested in real-life traffic, aided by a fiber-optic cable network and sensor systems slated for installation next year" -- although next week the truck will also start driving on the Ohio Turnpike.

[Dec 26, 2016] Stephen Hawking: Automation and AI Is Going To Decimate Middle Class Jobs

Dec 26, 2016 | tech.slashdot.org
(businessinsider.com) 468

Posted by BeauHD on Friday December 02, 2016 @05:00PM from the be-afraid-very-afraid dept.

An anonymous reader quotes a report from Business Insider:

In a column in The Guardian , the world-famous physicist wrote that "the automation of factories has already decimated jobs in traditional manufacturing, and the rise of artificial intelligence is likely to extend this job destruction deep into the middle classes , with only the most caring, creative or supervisory roles remaining." He adds his voice to a growing chorus of experts concerned about the effects that technology will have on workforce in the coming years and decades. The fear is that while artificial intelligence will bring radical increases in efficiency in industry, for ordinary people this will translate into unemployment and uncertainty, as their human jobs are replaced by machines.

Automation will, "in turn will accelerate the already widening economic inequality around the world," Hawking wrote. "The internet and the platforms that it makes possible allow very small groups of individuals to make enormous profits while employing very few people. This is inevitable, it is progress, but it is also socially destructive." He frames this economic anxiety as a reason for the rise in right-wing, populist politics in the West: "We are living in a world of widening, not diminishing, financial inequality, in which many people can see not just their standard of living, but their ability to earn a living at all, disappearing. It is no wonder then that they are searching for a new deal, which Trump and Brexit might have appeared to represent." Combined with other issues -- overpopulation, climate change, disease -- we are, Hawking warns ominously, at "the most dangerous moment in the development of humanity." Humanity must come together if we are to overcome these challenges, he says.

[Dec 26, 2016] Many CEOs Believe Technology Will Make People Largely Irrelevant

Notable quotes:
"... The firm says that 44 percent of the CEOs surveyed agreed that robotics, automation and AI would reshape the future of many work places by making people "largely irrelevant." ..."
Dec 26, 2016 | it.slashdot.org
(betanews.com) 541

Posted by msmash on Monday December 05, 2016 @02:20PM from the shape-of-things-to-come dept.

An anonymous reader shares a report on BetaNews:

Although artificial intelligence (AI), robotics and other emerging technologies may reshape the world as we know it, a new global study has revealed that the many CEOs now value technology over people when it comes to the future of their businesses . The study was conducted by the Los Angeles-based management consultant firm Korn Ferry that interviewed 800 business leaders across a variety of multi-million and multi-billion dollar global organizations.

The firm says that 44 percent of the CEOs surveyed agreed that robotics, automation and AI would reshape the future of many work places by making people "largely irrelevant."

The global managing director of solutions at Korn Ferry Jean-Marc Laouchez explains why many CEOs have adopted this controversial mindset, saying:

"Leaders may be facing what experts call a tangibility bias. Facing uncertainty, they are putting priority in their thinking, planning and execution on the tangible -- what they can see, touch and measure, such as technology instruments."

[Dec 26, 2016] Microsoft Researchers Offer Predictions For AI, Deep Learning

Dec 26, 2016 | hardware.slashdot.org
(theverge.com) 102

Posted by BeauHD on Tuesday December 06, 2016 @10:30PM from the what-to-expect dept.

An anonymous reader quotes a report from The Verge:

Microsoft polled 17 women working in its research organization about the technology advances they expect to see in 2017 , as well as a decade later in 2027. The researchers' predictions touch on natural language processing, machine learning, agricultural software, and virtual reality, among other topics. For virtual reality, Mar Gonzalez Franco , a researcher in Microsoft's Redmond lab, believes body tracking will improve next year, and then over the next decade we'll have "rich multi-sensorial experiences that will be capable of producing hallucinations which blend or alter perceives reality."

Haptic devices will simulate touch to further enhance the sensory experience. Meanwhile, Susan Dumais , a scientist and deputy managing director at the Redmond lab, believes deep learning will help improve web search results next year.

In 2027, however, the search box will disappear, she says.

It'll be replaced by search that's more "ubiquitous, embedded, and contextually sensitive." She says we're already seeing some of this in voice-controlled searches through mobile and smart home devices.

We might eventually be able to look things up with either sound, images, or video. Plus, our searches will respond to "current location, content, entities, and activities" without us explicitly mentioning them, she says.

Of course, it's worth noting that Microsoft has been losing the search box war to Google, so it isn't surprising that the company thinks search will die. With global warming as a looming threat, Asta Roseway , principal research designer, says by 2027 famers will use AI to maintain healthy crop yields, even with "climate change, drought, and disaster."

Low-energy farming solutions, like vertical farming and aquaponics, will also be essential to keeping the food supply high, she says. You can view all 17 predictions here

.

[Dec 26, 2016] Neoliberalims led to impoverishment of lower 80 pecent of the USA population with a large part of the US population living in a third world country

Notable quotes:
"... Efforts which led to impoverishment of lower 80% the USA population with a large part of the US population living in a third world country. This "third world country" includes Wal-Mart and other retail employees, those who have McJobs in food sector, contractors, especially such as Uber "contractors", Amazon packers. This is a real third world country within the USA and probably 50% population living in it. ..."
"... While conversion of electricity supply from coal to wind and solar was more or less successful (much less then optimists claim, because it requires building of buffer gas powered plants and East-West high voltage transmission lines), the scarcity of oil is probably within the lifespan of boomers. Let's say within the next 20 years. That spells deep trouble to economic growth as we know it, even with all those machinations and number racket that now is called GDP (gambling now is a part of GDP). And in worst case might spell troubles to capitalism as social system, to say nothing about neoliberalism and neoliberal globalization. The latter (as well as dollar hegemony) is under considerable stress even now. But here "doomers" were wrong so often in the past, that there might be chance that this is not inevitable. ..."
"... Shale gas production in the USA is unsustainable even more then shale oil production. So the question is not if it declines, but when. The future decline (might be even Seneca Cliff decline) is beyond reasonable doubt. ..."
Dec 26, 2016 | economistsview.typepad.com

ilsm -> pgl... December 26, 2016 at 05:12 AM

"What is good for wall st. is good for America". The remains of the late 19th century anti trust/regulation momentum are democrat farmer labor wing in Minnesota, if it still exists. An example: how farmers organized to keep railroads in their place. Today populists are called deplorable, before they ever get going.

And US' "libruls" are corporatist war mongers.

Used to be the deplorable would be the libruls!

Division!

likbez -> pgl...

I browsed it and see more of less typical pro-neoliberal sentiments, despite some critique of neoliberalism at the end.

This guy does not understand history and does not want to understand. He propagates or invents historic myths. One thing that he really does not understand is how WWI and WWII propelled the USA at the expense of Europe. He also does not understand why New Deal was adopted and why the existence of the USSR was the key to "reasonable" (as in "not self-destructive" ) behaviour of the US elite till late 70th. And how promptly the US elite changed to self-destructive habits after 1991. In a way he is a preacher not a scientist. So is probably not second rate, but third rate thinker in this area.

While Trumpism (aka "bastard neoliberalism") might not be an answer to challenges the USA is facing, it is definitely a sign that "this time is different" and at least part of the US elite realized that it is too dangerous to kick the can down the road. That's why Bush and Clinton political clans were sidelined this time.

There are powerful factors that make the US economic position somewhat fragile and while Trump is a very questionable answer to the challenges the USA society faces, unlike Hillary he might be more reasonable in his foreign policy abandoning efforts to expand global neoliberal empire led by the USA.

Efforts which led to impoverishment of lower 80% the USA population with a large part of the US population living in a third world country. This "third world country" includes Wal-Mart and other retail employees, those who have McJobs in food sector, contractors, especially such as Uber "contractors", Amazon packers. This is a real third world country within the USA and probably 50% population living in it.

Add to this the decline of the US infrastructure due to overstretch of imperial building efforts (which reminds British empire troubles).

I see several factors that IMHO make the current situation dangerous and unsustainable, Trump or no Trump:

1. Rapid growth of population. The US population doubled in less them 70 years. Currently at 318 million, the USA is the third most populous country on earth. That spells troubles for democracy and ecology, to name just two. That might also catalyze separatists movements with two already present (Alaska and Texas).

2. Plato oil. While conversion of electricity supply from coal to wind and solar was more or less successful (much less then optimists claim, because it requires building of buffer gas powered plants and East-West high voltage transmission lines), the scarcity of oil is probably within the lifespan of boomers. Let's say within the next 20 years. That spells deep trouble to economic growth as we know it, even with all those machinations and number racket that now is called GDP (gambling now is a part of GDP). And in worst case might spell troubles to capitalism as social system, to say nothing about neoliberalism and neoliberal globalization. The latter (as well as dollar hegemony) is under considerable stress even now. But here "doomers" were wrong so often in the past, that there might be chance that this is not inevitable.

3. Shale gas production in the USA is unsustainable even more then shale oil production. So the question is not if it declines, but when. The future decline (might be even Seneca Cliff decline) is beyond reasonable doubt.

4. Growth of automation endangers the remaining jobs, even jobs in service sector . Cashiers and waiters are now on the firing line. Wall Mart, Shop Rite, etc, are already using automatic cashiers machines in some stores. Wall-Mart also uses automatic machines in back office eliminating staff in "cash office".

Waiters might be more difficult task but orders and checkouts are computerized in many restaurants. So the function is reduced to bringing food. So much for the last refuge of recent college graduates.

The successes in speech recognition are such that Microsoft now provides on the fly translation in Skype. There are also instances of successful use of computer in medical diagnostics. https://en.wikipedia.org/wiki/Computer-aided_diagnosis

IT will continue to be outsourced as profits are way too big for anything to stop this trend.

[Dec 26, 2016] Michigan Lets Autonomous Cars On Roads Without Human Driver

Notable quotes:
"... Companies can now test self-driving cars on Michigan public roads without a driver or steering wheel under new laws that could push the state to the forefront of autonomous vehicle development. ..."
Dec 26, 2016 | tech.slashdot.org
(go.com) 166

Posted by msmash on Friday December 09, 2016 @01:00PM from the it's-coming dept.

Companies can now test self-driving cars on Michigan public roads without a driver or steering wheel under new laws that could push the state to the forefront of autonomous vehicle development.

From a report on ABC:

The package of bills signed into law Friday comes with few specific state regulations and leaves many decisions up to automakers and companies like Google and Uber. It also allows automakers and tech companies to run autonomous taxi services and permits test parades of self-driving tractor-trailers as long as humans are in each truck . And they allow the sale of self-driving vehicles to the public once they are tested and certified, according to the state. The bills allow testing without burdensome regulations so the industry can move forward with potential life-saving technology, said Gov. Rick Snyder, who was to sign the bills. "It makes Michigan a place where particularly for the auto industry it's a good place to do work," he said.

[Dec 26, 2016] Googles DeepMind is Opening Up Its Flagship Platform To AI Researchers Outside the Company

Dec 26, 2016 | tech.slashdot.org
(businessinsider.com) 22

Posted by msmash on Monday December 05, 2016 @12:20PM from the everyone-welcome dept.

Artificial intelligence (AI) researchers around the world will soon be able to use DeepMind's "flagship" platform to develop innovative computer systems that can learn and think for themselves .

From a report on BusinessInsider:

DeepMind, which was acquired by Google for $400 million in 2014, announced on Monday that it is open-sourcing its "Lab" from this week onwards so that others can try and make advances in the notoriously complex field of AI.

The company says that the DeepMind Lab, which it has been using internally for some time, is a 3D game-like platform tailored for agent-based AI research. [...]

The DeepMind Lab aims to combine several different AI research areas into one environment. Researchers will be able to test their AI agent's abilities on navigation, memory, and 3D vision, while determining how good they are at planning and strategy.

[Dec 26, 2016] Here is my top 5 backup tools in Linux

March 26, 2016 OSTechNix

Data is the backbone of a Company. So, performing backup on regular intervals is one of the vital role of a system administrator. Here is my favourite five backup tools that I use mostly. I won't say these are the best, but these are the backup tools which I considered first when it comes to data backup.

Let me explain some of my preferred backup tools.

1. BACULA

BACULA is a power full backup tool . It is easy to use and efficient in recovering of loss data and damaged files in the local system and remotely. It having rich user interface( UI ) . It works on different cross platforms like windows, and Mac OS X.

Concerning about BACULA features, I can list the following:

  1. SD-SD replication.
  2. Enterprise binaries avaliable for univention.
  3. Restore performance improved for hard data files.
  4. Periodic status on running jobs in Director status report.

BACULA has the following components.

2. FWBACKUPS

FWBACKUPS is the easiest of all backup tools in linux. It having the rich user interface, and also it is a cross platform tool.

One of the notable feature of FWBACKUPS is remote backup. We can backup data from various systems remotely.

FWBACKUPS having some features are listed below.

  1. Simple Interface – Backup and restoring the documents is simple for user.
  2. Cross – platform – It's supports different platforms like windows, and Mac OS X. It restores the data on one system and restores into another system.
  3. Remote backup – All types of files can handle remotely.
  4. Scheduled Backups – Run a backup once or periodically.
  5. Speed – Backups moves faster by copying only the changes.
  6. Organized and clean – It takes care about organized data and removal of expired one. It list the backup to restore from which list of date.
3. RSYNC

rsync

RSYNC is a widely used tool for backups in linux. It is a command line backup tool. RSYNC is used to collect data remotely and locally. It is mainly used for automated backup. We can automate backup jobs with scripts.

Some of the notable features are listed below:

  1. It can update whole directory trees and filesystems.
  2. It uses ssh, rsh or direct sockets as the transport.
  3. Supports anonymous rsync which is ideal for mirroring.
  4. We can set bandwidth limit and file size.
4. URBACKUP

URBACKUP is a client/server backup system. It's efficient in client/server backup system for both windows and linux environments. File and image backups are made while the system is running without interrupting current process.

Here is the some features of this tool:

  1. whole partition can be saved as single directory.
  2. Image and file backup are made while system is running.
  3. Fast file and image transmission.
  4. Clients have the flexibility to change the settings like backup frequency. Next to no configuration.
  5. Web interface of URBACKUP is good in showing the status of the clients, current status of backup issues.
5. BACKUP PC

backuppc

BACKUP PC is high performance, enterprise-grade backup tool. It is a high configurable and easy to install, use and maintain.

It reduces the cost of the disks and raid system. BACKUP PC is written in perl language and extracts data using Samba service.

It is robust, reliable, well documented and freely available as open source on Sourceforge .

Features:

  1. No client side software needed. The standard smb protocol is used to extract backup data.
  2. A powerful web interface provides log details to view log files, configuration, current status and allows user to initiate and cancelled backups and browse and restore files from backups.
  3. It supports mobile environment where laptops are only intermittently connected to the network and have dynamic IP address.
  4. Users will receive email remainders if their pc has not recently been backed up.
  5. Open source and freely available under GPL.

These are the top backup tools that I use mostly. What's your favourite? Let us know in the comment section below.

Thanks for stopping by.

Cheers!

[Dec 26, 2016] Interns At Tech Companies Are Better Paid Than Most American Workers

Dec 26, 2016 | tech.slashdot.org
(qz.com) 158 Posted by msmash on Monday December 05, 2016 @03:00PM from the something-worth-pondering dept. According to a survey conducted by Jesse Collins, a senior at Purdue University and former Yelp intern, interns at tech companies make much more money on an annualized basis than workers in the vast majority of other occupations . From a report on Quartz: About 300 of the nearly 600 people who responded to the survey said they had received internship offers from big companies like Facebook, Twitter, Yelp, and Goldman Sachs for 2017. On average, the internship recipients said they would be paid $6,500 per month, the equivalent of $78,000 per year (the survey is still open, so results may change). Many also said they would receive more than $1,000 worth of stipends per month for housing and travel or signing bonuses. Internships typically run for a summer, but we've annualized the numbers. If the average intern who responded to Collins' survey were to work for a year, he would make $30,000 more than the average annual income for all occupations in the U.S., which is $48,000. Of the 1,088 occupation categories within which the Bureau of Labor Statistics tracks average income, workers in only about 200 of them on average make more money in a year than the intern would.

[Dec 05, 2016] Why I left Google -- discussion

Notable quotes:
"... "Google was the rich kid who, after having discovered he wasn't invited to the party, built his own party in retaliation," Whittaker wrote. "The fact that no one came to Google's party became the elephant in the room." ..."
"... Isn't it inevitable that Google will end up like Microsoft. A brain-dead dinosaur employing sycophantic middle class bores, who are simply working towards a safe haven of retirement. In the end Google will be passed by. It's not a design-led innovator like Apple: it's a boring, grey utilitarian, Soviet-like beast. Google Apps are cheap - but very nasty - Gmail is a terrible UI - and great designers will never work for this anti-design/pro-algorithms empire. ..."
"... All of Google's products are TERRIBLE except for Gmail, and even that is inferior to Outlook on the web now. ..."
"... I used Google Apps for years, and Google just doesn't listen to customers. The engineers that ran the company needed some corporate intervention. I just think Larry Page tried to turn Google into a different company, rather than just focusing the great ideas into actually great products. ..."
"... It seems the tech titans all have this pendulum thing going on. Google appears to be beginning its swing in the "evil" direction. ..."
"... You claim old Google empowered intelligent people to be innovative, with the belief their creations would prove viable in the marketplace. You then go on to name Gmail and Chrome as the accomplishments of that endeavour. Are you ****** serious? ..."
"... When you arrived at Google it had already turned the internet into a giant spamsense depository with the majority of screen real estate consumed by Google's ads. The downhill spiral did not begin with Google+, but it may end there. On a lighter note, you are now free. Launch a start-up and fill the gaping hole which will be left by the fall of the former giant. ..."
"... Great post. Appreciate the insights the warning about what happens when bottom-up entrepreneurship loses out to top-down corporate dictums. ..."
"... The ability to actually consume shared content in an efficient and productive manner is still as broken as ever. They never addressed the issue in Buzz and still haven't with G+ despite people ranting at them for this functionality forever. ..."
"... Sounds like Google have stopped focusing on what problem they're solving and moving onto trying to influence consumer behaviour - always a much more difficult trick to pull off. Great article - well done for sharing in such a humble and ethical manner. Best of luck for the future. ..."
Apr 02, 2012 | JW on Tech

Whittaker, who joined Google in 2009 and left last month, described a corporate culture clearly divided into two eras: "Before Google+," and "After."

"After" is pretty terrible, in his view.

Google (GOOG, Fortune 500) once gave its engineers the time and resources to be creative. That experimental approach yielded several home-run hits like Chrome and Gmail. But Google fell behind in one key area: competing with Facebook.

That turned into corporate priority No. 1 when Larry Page took over as the company's CEO. "Social" became Google's battle cry, and anything that didn't support Google+ was viewed as a distraction.

"Suddenly, 20% meant half-assed," wrote Whittaker, referring to Google's famous policy of letting employees spend a fifth of their time on projects other than their core job. "The trappings of entrepreneurship were dismantled."

Whittaker is not the first ex-Googler to express that line of criticism. Several high-level employees have left after complaining that the "start-up spirit" of Google has been replaced by a more mature but staid culture focused on the bottom line.

The interesting thing about Whittaker's take is that it was posted not on his personal blog, but on an official blog of Microsoft (MSFT, Fortune 500), Google's arch nemesis.

Spokesmen from Microsoft and Google declined to comment.

The battle between Microsoft and Google has heated up recently, as the Federal Trade Commission and the European Commission begin to investigate Google for potential antitrust violations. Microsoft, with its Bing search engine, has doubled its share of the search market since its June 2010 founding, but has been unsuccessful at taking market share away from Google.

Microsoft is increasingly willing to call out Google for what it sees as illicit behavior. A year ago, the software company released a long list of gripes about Google's monopolistic actions, and last month it said Google was violating Internet Explorer users' privacy.

Despite his misgivings about what Google cast aside to make Google+ a reality, Whittaker thinks that the social network was worth a shot. If it had worked -- if Google had dramatically changed the social Web for the better -- it would have been a heroic gamble.

But it didn't. It's too early to write Google+ off, but the site is developing a reputation as a ghost town. Google says 90 million people have signed up, but analysts and anecdotal evidence show that fairly few have turned into heavy users.

"Google was the rich kid who, after having discovered he wasn't invited to the party, built his own party in retaliation," Whittaker wrote. "The fact that no one came to Google's party became the elephant in the room."

Ian Smith:

Isn't it inevitable that Google will end up like Microsoft. A brain-dead dinosaur employing sycophantic middle class bores, who are simply working towards a safe haven of retirement. In the end Google will be passed by. It's not a design-led innovator like Apple: it's a boring, grey utilitarian, Soviet-like beast. Google Apps are cheap - but very nasty - Gmail is a terrible UI - and great designers will never work for this anti-design/pro-algorithms empire.

Steve

I have to be honest with you. All of Google's products are TERRIBLE except for Gmail, and even that is inferior to Outlook on the web now.

I used Google Apps for years, and Google just doesn't listen to customers. The engineers that ran the company needed some corporate intervention. I just think Larry Page tried to turn Google into a different company, rather than just focusing the great ideas into actually great products.

Matt:

It seems the tech titans all have this pendulum thing going on. Google appears to be beginning its swing in the "evil" direction. Apple seems like they're nearing the peak of "evil".

And Microsoft seems like they're back in the middle, trying to swing up to the "good" side. So, if you look at it from that perspective, Microsoft is the obvious choice.

Good luck!

VVR:

The stark truth in this insightful piece is the stuff you have not written..

Atleast you had a choice in leaving google. But we as users don't.

I have years of email in Gmail and docs and youtube etc. I can't switch.

"Creepy" is not the word that comes to mind when Ads for Sauna, online textbooks, etc suddenly begin to track you, no matter which website you visit.

You know you have lost when this happens..

David:

A fascinating insight, I think this reflects what a lot of people are seeing of Google from the outside. It seems everybody but Page can see that Google+ is - whilst technically brilliant - totally superfluous; your daughter is on the money. Also apparent from the outside is the desperation that surrounds Google+ - Page needs to face facts, hold his hands up and walk away from Social before they loose more staff like you, more users and all the magic that made Google so great.

Best of luck with your new career at Microsoft, I hope they foster and encourage you as the Google of old did.

Raymond Traylor:

I understand Facebook is a threat to Google search but beating Facebook at their core competency was doomed to fail. Just like Bing to Google. I was so disappointed in Google following Facebook's evil ways of wanting to know everything about me I've stopped using their services one at a time, starting with Android.

I am willing to pay for a lot of Google's free service to avoid advertising and harvesting my private data.

root

You claim old Google empowered intelligent people to be innovative, with the belief their creations would prove viable in the marketplace. You then go on to name Gmail and Chrome as the accomplishments of that endeavour. Are you ****** serious?

Re-branding web based email is no more innovative than purchasing users for your social networking site, like Facebook did. Same for Chrome, or would you argue Google acquiring VOIP companies to then provide a mediocre service called Google Voice was also innovative?

When you arrived at Google it had already turned the internet into a giant spamsense depository with the majority of screen real estate consumed by Google's ads. The downhill spiral did not begin with Google+, but it may end there. On a lighter note, you are now free. Launch a start-up and fill the gaping hole which will be left by the fall of the former giant.

RBLevin:

Great post. Appreciate the insights the warning about what happens when bottom-up entrepreneurship loses out to top-down corporate dictums.

Re: sharing, while I agree sharing isn't broken (heck, it worked when all we had was email), it certainly needs more improvement. I can't stand Facebook. Hate the UI, don't care for the culture. Twitter is too noisy and, also, the UI sucks. I'm one of those who actually thinks Google+ got 21st century BBSing right.

But if that's at the cost of everything else that made Google great, then it's a high price to pay.

BTW, you can say a lot of these same things about similar moves Microsoft has made over the years, where the top brass decided they knew better, and screwed over developers and their investments in mountains of code.

So, whether it happens in an HR context or a customer context, it still sucks as a practice.

bound2run:

I have made a concerted effort to move away from Google products after their recent March 1st privacy policy change. I must say the Bing is working just fine for me. Gmail will be a bit tougher but I am making strides. Now I just need to dump my Android phone and I will be "creepy-free" ... for the time being.

Phil Ashman:

The ability to actually consume shared content in an efficient and productive manner is still as broken as ever. They never addressed the issue in Buzz and still haven't with G+ despite people ranting at them for this functionality forever.

Funny that I should read your post today as I wrote the following comment on another persons post a couple days back over Vic's recent interview where someone brought up the lack of a G+ API:

"But if it were a social network.......then they are doing a pretty piss poor job of managing the G+ interface and productive consumption of the stream. It would be nice if there was at least an API so some 3rd party clients could assist with the filtering of the noise, but in reality the issue is in the distribution of the stream. What really burns me is that it wouldn't be that hard for them to create something like subscribable circles.

Unfortunately the reality is that they just don't care about whether the G+ stream is productive for you at the moment as their primary concern isn't for you to productively share and discuss your interests with the world, but to simply provide a way for you to tell Google what you like so they can target you with advertising. As a result, the social part of Google+ really isn't anything to shout about at the moment."

You've just confirmed my fear about how the company's focus has changed.

Alice Wonder:

Thanks for this. I love many of the things Google has done. Summer of code, WebM, Google Earth, free web fonts, etc.

I really was disappointed with Google+. I waited for an invite, and when I finally got one, I started to use it. Then the google main search page started to include google+ notifications, and the JS crashed my browser. Repeatedly. I had to clear my cache and delete my cookies just so google wouln't know it was me and crash search with a notification. They fixed that issue quickly but I did not understand why they would risk their flagship product (search) to promote google plus. The search page really should be a simple form.

And google plus not allowing aliases? Do I want a company that is tracking everything I do centrally to have my real name with that tracking? No. Hence I do not use google+ anymore, and am switching to a different search engine and doing as little as I can with google.

I really don't like to dislike google because of all they have done that was cool, it is really sad for me to see this happening.

Mike Whitehead

Sounds like Google have stopped focusing on what problem they're solving and moving onto trying to influence consumer behaviour - always a much more difficult trick to pull off. Great article - well done for sharing in such a humble and ethical manner. Best of luck for the future.

jmacdonald 14 Mar 2012 4:07 AM great write-up

personally i think that google and facebook have misread the sociological trend against the toleration of adverts, to such an extent that if indeed google are following the 'facebook know everything and we do too' route, i suspect both companies may enter into issues as the advertising CPMs fall and we're left with us wretched consumers who find ways around experiences that we don't want

more on this stuff here: www.jonathanmacdonald.com

and here: www.jonathanmacdonald.com

for anyone that cares about that kinda angle

Mahboob Ihsan:

Google products are useful but probably they could have done more to improve the GUI, Standardization and Usability. You can continue to earn business in short term enjoying your strategic advantage as long as you don't have competitors. But as soon as you have just one competitor offering quality products at same cost, your strategic advantage is gone and you have to compete through technology, cost and quality. Google has been spreading its business wings to so many areas, probably with the single point focus of short term business gains. Google should have learnt from Apple that your every new offering should be better (in user's eye) than the previous one.

Victor Ramirez:

Thanks for the thoughtful blog post. Anybody who has objectively observed Google's behavior and activity over the past few years has known that Google is going in this direction. I think that people have to recognize that Google, while very technically smart, is an advertising company first and foremost. Their motto says the right things about being good and organizing the world's information, but we all know what Google is honestly interested in. The thing that Google is searching for, more than almost anything else, is about getting more data about people so they can get people better ads they'll be more likely to click on so they make more money. Right now, Google is facing what might be considered an existential threat from Facebook because they are the company that is best able to get social data right now. Facebook is getting so much social data that odds are that they're long-term vision is to some point seriously competing in search using this social data that they have. Between Facebook's huge user-base and momentum amongst businesses (just look at how many Super Bowl ads featured Facebook pages being promoted for instance, look at the sheer number of companies listed at www.buyfacebookfansreviews.com that do nothing other than promote Facebook business pages, and look at the biggest factor out there - the fact that Facebook's IPO is set to dominate 2012) I think that Facebook has the first legitimate shot of creating a combination of quality results and user experience to actually challenge Google's dominance, and that's pretty exciting to watch. The fact that Google is working on Google+ so much and making that such a centerpiece of their efforts only goes to illustrate how critical this all is and how seriously they take this challenge from Facebook into their core business. I think Facebook eventually enters the search market and really disrupts it and it will be interesting to see how Google eventually acts from a position of weakness.

Keith Watanabe:

they're just like any company that gets big. you end up losing visibility into things, believe that you require the middle management layer to coordinate, then start getting into the battlegrounds of turf wars because the people hired have hidden agendas and start bringing in their army of yes men to take control as they attempt to climb up the corporate ladder. however, the large war chest accumulated and the dominance in a market make such a company believe in their own invulnerability. but that's when you're the most vulnerable because you get sloppy, forget to stop and see the small things that slip through the cracks, forget your roots and lose your way and soul. humility is really your only constant savior.

btw, more than likely Facebook will become the same way. And any other companies who grow big. People tend to forget about the days they were struggling and start focusing on why they are so great. You lose that hunger, that desire to do better because you don't have to worry about eating pinches of salt on a few nibbles of rice. This is how civilization just is. If you want to move beyond that, humans need to change this structure of massive growth -> vanity -> decadence -> back to poverty.

Anon:

This perceived shift of focus happens at every company when you go from being an idealistic student to becoming an adult that has to pay the bills. When you reach such a large scale with so much at stake, it is easy to stop innovating. It is easy to get a mix of people who don't share the same vision when you have to hire on a lot of staff. Stock prices put an emphasis on perpetual monetization. Let's keep in mind that Facebook only recently IPO'd and in the debate for personal privacy, all the players are potentially "evil" and none of them are being held to account by any public policy.

The shutdown of Google Labs was a sad day. Later the shutdown of Google Health I thought was also sad as it was an example of a free service already in existence, akin to what Ontario has wasted over $1 billion on for E-Health. Surely these closures are a sign that the intellectual capital in the founders has been exhausted. They took their core competencies to the maximum level quickly, which means all the organic growth in those areas is mostly already realized.

There needs to be some torch passing or greater empowerment in the lower ranks when things like this happen. Take a look at RIM. Take a look at many other workplaces. It isn't an isolated incident. There are constantly pressures between where you think your business should go, where investors tell you to go, and where the industry itself is actually headed. This guy is apparently very troubled that his name is attached to G+ development and he is trying to distance himself from his own failure. Probably the absence of Google Labs puts a particular emphasis on the failure of G+ as one of the only new service projects to be delivered recently.

After so much time any company realizes that new ideas can only really come with new people or from outside influences. As an attempt to grow their business services via advertising, the idea that they needed to compete with Facebook to continue to grow wasn't entirely wrong. It was just poorly executed, too late, and at the expense of potentially focusing their efforts on doing something else under Google Labs that would have been more known as from them (Android was an acquisition, not organically grown internally). There is no revolution yet, because Facebook and Google have not replaced any of each others services with a better alternative

The complaints in the final paragraph of the blog regarding privacy are all complaints about how much Google wants to be Facebook. Thing is that Google+ just like all the aforementioned services are opt-in services with a clear ToS declared when you do so, even if you already have a Google account for other services. The transparency of their privacy policy is on par if not better than most other competing service providers. The only time it draws criticism is when some changes have been made to say that if you use multiple services, they may have access to the same pool of information internally. It's a contract and it was forced to be acknowledged when it changed. When advertising does happen it is much more obvious to me that it is advertising via a Google service, than when Facebook decides to tell me who likes what. Not to give either the green light here; but the evolution is one of integrating your network into the suggestions, and again, it isn't isolated to any one agency.

One way to raise and enforce objections to potential mishandling of information is to develop a blanket minimum-requirement on privacy policy to apply to all businesses, regarding the handling of customer information. We are blind if we think Google+ and Facebook are the only businesses using data in these ways. This blanket minimum requirement could be voluntarily adopted via 3rd party certification, or it could be government enforced; but the point is that someone other than the business itself would formulate it, and it must be openly available to debate and public scrutiny/revision. It is a sort of "User License Agreement" for information about us. If James Whittaker left to partake in something along these lines, it sure would make his blog entry more credible, unless Microsoft is focused so much more greatly on innovation than the profit motive.

It is also important for customers and the general public not to get locked into any kind of brand loyalty. One problem is Facebook is a closed proprietary system with no way to forward or export the data contained within it to any comparable system. Google is a mish-mash of some open and some closed systems. In order for us as customers to be able to voice our opinions in a way that such service providers would hear, we must be provided alternatives and service portability.

As an example of changing service providers, there has been an exodus of business customers away from using Google Maps as they began charging money to businesses that want to use the data to develop on top of it. I think that this is just the reality of a situation when you have operating costs for a service that you need to recoup; but there is a royalty-free alternative like Open Street Map (which Apple has recently ripped off by using Open Street Map data without attribution).

Google won't see the same meteoric growth ever again. It probably is a less fun place for a social media development staffer to work at from 2010 to present, than it was from 2004 - 2010 (but I'm betting still preferable to FoxConn or anything anywhere near Balmer).

Linda R. Tindall :

Thank you for your honest comments Mr. Whittaker. And yes, Google is not like it was before..

It is Scary, Google may destroy anyone online business overnight!

Google penalize webmasters if they don't like a Website for any reason. They can put out anyone they want out of business. How does Google judge a webmaster's?

Google's business isn't anymore the search engine. Google's business is selling and displaying ads.

GOOGLE becomes now the Big Brother of the WWW. I think it is scary that Google has so much power. Just by making changes, they can ruin people's lives.

As it turned out, sharing was not broken. Sharing was working fine and dandy, Google just wasn't part of it. People were sharing all around us and seemed quite happy. A user exodus from Facebook never materialized. I couldn't even get my own teenage daughter to look at Google+ twice, "social isn't a product," she told me after I gave her a demo, "social is people and the people are on Facebook."

Google was the rich kid who, after having discovered he wasn't invited to the party, built his own party in retaliation. The fact that no one came to Google's party became the elephant in the room.

[Nov 12, 2016] How a Programmer Gets By On $16K-Yr He Moves to Malaysia

Slashdot

Interview With A Digital Migrant Meet John Hunter - Unchartered Waters

John Hunter

"Move to low cost-of-living area of the world, set up shop working remote, work ten hours a week while building a huge nest egg."

Whole books have been published on this model, along with terms like "The Nouveau Rich", people who get to earn wealth while enjoying the easy life.

And yet …

It seems to never actually happen.

Or, at least, it doesn't seem to happen much. Often the people living the "Jimmy Buffett Life" are already millionaires living off interest. Often the person speaking is selling something (perhaps a dream) more than a reality. We can do better.

Then I met John Hunter and learned about his technology business.

John is not independently wealthy. He did not have a big IPO, and does not have have a revenue stream. Nor does he have a best-selling book on, say, how to live cheap.

Instead, he was a practicing programmer and IT program manager who moved from Virginia to Malaysia, on the expectation of taking a year long "sabbatical," and, if he could find a way to make it work, to stay a bit longer.

And Now. John has been in Malaysia for a bit over a year now, with no sign of returning anytime soon.

I thought he was worth talking to, and sharing here.

by Stiletto (12066) writes: on Monday March 18, @06:44PM ( #43208399)

It's not causal. Working long hours does not cause you to be highly paid or wealthy. If that were true, all a vegetable picker would have to do is work 120 hours a week and retire in comfort. A CEO does not make 800X what his average staff makes because he works 800 times as long.

Sadly, on average, the most accurate predictor of someone's income is their father's income.

Re:30 hours per week? (Score:5, Insightful)

by Anonymous Coward on Monday March 18, @08:33PM (#43209445)

It does if you count high school and college and grad school. Tell me, how hard do you think those vegetable pickers were working when they were 16? Were they staying late after school to learn Calculus, or were they cutting class to get high with their homies? Working hard and being smart actually matter. Anyone who says otherwise has never tried either working hard or being smart.

They were probably too busy picking vegetables at 16. They probably study when they can. College? Grad School? With fruit pickin' parents? You really are disconnected from the real world down here aren't you.

Funny, most business owners/CEO's I've met are decent with basic algebra but weak when it comes to calculus, trig, etc.

A.) They know people. (usually part of a boys club at an expensive university)
B.) Have rich fathers
C.) Work hard.

Pick any two of the above and it will fit most CEO's... they also have to be willing to make hard choices at the expense of others to further their agenda.... or perform some CYA.

No, it's because what he knows is 800x as valuable. Not all work is equal. That's one of the many flaws in Marx's philosophy.

So Carly Fiorina's contributions were worth more than a seasoned electronics engineer with 25 years of experience? I think not.

A CEO is a corporate face undeserving of being put on a pedestal unless they built the company they are running with their bare hands in the beginning.

Sadly, on average, the most accurate predictor of someone's income is their father's income.

That's because income is dependent on intelligence and hard work. Intelligence is highly heritable and appreciation for hard work is handed down in successful families.

Spoken like a true wannabe aristocrat. The possibility of being intelligent may be inherited, but actual intelligence isn't. Most trust fund babies I've ever met have been pretty useless except for office political gain. Breeding has nothing to do with being fit for the job.

The hard work that's handed down is soaked in the blood of the people who actually worked for it that were desperate enough to allow themselves to be exploited.

Re:30 hours per week? (Score:5, Funny)

by ColdSam (884768) on Monday March 18, @08:55PM (#43209631)

Ok let's test you theory:

Nicely done. Your rigorous analysis sure proved him wrong.

It takes skill, hard work, ruthless ambition and extreme good luck to get rich and stay there.

Is it okay if we use your method on your own theory?

Paris Hilton - nope

George Bush - nope

gordo3000 Re:30 hours per week? (Score:3)

do you have any sources that say the majority of wealth in this country was inherited? A skimming the forbes richest list show quite a few people who didn't get there via just cashing an inheritance and others who did (walton's family, as the obvious top examples).

I've generally found in 3rd world countries the amount that inheritance matters and who you know matters far more than in the US. In fact, I think the US is the most democratic on this (I've traveled quite a bit and lived abroad, but I've never done hard research, so I'm not saying this is research). But that is relative across countries, to speak towards working towards your wealth and inheriting in an absolute manner is a hard research question. here is what I mean:

Most upper income people I know got there by hard work, but that is not saying parents didn't help them a lot and that those advantages can be passed down. I know doctors and lawyers (as two high income profession classes), all worked hard to get there, some had lots of family money getting them through, others didn't. What I have found is that having family money helps make up for lower qualifications, making upper income a sticky level to be at (though this doesn't speak to super rich families where children never need to work). A great example is poor performance in college. Upper income people can afford to have their children stay in college longer, pay more for tuition, or send them abroad for medical school to get them to being a doctor. Lower income people just have to fight it out onshore.

Re:30 hours per week? (Score:4, Interesting)

by SpaceMonkies (2868125) on Monday March 18, @09:19PM (#43209821)

If you can make $15/hr remotely, I'd suggest Montenegro. Find a place near the sea, you got it made. You might have to work at getting a really great broadband deal, but there are some to be had. If you're single, the women there are beautiful and have sexy accents, you've got the sea and off-season the tourists go away and you can really enjoy the good life. You're a short hop from shopping in Italy, skiing in the Alps and you're still not in the EU (yet). Learn to play tuba in a Balkan horn band. Drink lots of coffee and slivovitza. Go out in your backyard and pick fresh figs for breakfast. Even if swimming in crystal-blue seas is not your idea of fun, you can set yourself down in a sidewalk cafe and watch one Mila Jovovic after another walk by. And there's none of the snobbiness of Western Europe.

Re:

I know that a lot of janitors/custodial service as well as bus drivers that do that. Some people in retail and "health care"/assisted living places work that much too. White collar workers mostly have it "easier" because their jobs pay far more. 50 hours is a bit low, since many of them work weekends too.

udachny

Entrepreneurs!

Entrepreneurs never stop working, off hours, weekends, holidays, those are just words, they don't mean anything when you run a business.

ph1ll

... and I can't find a country called Malysia (please note, editors: it's Malaysia).

I know Malaysia well (even though I live in the UK). I first went there in '97 and married a Malaysian-born woman. Some observations: They really like and respect white people. They don't particularly like Chinese people (my wife is half Chinese so I see rampant discrimination against this large minority - about 25% of Malaysia's population - all the time). The weather is great (although sometimes a little too humid).

Kuala Lumpur is a very advanced city that can compare to anything in the West. Broadband speeds are so-so according to my cousin-in-law. There appears to be a demand for good engineers (according to another cousin-in-law, a Chinese who studied IT in England). So, assuming you can get a visa, getting some interesting work shouldn't be too hard. The political situation there is... interesting. But I get the impression that if you don't cause trouble you will be left alone - especially if you are white.

HTH

Re:What article (Score:5, Interesting)

tlhIngan:

I know Malaysia well (even though I live in the UK). I first went there in '97 and married a Malaysian-born woman. Some observations: They really like and respect white people. They don't particularly like Chinese people (my wife is half Chinese so I see rampant discrimination against this large minority - about 25% of Malaysia's population - all the time). The weather is great (although sometimes a little too humid). Kuala Lumpur is a very advanced city that can compare to anything in the West. Broadband speeds are so-so according to my cousin-in-law. There appears to be a demand for good engineers (according to another cousin-in-law, a Chinese who studied IT in England). So, assuming you can get a visa, getting some interesting work shouldn't be too hard. The political situation there is... interesting. But I get the impression that if you don't cause trouble you will be left alone - especially if you are white.

HTH

The reason is that after the war or so, the first people to start running businesses and such were Chinese (most likely chased out from Singapore by the Japanese), and they got very rich doing so.

The government exploits the fact that a lot of Malaysians are jealous of the Chinese for being successful (which happens because they worked hard at building businesses and such) , so they put up huge campaigns of national identity and such to encourage hatred of the Chinese. However, they government doesn't really do anything about it (they can't - said Chinese businesses pay a good amount of tax and employ a lot of Malays). So basically the Chinese are demonized for being successful and "exploiting" Malays

If you're white, you're usually a tourist or an investor, so you're treated well to get at your $$$. If you're a Chinese investor with $$$, everyone eyes you like you're going to enslave them.

The government feeds off this sentiment and basically just fans the flames. There's no real democracy (there is voting, but the opposition is usually highly discredited, or even arrested if they have a chance of winning - being a Muslim state, there are plenty of "crimes" that one can accuse the Opposition of).

Kagato

That closes the loop on what I noticed about the Chinese in Singapore hating the Japanese. I actually witnessed a shop keeper play dumb with a Japanese trying to buy something using Engrish. Old Japanese guy stormed out in frustration. I go to buy something, no problem, he explained the other guy was Japanese.

I don't know if I would choose Malaysia or Singapore though. Both are kind of strict countries if you run afoul of the local powers that be. Fun to visit, not so much on the living there. I'd hit up Belize. Nice locals, cheap and only 1 hour plane ride to Miami if the shit goes down.

[Nov 12, 2016] How Did You Become a Linux Professional - Slashdot

linux.slashdot.org
August 26, 2012 | Ask Slashdot

fahrbot-bot

Just another tool in the box...

August 26, 2012

I've spent 1/2 my 25+ year career as a "Unix" (you know what I mean) system administrator and the other 1/2 as a Unix system programmer, sometimes application programmer, all with a little (sigh) DOS/Windows thrown in. I've worked on just about every flavor of Unix running on PC class to Cray-2 hardware, usually several at any one time. For most of that time, there were no books on the topics, just man pages and the compiler. Linux is just another tool in my toolbox.

It seems almost universal that every prospective employer only sees the "other" half of my experience - We want a sysadmin, but you're a programmer. We want a programmer, but you're a sysadmin. I simply tell them I do both and I do both well. Resume and references speak for themselves.

I got my first jobs at my university doing LISP research and working in the CS office. First real job because employer liked my school experience (did more than just took classes). It was small company and I did both system programming/admin (on 8 different versions of Unix). Second job, I bumped into professor from school and got job as both Unix system admin/programmer at NASA Langley (super computing network) and another contractor as sysadmin (100+ Sun/SGI workstations); then The New York Times for a few years as Unix sysadmin; now defense contractor (can't say who) for 11 years, because of friend from very first job. Now I work on primarily Solaris, Linux and (sigh, still) Windows systems as a system/application programmer - in about 10 different programming languages - and sysadmin when needed.

All in all, you learn what you need to know and what interests you - sometimes the weirder the better. You never know where it will lead.

There is one definitive: I hate Windows, especially Windows 7 - or as I call it "Windows for Dummies".

CAIMLAS (41445) writes:
it's a bag of tricks (Score:2)
'Linux professional,' I mean anyone in a paid IT position who uses or administers Linux systems on a daily basis.

Being a "Linux Professional" (or as people tend to more often call me, "Linux Guru", damn them) is more about a broad and deep level of experience than it is about 'knowing linux'. For instance, you're going to know the inner workings of how many protocols work; you're going to know how to build your own Linux distro (more or less), and you're going to know how hardware behaves properly. There are many 'professionals' who don't know this, but if you're specializing you've got to know pretty much everything.

Think: RHCE or similar.

Over the past five years, I've developed an affection for Linux, and use it every day as a freelance IT consultant. I've built a breadth of somewhat intermediate skills, using several distros for everything from everyday desktop use, to building servers from scratch, to performing data recovery. I'm interested in taking my skills to the next level - and making a career out of it - but I'm not sure how best to appeal to prospective employers, or even what to specialize in

You'll become a generalist unless you become a "Postfix Administrator" or something like that. That's the most likely first step. You will pick up your specialty over the years, largely depending on which type of systems you're working on.

(I refuse to believe the only option is 'sysadmin,' though I'm certainly not opposed to that).

That's not the only option, but it's the main and first one you'll have to master. Being an architect or systems specialist (mail, dns, filesystems, whatever) is the next step up. It takes a while to get there, and usually requires either a specialized company dealing exclusively with something in that domain, a very large corporation, or contracting.

Specifically, I'm interested in what practical steps I can take to build meaningful skills that an employer can verify, and will find valuable.

This is sorta "LOL". You assume that your employer cares more than anything other than a stable work history and/or specifically applicable experience to what you will be doing on a day in and out basis. It is a rare IT manager who cares more about this, even. Being highly skilled and capable, in a field where your skillset is in demand, is entirely different than being employable doing said work.

So, what do you do, and how did you get there? How did you conquer the catch-22 of needing experience to get the position that gives you the experience to get the position?

You know the right people, or you luck out and get a job in the field right after school. Part of lucking out is knowing the right people.

Every single IT job I've gotten has either been due to the employer being desperate because they have someone vacating a crucial position or expansive growth they can't manage, or through a friend. I've also not gotten jobs through friends, after failing interviews (not enough experience in such-and-such technology or the snap-judgement IT Director not liking me, or any number of other things.)

Did you get certified, devour books and manpages, apprentice under an expert, some combination of the above, or something else entirely?"

Everyone is different in this regard. I personally got a 4 year degree and spent many, many long hours devouring man pages, chatting on technical IRC, experimenting/pushing my envelope, and reading in general. That's the easy part. The hardest part of all of it is breaking into a linux-oriented job, IMO. If you're not in the right market, you've got to get yourself to that market before any of your experience even matters. Knowing the right people is, IMO, key. Personally, it took approximately 5 years of constant trying, experimentation with what works, etc. to get my first 'linux' job - and that was primarily a FreeBSD job, at that. Now I'm working primarily with AIX (just several years later). There is no golden bullet, here, and you will probably find it almost impossible to find your 'ideal' job. (Mine would be doing systems engineering/administration/management in an academic/scientific setting. I've done it for a year so far, and would love to get back to doing it again.)

[Nov 12, 2016] 9 Reasons Why Application Developers Think Their Manager Is Clueless

Aug 15, 2012 | Business Technology Leadership

"My manager is clueless." These are words you don't want to hear if you want to earn the respect of your application development professionals. So how do you avoid being a clueless manager? Steer clear of these behaviors:

1. The manager is a control nut. If you want to be a Controller then get a job in the accounting department. Okay, so maybe you are not a certifiable control nut. Maybe it is just a strategy you are employing because your direct reports can't get the job done. If this is the case, then control is not the solution. Have the courage to replace those managers that aren't strong. Control won't work in the long run anyway.

2. The manager is aloof. Stop thinking about your golf game. You may have a great team-strong individual managers and team chemistry-but your leadership is still necessary to keep things on course (not the golf course). Besides, no matter how much you practice, your golf game will still be mediocre, but you can be at the top of your game as manager if you work at it.

3. The manager gulps vendor Kool-Aid. Did you know that there are more than 34,750 registered lobbyists in Washington, D.C., for just 435 representatives and 100 senators? That's 64 lobbyists for each congressperson. I wonder how many vendor account managers there are per manager. You are smart enough to know that vendors are trying to sell you and you won't be fooled wholesale. Yeah right. Their influence can eat away at you without you even realizing it. Be even more skeptical than you are now. Just say no.

4. The manager is a technical dinosaur. Unless you are running for president of the United States, experience does matter. Technology has changed since you were writing RPG on the mainframe umpteen years ago. And for you younger guys who made your bones writing VB or Java Web apps, make sure you know why there is so much buzz about Ruby on Rails and multicore programming. Your ability to talk tech will go a long way to earning the respect of application development professionals.

5. The manager thinks changes can happen overnight. Sorry to have to break this to you: You are not a wizard and your magic wand doesn't work.

6. The manager doesn't know the difference between resources and talent. The fastest way to lose respect is to put clueless managers in charge. Clueless managers equal clueless managers. Can you ever imagine Doc Rivers, coach of the 2008 world champion Boston Celtics, talking about player resources like they were interchangeable? "I need two guard resources." "I need a center resource." No. Talent and teamwork make winning teams. Talent matters. Don't pay lip-service to talent. Find a way to locate and use the talent in your organization. You will only be as good as the team you assemble.

7. The manager collaborates to death. Whether it is the character flaw of being indecisive or some middle-school notion of democracy, you are in charge. Collaboration is critical, but you also need to make the right decision at the right time. Collaborate like Captain Kirk. "Spock?" "Bones?" He gets opinions from his experts but there is never any question about who will make the final decision. And, if you never watched Star Trek then you shouldn't even be a manager.

The real reason managers are clueless (Score:1)
by roster238 (969495) on Tuesday July 01, @08:33PM (#24024329)

The reason why managers are normally clueless is that you have to be competent to recognize incompetence. The folks who normally select senior IT managers are business people who have no understanding of technology whatsoever. The are in fact "technically" incompetent. They choose the person who scares them least.

They want the person with whom that can communicate without being bogged down with any complexities or difficult concepts. They will stay far away from the most knowledgeable and experienced people in favor of someone who laughs and jokes with them about the big game, a television show, or anything else they can understand. They will reject someone with the capability to understand a system from the ground up as not seeing the "big picture". This is why senior IT managers are often unsure that they are getting the bang for the buck that an IT department should bring to the table. It's because they pick the least qualified person to lead based on their personality rather than ability.

On the positive side the senior IT manager who wants to keep his job will find a few key people who really know what they doing and throw money at them to ensure that they don't leave. They are the folks who really run the department by proxy.

[Nov 12, 2016] Business Has Killed IT With Overspecialization

Apr 07, 2010 | Enterprise Networking Planet

What happened to the old "sysadmin" of just a few years ago? We've split what used to be the sysadmin into application teams, server teams, storage teams, and network teams. There were often at least a few people, the holders of knowledge, who knew how everything worked, and I mean everything. Every application, every piece of network gear, and how every server was configured -- these people could save a business in times of disaster.

Now look at what we've done. Knowledge is so decentralized we must invent new roles to act as liaisons between all the IT groups. Architects now hold much of the high-level "how it works" knowledge, but without knowing how any one piece actually does work. In organizations with more than a few hundred IT staff and developers, it becomes nearly impossible for one person to do and know everything. This movement toward specializing in individual areas seems almost natural. That, however, does not provide a free ticket for people to turn a blind eye.

Specialization

You know the story: Company installs new application, nobody understands it yet, so an expert is hired. Often, the person with a certification in using the new application only really knows how to run that application. Perhaps they aren't interested in learning anything else, because their skill is in high demand right now. And besides, everything else in the infrastructure is run by people who specialize in those elements. Everything is taken care of.

Except, how do these teams communicate when changes need to take place? Are the storage administrators teaching the Windows administrators about storage multipathing; or worse logging in and setting it up because it's faster for the storage gurus to do it themselves? A fundamental level of knowledge is often lacking, which makes it very difficult for teams to brainstorm about new ways evolve IT services. The business environment has made it OK for IT staffers to specialize and only learn one thing.

If you hire someone certified in the application, operating system, or network vendor you use, that is precisely what you get. Certifications may be a nice filter to quickly identify who has direct knowledge in the area you're hiring for, but often they indicate specialization or compensation for lack of experience.

Resource Competition

Does your IT department function as a unit? Even 20-person IT shops have turf wars, so the answer is very likely, "no." As teams are split into more and more distinct operating units, grouping occurs. One IT budget gets split between all these groups. Often each group will have a manager who pitches his needs to upper management in hopes they will realize how important the team is.

The "us vs. them" mentality manifests itself at all levels, and it's reinforced by management having to define each team's worth in the form of a budget. One strategy is to illustrate a doomsday scenario. If you paint a bleak enough picture, you may get more funding. Only if you are careful enough to illustrate the failings are due to lack of capital resources, not management or people. A manager of another group may explain that they are not receiving the correct level of service, so they need to duplicate the efforts of another group and just implement something themselves. On and on, the arguments continue.

Most often, I've seen competition between server groups result in horribly inefficient uses of hardware. For example, what happens in your organization when one team needs more server hardware? Assume that another team has five unused servers sitting in a blade chassis. Does the answer change? No, it does not. Even in test environments, sharing doesn't often happen between IT groups.

With virtualization, some aspects of resource competition get better and some remain the same. When first implemented, most groups will be running their own type of virtualization for their platform. The next step, I've most often seen, is for test servers to get virtualized. If a new group is formed to manage the virtualization infrastructure, virtual machines can be allocated to various application and server teams from a central pool and everyone is now sharing. Or, they begin sharing and then demand their own physical hardware to be isolated from others' resource hungry utilization. This is nonetheless a step in the right direction. Auto migration and guaranteed resource policies can go a long way toward making shared infrastructure, even between competing groups, a viable option.

Blamestorming

The most damaging side effect of splitting into too many distinct IT groups is the reinforcement of an "us versus them" mentality. Aside from the notion that specialization creates a lack of knowledge, blamestorming is what this article is really about. When a project is delayed, it is all too easy to blame another group. The SAN people didn't allocate storage on time, so another team was delayed. That is the timeline of the project, so all work halted until that hiccup was restored. Having someone else to blame when things get delayed makes it all too easy to simply stop working for a while.

More related to the initial points at the beginning of this article, perhaps, is the blamestorm that happens after a system outage.

Say an ERP system becomes unresponsive a few times throughout the day. The application team says it's just slowing down, and they don't know why. The network team says everything is fine. The server team says the application is "blocking on IO," which means it's a SAN issue. The SAN team say there is nothing wrong, and other applications on the same devices are fine. You've ran through nearly every team, but without an answer still. The SAN people don't have access to the application servers to help diagnose the problem. The server team doesn't even know how the application runs.

See the problem? Specialized teams are distinct and by nature adversarial. Specialized staffers often relegate themselves into a niche knowing that as long as they continue working at large enough companies, "someone else" will take care of all the other pieces.

I unfortunately don't have an answer to this problem. Maybe rotating employees between departments will help. They gain knowledge and also get to know other people, which should lessen the propensity to view them as outsiders

[Nov 12, 2016] Is the Information Technology Revolution Over

The article was a little too early call... But in 2016 it is definitely over. And degenerated into perversions...
Notable quotes:
"... The problem as I see it and it seems to be everywhere is not nearly enough people paying for custom software development done in house. ..."
"... I used to lead teams that built custom software for internal use. The ROI can be high under the right conditions, but Commercial Off The Shelf is often a better solution. ..."
"... Another reason to avoid custom software is risk mitigation. A business that relies on software built by one "someone on staff" is running a real risk if/when that someone leaves. ..."
"... Vendor which try to capitalize on ignorance of a typical IT management layer. ..."
"... One Unix group manager in a large company that I used to know for example did not understand the fact that IBM Power servers and Intel servers are based on CPU with two different architectures. When at the meeting I realized that my jaw simply dropped. ..."
"... For example for older people a cutting edge computer technology can probably provide the level of service comparable with the level of service of nursing home ..."
"... I also don't think it's even close to over not just because of robotics but also because of machine learning. ..."
Jun 04, 2013 | Economist's View

Quick one, then I have to figure out how to get to Toulouse (missed connection, in Paris now ... but should be able to get there ... long day so far):

Is the Information Technology Revolution Over?, by David M. Byrne, Stephen D. Oliner, and Daniel E. Sichel, FRB: Abstract: Given the slowdown in labor productivity growth in the mid-2000s, some have argued that the boost to labor productivity from IT may have run its course. This paper contributes three types of evidence to this debate. First, we show that since 2004, IT has continued to make a significant contribution to labor productivity growth in the United States, though it is no longer providing the boost it did during the productivity resurgence from 1995 to 2004. Second, we present evidence that semiconductor technology, a key ingredient of the IT revolution, has continued to advance at a rapid pace and that the BLS price index for microprocesssors may have substantially understated the rate of decline in prices in recent years. Finally, we develop projections of growth in trend labor productivity in the nonfarm business sector. The baseline projection of about 1¾ percent a year is better than recent history but is still below the long-run average of 2¼ percent. However, we see a reasonable prospect--particularly given the ongoing advance in semiconductors--that the pace of labor productivity growth could rise back up to or exceed the long-run average. While the evidence is far from conclusive, we judge that "No, the IT revolution is not over."

Comments

Darryl FKA Ron said...
The pickup reflects ongoing advances in IT and an assumption that those gains and innovations in other sectors spur some improvement in multifactor productivity (MFP) growth outside of the IT sector relative to its tepid pace from 2004 to 2012.5 These developments feed through the economy to provide a modest boost to labor productivity growth.

[Using technology to replace people or make them more productive, generally considered the same thing, is one form of productivity increasing technology integration. Automating accounting functions from producing bills to meter reading or selling your goods on the WWW were examples of the revolution of picking low hanging fruit. Using technology to manage systems in ways that people could not realistically accomplish is another way of increasing productivity. From running power production and distribution, traffic lights, and just in time manufacturing integrated with ERP accounting systems from order entry through to shipping and general ledger were other ways of increasing productivity. The green fields of labor replacement have largely been sewn. The green fields of automated systems management are without end. Economists have a limited lens into operations with metrics that often confuse value and price.]

Darryl FKA Ron said in reply to Darryl FKA Ron...
IOW, the MFP is underpriced because its marginal benefits get absorded by price competition.
squidward said in reply to Darryl FKA Ron...
Economists have a limited lens into operations with metrics that often confuse value and price.

That can be very true when looking at it qualitatively. Google or Amazon could be loading your browser with cookies and data mining your online habits to maximize sales. Their increases in revenue don't help the average consumer much other than consume more. It's not quite the same brave new world we had with the advent of online banking, bill paying and 24 hr shopping with home delivery.

I would have to agree that now marginal increases in productivity due to IT aren't giving the same marginal increases in value to the end consumer.

john personna said...
As I understand it, middle class incomes have fallen as productivity has risen. Doesn't that make a productivity centered view much less interesting to compassionate observers?
reason said in reply to john personna...
Not sure, but it makes redistribution more interesting.
Fred C. Dobbs said...

Yesterday I watched three techs work for two hours to get a laser printer going again that had been working fine Friday, but now was down due to 'network problems', so I would have to say, yeah, it could be over.

Julio said in reply to Fred C. Dobbs...
I watched the same scene twenty years ago, and it was a crappy dot-matrix printer that cost ten times as much.

What does it all mean?

john personna said in reply to Julio...
I suspect that techs stretched out a ticket, then and now.
Fred C. Dobbs said in reply to Fred C. Dobbs...

Such problems, which are infrequent, seem to be invariably network-related. Network support techs
are in short supply so 'everything else' is always tried first, even when that doesn't make much sense.

KJMClark said...

If economists at the FRB are still getting paid to ruminate on whether the IT revolution is over, then the IT revolution is not over. We'll know it's nearly over when we're replacing all but the top level of economists with intelligent software. (The top level will be helping the top-level software developers write the software.) We'll know it's completely over when the politicians decide we've had enough of the experiment of replacing people with machines. Or, rather, when we get the intelligent machines' responses to the politicians saying we're done.

ezra abrams said...

ya know, if u economist ever got outta yr offices, and did some real work...
You would find that the gains yet to be realized from the IT revolution are IMMENSE
People like myself, highly paid and educated PhDs, we can all be dispensed with
or, how about an earpiece that in real time tells a trial lawyer, *while s/he is in court*, what ruling he needs to cite to rebut a just made oral argument...

or CAD software that allows automotive design engineers to shave 10% of the weight of a car in an iteritve fashion (lower the weight of one component; therefore suspension can be less sturdy, in turn you then need less horsepower due to the lower wieght...)

or software that can solve Navier stokes for non laminair flow at high reynolds number

i mean, seriously, the it revolution is over ?
I'm sure they said the same thing about RailRoads and the tansportation revolution in, say 1890

jt said...

Couldn't automation lead to declining gross productivity via underemployment of displaced workers into low productivity jobs (services)? In fact, one could argue that the dual mandate of monetary policy will produce exactly this outcome. Labor income distributions seem to confirm this split in the labor market. The more interesting comparison, is in industry that use a lot of IT, has their productivity improved?

Michael Gamble:

Computer technology with out software is just a paperweight. The technology revolution is stalled but not over. The problem as I see it and it seems to be everywhere is not nearly enough people paying for custom software development done in house.

Every company any bigger than a few employees needs someone on staff who manages software purchases, installation and the creation of custom software to make the bridge between a lot of chimerical software, but no one wants to pay for it.

I don't blame them mind you there has been a lot of over hyping going on by the big players in the industry for years. And looking around you would think your entire business could be run from your phone with all the advertising, technology and hype put in to mobile phones?

Observer said in reply to Michael Gamble...
I used to lead teams that built custom software for internal use. The ROI can be high under the right conditions, but Commercial Off The Shelf is often a better solution. It depends. My off the cuff guess is that custom is hard to justify for a small business, except in very special cases.

Custom software that actually supports business critical processes tends to be quite expensive, with reason.

Another reason to avoid custom software is risk mitigation. A business that relies on software built by one "someone on staff" is running a real risk if/when that someone leaves.

Part of the unwillingness to pay is that people's expectations are influenced by consumer software price points, $0 to a few hundred dollars.

Jerry said...
A secondary inhibiter might be the growing mess in the software patent world. There seems to be an increasing reluctance to make an investment because of the patent toes that might get tweaked.
squidward said in reply to Jerry...

This is very true, the whole tech industry is a mess with antiquated patent law. You have to wonder how many products aren't being brought to market because a smaller company can't afford to get in a heavy weight patent brawl a la Samsung v. Apple.

I'm no expert but I have always wondered why writing code isn't more analogous to copyrighting than inventing and patenting. If we could incentivize more open source we could have more innovation.

Second Best said...

For the US, much of the IT revolution was over after the major carriers killed the internet revolution in its tracks, and it ain't coming back anytime soon.

See 'Captive Audience' by Susan Crawford.

kievite said...

Situation is enterprise datacenters definitely corresponds to definition of stagnation. We see a lot of cost cutting.

Percentage of custom software is small and getting smaller as Observer already noted above. Programmers are disappearing from the enterprise IT departments.

At the same time a new dangerous trend is in place. Some "of-the-shelf" packages on which enterprise depends are problematic with some subsystems close to junk or even harmful (SAP/R3, some IBM products, etc). Moreover there is now a new type of enterprise software vendors who are specializing in selling completely useless or even harmful software on the pure strength of marketing (plus fashion). Vendor which try to capitalize on ignorance of a typical IT management layer.

Like Kolmogorov once said "You can't overestimate the level of ignorance of the audience". That was about different audience, but fully applicable here. So snake oil salesmen in IT are making good money, may be better than honest sailmen.

But the problems with "off-the-shelf" packages are increasing due to their often unwarranted complexity (which serves mainly as barrier of entry for competitors), or just complexity for the sake of complexity.

This and the fact that generally software is a the most complex artifact invented by mankind lead to the level of understanding of existing packages and operating systems that can be called dismal. Even people who "should know" often look like coming from the pages of "The Good Soldier Švejk" or "Catch 22". One Unix group manager in a large company that I used to know for example did not understand the fact that IBM Power servers and Intel servers are based on CPU with two different architectures. When at the meeting I realized that my jaw simply dropped.

People who saw the software evolution from its humble beginning and can understand internals and nature of compromises taken in existing hardware and software are now close to retirement and in the new generation such people are exceedingly rare.

That is true for operating systems such as Linux or Solaris, this is even more true for web-related software such as Apache, MediaWiki, Frontpage, etc. As a result a lot of things are "barely run" and a lot of system are bought just because people have no clue that already bought systems can perform the same functions.

This is also true about Office, especially Excel. One think that I noticed that the level of knowledge of Excel is really dismal across the enterprise. I would agree with "squidward" that for office (but only for office) "I would have to agree that now marginal increases in productivity due to IT aren't giving the same marginal increases in value to the end consumer. ". But that's for office only. Cars, homes, etc are still "terrra incognita".

But while internally everything looks rotten, externally situation looks different: there is unending assault of automation on existing jobs. So JT is on something when he asks the question "The more interesting comparison, is in industry that use a lot of IT, has their productivity improved?" Yes and to the extend that many workforce cuts are permanent and moreover cuts might continue.

As for statement "For the US, much of the IT revolution is over" I doubt it. Computer will continue to eat jobs. The "cutting edge" simply moved elsewhere and one hot area are various robotic systems. Here is one example:

"IBM is using robots based on iRobot Create, a customizable version of the Roomba vacuum cleaner, to measure temperature and humidity in data centers. The robot looks for cold zones (where cold air may be going to waste instead of being directed to the servers) and hotspots (where the air circulation may be breaking down. IBM is putting the robots to commercial use at partners - while EMC is at an early stage on a strikingly similar project."

Both home and datacenter are huge application areas. Even in consumer electronics what we have is still very primitive in comparison with what is possible on the current hardware. That is true for smartphones, tablets and other mass gargets. And it is even more true for home. For example for older people a cutting edge computer technology can probably provide the level of service comparable with the level of service of nursing home. Automated cook who accepts a simple menu and deliver dishes is already feasible automation. Currently the cost will be high but gradually it will drop and quality of service improves.

One interesting area is saving energy. How many people here have home network which integrates thermostat, outdoor and indoor lights, and security system. And probably nobody here has computer automated shades on windows.

Autonomous datacenters with robot service are in my opinion an interesting development which in many cases can serve as "distributed cloud".

Sunny Liu said in reply to kievite...

I agree. That was a wonderfully informative post, and thank you for that. I also don't think it's even close to over not just because of robotics but also because of machine learning. The implications for data mining and big data are enormous, and the research being done in those fields are still yet to be fully utilized.

Right now, machine learning and big data are advancing research in biology, but what about the optimizations possible in pharmaceuticals or manufacturing?

geoff

not sure if this qualifies as IT. Not even sure if that is a meaningful distinction, but as a distributor I can't help thinking that 3D printers will have a profound impact on both manufacturing and distribution.

http://www.businessweek.com/articles/2013-05-16/bloomberg-view-why-3d-printing-can-make-the-world-a-better-place

The issue now becomes whether the technology will transform manufacturing more broadly. At the moment, 3D printing is a small part of the economy. The printers are typically slow, and the material they use is expensive and inconsistent. As the industry advances, however, printing on demand could reduce assembly lines, shorten supply chains, and largely erase the need for warehouses for many companies. Cutting back on shipping and eliminating the waste and pollution of traditional manufacturing could be an environmental boon.

Kaleberg

The software revolution is over, just the same way the written word revolution ended in 1900.

Tom Shillock:

Does increased labor productivity increase living standards? What if the products are lower quality and either must be disposed of sooner than if they were of higher quality or incur greater lifecycle service costs than if they were of higher quality? If increased productivity degrades the natural environment (air, water, soil, food) are living standards increased? If productivity numbers increase because more people are unemployed or under employed or suffer stagnant or lower wages from globalization has the standard of living increased?

Looking at aggregate data glosses losers and winners. For the last three decades the winners are relatively fewer in number but grabbing a greater share of GDP, while the loser are vastly greater in number and have fewer opportunities to be winners other than through random luck (marriage, inheritance, connections).

Is IT innovation necessarily good or more productive? The increase in semiconductor performance (pick your metric) at a given price, or even lower price, will not increase productivity if you already have all the IT performance you can use. In some cases, it might reduce productivity. E.g., having larger hard drives means more data will be collected (data collects to fill the space available: Shillock's Second Law of Storage). Unless one has efficient methods to keep track of it all then searching for it will reduce productivity. Also, access and seek times have not improved linearly with capacity. Another hit to productivity from IT. This is why the Fed's incorporation of a "hedonic index" for microprocessors into its price index muddies the water (c.f. p. 8)

What if some IT innovation gives a company a competitive edge such as happened in finance? Others are compelled to adopt it ASAP but that does not necessarily make the financial industry more productive. Indeed, it facilitated fraud on a massive scale that caused the Great Recession while leaving insiders vastly wealthier for it. It could be argued that IT innovations have facilitated the increasing and increasingly server financial crises over the past three decades, if only because they create the delusion among users that they know more than they do thereby feeding their hubris.

The Dictatorship of Data http://www.technologyreview.com/news/514591/the-dictatorship-of-data/

Most of the gains to productivity from IT are in two areas. First, is that large financial and reservation systems. These run on IBM mainframes. The second is the application of smaller computers to manufacturing to run tools. The vast amount of PC level IT probably reduces productivity because most people do not know how to use it. Word processors allow more people to take more time making their emails and interoffice memos grammatically better and with fewer spelling errors. Most people have little clue how to use spreadsheets, even Rinehart and Rogoff were challenged. PowerPoint and similar PC apps are great for enabling the incompetent and ignorant to appear otherwise, which accounts for their popularity. They are the lingua franca of IBM. So far Facebook, Twitter, LinkedIn, etc. are a great waste of time for the neurotically self-conscious and self-important.

Economics is a literary genre in which contestants focus only on the numbers and usually those they like then use their imaginations to spin stories about the numbers. Audiences then vote on which story they find most pleasing.

[Nov 04, 2016] Coding Style rear-rear Wiki

Reading rear sources is an interesting exercise. It really demonstrates attempt to use "reasonable' style of shell programming and you can learn a lot.
Nov 04, 2016 | github.com

Relax-and-Recover is written in Bash (at least bash version 3 is needed), a language that can be used in many styles. We want to make it easier for everybody to understand the Relax-and-Recover code and subsequently to contribute fixes and enhancements.

Here is a collection of coding hints that should help to get a more consistent code base.

Don't be afraid to contribute to Relax-and-Recover even if your contribution does not fully match all this coding hints. Currently large parts of the Relax-and-Recover code are not yet in compliance with this coding hints. This is an ongoing step by step process. Nevertheless try to understand the idea behind this coding hints so that you know how to break them properly (i.e. "learn the rules so you know how to break them properly").

The overall idea behind this coding hints is:

Make yourself understood

Make yourself understood to enable others to fix and enhance your code properly as needed.

From this overall idea the following coding hints are derived.

For the fun of it an extreme example what coding style should be avoided:

#!/bin/bash for i in `seq 1 2 $((2*$1-1))`;do echo $((j+=i));done



   

Try to find out what that code is about - it does a useful thing.

Code must be easy to read Code should be easy to understand

Do not only tell what the code does (i.e. the implementation details) but also explain what the intent behind is (i.e. why ) to make the code maintainable.

Here the initial example so that one can understand what it is about:

#!/bin/bash # output the first N square numbers # by summing up the first N odd numbers 1 3 ... 2*N-1 # where each nth partial sum is the nth square number # see https://en.wikipedia.org/wiki/Square_number#Properties # this way it is a little bit faster for big N compared to # calculating each square number on its own via multiplication N=$1 if ! [[ $N =~ ^[0-9]+$ ]] ; then echo "Input must be non-negative integer." 1>&2 exit 1 fi square_number=0 for odd_number in $( seq 1 2 $(( 2 * N - 1 )) ) ; do (( square_number += odd_number )) && echo $square_number done

Now the intent behind is clear and now others can easily decide if that code is really the best way to do it and easily improve it if needed.

Try to care about possible errors

By default bash proceeds with the next command when something failed. Do not let your code blindly proceed in case of errors because that could make it hard to find the root cause of a failure when it errors out somewhere later at an unrelated place with a weird error message which could lead to false fixes that cure only a particular symptom but not the root cause.

Maintain Backward Compatibility

Implement adaptions and enhancements in a backward compatible way so that your changes do not cause regressions for others.

Dirty hacks welcome

When there are special issues on particular systems it is more important that the Relax-and-Recover code works than having nice looking clean code that sometimes fails. In such special cases any dirty hacks that intend to make it work everywhere are welcome. But for dirty hacks the above listed coding hints become mandatory rules:

For example a dirty hack like the following is perfectly acceptable:

# FIXME: Dirty hack to make it work # on "FUBAR Linux version 666" # where COMMAND sometimes inexplicably fails # but always works after at most 3 attempts # see http://example.org/issue12345 # Retries should have no bad effect on other systems # where the first run of COMMAND works. COMMAND || COMMAND || COMMAND || Error "COMMAND failed."

Character Encoding

Use only traditional (7-bit) ASCII charactes. In particular do not use UTF-8 encoded multi-byte characters.

Text Layout Variables Functions Relax-and-Recover functions

Use the available Relax-and-Recover functions when possible instead of re-implementing basic functionality again and again. The Relax-and-Recover functions are implemented in various lib/*-functions.sh files .

test, [, [[, (( Paired parenthesis See also

[Oct 12, 2016] Crash: how computers are setting us up for disaster

Oct 12, 2016 | www.theguardian.com

Guardian (resilc). Today's must read.

[Sep 12, 2016] Linux.com CLI Magic Bash complete

Notable quotes:
"... Shashank Sharma is studying for a degree in computer science. He specializes in writing about free and open source software for new users. He is the co-author of Beginning Fedora , published by Apress. ..."
Sep 12, 2016 | www.linux.com

CLI Magic: Bash complete

By Shashank Sharma on May 08, 2006 (8:00:00 AM)


Printer friendly page Print
Comment on this article Comments

The auto complete feature of the Bourne Again SHell makes bash one of the most loved and newbie-friendly Linux shells. Just by pressing the Tab key you can complete commands and filenames. Press the Tab key twice and all files in the directory get displayed. But you can do more with autocomplete -- such as associating file types with applications, and automatically designating whether you're looking for directories, text, or MP3 files. With simple commands such as complete and the use of Escape sequences, you can save time and have fun on the command line. You can use the dollar sign ($) , tilde (~) , and at (@) characters along with the Tab key to get quick results in autocomplete.

For instance, if you want to switch to the testing subdirectory of your home directory, you can either type cd /ho[Tab]/tes[Tab] to get there, or use the tilde -- cd ~tes[Tab] . If the partial text -- that is, the portion before you press Tab -- begins with a dollar sign, bash looks for a matching environment variable. The tilde tells bash to look for a matching user name, and the at-sign tells it to look for a matching hostname.

Escaping is good

The Tab key can complete the names of commands, files, directories, users, and hosts. Sometimes, it is overkill to use the Tab key. If you know that you are looking for a file, or only user names, then use the Escape key instead for completion, as it limits bash's completion field.

You can use several Escape key combinations to tell bash what you are looking for. Invoke Escape key combinations by pressing a key while keeping the Escape key pressed. When looking for a file, you can use the Esc-/ (press / along with Escape) key combination. This will attempt filename completion only. If you have one file and one directory beginning with the letter 'i,' you will have to press the Tab key twice to see all the files:

$ less i <tab><tab>
ideas im articles/

When you type less i and press '/' while keeping the Escape key pressed, bash completes the filename to 'ideas.'

While Control key combinations work no matter how long you keep the Ctrl key pressed before pressing the second key, this is not the case with Escape key sequences. The Esc-/ sequence will print out a slash if you delay in pressing the / key after you press the Escape key.

You can also use Escape along with the previously discussed $ , ~ , and @ keys. Esc-$ , for example, completes only variable names. You can use Esc-! when you wish to complete command names. Of course you need to press the Shift key in order to use any of the "upper order" characters.

Wildcard expansion
The asterisk (*), caret (^), and question mark (?) are all wildcard characters. All *nix shells can perform wildcard expansion. Use the asterisk wildcard if you don't wish to view any hidden files. ls * would display all files and directories in the current directory except for those beginning with a dot (hidden).

If you wish to view only files with five-letter names beginning with a given letter, use the question mark wildcard. ls p???? will display only files with names with five letter files starting with 'p.' You can also use the square brackets for filename completion. ls [a-d]* displays all files and directories that begin with any letter between 'a' and 'd.' The caret wildcard excludes files. mv *[^.php] ../ would move all files to the parent directory, exculding those with .php extension.

Even smarter completion

By default, Tab completion is quite dim-witted. This is because when you have already typed cd down before pressing Tab, you'd expect bash to complete only directory names. But bash goes ahead and displays all possible files and directories that begin with 'down.'

You can, however, convert bash into a brilliant command-reading whiz. As root, edit the /etc/bash.bashrc file. Scroll down to the end of the file till you see the section:

# enable bash completion in interactive shells #if [ -f /etc/bash_completion ]; then # . /etc/bash_completion #fi

Uncomment this section and voilà, you have given bash powers far beyond your imagination! Not only is bash now smart enough to know when to complete only directory names, it can also complete man pages and even some command arguments.

Don't despair if you don't have root previleges. Just edit the last section of your ~/.bashrc file.

Associating application with file types

The complete command in bash lets you associate file types with certain applications. If after associating a file type to an application you were to write the name of the application and press Tab, only files with associated file types would be displayed.

complete -G "*.txt" gedit would associate .txt files with gedit. The downfall of using complete is that it overwrites bash's regular completion. That is, if you have two files named invoice.txt and ideas.txt, gedit [Tab][Tab] displays both the files, but gedit inv[Tab] , which should complete to invoice.txt, no longer works.

complete associations last only for the current bash session. If you exit and open a console, gedit will no longer be associated with .txt files. You need to associate file types to applications each time you start a new console session.

For permanent associations, you need to add the command to one of the bash startup scripts, such as ~/.bashrc. Then, whenever you are at the console, gedit will be associated with .txt files.

Shashank Sharma is studying for a degree in computer science. He specializes in writing about free and open source software for new users. Shashank Sharma is studying for a degree in computer science. He specializes in writing about free and open source software for new users. He is the co-author of Beginning Fedora , published by Apress.

[Aug 18, 2016] Neoliberals often argue that people should be glad to lose employment at 50 so that people from other countries can have higher incomes

Notable quotes:
"... Neoliberals often argue that people should be glad to lose employment at 50 so that people from other countries can have higher incomes, and leftists often agree because hey "free movement" and because after all the professional class jobs aren't at risk ..."
"... "I think Trump is afraid the imperial global order presided by the US is about to crash and thinks he will be able to steer the country into a soft landing by accepting that other world powers have interests, by disengaging from costly and humiliating military interventions, by re-negotiating trade deals, and by stopping the mass immigration of poor people." ..."
"... No one has literally argued that people should be glad to lose employment: that part was hyperbole. But the basic argument is often made quite seriously. See e.g. outsource Brad DeLong . ..."
"... To the guy who asked- poor white people keep voting Republican even though it screws them because they genuinely believe that the country is best off when it encourages a culture of "by the bootstraps" self improvement, hard work, and personal responsibility. They view taxing people in order to give the money to the supposedly less fortunate as the anti thesis of this, because it gives people an easy out that let's them avoid having to engage in the hard work needed to live independently. ..."
"... Attempts at repairing historical racial inequity read as cheating in that paradigm, and even as hostile since they can easily observe white people who are just as poor or poorer than those who racial politics focuses upon. ..."
"... The extent to which "poor white people" vote against their alleged economic interests is overblown. To a large extent, they do not vote at all nor is anyone or anything on the ballot to represent their interests. And, yes, they are misinformed systematically by elites out to screw them and they know this, but cannot do much to either clear up their own confusion or fight back. ..."
"... The mirror image problem - of elites manipulating the system to screw the poor and merely middle-class - is daily in the news. Both Presidential candidates have been implicated. So, who do you recommend they vote for? ..."
"... My understanding is trumps support disproportionately comes from the small business owning classes, Ie a demographic similar to the petite bourgeoisie who have often been heavily involved in reactionary movements. This gets oversold as "working class" when class is defined by education level rather than income. ..."
"... Eric Berne, in The Structures and Dynamics of Organizations and Groups, proposed that among the defining characteristics of a coherent group is an explicit boundary which determines whether an individual is a member of the group or not. (If there is no boundary, nothing binds the assemblage together; it is a crowd.) The boundary helps provide social cohesion and is so important that groups will create one if necessary. Clearly, boundaries exclude as well as include, and someone must play the role of outsider. ..."
"... I am somewhat suspicious of leaving dominating elites out of these stories of racism as an organizing principle for political economy or (cultural) community. ..."
"... Racism served the purposes of a slaveholding elite that organized political communities to serve their own interests. (Or, vis a vis the Indians a land-grab or genocide.) ..."
"... Racism serves as an organizing principle. Politically, in an oppressive and stultifying hierarchy like the plantation South, racism not incidentally buys the loyalty of subalterns with ersatz status. ..."
"... For a time, the balkanization of American political communities by race, religion and ethnicity was an effective means to the dominance of an tiny elite with ties to an hegemonic community, but it backfired. Dismantling that balkanization has left the country with a very low level of social affiliation and thus a low capacity to organize resistance to elite depredations. ..."
crookedtimber.org

Rich Puchalsky 08.04.16 at 11:40 am 177

fn: "Of course there is a subtext to these racist hate campaigns that someone else here raised and rich ran with a bit, which is the hatred of the unemployed. I think a lot of people voting leave imagine that the next thing on the agenda is slashing the dole to force poor white people to do the work the Eastern Europeans did. "

Yes, in part. In part, also, people imagine that poor citizens will get jobs that previously were done by migrants. This has a hatred of slackers element that is bad, but as economics, it's pretty well-founded that if you reduce the size of the labor pool relative to the population then unemployment will go down and wages will go up. Neoliberals often argue that people should be glad to lose employment at 50 so that people from other countries can have higher incomes, and leftists often agree because hey "free movement" and because after all the professional class jobs aren't at risk. But strangely enough some people seem to resent this.

Layman 08.04.16 at 11:48 am 178

Lupita: "I think Trump is afraid the imperial global order presided by the US is about to crash and thinks he will be able to steer the country into a soft landing by accepting that other world powers have interests, by disengaging from costly and humiliating military interventions, by re-negotiating trade deals, and by stopping the mass immigration of poor people."

... ... ...

Rich Puchalsky 08.04.16 at 12:03 pm

"I can't recall any particular instance where someone made this argument."

No one has literally argued that people should be glad to lose employment: that part was hyperbole. But the basic argument is often made quite seriously. See e.g. outsource Brad DeLong.

engels 08.04.16 at 12:25 pm

While this may be the effect of some neoliberal policies, I can't recall any particular instance where someone made this argument

Maybe this kind of thing rom Henry Farrell? (There may well be better examples.)

Is some dilution of the traditional European welfare state acceptable, if it substantially increases the wellbeing of current outsiders (i.e. for example, by bringing Turkey into the club). My answer is yes, if European leftwingers are to stick to their core principles on justice, fairness, egalitarianism etc…

http://crookedtimber.org/2005/05/31/talking-turkey-over-welfare/

Lupita 08.04.16 at 2:42 pm

Large numbers of low-income white southern Americans consistently vote against their own economic interests. They vote to award tax breaks to wealthy people and corporations, to cut unemployment benefits, to bust unions, to reward companies for outsourcing jobs, to resist wage increases, to cut funding for health care for the poor, to cut Social Security and Medicare, etc.

The same thing has happened in Mexico with neoliberal government after neoliberal government being elected. There are many democratically elected neoliberal governments around the world.

Why might this be?

In the case of Mexico, because Peña Nieto's wife is a telenovela star. How cool is that? It places Mexico in the same league as 1st world countries, such as France, with Carla Bruni.


Patrick 08.04.16 at 4:32 pm

To the guy who asked- poor white people keep voting Republican even though it screws them because they genuinely believe that the country is best off when it encourages a culture of "by the bootstraps" self improvement, hard work, and personal responsibility. They view taxing people in order to give the money to the supposedly less fortunate as the anti thesis of this, because it gives people an easy out that let's them avoid having to engage in the hard work needed to live independently.

They see it as little different from letting your kid move back on after college and smoke weed in your basement. They don't generally mind people being on unemployment transitionally, but they're supposed to be a little embarrassed about it and get it over with as soon as possible.

They not only worry that increased government social spending will incentivize bad behavior, they worry it will destroy the cultural values they see as vital to Americas past prosperity. They tend to view claims about historic or systemic injustice necessitating collective remedy because they view the world as one in which the vagaries of fate decree that some are born rich or poor, and that success is in improving ones station relative to where one starts.

Attempts at repairing historical racial inequity read as cheating in that paradigm, and even as hostile since they can easily observe white people who are just as poor or poorer than those who racial politics focuses upon. Left wing insistence on borrowing the nastiest rhetoric of libertarians ("this guy is poor because his ancestors couldn't get ahead because of historical racial injustice so we must help him; your family couldn't get ahead either but that must have been your fault so you deserve it") comes across as both antithetical to their values and as downright hostile within the values they see around them.

All of this can be easily learned by just talking to them.

It's not a great world view. It fails to explain quite a lot. For example, they have literally no way of explaining increased unemployment without positing either that everyone is getting too lazy to work, or that the government screwed up the system somehow, possibly by making it too expensive to do business in the US relative to other countries. and given their faith in the power of hard work, they don't even blame sweatshops- they blame taxes and foreign subsidies.

I don't know exactly how to reach out to them, except that I can point to some things people do that repulse them and say "stop doing that."


bruce wilder 08.04.16 at 5:50 pm

The extent to which "poor white people" vote against their alleged economic interests is overblown. To a large extent, they do not vote at all nor is anyone or anything on the ballot to represent their interests. And, yes, they are misinformed systematically by elites out to screw them and they know this, but cannot do much to either clear up their own confusion or fight back.

The mirror image problem - of elites manipulating the system to screw the poor and merely middle-class - is daily in the news. Both Presidential candidates have been implicated. So, who do you recommend they vote for?

There is serious deficit of both trust and information among the poor. Poor whites hardly have a monopoly; black misleadership is epidemic in our era of Cory Booker socialism.


bruce wilder 08.04.16 at 7:05 pm

Politics is founded on the complex social psychology of humans as social animals. We elevate it from its irrational base in emotion to rationalized calculation or philosophy at our peril.


T 08.04.16 at 9:17 pm

@Layman
I think you're missing Patrick's point. These voters are switching from one Republican to another. They've jettisoned Bush et. al. for Trump. These guys despise Bush. They've figured out that the mainstream party is basically 30 years of affinity fraud. So, is your argument is that Trump even more racist? That kind of goes against the whole point of the OP. Not saying that race doesn't matter. Of course it does. But Trump has a 34% advantage in non-college educated white men. It just isn't the South. Why does it have to be just race or just class?


Ronan(rf) 08.04.16 at 10:35 pm

"I generally don't give a shit about polls so I have no "data" to evidence this claim, but my guess is the majority of Trump's support comes from this broad middle"

My understanding is trumps support disproportionately comes from the small business owning classes, Ie a demographic similar to the petite bourgeoisie who have often been heavily involved in reactionary movements. This gets oversold as "working class" when class is defined by education level rather than income.

This would make some sense as they are generally in economically unstable jobs, they tend to be hostile to both big govt (regulations, freeloaders) and big business (unfair competition), and while they (rhetorically at least) tend to value personal autonomy and self sufficiency , they generally sell into smaller, local markets, and so are particularly affected by local demographic and cultural change , and decline. That's my speculation anyway.

T 08.05.16 at 3:12 pm

@patrick @layman

Patrick, you're right about the Trump demographic. https://fivethirtyeight.com/features/the-mythology-of-trumps-working-class-support/

Layman - Why are these voters switching from Bush et al to Trump? Once again, Corey's whole point is that there is very little difference between the racism of Trump and the mainstream party since Nixon. Is Trump just more racist? Or are the policies of Trump resonating differently than Bush for reasons other than race? Are the folks that voted for the other candidates in the primary less racist so Trump supporters are just the most racist among Republicans? Cruz less racist? You have to explain the shift within the Republican party because that's what happened.


Anarcissie 08.06.16 at 3:00 pm

Faustusnotes 08.06.16 at 1:50 pm @ 270 -

Eric Berne, in The Structures and Dynamics of Organizations and Groups, proposed that among the defining characteristics of a coherent group is an explicit boundary which determines whether an individual is a member of the group or not. (If there is no boundary, nothing binds the assemblage together; it is a crowd.) The boundary helps provide social cohesion and is so important that groups will create one if necessary. Clearly, boundaries exclude as well as include, and someone must play the role of outsider. While Berne's theories are a bit too nifty for me to love them, I have observed a lot of the behaviors he predicts. If one wanted to be sociobiological, it is not hard to hypothesize evolutionary pressures which could lead to this sort of behavior being genetically programmed. If a group of humans, a notably combative primate, does not have strong social cohesion, the war of all against all ensues and everybody dies. Common affections alone do not seem to provide enough cohesion.

In an earlier but related theory, in the United States, immigrants from diverse European communities which fought each other for centuries in Europe arrived and managed to now get along because they had a major Other, the Negro, against whom to define themselves (as the White Race) and thus to cohere sufficiently to get on with business. The Negro had the additional advantage of being at first a powerless slave and later, although theoretically freed, was legally, politically, and economically disabled - an outsider who could not fight back very effectively, nor run away. Even so, the US almost split apart and there continue to be important class, ethnic, religious, and regional conflicts. You can see how these two theories resonate.

It may be that we can't have communities without this dark side, although we might be able to mitigate some of its destructive effects.


bruce wilder 08.06.16 at 4:28 pm

I am somewhat suspicious of leaving dominating elites out of these stories of racism as an organizing principle for political economy or (cultural) community.

Racism served the purposes of a slaveholding elite that organized political communities to serve their own interests. (Or, vis a vis the Indians a land-grab or genocide.)

Racism serves as an organizing principle. Politically, in an oppressive and stultifying hierarchy like the plantation South, racism not incidentally buys the loyalty of subalterns with ersatz status. The ugly prejudices and resentful arrogance of working class whites is thus a component of how racism works to organize a political community to serve a hegemonic master class. The business end of racism, though, is the autarkic poverty imposed on the working communities: slaves, sharecroppers, poor blacks, poor whites - bad schools, bad roads, politically disabled communities, predatory institutions and authoritarian governments.

For a time, the balkanization of American political communities by race, religion and ethnicity was an effective means to the dominance of an tiny elite with ties to an hegemonic community, but it backfired. Dismantling that balkanization has left the country with a very low level of social affiliation and thus a low capacity to organize resistance to elite depredations.

engels 08.07.16 at 1:02 am

But how did that slavery happen

Possible short answer: the level of technological development made slavery an efficient way of exploiting labour. At a certain point those conditions changed and slavery became a drag on further development and it was abolished, along with much of the racist ideology that legitimated it.


Lupita 08.07.16 at 3:40 am

But how did that slavery happen

In Mesoamerica, all the natives were enslaved because they were conquered by the Spaniards. Then, Fray Bartolomé de las Casas successfully argued before the Crown that the natives had souls and, therefore, should be Christianized rather than enslaved. As Bruce Wilder states, this did not serve the interests of the slaveholding elite, so the African slave trade began and there was no Fray Bartolomé to argue their case.

It is interesting that while natives were enslaved, the Aztec aristocracy was shipped to Spain to be presented in court and study Latin. This would not have happened if the Mesoamericans were considered inferior (soulless) as a race. Furthermore, the Spaniards needed the local elite to help them out with their empire and the Aztecs were used to slavery and worse. This whole story can be understood without recurring to racism. The logic of empire suffices.

[Jul 14, 2016] mc-4.8.17 released

Notable quotes:
"... Copy & move operations now use an adaptive buffer, just like the corresponding coreutils commands, which will significantly improve the performance (hopefully!) for many of our users. ..."
www.midnight-commander.org

This is a maintenance release that includes bugfixes for a bunch of very annoying bugs that surfaced in the previous version (FISH, patchfs, segfault and tcsh detection on FreeBSD) and brings several new features.

Copy & move operations now use an adaptive buffer, just like the corresponding coreutils commands, which will significantly improve the performance (hopefully!) for many of our users. Move to the new high-level mouse API has not only simplified our code, but also resolved a number of long-standing mouse bugs. Finally, the new panel centered scrolling mode is weird, but fun; try it out!

For a detailed list of changes since the last version, please refer to the release notes.

Download page: http://ftp.midnight-commander.org/?C=N;O=D

Major changes since 4.8.16

Core

VFS

Editor

Misc

[Jul 14, 2016] Maintenance release of MC with LUA support

typo.co.il
A new release of mc^2 is out. It's mainly a maintenance release,
so there aren't many exciting new features.

    http://www.typo.co.il/~mooffie/mc-lua/docs/html/

News:

The C side:

  - The branch is rebased against mc 4.8.17.

The Lua side:

  - A few minor bug fixes.

  - New module: "dynamic skin"

    It lets you change the skin automatically depending on
    the directory you're in.

    So, for example, when you're examining an old backup disk
    you've mounted, or when you're on a remote machine, or when
    you're browsing a panelized or filtered listing, or when
    you're in a read-only directory, you can get a very
    noticeable visual indication reminding you of this.

  - New module: "colon"

    It lets you type :commands :like :these on the
    command-line (or in the editor). Like in 'vi'.
    E.g., you can rename files by typing:

        :s/\.jpe?g/jpg/i

    (This launches Visual Rename, where you can inspect
    the changes before committing them.)

  - The snapshots module can now save/restore panelized listings.
_______________________________________________
mc-devel mailing list
https://mail.gnome.org/mailman/listinfo/mc-devel

[Jun 26, 2016] Hillary Clinton email scandal shines light on specter of shadow IT

Notable quotes:
"... Surely, The State Department had an enterprise-grade email solution in place in 2013. We can only hope that Clinton protected her personal accounts with something more sophisticated than "Chelsea1980". ..."
"... 52% of IT executives said they don't have processes in place to manage outside sources, such as Dropbox in Vision Solutions' 2015 State of Resilience Report . Meanwhile, 70% of employees that use Dropbox do so solely for work, according to a 2013 Forrester report, and shadow IT appeared as a concern for the first time in the 2015 SIM IT Trends Study . ..."
"... Former FBI Director Robert Mueller said in 2012, "There are only two types of companies: those that have been hacked and those that will be." ..."
"... What kind of security risks does shadow IT create for your organisation? What happens when an employee uses the same password for both personal and enterprise accounts and hackers target that person's personal account? ..."
"... Their low-security Google Drive password just created a big headache for your organisation. ..."
"... Sourced from Bob Dvorak, founder and president, ..."
Mar 6, 2015 | Information Age

Consumer-grade and insecure applications can make headlines – and not in a good way

...This revelation should have public and private sector IT pros questioning their policies and practice around shadow IT – those programs outside of the formal control of the information technology department.

The Times wrote: "Her expansive use of the private account was alarming to current and former National Archives and Records Administration officials and government watchdogs, who called it a serious breach."

Surely, The State Department had an enterprise-grade email solution in place in 2013. We can only hope that Clinton protected her personal accounts with something more sophisticated than "Chelsea1980".

IT has an important job, and keeping tabs on the personal email accounts of executives or high-ranking officials should be the least of their worries. However, with 783 reported data breaches in 2014, according to The Identity Theft Resources Center, shadow IT is a strategic IT issue that is too important to ignore.

The topic raises an important issue around policy and practice of shadow IT, individual or departmental use of consumer-grade applications, such as personal email accounts, and cloud storage, departmental (or individual) SaaS accounts, even employee social media activity. All fall within this category in an age where the lines between work life and personal life are increasingly blurred.

While there may be individual, departmental or even organisational benefits to some elements of shadow IT, there are both operational and security risks associated with it and professionals' use of consumer grade tools for email, cloud storage and other services. CIOs and IT leaders need to be vigilant in developing, instituting and enforcing corporate IT governance policies and procedures.

52% of IT executives said they don't have processes in place to manage outside sources, such as Dropbox in Vision Solutions' 2015 State of Resilience Report. Meanwhile, 70% of employees that use Dropbox do so solely for work, according to a 2013 Forrester report, and shadow IT appeared as a concern for the first time in the 2015 SIM IT Trends Study.

...Gartner reported in its 2015 CIO Agenda that shadow IT consumes as much as 20% of a company's IT resources and, for the first time, respondents to the SIM IT Trends Study included shadow IT among their list of management concerns.

So what happens when Dropbox experiences downtime, as it did in January of last year? How do businesses react? What happens to the customer data, financial data or important documents they stored there?

When nearly two-thirds of organisations using the cloud reported not having HA or DR solutions for their enterprise applications, according to Vision Solutions, you can imagine how low the number must be for companies actively able to recover from, or are even monitoring, employee activity in the cloud.

The small matter of security

Former FBI Director Robert Mueller said in 2012, "There are only two types of companies: those that have been hacked and those that will be."

What kind of security risks does shadow IT create for your organisation? What happens when an employee uses the same password for both personal and enterprise accounts and hackers target that person's personal account?

Their low-security Google Drive password just created a big headache for your organisation.

You may not face a public records request that brings the specter of shadow IT in your organisation to light, but publicly traded corporations have internal control requirements to consider and private companies are notoriously protective of their intellectual property and confidential information.

All it takes is one instance and your company can be front-page news – and not in a good way.

Sourced from Bob Dvorak, founder and president, KillerIT

[Jun 26, 2016] Hillary Clinton takes shadow IT mainstream

Notable quotes:
"... Now let's strip away all the politics, sniping and legality over Clinton's email practices. What you have is shadow IT for official business and a State Department without the IT clout to stop it. You could argue with all the NSA snooping that Clinton's own email infrastructure was warranted. ..."
"... Security issues often are tossed aside for convenience. For Clinton it was a homemade email server. For the rest of us it's a personal cloud storage account. ..."
"... In the end, the Clinton email flap will play out for months. There will be hearings and non-stop election coverage about it. Just keep in mind what you're witnessing is shadow IT at a grand scale ..."
March 4, 2015 | ZDNet / Between the Lines

Hillary Rodham Clinton is in one big email mess, but if you zoom out and look at her as any other employee you have a leading example of shadow IT at play.

Hillary Rodham Clinton reportedly ran her own email server out of her house and now is in the middle of political firestorm. For our purposes, Clinton has provided us with the most high-profile case of shadow IT practices. And the first lesson of shadow IT is that the techies aren't going to push around the top execs. For the folks in business tech, the concept of shadow IT isn't exactly new. You're the CIO. Your other C-level peers have had their own cloud services provisioned for years. Developers have Amazon's cloud on a corporate Amex. It started with an innocuous printer under a desk. Then went to a server. Then smartphones to cloud services. People bring their own devices, apps and business practices with them to work.

Hell, the poor CIO is just finding out about some of these things.

Enter Clinton. According to the Associated Press, Clinton ran her own email as a Cabinet-level official. Enter records laws and all sorts of concerns. On the bright side, Clinton at least wasn't using a public email server. She at least earns some techie props for that.

Now let's strip away all the politics, sniping and legality over Clinton's email practices. What you have is shadow IT for official business and a State Department without the IT clout to stop it. You could argue with all the NSA snooping that Clinton's own email infrastructure was warranted.

Boil this down to Clinton as an employee and you have the following.

  1. Clinton was a top exec and those folks often get to push IT around. How do you think the iPad and iPhone became an enterprise juggernaut? You guessed it. The CEO wanted one.
  2. The email infrastructure Clinton ran was techie, but how many of you are conducting work on personal accounts? Thought so. You may not have federal records laws, but you're ignoring IT policies almost daily.
  3. Security issues often are tossed aside for convenience. For Clinton it was a homemade email server. For the rest of us it's a personal cloud storage account.

In the end, the Clinton email flap will play out for months. There will be hearings and non-stop election coverage about it. Just keep in mind what you're witnessing is shadow IT at a grand scale

[Jun 26, 2016] Hillary clintons private email account hacked the perils of shadow IT

www.tripwire.com

According to the Washington Post, the worst scenario may have come true when hacker "Guccifer" reportedly released several emails pertaining to Benghazi, which appear to be between Sidney Blumenthal and Hillary Clinton at the "clintonemail.com" domain. The domain was registered January 2009 through Network Solutions.

clintondomain

Looking a bit deeper at the MX records for the domain they map to a service run by McAfee:

clintonemail2

MX Logic was acquired by McAfee in June of 2009 and is now part of McAfee's SaaS offerings. So, it looks like someone knew what they were doing at some level to modify the MX records to use McAfee's service.

However, the risk of this email account being compromised is significant and one wonders who else aside from Guccifer may have had access to sensitive communications.

Before we pick on Hillary Clinton too much, we should evaluate how common this practice is. If the goal is to circumvent a regulatory requirement and is putting communications at risk, these shadow IT practices should be evaluated government-wide.

tonyE

This is a security breakdown at a very high level.

From personal experience I can tell you that any emails that are classified MUST be routed via very specific networks.

For the SecState to use a private network is a breakdown of security at the HIGHEST LEVEL.

She is guilty of a very serious crime, there is simply no way for her to excuse herself.

Also, how about all the people who were communicating with her? Surely they knew they were breaking the law... ( and I&#039m not talking about the records, I&#039m talking about a security breach at the highest level of our nation).

[Jun 25, 2016] Clinton's e-mail scandal another case of the entitled executive syndrome

Notable quotes:
"... And lest we forget, well before Clinton came to the State Department, members of the George W. Bush administration used a private e-mail server (at gwb43.com) run and paid for by the Republican National Committee-at least 88 accounts were set up for Bush administration officials in order to bypass the official White House e-mail system and avoid the regulations around presidential record retention, the Federal Records Act, and the Hatch Act (which bans the use of government e-mail accounts for political purposes, among other things). In the process of using that system, more than 5 million e-mail messages were "lost," which led to the resignation of a number of White House officials, including Deputy Chief of Staff Karl Rove. None of the e-mails for 51 of the 88 accounts was preserved by the RNC. ..."
"... Clinton was well aware of the Bush administration e-mail fiasco before she was nominated and confirmed as Secretary of State. She even told the State Department's assistant secretary for diplomatic security that she "gets it" after being briefed on why there were problems with her using a BlackBerry. ..."
"... Sure, the State Department's IT support is not exactly customer-centric. But its IT department has supported BlackBerry devices for unclassified e-mail in the past, and if Clinton could have dealt with sticking to using a computer while inside the State Department secure compartmented information facility (SCIF) and using a BlackBerry for unclassified e-mail, the State Department could have probably accommodated her. It was purely about Clinton's discomfort about using a PC for e-mail and her desire to use e-mail just like she did while running for office. ..."
"... So, as the State Department Office of the Inspector General reported, she paid a State Department staffer (who had worked for her directly in the past) off the books to create a shadow e-mail service of her own, and she used a personal BlackBerry not configured to State Department security standards to carry out official business. Having had a BlackBerry and the full control offered by private e-mail service during her presidential campaign in 2008, Clinton knew what she wanted, and she was going to have it whether it was approved or not. And she provided the same shadow e-mail service to her core staff as well-taking all of their communications off the grid and out of federal oversight. ..."
"... Besides, Clinton's excuse basically boils down to this: other people broke the rules, so she should have been allowed to as well. It's the entitled executive syndrome writ large. ..."
arstechnica.com

A certain class of executives wants a specific phone supported or special IT support for their chosen staff, and they want it now, rules and regulations be damned. "Yes" is the only answer they ever hear, and they will keep asking until they hear it-either from the IT department or from someone who will do it for them on the side. When I worked in IT, particularly when I moved up to a role as a "director of IT strategy" at a previous employer, these requests for special treatment happened so frequently we started calling it the "entitled executive syndrome." No matter how many times I explained the laws of physics and the limits of our budget and capabilities, I was told to find a way to make it happen… or come up with a creative workaround.

Sure, there's often a reason for dissatisfaction with the organizational norm. But skirting the norm can create all sorts of regulatory and legal headaches-Sarbanes-Oxley-related ones are the most common in the corporate IT world. Looking at the government sector, shadow IT has constantly gotten people in trouble for a host of other reasons: federal records laws, Federal Information Security Management Act (FISMA) violations, and privacy violations. For example, in 2010, doctors at a Department of Veterans Affairs got caught using Google and Yahoo cloud calendar services to schedule surgeries, breaching the security of health care data. They used it because it was more convenient than the VA's internal shared calendar system.

And lest we forget, well before Clinton came to the State Department, members of the George W. Bush administration used a private e-mail server (at gwb43.com) run and paid for by the Republican National Committee-at least 88 accounts were set up for Bush administration officials in order to bypass the official White House e-mail system and avoid the regulations around presidential record retention, the Federal Records Act, and the Hatch Act (which bans the use of government e-mail accounts for political purposes, among other things). In the process of using that system, more than 5 million e-mail messages were "lost," which led to the resignation of a number of White House officials, including Deputy Chief of Staff Karl Rove. None of the e-mails for 51 of the 88 accounts was preserved by the RNC.

Clinton was well aware of the Bush administration e-mail fiasco before she was nominated and confirmed as Secretary of State. She even told the State Department's assistant secretary for diplomatic security that she "gets it" after being briefed on why there were problems with her using a BlackBerry.

As previous e-mails obtained through Freedom of Information Act requests have shown, Clinton pushed hard to get the State Department's information security officers to approve her use of a mobile device for e-mail and do it from inside the State Department's secure executive suite-largely on the grounds that she was uncomfortable using a PC. The National Security Agency suggested she use an approved secure device capable of doing Secret-level classified e-mail as well as official unclassified e-mail. But the State Department was unprepared for the cost of supporting such a device, and its IT department didn't have the resources (nor, likely, the skills) in-house to support it.

Sure, the State Department's IT support is not exactly customer-centric. But its IT department has supported BlackBerry devices for unclassified e-mail in the past, and if Clinton could have dealt with sticking to using a computer while inside the State Department secure compartmented information facility (SCIF) and using a BlackBerry for unclassified e-mail, the State Department could have probably accommodated her. It was purely about Clinton's discomfort about using a PC for e-mail and her desire to use e-mail just like she did while running for office.

So, as the State Department Office of the Inspector General reported, she paid a State Department staffer (who had worked for her directly in the past) off the books to create a shadow e-mail service of her own, and she used a personal BlackBerry not configured to State Department security standards to carry out official business. Having had a BlackBerry and the full control offered by private e-mail service during her presidential campaign in 2008, Clinton knew what she wanted, and she was going to have it whether it was approved or not. And she provided the same shadow e-mail service to her core staff as well-taking all of their communications off the grid and out of federal oversight.

Clinton's excuse for her decision, which she now calls a mistake, was:

But no other secretary of state before her used e-mail as heavily, and the regulations regarding preserving e-mail records have changed over the past two decades. Condoleezza Rice did not use a personal e-mail account, according to the OIG report; she used a BlackBerry, but it was State Department issued. Madeline Albright never even sent e-mails. And while Colin Powell did use a personal e-mail account, the State Department was just getting Internet-connected e-mail at the time (on a system called OpenNet).

Besides, Clinton's excuse basically boils down to this: other people broke the rules, so she should have been allowed to as well. It's the entitled executive syndrome writ large.

[Jun 25, 2016] Clinton email flap highlights issues of shadow IT

Notable quotes:
"... "The reality is that every organization has a BYOD program - whether they think they do or not," Stevens said. "Now's the time to shore up the systems and enable mobility without sacrificing security." ..."
March 4, 2015 | .federaltimes.com

"I can recall no instance in my time at the National Archives when a high-ranking official at an executive branch agency solely used a personal email account for the transaction of government business," former NARA Director of Litigation Jason Baron told the Times.

While pundits and politicians are debating the ethics and legality of this, it also raises questions about the security of Clinton's communications.

"This news is yet another example of the lines blurring between work and personal lives and should serve as a wake-up call to federal IT departments," said Bob Stevens, vice president of federal systems at Lookout. "This trend towards mobility has clear benefits but it also adds a nuanced layer to not just email security, but all security."

Stevens noted that mobile devices, by their nature, move about and touch multiple networks as they do so. Since some networks are less secure than others, it becomes even more important to use secure programs and services to communicate.

"The reality is that every organization has a BYOD program - whether they think they do or not," Stevens said. "Now's the time to shore up the systems and enable mobility without sacrificing security."

Subsequent reports revealed that Clinton maintained her own server, but whether that server was more or less secure than commercial or federal email offerings is still unknown.

[Jun 25, 2016] The Perils of Shadow IT Your Most Senior Executives Are Doing It

Notable quotes:
"... In fact, according to the survey respondents, the average company already uses 20+ SaaS applications - think about it: Asana, Dropbox, Skype, Basecamp, Apple iCloud, Gmail, LastPass, not to mention your Facebooks and Twitters. But of those 20 or so SaaS platforms, more than 7 are non-approved. So, "…upwards of 35 percent of all SaaS apps in your company are purchased and used without oversight." ..."
"... Instead of losing sleep over perceived risk, companies must develop clear and concise policies governing cloud computing and SaaS usage. And don't stone me for saying it, but IT departments shouldn't exclusively own this exercise. Today, most executive level employees are well versed in SaaS, and they are probably well aware of what systems and platforms their teams are using day to day. ..."
duckduckgo.com
They say any press is good press, and the ruling is still out as to whether or not Hillary Clinton knowingly broke any laws when she used a private, home based email account for official State business as Secretary of State. She admitted on Tuesday that she had made a mistake and should've created two email accounts: a government one and a personal one. Still, one thing is clear: When the story broke last week, the entire world was talking about the latest threat to corporate security: shadow IT.

For those of you heavily immersed in the tech side of running a business, this won't be news to you. But for many business executives and CEOs the idea of classified information being run through outside servers or software can be chilling.

Basically, Shadow IT, also known as Stealth IT, describes solutions and SaaS, specified and deployed by departments other than the organizations own IT department.

As far back as 2012, IT research and advisory company Gartner was predicting that 35 percent of enterprise IT expenditures for most organizations would be managed outside the IT department's budget by 2015. Surely today, based on the innovations in technology which have occurred in 2012, that number's even higher.

And if you think the blame lies with those hipster millennials and their "always on" lifestyle, you would be wrong.

The Enemy Is Us

According to a 2014 study by Stratecast and Frost & Sullivan and based on input from organizations in the United States, United Kingdom, Australia and New Zealand, the biggest users of Shadow IT services are IT executives and employees.

Now extrapolate that fact across your organization, to other executives, managers, and employees, and you can see just how quickly those numbers begin to add up.

In fact, according to the survey respondents, the average company already uses 20+ SaaS applications - think about it: Asana, Dropbox, Skype, Basecamp, Apple iCloud, Gmail, LastPass, not to mention your Facebooks and Twitters. But of those 20 or so SaaS platforms, more than 7 are non-approved. So, "…upwards of 35 percent of all SaaS apps in your company are purchased and used without oversight."

So, if you can't blame the millennials, who or what can you blame?

You can blame technology.

Get Off'a My Cloud

More to the point, you can blame the rise of cloud computing. As with most things in life, that which can benefit us the most, can also harm us.

With more and more companies adopting BYOD policies (often also referred to as BYOC, or cloud), it's no surprise that Shadow IT isn't really in the shadows anymore. Which probably isn't news to any of you.

In fact, as the study discovered, Shadow IT is now being perceived as an important step in innovation, opening new channels of development for businesses, and reducing overall costs.

Here's why:

Of course, these are in addition to the direct benefits to a corporate IT department: No monies paid out in development costs, maintenance, testing, upgrades capacity planning, or performance management. Plus, backup and recovery of data and infrastructure is generally also the responsibility of the platform's vendor.

Manage Your Risk

So, where does that leave us? With remote working, job sharing, file sharing, and BYOD policies becoming commonplace, along with the rise of mobile and the ever evolving technological advances happening around us daily, it's a little too late to shut that barn door.

And, contrary to how nefarious the term Shadow IT "feels," it appears most employees who "go rogue" and use unapproved SaaS during work hours are doing so with the best of intentions: They simply want to do their jobs, as efficiently and as cost effectively as possible. What's not to like about that?

They're not doing it just because, either. These are generally speaking a smart group of people who want to get things done. They cite reasons like quickly gaining access to the right tools, overall comfort level with certain apps and platforms, and, perhaps most importantly, the desire to avoid a steep learning curve and the waste of time conquering such a learning curve entails if forced to adopt something new.

I think the responsibility today in handling cloud computing and unregulated corporate SaaS usage lies squarely with each organization. As we need to look inward to see who's really performing this Shadow IT (our own executive, managers, and IT people), we also need to look inward when it comes to corporate policies and guidelines. Because most companies today don't have any.

Instead of losing sleep over perceived risk, companies must develop clear and concise policies governing cloud computing and SaaS usage. And don't stone me for saying it, but IT departments shouldn't exclusively own this exercise. Today, most executive level employees are well versed in SaaS, and they are probably well aware of what systems and platforms their teams are using day to day.

The ideal approach to Shadow IT is to collaborate. We've got to break down silos between IT and the rest of the organization, and involve all areas of your organization to work together to create best practices and help put the right policies in place to minimize corporate risk. Think outside the box. Remain flexible. Be prepared to drop old-school "firewall" thinking. And remember, the end-goal really is to improve business outputs and add to the bottom line of the organization.

Was Clinton breaking the law with her Shadow IT efforts? I don't know. The State Department's email system is known to be vulnerable to hackers. But what I do know is she was leaps and bounds ahead of Romney and Palin, who conducted official business on free email services from Microsoft Corp. and Yahoo Inc.

Sometimes, perspective really is everything.

What do you think? Are you aware of any Shadow IT occurring in your organization? What do you think would be the most important things to include in policies and guidelines supporting SaaS usage? I would love to know your thoughts in the comment section.

This post was written as part of the Dell Insight Partners program, which provides news and analysis about the evolving world of tech. For more on these topics, visit Dell's thought leadership site PowerMore . Dell sponsored this article, but the opinions are my own and don't necessarily represent Dell's positions or strategies.

[Jun 25, 2016] Hillary Clinton's Shadow IT Problem

Notable quotes:
"... Again, the point here is not that Clinton should have ditched the secure, government system in order to use her phone of choice. In her circumstances, the security concerns should have outweighed her personal comfort. But for many, the desire to stick with tech that they know and love is often counter to logic, efficiency, security and policy. And most of us work in environments where bucking the system isn't quite as dire as it could be for the nation's top diplomat. ..."
"... " Shadow IT " is technology that users install without company approval because they prefer it to what's offered. What I know is that I can't secure my network if it's packed with technology that my users hate. ..."
March 20, 2016 | Techcafeteria
...Judicial Watch, a conservative foundation looking for evidence that Clinton broke laws in her handling of the email, received some fascinating information in response to a recent FOIA request.

Upon joining the State Department in early 2009, Clinton immediately requested a Blackberry smartphone. Having used one extensively during her 2008 Presidential campaign, she, like almost every attorney in that decade, had fallen in love with her Blackberry, hence the request. After all, Condoleezza Rice, her predecessor as Secretary of State, had used one. President Obama had a special secure one that the NSA had developed for him. But they said no. Even after being called to a high level meeting with Clinton's top aide and five State Department officials, they still said no.The NSA offered Clinton an alternative. But it was based on Windows CE, a dramatically different, less intuitive smartphone operating system. A month later, Clinton started using her own server. Judicial Watch claims that this info proves that Clinton knew that her email was not secure, but I think that she has already admitted that. But it also reveals something much more telling.

As a three plus decade technology Director/CIO (working primarily with Attorneys), I can tell you that people get attached to specific types of technology. I know a few Attorneys who still swear to this day that Wordperfect 5.1 for DOS was the best word processing software ever released. And there are millions who will tell you that their Blackberry was their virtual right arm in the 2000's.

How devoted are people to their favorite applications and devices? I worked for a VP who was only comfortable using Word, so when she did her quarterly reports to the board, she had her assistant export huge amounts of information from our case management system. Then she modified all of it in Word. Once delivered, she had her assistant manually update the case management system in order to incorporate her changes. Efficient? Not at all. But she loved herself some Word. I've seen staff using seven year old laptops because they know them and don't want to have to learn and set up a new one. And it wasn't until the bitter end of 2014 that both my boss and my wife finally gave in and traded up their Blackberries for iPhones.

Again, the point here is not that Clinton should have ditched the secure, government system in order to use her phone of choice. In her circumstances, the security concerns should have outweighed her personal comfort. But for many, the desire to stick with tech that they know and love is often counter to logic, efficiency, security and policy. And most of us work in environments where bucking the system isn't quite as dire as it could be for the nation's top diplomat.

"Shadow IT" is technology that users install without company approval because they prefer it to what's offered. What I know is that I can't secure my network if it's packed with technology that my users hate. Smart people will bypass that security in order to use the tools that work for them. An approach to security that neglects usability and user preference is likely to fail. In most cases, there are compromises that can be made between IT and users that allow secure products to be willingly adopted. In other cases, with proper training, hand-holding, and executive sponsorship, you can win users over. But when we are talking about Blackberries in the last decade, or the iPhone in this one, we have to acknowledge that the popularity of the product is a serious factor in adoption that technologists can't ignore. And if you don't believe me, just ask Hillary Clinton.

[Jun 25, 2016] How to Turn Hillary Clintons Shadow IT Habits into Opportunity

Notable quotes:
"... Hillary Clinton has quickly become the public face of so-called "shadow IT" practices, which already affects almost every organization ..."
"... In other words, shadow IT is the unapproved, unmanaged solution that frustrated employees (and government officials) turn to when official systems don't meet their needs. In Chua's view, it's simply a good idea to take this bull by the horns, identify the pain points people are trying to avoid, and meet those needs through official channels instead. ..."
"... "Heavy-handed approaches are not going to eliminate shadow IT, it'll just go farther underground," ..."
"... In other words, a light touch might do wonders to tame the shadow IT beast even where strict policy edicts fail. And this lesson needs to be absorbed by a very large audience. ..."
The Motley Fool

... there's also a big upside to Clinton's home-brew email solution getting national attention. Hillary Clinton has quickly become the public face of so-called "shadow IT" practices, which already affects almost every organization -- from small and medium businesses to enterprise-class giants, and onward to the government behemoth. It's high time investors and business managers take a closer look at this trend, so let's thank her for opening the debate.

... ... ...

"I think we're at a point in time where companies can no longer ignore shadow IT," Chua said. "They need to put official policies in place, start talking to employees about what they need, make sure that these needs are aligned with the business.

"If they don't, then people can start creating their own solutions and create this whole shadow IT problem."

In other words, shadow IT is the unapproved, unmanaged solution that frustrated employees (and government officials) turn to when official systems don't meet their needs. In Chua's view, it's simply a good idea to take this bull by the horns, identify the pain points people are trying to avoid, and meet those needs through official channels instead.

"This is definitely an opportunity to sit up and take action," Chua explained. "The IT industry is moving away from cookie-cutter solutions with help desk tickets and red tape around everything. This debate gives IT departments a chance to say, 'Hey, different business units have different needs. I'm going to create a baseline framework, but I'll be agile and respond to the various needs of different units.'"

Chua's comments underscore a growing sentiment among IT industry professionals. Talking to the CIO magazine this week, Deputy Chief Technology Officer Steve Riley of data networking specialist Riverbed Technology (NASDAQ:RVBD) expanded on the problem. "Heavy-handed approaches are not going to eliminate shadow IT, it'll just go farther underground," Riley said. "There's no positive outcome for being a disciplinarian about something like this. You might end up with services that are even more dangerous, where people now actively seek to circumvent policies."

How the solutions fit the problem

In other words, a light touch might do wonders to tame the shadow IT beast even where strict policy edicts fail. And this lesson needs to be absorbed by a very large audience.

According to Softchoice's data collection, over 80% of organizations -- businesses, corporations, churches, you name it -- already see some members stepping outside the formal IT structure to enjoy the convenience of cloud-based public services.

Google (NASDAQ:GOOG) (NASDAQ:GOOGL) is a popular provider with tools including Gmail, Google Docs, and Google Calendar. Microsoft (NASDAQ:MSFT) might lose some software license sales to other cloud providers, but its Windows Azure and SkyDrive services are also leaders in their own right.

Sure, some of these service choices already have the official support of the IT department. But one-third of all users in a large Softchoice audit program recently reported employing tools such as SkyDrive or Google Calendar at work -- without so much as notifying the IT department.

The shadow IT market seems to open up a very large business opportunity for software-as-a-service providers such as Google and Microsoft. Managing these tools in a properly approved and budgeted fashion will help in closing boatloads of security and transparency concerns. And that way, they could soak up the demand for unofficial email servers and unapproved data warehouses running in some random employee's garage, beyond the reach of corporate firewalls.

Final words

A flexible approach to systems management can help businesses and government agencies make the most of their resources. There will always be rogue systems and maverick users, but acknowledging this reality can help contain the problem -- and maybe turn it into a strength instead.

Sweeping shadow IT under the rug, on the other hand, only opens up the door to more security leaks and the next Clinton-style transparency scandal.

[May 28, 2016] How Can Older Workers Compete In An Economy That Values Youth

www.zerohedge.com

Zero Hedge

...

Workers of all ages are caught in a vice. Older workers need to keep working longer in an economy which values younger workers (and their cheaper healthcare premiums). Younger workers are caught in the vice of "you don't have enough experience" and "how do I get experience if nobody will hire me?"

Middle-aged workers are caught between the enormous Millennial generation seeking better jobs and the equally numerous baby Boom generation seeking to work a few more years to offset their interest-starved retirement funds. (Thank you, predatory and rapacious Federal Reserve for siphoning all our retirement fund interest to your cronies the Too Big to Fail Banks.)

Workers 55 and older are undeniably working longer. Here is the labor participation rate for 55+ workers:

... And here's why so many workers have to work longer--earned income's share of the GDP has been in a free-fall for decades as Fed-funded financiers and corporations skim an ever greater share of the nation's GDP.

I am 62, very much an older worker with a startling 46 years in the work force (first formal paycheck, 1970 from Dole Pineapple). (Thanks to the Fed's zero-interest rate policy, I should be able to retire at 93 or so--unless the Fed imposes a negative-rate policy on me and the other serfs.)

But I recall with painful clarity the great hardships and difficulties I experienced in the recessions of 1973-74, 1981-82 and 1990-91 when I was in younger demographics. My sympathies are if anything more with younger workers, as it is increasingly difficult to get useful on-the-job experience if you're starting out.

That said, here are some suggestions for 55+ workers seeking to find work in a very competitive job/paid work market.

1. Target sectors that haven't changed much. There's a reason so many older guys find a niche in Home Depot and Lowe's--power saws, lumber, appliances, etc. haven't changed that much (except their quality has declined) for 40 years.

The same can be said of many areas of retail sales, house-cleaning, caring for children, etc.

Everyone knows the young have an advantage in sectors dominated by fast-changing technology, so avoid those sectors and stick to sectors where your knowledge and experience is still applicable and valued by employers.

2. If at all possible, get your healthcare coverage covered by a spouse or plan you pay. Those $2,000/month premiums for older workers are a big reason why employers would rather hire a $200/month premium younger worker, or limit the hours of older workers to part-time so no healthcare coverage is required.

Telling an employer you already have healthcare coverage may have a huge impact on your chances of getting hired.

3. If you have any computer-network-social media skills, you can get paid to help everyone 55+ with fewer skills. Your computer skills may not be up to the same level as a younger person's, but they are probably far more advanced than other 55+ folks. Many older people are paying somebody $35/hour or more to help them set up email, fix their buggy PCs and Macs, get them started on Facebook, etc. It might as well be you.

4. Focus on fields where managerial experience and moxie is decisive. Even highly educated young people have a tough time managing people effectively because they're lacking experience. Applying biz-school case studies to the real world isn't as easy as it looks. (I found apologizing to my older employees necessary and helpful. Do they teach this in biz school? I doubt it.)

The ability to work with (and mentor) a variety of people is an essential skill, and it's one that tends to come with age and experience.

5. Reliability matters. The ability to roll with the punches, show up on time, do what's needed to get the job done, and focus on outcomes rather than process are still core assets in a work force.

Being 55+ doesn't automatically mean someone has those skills, but they tend to come with decades of work.

6. If nobody will hire you, start your own enterprise to fill scarcities and create value in your community. The classic example is a handyperson, as it's very difficult for a young person to acquire the spectrum of experience needed to efficiently assess a wide array of problems and go about fixing them.

#3 above is another example of identifying one's strengths and then seeking a scarcity to fill. Value, profits and high wages flow to scarcity. Don't try to compete in supplying what's abundant; seek out scarcities and work on addressing those in a reliable fashion.

Every age group has its strengths and weaknesses, and the task facing all of us is to 1) identify scarcities we can fill and 2) seek ways to play to our strengths.

Shizzmoney

That's easy: the elitist old people in power will start a war, force the young people into that war, where they will all be killed and the old people get their jobs.

Also, for those young people who protest the war, the government and corporate military security forces will detain and kill them, too.

Problem solved!

Normalcy Bias

Just wait 'til they get a load of our Mighty Morphin Tranny Ranger Battalion!

KnuckleDragger-X

+1000 for that one.... :-))

KnuckleDragger-X

Bob Seger: Ballad of the Yellow Berets


https://www.youtube.com/watch?v=WP6Xt3GBhdQ

Syrin

Exactly. Value youth? Is that why we saddle them with $250,000 worth of student loan debt and a degree in women's studies to find no jobs because we let in illegals and skilled workers with Visas from foreign countries? Seems like we hate our youth. Of course, they deserve it since they have been focused on being social justice fucktards rather than getting any marketable skills and paying attention to what the gov't is doing to their future. Schadenfreude.

deja

No, they are stupid enough to saddle themselves with $250,000 worth of student loan debt for a degree in womens' studies.

cougar_w

The OP doesn't make much sense to me. Most of the work people my age do, the young people either don't want or are not qualified for. Maintaining vital COBOL apps or air traffic controller software from the 70's? Really? And the ones are, they don't mind working with older employees and seem to enjoy our "gravity".

I work in IT so maybe things are a bit different. Grey beards are huge around here and always will be.

But this has been a challenge for centuries, young people have to find their own way and "their way" (being probably a dream from childhood or an inspiration from a college professor) might not be practical at first. They bounce around a little until marriage hits them and then they find something that works for supporting a family. Same as it ever was. The idea that "their way" is some kind of unswerving life's mission is usually part of the corporate "just do it" meme that sells $400 specialty running shoes. Yeah whatever, just figure it out actually, life will tell you what you are supposed to be doing, and who you are supposed to be doing it with.

GeezerGeek

The market for COBOL programmers had a sudden surge around Y2K, but only certain industries still maintain their old COBOL apps. Curiously, a certain computer/software has recently tried pushing a visual version of COBOL, much like Gates did when he came out with Visual Basic back in the early 1990s. I retired after 40 years in IT in 2011, so I am a bit out of touch where COBOL is concerned. Does anyone even teach it anymore in college? Maybe if someone modified it to create phone apps and games it would once again be popular.

Abbie Normal

Then it's a good thing I didn't follow my undergrad English Prof's advice and switch my major from science to arts, because he thought there was some "real intelligence" in my writing style that even his grad students lacked. Maybe I should look him up....

eatthebanksters

I have two buddies, one a 61 year old attonery who has never lost a case and the other a 59 year old facilities director. The lawyer has been seeking work for 6 years and has pretty much given up...he can't even get hired at lesser jobs because he is overqualified and 'will leave when something better comes along'. The facilities director has a great resume and knows his stuff but has been out of work for almost two years. He has come in 'second' more times than I can count. He is working od jobs and living with a friends mother, exchanging work on the house for rent and meals. Welcome to Obama's economy.

N0TaREALmerican

He'd work if he'd accept less money, but he feels "entitled to earn what HE thinks he's worth". Just another lazy old-fart who feels the world owns him something. Welcome to a competitive economy old-fart, nobody said life was fair. Stop bitching and work for less.

mary mary

If you ever need an attorney, you might look for an experienced attorney who worked so hard that he never lost a case.

If you ever inherit a zillion bucks and buy a bunch of properties, you might confer with an experienced facility manager who actually managed a bunch of properties.

I doubt an attorney who never lost a case achieved that record by going around saying, "somebody owes me something".

I doubt a facilities manager who managed a bunch of properties achieved that by going around saying, "somebody owes me something".

Baa baa

What a load of crap. Most will take anything. I know, I am one. Don't lecture me about being "entitled" you punk. Your post reeks of the entitlement generation. Slug through 50 years of working, rearing a family, kids to college... I am beginning to wonder if the hundreds of thousands spent on the education and well-being of your ingrate ass was a misallocation of funds.

corporatewhore

Give credit where credit is due. This inability to find work at an older age has been going on for years and can't be blamed on Obama. Senior buyers at Macy's, older workers at Monsanto or television weather people at KSDK in St Louis all suffer the same fate. Labor cost and benefits are all less for the younger generation no matter what level of experience or capability. We develop a mindset throughout our productive career that we are indispensable and worth it because of our knowledge, contacts and industry wherewithal. It's all an illusion and we are NOT prepared or equipped to face the reality at an older age that we are completely dispensable.

At an older age if you want meaning you have to find it and think out of the paradigm that you've been led to believe is real. No one owes you anything for your experience or wealth of knowledge. Figure it out and rethink yourself as to what you love to do and want to do not what you must do to make money.

At 58 in 2008 I was fucked over by my corporation and wallowed in miserableness and poverty while i worked every contact and firm I knew. Nothing resulted. I had to work 3 part time jobs until I earned 2 full time ones and work over 90 hours per week because I enjoy it. It is work that covers the bills and allows me to create what I want to work on for the future while I still can walk think and breathe.

Best advice to your children: Go in business for yourself because just as it happened to me, it will happen to you when you become 55.

Nobody For President

Thanks for that, corporate whore. That sounds like an honest reprise of an incredibly hard time in your life, and I totally agree. I'm telling all (4) my grandkids, from 7 to 20, to live your life, not someone else's. The oldest one gets it, and I think the other ones will also, if I live long enough, because I walked that walk.

I'm old, and work full time (more or less) and make a living - not a killing, but a living - at it.

nuubee

Good news old people, the economy currently doesn't value anything you can produce, unless you can print money.

Cautiously Pessimistic

You get up every morning
From your 'larm clock's warning
Take the 8:15 into the city
There's a whistle up above
And people pushin', people shovin'
And the girls who try to look pretty
And if your train's on time
You can get to work by nine

... ... ...

mary mary

MSM says Baby Boomers "have stolen everything", but in fact Baby Boomers are having to extend their careers because they're broke. This is the easily foreseeable result of 20+ years of the Fed keeping interest rates artificially low, making Baby Boomers suffer the double-whammy of (1) not having their deferred income (pensions) grow, while (2) inflation in fact continued at 6% annual, thanks also to the Fed keeping interest rates artificially low.

Yes, someone "have stolen everything". That someone is the owners of the Fed.

[May 21, 2016] NetSlaves: True Tales of Working the Web

www.amazon.com
Arthur Lindsey III , April 16, 2003
A 246 Page "Support Group"

Being an unemployed techie myself, I cannot begin to describe what a godsend this book is. NETSLAVES finally reveals the truth about what it is to be part of what is likely the most under-appreciated sect of the working class. The stale stories of "dorm-room success" have been supplanted by the pathetically sad/darkly humorous accounts of those who have been saddled with with million-dollar job titles, bleeding ulcers, and ramen noodle grocery budgets. NETSLAVES is an entertaining and enligtening read, written by two men who have actually been passengers in every sewer pipe that is The new-media industry. This book is a must for every modern library, as it can be considered a "warning shot" for those with IT aspirations, or as a source of vindication for those of us who have been dismissed and trampled on. Bravo!

A customer, November 24, 1999

Handwriting on the Wall

NetSlaves tells it like it is for the millions of us on the business end of the IPO and monopoly screwdrivers. Apply these lessons to the law, publishing, automotive, chemical, airline industries, etc., etc. This book is not just a cerebral and satirical indictment of the internet industry.

It is a comment on upper and middle management corporate business practices in general, and the dismal fate of the vast armies of workers used as cannon fodder since day one for the follies of unscrupulous robber barons; or morons who just happen to find themselves in the right place at the right time to make market killings; or Scrooges who will never learn what it is to have a heart. Baldwin and Lessard are heirs to the muckrakers of the early 20th Century. Corporate E-merica, take heed.

[Apr 30, 2016] My Top 5 Movies About Unemployment

Christianity Today
Erin Brockovich

2000 | Rated R
directed by Steven Soderbergh
Based on the true story of an unemployed mother of three who forced her way into a job as a legal clerk and built an anti-pollution case against a California utility company. Erin Brockovich has become a name for someone with tenacity and perseverance.

The Journey of Natty Gann

1985 | Rated PG
directed by Jeremy Kagan
Disney's family-friendly adventure demonstrates how tough the Great Depression was on kids, namely the teenage girl of the title who journeys across America to reunite with her father. Grounded by strong performances, including a young John Cusack, this gem serves as a fine introduction of a difficult subject to younger viewers.

Tootsie

1982 | Rated PG
directed by Sydney Pollack
This light-hearted, quirky comedy stars Dustin Hoffman as an unemployed actor who pretends to be a woman for a full-time role in a soap opera. Beneath the hilarity is a sobering reminder that landing a job sometimes requires thinking outside the box, to say the least.

Up in the Air

2009 | Rated R
directed by Jason Reitman
George Clooney is stellar as a veteran hatchet man who has lost his ability to form meaningful relationships, living a life on the road. Ultimately this is a poignant drama about identity and what defines us. If we are nothing more than our occupation, what remains when that is gone?

Russ Breimeier, a freelance film critic who lives in Indianapolis, was unemployed for two years until recently landing a part-time job.

[Apr 30, 2016] Over 50, Unemployed, Depressed and Powerless

Notable quotes:
"... I know what it feels like to be marginalized because you're out of work. To be judged by others as if there's something wrong with you. To grow increasingly depressed, demoralized and despairing as three months turns into six months and that goes on for a year or more; as rejection after rejection becomes crushing, humiliating, and leaves you feeling worthless. ..."
"... All money-related impacts aside, you lose confidence. You wear out. You start to give up ..."
Daily Plate of Crazy

Are you over 50, unemployed, depressed and feeling powerless? For that matter, are you any age and feeling hopeless because you can't seem to land a job?

Frustrated Middle Age Man

The recession may be officially over, and for some segments of the population, things are looking up. But too many are still sinking or hanging on by the skin of their teeth. Long-term unemployment or underemployment has become a way of life.

This issue, for me, is personal.

I know what it feels like to be marginalized because you're out of work. To be judged by others as if there's something wrong with you. To grow increasingly depressed, demoralized and despairing as three months turns into six months and that goes on for a year or more; as rejection after rejection becomes crushing, humiliating, and leaves you feeling worthless.

All money-related impacts aside, you lose confidence. You wear out. You start to give up. And you don't even make it into the "statistics." It's been too long since your last employment relationship.

Overqualified, Over-Educated, Over 50

Despite my fancy educational background and shiny corporate career history, for a number of years I was unable to obtain work that was even remotely close to using my skills. Paying me a living wage? Let's not even discuss it. I must have applied to 100 positions over the course of several years, attended the usual networking events, and schmoozed every contact I could come up with.

No go. I suffered from the three O's: Overqualified, Over-educated and Over 50, though I may not have looked it. That last? If you ask me, age was the kicker. Throughout that period, as post-divorce skirmishes continued to flare (further complicating matters), I nonetheless took every project I could eke out of the woodwork, supplemented by debt.

Hello, bank bail-out? How about a few bucks for those of us who foot the bill in tax dollars?

The Borrowing Trap

Now and then, an acquaintance will make an off-hand remark about those who borrow money or live on credit cards. The assumption is that credit purchases are frivolous, or that the person who racks up consumer debt does so out of irresponsibility and poor judgment.

Never assume. Yours truly? I borrowed to put food on the table. I borrowed to pay for school supplies for my kids. I borrowed to enable them to take advantage of academic opportunities that they earned through their own hard work. I also counted my blessings. While I had no family to assist, my kids were healthy and doing well, I was basically healthy despite chronic pain, and I was able to use credit. Borrowing is a double-edged sword of course, especially if it continues for an extended period. But for my little household, debt was the only path to survival. For all I know, it will be again.

Fighting Your Way Back

These days? I still live on a tight budget, I dream of recovering from the years of financial devastation "someday," and I take every gig I can get. Willingly. I've gained new skills along the way and continue to refine them, I'm always looking for another project and thrilled when I nab one, and I'm accustomed to a 12- to 14-hour workday. I put in long hours throughout my corporate career and I have no problem doing so now. In fact, I'm grateful for these workdays and I take none of them for granted. Moreover, I suggest that few of us should take our sources of income as a given.

You know the expression - "There but for the grace of God go I." Misfortune can visit any one of us. Layoff. Accident or illness. Gray divorce. The phone call or email with no warning, saying "you're done" as you're replaced by someone 20 years younger.

And yes, I've internalized the wisdom of this little gem: "If opportunity doesn't knock, build a door." But I also know it isn't always possible, and the secret to success is not as simple as hard work. It's aided by the assistance of others, not to mention - luck.

Unemployed and Depressed

Forbes reminds us of the clear links between unemployment and depression, which isn't to say that underemployment or hating your job is a picnic.

Forbes staff writer Susan Adams cites a Gallup poll as follows:

The longer that Americans are unemployed, the more likely they are to report signs of poor psychological well-being," says the study. "About one in five Americans who have been unemployed for a year or more say they currently have or are being treated for depression - almost double the rate among those who have been unemployed for five weeks or less.

She goes on to note:

The long-term unemployed, unfortunately, have good reason to be depressed. They suffer plenty of discrimination in the job market. A 2012 study by economist Rand Ghayad found that employers preferred candidates with no relevant experience, but who had been out of work for less than six months, to those with experience who had been job hunting for longer than that.

.... ... ...

How many of you have found yourselves laid off and unable to get another job? How many of you are struggling in midlife to create a career where once you were responsible for taking care of a family?
•How many of you have knocked on doors and connected until your blue in the face, only to give up?
•How many of you have drained away any savings you may have had or incurred crushing debt?
•Have you had more success at creating new ventures for yourself - a business or freelance work?
•Were you able to rely on the assistance of family or friends for a temporary period?
•If you're over 50, have you found it harder? Have you had an experience similar to Cindy's?

I'm certain that many of you have fought your way back; I'm still fighting after years, but I have seen progress. Slower than I'd like, but progress all the same.

If someone helped you out, have you paid it forward by making connections for others?

Please do read this comment from Cindy. I have responded as best I can. I'm sure she would welcome your suggestions.

A Note on Despair

To be in this position - wanting to work, needing to work, knowing you still have much to contribute but never getting a foot in the door - is deeply frustrating, horribly depressing, and leaves us feeling powerless. Add up these elements and you have the formula for despair.

It's brutally hard to fight your way back from despair. But sometimes, an act of compassion can help.

I've been on the receiving end of those incredible kindnesses - from strangers, from readers, and from one friend in particular, herself too long living on the edge.

One small act of compassion can breathe new hope into the worst situation. And here's what I know with 100% certainty. We may be unemployed, we may be depressed… but we aren't powerless if we come together and try to help one another.

... ... ...

[Apr 30, 2016] Over 50 and unemployed Don't panic!

January 3, 2012 | Palmetto Workforce Connections

When you find yourself over 50 and unemployed, the thought of finding another job may seem daunting and hopeless.

It is quite easy to become discouraged because many people fear being stereotyped because of their age, the tough job market, or the prospect of being interviewed by someone half their age. However, there are some things the older unemployed should keep in mind while on the job search. Using the following tips will increase your chances of a short job search and create an overall more pleasant experience.

  1. Quit telling yourself that no one hires older workers. This is simply just not true. In some cases older workers have to exert more effort to overcome discrimination, but this is certainly not the case for every employer. There are even entire websites with jobs posted specifically for older workers, and a quick Google search will render you a list of those websites. Take advantage of such resources!
  2. Take advantage of new technology. Learn to blog and micro-blog, via Twitter, about your profession and interests. You should even create a LinkedIn profile (a website similar to Facebook yet has a more career oriented function) to assist it meeting people in your desired field. All of which will help you stay fine tuned on your skills, while developing new ones. Learning to use social networking will indicate to potential employers that you can adapt to change and learn new things, particularly technology, fairly quickly.
  3. Use all those hard earned contacts. Using contacts, no matter how far in the past they rest, is nothing to be ashamed of! You've probably spent most of your life working, and meeting a lot of people along the way. It is completely acceptable to reach out to former colleagues, class mates, co-workers and employers for job possibilities. Using resources like Facebook or LinkedIn are great ways to find those long lost contacts as well. Chances are they would love to hear from you and help you out if possible.
  4. Don't clutter your resume. Your resume should be tailored to each and every job you apply for. While it is important to showcase your talent and skills, how you present the information is equally important. This means keep it straight to the point and relate your past experience to the skills necessary for the job you are applying for. Essentially, don't do a history dump of every job you've ever had, instead, make each word count!
  5. Don't act superior to the interviewer. It is likely that the people interviewing you will be younger than you. But this does not mean you should look down upon them. Obviously they have earned their position, and if you play your cards right, in due time, you will earn yours! Even if you've worked more years than your interviewer has been alive, it's not okay to tell him or her that you can "teach" them anything. A better idea would be to state your experience working in a multi-generational work place.

Use these tips to help make your job search less stressful and more positive. Whatever you do, don't throw in the towel before you've even tried. Your experience and knowledge will be recognized. All you need is the right employer to identify it.

[Apr 30, 2016] Over 50 and out of work Program seeks to help long-term unemployed

Nov 16 2013 | NBC News
When Bret Lane was laid off from his telecommunications sales job after 16 years, he wasn't worried. He'd never been unemployed for more than a few days since he started working as a teenager. But months passed, and he couldn't find a job. One day, he heard the Purina plant in his Turlock, Calif., neighborhood was hiring janitors for $14 an hour. When he arrived early at 4 a.m., he counted more than 400 people lined up to interview.

"That's when I realized things had gotten serious," said Lane, 53, who called being out of work "pure hell."

Lane's experience is hardly unique. As of September 2013, 4 million people had been unemployed for six months or more. The economy has been slow to regain the 8.7 million jobs lost during the Great Recession, making prospects grim for many of the long-term unemployed.

Older workers like Lane make up a larger percentage of the persistently jobless than ever before. Nearly 40 percent of unemployed workers are over the age of 45 - a 30 percent rise from the 1980s. And for this group, the job hunt can be particularly long and frustrating. Unemployed people aged 45-54 were jobless for 45 weeks on average, and those 55 to 64 were jobless for 57 weeks, according to an October 2013 Associated Press-NORC Center for Public Affairs Research poll.

Younger workers didn't have such a hard time, perhaps because many employers perceive them to be more energetic or productive than older workers, said Linda Barrington, an economist at Cornell University's Institute for Compensation Studies. Employers "acting on such inaccurate assessments or stereotypes is what benefits younger workers and disadvantages older workers," she said.

Addressing the emotional side of unemployment

An innovative program based in Bridgeport, Conn., is helping to get those who are over 50 and unemployed for long periods back into the market. Platform to Employment started in 2011 when a Connecticut job center called the WorkPlace was overwhelmed by calls from "99ers"-people who had been unemployed for 99 weeks, exhausting their unemployment benefits-many of whom were older workers.

The exact number of 99ers across the country is unknown; the Bureau of Labor Statistics hasn't distinguished between 99ers and those out of work for a year since 2010, an oversight that some say renders this group even more politically invisible. Already, the long-term unemployed face biases in hiring. It's both legal and common for employers to write "unemployed need not apply" on job postings.

There has been virtually no public policy tackling long-term unemployment since the recession hit, said P2E founder Joe Carbone, and his program seeks to fill that gap. "These people have lost access to opportunity, which is a basic American tenet," said Carbone. "We find a way to make them competitive and feel hopeful."

P2E is an intensive, individualized five-week bootcamp that teaches job skills and works to build job-seekers' confidence and emotional health. "We acknowledge that there are serious emotional issues for people who'd been unemployed for that long," Carbone said.

The privately-funded program makes deals with businesses who hire P2E graduates for "internships," a few-week trial period for the would-be employee, whose salary is subsidized by the WorkPlace. Often, it leads to full-time work. According to P2E, 80 percent of their participants have been granted trial periods, and of those, more than 85 percent have been hired by employers.

Accepting a new economic reality

Bret Lane washes out his coffee pot at his home after a shift at a call center in San Diego, Calif., on Oct. 31. Lane was laid off after 16 years as a salesman in telecommunications and was unemployed until he got a job at a call center. Sandy Huffaker / Getty Images for NBC News

The program has spread to 10 other cities across the United States, including San Diego, where Lane, a P2E graduate, has been employed full-time at a call center since May. After a year and nine months of unemployment, Lane sold his two-bedroom house, pared down his possessions to fit in a 5x10 storage unit, and drove to San Diego to live with his sister. That's when he saw an ad in the paper for Platform to Employment.

He learned how to make his online resume more searchable by adding keywords, as well as how to create an impressive LinkedIn profile. "It also occurred to me that I was being discriminated against" because of age, rather than being rejected for not being good enough. Lane now makes about half of his previous salary and still lives with his sister, but he's "happy to be working again."

This acceptance of a new economic reality is at the heart of P2E; the program isn't solving the problems of precarity, real-wage decline, or manufacturing losses so much as doing damage control.

"I'd say 100 percent of the people who went through Platform are making less than they did previously," said Carbone. "We get them prepared for the fact that their standard of living will go down, that they probably have to change careers."

This guidance is necessary, Barrington said. "A lot of [the long-term unemployed] came into the workforce still thinking you could work for the same company for your whole life," she said. "Someone has to sit you down and tell you that's not going to happen."

She added that businesses need to be reminded of the value of older workers, who often bring intangible skills, such as punctuality, responsibility, and "being able to write a memo," that younger employees may not yet have.

Heidi DeWyngaert, President of Bankwell, a holding company of several banks in Connecticut, said one of her banks hired an older worker from P2E who is succeeding on the job precisely for these reasons. "She's mature, reliable and responsible with a great attitude," said DeWyngaert.

The program has gained so much prominence that it's become competitive in its own right. Early last year, after P2E was featured on 60 Minutes, the Bridgeport office was flooded with inquiries. The program routinely gets 1,000 applicants for around 20 spots.

Hoping to spark a national conversation

Vanessa Jackson, 57, saw the segment and kept track of P2E's growth until it expanded to her area in Chicago. Jackson had been unemployed off and on since 2008, when she lost her $100,000 job as a marketing manager during a corporate downsizing. "I thought, of course, I would get another comparable job," she said.

But it didn't happen. She decided to get an MBA to "ride out the recession," but that just landed her more debt. She finally got a part-time job as a deli clerk, until she broke her arm and went on disability for 10 months. Her $300,000 401(k) account dwindled to $60,000. She sold her house in the suburbs and moved in with her boyfriend on the South Side of Chicago.

"It was the most desperate thing in the world," Jackson said. It pained her to remember the days when recruiters would tell her she was one of "the top African-American women in marketing."

P2E "revived my energy," she said. "It lifted the depression that was very much there."

Jackson now works part-time as a project coordinator at a home care service agency for $13 an hour, which she admits is inadequate for her level of education. Still, she almost missed out on the opportunity. When P2E came to Chicago earlier this year, she wasn't selected at first. "It felt like applying for a job in itself," she said. "I beseeched [Chicago program manager Michael Morgan]. He said 'I admire your ambition' and let me in."

Carbone is all too aware of P2E's limited reach. "We've helped hundreds of people, but that doesn't put even a small dent in the amount who need help," he said. Carbone hopes to spark a national conversation and, eventually, get the attention of Washington.

"Let's be clear," Carbone said. "I wouldn't be doing this if there were appropriate and relevant government policies."

[Apr 21, 2016] For Mental Health, Bad Job Worse than No Job

Notable quotes:
"... Unemployed people who found a job that rated well in these areas reported a substantial improvement in their mental health. By contrast, newly employed people who felt overwhelmed, insecure about their employment, underpaid, and micromanaged reported a sharp decline in their mental health, including increased symptoms of depression and anxiety. Even those who couldn't find a job fared better. This last finding was "striking," Butterworth says. "This runs counter to a common belief that any job offers psychological benefits for individuals over the demoralizing effects of unemployment." ..."
"... Policymakers should address the impact that the workplace has on mental - and not just physical - health, Butterworth says. "In the same way that we no longer accept workplaces that are physically unsafe or in which employees are exposed to dangerous or toxic substances, there could be a greater focus on ensuring a more positive psychosocial environment at work." ..."
March 14, 2011 | Health.com

With unemployment still high, job seekers who have been discouraged by a lack of work might be inclined to take the first opportunity they're offered. That will help pay the bills, but it could cause other problems: A new study suggests that some jobs are so demoralizing they're actually worse for mental health than no [...]

With unemployment still high, job seekers who have been discouraged by a lack of work might be inclined to take the first opportunity they're offered. That will help pay the bills, but it could cause other problems: A new study suggests that some jobs are so demoralizing they're actually worse for mental health than not working at all.

The findings add a new wrinkle to the large body of research showing that being out of work is associated with a greater risk of mental health problems. In the study, which followed more than 7,000 Australians over a seven-year period, unemployed people generally reported feeling calmer, happier, less depressed, and less anxious after finding work, but only if their new jobs were rewarding and manageable.

"Moving from unemployment to a poor-quality job offered no mental health benefit, and in fact was more detrimental to mental health than remaining unemployed," says the lead author of the study, Peter Butterworth, PhD, a senior research fellow at the Centre for Mental Health Research at the Australian National University, in Canberra.

The study was published in the journal Occupational and Environmental Medicine.

Butterworth and his colleagues analyzed data from an annual survey in which participants described their mental state, their employment status, and-for those with a job-details of the working conditions that they enjoyed (or didn't enjoy, as the case may be). The survey respondents were asked how strongly they agreed with statements such as "My job is complex and difficult" and "I worry about the future of my job."

The researchers focused on four job characteristics that are closely linked with mental health: the complexity and demands of the work, job security, compensation, and job control (i.e., the freedom to decide how best to do the job, rather than being ordered around).

Unemployed people who found a job that rated well in these areas reported a substantial improvement in their mental health. By contrast, newly employed people who felt overwhelmed, insecure about their employment, underpaid, and micromanaged reported a sharp decline in their mental health, including increased symptoms of depression and anxiety. Even those who couldn't find a job fared better.

This last finding was "striking," Butterworth says. "This runs counter to a common belief that any job offers psychological benefits for individuals over the demoralizing effects of unemployment."

Although certain types of jobs-such as working in a customer-service call center-are more likely to be downers, the working environment tends to have a greater impact on mental health than the job description itself, Butterworth adds.

Managers are especially important to employee well-being, says Robert Hogan, PhD, an expert on personality in the workplace and a former chair of the department of psychology at the University of Tulsa. "Bad bosses will make anybody unhappy," Hogan says. "Stress comes from bad managers."

Policymakers should address the impact that the workplace has on mental - and not just physical - health, Butterworth says. "In the same way that we no longer accept workplaces that are physically unsafe or in which employees are exposed to dangerous or toxic substances, there could be a greater focus on ensuring a more positive psychosocial environment at work."

[Mar 23, 2016] How to Lay Off An Employee - the Right Way

Their dirty ways: " One day last October, when employees at Cenovus Energy showed up at the office, many discovered that they couldn't access their computer files on the company's internal system. That's how they found out they were being laid off.
Alberta Oil Magazine

How to Lay Off An Employee - the Right Way

Companies that fought to attract and keep staff have been learning the hard way how to shed them in a hurry. But that doesn't mean it can't – and shouldn't – be done right

One day last October, when employees at Cenovus Energy showed up at the office, many discovered that they couldn't access their computer files on the company's internal system. That's how they found out they were being laid off. Two months earlier, employees at Hutchison Ports Australia in Sydney and Brisbane got a text message, then an email in the middle of the night inviting them to a beachside hotel. They, too, were being laid off.

Cenovus called its move a mistake. Hutchison Ports Australia said it had begun its consultations with staff and unions regarding redundancies in June. Whatever the explanation, companies need to start approaching layoffs more carefully. And though everyone in the energy business is hoping the bloodletting is over, if it isn't, there are ways to soften the blow of layoffs, and do them fairly and transparently.

Communication

A company should keep its employees informed of the economic forces acting on the business and their employment prospects, says Martin Birt, president of HRaskme.com and a human resources consultant with 30 years in the business. "Closures…should never, in my view, be a surprise," he says. Neither should layoffs. You can communicate messages with your employees such as how decisions will be made in what Birt calls a "long-game communications plan," a set of HR principles that will be applied should anything be decided regarding the company's long-term employment potential. That way, employees have some context as to what to expect when market circumstances change.

If you choose not to share your long-game communications plan in your employee manual, when speaking to the people you're laying off, at least communicate how, why and when you made the decision, Birt says. Your actions will get back to suppliers, contractors and layoff survivors. And if you've communicated fairly and awarded appropriate compensation and benefits, the external environment will understand what kind of corporate citizen you are.

Listen to Your Experts

Involve the correct teams – operations, human resources and legal – and involve them as early in the decision-making process as possible, says Birt. These teams will protect you as a corporation from any liability associated with a layoff.

Soften the Blow

Henry Hornstein is an assistant professor at Sault Ste. Marie, Ontario's Algoma University, specializing in organizational change management. In the early 1990s, he was among the staff let go from Imperial Oil's Strathcona refinery. "It was not pleasant," says Hornstein, "but the way Imperial Oil handled that at the time cushioned the blow." They provided him with a year's worth of salary and benefits, and services with an outplacement firm. These included resumé writing, interview training and networking support. "Rather than treating people as commodities, people are treated compassionately," says Hornstein of the experience. You can provide your employees with psychological support in addition to proper severance, benefits and outplacement services, he says. Consider offering group meetings where people can talk to others about the negative psychological impacts of downsizing that they've experienced. "Downsizing is a significant assault on an individual's self-esteem…Everybody has a story, and when somebody is downsized, the organization can [seem to] take an approach that they don't care what the background story is, they just want to get rid of the people." says Hornstein. Birt agrees, saying companies should be prepared to offer an employee assistance program (EAP), a short-term counseling service for employees in need of support. This can also add a buffer against the company's liability.

Confidentiality

Having said that, to maintain confidentiality, limit the planning group to only those whose participation is necessary, says Birt. Consider using specific project-related confidentiality agreements, as well, and clearly describe the consequences for breaching confidentiality. He also suggests reminding participants with pre-existing confidentiality agreements of the terms of those agreements. If you are a publicly traded company, you should know if you are required to first inform the markets of your actions. If that is the case, managers must be prepared to communicate with employees immediately after informing the markets.

Finalize the Details

Before you deliver the news of layoffs, finalize all the details with human resources and legal, including severance, benefits and pension entitlements, says Birt. You'll be prepared to immediately answer individual questions. Everything you say orally in a termination meeting should be captured in a termination letter as well, he says. However, give terminated employees a few days to review their termination package and ask any questions, says Fraser Johnson, a professor at the Ivey School of Business in London, Ontario. "As soon as you hear the words that you're being laid off, your mind might go blank," he says.

Share the Pain

Rather than targeting employees with layoffs, share the cuts across the corporation, just as Canadian Natural Resources did when all staff pay was cut by up to 10 percent. Or, introduce flexible work arrangements like part-time work, voluntary leaves of absence, or deferred compensation in which an employee can work full-time at 80 percent salary for several years before taking a paid sabbatical.

[Mar 04, 2016] What do you call a 50 year old engineer?

www.nakedcapitalism.com

Synoia , March 3, 2016 at 10:25 am

Q: What do you call a 50 year old engineer?

A: Unemployed.

[Jan 10, 2016] The Death of the Professional Are Doctors, Lawyers and Accountants Becoming Obsolete

Notable quotes:
"... Adapted from the new book The Future of the Professions by Richard Susskind Daniel Susskind (Oxford University Press, 2015).Originally published at Alternet ..."
"... The proof is in how there is one premium cost if the medical provider is on their own and magically it is cheaper if theu are part of a group or hospital.. Same doctor…same practices…lower rates…prima facia evidence of insurance company rate fraud… ..."
"... Re solidarity, you might be surprised. One reason law school enrollments are down is that it is becoming public knowledge that employment for graduates in upwardly mobile career positions is way down … ..."
"... Many are shunted into low level proletarian type legal work, churning out evidence for use in lawsuits owned and managed by large firms. Lawyers who do this earn less then a good paralegal with less job security and no benefits. ..."
"... So much of the 'grunt work' of professions – once the entry and training province of new graduates – is now being done overseas by shops that specialize in legal research, or reading x-rays, or accounting and tax preparation. ..."
"... The 'grunt work' that grounds one in the full knowledge of the profession and how it works is slowly removed from the profession. That omission leaves future practitioners with an incomplete understanding. ..."
"... This loss makes them more reliant on big data as both assistant and excuse/defense, and makes them less master craftsmen (if I may use the term without giving offense) and more the front-end interface of one-size-fits-all processes. Very good for corporate profits. Not so good for the professions or their clients. ..."
"... Long ago, firms started off-shoring basic, tedious, repetitive tasks, generally considered as unrewarding, such as software testing or error correction to India. The idea was to focus on "high added-value" jobs such as system architects or project management, and leave low-level operations, supposedly requiring less qualifications, to cheaper Indian contractors. Decades later, there is a shortage of qualified people for those high-skilled jobs - precisely because fewer and fewer young people have had the possibility ..."
"... The result? It is now necessary to import expensive project managers and system architects from foreign countries. ..."
"... Bottom line: the race to the bottom for wages is "on". ..."
"... Professionals would be the next logical choice of squeezing cost out of work; unions, middle management, big industry, airlines, manufacturing and construction have all paid their price at the alter of the 1%. ..."
"... I also agree with the concept of there being less for the bottom 90% to spend. And as more automation kicks in, there will be even less bad choice jobs for these folks to scramble for. Just waiting for truck drivers to be slowly replaced with auto-drive trucks. ..."
"... "…. Prefer a fence at the top of the cliff to an ambulance at the bottom…" ..."
"... The rich and the truly rich will always have skilled, artistic human professionals to serve their personally tailored bespoke needs. It is the rest of us who will be assigned the doctorobots, the lawyer machines, etc. ..."
"... Part of the "crapification of everything" … except for managers and owners, it is part of their cost cutting plan. ..."
"... First they came for the blue collar workers, and I did nothing? Then they came for the white collar workers, and I did nothing? Now they are coming for the professionals, and they are laughing at my passivity? ..."
"... They have played all the classes, higher than the one they are currently discarding, and the remaining consumers are happy to throw their neighbors under the bus. But your turn will come. Karma. ..."
"... Once corporations start setting guidelines and dictating the drugs you can and can't use for treatment, do you think they'll do it according to what's cost effective and least risky for the patient based on current science or do you think they'll do it based on their own profits? ..."
"... The shift from reactive to proactive ..."
"... Proactive cannibalism ..."
www.nakedcapitalism.com

naked capitalism

Posted on January 9, 2016 by Yves Smith Yves here. Many members of the top 10% regard their role in society as relatively secure, particularly if the are in a niche that serves the capital-deploying 1% or better yet, 0.1%. But a new book suggest their position is not secure. And trends in motion confirm this dour reading, such as the marked decline in law school enrollments, and the trend in the US to force doctors to practice out of hospitals or HMOs, where they are salaried and are required to adhere to corporate care guidelines. For instance, my MD is about to have her practice bought out, and is looking hard as to whether she can establish a concierge practice. Mind you, she appears regularly on TV and writes a monthly column for a national magazine [not that is how I found her or why I use her]. Yet she has real doubts as to whether she can support all the overhead. If someone with a profile can't make a go at it solo in a market like Manhattan, pray tell, who can?

Adapted from the new book The Future of the Professions by Richard Susskind & Daniel Susskind (Oxford University Press, 2015).Originally published at Alternet

The end of the professional era is characterized by four trends: the move from bespoke service; the bypassing of traditional gatekeepers; a shift from a reactive to a proactive approach to professional work; and the more-for-less challenge.

The Move From Bespoke (Custom) Service

For centuries, much professional work has been handled in the manner of a craft. Individual experts and specialists-people who know more than others-have offered an essentially bespoke service ("bespoke" is British for "custom"). In the language of the tailor, their product has been "made-to-measure" rather than "off-the-peg." For each recipient the service has been disposable (used once only), handcrafted ordinarily by a solitary scribe or sole trusted adviser, often in the spirit of an artist who starts each project afresh with a blank canvas.

Our research strongly suggests that bespoke professional work in this vein looks set to fade from prominence, as other crafts (like tailoring and tallow chandlering) have done over the centuries. Significant elements of professional work are being routinized: in checklists, standard form materials, and in various sorts of systems, many of which are available online. Meanwhile, the work that remains for human beings to handle conventionally is often not conducted by individual craftspeople, but collaboratively in teams, sometimes collocated, but more often virtually. And, with the advance of increasingly capable machines, some work may not be conducted by human beings at all.

Just as we witnessed the "death of gentlemanly capitalism" in the banks in the 1980s, we seem to be observing a similar decline in bespoke professionalism.

The Bypassed Gatekeepers

In the past, when in need of expert guidance we turned to the professions. Their members knew things that others did not, and we drew on their knowledge and experience to solve our problems. Each profession acted as a "gatekeeper" of its own, distinct body of practical expertise. Today this set-up is under threat.

We are already seeing some work being wrested from the hands of traditional professions. Some of the competition is coming from within. We observe professionals from different professions doing each other's work. They even speak of "eating one another's lunch." Accountants and consultants, for example, are particularly effective at encroaching on the business of lawyers and actuaries. We also see intra-professional friction, when, for example, nurses take on work that used to be exclusive to doctors, or paralegals are engaged to perform tasks that formerly were the province of lawyers.

But the competition is also advancing from outside the traditional boundaries of the professions-from new people and different institutions. We see a recurring need to draw on people with very different skills, talents, and ways of working. Practicing doctors, priests, teachers, and auditors did not, for example, develop the software that supports the systems that we describe. Stepping forward instead are data scientists, process analysts, knowledge engineers, systems engineers, and many more. Today, professionals still provide much of the content, but in time they may find themselves down-staged by these new specialists. We also see a diverse set of institutions entering the fray-business process outsourcers, retail brands, Internet companies, major software and service vendors, to name a few. What these providers have in common is that they look nothing like twentieth-century doctors, accountants, architects, and the rest.

More than this, human experts in the professions are no longer the only source of practical expertise. There are illustrations of practical expertise being made available by recipients of professional work-in effect, sidestepping the gatekeepers. On various platforms, typically online, people share their past experience and help others to resolve similar problems. These "communities of experience," as we call them, are springing up across many professions (for example, PatientsLikeMe and the WebMD communities in medicine). We say more about them in a moment. More radical still are systems and machines that themselves generate practical expertise. These are underpinned by a variety of advanced techniques, such as Big Data and artificial intelligence. These platforms and systems tend not to be owned and run by the traditional professions. Whether those who do so will in turn become "new gatekeepers" is a subject of some concern.

The keys to the kingdom are changing. Or, if not changing, they are at least being shared with others.

Jim Haygood ,, January 9, 2016 at 8:57 pm

'medium and large corporations are also struggling to deal with increasing regulation'

My claim is that large corporations don't "struggle to deal with" regulation - they write it.

Case in point, Obamacare was drafted by Liz Fowler, formerly of WellPoint.

alex morfesis , January 10, 2016 at 12:05 am

You nailed it on medical professionals…would like to add, that at least here in flori duh there seems to be massive pricing fraud by malpractice and liability insurance providers which state regulators allow to continue to force small or single practitioners to join groups by financial obliteration…at least in floriduh, there is the usual massive distortion suggesting insurance companies are paying out huge amounts when there in fact seems to be collusion amongst insurance companies neglecting the legal requirement to try to settle on good faith and end up forcing people to settle for pennies on the dollar…yet the insurance companies keep picking the pockets of medical professionals

The proof is in how there is one premium cost if the medical provider is on their own and magically it is cheaper if theu are part of a group or hospital.. Same doctor…same practices…lower rates…prima facia evidence of insurance company rate fraud…

jrs , January 9, 2016 at 3:49 pm

Yes some of it is only logical though, if masses of the population see their income declining and yet the costs of medical care keeps increasing eventually noone can afford to see the doctor never mind the ACA etc.. And it can get to be this way with a lot of professional services less urgent and distorted than medical care, like soon noone can afford an accountant, you use turbo tax, a lawyer – no middle class people start to make their own wills. Many professions seek ever further protections of government for their guilds (more and more requirements to practice to try to preserve their privilege) and yet with nothing protecting the income of the other 80% (read: unions, that would be their role) unless they plan to only serve the fellow 20% …

So solidarity? Yea, but making the solidarity argument with many (not all) members of such professions is a waste of time as they instinctively side with the 1s.

Local to Oakland , January 9, 2016 at 4:13 pm

Re solidarity, you might be surprised. One reason law school enrollments are down is that it is becoming public knowledge that employment for graduates in upwardly mobile career positions is way down …

Many are shunted into low level proletarian type legal work, churning out evidence for use in lawsuits owned and managed by large firms. Lawyers who do this earn less then a good paralegal with less job security and no benefits.

ilporcupine , January 9, 2016 at 4:33 pm

It has been said Paralegals are being squeezed out, to make way for the huge increase in law graduates from prior class booms. Why not use cheap lawyers, with better credential, and desperate for employment?

flora , January 9, 2016 at 5:39 pm

So much of the 'grunt work' of professions – once the entry and training province of new graduates – is now being done overseas by shops that specialize in legal research, or reading x-rays, or accounting and tax preparation.

There are 3 downsides to this, in my opinion. New college grads have fewer entry slots. The 'grunt work' that grounds one in the full knowledge of the profession and how it works is slowly removed from the profession. That omission leaves future practitioners with an incomplete understanding.

This loss makes them more reliant on big data as both assistant and excuse/defense, and makes them less master craftsmen (if I may use the term without giving offense) and more the front-end interface of one-size-fits-all processes. Very good for corporate profits. Not so good for the professions or their clients.

guest , January 9, 2016 at 6:25 pm

Big Data is not a solution.

Your first two points (no entry-level jobs for beginners, no acquisition of professional basics) are essential - and their detrimental effects are already painfully felt in some professions.

Case in point: software development.

Long ago, firms started off-shoring basic, tedious, repetitive tasks, generally considered as unrewarding, such as software testing or error correction to India. The idea was to focus on "high added-value" jobs such as system architects or project management, and leave low-level operations, supposedly requiring less qualifications, to cheaper Indian contractors. Decades later, there is a shortage of qualified people for those high-skilled jobs - precisely because fewer and fewer young people have had the possibility to

(a) start in the profession at entry-level positions (when job postings all require qualifications as senior software engineer and five years experience, what do you do?)

(b) learn the ropes and practice the skills from the ground up (the necessary step before rising in the professional hierarchy).

The result? It is now necessary to import expensive project managers and system architects from foreign countries.

From what I read, the UK has been especially hit by this phenomenon, because it was particularly enthusiastic about off-shoring IT to India.

polecat , January 9, 2016 at 8:18 pm

Uhm…..oh wait…uh ..I know….uh…Brondo's got what plants need…..right?

Phil , January 10, 2016 at 2:34 am

Attorney's work is being automated and outsourced. For more on one aspect of outsourcing:

http://www.nytimes.com/2010/08/05/business/global/05legal.html?_r=2

I can't find the cite, but last year I read that some of the Indian companies that American law firms have outsourced to are now moving offices "stateside" to hire American attorneys, here.

Bottom line: the race to the bottom for wages is "on". Add to this job automation that will only get more efficient, over time.
http://www.futuretech.ox.ac.uk/news-release-oxford-martin-school-study-shows-nearly-half-us-jobs-could-be-risk-computerisation

armchair , January 9, 2016 at 5:17 pm

The Washington State Bar has initiated a legal technician program , and I find the timing questionable, even if the premise of the program is good-hearted. As the market is awash in underemployed, licensed attorneys, the Bar is going ahead and turning veteran paralegals into the people to undercut the market even further. It seems like bad timing to let someone who has years of experience, and no law school debt get over on a bunch law school grads who are facing a life of being hounded for their debts. I spoke to someone at the Bar who made a good defense, that the legal technician is like an ARNP. Only later did it occur to me that there are very few out-of-work doctors.

From another perspective, the legal technician answers another problem of the collapsing paralegal market. Much of the collapse has been driven by advances in document management, especially scanning that 'reads' the text and makes it searchable. But hey, here is a shiny new program. Go ahead and set up a parenting plan with your abusive ex for $75! What could go wrong?

The key to really get the legal field de-humanized would be robot judges and robotic juries. I hope someone is already working on it.

polecat , January 9, 2016 at 8:26 pm

Don't worry…what's old is new again. At some point in the future we'll all be scratching glyphs on clay tablets….once the 2nd law of thermodynamics really kicks in………..plenty of work then!

armchair , January 9, 2016 at 9:02 pm

Work! What about George Jetson? The go west value system we are stuck with these days is almost perfectly incompatible with a future that requires very little human labor.

MyWag , January 9, 2016 at 5:33 pm

Professionals would be the next logical choice of squeezing cost out of work; unions, middle management, big industry, airlines, manufacturing and construction have all paid their price at the alter of the 1%.

Public sector unions are hanging on but as the majority of local & state taxpayers have less to give, these wages, benefits and especially pensions will be cut. Those earning less and less will gleefully pull down those public employees who are 'living like kings'.

I also agree with the concept of there being less for the bottom 90% to spend. And as more automation kicks in, there will be even less bad choice jobs for these folks to scramble for. Just waiting for truck drivers to be slowly replaced with auto-drive trucks.

This leads us to an enhanced confrontation at the Federal level on how to go forward. The earned income tax credit, a good concept also under siege, I believe, will have to be supplemented with a minimum guaranteed income.

By this time, 20 years, the DEMs will be the party of business and the GOP will be entirely dependent on fed govt subsidies. Oh the irony.

Brooklin Bridge , January 10, 2016 at 12:00 pm

By this time 20 years, the GOP will be saying, "I told you so", regarding Global Warming.

Ptup , January 9, 2016 at 6:12 pm

Reading Rise of a The Robots right now, and the law and accounting profession have and will continue to be hurt hard by computers armed with big data, and the education and medical profession are next. Has to be. It's already a travesty that education and medical costs continue to rise as incomes stagnate and drop, and that just cannot continue. Well, maybe it can, until all of those guns out there are used by the people as they rise up. Look at the buffoon who many are considering for the Republican nominee, more out of blind, misinformed anger, than anything. Scary.

RBHoughton , January 9, 2016 at 7:31 pm

"…. Prefer a fence at the top of the cliff to an ambulance at the bottom…"

You have a delightful way with words Yves. Many thanks.

different clue , January 9, 2016 at 9:19 pm

The rich and the truly rich will always have skilled, artistic human professionals to serve their personally tailored bespoke needs. It is the rest of us who will be assigned the doctorobots, the lawyer machines, etc.

James Koss , January 10, 2016 at 11:13 am

The French phrase "Everything changes and remains the same" remains true today.

Whereas today the top of society has its professionals to isolate and protect them from the remainder of the population and the rules nobility and the church had its knights, nobles, obedient serfs and peasants to fight and protect "their" nobility. Names and titles changed but the rules remained. Those who have will get those who don't will not.

Inverness , January 10, 2016 at 11:29 am

Correct. The same applies in education. The wealthy know what kinds of schools serve their children best: those with better teacher to student ratios, rich arts curricula, and a progressive approach to instruction. Just see what Obama's kids got at their fancy Quaker school. The rest get standardized lesson plans, big class sizes, deep cuts in music and the arts, and high-stakes testing.

They can privatize their lives; we cannot.

Disturbed Voter , January 9, 2016 at 10:42 pm

Part of the "crapification of everything" … except for managers and owners, it is part of their cost cutting plan.

Why would you trust a medical system run by politicians and insurance companies … a system promoted by those same managers and owners. Like hiring the Three Stooges as your plumber, electrician and roofer. Gullibility will be the death of us … that and malice.

First they came for the blue collar workers, and I did nothing? Then they came for the white collar workers, and I did nothing? Now they are coming for the professionals, and they are laughing at my passivity?

They have played all the classes, higher than the one they are currently discarding, and the remaining consumers are happy to throw their neighbors under the bus. But your turn will come. Karma.

flora , January 10, 2016 at 2:19 am

In Oregon some doctors are unionizing to resist medical assembly line medicine.
From NYTimes:

Doctors Unionize to Resist the Medical Machine

"Dr. Alexander and his colleagues say they are in favor of efficiency gains. It's the particular way the hospital has interpreted this mandate that has left them feeling demoralized. If you talk to them for long enough, you get the distinct feeling it is not just their jobs that hang in the balance, but the loss of something much less tangible - the ability of doctors everywhere to exercise their professional judgment."

http://www.nytimes.com/2016/01/10/business/doctors-unionize-to-resist-the-medical-machine.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=second-column-region&region=top-news&WT.nav=top-news&_r=0

digi_owl , January 10, 2016 at 4:12 am

I find myself thinking about an episode of the original Connections series, that was produced in the 70s.

There it was mused about how corporate management would idle their days away waiting for the computer in the basement to crunch the numbers and come up with company decisions they were then to implement.

Instead what happened was that the professional managerial class, the MBAs, dug in while computers instead replaced the laborers via robotics.

Jesper , January 10, 2016 at 6:55 am

Or shorter: The common argument that 'we (by that I mean you) have to become more employable' is about to hit home among the people with long education. Will they recognize the similarity to what has already happened to others and/or will they themselves make themselves more 'employable'?

financial matters , January 10, 2016 at 8:11 am

I think one of the major consequences we are seeing as a result of a misguided professional system is the lack of basic legal services for millions of people. This resulted in people being thrown out of their homes as the result of very obvious fraud and yet having no recourse unless they were able to spend hundreds of thousands of dollars on legal fees.

I think the popular new series 'Making of a Murderer' emphasizes this problem. I don't think a show that emphasizes the problems that the very poor have with justice from the lack of being able to pay for legal services would have been this popular 10 years ago.

financial matters , January 10, 2016 at 8:17 am

I think this would require a 'single payer' legal system similar to the need for a single payer medical system.

Wade Riddick , January 10, 2016 at 8:53 am

Once corporations start setting guidelines and dictating the drugs you can and can't use for treatment, do you think they'll do it according to what's cost effective and least risky for the patient based on current science or do you think they'll do it based on their own profits?

What happens when they own their own pharmacies – as they're all scrambling to do right now – and try to jack up reimbursement through that unit too? Do you think patients were served when Philidor started (criminally) altering scripts and making substitutions?

For profit healthcare is really sickcare, isn't it? Why cure a disease when treating it brings in more revenue? Why sell cheap human insulin when you can patent a variety on the molecule, jack up the price and carve up the market?

Keep the sucker paying the vig…

These guys aren't adopting better guidelines for treating chronic disease based on the best available science. In fact, as they corporatize they're getting worse. I've talked to these clowns. They're typically ten years behind the state of the art in their field. Patients do the reading and then they stare at us like we're morons. Fifteen years later they swear they knew the truth all along.

If these corporate suits are setting the guidelines for care, how come there's no common national board standard for care, no portfolio investment model approach where they model the disease with the best available experts, determine how to intervene in the various genetic pathways that are perturbed and then pick the simplest, cheapest methods/chemicals to try first?

That sounds like a pretty reasonable, scientific approach to treatment – but, if that's your standard, then these people are in breech of fiduciary duty left and right and it all has to do with that old canard "maximizing shareholder value." What about maximizing customer service? Corporate medicine will lead to tobacco-level deaths. I know doctors who have been personally injured in this system already. Corporations want to avoid risk to their profit – *not* their patient. Imagine what *those* mandatory arbitration clauses are going to look like. Imagine what the sequel to _Merchants of Doubt_ will look like in the era of corporate medicine and Supreme Court decisions that bust doctors' unions.

I'm still burning from Peter Thiel's comments on monopolies in the New York Times this morning. Does he have any clue how bad the service is in regional hospital cartels already and how fast prices are rising?

It's not even a matter of price in the drug markets now. It's basic availability. Aside from the persistent shortages of cheap, effective generics due to the kickback scheme in PMOs/PBMs, we now have explicit regulatory interference. The FDA has been moving to withdraw entire lines of medication from compounding pharmacies even when there's no rival big pharma product competing against them or any indication of patient risk. These are decades-old treatments. (It's the CDC's job to set treatment guidelines, by the way, not the FDA's).

It's just a knee-jerk reaction at this point to protect imaginary future profits, I suppose. You can't make up this stuff. The FDA has even imposed a 30% sales volume rule for "safety." It has nothing to do with purity or contamination of compounded products. If Tesla sold exploding cars, how would restricting 30% of their sales volume to California improve consumer safety? It's clearly a market-rigging reg – and it's because the corporate medicine lobby wants it.

What does this have to do with corporate medicine? Compounding pharmacies in big chain hospitals – which are often pitifully narrow in their professional scope – are all magically exempt (oligopolistic and more expensive too). Isn't that wonderful?

The current corporatization of medicine rests on the notion that the chief challenge faced by those of us with serious illnesses is that we simply don't read enough fine print or fill out enough paperwork.

If you think that corporations have done a fine job handling your retirement investments in this era of lax accounting standards, wait until you see what they do with your actual body.

Brooklin Bridge , January 10, 2016 at 1:00 pm

Exceptional comment!

Brooklin Bridge , January 10, 2016 at 12:18 pm

This article is based on the faulty perception that this is all normal benign efficiency working it's way out of an antiquated system, perhaps with a few -to be expected- hiccups. It isn't.

What we are experiencing is wholesale greed and corruption on an international scale working it's way into the core of our civilization like mold or cancer, and perverting technology as well as the process of social change and adjustment to that change – for it's exclusive benefit – as it goes. It is unconscionable that we could call this progress or adjustment in anything but the most cruelly ironic sense.

The shift from reactive to proactive my foot! 60 years ago doctors were getting out proactive messages far better than today via education, television, the media and so on. And they gave a damn!!! Today, insurance companies are devising ever new ways to minimize what they spend on your care, maximize what they charge you for it, and call it, "proactive." Proactive theft, or genocide for fun and profit, would be closer to the mark.

Brooklin Bridge , January 10, 2016 at 12:26 pm

Proactive cannibalism also comes to mind…

[Jan 09, 2016] How people survive when economy dies

Notable quotes:
"... His motto (which he shares with me every other day), is, Never throw anything away . And damned if he cant find exactly what I need when I come over to scrounge at what I call, Our Store . ..."
"... We wear our clothes out and watch what we buy. We dont travel by air, and limit trips to local visits with family. In the future, perhaps this will be the norm for most instead of todays extravagent consumption which is thought of as normal for Canadians and a birthright. ..."
"... Perhaps as a group, as a species, it seems as if we never learn and make the same and even greater mistakes over and over. But as individuals we can try to do things better, live better; until we cant go on. That is my plan, and I am sticking with it until I cant go on. ..."
"... the banjo prevailed and damned if it isnt getting better. I am well on the way to mastering (to use that term loosely) an Iris Dement version of Leaning on the Everlasting Arms , I can See Clearly Now by Johnny Nash, and I Believe In You by Don Williams. I think I have sub-conciously chosen these songs to combat my own confessions of doom and gloom. ..."
peakoilbarrel.com

aws. , 01/08/2016 at 10:50 pm

Back in early 2012, the Premier of Ontario suggested that the loonie (Canadian dollar) was becoming a petro-dollar. He was slapped down by the Cons, and walked back his comment.
"That has knocked the wind out of Ontario exporters and manufacturing in particular," explained McGuinty.

"The only reason the dollar is high - it's a petro dollar , right? It's been driven by the global demand for oil and gas to be sourced in Western Canada.

"So if I had my preferences, as to whether we have a rapidly growing oil-and-gas sector in the West or a lower dollar benefiting Ontario, I'll tell you where I'd stand - with the lower dollar."

Canadian Dollar's Worst Rout Ever Raises Petro-State Worries

Ari Altstedter, Bloomberg, January 4, 2016

By the middle of 2014, oil's share of Canada's total exports reached 19 percent from about 6 percent a decade earlier. Meanwhile, the Ontario-based auto industry was seeing its share of the export pie fall to to 14 percent from 22 percent. The heavier reliance on crude became an issue in last October's national election, as Harper and his Western-based Conservatives were accused by all their opponents of having favored oil to the detriment of other regions.

In the process, the Canadian dollar had effectively joined the ranks of petro-currencies. The correlation between movements in the price of oil and the loonie has increased five-fold since 2000, according to data compiled by Bloomberg. In 2015, while all commodity-exporting countries faced currency pressure, the Canadian dollar was more sensitive to oil price movements than such petro-states as Mexico, Norway and Russia.

I don't actually see low oil prices re-balanacing the Canadian economy. The cure for low oil prices is low oil prices. Would you invest in an export dependent industry in Canada? One can reasonably model a scenario where the price of oil goes up to $100/barrel, the loonie returns to parity with the greenback… and an exporters competitiveness disappears with our petro-loonie's parity with the U.S. dollar.

Being a petro-state makes for a pretty ugly domestic economy.

Paulo , 01/08/2016 at 11:40 pm
AWS (and Ron)

I believe the decline will bring us Canadians back to our roots and strengths. Personally, I have been disgusted with our past 30 year transformation into urban consumers, no matter what part of the country we live in.

I remember my Grandparents playing penny poker on winter evenings. I grew up with stories of the Depression. While I am 60, my good friend down the road is 75. He often tells me about living in our Valley from '46 onwards…..a time of bailing water from the river into a 45 for home supply, canned venison and salmon for winter, oil lamps because Hydro did not arrive until the latter '60s. His Dad built up a sawmill and his folks provided room and board for 'the crew'. His mom washed their clothing by hand on Saturdays and finally got a gas powered ringer washer to make it easier. Nowadays, he scrounges scrap steel (old bedframes and the like) from the recycling bins for our welding projects. He helps me make up power saw chains from scraps and pieces. His motto (which he shares with me every other day), is, "Never throw anything away". And damned if he can't find exactly what I need when I come over to scrounge at what I call, "Our Store".

I don't know what will happen to the Vancouverites or Torontonians when property values dive. I imagine that many will lose everything they think they have, (when their debt bomb blows). I guess then we will see what people are made of. Will they whine? Or will they pick themselves up and make the best of it?

As for Ron's post, it is similar to one last year. I could hardly read that one as well. Yes, there are deserts and sewers made by man, and that will be the best that many can hope for. But what is your sphere of influence and power to change things? I have replanted several thousand trees on our property and let most of it regen into a bramble-filled mixed forest. I have put in a pond that trout have found from the drainage ditches and flooded wetlands next door. I have cut trails for deer and elk crossing routes. We grow our gardens without pesticides and with as much compost as possible. We wear our clothes out and watch what we buy. We don't travel by air, and limit trips to local visits with family. In the future, perhaps this will be the norm for most instead of today's extravagent consumption which is thought of as normal for Canadians and a birthright.

Ron, the facts are glum. Your story is true. I accept that. What I don't accept is allowing it to bring me down and giving up….on myself, my loved ones, and my people. Perhaps as a group, as a species, it seems as if we never learn and make the same and even greater mistakes over and over. But as individuals we can try to do things better, live better; until we can't go on. That is my plan, and I am sticking with it until I can't go on.

I am teaching myself to play the banjo. (My blessed wife is so so patient). Today, the weather was cold and foggy, my lumber is frozen into ice lumps, and I was quite house bound being sick of crunching around my frozen yard trying to be productive. So, the banjo prevailed and damned if it isn't getting better. I am well on the way to mastering (to use that term loosely) an Iris Dement version of "Leaning on the Everlasting Arms", "I can See Clearly Now" by Johnny Nash, and "I Believe In You" by Don Williams. I think I have sub-conciously chosen these songs to combat my own confessions of doom and gloom. Sometimes, it is all we have. For you post readers, I will provide the Youtube links for a pick-me-up. A nice glass of whiskey makes for a good listening partner.

regards and thanks for your heartfelt honesty and efforts

https://www.youtube.com/watch?v=ZGbxrNqK4-4
https://www.youtube.com/watch?v=pSsqWHtg7Ig
https://www.youtube.com/watch?v=Biz5kBIAtic

[Jan 09, 2016] Overshoot The Ecological Basis of Revolutionary Change

www.amazon.com

Amazon.com

By J. Mann on April 8, 2005

Masterpiece, offers solution for THE problem of our time/div> I am astonished at the quality of this book, which is about the eighth book in a personal reading program that included Paul Roberts' The End of Oil, Kenneth Deffeyes' Beyond Oil, Jared Diamon's Collapse, Cottrell's Energy and Society, Michael Klare's Blood and Oil, and others, all extremely good and relevant books. The task this author undertakes is to help readers find a new perspective from which to constructively and usefully interpret inevitable and major changes the world around us. By taking this approach, the author is providing the very essential tool we need to cope with these changes.

The issue is our ecological footprint.

Catton uses the term "Age of Exuberance" to represent the time since 1492 when first a newly discovered hemisphere and then the invention of fossil-fuel-driven machines allowed Old-World humans to escape the constraints imposed by a population roughly at earth's carrying capacity, and instead to grow (and philosophize and emote) expansively. He then reminds us that we are soon to be squeezed by the twin jaws of excessive population and exhausted resources, as our current population is utterly dependent on the mining and burning of fossil energy and its use to exploit earth's resources in general. In spring 2005, the buzz about "the end of cheap energy" is reaching quite a pitch, and when and if the "peak oil" scenario (or other environmental limit-event) is reached, the impact on our social / political world will be enormous. Already the US is brandishing and using its superior weaponry to sieze control of oil assets; this same kind of desperate struggle may well erupt at all levels of society if we don't find a way to identify the problem, anticipate its consequences, and find solutions. Catton offers a perspective based on biology / ecology -- not bad, since we are indeed animals in an ecology and we are indeed subject to the iron laws of nature and physics.

With this perspective we can avoid ending up screaming nonsense at each other when changes begin to get scary. My urgent recommendation is, read this G.D. book and do it now.

[Jan 09, 2016] Even a cursory look at things reveals the overwhelming scope of things and quickly leads to despair

Notable quotes:
"... It has to be explained that Stoics believe that nothing external to the individual is secure, and thus the truly important thing is virtue, based on ethics and moral. ..."
"... Stoicism is the appropriate philosophy for what awaits us. It brings out the best of us and it eases the anguish. The illusion of control is our worst enemy. Matters are completely out of our control and Nature will deal with them as she pleases. ..."
peakoilbarrel.com
Javier , 01/09/2016 at 5:29 am

I wholeheartedly agree that even a cursory look at things reveals the overwhelming scope of things and quickly leads to despair.

It doesn't have to lead to despair. I recommend Stoicism , which is the way Greeks and Romans coped with their own decline.

In the words of Seneca:

"Let Nature deal with matter, which is her own, as she pleases; let us be cheerful and brave in the face of everything, reflecting that it is nothing of our own that perishes." (De Provid. v.8)

It has to be explained that Stoics believe that nothing external to the individual is secure, and thus the truly important thing is virtue, based on ethics and moral. Virtue can not be taken from an individual whatever the circumstances, and helps him deal with adversity. That is what Seneca means with "nothing of our own that perishes" .

Stoicism is the appropriate philosophy for what awaits us. It brings out the best of us and it eases the anguish. The illusion of control is our worst enemy. Matters are completely out of our control and Nature will deal with them as she pleases.

[Dec 26, 2015] BOLD ENDEAVORS Lessons from Polar and Space Exploration

Amazon.com
Britta Sahlgren, October 16, 1999
An intriguing story of human relationships in the extreme.

Bold Endeavors by Jack Stuster proved to be a real page-turner! Since childhood reading about adventures and explorers had been my favorite literature. In this book the persons behind these endeavors came to life. They were of flesh and blood and you as a reader took part of their everyday life, their hardships and personal problems. A thrilling experience. A lesson in the importance of relationships not only among people in isolation A lesson of use at job interviews, schools and even in families. I am thankful for an added knowledge and understanding of the many problems associated with these Endeavors. This book should be a "must" to all young people.

[Dec 26, 2015] Is the Unemployment Rate Tied to the Divorce Rate

Notable quotes:
"... but only before 1980 ..."
"... but only when she had a job. ..."
Science of Relationships
The first study considers government data from all 50 U.S. states between the years 1960 and 2005.1 The researchers predicted that higher unemployment numbers would translate to more divorces among heterosexual married couples. Most of us probably would have predicted this too based on common sense-you would probably expect your partner to be able to hold down a job, right? And indeed, this was the case, but only before 1980. Surprisingly, since then, as joblessness has increased, divorce rates have actually decreased.

How do we explain this counterintuitive finding? We don't know for sure, but the researchers speculate that unemployed people may delay or postpone divorce due to the high costs associated with it. Not only is divorce expensive in terms of legal fees, but afterward, partners need to pay for two houses instead of one. And if they are still living off of one salary at that point, those costs may be prohibitively expensive. For this reason, it is not that uncommon to hear about estranged couples who can't stand each other but are still living under the same roof.

The second study considered data from a national probability sample of over 3,600 heterosexual married couples in the U.S. collected between 1987 and 2002. However, instead of looking at the overall association between unemployment and marital outcomes, they considered how gender and relationship satisfaction factored into the equation. 2

They also looked at marital breakup more generally, focusing on when couples decided to end their relationships (not necessarily if or when they got divorced). Their findings revealed that when men were unemployed, the likelihood that either spouse would leave the marriage increased. What about the woman's employment status? For husbands, whether their wife was employed or not was seemingly unimportant-it was unrelated to their decision to leave the relationship. It did seem to matter for wives, though, but it depended upon how satisfied they were with the marriage.

When women were highly satisfied, they were inclined to stay with their partner regardless of whether they had employment. However, when the wife's satisfaction was low, she was more likely to exit the relationship, but only when she had a job.

[Dec 06, 2015] Bash For Loop Examples

A very nice tutorial by Vivek Gite (created October 31, 2008 last updated June 24, 2015). His mistake is putting new for loop too far inside the tutorial. It should emphazied, not hidden.
June 24, 2015 | cyberciti.biz

... ... ...

Bash v4.0+ has inbuilt support for setting up a step value using {START..END..INCREMENT} syntax:

#!/bin/bash
echo "Bash version ${BASH_VERSION}..."
for i in {0..10..2}
  do
     echo "Welcome $i times"
 done

Sample outputs:

Bash version 4.0.33(0)-release...
Welcome 0 times
Welcome 2 times
Welcome 4 times
Welcome 6 times
Welcome 8 times
Welcome 10 times

... ... ...

Three-expression bash for loops syntax

This type of for loop share a common heritage with the C programming language. It is characterized by a three-parameter loop control expression; consisting of an initializer (EXP1), a loop-test or condition (EXP2), and a counting expression (EXP3).

for (( EXP1; EXP2; EXP3 ))
do
	command1
	command2
	command3
done

A representative three-expression example in bash as follows:

#!/bin/bash
for (( c=1; c<=5; c++ ))
do
   echo "Welcome $c times"
done
... ... ...

Jadu Saikia, November 2, 2008, 3:37 pm

Nice one. All the examples are explained well, thanks Vivek.

seq 1 2 20
output can also be produced using jot

jot – 1 20 2

The infinite loops as everyone knows have the following alternatives.

while(true)
or
while :

//Jadu

Andi Reinbrech, November 18, 2010, 7:42 pm
I know this is an ancient thread, but thought this trick might be helpful to someone:

For the above example with all the cuts, simply do

set `echo $line`

This will split line into positional parameters and you can after the set simply say

F1=$1; F2=$2; F3=$3

I used this a lot many years ago on solaris with "set `date`", it neatly splits the whole date string into variables and saves lots of messy cutting :-)

… no, you can't change the FS, if it's not space, you can't use this method

Peko, July 16, 2009, 6:11 pm
Hi Vivek,
Thanks for this a useful topic.

IMNSHO, there may be something to modify here
=======================
Latest bash version 3.0+ has inbuilt support for setting up a step value:

#!/bin/bash
for i in {1..5}
=======================
1) The increment feature seems to belong to the version 4 of bash.
Reference: http://bash-hackers.org/wiki/doku.php/syntax/expansion/brace
Accordingly, my bash v3.2 does not include this feature.

BTW, where did you read that it was 3.0+ ?
(I ask because you may know some good website of interest on the subject).

2) The syntax is {from..to..step} where from, to, step are 3 integers.
You code is missing the increment.

Note that GNU Bash documentation may be bugged at this time,
because on GNU Bash manual, you will find the syntax {x..y[incr]}
which may be a typo. (missing the second ".." between y and increment).

see http://www.gnu.org/software/bash/manual/bashref.html#Brace-Expansion

The Bash Hackers page
again, see http://bash-hackers.org/wiki/doku.php/syntax/expansion/brace
seeems to be more accurate,
but who knows ? Anyway, at least one of them may be right… ;-)

Keep on the good work of your own,
Thanks a million.

- Peko

Michal Kaut July 22, 2009, 6:12 am
Hello,

is there a simple way to control the number formatting? I use several computers, some of which have non-US settings with comma as a decimal point. This means that
for x in $(seq 0 0.1 1) gives 0 0.1 0.2 … 1 one some machines and 0 0,1 0,2 … 1 on other.
Is there a way to force the first variant, regardless of the language settings? Can I, for example, set the keyboard to US inside the script? Or perhaps some alternative to $x that would convert commas to points?
(I am sending these as parameters to another code and it won't accept numbers with commas…)

The best thing I could think of is adding x=`echo $x | sed s/,/./` as a first line inside the loop, but there should be a better solution? (Interestingly, the sed command does not seem to be upset by me rewriting its variable.)

Thanks,
Michal

Peko July 22, 2009, 7:27 am

To Michal Kaut:

Hi Michal,

Such output format is configured through LOCALE settings.

I tried :

export LC_CTYPE="en_EN.UTF-8″; seq 0 0.1 1

and it works as desired.

You just have to find the exact value for LC_CTYPE that fits to your systems and your needs.

Peko

Peko July 22, 2009, 2:29 pm

To Michal Kaus [2]

Ooops – ;-)
Instead of LC_CTYPE,
LC_NUMERIC should be more appropriate
(Although LC_CTYPE is actually yielding to the same result – I tested both)

By the way, Vivek has already documented the matter : http://www.cyberciti.biz/tips/linux-find-supportable-character-sets.html

Philippe Petrinko October 30, 2009, 8:35 am

To Vivek:
Regarding your last example, that is : running a loop through arguments given to the script on the command line, there is a simplier way of doing this:
# instead of:
# FILES="$@"
# for f in $FILES

# use the following syntax
for arg
do
# whatever you need here – try : echo "$arg"
done

Of course, you can use any variable name, not only "arg".

Philippe Petrinko November 11, 2009, 11:25 am

To tdurden:

Why would'nt you use

1) either a [for] loop
for old in * ; do mv ${old} ${old}.new; done

2) Either the [rename] command ?
excerpt form "man rename" :

RENAME(1) Perl Programmers Reference Guide RENAME(1)

NAME
rename – renames multiple files

SYNOPSIS
rename [ -v ] [ -n ] [ -f ] perlexpr [ files ]

DESCRIPTION
"rename" renames the filenames supplied according to the rule specified
as the first argument. The perlexpr argument is a Perl expression
which is expected to modify the $_ string in Perl for at least some of
the filenames specified. If a given filename is not modified by the
expression, it will not be renamed. If no filenames are given on the
command line, filenames will be read via standard input.

For example, to rename all files matching "*.bak" to strip the
extension, you might say

rename 's/\.bak$//' *.bak

To translate uppercase names to lower, you'd use

rename 'y/A-Z/a-z/' *

- Philippe

Philippe Petrinko November 11, 2009, 9:27 pm

If you set the shell option extglob, Bash understands some more powerful patterns. Here, a is one or more pattern, separated by the pipe-symbol (|).

?() Matches zero or one occurrence of the given patterns
*() Matches zero or more occurrences of the given patterns
+() Matches one or more occurrences of the given patterns
@() Matches one of the given patterns
!() Matches anything except one of the given patterns

source: http://www.bash-hackers.org/wiki/doku.php/syntax/pattern

Philippe Petrinko November 12, 2009, 3:44 pm

To Sean:
Right, the more sharp a knife is, the easier it can cut your fingers…

I mean: There are side-effects to the use of file globbing (like in [ for f in * ] ) , when the globbing expression matches nothing: the globbing expression is not susbtitued.

Then you might want to consider using [ nullglob ] shell extension,
to prevent this.
see: http://www.bash-hackers.org/wiki/doku.php/syntax/expansion/globs#customization

Devil hides in detail ;-)

Dominic January 14, 2010, 10:04 am

There is an interesting difference between the exit value for two different for looping structures (hope this comes out right):
for (( c=1; c<=2; c++ )) do echo -n "inside (( )) loop c is $c, "; done; echo "done (( )) loop c is $c"
for c in {1..2}; do echo -n "inside { } loop c is $c, "; done; echo "done { } loop c is $c"

You see that the first structure does a final increment of c, the second does not. The first is more useful IMO because if you have a conditional break in the for loop, then you can subsequently test the value of $c to see if the for loop was broken or not; with the second structure you can't know whether the loop was broken on the last iteration or continued to completion.

Dominic January 14, 2010, 10:09 am

sorry, my previous post would have been clearer if I had shown the output of my code snippet, which is:
inside (( )) loop c is 1, inside (( )) loop c is 2, done (( )) loop c is 3
inside { } loop c is 1, inside { } loop c is 2, done { } loop c is 2

Philippe Petrinko March 9, 2010, 2:34 pm

@Dmitry

And, again, as stated many times up there, using [seq] is counter productive, because it requires a call to an external program, when you should Keep It Short and Simple, using only bash internals functions:


for ((c=1; c<21; c+=2)); do echo "Welcome $c times" ; done

(and I wonder why Vivek is sticking to that old solution which should be presented only for historical reasons when there was no way of using bash internals.
By the way, this historical recall should be placed only at topic end, and not on top of the topic, which makes newbies sticking to the not-up-to-date technique ;-) )

Sean March 9, 2010, 11:15 pm

I have a comment to add about using the builtin for (( … )) syntax. I would agree the builtin method is cleaner, but from what I've noticed with other builtin functionality, I had to check the speed advantage for myself. I wrote the following files:

builtin_count.sh:

#!/bin/bash
for ((i=1;i<=1000000;i++))
do
echo "Output $i"
done

seq_count.sh:

#!/bin/bash
for i in $(seq 1 1000000)
do
echo "Output $i"
done

And here were the results that I got:
time ./builtin_count.sh
real 0m22.122s
user 0m18.329s
sys 0m3.166s

time ./seq_count.sh
real 0m19.590s
user 0m15.326s
sys 0m2.503s

The performance increase isn't too significant, especially when you are probably going to be doing something a little more interesting inside of the for loop, but it does show that builtin commands are not necessarily faster.

Andi Reinbrech November 18, 2010, 8:35 pm

The reason why the external seq is faster, is because it is executed only once, and returns a huge splurb of space separated integers which need no further processing, apart from the for loop advancing to the next one for the variable substitution.

The internal loop is a nice and clean/readable construct, but it has a lot of overhead. The check expression is re-evaluated on every iteration, and a variable on the interpreter's heap gets incremented, possibly checked for overflow etc. etc.

Note that the check expression cannot be simplified or internally optimised by the interpreter because the value may change inside the loop's body (yes, there are cases where you'd want to do this, however rare and stupid they may seem), hence the variables are volatile and get re-evaluted.

I.e. botom line, the internal one has more overhead, the "seq" version is equivalent to either having 1000000 integers inside the script (hard coded), or reading once from a text file with 1000000 integers with a cat. Point being that it gets executed only once and becomes static.

OK, blah blah fishpaste, past my bed time :-)

Cheers,
Andi

Anthony Thyssen June 4, 2010, 6:53 am

The {1..10} syntax is pretty useful as you can use a variable with it!

limit=10
echo {1..${limit}}
{1..10}

You need to eval it to get it to work!

limit=10
eval "echo {1..${limit}}"
1 2 3 4 5 6 7 8 9 10

'seq' is not avilable on ALL system (MacOSX for example)
and BASH is not available on all systems either.

You are better off either using the old while-expr method for computer compatiblity!

   limit=10; n=1;
   while [ $n -le 10 ]; do
     echo $n;
     n=`expr $n + 1`;
   done

Alternativally use a seq() function replacement…

 # seq_count 10
seq_count() {
  i=1; while [ $i -le $1 ]; do echo $i; i=`expr $i + 1`; done
}
# simple_seq 1 2 10
simple_seq() {
  i=$1; while [ $i -le $3 ]; do echo $i; i=`expr $i + $2`; done
}
seq_integer() {
    if [ "X$1" = "X-f" ]
    then format="$2"; shift; shift
    else format="%d"
    fi
    case $# in
    1) i=1 inc=1 end=$1 ;;
    2) i=$1 inc=1 end=$2 ;;
    *) i=$1 inc=$2 end=$3 ;;
    esac
    while [ $i -le $end ]; do
      printf "$format\n" $i;
      i=`expr $i + $inc`;
    done
  }

Edited: by Admin – added code tags.

TheBonsai June 4, 2010, 9:57 am

The Bash C-style for loop was taken from KSH93, thus I guess it's at least portable towards Korn and Z.

The seq-function above could use i=$((i + inc)), if only POSIX matters. expr is obsolete for those things, even in POSIX.

Philippe Petrinko June 4, 2010, 10:15 am

Right Bonsai,
( http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_06_04 )

But FOR C-style does not seem to be POSIXLY-correct…

Read on-line reference issue 6/2004,
Top is here, http://www.opengroup.org/onlinepubs/009695399/mindex.html

and the Shell and Utilities volume (XCU) T.OC. is here
http://www.opengroup.org/onlinepubs/009695399/utilities/toc.html
doc is:
http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap01.html

and FOR command:
http://www.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_09_04_03

Anthony Thyssen June 6, 2010, 7:18 am

TheBonsai wrote…. "The seq-function above could use i=$((i + inc)), if only POSIX matters. expr is obsolete for those things, even in POSIX."

I am not certain it is in Posix. It was NOT part of the original Bourne Shell, and on some machines, I deal with Bourne Shell. Not Ksh, Bash, or anything else.

Bourne Shell syntax works everywhere! But as 'expr' is a builtin in more modern shells, then it is not a big loss or slow down.

This is especially important if writing a replacement command, such as for "seq" where you want your "just-paste-it-in" function to work as widely as possible.

I have been shell programming pretty well all the time since 1988, so I know what I am talking about! Believe me.

MacOSX has in this regard been the worse, and a very big backward step in UNIX compatibility. 2 year after it came out, its shell still did not even understand most of the normal 'test' functions. A major pain to write shells scripts that need to also work on this system.

TheBonsai June 6, 2010, 12:35 pm

Yea, the question was if it's POSIX, not if it's 100% portable (which is a difference). The POSIX base more or less is a subset of the Korn features (88, 93), pure Bourne is something "else", I know. Real portability, which means a program can go wherever UNIX went, only in C ;)

Philippe Petrinko November 22, 2010, 8:23 am

And if you want to get rid of double-quotes, use:

one-liner code:
while read; do record=${REPLY}; echo ${record}|while read -d ","; do field="${REPLY#\"}"; field="${field%\"}"; echo ${field}; done; done<data

script code, added of some text to better see record and field breakdown:

#!/bin/bash
while read
do
echo "New record"
record=${REPLY}
echo ${record}|while read -d ,
do
field="${REPLY#\"}"
field="${field%\"}"
echo "Field is :${field}:"
done
done<data

Does it work with your data?

- PP

Philippe Petrinko November 22, 2010, 9:01 am

Of course, all the above code was assuming that your CSV file is named "data".

If you want to use anyname with the script, replace:

done<data

With:

done

And then use your script file (named for instance "myScript") with standard input redirection:

myScript < anyFileNameYouWant

Enjoy!

Philippe Petrinko November 22, 2010, 11:28 am

well no there is a bug, last field of each record is not read – it needs a workout and may be IFS modification ! After all that's what it was built for… :O)

Anthony Thyssen November 22, 2010, 11:31 pm

Another bug is the inner loop is a pipeline, so you can't assign variables for use later in the script. but you can use '<<<' to break the pipeline and avoid the echo.

But this does not help when you have commas within the quotes! Which is why you needed quotes in the first place.

In any case It is a little off topic. Perhaps a new thread for reading CVS files in shell should be created.

Philippe Petrinko November 24, 2010, 6:29 pm

Anthony,
Would you try this one-liner script on your CSV file?

This one-liner assumes that CSV file named [data] has __every__ field double-quoted.


while read; do r="${REPLY#\"}";echo "${r//\",\"/\"}"|while read -d \";do echo "Field is :${REPLY}:";done;done<data

Here is the same code, but for a script file, not a one-liner tweak.


#!/bin/bash
# script csv01.sh
#
# 1) Usage
# This script reads from standard input
# any CSV with double-quoted data fields
# and breaks down each field on standard output
#
# 2) Within each record (line), _every_ field MUST:
# - Be surrounded by double quotes,
# - and be separated from preceeding field by a comma
# (not the first field of course, no comma before the first field)
#
while read
do
echo "New record" # this is not mandatory-just for explanation
#
#
# store REPLY and remove opening double quote
record="${REPLY#\"}"
#
#
# replace every "," by a single double quote
record=${record//\",\"/\"}
#
#
echo ${record}|while read -d \"
do
# store REPLY into variable "field"
field="${REPLY}"
#
#
echo "Field is :${field}:" # just for explanation
done
done

This script named here [cvs01.sh] must be used so:

cvs01.sh < my-cvs-file-with-doublequotes

Philippe Petrinko November 24, 2010, 6:35 pm

@Anthony,

By the way, using [REPLY] in the outer loop _and_ the inner loop is not a bug.
As long as you know what you do, this is not problem, you just have to store [REPLY] value conveniently, as this script shows.

TheBonsai March 8, 2011, 6:26 am
for ((i=1; i<=20; i++)); do printf "%02d\n" "$i"; done

nixCraft March 8, 2011, 6:37 am

+1 for printf due to portability, but you can use bashy .. syntax too

for i in {01..20}; do echo "$i"; done

TheBonsai March 8, 2011, 6:48 am

Well, it isn't portable per se, it makes it portable to pre-4 Bash versions.

I think a more or less "portable" (in terms of POSIX, at least) code would be

i=0
while [ "$((i >= 20))" -eq 0 ]; do
  printf "%02d\n" "$i"
  i=$((i+1))
done

Philip Ratzsch April 20, 2011, 5:53 am

I didn't see this in the article or any of the comments so I thought I'd share. While this is a contrived example, I find that nesting two groups can help squeeze a two-liner (once for each range) into a one-liner:

for num in {{1..10},{15..20}};do echo $num;done

Great reference article!

Philippe Petrinko April 20, 2011, 8:23 am

@Philip
Nice thing to think of, using brace nesting, thanks for sharing.

Philippe Petrinko May 6, 2011, 10:13 am

Hello Sanya,

That would be because brace expansion does not support variables. I have to check this.
Anyway, Keep It Short and Simple: (KISS) here is a simple solution I already gave above:

xstart=1;xend=10;xstep=1
for (( x = $xstart; x <= $xend; x += $xstep)); do echo $x;done

Actually, POSIX compliance allows to forget $ in for quotes, as said before, you could also write:

xstart=1;xend=10;xstep=1
for (( x = xstart; x <= xend; x += xstep)); do echo $x;done

Philippe Petrinko May 6, 2011, 10:48 am

Sanya,

Actually brace expansion happens __before__ $ parameter exapansion, so you cannot use it this way.

Nevertheless, you could overcome this this way:

max=10; for i in $(eval echo {1..$max}); do echo $i; done

Sanya May 6, 2011, 11:42 am

Hello, Philippe

Thanks for your suggestions
You basically confirmed my findings, that bash constructions are not as simple as zsh ones.
But since I don't care about POSIX compliance, and want to keep my scripts "readable" for less experienced people, I would prefer to stick to zsh where my simple for-loop works

Cheers, Sanya

Philippe Petrinko May 6, 2011, 12:07 pm

Sanya,

First, you got it wrong: solutions I gave are not related to POSIX, I just pointed out that POSIX allows not to use $ in for (( )), which is just a little bit more readable – sort of.

Second, why do you see this less readable than your [zsh] [for loop]?

for (( x = start; x <= end; x += step)) do
echo "Loop number ${x}"
done

It is clear that it is a loop, loop increments and limits are clear.

IMNSHO, if anyone cannot read this right, he should not be allowed to code. :-D

BFN

Anthony Thyssen May 8, 2011, 11:30 pm

If you are going to do… $(eval echo {1..$max});
You may as well use "seq" or one of the many other forms.
See all the other comments on doing for loops.

Tom P May 19, 2011, 12:16 pm

I am trying to use the variable I set in the for line on to set another variable with a different extension. Couldn't get this to work and couldnt find it anywhere on the web… Can someone help.

Example:

FILE_TOKEN=`cat /tmp/All_Tokens.txt`
for token in $FILE_TOKEN
do
A1_$token=`grep $A1_token /file/path/file.txt | cut -d ":" -f2`

my goal is to take the values from the ALL Tokens file and set a new variable with A1_ infront of it… This tells be that A1_ is not a command…

[Dec 05, 2015] How to forcefully unmount a Linux disk partition

January 27, 2006 | www.cyberciti.biz

... ... ...

Linux / UNIX will not allow you to unmount a device that is busy. There are many reasons for this (such as program accessing partition or open file) , but the most important one is to prevent the data loss. Try the following command to find out what processes have activities on the device/partition. If your device name is /dev/sdb1, enter the following command as root user:

# lsof | grep '/dev/sda1'
Output:
vi 4453       vivek    3u      BLK        8,1                 8167 /dev/sda1

Above output tells that user vivek has a vi process running that is using /dev/sda1. All you have to do is stop vi process and run umount again. As soon as that program terminates its task, the device will no longer be busy and you can unmount it with the following command:

# umount /dev/sda1
How do I list the users on the file-system /nas01/?

Type the following command:

# fuser -u /nas01/
# fuser -u /var/www/
Sample outputs:
/var/www:             3781rc(root)  3782rc(nginx)  3783rc(nginx)  3784rc(nginx)  3785rc(nginx)  3786rc(nginx)  3787rc(nginx)  3788rc(nginx)  3789rc(nginx)  3790rc(nginx)  3791rc(nginx)  3792rc(nginx)  3793rc(nginx)  3794rc(nginx)  3795rc(nginx)  3796rc(nginx)  3797rc(nginx)  3798rc(nginx)  3800rc(nginx)  3801rc(nginx)  3802rc(nginx)  3803rc(nginx)  3804rc(nginx)  3805rc(nginx)  3807rc(nginx)  3808rc(nginx)  3809rc(nginx)  3810rc(nginx)  3811rc(nginx)  3812rc(nginx)  3813rc(nginx)  3815rc(nginx)  3816rc(nginx)  3817rc(nginx)

The following discussion allows you to unmount device and partition forcefully using mount or fuser Linux commands.

Linux fuser command to forcefully unmount a disk partition

Suppose you have /dev/sda1 mounted on /mnt directory then you can use fuser command as follows:

WARNING! These examples may result into data loss if not executed properly (see "Understanding device error busy error" for more information).

Type the command to unmount /mnt forcefully:

# fuser -km /mnt
Where,

Linux umount command to unmount a disk partition.

You can also try the umount command with –l option on a Linux based system:

# umount -l /mnt
Where,

If you would like to unmount a NFS mount point then try following command:

# umount -f /mnt
Where,

Please note that using these commands or options can cause data loss for open files; programs which access files after the file system has been unmounted will get an error.

See also:

[Nov 23, 2015] Game of musical chairs became more difficult: the US economy seems to put out fewer and fewer chairs.

Notable quotes:
"... A good economy compensates for much social dysfunction. ..."
"... More than that, it prevents the worst of behaviors that are considered an expression of dysfunction from occurring, as people across all social strata have other things to worry about or keep them busy. Happy people dont bear grudges, or at least they are not on top of their consciousness as long as things are going well. ..."
"... This could be seen time and again in societies with deep and sometimes violent divisions between ethnic groups where in times of relative prosperity (or at least a broadly shared vision for a better future) the conflicts are not removed but put on a backburner, or there is even finally reconciliation, and then when the economy turns south, the old grudges and conflicts come back (often not on their own, but fanned by groups who stand to gain from the divisions, or as a way of scapegoating) ..."
"... backwaters of America, that economy seems to put out fewer and fewer chairs. ~~Harold Pollack~ ..."
"... Going up through the chairs has become so impossible for those on the slow-track. Not enough slots for all the jokers within our once proud country of opportunities, ..."
"... George Orwell: I doubt, however, whether the unemployed would ultimately benefit if they learned to spend their money more economically. ... If the unemployed learned to be better managers they would be visibly better off, and I fancy it would not be long before the dole was docked correspondingly. ..."
"... Perhaps you are commenting on the aspect that when (enough) job applicants/holders define down their standards and let employers treat them as floor mats, then the quality of many jobs and the labor relations will be adjusted down accordingly, or at the very least expectations what concessions workers will make will be adjusted up. That seems to be the case unfortunately. ..."
economistsview.typepad.com
Avraam Jack Dectis said...
A good economy compensates for much social dysfunction.

A bad economy moves people toward the margins, afflicts those near the margins and kills those at the margins.

This is what policy makers should consider as they pursue policies that do not put the citizen above all else.

cm -> Avraam Jack Dectis...
"A good economy compensates for much social dysfunction."

More than that, it prevents the worst of behaviors that are considered an expression of dysfunction from occurring, as people across all social strata have other things to worry about or keep them busy. Happy people don't bear grudges, or at least they are not on top of their consciousness as long as things are going well.

This could be seen time and again in societies with deep and sometimes violent divisions between ethnic groups where in times of relative prosperity (or at least a broadly shared vision for a better future) the conflicts are not removed but put on a backburner, or there is even "finally" reconciliation, and then when the economy turns south, the old grudges and conflicts come back (often not on their own, but fanned by groups who stand to gain from the divisions, or as a way of scapegoating)

Dune Goon said...

"backwaters of America, that economy seems to put out fewer and fewer chairs." ~~Harold Pollack~

Going up through the chairs has become so impossible for those on the slow-track. Not enough slots for all the jokers within our once proud country of opportunities, not enough elbow room for Daniel Boone, let alone Jack Daniels! Not enough space in this county to wet a tree when you feel the urge! Every tiny plot of space has been nailed down and fenced off, divided up among gated communities. Why?

Because the 1% has an excessive propensity to reproduce their own kind. They are so uneducated about the responsibilities of birth control and space conservation that they are crowding all of us off the edge of the planet. Worse yet we have begun to *ape our betters*.

"We've only just begun!"
~~The Carpenters~

William said...

"Many of us know people who receive various public benefits, and who might not need to rely on these programs if they made better choices, if they learned how to not talk back at work, if they had a better handle on various self-destructive behaviors, if they were more willing to take that crappy job and forego disability benefits, etc."

George Orwell: "I doubt, however, whether the unemployed would ultimately benefit if they learned to spend their money more economically. ... If the unemployed learned to be better managers they would be visibly better off, and I fancy it would not be long before the dole was docked correspondingly."

cm said in reply to William...

A valid observation, but what you are commenting on is more about getting or keeping a job than managing personal finances.

Perhaps you are commenting on the aspect that when (enough) job applicants/holders define down their standards and let employers treat them as floor mats, then the quality of many jobs and the labor relations will be adjusted down accordingly, or at the very least expectations what concessions workers will make will be adjusted up. That seems to be the case unfortunately.

[Nov 14, 2015] Students across U.S. march over debt, free public college

Neoliberal college is not about education. It is about getting wealthy a head start to enforce and strengthen separation between the elite and the rest. Other can only complain... But that e fact is the many large companies invite for interview for open positions only of Ivy Leagues graduates. Other do not need to apply. So it is mostly about "Class A" and "Class B" citizens. Talent and hard work can buy buy a ticket for vertical mobility (see some stories below), but that was true in any society. Actually mobility in the USA is below average, despite MSM non-stop brainwashing of the USA citizens about "the land of opportunities", the "American Dream", etc. And exorbitant salaries of University brass is a norm now. you can't change that without changing the neoliberal system as a whole. they are no longer bound by academic ethics. Like Wall Streeters, want to get the most of the life, no matter by what means (the end justifies the means mentality). They are masters of the universe. Others (aka suckers) can go to hell.
Notable quotes:
"... Dealing with swiftly mounting student loan debt has been a focus of candidates vying for the White House in 2016. Democratic hopeful Bernie Sanders has vowed to make tuition free at public universities and colleges, and has pledged to cut interest rates for student loans. ..."
"... I can see having a low, federally-subsidized interest rate on these loans....which I seem to recall having on some of my loans, but anyone wanting anything for free can take a hike, IMHO. ..."
"... Ever visit a university in a country that has free college education for its' citizens? It's pretty austere. These kids need to think past the clever sound bytes and really consider the effect of what they are asking for. ..."
"... What really needs to be addressed is the skyrocketing cost of college education PERIOD! At the rate it's going up pretty soon only the children of billionaires will be able to afford to go to college. ..."
"... College tuition cannot be allowed to just continue to escalate. ..."
"... If a high school grad can't explain in detail how much cash is needed, and how spending all that cash and time for education is going to provide a positive return on investment, he or she should not be going to college. This should be near the top of things that teens learn in high school. ..."
"... I get really cynical about all graduates claiming they had no idea how much their loans were going to cost them. ..."
"... If you didn't bother to read your loan docs before signing, or research likely monthly payments for your loan, that's your fault! ..."
"... College costs went up far faster than inflation, often because colleges built fancy sports and living facilities...because they figured out these same millennials pick colleges based on those things. ..."
"... The standard tours take students through fancy facilities that have nothing to do with quality of education. Add declining teaching loads that have decreased from 12 class hours to 3 class hours per week for a professor in the past 25 years and the rise in overhead for non-academic administration overhead positions like chief diversity and inclusion officer and you have expensive college. ..."
"... These 'loans' are now almost all, Pell Grant underwritten. Cannot Bankrupt on, co-signers and students can lose their Social Security money if defaulting. 1.5 trillion$ of these loans have been packaged, like Home Loans, derivative. What happens to peoples retirement accounts when their Funds have investments in them, what happens to the Primary Dealers when the derivatives bubble bursts? ..."
"... Where it is free, but only to the select, the performers, most American Students would not qualify in other countries for advanced Ed. ..."
Nov 14, 2015 | news.yahoo.com

Students held rallies on college campuses across the United States on Thursday to protest ballooning student loan debt for higher education and rally for tuition-free public colleges and a minimum wage hike for campus workers.

The demonstrations, dubbed the Million Student March, were planned just two days after thousands of fast-food workers took to the streets in a nationwide day of action pushing for a $15-an-hour minimum wage and union rights for the industry.

About 50 students from Boston-area colleges gathered at Northeastern University carrying signs that read "Degrees not receipts" and "Is this a school or a corporation?"

"The student debt crisis is awful. Change starts when people demand it in the street. Not in the White House," said Elan Axelbank, 20, a third year student at Northeastern, who said he was a co-founder of the national action.

... ... ...

"I want to graduate without debt," said Ashley Allison, a 22-year-old student at Boston's Bunker Hill Community College, at the Northeastern rally. "Community college has been kind to me, but if I want to go on, I have to take on debt."

Dealing with swiftly mounting student loan debt has been a focus of candidates vying for the White House in 2016. Democratic hopeful Bernie Sanders has vowed to make tuition free at public universities and colleges, and has pledged to cut interest rates for student loans.

... ... ...

Andrew Jackson

Free taxpayer supported public education means more college administrators earning $200,000 or more, more faculty earning $100,000 or more working 8 months a year and more $300 textbooks. Higher education costs are a direct correlation to Federal Student Loans subsiding college bureaucracies, exorbitant salaries for college administrators and faculty.

terrance

What fantasy world do these people live in. There is nothing for free and if you borrow tens of thousands of dollars you can't expect later that someone else will pick up your tab. Pucker up bucky, it is your responsibility.

Furthermore, a lot of this money didn't go to education. I have read where people went back to school so they could borrow money to pay their rent, or even their car payments. As for 15 dollars an hour to sling burgers, grow up.

sjc

Having been out of college for a few years, I am curious. I went to a State University. Tuition was high, I had to take loans, I drove a cheap 10 yr old vehicle, but it didn't kill me. My total debt was about the price of a decent new car back then.

Today, the average student loan debt after graduation is just under $30,000. Around the price of a new car. And these kids are trying to tell us that this is too much of a burden??? Look around any campus these days, and you will see lots of $30,000 cars in the parking lots.

I can see having a low, federally-subsidized interest rate on these loans....which I seem to recall having on some of my loans, but anyone wanting anything for free can take a hike, IMHO.

Meed

Careful what you wish for, kiddies. It's simple math and simple economics (things I learned in school while studying instead of protesting). Every university has a maximum number of students it can support, based on the number and capacity of dorms, classrooms, faculty, etc.

The tuition rates have always closely matched the amount of easily-accessed loans available - the easier the access to loans, the higher tuition is. The simple reason is that the universities raise tuition rates to manage the demand for their limited resources, and can always raise rates when there is more demand than there are openings for incoming students.

Thanks to the windfall from that high tuition, today's universities have student unions, recreation facilities, gyms, pools, and lots of amenities to attract students. Imagine what they will offer when they can't jack up the tuition. Ever visit a university in a country that has "free" college education for its' citizens? It's pretty austere. These kids need to think past the clever sound bytes and really consider the effect of what they are asking for.

matthew

Oddly enough, a majority of these students attend colleges who has sport teams sponsored by Nike, Under Armour, Adidas, or Reebok. So, should theses companies atop providing the uniforms and equipment free of charge and donate the money to make more scholarships available? Then the student athletes can purchase their own gear on their own dime. Where one group attains, another must lose. Let this be debated on college campuses and watch the students divide themselves. We will find out what is most important to them.

JB

What really needs to be addressed is the skyrocketing cost of college education PERIOD! At the rate it's going up pretty soon only the children of billionaires will be able to afford to go to college.

Some junk yard dog investigative journalist needs to dig into the rising cost of college education and identify the cause. Once the cause are understood then something can be done to make college more affordable. College tuition cannot be allowed to just continue to escalate.

just sayin'

Seriously how do we let our children out of high school without enough information to decide if going to college is actually a good investment? If a high school grad can't explain in detail how much cash is needed, and how spending all that cash and time for education is going to provide a positive return on investment, he or she should not be going to college. This should be near the top of things that teens learn in high school.

pcs

I get really cynical about all graduates claiming they had no idea how much their loans were going to cost them. I mean, they had enough math skills to be accepted, then graduate, from college. If you didn't bother to read your loan docs before signing, or research likely monthly payments for your loan, that's your fault!

E

College costs went up far faster than inflation, often because colleges built fancy sports and living facilities...because they figured out these same millennials pick colleges based on those things. If you tour colleges, and I toured many in the past few years with my kids, you don't see a classroom or lab unless you ask.

The standard tours take students through fancy facilities that have nothing to do with quality of education. Add declining teaching loads that have decreased from 12 class hours to 3 class hours per week for a professor in the past 25 years and the rise in overhead for non-academic administration overhead positions like "chief diversity and inclusion officer" and you have expensive college.

If students want a cheap education, go to the junior college for general ed classes then transfer to a four-year school. It is not glamorous but it yields a quality education without a fortune in debt.

Rich

Getting an education is obviously the biggest scam in history!!!! Look at who controls education. Look at all the Universities presidents last names then you will know what they are. I can't say it here on Yahoo because they will take my comments out for speaking the truth. These presidents make millions of $$$$$ a year off of students and parents who are slaves and work hard to pay those tuitions. Not only that but look at the owners last names of the Loan

50 CAL

Universities are money munching machines with no regard for how the students will repay the loans. Universities annually raise tuition rates(much of which is unnecessary) with no regard of how these young minds full of mush are going to repay the crushing debt, nor do they care. Locally one university just opened a 15 million dollar athletic center, which brings up the question, why did they need this? With that kind of cash to throw around, what wasn't at least some used to keep tuitions affordable?

Mike D

These 'loans' are now almost all, Pell Grant underwritten. Cannot Bankrupt on, co-signers and students can lose their Social Security money if defaulting. 1.5 trillion$ of these loans have been packaged, like Home Loans, derivative.

What happens to peoples retirement accounts when their Funds have investments in them, what happens to the Primary Dealers when the derivatives bubble bursts?

How are these loans to be made 'free' if existing loans bear interest? If the student of 'free education' defaults, doesn't graduate, will he owe money-will his parent, or will the 'free school' simply become a dumping ground for the youth without direction, simply housed in college's dorm rooms?
Lots of questions and two things to keep in mind, the Banks and Teaching institutes love the idea of 'free', the students are believing there might be a free ride.. ignoring schools and Banks don't, won't and never do anything for free.

This is not going to turn out well for consumers. Sure, Household payments of Education may drop, but the Institution of Education cannot keep even its slim success rate it has now. I don't know how educators managed to turn education into a purely self gratifying industry, giving anything to purchasers they wished for that Education loan, but never ever ever, has underwriting by the Central improved the quality of business. Complete underwriting of the important system of education at the Fed level will be a disaster.

There will be almost zero accountability for institutes and students, we will have a more expensive system that turns out the worst grads.

Don't try believing that other countries abilities with free Ed can be duplicated here.. not without serious socialism, a condition where qualifying for Ed advancement is determined by the Central.

Where it is free, but only to the select, the performers, most American Students would not qualify in other countries for advanced Ed. Blanket quals are almost a condition here, American Students are in for a serious surprise. They will not be so able to buy/loan their way to college and have to excel to get into college.

The joke is on the American student.

Jim

i was one of seven children- i worked my way through four years of undergrad and three years of grad school with my parents only being able to pay health insurance and car insurance- i worked shelving books, busing tables, delivering pizzas and for the last five years as a parimutuel clerk at dog and horsetracks- i never got to go on spring break, do a semester at sea or take classes in europe- i graduated debt free from public universities- have no sympathy for a bunch of whiny brats who have to drive better cars than their professors and believe they are entitled to special treatment- get a job and quit acting like a bunch of welfare queens who feel they deserve entitlements

Linda

My son is in college. Because grandpa saved his money over the years, he volunteered to pay for college costs. We hope to continue the tradition with our grandchildren and carefully save our money as well. We don't live high or purchase new. He will graduate zero dollars in debt.

My son's college roommate comes from a very wealthy family. They own a plane - two houses - dad works on Wall Street - mom is a Doctor. He has to pay for his own education and gets loans for everything. His parents simply don't have the cash to pay for his education.

It's priorities people! If something is worth it, you'll make it happen.

[Nov 11, 2015] Paul Krugman: Despair, American Style

Notable quotes:
"... In a recent interview Mr. Deaton suggested that middle-aged whites have "lost the narrative of their lives." That is, their economic setbacks have hit hard because they expected better. Or to put it a bit differently, we're looking at people who were raised to believe in the American Dream, and are coping badly with its failure to come true. ..."
"... the truth is that we don't really know why despair appears to be spreading across Middle America. But it clearly is, with troubling consequences for our society... ..."
"... Some people who feel left behind by the American story turn self-destructive; others turn on the elites they feel have betrayed them. ..."
"... What we are seeing is the long term impacts of the Reagan Revolution. ..."
"... The affected cohort here is the first which has lived with the increased financial and employment insecurity that engendered, as well as the impacts of the massive offshoring of good paying union jobs throughout their working lives. Stress has cumulative impacts on health and well-being, which are a big part of what we are seeing here. ..."
"... Lets face it, this Fed is all about goosing up asset prices to generate short term gains in economic activity. Since the early 90s, the Fed has done nothing but make policy based on Wall Streets interests. I can give them a pass on the dot com debacle but not after that. This toxic relationship between wall street and the Fed has to end. ..."
"... You are the guys who do not consider the counterfactual where higher rates would have prevented the housing bubble in 2003-05 and that produced the great recession in the first place. ..."
"... Inequality has been rising globally, almost regardless of trade practices ..."
"... It is not some unstoppable global trend. This is neoliberal oligarchy coup détat. Or as it often called a quite coup . ..."
"... First of all, whether a job can or is offshored has little to do with whether it is low skilled but more with whether the workflow around the job can be organized in such a way that the job can be offshore. This is less a matter of skill level and more volume and immediacy of interaction with adjacent job functions, or movement of material across distances. ..."
"... The reason wages are stuck is that aggregate jobs are not growing, relative to workforce supply. ..."
"... BTW the primary offshore location is India, probably in good part because of good to excellent English language skills, and Indias investment in STEM education and industry (especially software/services and this is even a public stereotype, but for a reason). ..."
"... Very rough figures: half a million Chicago employees may make less than $800 a week -- almost everybody should earn $800 ... ..."
"... Union busting is generally (?) understood as direct interference with the formation and operation of unions or their members. It is probably more common that employers are allowed to just go around the unions - right to work , subcontracting non-union shops or temp/staffing agencies, etc. ..."
"... Why would people join a union and pay dues when the union is largely impotent to deliver, when there are always still enough desperate people who will (have to) take jobs outside the union system? Employers dont have to bring in scabs when they can legally go through unencumbered subcontractors inside or outside the jurisdiction. ..."
"... The anti-knowledge of the elites is worth reading. http://billmoyers.com/2015/11/02/the-anti-knowledge-of-the-elites/ When such herd instinct and institutional overbearance connects with the credibility trap, the results may be impressive. http://jessescrossroadscafe.blogspot.com/2015/11/gold-daily-and-silver-weekly-charts-pop.html ..."
"... Suicide, once thought to be associated with troubled teens and the elderly, is quickly becoming an age-blind statistic. Middle aged Americans are turning to suicide in alarming numbers. The reasons include easily accessible prescription painkillers, the mortgage crisis and most importantly the challenge of a troubled economy. The Center for Disease Control and Prevention claims suicide rates now top the number of deaths due to automobile accidents. ..."
"... The suicide rate for both younger and older Americans remains virtually unchanged, however, the rate has spiked for those in middle age (35 to 64 years old) with a 28 percent increase (link is external) from 1999 to 2010. ..."
"... When few people kill themselves on purpose or die from self-inflicted but probably unintended harms (e.g. organ failure or accidental death caused by substance abuse), it can be shrugged off as problems related to the individual (more elaboration possible but not necessary). ..."
"... When it becomes a statistically significant phenomenon (above-noise percentage of total population or demographically identifiable groups), then one has to ask questions about social causes. My first question would be, what made life suck for those people ? What specific instrument they used to kill themselves would be my second question (it may be the first question for people who are charged with implementing counter measures but not necessarily fixing the causes). ..."
"... Since about the financial crisis (Im not sure about causation or coincidence - not accidental coincidence BTW but causation by the same underlying causes), there has been a disturbing pattern of high school students throwing themselves in front of local trains. At that age, drinking or drugging oneself to death is apparently not the first choice . Performance pressure *related to* (not just and ) a lack of convincing career/life prospects has/have been suspected or named as a cause. I dont think teenagers suddenly started to jump in front of trains that have run the same rail line for decades because of the usual and centuries to millennia old teenage romantic relationship issues. ..."
November 9, 2015 | economistsview.typepad.com

"There is a darkness spreading over part of our society":

Despair, American Style, by Paul Krugman, Commentary, NY Times: A couple of weeks ago President Obama mocked Republicans who are "down on America," and reinforced his message by doing a pretty good Grumpy Cat impression. He had a point: With job growth at rates not seen since the 1990s, with the percentage of Americans covered by health insurance hitting record highs, the doom-and-gloom predictions of his political enemies look ever more at odds with reality.

Yet there is a darkness spreading over part of our society. ... There has been a lot of comment ... over a new paper by the economists Angus Deaton (who just won a Nobel) and Anne Case, showing that mortality among middle-aged white Americans has been rising since 1999..., while death rates were falling steadily both in other countries and among other groups in our own nation.

Even more striking are the proximate causes of rising mortality. Basically, white Americans are, in increasing numbers, killing themselves... Suicide is way up, and so are deaths from drug poisoning and ... drinking... But what's causing this epidemic of self-destructive behavior?...

In a recent interview Mr. Deaton suggested that middle-aged whites have "lost the narrative of their lives." That is, their economic setbacks have hit hard because they expected better. Or to put it a bit differently, we're looking at people who were raised to believe in the American Dream, and are coping badly with its failure to come true.

That sounds like a plausible hypothesis..., but the truth is that we don't really know why despair appears to be spreading across Middle America. But it clearly is, with troubling consequences for our society...

I know I'm not the only observer who sees a link between the despair reflected in those mortality numbers and the volatility of right-wing politics. Some people who feel left behind by the American story turn self-destructive; others turn on the elites they feel have betrayed them. No, deporting immigrants and wearing baseball caps bearing slogans won't solve their problems, but neither will cutting taxes on capital gains. So you can understand why some voters have rallied around politicians who at least seem to feel their pain.

At this point you probably expect me to offer a solution. But while universal health care, higher minimum wages, aid to education, and so on would do a lot to help Americans in trouble, I'm not sure whether they're enough to cure existential despair.

bakho said...

There are a lot of economic dislocations that the government after the 2001 recession stopped doing much about it. Right after the 2008 crash, the government did more but by 2010, even the Democratic president dropped the ball. and failed to deliver. Probably no region of the country is affected more by technological change that the coal regions of KY and WV. Lying politicians promise a return to the past that cannot be delivered. No one can suggest what the new future will be. The US is due for another round of urbanization as jobs decline in rural areas. Dislocation forces declining values of properties and requires changes in behavior, skills and outlook. Those personal changes do not happen without guidance. The social institutions such as churches and government programs are a backstop, but they are not providing a way forward. There is plenty of work to be done, but our elites are not willing to invest.

DrDick -> bakho...

The problem goes back much further than that. What we are seeing is the long term impacts of the "Reagan Revolution."

The affected cohort here is the first which has lived with the increased financial and employment insecurity that engendered, as well as the impacts of the massive offshoring of good paying union jobs throughout their working lives. Stress has cumulative impacts on health and well-being, which are a big part of what we are seeing here.

ilsm said...

Thuggee doom and gloom is about their fading chance to reinstate the slavocracy.

The fever swamp of right wing ideas is more loony than 1964.

Extremism is the new normal.

bmorejoe -> ilsm...

Yup. The slow death of white supremacy.

Peter K. -> Anonymous...

If it wasn't for monetary policy things would be even worse as the Republicans in Congress forced fiscal austerity on the economy during the "recovery."

sanjait -> Peter K....

That's the painful irony of a comment like that one from Anonymous ... he seems completely unaware that, yes, ZIRP has done a huge amount to prevent the kind of problems described above. He like most ZIRP critics fails to consider what the counterfactual looks like (i.e., something like the Great Depression redux).

Anonymous -> sanjait...

You are the guys who do not consider the counterfactual where higher rates would have prevented the housing bubble in 2003-05 and that produced the great recession in the first place. Because preemptive monetary policy has gone out of fashion completely. And now we are going to repeat the whole process over when the present bubble in stocks and corporate bonds bursts along with the malinvestment in China, commodity exporters etc.

Peter K. -> Anonymous...

"liquidate labor, liquidate stocks, liquidate farmers, liquidate real estate... it will purge the rottenness out of the system. High costs of living and high living will come down. People will work harder, live a more moral life. Values will be adjusted, and enterprising people will pick up from less competent people."

sanjait -> Anonymous...

"You want regulation? I would like to see

1) Reinstate Glass Steagall

2) impose a 10bp trans tax on trading financial instruments."

Great. Two things with zero chance of averting bubbles but make great populist pablum.

This is why we can't have nice things!

"3) Outlaw any Fed person working for a bank/financial firm after they leave office."

This seems like a decent idea. Hard to enforce, as highly intelligent and accomplished people tend not to be accepting of such restrictions, but it could be worth it anyway.

likbez -> sanjait...

" highly intelligent and accomplished people tend not to be accepting of such restrictions, but it could be worth it anyway."

You are forgetting that it depends on a simple fact to whom political power belongs. And that's the key whether "highly intelligent and accomplished people" will accept those restrictions of not.

If the government was not fully captured by financial capital, then I think even limited prosecution of banksters "Stalin's purge style" would do wonders in preventing housing bubble and 2008 financial crush.

Please try to imagine the effect of trial and exile to Alaska for some period just a dozen people involved in Securitization of mortgages boom (and those highly intelligent people can do wonders in improving oil industry in Alaska ;-).

Starting with Mr. Weill, Mr. Greenspan, Mr. Rubin, Mr. Phil Gramm, Dr. Summers and Mr. Clinton.

Anonymous -> Peter K....

"2003-2005 didn't have excess inflation and wage gains."

Monetary policy can not hinge just on inflation or wage gains. Why are wage gains a problem anyway?

Lets face it, this Fed is all about goosing up asset prices to generate short term gains in economic activity. Since the early 90s, the Fed has done nothing but make policy based on Wall Street's interests. I can give them a pass on the dot com debacle but not after that. This toxic relationship between wall street and the Fed has to end.

You want regulation? I would like to see
1) Reinstate Glass Steagall
2) impose a 10bp trans tax on trading financial instruments.
3) Outlaw any Fed person working for a bank/financial firm after they leave office. Bernanke, David Warsh etc included. That includes Mishkin getting paid to shill for failing Iceland banks or Bernanke making paid speeches to hedge funds.


Anonymous -> EMichael...

Fact: there was a housing bubble that most at the Fed (including Bernanke) denied right upto the middle of 2007
Fact: Yellen, to her credit, has admitted multiple times over the years that low rates spur search for yield that blows bubbles
Fact: Bursting of the bubble led to unemployment for millions and U3 that went to 10%

what facts are you referring to?

EMichael -> Anonymous...

That FED rates caused the bubble.

to think this you have to ignore that a 400% Fed Rate increase from 2004 to 2005 had absolutely no effect on mortgage originations.

Then of course, you have to explain why 7 years at zero has not caused another housing bubble.

https://research.stlouisfed.org/fred2/series/FEDFUNDS

Correlation is not causation. Lack of correlation is proof of lack of causation.

pgl -> Anonymous...

"You are the guys who do not consider the counterfactual where higher rates would have prevented the housing bubble in 2003-05 and that produced the great recession in the first place."

You are repeating the John B. Taylor line about interest rates being held "too low and too long". And guess what - most economists have called Taylor's claim for the BS it really is. We should also note we never heard this BS when Taylor was part of the Bush Administration. And do check - Greenspan and later Bernanke were raising interest rates well before any excess demand was generated which is why inflation never took off.

So do keep repeating this intellectual garbage and we keep noting you are just a stupid troll.

anne -> anne...
http://www.pnas.org/content/early/2015/10/29/1518393112

September 17, 2015

Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century
By Anne Case and Angus Deaton

Midlife increases in suicides and drug poisonings have been previously noted. However, that these upward trends were persistent and large enough to drive up all-cause midlife mortality has, to our knowledge, been overlooked. If the white mortality rate for ages 45−54 had held at their 1998 value, 96,000 deaths would have been avoided from 1999–2013, 7,000 in 2013 alone. If it had continued to decline at its previous (1979‒1998) rate, half a million deaths would have been avoided in the period 1999‒2013, comparable to lives lost in the US AIDS epidemic through mid-2015. Concurrent declines in self-reported health, mental health, and ability to work, increased reports of pain, and deteriorating measures of liver function all point to increasing midlife distress.

Abstract

This paper documents a marked increase in the all-cause mortality of middle-aged white non-Hispanic men and women in the United States between 1999 and 2013. This change reversed decades of progress in mortality and was unique to the United States; no other rich country saw a similar turnaround. The midlife mortality reversal was confined to white non-Hispanics; black non-Hispanics and Hispanics at midlife, and those aged 65 and above in every racial and ethnic group, continued to see mortality rates fall. This increase for whites was largely accounted for by increasing death rates from drug and alcohol poisonings, suicide, and chronic liver diseases and cirrhosis. Although all education groups saw increases in mortality from suicide and poisonings, and an overall increase in external cause mortality, those with less education saw the most marked increases. Rising midlife mortality rates of white non-Hispanics were paralleled by increases in midlife morbidity. Self-reported declines in health, mental health, and ability to conduct activities of daily living, and increases in chronic pain and inability to work, as well as clinically measured deteriorations in liver function, all point to growing distress in this population. We comment on potential economic causes and consequences of this deterioration.

ilsm -> Sarah...

Murka is different. Noni's plan would work if it were opportune for the slavocracy and the Kochs and ARAMCO don't lose any "growth".

Maybe cost plus climate repair contracts to shipyards fumbling through useless nuclear powered behemoths for war plans made in 1942.

Someone gotta make big money plundering for the public good, in Murka!

CSP said...

The answers to our malaise seem readily apparent to me, and I'm a southern-born white male working in a small, struggling Georgia town.

1. Kill the national war machine
2. Kill the national Wall Street financial fraud machine
3. Get out-of-control mega corporations under control
4. Return savings to Main Street (see #1, #2 and #3)
5. Provide national, universal health insurance to everyone as a right
6. Provide free education to everyone, as much as their academic abilities can earn them
7. Strengthen social security and lower the retirement age to clear the current chronic underemployment of young people

It seems to me that these seven steps would free the American people to pursue their dreams, not the dreams of Washington or Wall Street. Unfortunately, it is readily apparent that true freedom and real individual empowerment are the last things our leaders desire. Shame on them and shame on everyone who helps to make it so.

DeDude -> CSP...

You are right. Problem is that most southern-born white males working in a small, struggling Georgia town would rather die than voting for the one candidate who might institute those changes - Bernie Sanders.

The people who are beginning to realize that the american dream is a mirage, are the same people who vote for GOP candidates who want to give even more to the plutocrats.

kthomas said...

The kids in Seattle had it right when WTO showed up.


Why is anyone suprised by all this?

We exported out jobs. First all the manufacturing. Now all of the Service jobs.


But hey...we helped millions in China and India get out of poverty, only to put outselves into it.


America was sold to highest bidder a long long time ago. A Ken Melvin put it, the chickens came home to roost in 2000.

sanjait -> kthomas...

So you think the problem with America is that we lost our low skilled manufacturing and call center tech support jobs?

I can sort of see why people assume that "we exported out jobs" is the reason for stagnant incomes in the U.S., but it's still tiresome, because it's still just wrong.

Manufacturing employment crashed in the US mostly because it has been declining globally. The world economy is less material based than ever, and machines do more of the work making stuff.

And while some services can be outsourced, the vast majority can't. Period.

Inequality has been rising globally, almost regardless of trade practices. The U.S. has one of the more closed economies in the developed world, so if globalization were the cause, we'd be the most insulated. But we aren't, which should be a pretty good indication that globalization isn't the cause.

cm -> sanjait...

Yes, the loss of "low skilled" jobs is still a loss of jobs. Many people work in "low skilled" jobs because there are not enough "higher skill" jobs to go around, as most work demanded is not of the most fancy type.

We have heard this now for a few decades, that "low skilled" jobs lost will be replaced with "high skill" (and better paid) jobs, and the evidence is somewhat lacking. There has been grow