Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Systemd bulletin, 2019

Home 2020 2019 2018 2017 2016 2015

For the list of top articles see Recommended Links section


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Jan 26, 2019] Systemd developers don't want to replace the kernel, they are more than happy to leverage Linus's good work on what they see as a collection of device driver

Jan 26, 2019 | blog.erratasec.com

John Morris said...

They don't want to replace the kernel, they are more than happy to leverage Linus's good work on what they see as a collection of device drivers. No, they want to replace the GNU/X in the traditional Linux/GNU/X arrangement. All of the command line tools, up to and including bash are to go, replaced with the more Windows like tools most of the systemd developers grew up on, while X and the desktop environments all get rubbished for Wayland and GNOME3.

And I would wish them luck, the world could use more diversity in operating systems. So long as they stayed the hell over at RedHat and did their grand experiment and I could still find a Linux/GNU/X distribution to run. But they had to be borg and insist that all must bend the knee and to that I say HELL NO!

[Jan 26, 2019] The coming enhancement to systemd

Jan 26, 2019 | blog.erratasec.com

Siegfried Kiermayer said...

I'm waiting for pulse audio being included in systemd to have proper a boot sound :D

[Jan 26, 2019] SysVinit to Systemd Cheatsheet

Apr 15, 2015 | FedoraProject
Sysvinit Command Systemd Command Notes
service frobozz start systemctl start frobozz Used to start a service (not reboot persistent)
service frobozz stop systemctl stop frobozz Used to stop a service (not reboot persistent)
service frobozz restart systemctl restart frobozz Used to stop and then start a service
service frobozz reload systemctl reload frobozz When supported, reloads the config file without interrupting pending operations.
service frobozz condrestart systemctl condrestart frobozz Restarts if the service is already running.
service frobozz status systemctl status frobozz Tells whether a service is currently running.
ls /etc/rc.d/init.d/ systemctl (or) systemctl list-unit-files --type=service (or)
ls /lib/systemd/system/*.service /etc/systemd/system/*.service
Used to list the services that can be started or stopped
Used to list all the services and other units
chkconfig frobozz on systemctl enable frobozz Turn the service on, for start at next boot, or other trigger.
chkconfig frobozz off systemctl disable frobozz Turn the service off for the next reboot, or any other trigger.
chkconfig frobozz systemctl is-enabled frobozz Used to check whether a service is configured to start or not in the current environment.
chkconfig --list systemctl list-unit-files --type=service (or) ls /etc/systemd/system/*.wants/ Print a table of services that lists which runlevels each is configured on or off
chkconfig frobozz --list ls /etc/systemd/system/*.wants/frobozz.service Used to list what levels this service is configured on or off
chkconfig frobozz --add systemctl daemon-reload Used when you create a new service file or modify any configuration

[Jan 26, 2019] Errata Security About the systemd controversy...

Aug 30, 2015 | www.agwa.name

This is the core system within systemd that allows different bits of userspace to talk to each other. But it's got problems. A demonstration of the D-Bus problem is the recent Jeep hack by researchers Charlie Miller and Chris Valasek. The root problem was that D-Bus was openly (without authentication) accessible from the Internet.

Likewise, the "AllJoyn" system for the "Internet of Things" opens up D-Bus on the home network. D-Bus indeed simplifies communication within userspace, but its philosophy is to put all your eggs in one basket, then drop the basket.

[Jan 26, 2019] Systemd is not Magic Security Dust by Andrew Ayer

Oct 02, 2016 | www.agwa.name

Systemd maintainer David Strauss has published a response to my blog post about systemd . The first part of his post is replete with ad hominem fallacies, strawmen, and factual errors. Ironically, in the same breath that he attacks me for not understanding the issues around threads and umasks, he betrays an ignorance of how the very project which he works on uses threads and umasks . This doesn't deserve a response beyond what I've called out on Twitter.

In the second part of his blog post, Strauss argues that systemd improves security by making it easy to apply hardening techniques to the network services which he calls the "keepers of data attackers want." According to Strauss, I'm "fighting one of the most powerful tools we have to harden the front lines against the real attacks we see every day." Although systemd does make it easy to restrict the privileges of services, Strauss vastly overstates the value of these features.

The best systemd can offer is whole application sandboxing. You can start a daemon as a non-root user, in a restricted filesystem namespace, with mandatory access control. Sandboxing an entire application is an effective way to run potentially malicious code, since it protects other applications from the malicious one. This makes sandboxing useful on smartphones, which need to run many different untrustworthy, single-user applications. However, since sandboxing a whole application cannot protect one part of the application from a compromise of a different part, it is ineffective at securing benign-but-insecure software, which is the problem faced on servers. Server applications need to service requests from many different users. If one user is malicious and exploits a vulnerability in the application, whole application sandboxing doesn't protect the other users of the service.

For concrete examples, let's consider Apache and Samba, two daemons which Strauss says would benefit from systemd's features.

First Apache. You can start Apache as a non-root user provided someone else binds to ports 443 and 80. You can further sandbox it by preventing it from accessing parts of the filesystem it doesn't need to access. However, no matter how much you try to sandbox Apache, a typical setup is going to need a broad amount of access to do its job, including read permission to your entire website (including password-protected parts) and access to any credential (database password, API key, etc.) used by your CGI, PHP, or similar webapps.

Even under systemd's most restrictive sandboxing, an attacker who gains remote code execution in Apache would be able to read your entire website, alter responses to your visitors, steal your HTTPS private keys, and gain access to your database and any API consumed by your webapps. For most people, this would be the worst possible compromise, and systemd can do nothing to stop it. Systemd's sandboxing would prevent the attacker from gaining access to the rest of your system (absent a vulnerability in the kernel or systemd), but in today's world of single-purpose VMs and containers, that protection is increasingly irrelevant. The attacker probably only wants your database anyways.

To provide a meaningful improvement to security without rewriting in a memory-safe language, Apache would need to implement proper privilege separation. Privilege separation means using multiple processes internally, each running with different privileges and responsible for different tasks, so that a compromise while performing one task can't lead to the compromise of the rest of the application. For instance, the process that accepts HTTP connections could pass the request to a sandboxed process for parsing, and then pass the parsed request along to yet another process which is responsible for serving files and executing webapps. Privilege separation has been used effectively by OpenSSH, Postfix, qmail, Dovecot, and over a dozen daemons in OpenBSD . (Plus a couple of my own: titus and rdiscd .) However, privilege separation requires careful design to determine where to draw the privilege boundaries and how to interface between them. It's not something which an external tool such as systemd can provide. (Note: Apache already implements privilege separation that allows it to process requests as a non-root user, but it is too coarse-grained to stop the attacks described here.)

Next Samba, which is a curious choice of example by Strauss. Having configured Samba and professionally administered Windows networks, I know that Samba cannot run without full root privilege. The reason why Samba needs privilege is not because it binds to privileged ports, but because, as a file server, it needs the ability to assume the identity of any user so it can read and write that user's files. One could imagine a different design of Samba in which all files are owned by the same unprivileged user, and Samba maintains a database to track the real ownership of each file. This would allow Samba to run without privilege, but it wouldn't necessarily be more secure than the current design, since it would mean that a post-authentication vulnerability would yield access to everyone's files, not just those of the authenticated user. (Note: I'm not sure if Samba is able to contain a post-authentication vulnerability, but it theoretically could. It absolutely could not if it ran as a single user under systemd's sandboxing.)

Other daemons are similar. A mail server needs access to all users' mailboxes. If the mail server is written in C, and doesn't use privilege separation, sandboxing it with systemd won't stop an attacker with remote code execution from reading every user's mailbox. I could continue with other daemons, but I think I've made my point: systemd is not magic pixie dust that can be sprinkled on insecure server applications to make them secure. For protecting the "data attackers want," systemd is far from a "powerful" tool. I wouldn't be opposed to using a library or standalone tool to sandbox daemons as a last line of defense, but the amount of security it provides is not worth the baggage of running systemd as PID 1.

Achieving meaningful improvement in software security won't be as easy as adding a few lines to a systemd config file. It will require new approaches, new tools, new languages. Jon Evans sums it up eloquently :

... as an industry, let's at least set a trajectory . Let's move towards writing system code in better languages, first of all -- this should improve security and speed. Let's move towards formal specifications and verification of mission-critical code.

Systemd is not part of this trajectory. Systemd is more of the same old, same old, but with vastly more code and complexity, an illusion of security features, and, most troubling, lock-in. (Strauss dismisses my lock-in concerns by dishonestly claiming that applications aren't encouraged to use their non-standard DBUS API for DNS resolution. Systemd's own documentation says "Usage of this API is generally recommended to clients." And while systemd doesn't preclude alternative implementations, systemd's specifications are not developed through a vendor-neutral process like the IETF, so there is no guarantee that other implementers would have an equal seat at the table.) I have faith that the Linux ecosystem can correct its trajectory. Let's start now, and stop following systemd down the primrose path.

[Jan 26, 2019] Systemd Flaw Leaves Linux Distributions Scrambling to Patch by Lucian Constantin

Jul 03, 2017 | thenewstack.io
Ubuntu, Fedora, Arch Linux and other Linux distributions have released patches for a serious arbitrary code execution vulnerability that could be exploited through malicious Domain Name System (DNS) packets.

The flaw was found in systemd-resolved , a service that's part of the systemd initialization system adopted by many Linux distributions in recent years. The resolved service provides network name resolution to local applications by querying DNS servers.

The vulnerability, tracked as CVE-2017-9445 , was discovered by Chris Coulson , a software engineer at Canonical and member of the Ubuntu team, who noticed that when dealing with certain data packet sizes, systemd-resolved fails to allocate a sufficiently large buffer.

"A malicious DNS server can exploit this by responding with a specially crafted TCP payload to trick systemd-resolved to allocate a buffer that's too small, and subsequently write arbitrary data beyond the end of it," Coulson said in an advisory posted on the Open Source Security mailing list.

This could be exploited to crash the systemd-resolved daemon or to execute potentially malicious code in its context.

There are multiple ways in which an attacker could send malicious DNS packets to a Linux system with systemd-resolved running. One of them is by launching a man-in-the-middle attack on an insecure wireless network or through a compromised router.

Fortunately, not all Linux systems are affected because some distributions don't use systemd and even among those that do, not all of them include systemd-resolved. For example, SUSE and openSUSE distributions don't ship this component and, while Debian 9 (Stretch) includes it, the service is not enabled by default . The previous Debian versions don't have the vulnerable code at all.

Red Hat rated this vulnerability as important and assigned it a Common Vulnerability Scoring System (CVSS) score of 7.5, but determined that it does not affect the versions of systemd shipped with Red Hat Enterprise Linux 7. Fedora, however, is affected and has issued patches .

Ubuntu , Arch Linux and probably other distributions are also affected. Users should check if they have any updates pending for systemd and should deploy the patches as soon as possible. According to Coulson, the flaw was likely introduced in systemd version 223 in 2015 and affects all versions up to and including 233.

[Jan 26, 2019] Three security bugs found in the popular Linux suite systemd by Pierluigi Paganini

Jan 10, 2019 | securityaffairs.co
Security firm Qualys has disclosed three flaws (CVE-2018-16864, CVE-2018-16865, and CVE-2018-16866 ) in a component of systemd , a software suite that provides fundamental building blocks for a Linux operating system used in most major Linux distributions.

The flaws reside in the systemd journald , a service of the systemd that collects and stores logging data.

Both CVE-2018-16864 and CVE-2018-16865 bugs are memory corruption vulnerabilities, while the CVE-2018-16866 is an out of bounds issue that can lead to an information leak.

Security patches for the three vulnerabilities are included in distro repository since the coordinated disclosure, but some Linux distros such as some versions of Debian remain vulnerable. The flaws cannot be exploited in SUSE Linux Enterprise 15, openSUSE Leap 15.0, and Fedora 28 and 29 because their code is compiled with GCC's -fstack-clash-protection option.

[Jan 26, 2019] Systemd flaw could cause the crash or hijack of vulnerable Linux machines by Pierluigi Paganini

Notable quotes:
"... is vulnerable to an out-of-bounds heap write in the DHCPv6 client when handling options sent by network adjacent DHCP servers. ..."
"... could exploit this via malicious DHCP server to corrupt heap memory on client machines, resulting in a denial of service or potential code execution." reads the advisory published by Red Hat. ..."
Oct 29, 2018 | securityaffairs.co

Both Ubuntu and Red Hat Linux published a security advisory on the issue. summary :

" systemd networkd is vulnerable to an out-of-bounds heap write in the DHCPv6 client when handling options sent by network adjacent DHCP servers. A attacker could exploit this via malicious DHCP server to corrupt heap memory on client machines, resulting in a denial of service or potential code execution." reads the advisory published by Red Hat.

"Felix Wilhelm discovered that systemd-networkd's dhcp6 client could be made to write beyond the bounds (buffer overflow) of a heap allocated buffer when responding to a dhcp6 server with an overly-long server-id parameter." reads the advisory published by Ubuntu.

The author of Systemd, Leonard Poettering, promptly published a security fix for Systemd-based Linux system relying on systemd-networkd.

[Jan 25, 2019] Some systemd problems that arise in reasonably complex datacenter environment

May 10, 2018 | theregister.co.uk
Thursday 10th May 2018 16:34 GMT Nate Amsden

as a linux user for 22 users

(20 of which on Debian, before that was Slackware)

I am new to systemd, maybe 3 or 4 months now tops on Ubuntu, and a tiny bit on Debian before that.

I was confident I was going to hate systemd before I used it just based on the comments I had read over the years, I postponed using it as long as I could. Took just a few minutes of using it to confirm my thoughts. Now to be clear, if I didn't have to mess with the systemd to do stuff then I really wouldn't care since I don't interact with it (which is the case on my laptop at least though laptop doesn't have systemd anyway). I manage about 1,000 systems running Ubuntu for work, so I have to mess with systemd, and init etc there.

If systemd would just do ONE thing I think it would remove all of the pain that it has inflicted on me over the past several months and I could learn to accept it.

That one thing is, if there is an init script, RUN IT. Not run it like systemd does now. But turn off ALL intelligence systemd has when it finds that script and run it. Don't put it on any special timers, don't try to detect if it is running already, or stopped already or whatever, fire the script up in blocking mode and wait till it exits.

My first experience with systemd was on one of my home servers, I re-installed Debian on it last year, rebuilt the hardware etc and with it came systemd. I believe there is a way to turn systemd off but I haven't tried that yet. The first experience was with bind. I have a slightly custom init script (from previous debian) that I have been using for many years. I copied it to the new system and tried to start bind. Nothing. I looked in the logs and it seems that it was trying to interface with rndc(internal bind thing) for some reason, and because rndc was not working(I never used it so I never bothered to configure it) systemd wouldn't launch bind. So I fixed rndc and systemd would now launch bind, only to stop it within 1 second of launching. My first workaround was just to launch bind by hand at the CLI (no init script), left it running for a few months. Had a discussion with a co-worker who likes systemd and he explained that making a custom unit file and using the type=forking option may fix it.. That did fix the issue.

Next issue came up when dealing with MySQL clusters. I had to initialize the cluster with the "service mysql bootstrap-pxc" command (using the start command on the first cluster member is a bad thing). Run that with systemd, and systemd runs it fine. But go to STOP the service, and systemd thinks the service is not running so doesn't even TRY to stop the service(the service is running). My workaround for my automation for mysql clusters at this point is to just use mysqladmin to shut the mysql instances down. Maybe newer mysql versions have better systemd support though a co-worker who is our DBA and has used mysql for many years says even the new Maria DB builds don't work well with systemd. I am working with Mysql 5.6 which is of course much much older.

Next issue came up with running init scripts that have the same words in them, in the case of most recently I upgraded systems to systemd that run OSSEC. OSSEC has two init scripts for us on the server side (ossec and ossec-auth). Systemd refuses to run ossec-auth because it thinks there is a conflict with the ossec service. I had the same problem with multiple varnish instances running on the same system (varnish instances were named varnish-XXX and varnish-YYY). In the varnish case using custom unit files I got systemd to the point where it would start the service but it still refuses to "enable" the service because of the name conflict (I even changed the name but then systemd was looking at the name of the binary being called in the unit file and said there is a conflict there).

fucking a. Systemd shut up, just run the damn script. It's not hard.

Later a co-worker explained the "systemd way" for handling something like multiple varnish instances on the system but I'm not doing that, in the meantime I just let chef start the services when it runs after the system boots(which means they start maybe 1 or 2 mins after bootup).

Another thing bit us with systemd recently as well again going back to bind. Someone on the team upgraded our DNS systems to systemd and the startup parameters for bind were not preserved because systemd ignores the /etc/default/bind file. As a result we had tons of DNS failures when bind was trying to reach out to IPv6 name servers(ugh), when there is no IPv6 connectivity in the network (the solution is to start bind with a -4 option).

I believe I have also caught systemd trying to mess with file systems(iscsi mount points). I have lots of automation around moving data volumes on the SAN between servers and attaching them via software iSCSI directly to the VMs themselves(before vsphere 4.0 I attached them via fibre channel to the hypervisor but a feature in 4.0 broke that for me). I noticed on at least one occasion when I removed the file systems from a system that SOMETHING (I assume systemd) mounted them again, and it was very confusing to see file systems mounted again for block devices that DID NOT EXIST on the server at the time. I worked around THAT one I believe with the "noauto" option in fstab again. I had to put a lot of extra logic in my automation scripts to work around systemd stuff.

I'm sure I've only scratched the surface of systemd pain. I'm sure it provides good value to some people, I hear it's good with containers (I have been running LXC containers for years now, I see nothing with systemd that changes that experience so far).

But if systemd would just do this one thing and go into dumb mode with init scripts I would be quite happy.

[Jan 25, 2019] SystemD vs Solaris 10 SMF

"Shadow files" approach of Solaris 10, where additional functions of init are controlled by XML script that exist in a separate directory with the same names as init scripts can be improved but architecturally it is much cleaner then systemd approach.
Notable quotes:
"... Solaris has a similar parallellised startup system, with some similar problems, but it didn't need pid 1. ..."
"... Agreed, Solaris svcadm and svcs etc are an example of how it should be done. A layered approach maintaining what was already there, while adding functionality for management purposes. Keeps all the old text based log files and uses xml scripts (human readable and editable) for higher level functions. ..."
"... AFAICT everyone followed RedHat because they also dominate Gnome, and chose to make Gnome depend on systemd. Thus if one had any aspirations for your distro supporting Gnome in any way, you have to have systemd underneath it all. ..."
Jan 25, 2019 | theregister.co.uk

Re: Poettering still doesn't get it... Pid 1 is for people wearing big boy pants.

SystemD is corporate money (Redhat support dollars) triumphing over the long hairs sadly. Enough money can buy a shitload of code and you can overwhelm the hippies with hairball dependencies (the key moment was udev being dependent on systemd) and soon get as much FOSS as possible dependent on the Linux kernel.

This has always been the end game as Red Hat makes its bones on Linux specifically not on FOSS in general (that say runs on Solaris or HP-UX).

The tighter they can glue the FOSS ecosystem and the Linux kernel together ala Windows lite style the better for their bottom line. Poettering is just being a good employee asshat extraordinaire he is.

Re: Ahhh SystemD

I honestly would love someone to lay out the problems it solves.

Solaris has a similar parallellised startup system, with some similar problems, but it didn't need pid 1.

Re: Ahhh SystemD

Agreed, Solaris svcadm and svcs etc are an example of how it should be done. A layered approach maintaining what was already there, while adding functionality for management purposes. Keeps all the old text based log files and uses xml scripts (human readable and editable) for higher level functions.

Afaics, systemd is a power grab by Red Hat and an ego trip for it's primary developer.

Dumped bloatware Linux in favour of FreeBSD and others after Suse 11.4, though that was bad enough with Gnome 3...

starbase7, Thursday 10th May 2018 04:36 GMT

SMF?

As an older timer (on my way but not there yet), I never cared for the init.d startup and I dislike the systemd monolithic architecture.

What I do like is Solaris SMF and wish Linux would have adopted a method such as or similar to that. I still think SMF was/is a great comprise to the init.d method or systemd manor.

I used SMF professionally, but now I have moved on with Linux professionally as Solaris is, well, dead. I only get to enjoy SMF on my home systems, and savor it. I'm trying to like Linux over all these years, but this systemd thing is a real big road block for me to get enthusiastic.

I have a hard time understanding why all the other Linux distros joined hands with Redhat and implemented that thing, systemd. Sigh.

Anonymous Coward, Thursday 10th May 2018 04:53 GMT

Re: SMF?

You're not alone in liking SMF and Solaris.

AFAICT everyone followed RedHat because they also dominate Gnome, and chose to make Gnome depend on systemd. Thus if one had any aspirations for your distro supporting Gnome in any way, you have to have systemd underneath it all.

RedHat seem to call the shots these days as to what a Linux distro has. I personally have mixed opinions on this; I think the vast anarchy of Linux is a bad thing for Linux adoption ("this is the year of the Linux desktop" don't make me laugh), and Linux would benefit from a significant culling of the vast number of distros out there. However if that did happen and all that was left was something controlled by RedHat, that would be a bad situation.

Steve Davies, Thursday 10th May 2018 07:30 GMT 3

Re: SMF?
Remember who 'owns' SMF... namely Oracle. They may well have made it impossible for anyone to adopt. That stance is not unknown now is it...?

As for systemd, I have bit my teeth and learned to tolerate it. I'll never be as comfortable with it as I was with the old init system but I did start running into issues especially with shutdown syncing with it on some complex systems.

Still not sure if systemd is the right way forward even after four years.

Daggerchild, Thursday 10th May 2018 14:30 GMT

Re: SMF?
SMF should be good, and yet they released it before they'd documented it. Strange priorities...

And XML is *not* a config file format you should let humans at. Finding out the correct order to put the XML elements in to avoid unexplained "parse error", was *not* a fun game.

And someone correct me, but it looks like there are SMF properties of a running service that can only be modified/added by editing the file, reloading *and* restarting the service. A metadata and state/dependency tracking system shouldn't require you to shut down the priority service it's meant to be ensuring... Again, strange priorities...

12 1 Reply
Friday 11th May 2018 07:55 GMTonefangSilver badge
Reply Icon
FAIL
Re: SMF?
"XML is *not* a config file format you should let humans at"

XML is a format you shouldn't let computers at, it was designed to be human readable and writable. It fails totally.

5 1 Reply
Friday 6th July 2018 12:27 GMTHans 1Silver badge
Reply Icon
Re: SMF?
Finding out the correct order to put the XML elements in to avoid unexplained "parse error", was *not* a fun game.

Hm, you do know the grammar is in a dtd ? Yes, XML takes time to learn, but very powerful once mastered.

0 1 Reply
Thursday 10th May 2018 13:24 GMTCrazyOldCatManSilver badge
Reply Icon
Re: SMF?
I have a hard time understanding why all the other Linux distros joined hands with Redhat and implemented that thing, systemd

Several reasons:

A lot of other distros use Redhat (or Fedora) as their base and then customise it.

A lot of other distros include things dependant on systemd (Gnome being the one with biggest dependencies - you can just about to get it to run without systemd but it's a pain and every update will break your fixes).

Redhat has a lot of clout.

6 3

[Jan 21, 2019] System Down A systemd-journald exploit Hacker News

Notable quotes:
"... Also, as I understand it the exploit would not exist if it was literally just outputting log lines to a file in /var/log/systemd/ ? ..."
Jan 21, 2019 | news.ycombinator.com
segfaultbuserr 11 days ago [-]

Yet another proof for the following:

1. It's reasonable to claim that amd64 (x86_64) is more secure than x86. x86_64 has larger address space, thus higher ASLR entropy. The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64. If some alert systems have been deploy on the server (attacks need to keep crashing systemd-journald in this process), it buys time. In other cases, it makes exploitation infeasible.

2. CFLAGS hardening works, in addition to ASLR, it's the last line of defense for all C programs. As long as there are still C programs running, patching all memory corruption bugs is impossible. Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage. All hardening flags should be turned on by all distributions, unless there is a special reason. Fedora turned "-fstack-clash-protection" on since Fedora 28 ( https://fedoraproject.org/wiki/Changes/HardeningFlags28 ).

If you are releasing a C program on Linux, please consider the following,

    -D_FORTIFY_SOURCE=2         glibc hardening

    -Wp,-D_GLIBCXX_ASSERTIONS   glibc++ hardening

    -fstack-protector-strong    stack smash protection

    -fstack-clash-protection    stack clash protection

    -fPIE -pie                  better ASLR protection

    -Wl,-z,noexecstack          don't allow code on stack

    -Wl,-z,relro                ELF hardening

    -Wl,-z,now                  ELF hardening
Major Linux distributions, including Fedora, Debian, Arch Linux, openSUSE are already doing it. Similarly, Firefox and Chromium are using many of these flags too. Unfortunately, Debian did not use `-fstack-clash-protection` and got hit by the exploit, because it was only added since GCC 8.

For a more comprehensive review, check

* Recommended compiler and linker flags for GCC:

https://developers.redhat.com/blog/2018/03/21/compiler-and-l...

* Debian Hardening

https://wiki.debian.org/Hardening

lmm 11 days ago [-]

"Proof" suggests a level of absolute confidence that this example certainly does not give.

> The exploit needs 10 minutes to crack ASLR on x86, but 70 minutes on amd64.

Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"?

> Using mitigation techniques and sandbox-based isolation are the only two ways to limit the damage.

I'm not at all convinced that mitigation techniques represent a real improvement in security, because by definition a mitigation technique is not backed by a solid model. If you're letting an attacker control the modification of memory that your security model assumes isn't modifiable, how confident can you be that ad-hoc mitigations for all the ways you could think of to exploit that cover all the possible ways to exploit that? E.g. I can remember a time when ASLR was touted as a solution to C's endemic security vulnerabilities; now cracking ASLR as part of vulnerability exploitation is routine, as seen here. Mitigations appear to give a security improvement because an app with mitigations is no longer the low-hanging fruit, but I suspect this is a case of "you don't have to outrun the bear": as long as there are C programs without mitigations, attackers will go after those first. That's different from saying that mitigations provide substantial protection.

FakeComments 11 days ago [-]

The hands-on-keyboard SLA for a lot of on-calls is 30 minutes.

So in an "attack was detected, break all the glass" scenario, the difference between 10 and 70 minutes is sufficient to allow human operators to render the attack moot by offlining its target, while the attackers are still trying to break through API servers.

At both big corps I've been at, the incident response plan for an exfiltration attack on customer data was invalidate DB creds and take the system down ourselves.

Better to be out of service than lose custody of customer data.

reply dTal 11 days ago [-]

>Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"?

How about an intrusion detection system that flags up a human response? 10 minutes is hardly any time at all to respond, an hour gives you a chance to roll out of bed.

reply segfaultbuserr 11 days ago [-]

PaX offers an anti-bruteforce protection: if the kernel discovers a crash, the `fork()` syscall of the parent process is blocked for 30 seconds for each failed attempt, the attacker is going to have a hard time beating 32-bit entropy. Meanwhine, it also writes a critical-level message to the kernel logbuffer to notify sysadmins, and possibly uncover the 0day exploit the attacker has used.

wstuartcl 11 days ago [-]

I guess, as long as the IDS senses the attack in progress quickly -- my gut is this type of attack would be hard to detect until the outcome was achieved. More likely the initial entry would be the detected event(s) -- in which case yeah the extra time gives some safety net.

In either case, it still feels like pulling all things into systemd creates a much harder to protect surface area on systems. Why should init care if your logger crashes, let alone take down init with it? I am not a anti-systemd person but I honestly do see the tradeoffs of the "let me do it all" architecture as a huge penalty.

viraptor 11 days ago [-]

> Why should init care if your logger crashes

It cares in the same way it cares about all the other processes. There's nothing systemd-specific here. Journald service is configured to restart of crash, same as many other services.

It's not taking down init when journald crashes either.

dane-pgp 11 days ago [-]

> There's nothing systemd-specific here.

Well, except journald itself.

fao_ 11 days ago [-]

> In either case, it still feels like pulling all things into systemd creates a much harder to protect surface area on systems. Why should init care if your logger crashes, let alone take down init with it? I am not a anti-systemd person but I honestly do see the tradeoffs of the "let me do it all" architecture as a huge penalty.

100% this. Also, as I understand it the exploit would not exist if it was literally just outputting log lines to a file in /var/log/systemd/ ?

EDIT: Also as I understand it, appending directly to a file is just as stable as the journald approach, given that many, many disk controllers and kernels are known to lie about whether they have actually flushed their cache to disk (actually moreso, because the binary format of journald is arguably more difficult to recover into proper form than a timestamped plaintext -- please correct me if I'm wrong, though!!)

viraptor 11 days ago [-]

> the binary format of journald is arguably more difficult to recover into proper form than a timestamped plaintext -- please correct me if I'm wrong, though!!

It depends what you mean by recover. To get the basic plaintext, you can pretty much run "strings" on the journal file and grep for "MESSAGE=". It's append-only so the entries are in order. Just because it's a binary file doesn't mean the text itself is mangled. (Unless you enable compression)

The reference may look complicated https://www.freedesktop.org/wiki/Software/systemd/journal-fi... but that's all extra features you may ignore for "recovery in emergency".

loeg 11 days ago [-]

> Why should init care if your logger crashes, let alone take down init with it?

They're separate processes. Logger crashes do not take down init.

> I am not a anti-systemd person

Whether you are or not, you are (inadvertently) repeating misinformation about it.

okket 11 days ago [-]

A server suddenly spiking full load for over an hour should raise alarms. It shows up even in the dumbest of 5 minute averaging charting tools.

A few minutes of high load can easily get overlooked.

geggam 11 days ago [-]

Enterprise systems or any large scale stack can have one running like this where people dismiss it for an hour. Some systems run hard like this by default. See Transcoding

pixl97 11 days ago [-]

Also, Weekend and Christmas attacks. In the field we are seeing more attacks with a valid username and pass occur at times when a sysadmin may not be on call.

segfaultbuserr 11 days ago [-]

> Is there any realistic threat model under which the difference between 10 minutes and 70 minutes is the difference between "insecure" and "secure"?

Time is given here just for an example. To crack systemd, it only takes 70 minutes, but in general, bruteforcing ASLR on 64-bit systems can take as few as 1.3 hours but as many as 34.1 hours, depending on the nature of bug. On the other hand, the ~20-bit of entropy on 32-bit systems is trivial to crack in 10 minutes for nearly all cases, and does not provide an adequate security margin.

Oon a 64-bit system there is ~32-40 bit of ASLR entropy available for a PIE program. It forces an attacker to brute-force it. Unlike other protections, no matter how is the system cleverly analyzed beforehand, it taxes the exploit by forcing it to solve a computational puzzle. This fact alone, is enough to stop many "Morris Worm"-type remote exploitations (they have suddenly became a serious consideration, given the future of IoT), since an exploit takes months or years to crack a single machine.

If it's not enough (it is not, I acknowledge ASLR by itself cannot be enough), an intrusion detection system should be used, and it already has used by many. For example, PaX offers an optional, simple yet effective anti-bruteforce protection: if the kernel discovers a crash, the `fork()` attempt of the parent process is blocked for 30 seconds. It takes years before an attacker is able to overcome the randomization (so the attacker is likely to try something else). In addition, it also writes a critical-level message to the kernel logbuffer, the sysadmin can be notified, and possibly uncover the 0day exploit the attacker has used. I'd call it a realistic threat model.

Finally, information leaks is a great concern here. Kernels and programs are leaking memory address like a sieve, and effectively making ASLR useless. Linux kernel is already actively plugging these holes (but with limited effectiveness, HardenedBSD should be the future case-study), so should other programs.

> e.g. I can remember a time when ASLR was touted as a solution to C's endemic security vulnerabilities; now cracking ASLR as part of vulnerability exploitation is routine, as seen here.

You can make the same comment on NX bit, or W^X/PaX, or BSD jail, or SMAP/SMEP (in recent Intel CPUs), or AppArmor, or SELinux, or seccomp(), or OpenBSD's pledge(), or Control Flow Integrity, or process-based sandboxing in web browsers, or virtual machine-based isolation.

Better defense leads to better attacks, and it in turns leads to better defense. By playing the game, it may not be possible to win, but by not playing it, losing the game is guaranteed. In this case, systemd is exploitable despite ASLR, due to a relatively new exploit technique called "Stack Clash", and for this matter, GCC has already updated its -fstack-check to the new -fstack-clast-protection long before the systemd exploit was discovered. If this mitigation has been used (like, by Fedora and openSUSE), it causes simply a crash, and is not exploitable. At least before the attacker finds another way round.

Early kernels and web browsers have no memory and exploit protections whatsoever: a single wrong pointer dereference or buffer overflow is enough to completely takeover the system. Nowadays, an attack needs to overcome at least NX, ASLR, sandboxing, and compiler-level mitigation, and we still see exploits. So the conclusion is all mitigations are completely useless? If it's your opinion, I'm fine to agree to your disagreement, many sensitive C programs need to be written in a memory-safe language anyway. But as I see it, as long as there are still C programs running with undiscovered vulnerabilities, and as long as attackers have to add more and more up-to-date workarounds and cracking techniques (ROP, anyone? but now the most sophisticated attackers are moving to DATA-ONLY attacks) to their exploit checklist, then we are not losing the race by increasing the cost of attacks.

On the other hand, if an attacker don't have to use an up-to-date cracking techniques, then we have serious problems. For example, broken and incomplete mitigation is often seen in the real word, and it's the real trouble. Recently, it has been discovered that the ASLR implementation in the MinGW toolchain is broken, allowing attackers to exploit VLC using shellcode tricks from the 2000s ( https://insights.sei.cmu.edu/cert/2018/08/when-aslr-is-not-r... ). And we still see broken NX bit protection and the total absence of any ASLR, or -fstack-protector in ALL home routers ( https://cyber-itl.org/2018/12/07/a-look-at-home-routers-and-... ).

The principle of Defense-in-Depth is that, if the enemies are powerful enough, it's inevitable all protections will be overcame. Like the Swiss Cheese Model ( https://en.wikipedia.org/wiki/Swiss_cheese_model ), a cliche in accident analysis, eventually there will be something that managed to find a hole in every layer of defense and pass though. What we can do, is to do our best at each layer of defense to prevent the preventable incidents, and adding more layers when the technology permits us.

My final words are: at least, do something. ASLR is already implemented as a prototype, analyzed, and exploited by clever hackers back in 2002 ( http://phrack.org/issues/59/9.html ), but only seen major adoptions ten years later. It would be a surprise if ASLR-breaking techniques has not improved given the inaction of most vendors.

> "Proof" suggests a level of absolute confidence that this example certainly does not give.

I agree. I should've use "given more empirical evidences" instead of "given a proof".

For real security, I believe memory-safe programming (e.g. Rust), and formal verification (e.g seL4) are the way forward, although they still have a long way to go.

lmm 10 days ago [-]

> You can make the same comment on NX bit, or W^X/PaX, or BSD jail, or SMAP/SMEP (in recent Intel CPUs), or AppArmor, or SELinux, or seccomp(), or OpenBSD's pledge(), or Control Flow Integrity, or process-based sandboxing in web browsers

I can, and I would.

> or virtual machine-based isolation

A little different because a VM can be designed to offer a rigid security boundary (with a solid model behind it) rather than as an ad-hoc mitigation technique.

> So the conclusion is all mitigations are completely useless? If it's your opinion, I'm fine to agree to your disagreement, many sensitive C programs need to be written in a memory-safe language anyway. But as I see it, as long as there are still C programs running with undiscovered vulnerabilities, and as long as attackers have to add more and more up-to-date workarounds and cracking techniques (ROP, anyone? but now the most sophisticated attackers are moving to DATA-ONLY attacks) to their exploit checklist, then we are not losing the race by increasing the cost of attacks.

> The principle of Defense-in-Depth is that, if the enemies are powerful enough, it's inevitable all protections will be overcame. Like the Swiss Cheese Model ( https://en.wikipedia.org/wiki/Swiss_cheese_model ), a cliche in accident analysis, eventually there will be something that managed to find a hole in every layer of defense and pass though. What we can do, is to do our best at each layer of defense to prevent the preventable incidents, and adding more layers when the technology permits us.

> For real security, I believe memory-safe programming (e.g. Rust), and formal verification (e.g seL4) are the way forward, although they still have a long way to go.

I think the defense in depth / swiss cheese approach has shown itself to be a failure, and exploit mitigation techniques have been a distraction from real security. It's worth noting that systemd is both recently developed and aggressively compatibility-breaking; there really is no excuse for it to be written in C, mitigations or no. Even if you don't think Rust was mature enough at that point, there were memory-safe languages that would have made sense (OCaml, Ada, ...). Certainly there's always more to be done, but I really don't think there's anything that would block the adoption of these languages and techniques if the will was there.

nickpsecurity 11 days ago [-]

Before the critique, I want to thank you all the detailed information (esp compiler tips) you're putting out on the thread for everyone. :)

"You can make the same comment on NX bit, or W^X/PaX, or BSD jail, or SMAP/SMEP (in recent Intel CPUs), or AppArmor, or SELinux, or seccomp(), or OpenBSD's pledge(), or Control Flow Integrity, or process-based sandboxing in web browsers, or virtual machine-based isolation."

You can indeed say that about all those systems since they mix insecure, bug-ridden code with probabilistic and tactical mechanisms that they prey will stop hackers. In high-assurance security, the focus was instead to identify each root cause, prevent/detect/fail-safe on it with some method, and add automation where possible for these. Since a lot of that is isolation, I'd say the isolation based method would be separation kernels running apps in their own compartments or in deprivileged, user-mode VM's. Genode OS is following that path with stuff like seL4, Muen, and NOVA running undearneath. First two are separation kernels, NOVA just correctnes focused with high-assurance, design style.

Prior systems designed like those did excellent in NSA pentesting whereas the UNIX-based systems with extensions like MAC were shredded. All we're seeing is a failure to apply the lessons of the past in both hardware and software with predictable results.

"Better defense leads to better attacks, and it in turns leads to better defense. By playing the game, it may not be possible to win, but by not playing it, losing the game is guaranteed. "

Folks using stuff like Ada, SPARK, Frama-C w/ sound analyzers, Rust, Cryptol, and FaCT are skipping playing the game to just knock out all the attack classes. Plus, memory-safety methods for legacy code like SAFEcode in SVA-OS or Softbound+CETS. Throw in Data-Flow Integrity or Information-Flow Control (eg JIF/SIF languages). Then, you just have to increase hardware spending a bit to make up for the performance penalty that comes with your desired level of security. Trades a problem that takes geniuses decades to solve for one an average, IT person with an ordering guide can handle quickly on eBay. Assuming the performance penalty even matters given how lots of code isn't CPU-bound.

I'd rather not play the "extend and obfuscate insecure stuff for the win" game if possible since defenders have been losing it consistently for decades. Obfuscation should just be an extra measure on top of methods that eliminate root causes to further frustrate attackers. Starting with most cost-effective for incremental progress like memory-safe languages, contracts, test generation, and static/dynamic analysis. The heavyweight stuff on ultra-critical components such as compilers, crypto/TLS, microkernels, clustering protocols, and so on. We already have a lot of that, though.

"For real security, I believe memory-safe programming (e.g. Rust), and formal verification (e.g seL4) are the way forward, although they still have a long way to go. "

Well, there you go saying it yourself. :)

"Early kernels and web browsers have no memory and exploit protections whatsoever"

Yeah, we pushed for high-assurance architecture to be applied there. Chrome did a weakened version of OP. Here's another design if you're interested in how to solve... attempt to solve... that problem:

https://www.usenix.org/legacy/events/osdi10/tech/full_papers...

wahern 11 days ago [-]

FWIW, the stack vulnerabilities here aren't just a C problem. Most languages, including every language relying on LLVM and GCC until the most recent versions, failed to perform stack probing.

I hesitate to call stack probing "hardening". IMO it's better understood as a failure by compilers to emit proper code in the first place, and it's been a glaringly obvious deficiency for years if not decades.

Continued

Recommended Links

Google matched content

Softpanorama Recommended

Top articles

Oldies But Goodies

[Jan 29, 2019] RHEL7 is a fine OS, the only thing it s missing is a really good init system.

Sites



Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D


Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March, 05, 2020