Welcome To 2024, The SSLVPN Chaos Continues - Ivanti CVE-2023-46805 & CVE-2024-21887
Did you have a good break? Have you had a chance to breathe? Wake up.
It’s 2024, and the chaos continues - thanks to Volexity (Volexity’s writeup), the industry has been alerted to in-the-wild exploitation of 2 incredibly serious 0days (CVE-2023-46805 and CVE-2024-21887 - two bugs, Command Injection and Authentication Bypass) in Ivanti (also known as Pulse Secure) Connect Secure (ICS) and Ivanti Policy Secure appliances - facilitating a full-device compromise and takeover.
CVE-2023-46805 & CVE-2024-21887 have been widely reported in the media as being utilised by nation-state-linked APT groups to compromise Ivanti appliances.
We’ve made it no secret - we (watchTowr) hate SSLVPN appliances. Not the concept of them, but that they all appear to have been constructed with the code equivalent of string, stamped with the word ‘secure’ and then just left to decay for 20 years.
What makes this situation even more “offensive” is Ivanti’s response (or lack of) to these vulnerabilities especially given the context - at the time of writing, a mitigation XML file is all that is available, with staggered patches available from the 22nd Jan 2024 per Ivanti. Yes, really.
We’re tired of the lack of responsibility taken by organisations when their devices are the literal gate between the Internet and their internal networks and hold an incredibly critical, and sensitive position in any network architecture.
Here at watchTowr, our job is to tell the organisations we work with whether they’re affected. Thus, we dived in.
If you haven’t read Volexity’s write-up yet, I’d advise reading it first for background information.
What Are The Bugs
As we stated earlier, there are two bugs at play here;
- CVE-2023-46805 and,
- CVE-2024-21887
Both are chained together, with CVE-2023-46805 allowing an unauthenticated Internet-based attacker to elevate to execute administrative functionality (bypass authentication), and CVE-2024-21887 allowing Command Execution via Command Injection within vulnerable administrative functionality.
Ivanti has tried very hard to make understanding these vulnerabilities as difficult as possible, distributing only an XML ‘mitigation’ within a private customer portal, and providing no actual patch yet.
Our Approach To Deciphering
Our usual approach with this kind of bug is straightforward - copy all the files from the target appliance, apply the patch, and then compare files. The fixes should make themselves known in the diff, or so the theory goes.
In this case, there is;
- No actual patch
- An (encrypted) mitigation XML only
As we dived in, we noted that our approach was hampered slightly by the full-disk encryption that Ivanti use to secure their product - we can’t simply boot into another OS and mount the disks, as they are encrypted with LUKS. Our usual approach would be to extract FDE keys from GRUB, which would typically mount the root device before passing execution to the kernel, but in this case, this was fruitless.
Some further investigation revealed that even the initrd booted is encrypted, which is interesting in itself, and suggests that the kernel itself has been modified to decrypt the image on the fly.
The first thing to try here is the old-faithful init=/bin/sh kernel command line argument, which will, when passed from GRUB, start a shell on the target just after the initrd has been mounted. Once the initrd is mounted and we have a shell, we can simply observe the encryption keys and use them to cold-mount the disks.
Frustratingly, though, attempts to do this were ignored by the appliance, again suggesting a custom kernel. Time to look at that kernel a little closer.
What’s this we see?
__int64 __fastcall sub_FFFFFFFF826CC601(unsigned __int8 *a1)
{
__int64 i;
if ( strcmp(a1, "/bin/sh") )
qword_FFFFFFFF827E2030 = a1;
for ( i = 0LL; i != 31; ++i )
qword_FFFFFFFF82212168[i] = 0LL;
return 1LL;
}
is… is that a blacklist on the term /bin/sh?! Really?! What a bizarre check. It’s easily bypassed, of course, by specifying a slightly different (but equivalent) argument (such as //bin//sh).
Doing so drops us right into a recovery shell, where we can find the FDE keys in /etc/lvmkey, which can then be used to mount the encrypted partitions.
Our approach then is fairly simple - an image of a vulnerable device, an image of a mitigation-applied device, and.. compare!
Unfortunately for us, however, doing so reveals no useful changes between the ‘mitigation XML-applied’ appliances, and vulnerable appliances. Time to take a different tactic.
Perhaps instead of trying to diff the command line by ourselves, we should be paying more attention to what breadcrumbs the vendor has left for us. Let’s take a closer look at the advice that Ivanti has for applying the workaround.
Well, Ivanti’s documentation for the mitigation warns that (among other things) “Automation built with REST API for configuration and monitoring will be impacted”.
Perhaps we should shift our focus there - what can we see that’s changed in the REST API? Let’s fire off a bunch of requests to a ‘patched’ and a ‘vulnerable’ VM, and see if we can spot any divergences.
Aha! This is looking more like it!
There are a handful of API endpoints that now respond with the above message, stating that access has been ‘blocked by your administrator’, instead of their usual response.
It seems like we’ve found the endpoints that have been restricted by the patch.
Editors note: All details from this point onwards have been redacted due to some inner feeling of moral responsibility that has crept up on us, and not wishing to even possibly add to the current barrage of exploitation that neighbourhood APTs are in the midst of.
Detection Approach
It’s important to note here that these changes in API behaviour happen before any authentication has been carried out, which is a massive help for defenders - given this information, it is straightforward to detect appliances that have had the patch applied without needing to authenticate.
Requesting the endpoint /api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark without supplying any authentication info will respond with an empty 403 if the device is vulnerable:
$ curl -v <https://host/api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark>
...
< HTTP/1.1 403 Forbidden
< Transfer-Encoding: chunked
< X-XSS-Protection: 1
< Strict-Transport-Security: max-age=31536000
<
However, performing the same request on a mitigation XML-applied version yields a full HTML page, rendering as above:
$ curl -v <https://host/api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark>
...
< HTTP/1.1 403 Forbidden
< Content-Type: text/html; charset=utf-8
< Connection: close
< Pragma: no-cache
< Cache-Control: no-store
< Expires: -1
< Content-Length: 3015
< Strict-Transport-Security: max-age=31536000
<
<!-- Copyright (c) 2022 by Ivanti Inc. All rights reserved -->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name=robots content="none">
<link rel="icon" href="/dana-na/imgs/Product_favicon.png" type="image/png">
<title>Ivanti Connect Secure</title>
..truncated..
<div id="error_message_content" class="intermediate__content">
Access to the Web site is blocked by your administrator. Please notify your system administrator. Made request for $request to $host:$port
</div>
..truncated..
</html>
It is important to note that this is one of 'a few' detection mechanisms we've identified - but hold a genuine concern that further sharing would ease the reproduction steps for bad-actors that are likely also watching this situation.
Conclusion
Another day, another SSL VPN bug. Sigh.
It’s been a fun 48 hours for us - reproducing these vulnerabilities, building unauthenticated detection mechanisms and ensuring the attack surfaces we help protect aren’t affected.
We will share details - but while there is in-the-wild exploitation, and Ivanti has not even released a patch - it would be truly irresponsible of us to do so at this point. In this case, we leave you with the below image teasing the Command Injection vulnerability alone - just to keep you on the hook a little.
Our closing note would be that - and we’re sure there may be legitimate reasons - we are still more than a week away from an actual patch from a security vendor for a pair of vulnerabilities that are being used in the wild by nation-state-linked APT.
But, as an industry, here we are. This is a disappointing place to be.
Once real patches are released, in our usual fashion we will be releasing further details in all their gory (you will cry with us at how ridiculous this all is).
Until then, please ensure sure you apply the mitigation.release.20240107.1.xml that Ivanti provides, and be careful out there - the APT is looking for more vendors that don’t do basic security in their enterprise-grade products.
(P.S. Please also follow Ivanti’s advice and perform integrity checks on your device - applying the mitigation alone is not enough).
At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.
If you'd like to learn more about how the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, can support your organisation, please get in touch.