Looking at how FortiOS validates a VM license
In the previous post I got seven FortiOS
images open end to end: outer XOR, ext2 carve, FORT-RC4, and the ARM PRGA
shift that nobody seemed to have written down. That leaves you with a rootfs.
Buried in it is a 106 MB stripped PIE at /bin/init that still does most
of the interesting work on a FortiGate.
This writeup is about the licensing half of that binary. I wanted a concrete answer to a simple question: when you paste a license into FortiOS, what does the client actually do before it decides the document is good or bad? Admin guides and marketing pages talk around that. The code is less shy.
Target is /bin/init from FGT_VM64_KVM FortiOS 8.0.0 (BuildID
46d1bfe698f61975105570cafc449b47d9ac252a). On the RX LOAD, file offset
equals VA, which makes static work less painful than it has any right to be.
Method is the usual one for a stripped PIE: find a string, scan .text for
RIP-relative leas that resolve to it, Capstone the neighborhood, chase the
next string loads. If I cannot point at an instruction, I do not claim it.
One boundary up front. Issuing a document that passes integrity and bind is Fortinet's job. I mapped the client side up to that wall. No generator, no patch notes for turning checks off.
First, this is not FlexLM
People still call any enterprise license file "FlexLM." FortiOS is not Flexera FlexNet Publisher. FortiFlex is a Fortinet product name for flexible VM entitlement delivery. It still lands in the same install path as everything else: you end up with a token or a PEM-ish document and the client runs it through the same pipeline.
What the binary implements is closer to a few independent questions that happen to share one product:
identity serial, UUID, platform string, model
platform what may this instance run as? (VM: /data/etc/vm.lic)
FortiCare who owns the serial for registration / support?
FortiGuard which DBs and features are entitled, until when?
VDOM how many virtual domains are allowed?
That split matters for used hardware. A permanent-eval story that exists for
some VM SKUs is not the same thing as "eBay chassis boots fully unlocked."
Physical identity is the burned serial. FortiCare and FortiGuard contracts
ride on that. The free eval .lic path is a VM platform document class.
You can read that out of the code paths without believing the brochure.
The header check is almost too clean
The nicest function in this whole mess starts at 0x1c47cd0. Arguments are
a buffer in rdi and a length in rsi. It walks off leading whitespace
with the bitmap constant 0x100002600 (the usual "is this a space-class
byte" bit test), then does two fixed-length compares:
0x1c47d56 lea rsi, "LICENSE-TOKEN:"
0x1c47d5d mov edx, 0xe ; 14
0x1c47d65 call [GOT] ; memcmp-class
0x1c47d6d je token_path ; logs "MIME parsed VM token"
0x1c47d73 lea rsi, "-----BEGIN FGT VM LICENSE-----"
0x1c47d7a mov edx, 0x1e ; 30, exact string length
0x1c47d82 call [GOT] ; same GOT slot
0x1c47d8a je license_path ; logs "MIME parsed VM license"
I double-checked the PEM lea the long way. Next RIP is 0x1c47d7a,
displacement is -0x18002c2, and that lands on string VA 0x447ab8.
The immediates are not approximate. They are the string lengths.
Both success edges fall into the same open path. That part is less "set a flag and hope" and more "stage the whole buffer to a fixed file":
0x1c47e0a lea rdi, "/etc/.cloudinit.lic"
0x1c47e11 lea rsi, "w"
0x1c47e18 call [GOT] ; fopen
0x1c47e1e mov r15, rax ; FILE*
0x1c47e25 mov byte ptr [out], 1 ; prefix matched (before null check)
0x1c47e42 test r15, r15
0x1c47e45 je fail ; "MIME failed to open file to write"
0x1c47e47 mov esi, 1
0x1c47e4c mov rdi, r14 ; buffer
0x1c47e4f mov rdx, rbx ; length
0x1c47e52 mov rcx, r15 ; FILE*
0x1c47e55 call [GOT] ; fwrite(buf, 1, len, fp)
0x1c47e60 mov rdi, r15
0x1c47e63 call [GOT] ; fclose
So after a recognized prefix, FortiOS writes the document to
/etc/.cloudinit.lic. That path has eleven RIP-relative lea sites in
.text. One is the write. Others are access/stat style checks (including an
OpenStack branch under /cdrom/openstack/content), cleanup next to the
string Cloudinit done, and pairing with
/etc/.cloudinit.script / .prescript. It is a staging object. The
long-lived store is elsewhere.
Success means reboot reason 2
There is a single lea to this string at 0x1c45662:
VM license install succeeded. Will reboot firewall.
The surrounding function re-checks LICENSE-TOKEN: with the same 14-byte
memcmp and whitespace strip. When a prior call returns zero, it frees its
buffers, prints the success line, and does something almost comically
literal:
0x1c45662 lea rdi, "VM license install succeeded. Will reboot firewall.\n"
0x1c4566b call log
0x1c45671 mov r12d, 2 ; reboot reason = 2
Reason 2 is not decoration. The reboot dispatcher around 0x1c45e30
compares against it:
0x1c45f72 cmp dword [rbp-0x30], 2
0x1c45f76 jne invalid
0x1c45f78 lea rdi, "Rebooting to install VM license\n"
0x1c45f81 call log
0x1c45f86 lea rdi, "Install VM license"
0x1c45f8d call 0x1c45da0 ; perform / reboot helper
...
0x1c45f92 lea rdi, "Invalid reboot reason: %d\n"
If you have ever installed a VM license and watched the box bounce, that is the reason code. Not a generic restart. A dedicated log line and a small integer.
Paths that get special treatment
A separate function at 0x56e4150 walks a candidate path and returns 0 or 1.
It runs a sequence of prefix compares with exact lengths:
len 0x0f /data/etc/cert/
len 0x13 /data/etc/vdlicense
len 0x10 /data/etc/vm.lic
len 0x05 /bin/
len 0x07 /lib/lib
len 0x07 /lib/ld-
/data/etc/vm.lic is the platform license path everyone expects. The same
binary still has name strings for helpers like licVM_SaveLicenseKey and
licVM_GetLicenseKeyFilename (present as C strings, not as lea'd log
messages). vdlicense sits right next to vm.lic in the compare list.
Putting those next to cert and loader prefixes is consistent with treating
them as privileged filesystem objects, not random user files.
Trial and FortiCare go over the network
At 0x1c47650, if nobody handed in an override host, the function copies a
default:
directregistration.fortinet.com
Two lea sites in the same function. Trial is not a local file invent either. The method name shows up as:
FortiGlobal.RequestTrialLicenseRequest
xref at 0x1c44357, in a region that formats identity bytes from a caller
buffer into the request. Flex-related names
(license-token, GetFlexVMLicenseKeyFile style strings) live in the same
surface. The shape is the same every time: client builds a named request,
Fortinet issues a document, client runs that document through the header and
stage path above.
A bad key is not just a printf
Invalid license key. lives at 0x3c7b0e, with a lea at
0x5998a4a inside a function that starts around 0x59988b0. In the same
function you get:
/etc/cert/self.crt
/etc/cert/self.key
mov edi, 0x46470002 ; FG-tagged constant
; writes into a context struct at +0x80c, +0x81f, +0x834, +0x9b, ...
So the failure path touches self-signed cert material and a FortiGate-tagged
context. Early return is 0xffffffc8 (-56) when edi <= 0. Whatever
you think of the product, this is not a log-only noop.
What I will stand behind, and what I will not
From disassembly alone I will claim the prefixes, the fopen/fwrite/fclose of
/etc/.cloudinit.lic, the open-fail log, the eleven lea sites on that
path, the install-success string and reboot reason 2, the path special cases
for vm.lic / vdlicense, the default FortiCare host, the trial method
name, and the invalid-key side effects.
I will not claim I have pinned the exact RSA/CMS/EVP routine that verifies the
PEM body, or the precise UUID compare operators inside bind, to a single
basic block off the header function. The binary does contain the usual
suspects in the same neighborhood (RSA_public_decrypt,
EVP_DigestVerify, PEM_read, PKCS7, CMS_*,
gpVerifyPkg, licVM_SaveLicenseKey, execute vm-license). That
establishes capability. It does not, by itself, give you a call graph edge you
can put in a paper without a dynamic trace or more work on the GOT. I am fine
leaving that as unfinished.
I am also not generating a document that passes integrity and bind for a serial, and I am not writing a bypass. The research value here is the map: accepted formats, privileged paths, hosts, reboot reason codes, failure side effects. That is enough for disclosure notes and interop documentation. It is deliberately not enough for a keygen.
The path in one place
buffer + len
strip whitespace
"LICENSE-TOKEN:" (14) -> log token
"-----BEGIN FGT VM LICENSE-----" (30) -> log license
|
v
*flag = 1
fopen /etc/.cloudinit.lic "w"
fail -> "MIME failed to open file to write"
ok -> fwrite whole buffer; fclose
|
v
install success -> reboot reason 2
"Rebooting to install VM license"
long-term store: /data/etc/vm.lic (and vdlicense)
online title: directregistration.fortinet.com
trial RPC name: FortiGlobal.RequestTrialLicenseRequest
fail path: "Invalid license key." + cert/context mutations
Addresses for anyone replaying this
binary /bin/init FGT_VM64_KVM 8.0.0 ~106MB stripped PIE
header 0x1c47cd0
write 0x1c47e0a fopen / 0x1c47e55 fwrite / 0x1c47e63 fclose
fail open 0x1c47ec6
success 0x1c45662 (reason 2)
reboot 0x1c45f78
paths 0x56e4150 (vm.lic @ 0x6041c4)
forticare 0x1c47650 (directregistration.fortinet.com)
trial 0x1c44357 (RequestTrialLicenseRequest)
invalid 0x59988b0 / 0x5998a4a
cloudinit 0x52ee4e (11 lea xrefs)