FLEXlm latest information by CrackZ

20/11/04 - Due to a legitimate request from WISE
Software (a developer of one of the programs featured in this
document), the programs name and license features have been removed
and replaced with *censored* to avoid any legal issues. Note that
this should not detract from the fact that FLEXlm is a lousy protection.

Introduction

This document discusses the reversing of 3 FLEXlm protected
programs. This essay is a classic example of where 1 vendors choosing
not to buy into Macrovisions 'enhanced security' aided in the
discovery of a very simple (and virtually generic) technique to
bypass vendors that had. Throughout this essay I use *
Tip * sections to help you through.

* Tip * - Use lmtools.exe, Utilities
tab (sometimes shipped with your target application or inside
the FLEXlm SDK to identify the version number of the FLEXlm library
used). Files with FLEXlm can also be searched with something like
UltraEdit for the strings 'lmgr.lib' or 'liblmgr.a'.

I started out on this project with several pieces of background
information.

Bitplane Imaris - I knew the vendor keys, seeds and
feature names all from v3.1/2 (which used FLEXlm v7.0e) and none
of the newer FLEXlm features, I recommend downloading the files
from the Bitplane WWW archive here.

*censored* v13.0 - In my possession was a patch
which apparently deprotected the program, initial analysis of
the changes confirmed to me that the FLEXlm routines had not been
patched, instead higher level checks had been forced.

SDS 2 v6.318 - In my possession was a 'warez released'
crack and a (presumed invalid) FLEXlm license. I knew from an
end-user that SDS had switched to using a new style FLEXlm SIGN=
license several versions before.

I started out with Bitplane Imaris, being that I knew the vendor
keys, seeds and license file format, hoping this would make it
easier to identify everything that was going on. I constructed
a dummy license like below (based on the format I knew in the
previous version, also assuming the vendor name 'bitplane' had
not changed, since a daemon of the same name ships still with
the program this seems like a good assumption) :

In the past with Imaris, a fake license had always given a
friendly message box complete with the feature name the program
desired and the FLEXlm error code, no longer is this the case,
instead we are just booted to demo mode. We need to find _lc_checkout().

Finding _lc_checkout()

Finding _lc_checkout() (when you know how) is actually very
simple, however I hacked about inside my targets inside SoftICE
instead of realising the one place it had to be and would be identified
by IDA, lmgr.lib, there we go good developer link in our lmgr.lib
and you too can tell everyone where your checkout routine is as
well. IDA does a really fantastic job of decompiling lmgr.lib
(be sure to select lm_ckout.obj for the object since there are
more than 60 packed inside, you can examine the others later if
you are so inclined). Inside you find the string reference 'lm_ckout.c',
this cross references pretty much exactly (right down to the offsets
of each part of the routine) with the checkout code found in all
3 of my targets. From here its a short step to identifying the
real call doing the work _l_checkout().

* Tip * Note you can generate
signatures from lmgr.lib using IDA's FLAIR tools, however the
signatures are not without faults and often miss key functions,
_l_sg() will almost certainly be missed due to its similarity
to the redundant _l_svk() which GlobeTrotter leave in for obfuscation.
I recommend using my 'lmgr.lib in IDA' approach.

From here we get the unsurprising version number (4.0) and
the feature name ImarisBase as well as a job structure. Looking
beneath _l_checkout() (inside the disassembly of lmgr.lib) we
can quickly trace down an old friend, _l_sg(). Currently _l_checkout()
refuses our fake license with the error -8 (the infamous LM_BADCODE).
Using Hiew or any other good hex editor we do a search for F8
FF FF FF (-8 in reverse byte order) to see likely places this
is set.

As it turns out we end up with 3 locations; in Imaris these
are 5F7282, 5F77D7 & 5F8AE5, the first 2 are referenced inside
_l_good_lic_key(), the latter inside _l_ckout_ok(), we could patch
these but for now lets have a trace through _l_checkout(), the
initial order of events is as follows :

Plugging these values into calcseed.exe we successfully derive
0x0A192837 (seed 1) & 0xF0E1D2C3 (seed 2), these match the
seeds from the previous version. Generating vendor keys using
an updated version of lmkg.exe (courtesy of good friend Nolan
Blender or using lmv8gen.exe (found somewhere on my FLEXlm page))
we can generate licenses (note that you'll need to insert defines
for ENCRYPTION_SEED1 & ENCRYPTION_SEED2 or copy them from
an older lm_code.h), I also found in later SDK's LM_SEED's must
also be defined, although they won't be used if you set LM_STRENGTH
appropriately.

Bitplane will now check this license out successfully, this
indicates that it does not use any of the new enhanced FLEXlm
features even though all of the routines are not compiled out,
you can now freely generate licenses for the remainder of the
features, accessible from the very convenient license option.

A Snag

FLEXlm couldn't be this easy?, surely?, lets try the technique
above with *censored*.....As it turns out *censored*
tries to check out 7 features :

I was informed each of these corresponds to an edition type
of *censored*, so only 1 feature should actually be checked
out successfully. I'd generated my license for *censored*
and _l_checkout() rejected it with error -8 (LM_BADCODE), the
other features failed with -5 (LM_NOFEATURE), no real surprise
there, crucially we now know there must therefore be something
else beneath _l_checkout() that determines that the license is
invalid.

I was pretty confident that the reason the license was rejected
was due to the new ECC security, since this has been well implemented
(Certicom did their job correctly), there is no hope of finding
the real LM_SEED's. A brute force attack would be 2^96, inpracticle
to say the least (and rendered further so by the speed of the
authentication code). There are now 2 things to do, first we can
probably tease from the FLEXlm library the -8 error (0xFFFFFFF8)
and find where its set (like Imaris, a hex editor will do for
finding those places in *censored*), as it turns out there
are 6 address candidates, simple testing would quickly find the
correct one. Or, secondly, we can just trace on from _l_sg() with
*censored*.

Tracing onwards I simply compared the code flow from Imaris
with that of *censored*, both targets next call _l_ckout_crypt()
and code flow differs soon after a call to _real_crypt(), modifying
execution at this level produces a null pointer error. With Imaris
code flow is directed towards a familiar side-by-side comparison
of the real license with the one obtained from the license file,
this implies that _real_crypt() and routines below must be deciding
whether to perform the old style checkout or the new style authentication.
Lets look at the piece of code that controls this (from Imaris)
:

The value of EDX at 5F9B56 controls which type of license 'check
out' will be performed, with Imaris the value was 0x1048C0 and
with *censored* 0x48C0, if we now modify this value in
a live debugging session, *censored* will check out using
the old style FLEXlm encryption and we'll be licensed successfully.
The next step is to see where the FLEXlm routines set this flag.
With a simple bpm we can backtrace our value being set using an
'or ecx, 100000h' instruction inside _l_good_lic_key(), this is
referenced back from another flag checked just after _l_sg().

So if we've got a 0 here we perform the old checkout, we could
now just settle for patching here but lets backtrace yet again.
This results in 2 locations that write 1 to our [ecx+51Ch] flag,
inside *censored* these are 6AF454 (at the end of _l_init())
and 6B219C (at the end of _lc_new_job()).

Flag 1

6AF454 - Set to 1 using a static ADD ECX, 1 instruction at
6AF426, this could be patched to ADD ECX, 0 for a 1 byte change
however Imaris has the same reference so there must be a variation
in code path. By comparing the flow between Imaris and *censored*,
we discover the following 'switching' code :

This is the real switch we have to backtrace and its set deep
inside the _l_buf_36() (routine described below),
the only way I reliably found to locate where the static value
is stored is as follows.

i). Breakpoint the _l_buf_36() routine, do d *(esp+8) to display
in the data window the pointer to the vendor code structure, in
some instances you may need to pagein this address via SoftICE.

ii). Set a bpm w on [vendor code structure + 3Ch] and monitor
writes, anything other than zero should yield the static location
of the data being written there (note that you'll probably get
3 or so breaks on access before finding the right one), once we've
located the correct place we can make a small patch of the static
data.

This requires a patch, in Imaris this flag is AND'd with 0,
*censored* sets it to 1 & SDS to 2. If we make the
necessary changes *censored* will launch and license successfully.
I've developed a 'muster' which you might care to print and apply
to your next FLEXlm target. Lets apply my technique to SDS 2 v6.318.
(Please note that this method should not be totally relied upon
as a quick substitute to commenting your IDA database using lmgr.lib),
the screenshots below are taken from a fully commented .idb.

* Tip * It can be worthwhile loading
lm_crstr.obj in IDA to aid commenting of your target idb.

1. Locate _l_sg().

This seems to be trivial by searching for the hex constant
6F7330B8h (unsigned long x = 0x6f7330b8; /*- v8.x */) used at
the start of the routine, (note that there are 2 references to
this constant and it is the first one you find which is _l_sg(),
the other is inside _l_svk() purely for obfuscation). If you
feel so inclined you can label dword_161DEC4 as _l_n36_buff.

_l_sg = sub_DBFD0A

2. Locate _l_init().

_l_sg() should have 6 cross references. By quickly examing
the code immediately after each call to _l_sg() looking for the
check against the default seeds 0x12345678 & 0x87654321 _l_init()
can be identified.

By examining the other references to _l_sg(), _l_good_lic_key()
will be the only reference that calls _l_sg() twice, in my fully
commented .idb this is clear to see.

_l_good_lic_key = sub_DBE15D

4. _l_good_lic_key() has 9 xrefs of the form 3 xrefs inside
3 functions, the top most of these in the library is _lm_start_real(),
again in a fully commented .idb, this is clear to see.

_lm_start_real() = sub_00DBD112

The only xref to _lm_start_real() should be inside _l_checkout().

_l_checkout() = sub_DBCA0E

From _l_checkout() we can easily deduce _lc_checkout() by
finding the reference with lm_ckout.c at the start of the function
(the other reference to _l_checkout() is inside _l_reconnect()).
_lc_checkout() = sub_DBC940

5. _lc_checkout() has 2 cross references, the 2nd of which
is of the form mov dword ptr [reg32+30h], offset _lc_checkout()
(this is actually part of _InitLmInterface()), by scrolling up
here we can label the routine at [reg32+0Ch] (or the first entry)
as _lc_init(), this yields _lc_new_job() which call's _lc_init()
as sub_DC8610 and _l_n36_buf as dd 0E07100h.

*Tip* You can label all of the
functions from _InitLmInterface() by loading lmgr.lib into IDA
and selecting l_lock_load.obj. In later FLEXlm versions lc_auth_data()
which might well be used by a target can be found in l_check.obj.

6. In a live debugging session we set breakpoints on :

_l_n36_buf = E07100
_l_sg = DBFD3D
_lc_checkout = DBC99D

We construct a dummy license file, like so (note that you
probably ought to be able to ascertain the name of the vendor
daemon) :

FEATURE somefeature dsndata 1.0 permanent uncounted
0 \
HOSTID=ANY

From the break inside _l_n36_buf() we discover that our 2
bad flags are set at address E07112 & E0D4BD. From the break
inside _lc_checkout(), we get the feature name sds2 and the version
number 6.318, lets amend the license accordingly.

FEATURE sds2 dsndata 6.318 permanent uncounted 0 \
HOSTID=ANY

* Tip * Most FLEXlm targets vendor
daemons don't use the _l_n36_buf() configuration flags, this
can make them an easy way to recover seeds using lmgrd -z daemon_name
-c license file, especially if you don't get a break on _l_sg()
in your regular target or _l_checkout() returns EAX=-15.

You should now get a break on _l_sg() and the call to the
_l_n36_buff() decoding routine, this should enable you to recover
the seeds (0x987AC78F & 0xC8D6382B). Using lmv8gen you can
generate vendor keys and now compile lmcrypt.exe. This will enable
you to generate a valid license.

* Tip * By leaving your breakpoint
on _lc_checkout() active you should be able to recover any other
feature names checked out by your target application and thus
generate licenses for them.

With SDS 2 the approach I describe above lets _lc_checkout()
return 0, however the program crashes unexpectedly afterwards;
this was puzzling since the warez released version simply patches
sub_DBC940 (what we know to be _lc_checkout() to XOR EAX, EAX
/ RETN), exactly the same effect as our license. Also the license
with the release was evidently generated using the same key information
I recovered above (as lmcrypt didn't change it). SDS 2 appears
to be an anomalous case amongst FLEXlm targets, I tried 3 others
and couldn't replicate this behaviour, however its useful to know
that we can also use this somehow more 'brutal technique' of patching
_lc_checkout(). Since there is also integrity checking of the
file, patches were also made to static checksum data, this was
implemented separately by the developers obviously to prevent
file tampering.

Conclusion

I hadn't really looked at FLEXlm for quite some time and Macrovision
are to be commended for finally making a protection which is safe
from key generators, however, whilst they continue to support
their legacy mistakes, old style FLEXlm licenses can still be
generated and the application trivially patched to make the licensing
layer accept them, even if my technique was to be broken, it is
still far too easy to patch the _lc_checkout() API directly.

Whats in a SIGN?

The tutorial above describes a mechanism by which we can patch
the FLEXlm licensing layer to use the license key checkout (this
is the oldest and simplest of the FLEXlm validation methods and
was chosen purely for simplicity). The basic SIGN attribute was
added by Globetrotter at around v7.x (it has only 12 chars) and
offers merely an improved algorithm (perhaps more resistance to
brute forcing) and better seed hiding, these are about the only
'enhancements' if one wishes to call them that.

These days a lot of customers have switched to using the CRO
or TRO (counterfeit/tamper resistant options), really the same
thing under a new name. This offers customers the ability to generate
the newest style ECC SIGN licenses with strings starting at 58
chars, as far as I know and can verify there have been no successful
attacks against ECC FLEXlm which enables either complete recovery
of the private key or the LM_SEED's (I do not rule out however
that there is enough processing power somewhere to recover them).

The new SIGN length has resulted in most crackers choosing
to patch _lm_pubkey_verify() and generating a SIGN= license using
their own LM_SEED's. There is however an alternative approach
which involves forcing the licensing layer to do the old style
SIGN=12 chars checkout and it simply involves patching only the
2nd bad flag we found inside _l_n36_buf() (see above). We can
then recover the encryption seeds as before and generate a license
using either the SDK or Lmcryptgui available on this site. The
patch works by simply telling the licensing layer not to get the
address of _lm_pubkey_verify() which is checked shortly after
_l_sg().

Another important thing to note, its easy to verify if your
target will allow the old style SIGN= checkout, after _l_sg()
set a breakpoint on the mangled seeds in the vendor code structure,
if it hits the seeds are being recovered and you can generate
the old style standard SIGN, if not, you'll need a patch; after
a patch a breakpoint on the mangled seeds should hit, just prior
to them being recovered.