Ξ welcome to cryptostorm's member forums ~ you don't have to be a cryptostorm member to post here ΞΞ any OpenVPN configs found on the forum are likely outdated. For the latest, visit here or GitHub ΞΞ If you're looking for tutorials/guides, check out the new https://cryptostorm.is/#section6 Ξ

Encouraging best practices in the VPN industry via independent, community-certified verification of clean installers and clean basic service operations. Let's reward the good, and make the bad a little bit less tempting 〰 github repo 〰 #cleanVPN

Certificate Trust List is generally used during SSL/TLS handshake when Client Certificate Authentication comes in to picture. During SSL Handshake the server sends the client the list of the distinguished CA names that it supports as a part of Server Hello message. The Client uses this list to draw up a list of client certificates that is issued by any of the CA’s in the list i.e., only those client certificates which are issued by any of the CA’s in the CTL will be populated. Below is an example of how a CTL looks like

- If the DL is corrupt or times out (5 secs. or so), nothing happens and the process is not reproduced unless you restart your browser and open that website again

- If DL succeeds, some validation mechanism checks the SKI and fingerprint of the certificate (I wasn't able to figure out, what exactly happens, but I couldn't just present a different root certificate. Windows wouldn't accept this).

- If validation succeeds, the root cert is installed into the local trusted store

The process can be blocked either by disabling it via GPO (on Windows 8 via Registry Entry) or by pointing DNS for ctldl.windowsupdate.com to 127.0.0.1/blocking requests to ctldl.windowsupdate.com

If you open a website that Windows doesn't have a valid root cert for, that CA/Root cert will be looked up from the list (which is cached localy as far as I understood)

I'm still working to integrate the "Certificate Trust List" into this process, because that's the one that actually gets pulled during (for example) the process of executing the Kebrum installation binaries, above.

From the pcaps, here's what happens (apart from the pulling of the 'authroot.txt' file, which is outlined in full above:

1. A process within the instantiated installer initiates a DNS resolution query via...

GET /msdownload/update/v3/static/trustedr/en/authrootstl.cab HTTP/1.1\r\n

Here's a screenshot of the underlying wireshark'd packet flow:

Which IP address - 23.5.245.163 - is indeed located in an Akamai-controlled block. It's also an IP address that is associated with a mix of (Verisign-affiliated, as Symantic now owns GeoTrust, Thawte, RapidSSL, and Verisign CA businesses) hostnsames that have resolved to it in the last 18 months alone...

VirusTotal's passive DNS only stores address records. The following domains resolved to the given IP address.

An awful lot of nasty stuff has been communicating with that specific IP address recently (although it's of course quite possible that all that badware is simply doing routine background Windows cert-update stuff entirely unrelated to any badness, in which case this exact pattern would appear but would indicate nothing improper on the part of ):

Latest files submitted to VirusTotal that are detected by one or more antivirus solutions and communicate with the IP address provided when executed in a sandboxed environment.

I still don't understand what purpose the complex welter of subhostnames serves in this: wouldn't resolving crl.verisign.com to 23.5.245.163 directly be more secure, less complex, and easier to administer? I say that with an awareness of these things given our use of similar layered hostname-IP resolver pools (our Hostname Assignment Framework), so I'm as much curious as anything else - assuming it's legitimate, then there's some reason they add that extra layer of arbitrary subhostnames into the process flow). Of course, I do understand the benefits of putting akamai in as one lookup layer... sorta. Only not really.

Do you really want to outsource to an amorphous cloud-based CDN the delivery of... CRLs? Or in this case CTLs (certificate trust lists)? These aren't exactly multi-gigabyte streaming video files. They don't change every 5 minutes (do they?), and they aren't subjected to enormous pressure to cut every millisecond from RTT pingtimes (are they?). They're compact, relatively static files that are extremely - extremely! - security-relevant given that they control what root certs are or are not injected silently into the windows trust store locally on countless millions of PCs.

Subvert that process, and you've got one bad-assed exploit avenue. You can now do things with those PCs unimaginable without that subversion... like, ooooh, MiTM all of their https sessions invisibly, without any hint it's going on. At massive scale.

It would seem to me, naive as I am, that Symantec/Verisign/Thawte Corporation would - given that they are a Certificate Authority - want to actually manage the process of controlling and delivering those CRLs and CDLs themselves, in-house, firsthand. Managed closely, with audit trails and accountability. Maybe a hostname like svrintl-t1-aia.verisign.com helps that, via some internal tracking process. Seems alot of complexity just asking to be broken, to my way of thinking - but then again my world includes a different mix than that of the folks at Symantec who likely designed this.

Finally, for this point anyhow, I'm really not sure how one would do an audit trail given that this is Robtex's graphical representation of what happens to get a request from a Windows machine wanting a CRL (version 3, which dates back to 2006... because of course, that makes sense?) and asking the DNS system to give it an IP address to which packets can be sent:

. . .

Anyway, so presently we see that crl.verisign.com resolves via a somewhat complex, multi-step process, to IP address 23.5.245.163.

That's not always been the case; a look back through records shows that these mappings have been noted as in effect at in the past 18 months:

I haven't looked into all of these yet - they should all cleanly WHOIS to either Akamai, or Symantec... right?

I'd have more confidence - and perhaps not include the question mark - if stuff like this didn't show up in my research on this post. First, malware that explictly calls some of these verisign.com CRL hostnames:

Second, this anomaly when checking the WHOIS records for crl.verisign.com (yes, doing WHOIS lookups for subdomains is a bit nonstandard, but it can still provide hints of interesting things):

This, I don't even remember where I screenshotted it tbh, so I'm sticking it here to remind myself (or anyone else curious) to follow up and see what's in the file referenced. That version is also very old, isn't it?

Oh, and here's the "CSL" that comes out of that cab at the "windowsupdate.com" URL (which itself cascades, via a series of DNS zone file records returned in response to the initial resolver request, from CNAME www.download.windowsupdate.com to CNAME www.download.windowsupdate.nsatc.net - both of which are subdomains, not http-based resources... which is not supposed to be possible, is it? - to CNAME download.windowsupdate.com.edgesuite.net to CNAME a767.g.akamai.net which is actually - woohoo! - a real A Record listing IP address of 184.25.56.93, which is the actual IP address that actually provided the below-attached file to the test run we did of this installer - pulled out of the full-payload pcaps gathered during the process) discussed earlier in this thread...

..which, and you'll get used to reading this alot, I've not been smart enough to unpack using any tool I can find (though I'm not a Windows native, so perhaps it'll be easy for others - I hope so!). It's pretty big for a "certificate trust list," in any case. Not a CRL, remember... a trust list.

(incidentally, IP address 184.25.56.93 a little odd in that it's been the answer to DNS resolution requests for domains nextgennbn.gov.sg, abercrombie.com, caranddriver.com, rediff.com, mashable.com, a184-25-56-93.deploy.static.akamaitechnologies.com, and nfl.com... all since December of 2014. That's quite a record! Now Akamai is a CDN, of course, so they can remap IPs to client domains as often as they want, and perhaps they've been fast-fluxing this stuff in the last few months because, ummm... something something. Because of some obvious corporate reason I'm not smart enough to figure out by myself, obviously.)

Anyway, authroot.txt & authroot.stl both come down as confirmed by pcaps. They come from deep within Akamai, via a series of CNAME redirects that's pretty impressive. They end up at IP addresses that are impressively ecumenical in the sorts of domains (and subdomains) to which they'll answer.

And it's all related to the authority to inject root certificates into the Windows trust store arbitrarily, without any approval (or even admin access) from the user required. Which is to say: figure a way to hijack, even temporarily or ephemerally, that process and you've got the door open to root cert inserts at an unmatched scale... it makes Superfish seem a drop in the bucket, in comparison.

Has someone managed that trick? Is there evidence of it in these files? I honestly don't know. Personally, I can't rule it out based on the data I've collated thus far... nor can I say I've got the proverbial smoking gun that it has taken place (or is taking place). Additionally, it fails to pass my tech-world sniff test... admittedly not terribly objective, and far from a perfect metric: I'm a bit jaded after years of network security front lines & yes I have been known to see ghosts in the shadows when there's only shadows.

Even so, this is all... so terribly ripe for exploitation. Were it my job to figure out how to inject root certs into alot of Win machines without causing a fuss, I'd sure as heck be checking this whole complex, unstable, multi-entity, http-encoded mess really closely for any little corners to pry up and use along the way. That smells like ripe ground for exploits to me... and I've had enough time around exploits to have a half-decent gut sense on that sort of thing, albeit far from perfect.

Whatever the case, it's fascinating. It's very instructive, in terms of the layering of DNS and CA-based certification that produces certificates in the local trust store, and thus the magical "green lock in the browser" that means https is secure. Only, of course, it doesn't... not at all.

edited to add: these are recently-enumerated examples of malware that have the hostname "crl.verisign.com" embedded in their post-compile strings somewhere; as noted before, that could simply be a result of them including routine CRL administrative matters in pre-compile code that comes through as binaries and thus pops up as this kind of list (indeed, on the same virustotal page one can see a list of non-flagged binaries that include this URL in strings but haven't raised red flags on virus scanning sites - which may or may not indicate they're clean files, tbh)... but it's something I'd be remiss not to at least note. I've not gone to them and checked each one to see if they'd have a legitimate reason to be calling a verisign CRL hostname as part of their compiled-in duties...

Latest files that are detected by at least one antivirus solution and embed URL pattern strings with the domain provided.

Here's a confirmation from Malware Must Die that the hostname crl.comodoca.com is used to deliver a payload 'EssentialSSLCA.crl' - which then gets installed into the trust store, which then... it's quite a chain, isn't it?

This is a step in the process of the functioning of the Upatre trojan downloader - which in turn was (is?) a part of the P2P/Gameover family of botnet agents. So that fills in one more piece.

added: take a single step deeper into the P2P/Gameover botnet forensics and you run into this:

Well, how very interesting. Black Lotus is very, very closely affiliated with a particular VPN company most folks will know by name. What are the chances, eh? All those ISPs all around the world, and two wires cross right down in Texas under the name Black Lotus. Quite a coincidence, indeed.

Here's a search query on the "social" side of TechNet that turns up a vast pool of questions relating to this hostname; I've only just begun reading, but wanted to post out the full search so others have easy access meanwhile, as well:

List of predefined items that have been signed by a trusted entity; may consist of a list of filenames or a list of certificates; each item in the list has been approved by the signing entity.

Certificate trust lists (CTLs) are used by Microsoft IIS to store trusted websites and other addresses that require a secure connection.

I am limited... I use Windows 7 Home Premium, if I had Professional or higher, I could go further! Sorry PJ.Stopping short of installing Windows Server 2008 RC2, or upgrading to W7 Professional so I can make use of Group Policy Editor (one can be custom-injected into Home Premium, but its reach compared to Professional is quite limited), I'm down for the count mate. From what I saw, there were a hell of a lot of certificates in there, all languages...

Ok, so if the CTL is genuine it'll have a bunch of certificates in it that Windows can grab and assume are good/trusted... so merely seeing a bunch of certs in there isn't an automatic red flag.

However, from what little I can see in those screencaps, they don't look like the trusted root certs that are legit. Edited: looked again, aaaah... that's the cert that signed the CTL itself. Yikes. That's a fishycert, for sure. Not a good sign...

Can you dump the certs out of the file? They'll be in PEM or DER format, afaik, or worst-case Windows versions of them which are easy to convert.

And yes, those IPs all resolve to Akamai blocks, which again is at least surface-level legitimate: Microsoft does use Akamai's CDN to distribute some (or all?) Windows Update files... which as I said above, seems unwise to me, but what do I know of such things as managing Windows update production requirements? Not much, honestly.

That said, those IPs are the destination of resolution for some decidedly non-Windows and most likely non-legitimate domain names in time windows right up against when they show up as "legitimate" resolver answers for this windowsupdate hostname... and those IPs sure do end up serving alot of verified malware in those close time windows too.

My working hypothesis right now, which may be total bunk, is that there's a trick using AAAA/IP6 DNS lookups that enables this redirect trick. It's come up a few times in related research, and it makes sense: IP6 resolver pathways are preferred by most modern browsers, so if you can jump the AAAA records and get traffic headed to your machine, you're in good shape.

Note that all of the analysis in this thread is solely IP4-based. That's myopic. What's happening in IP6-space? I'm running off machines - indeed, the entire cryptostorm network - that's got IP6 hard-disabled, so I'm seeing at best a partial picture here. Most of the world will be IP6-active, and that presents some new angles to consider.

http://www.download.windowsupdate.com is a dodgy one... more so now than ever before due to the release of Windows 10. The long list of DNS addresses that Windows calls out to also contains the above address. I doubt I will go to W10 any time soon; I would much rather stick with W7 or just swap to Linux (after I buy some proper equipment like a 2nd rig and a flashable router).

My testing on W7 did silence a hell of a lot of DNS callouts, but there was one that I could not... yep, you guessed it; the one mentioned above. I included methods such as:

hexing out hardcoded addresses in DNSapi.dll for both Windows/system x86 and x64 folders,

disabling a buttload of Scheduled Tasks and Windows Services (watch out for Software Protection and SPP Notification - if set to disabled or manual will turn your legit Windows install into a pirated one),

HOST file population,

converting callout address to CIDR and inputting them into outbound block rules for the 3 Windows Services mentioned below,

among others...

Consider the above a lesson in idiocy; trying to circumvent callouts on the system you are using by installing apps and mods and tweaks on said system while trying to silence it. Ummmm, dickhead much? Mind you, it was for educational purposes so yeah, I don't mind taking the hit to pride. I managed to track down the callouts to 1 of 3 services (Network Location Awareness, Cryptographic Services, DNS Cache). I've had trouble disabling DNS Cache while maintaining a live browsing experience, so kept it on. Disabling NLA kills the Internet and I doubt I want to much around with Crypto stuff...

The only band-aid solution I could get to work was to use a program called Acryclic DNS Proxy, which provides wildcard support for HOSTS file. So in laymens terms, to nerf MS and Windows, simply attach *microsoft*, *windowsupdate* and *msftncsi* (this one pings DNS for net condition status) to the Acrylic HOSTS file and it's done. This can be extended to other stuff like *google* or 1e100.net, and *facebook* etc...

marzametal wrote:http://www.download.windowsupdate.com is a dodgy one... more so now than ever before due to the release of Windows 10. The long list of DNS addresses that Windows calls out to also contains the above address.

Keeping in mind that this hostname has been formally tied (per above posts) to APT-class malware command-&-control, that's a pretty worrisome thing to see. Though I'm a month late in commenting, do you perhaps have more data on this finding?

I've my own run-in with it recently, when researching a CRL - ocsp.thawte.com - that's showing up in certs but throws weird errors when contacted for testing (more on that in another thread, perhaps, someday... tl;dr is that CRLs are shady beyond belief, and best blocked as we do with CRLblock, network, wide).