Monday, March 20, 2017

The debunkings will continue...

The news of Trump's server making interesting outbound connections caught attention of many security researchers in October 2016 and many of us, nerds, spent at least some time checking IP addresses, domains and looking at the logs.

However, the logs that were kindly shared by Jean Camp brought more questions than answers. For example, we see a bunch of DNS lookups for the A records of MAIL1.TRUMP-EMAIL.COM , but not much more that would support the claims of the secret communications. A number of researchers looked at it and wrote detailed explanations of why it is just a marketing email server, unlikely to be used for clandestine communications, and why the DNS log correlation with the political events seems very circumstantial. The fact that there was not enough information to make a final conclusion allowed that story to simmer until it flared up again in March, 2017 when Trump made allegations about the Trump tower wiretapping.

The reason we are raising this story from the dead again is to provide additional evidence that the "Trump's server" used to be a marketing email server. We also offer our possible explanations to some of the events and question some premises and assumptions of the original disclosure. We may repeat a lot of good points made by Krypt3ia and Errata Security in order to turn this collection of events into to a more cohesive narrative.

Disclaimer: We analyzed the email messages, the leaked logs, public DNS and IP information. We seek technical correctness and will welcome additional data. Conclusions that were made in this article were not driven by political opinions, we did not vote for Trump and do not have any interests in Alfa Bank. If you find technical or factual errors, please let us know in comments or email.

Examples of emails sent from the server in 2011-2016

The samples of email messages below show that the server was used for sending newsletter offers for at least 5 years and likely longer. We have a number of samples and mail logs of spam messages dated March 7, 2011-February 29, 2016. Please see below the email screenshots, list of subjects along with the partial string from each header, headers and screenshots of two messages.

Examples of marketing emails from March 7, 2011 to Feb 29.2016

Variety of emails received from MAIL1.TRUMP-EMAIL.COM2011-2016

First message sample
available date: Mar.7, 2011

Last message available dated:
Feb. 29, 2016

Raw email header of last email
avail. Feb. 29, 2016

Before we go into technical details, here is a list of points in a Q&A form.

Q: Did Trump or his associates communicate with the Russian bank via his server?

A: The messages were sent from one DNS server (Alfa Bank) to another DNS server (Cendyn) asking for the IP address of mail1.trump-email.com. The leaked logs that contain these queries do not give enough data to substantiate such claims.

Listrak Conf. Booth

Q:Does that prove <insert anything related to Trump's claims about wiretapping, Russian computer hacking, Russian ties, etc? A: Despite various wild theories, the events described in the original post and the logs have no relation to the Trump's claims that his wires were crossed tapped. This post does not prove that he "has" or "has no" other connections to Russia or anything about Russian hacking or other foreign entities. "The server " has never been the primary reason for the listed allegations.Q:Can that server in Trump tower be possibly bugged by Obama, the British or hacked by
someone who wants to accuse the president in communications with Russia.A:"That server" is the same server we are talking about and it is not in the Trump tower. The server mail1.trump-email.com 66.216.133.29 was located in the Lititz, PA datacenter of a reputable digital marketing company Listrak contracted by Cendyn. Currently, the server with the IP address 66.216.133.29 is still in the datacenter and will be recycled for other needs. MAIL1.TRUMP-EMAIL.COM is pointing to a GoDaddy domain parking IP address (no actual server). TRUMP1.CONTACT-CLIENT.COM is still pointing to 66.216.133.29.Q:So, what happened then?A:

Mail flow before March 2016

From at least 2011 to March, 2016, Alfa Bank employees and many other recipients around the world received so called marketing emails (aka spam) from Trump Organization sent from MAIL1.TRUMP-EMAIL.COM. Digital marketing companies Cendyn and Listrak who provided the mailing services used their mail and DNS servers in Pennsylvania and Florida. Cendyn registered that domain for the Trump Organization, which already owns over 3500 domains (src. Domaintools). None of the servers were ever physically in the Trump's Tower.

In March 2016, Trump Organization changed the vendor and stopped using Cendyn's services. Since at least May 4, 2016 (earliest date in the logs), at least some of the companies that we believe received Trump spam in the past continued to make DNS lookup requests for IP address of MAIL1.TRUMP-EMAIL.COM. Alfa Bank and Spectrum Health made many more lookups than others. Other IP addresses belong to a quarantine appliance run by an Anti-Spam cloud filtering provider MailCleaner, eCommerce Corporation mail service, Australian company called Shiftcare (software for home care services), Hostedmail.com, DNS server for small business hosting.
They did not directly connect to MAIL1.TRUMP-EMAIL.COM. In addition, it is believed many other companies were seen by various ISP providers doing similar lookups.

DNS Lookups as seen in the logs until September 23, 2016
The circle "Logs that leaked" shows the conversation content
in the logs. This does not imply that the logs were stolen from
Cendyn's ns[1-3].cdcservices.com as this is not the only
source where they could come from. There are concerns about the source of the logs

The logs span the period from May 4, 2016 to Sept. 23 2016 and contain DNS lookup requests made by Alfa Bank's DNS servers and the companies mentioned. Some IP addresses in the logs are not actual DNS servers but gateway IP addresses for those networks.

Alfa Bank and other companies made daily (1-70+ a day) queries / DNS lookups asking for the IP address of MAIL1.TRUMP-EMAIL.COM that sent those spam emails, as seen in the email headers below.

These DNS lookups for domains and IPs inside messages that are not incoming but already delivered may be caused by any of the following: misconfigurations or glitches on email and mail filtering services, security appliances performing automated or search triggered lookups (DNS lookups on existing blacklists etc.), anti-spam mailbox store rescans, and endpoint level anti-spam products.
For example, anti-spam systems are known to try to resolve and lookup every IP address and DNS name in the email message header, which can sometimes trigger unintended unsubscribe actions. For example, IETF Request for Comments RFC8058 "Signaling One-Click Functionality for List Email Headers" released in Jan. 2017 specifies rules for the broadcast marketing companies to help cope with unintended unsubscribe actions caused by anti-spam systems.

The exact reason for lookups can be only guessed, since only the companies themselves would be able to tell which of their systems caused it, assuming enough associated internal logs were saved to correlate. The reasons could be different for all companies - some of them made lookups for LINKS.TRUMP-EMAIL.COM as all URLs in the emails used that subdomain. You can see example of those links in the header examples and in these Tweetbot posts.

On September 21, Alfa bank was reached for comments about the logs, which caused the number of lookups and their variety skyrocket as their security team started the investigation.

The author of the original disclosure states that the lookup errors started on September 22, 2016 because Cendyn removed the DNS zone for mail1.trump-email.com from ns1 and ns3.cdcservices.com. These were two Cendyn DNS servers in Ft.Lauderdale, FL. The second, ns2.cdcservices.com, is located in Boca Raton, Fl. Considering that Trump was not their client since March 2016, the hasty and belated removal was either co-incidence or reaction to being notified and realizing that the zone, or domain should have been removed long ago.

Passive DNS logs show only when the subdomain is first seen, not when created or assigned. The fact that TRUMP1.CONTACT-CLIENT.COM showed up in the passive DNS logs on Sept. 30 could be attributed to testing if the server is reachable using the new (or existing) freebie domain (Cendyn creates them for each customer), especially if they indeed still used it for CRM software that "CenDyn provides to the Trump Organization".

On September 27, Alfa Bank made a DNS request for the new TRUMP1.CONTACT-CLIENT.COM. Considering, that at that time the computer security department was performing investigation of the claims, it is not surprising. The domain was likely coaxed by various lookups and queries performed by their IT department. For example, you can see sudden appearance of queries for MAIL.TRUMP-EMAIL.COM (Mail without 1) from Alfa Bank 217.12.96.15 on September 22, which can be attributed to the investigation too.

Q: Did you see Alfa Bank's statement on March 17, 2017 that they were hacked and thus those connections to the Trump's server were made by hackers to look like Alfa Bank did it. (src. Circa)

A: It is possible to send a lot of DNS traffic, or other requests and perform an attack (DDoS or other) without actually "hacking" the victim. They were not "hacked" in this particular case, in the sense of someone infiltrating their network, nor do they say that. Alfa Bank received a lot of DNS queries and DNS replies to spoofed requests after the news came out. We are sure that many of those requests are the result of various researchers trying things. 1340 DNS queries is not a large number. And no, we didn't do it.

While it is possible to spoof DNS requests and make them look like they came from Alfa Bank, it is not a convincing theory for events before September 23, 2016. From the logs provided, there were 7 other companies seen over the course of 4.5 months doing the same type of lookups.

We think the DNS spoofing attacks that happened in 2017 as reported by Alfa Bank were spurred by all the news about the mysterious DNS communications channel used by Trump and Russians. Many researchers and hackers would try all kinds of queries to elicit server responses and some possibly tried to make it look like the 'secret' communications continue. The evidence of those research efforts can be seen on the Farsight pDNS search for TRUMP-EMAIL.COM, where some recent entries include 'new' subdomains like you see below. The cause for these is the fact that TRUMP-EMAIL.COM uses a wildcard DNS record, so queries for its random subdomains will resolve successfully and show up in the database (if seen by any pDNS sensors).

last seen

2017-03-17 21:18:09 -0000

thej35t3rpwns.trump-email.com.

A

184.168.221.46

We should note that Cendyn transferred the TRUMP-EMAIL.COM domain to Trump Organization on March 8, 2017, thus all attempts to resolve the domain since that date would return the IP address of GoDaddy domain parking server.

Claims and Counterclaims:

Before May 2016:

Claim 1:Trump campaign press secretary Hope Hicks: “First of all, it’s not a secret server. The email server, set up for marketing purposes and operated by a third-party, has not been used since 2010. The current traffic on the server from Alphabank’s [sic] IP address is regular DNS server traffic – not email traffic.” (Src. Guardian)

Response 1:

As you see in the last message header As you see in the last message header here, the last message was received from that server MAIL1.TRUMP-EMAIL.COM on the IP address 66.216.133.29, the last message was received from that server MAIL1.TRUMP-EMAIL.COM on 66.216.133.29 on February 29, 2016.(src. DeepEnd Research)

This tweetbot was still posting links from Trump Hotel's marketing emails in February with the last one on Feb. 29, 2016 (src. Twitter)

Cendyn acknowledged that the last marketing email it delivered for Trump's corporation was sent in March 2016" (Src. CNN)

May 2016 - September 23, 2016. Logs and log time period:

As it was already pointed out by many, the sever is located in a server farm that belongs to a hosting company and is one of many used by Cendyn (the company used by Trump Organization for mailing services). It is not more hidden than any server of any cloud services provider.

"The RData for this host were served by the Central Dynamics (CC-801) authority resolvers ns{1,2,3}.cdcservices.com."(src. GDD) < Central Dynamics (Cendyn) maintained DNS records for the domain just like they do for other customers and other domains they registered and maintained for Trump were:

"The scientists theorized that the Trump and Alfa Bank servers had a secretive relationship after testing the behavior of mail1.trump-email.com using sites like Pingability. When they attempted to ping the site, they received the message “521 lvpmta14.lstrk.net does not accept mail from you.” (src. LJean.com)

Response 3:

Robert Graham from Errata Security already explained that this is how Listrak configures email marketing servers. (src. Errata Security).

As for "outside of SPF range", Cendyn's SPF records for TRUMP-EMAIL.COM and CONTACT-CLIENT.COM (envelope sender) included MX, which is the same for all their domains - incoming.cdcservices.com . MX entry in SPF records makes it unnecessary to list all the IPs. The only downside and limitation about using MX entry instead of IPs is that it works only for servers that only do sending, not receiving - which is what that server was built to do. See the header here and note that Received-SPF: pass

"Strange combined domain name (mail.trump-email.com.moscow.alfaintra.net) seen in Alfa Bank logs mean "Moscow division of the INTERNAL Alfa Bank network most definitely has purposeful communications with a hostname registered by the Trump Organization. "(src. LJean.com)

Claim 8:CenDyn stated the reason they recreated a trump1.contact-client.com hostname pointing to this same IP address was for the Trump Organization to use the CRM software CenDyn provides to the Trump Organization." (src. LJean.com)

Response 8:

It is possible they needed to use TRUMP1.CONTACT-CLIENT.COM after they removed EMAIL1.TRUMP-EMAIL.COM We do not know when it happened. We know when TRUMP1.CONTACT-CLIENT.COM showed up in the DNS logs and passive DNS database, but it is not a direct evidence of the creation and assignment date.

Claim 9:"CenDyn states that their servers are not dedicated to a specific client. Yet the Internet-Wide Scan Data Repository (scans.io) data show that the hostname mail1.Trump-Email.com has been stable since at least 2013. It did not change for three years, then did change on on 23 September 2016. At the time of this writing, 2 October 2016, no other hostname has pointed to this IP 66.216.133.29:just trump1.contact-client.com and mail1.trump-email.com. So this IP address is associated with only that server. " (src. LJean.com)

Response 9:

This is correct. It appears that 66.216.133.29 was dedicated to Trump Organization. PTR records are still not updated.

Claim 10:DNS was possibly used to conceal data and commands within DNS traffic using the technique called DNS tunneling (as many ask on Twitter)

Response 10:

It does not seem to be the case, if based on the provided logs. They show "A" records only. "A" records are used for transferring only IP addresses. DNS tunneling would be possible if those were "TXT" or "CNAME" type records that can hold arbitrary non-formatted text strings. (Tunneling Data and Commands Over DNS to Bypass Firewalls by Lenny Zeltser)

September 21, 2016 - October 5, 2016 As requests for comments were sent to Alfa Bank

Claim 11:"When a reporter called Alfa Bank for comment on September 21, the zone for mail1.trump-email.com was removed from ns1 and ns3.cdcservices.com causing RCODE=2 (Server Failure), and ns2 returned empty referrals"(src. GDD) "One of the intriguing facts in my original piece was that the Trump server was shut down on Sept. 23, two days after the New York Times made inquiries to Alfa Bank (and a week before the Times reached out to Trump)." (src. Slate)Trump, CenDyn or some other party associated with the domain sought to erase the mail1.Trump-Emal.com host by deleting forward resolution zones. So the domain name was removed from the normal way one would look up a domain. However, the reverse delegation still exists as of 2 November 2016." (src. LJean.com)

Response 11:

The server as machine on 66.216.133.29 in the Listrak datacenter is still up so it was not shut down.
Passive DNS shows that "A" record MAIL1.TRUMP-EMAIL.COM was last seen on 66.216.133.29 on 2016-09-13. Since Trump company 'ditched' Cendyn in March 2016, eventual cleanup of DNS records had to happen - eventually. We don't know if they were contacted regarding the matter on or before September 22, 2016. If they were, it would be a normal knee-jerk reaction to the inquiry.
They removed records only from the Ft. Lauderdale servers (NS1 and NS3) but not NS2 in Boca Raton (different admins?). It was noted by many that they also forgot to remove PTR record for mail1.trump-email.com and it is still pointing to 66.216.133.29 even though A record was finally assigned to GoDaddy domain parking 184.168.221.22 on March 8, 2017 (after transferring domain back to Trump org).

Claim 12:"Alfa Bank knew that Trump renamed his host through ongoing email delivery and HELO/EHLO resolutions, or another channel. Trump and Alfa Bank have since coordinated their move to an office communications channel." (src. GDD)

Response 12:

Not sure what the author means by "an office communications channel". The requests for comments for the Alfa Bank were made on September 21, 2016. On September 27, 2016 the Alfa bank DNS server made a lookup for TRUMP1.CONTACT-CLIENT.COM. Considering that they did their investigation of the claims, it is not unexpected that their security people finally found and queried the other domain associated with the IP.

Claim 13:"The hostname trump1.contact-client.com appeared in the first passive DNS

Over 500 subdomains.
via PassiveTotal pDNS

database three days later, and still has not appeared in some passive collections." (src. GDD)

October 5, 2016 - March 8, 2017 Post-Disclosure

Claim 14:In March 2016, Cendyn said it "transferred back to" Trump's company the mail1.trump-email.com domain. (Src. CNN)

Response 14:

Yes, they did transfer the domain control on 2017-03-08. Since then, MAIL1.TRUMP-EMAIL.COM and all subdomains resolve to 184.168.221.46 - GoDaddy Parking (IP address for domains without associated hosting servers)

Claim 15:Alfa Bank claims that the recent attacks in February and March 2017 are intended to make it look they continue the secret communications with the Trump server.

Response 15:

2017-02-17 According to the Alfa Bank press release on 2017-03-17, on 2017-02-17 computers in USA sent requests to "Trump Organization server" and made it look like it came "from various variants of MOSCow.ALFAintRa.nET", thus the "Trump's server's" replies were sent to Alfa bank.(src. Alfa Bank and Circa).

The press releases often go through several layers of editing which could affect the technical accuracy of the text. For example, here we can assume that by the Trump Organization server they mean Cendyn's DNS server for MAIL1.TRUMP-EMAIL.COM and that server received DNS queries for MAIL1.TRUMP-EMAIL.COM that came from Alfa Bank spoofed IP addresses. DNS servers do not record domain names of incoming requestors, so it is not entirely clear where they saw MOSCow.ALFAintRa.nET. Not questioning the fact of the attack but it is hard to say what happened without actual logs or more technical data.

2017-03-11 and 2017-03-13 According to the Alfa Bank press release on 2017-03-17, on 2017-03-11 and 2017-03-13 their systems received 1340 DNS replies to the queries they did not send for mail.trump-email.com.moscow.alfaintra.net.(src. Alfa Bank and Circa)

Again, it looks like press release is lacking technical accuracy, which is ok.
In general, sending DNS request from spoofed IP addresses (crafted packets) is very easy. Often attackers use nonexistent subdomains to force their recursive DNS server to forward each of their queries to the authoritative DNS server for that domain instead of using cached answers, thus overloading it. DDoS does not seem to be the goal but more like malicious experimenting.

Claim 16:

But experts claim it is <unusual, odd.. etc>

Response 16:

In tech speak, epithets like "odd", "weird", "not normal" do not really mean clandestine or paranormal. These are highly technical terms meant to convey that existing evidence is too limited to allow one extrapolate the possible scenarios. I am not speaking for every comment out there but am suggesting not to jump to conclusions when a nerd calls something "odd".
Robert Graham comments on the experts' claims too (src. Errata Security)

Timeline of events 2007 - 2017

It would be beneficial, I think, to establish a timeline of the events that you see below and we will go over the milestones below.

2017-02-17 According to the Alfa Bank press release on 2017-03-17, computers in USA sent requests to "Trump Organization server" and made it look like it came "from MOSCow.ALFAintRa.nET", thus the "Trump's server's" replies were sent to Alfa bank.(src. Alfa Bank and Circle)

2017-03-04 - 29.133.216.66.in-addr.arpa. PTR for MAIL1.TRUMP-EMAIL.COM last seen on 66.216.133.2 (via dig -x)

2017-03-11 and 2017-03-13 According to the Alfa Bank press release on 2017-03-17, their systems received 1340 DNS replies to the queries they did not send for mail.trump-email.com.moscow.alfaintra.net.(src. Alfa Bank and Circle)