When Checking the Box Results in Two Zero Days and Root (CVE-2019-14257 and CVE-2019-14258)

August 21, 2019, Jakob Nelson, Associate, Coalfire Labs

Finding new bugs and exploiting them can be exciting and fun for a penetration tester. I was ecstatic to find my first two zero-days, and I used them to break a system from no access to root. This was a good day for me – but the story behind the story provides some real lessons enterprises can apply to their security programs.

Taking security seriously and continuously improving

As a penetration tester, we run into networks that have serious issues far too often. These are networks that cause a basic vulnerability scan to be filled with high-, medium-, and low-severity findings. Sadly, because time is a finite resource on an engagement, many clients don’t get the full value that they could out of a network pentest. These sorts of networks make it hard for a pentester to dig deep, because we spend a large amount of time just exploiting and reporting on the high volume caught by a vulnerability scan, even though many of the items will not likely lead to compromise during a penetration test.

On this most auspicious day of penetration testing, I had a client who takes security seriously. They had worked with Coalfire for a few years doing their PCI testing, and it was my second year in a row conducting the testing on their behalf. What set this situation apart was that the client implemented the changes from the previous year’s pentest report all the way down to the low-severity SSL findings. This year, the environment looked fairly clean. I had the time to go through and validate all the low-impact findings as usual, and then started looking at what was most interesting next.

Increased value

Because this client’s network was fairly clean, I was able to dig deeper and find an issue that was new and would not be found by any products or typical scanners. Scanners are great at finding known vulnerabilities; however, penetration tests are about showing the impact of vulnerabilities and identifying things the scanners can’t or don’t identify.

During a compliance engagement, the tester is focusing on getting at a target. For PCI-DSS tests, the objective is to get to the credit card data. If the network is clean, patched, and well configured, it can be difficult to get that initial foothold that sends a tester down the path that can lead to a demonstration of risk to the client.

How they were found

A standard part of penetration tests is to look at all the web services. One of these had a strange error: “HTTP ERROR: 405 Problem Accessing /. Reason: HTTP method GET is not supported by this URL.”

This was a get request, so the error stood out. Examining it with Burp, I tried to make a post request. This time the error changed to “Failed to parse XML-RPC request: Premature end of file.”

This was starting to sound like something a mentor here at Coalfire told me: “Sometimes an API will tell you what to do so you can exploit it.” With this in mind, I looked up an example of an XML RPC request. I put that in the body of the POST request, and another error was identified: “No such handler.”

Since I hadn’t looked at the rest of the server to figure out what this service was for, I let Burp Suite Pro run its scanner on the request. I saw that this was a Zenoss server, and there were other ports open.

Looking back at Burp, there was an XML XXE finding. I was staring at the output from /etc/passwd. At this point, I have a Linux system and the ability to read files. At the very least I’m happy to be making progress. Manually looking at the XXE, I viewed some other files and found that I could also list directories.

Once I could list directories through the web service, I was no longer blindly enumerating the OS. This made life much easier, and now I was convinced that I could get a shell on the system. Motivated by this, I kept up the enumeration. Looking at the directory listing of /home I only saw a couple of users. Checking each of the directories revealed that I had access as the Zenoss user. Another juicy tidbit showed up as well. I could see the /home/zenoss/.ssh/ directory, and there was an authorized_keys file. This is the file that allows a user to log in to a system over SSH using a private key. A separate configuration issue was present where, quite fortunately for me, the id_rsa.pub public key was also in the authorized_keys file. I read out the private key and logged in as the Zenoss user over SSH. This was a classic case of “that one guy built that one server and he’s been gone for . . . ”.

This was an anomaly on the network, so it provided more value. I was able to show them something that the usual tools couldn’t.

There was another user that I wanted to get into the home directory and see if there was anything that I could find that would get me into the CDE. This was the network monitoring system after all; who knows what kind of gems a pentester can find on a succulent target like this?

I wanted root. I started with the classic search for SUID and SGID binaries. The usual binaries showed up, but a couple of others were new to me. The one that led to compromise was pyraw. This was apparently part of the Zenoss distribution due to its location. I found that it would execute arbitrary python, but it dropped the SUID privileges before it executed the code I provided. I tried running strings on the binary, and it spit out some interesting python code.

I’m not an experienced reverse engineer, and Ghidra had just come out, so I figured I would give it a shot. I threw the pyraw executable in, and it spit out some decent looking C source. I did a quick search for the python code I had seen when running strings on the binary and started to get my bearings.

I traced the execution back, and it looked like the python code I had seen was run with elevated permissions, but the permissions were dropped after that, and then the user-provided code was run. I consulted my co-worker and generally brilliant researcher Victor Teissler (@VTeissler on Twitter). He confirmed that I was seeing what I thought I was seeing, and we got to work figuring out how to privesc to root.

Victor suggested that we could hijack the site.py library to execute arbitrary code before the privileges were dropped. We did some testing, and about 15 minutes after identifying the vulnerability, we had a working proof of concept that resulted in root permissions. First, we had to create a new directory structure to hold the modified site.py, but the exploit script was short and easy to understand.

That is how I ended up with two CVEs when doing a compliance penetration test. Naturally, I continued testing, finishing out the rest of the network. While the CVEs were great to get for me personally, I was happy that I could provide a solid pentest to the client and help them to shine a light in the dark corners of their environment. I was also pleased to work on a client project where the customer applied solid best practices, enabling the penetration test to provide them with full value.

The CVEs have been released and exploits have been published. We provided Zenoss with more than 90 days for responsible disclosure to fix the vulnerabilities and release a patch. When this post was published, no patch had been released, however, the manual remediation guidance provided by Zenoss is included at the end of this post. I would like to thank the Zenoss security point of contact for being a cordial and pleasant person to work with during the responsible disclosure process.

Responsible Disclosure Timeline

The initial discovery of the vulnerability occurred during the week of 4/15/2019. The client authorized Coalfire to pursue responsible disclosure on 4/22/2019. The first response from Zenoss, Inc., was on 4/24/2019, and Coalfire provided exploitation details and proof of concept code.

Zenoss asked that Coalfire include remediation guidance with this post. The following guidance has been provided by Zenoss, Inc.:

Manually replace "xmlrpc-common-3.0.jar" and "xmlrpc-server-3.0.jar" found at /opt/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.12.1-py2.7.egg/ZenPacks/zenoss/ZenJMX/lib/ with the latest (apache-xmlrpc-3.1.3) jars, found at http://archive.apache.org/dist/ws/xmlrpc/

Note: The file location of ZenPacks (/opt/zenoss/ZenPacks) may vary depending on ZSD version.

Note 2: The file location contains the version number of the ZenJMX ZenPack. In the above steps we use the latest version "ZenPacks.zenoss.ZenJMX-3.12.1-py2.7.egg"

Stopping/disabling the ZenJMX service / daemon

If you are not using the JMX monitoring capabilities provided by this ZenPack then you can safely stop/disable the daemon/service which will result in no longer listening on port 9988
In 4.2.5 remove the zenjmx daemon from $ZENHOME/etc/daemons.txt. Stop/kill the zenjmx daemon process.

In 5.0.0+, use serviced to edit the zenjmx service "Launch" setting to "manual" e.g. "serviced service edit zenjmx" and search the file for "Launch". Then stop the zenjmx service in the Control Center UI.

Remove the ZenJMX service / daemon

In 4.2.5 you can remove the ZenJMX Zenpack in the ZenPacks tab under Settings

In 5.0.0+, follow the current documentation on ZenPack removal. https://help.zenoss.com/zsd/RM/administering-resource-manager/extending-resource-manager-with-zenpacks/removing-a-zenpack

Note: Upgrading the version of ZSD may bring back the ZenJMX ZenPack. If that occurs, repeat the un-install.

The vulnerability is currently being worked on by our engineering team. We expect to release a fix late this year or early next year.

Please note that Zenoss 4.2.5 reached End of Maintenance on 2018-06-30 and the fix will only be included in versions after our newest release (6.4+) and then backported to versions still under maintenance (6.2, 6.3)

Starting with version 5.0.0, the Zenoss Service Dynamics product was rearchitected to move the daemon processes running on the host to individual docker containers called services. Consequently, the privilege escalation will occur within a running docker container. This provides only minimal mitigation as the attacker can then proceed to attack other containers or attempt to escape the container onto the host.

In the meantime, the following work around can be used to mitigate the exposed vulnerability in all versions.

Removal of pyraw

In 4.2.5, pyraw is used by the discovery feature. If you are not using the discovery feature then you can safely remove the binary pyraw.

In 5.0.0+, pyraw is no longer used and can be safely removed.

Exposing pyraw temporarily

When not needing the discovery feature, render the binary unusable
chmod 000 /opt/zenoss/bin/pyraw

If you need to use the discovery feature, Zenoss recommends you add back the needed permissions and then once done, re-render the file unusable chmod 04750 /opt/zenoss/bin/pyraw

Note: In version 5.0.0+ a change to a container can only be made permanent by committing the container snapshot with the change

Note 2: Upgrading the version of ZSD may bring back pyraw and/or the file permissions. If that occurs, repeat the removal / render unusable.

About

Coalfire Certificate Program

A Coalfire Certificate is proof-positive that information technology controls have been independently scanned, assessed or validated in accordance with the highest industry standards. Certificates can be displayed on websites or in print.

Coalfire started in 2001 with a simple idea – cyber threats are increasing, compliance mandates are getting more complicated, and a well-designed cybersecurity program can help fuel your overall success.

Coalfire helps organizations comply with global financial, government, industry and healthcare mandates while helping build the IT infrastructure and security systems that will protect their business from security breaches and data theft. The company is a leading provider of IT advisory services for security in retail, payments, healthcare, financial services, higher education, hospitality, government and utilities.

The Coalfire Board of Directors provides invaluable guidance for the organization and reflects Coalfire’s dedication to achieving success for our customers.

Coalfire’s executive leadership team comprises some of the most knowledgeable professionals in cybersecurity, representing many decades of experience leading and developing teams to outperform in meeting the security challenges of commercial and government clients. With diverse backgrounds in IT systems security, governmental security, compliance, and reducing risk while implementing the latest enabling technologies (such as the Cloud and IoT), our leaders understand the challenges customers face.

With a passion for quality, Coalfire uses a process-driven quality approach to improve the customer experience and deliver unparalleled results.

Created in honor of the late co-founder of Coalfire, the Richard E. Dakin Fund at The Denver Foundation is supporting scholarship programs at several universities for promising college students studying cybersecurity and related fields.

Security is a team game. If your organization values both independence and security, perhaps we should become partners.

The increased need for cyber security has become a common enterprise priority across the globe. However, industry requirements for effective cyber risk management are as distinct as the individual entities under fire. Enterprises and government organizations need more than an off-the-shelf audit to provide an effective threat assessment. They need industry- and organization-specific insights, tools and processes to protect digital assets and ensure compliance.

Coalfire can help cloud service providers prioritize the cyber risks to the company, and find the right cyber risk management and compliance efforts that keeps customer data secure, and helps differentiate products.

“Success” at a government entity looks different than at a commercial organization. Create cybersecurity solutions to support your mission goals with a team that understands your unique requirements.

The financial services industry was built upon security and privacy. As cyber-attacks become more sophisticated, a strong vault and a guard at the door won’t offer any protection against phishing, DDoS attacks and IT infrastructure breaches.

The continuum of care is a concept involving an integrated system of care that guides and tracks patients over time through a comprehensive array of health services spanning all levels of care. Interoperability is the central idea to this care continuum making it possible to have the right information at the right time for the right people to make the right decisions.

Maintaining network and data security in any large organization is a major challenge for information systems departments. However, in the higher education environment, the protection of IT assets and sensitive information must be balanced with the need for ‘openness’ and academic freedom; making this a more difficult and complex task.

When it comes to cyber threats, the hospitality industry is not a friendly place. Hotels and resorts have proven to be a favorite target for cyber criminals who are looking for high transaction volume, large databases and low barriers to entry.

The payments industry is undergoing rapid changes and unfortunately, an increasing risk for data breaches. Cyber criminals are growing increasingly businesslike, and payments leaders need to move quickly to cover their cyber risk.

The food and beverage industry is under attack from cyber criminals intent on stealing payment information. The food and beverage industry makes up the highest percentage of breach investigations, at nearly 73 percent, according to Visa.

The global retail industry has become the top target for cyber terrorists, and the impact of this onslaught has been staggering to merchants. To secure the complex IT infrastructure of a retail environment, merchants must embrace enterprise-wide cyber risk management practices that reduces risk, minimizes costs and provides security to their customers and their bottom line.

Private enterprises serving government and state agencies need to be upheld to the same information management practices and standards as the organizations they serve. Coalfire has over 16 years of experience helping companies navigate increasing complex governance and risk standards for public institutions and their IT vendors.

Technology innovations are enabling new methods for corporations and governments to operate and driving changes in consumer behavior. The companies delivering these technology products are facilitating business transformation that provides new operating models, increased efficiency and engagement with consumers as businesses seek a competitive advantage.

Cybersecurity has entered the list of the top five concerns for U.S. electric utilities, and with good reason. According to the Department of Homeland Security, attacks on the utilities industry are rising "at an alarming rate."

Cyber risk management, advisory, technology and compliance services. Manage risk and maximize return on investment to prevent data breaches and theft. Coalfire’s solutions are led by a team of industry experts that help enterprise organizations understand a wide range of compliance and risk management initiatives, which enables a consistent cybersecurity framework across the organization.

Expert assessments that provide an accurate understanding of what you are trying to protect, the inherent and residual cyber risk to your enterprise and the maturity of the your security program and underlying controls