Kubernetes privilege escalation and access to sensitive information in OpenShift products and services - CVE-2018-1002105

Public Date: November 26, 2018, 08:44
Updated September 3, 2021, 12:05 - Chinese, Simplified Japanese Korean
Resolved Status
Critical Impact

Insights vulnerability analysis

View exposed systems

A flaw has been detected in kubernetes which allows privilege escalation and access to sensitive information in OpenShift products and services.  This issue has been assigned CVE-2018-1002105 and has a security impact of Critical.

All 3.x versions of OpenShift Container Platform allow for compromise of pods (multiple running container instances) running on a compute node to which a pod is scheduled with normal user privilege.  This access could include access to all secrets, pods, environment  variables, running pod/container processes, and persistent volumes.

Additionally, on  OpenShift Container Platform versions 3.6 and higher, this vulnerability allows cluster-admin level access to any API hosted by an aggregated API server.  This includes the ‘metrics-service’ and ‘servicecatalog’ as possible targets. Cluster-admin level access to the service catalog allows creation of brokered services by an unauthenticated user with escalated privileges in any namespace and on any node. This could allow an attacker to deploy malicious code, or alter existing brokered services.

For OpenShift Dedicated environments, a regular user, with pod exec/attach/portforward permissions, can gain cluster-level administrative privileges on any compute node that can run that pod.  This includes exec access to all running workloads, all current secrets, logs, etc..  

Background Information

The OpenShift API Server is included in all OpenShift Container Platform installations and handles all the administration tasks for the cluster. Administrators and developers don’t usually call the API directly, but use the ‘oc’ binary, which calls the API server.

The API server provides various functionality, including the following oc commands:

  • oc exec
  • oc port-forward
  • oc rsh

It does this by acting as a reverse proxy to the kubelet running on the compute nodes. When connecting to the kubelet in order to fulfill any of the above commands, it opens a websocket connection which connects stdin, stdout, or stderr to the administrator or developer’s original call. For more information, read the Remote Commands section of the OpenShift Container Platform 3.11 architecture guide.

The API Server also acts as a reverse proxy when implementing the API Aggregation feature of Kubernetes. API aggregation enables the installation of additional Application Programming Interfaces (APIs) into the core API Server. Those additional APIs are referred to as API extensions by upstream Kubernetes. API Extensions allow an architect to extend the features of OpenShift Container Platform 3.

Acknowledgements

Red Hat would like to thank the Kubernetes Product Security Team for reporting this issue. Upstream acknowledges Darren Shepherd as the original reporter.

Additional References

Kubernetes Announce List

Video: Kubernetes Privilege Flaw Escallation Explained by Red Hat

Blog: The Kubernetes privilege escalation flaw: Innovation still needs IT security expertise

Blog: Understanding the critical Kubernetes privilege escalation flaw in OpenShift 3


Impacted Products

Red Hat Product Security has rated CVE-2018-1002105 as having a security impact of Critical.

The following Red Hat Product versions are impacted:

  • Red Hat OpenShift Container Platform 3.x
  • Red Hat OpenShift Online
  • Red Hat OpenShift Dedicated

Attack Description and Impact

The OpenShift API server is a component which handles the API requests for OpenShift Container Platform 3. The OpenShift API server had a vulnerability in the ‘upgradeawarehandler’ type which acts as a reverse proxy to the compute nodes and other services added as API Extensions. A flaw in the reverse proxy prevented it from closing connections from the downstream service when there was an error. This allowed the user making the API call to escalate their privileges.

There are 2 ways to use this vulnerability to attack OpenShift Container Platform, OpenShift Online, or OpenShift Dedicated. The first involves abusing pod exec, attach, or portforward privileges granted to a normal user, and the second involves attacking the API extensions feature which provides the service catalog and access to additional features in OpenShift Container Platform 3.6 and later.

If a user has pod exec/attach/portforward privileges for any pod, their privileges can be escalated to cluster-admin, and any API call to a compute node Kubelet api can be achieved. An API call to the Kubelet api allows the user to exec into any container running on the same node as the pod which they have privileges for, including privileged containers which have read/write access to the host filesystem.

The second attack doesn’t require any privileges and exploits the API extension feature used by ‘metrics-server’ and ‘servicecatalog’ in OpenShift Container Platform, OpenShift Online, and Dedicated. It allows an unauthenticated user to gain cluster-admin privileges to any API extension deployed to the cluster including ‘servicecatalog’. Cluster-admin access to ‘servicecatalog’ allows creation of service brokers in any namespace and on any node. 

Diagnose your vulnerability

To check the version of OpenShift container platform installed send an http request to the API server, such as (Note: The URL should be the same as the webconsole URL):

   curl https://openshift.example.com:8443/version/openshift | grep gitVersion

*port is configuration specific

Alternatively if you are logged in with the ‘oc’ command you can check the version with this command:

   oc version

Any versions of OpenShift Container Platform older than those listed below are vulnerable:

  • v3.11.43-1
  • v3.10.72-1
  • v3.9.51-1
  • v3.8.44-1
  • v3.7.72-1
  • v3.6.173.0.140-1
  • v3.5.5.31.80-1
  • v3.4.1.44.57-1
  • v3.3.1.46.45-1
  • v3.2.1.34-2

Take Action

Customers running affected versions of Red Hat products are strongly recommended to update them as soon as errata are available.  

OpenShift Online (Starter, Pro) have been remediated.  OpenShift Dedicated customers should speak with their support contact to confirm the status/schedule for these fixes. 

Updates for Affected Products

ProductPackageAdvisory/Update
OpenShift Container Platform v3.11kubernetesRHSA-2018:3537
OpenShift Container Platform v3.10kubernetesRHSA-2018:3549
OpenShift Container Platform v3.9kubernetesRHSA-2018:2908
OpenShift Container Platform v3.8kubernetesRHSA-2018:3551
OpenShift Container Platform v3.7kubernetesRHSA-2018:2906
OpenShift Container Platform v3.6kubernetesRHSA-2018:3598
OpenShift Container Platform v3.5kubernetesRHSA-2018:3624
OpenShift Container Platform v3.4kubernetesRHSA-2018:3752
OpenShift Container Platform v3.3kubernetesRHSA-2018:3754
OpenShift Container Platform v3.2kubernetesRHSA-2018:3742


Mitigation

Fixes for OpenShift Container Platform versions 3.2 and higher have been shipped to mitigate the flaw.   

Users who update to the latest versions do not need to apply further mitigations. However, if the updates cannot be applied, mitigating the issue is recommended. 

Mitigate the pod exec/attach/port-forward attack (OCP 3.2->3.11)

Change the default admin, and edit roles which have pod attach/exec/port-forward:

$oc edit clusterrole admin

Add the ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, and remove the pod attach/exec/portforward permissions. Example:

apiVersion: v1
kind: ClusterRole
metadata:
   annotations:
      openshift.io/description: A user that has edit rights within the project and can
        change the project's membership.
   creationTimestamp: 2018-11-21T01:34:04Z
   name: admin
   resourceVersion: "148192"
   selfLink: /oapi/v1/clusterroles/admin
   uid: 87dd0306-ed2d-11e8-9456-fa163e65ec84
rules:
- apiGroups:
  - ""
   attributeRestrictions: null
   resources:
  - pods
  - pods/proxy
   verbs:

Do the same for the ‘edit’ role which also has these permissions.

For a single user in a project:

Check the roles assigned to the user. In this case, the admin role is assigned to the quicklab user:
 
$ oc project myproj
$ oc adm policy who-can get pods/exec
Namespace: test
Verb:      get
Resource:  pods/exec
 
Users:  quicklab
        system:admin
        system:serviceaccount:kube-system:generic-garbage-collector
        system:serviceaccount:kube-system:namespace-controller
 
Groups: system:cluster-admins
        system:masters
 
 
$ oc get rolebindings
NAME                    ROLE                    USERS      GROUPS                          SERVICE ACCOUNTS   SUBJECTS
admin           /admin          quicklab                                                      
system:deployers        /system:deployer                                                   deployer           
system:image-builders   /system:image-builder                                              builder            
system:image-pullers    /system:image-puller               system:serviceaccounts:myproj
$ oc export clusterrole admin > admin-mitigate.yml

Edit admin-mitigate.yml by adding a ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, changing the name and removing the pod attach/exec/portforward permissions. Example:

apiVersion: v1
kind: ClusterRole
metadata:
  annotations:
    openshift.io/description: A user that has edit rights within the project and can
      change the project's membership.
    rbac.authorization.kubernetes.io/autoupdate: "false"
  creationTimestamp: null
  name: admin-mitigate
rules:
- apiGroups:
  - ""
  attributeRestrictions: null
  resources:
  - pods
  - pods/proxy
  verbs:
   ...

Create a new role with the new configuration:

$ cat admin-mitigate.yml | oc create -f -

Assign this role to any users who you don’t want to have those permissions:

$ oc adm policy remove-role-from-user admin quicklab
role "admin" removed: "quicklab"
$oc adm policy add-role-to-user admin-mitigate quicklab
role “admin-mitigate’”added: “quicklab”

Mitigating the API Extension attack (OCP 3.6 -> 3.11)

To mitigate the API extension attack until such time that a cluster upgrade to a patched version can be completed, it is possible to delete the affected API Extensions. Please review each services carefully to ensure this will not cause a loss of critical functionality, in which case alternate protections such as network level restrictions to known sources may be more appropriate.

Get a list of services which need to be disabled:

$ oc get apiservices -o=custom-columns=NAME:.metadata.name,SERVICE:.spec.service,STATUS:.status.conditions[0].type,VALUE:.status.conditions[0].status | grep -v '<nil>'

For example, on OCP 3.11 the services might include:

NAMESERVICESTATUSVALUE
v1beta1.servicecatalog.k8s.io map[name:apiserver namespace:kube-service-catalog]AvailableTrue

Disable the services listed in the output, for example:

$ oc delete svc apiserver -n kube-service-catalog

For this example, to re-enable service-catalog run the openshift-service-catalog playbook. Please refer to the product documentation to re-enable other services.

$ ansible-playbook [-i /path/to/inventory]
/usr/share/ansible/openshift-ansible/playbooks/openshift-service-catalog/config.yml

Comments