You can access this page also inside the Remote Desktop by using the icons on the desktop

  • Score
  • Questions and Answers
  • Preview Questions and Answers
  • Exam Tips
answers

CKS Simulator Kubernetes 1.30

https://killer.sh

 

Pre Setup

Once you've gained access to your terminal it might be wise to spend ~1 minute to setup your environment. You could set these:

Vim

The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

More setup suggestions are in the tips section.

 

 

Question 1 | Contexts

 

You have access to multiple clusters from your main terminal through kubectl contexts. Write all context names into /opt/course/1/contexts, one per line.

From the kubeconfig extract the certificate of user restricted@infra-prod and write it decoded to /opt/course/1/cert.

 

Answer:

Maybe the fastest way is just to run:

Or using jsonpath:

The content could then look like:

For the certificate we could just run

And copy it manually. Or we do:

Or even:

 

 

Question 2 | Runtime Security with Falco

 

Use context: kubectl config use-context workload-prod

 

Falco is installed with default configuration on node cluster1-node1. Connect using ssh cluster1-node1. Use it to:

  1. Find a Pod running image nginx which creates unwanted package management processes inside its container.

  2. Find a Pod running image httpd which modifies /etc/passwd.

Save the Falco logs for case 1 under /opt/course/2/falco.log in format:

No other information should be in any line. Collect the logs for at least 30 seconds.

Afterwards remove the threads (both 1 and 2) by scaling the replicas of the Deployments that control the offending Pods down to 0.

 

Answer:

Falco, the open-source cloud-native runtime security project, is the de facto Kubernetes threat detection engine.

NOTE: Other tools you might have to be familar with are sysdig or tracee

 

Use Falco as service

First we can investigate Falco config a little:

This is the default configuration, if we look into falco.yaml we can see:

This means that Falco is writing into syslog, hence we can do:

Yep, quite some action going on in there. Let's investigate the first offending Pod:

First Pod is webapi-6cfddcd6f4-ftxg4 in Namespace team-blue.

Second Pod is rating-service-68cbdf7b7-v2p6g in Namespace team-purple.

 

Eliminate offending Pods

The logs from before should allow us to find and "eliminate" the offending Pods:

 

Use Falco from command line

We can also use Falco directly from command line, but only if the service is disabled:

We can see that rule files are loaded and logs printed afterwards.

 

Create logs in correct format

The task requires us to store logs for "unwanted package management processes" in format time,container-id,container-name,user-name. The output from falco shows entries for "Error Package management process launched" in a default format. Let's find the proper file that contains the rule and change it:

Find the rule which looks like this:

Should be changed into the required format:

For all available fields we can check https://falco.org/docs/rules/supported-fields, which should be allowed to open during the exam.

Next we check the logs in our adjusted format:

This looks much better. Copy&paste the output into file /opt/course/2/falco.log on your main terminal. The content should be cleaned like this:

For a few entries it should be fast to just clean it up manually. If there are larger amounts of entries we could do:

The tool cut will split input into fields using space as the delimiter (-d""). We then only select the 9th field using -f 9.

 

Local falco rules

There is also a file /etc/falco/falco_rules.local.yaml in which we can override existing default rules. This is a much cleaner solution for production. Choose the faster way for you in the exam if nothing is specified in the task.

 

 

Question 3 | Apiserver Security

 

Use context: kubectl config use-context workload-prod

 

You received a list from the DevSecOps team which performed a security investigation of the k8s cluster1 (workload-prod). The list states the following about the apiserver setup:

  • Accessible through a NodePort Service

Change the apiserver setup so that:

  • Only accessible through a ClusterIP Service

 

Answer:

In order to modify the parameters for the apiserver, we first ssh into the master node and check which parameters the apiserver process is running with:

We may notice the following argument:

We can also check the Service and see it's of type NodePort:

The apiserver runs as a static Pod, so we can edit the manifest. But before we do this we also create a copy in case we mess things up:

We should remove the unsecure settings:

Once the changes are made, give the apiserver some time to start up again. Check the apiserver's Pod status and the process parameters:

The apiserver got restarted without the unsecure settings. However, the Service kubernetes will still be of type NodePort:

We need to delete the Service for the changes to take effect:

After a few seconds:

This should satisfy the DevSecOps team.

 

 

Question 4 | Pod Security Standard

 

Use context: kubectl config use-context workload-prod

 

There is Deployment container-host-hacker in Namespace team-red which mounts /run/containerd as a hostPath volume on the Node where it's running. This means that the Pod can access various data about other containers running on the same Node.

To prevent this configure Namespace team-red to enforce the baseline Pod Security Standard. Once completed, delete the Pod of the Deployment mentioned above.

Check the ReplicaSet events and write the event/log lines containing the reason why the Pod isn't recreated into /opt/course/4/logs.

 

Answer:

Making Namespaces use Pod Security Standards works via labels. We can simply edit it:

Now we configure the requested label:

This should already be enough for the default Pod Security Admission Controller to pick up on that change. Let's test it and delete the Pod to see if it'll be recreated or fails, it should fail!

Usually the ReplicaSet of a Deployment would recreate the Pod if deleted, here we see this doesn't happen. Let's check why:

There we go! Finally we write the reason into the requested file so that Mr Scoring will be happy too!

Pod Security Standards can give a great base level of security! But when one finds themselves wanting to deeper adjust the levels like baseline or restricted... this isn't possible and 3rd party solutions like OPA could be looked at.

 

 

Question 5 | CIS Benchmark

 

Use context: kubectl config use-context infra-prod

 

You're ask to evaluate specific settings of cluster2 against the CIS Benchmark recommendations. Use the tool kube-bench which is already installed on the nodes.

Connect using ssh cluster2-controlplane1 and ssh cluster2-node1.

On the master node ensure (correct if necessary) that the CIS recommendations are set for:

  1. The --profiling argument of the kube-controller-manager

  2. The ownership of directory /var/lib/etcd

On the worker node ensure (correct if necessary) that the CIS recommendations are set for:

  1. The permissions of the kubelet configuration /var/lib/kubelet/config.yaml

  2. The --client-ca-file argument of the kubelet

 

Answer:
Number 1

First we ssh into the master node run kube-bench against the master components:

We see some passes, fails and warnings. Let's check the required task (1) of the controller manager:

There we see 1.3.2 which suggests to set --profiling=false, so we obey:

Edit the corresponding line:

We wait for the Pod to restart, then run kube-bench again to check if the problem was solved:

Problem solved and 1.3.2 is passing:

 

Number 2

Next task (2) is to check the ownership of directory /var/lib/etcd, so we first have a look:

Looks like user root and group root. Also possible to check using:

But what has kube-bench to say about this?

To comply we run the following:

This looks better. We run kube-bench again, and make sure test 1.1.12. is passing.

Done.

 

Number 3

To continue with number (3), we'll head to the worker node and ensure that the kubelet configuration file has the minimum necessary permissions as recommended:

Also here some passes, fails and warnings. We check the permission level of the kubelet config file:

777 is highly permissive access level and not recommended by the kube-bench guidelines:

We obey and set the recommended permissions:

And check if test 2.2.10 is passing:

 

Number 4

Finally for number (4), let's check whether --client-ca-file argument for the kubelet is set properly according to kube-bench recommendations:

This looks passing with 4.2.3. The other ones are about the file that the parameter points to and can be ignored here.

To further investigate we run the following command to locate the kubelet config file, and open it:

The clientCAFile points to the location of the certificate, which is correct.

 

 

Question 6 | Verify Platform Binaries

 

(can be solved in any kubectl context)

 

There are four Kubernetes server binaries located at /opt/course/6/binaries. You're provided with the following verified sha512 values for these:

kube-apiserver f417c0555bc0167355589dd1afe23be9bf909bf98312b1025f12015d1b58a1c62c9908c0067a7764fa35efdac7016a9efa8711a44425dd6692906a7c283f032c

kube-controller-manager 60100cc725e91fe1a949e1b2d0474237844b5862556e25c2c655a33boa8225855ec5ee22fa4927e6c46a60d43a7c4403a27268f96fbb726307d1608b44f38a60

kube-proxy 52f9d8ad045f8eee1d689619ef8ceef2d86d50c75a6a332653240d7ba5b2a114aca056d9e513984ade24358c9662714973c1960c62a5cb37dd375631c8a614c6

kubelet 4be40f2440619e990897cf956c32800dc96c2c983bf64519854a3309fa5aa21827991559f9c44595098e27e6f2ee4d64a3fdec6baba8a177881f20e3ec61e26c

Delete those binaries that don't match with the sha512 values above.

 

Answer:

We check the directory:

To generate the sha512 sum of a binary we do:

Looking good, next:

Okay, next:

Also good, and finally:

Catch! Binary kubelet has a different hash!

 

But did we actually compare everything properly before? Let's have a closer look at kube-controller-manager again:

Edit to only have the provided hash and the generated one in one line each:

Looks right at a first glance, but if we do:

This shows they are different, by just one character actually.

To complete the task we do:

 

 

Question 7 | Open Policy Agent

 

Use context: kubectl config use-context infra-prod

 

The Open Policy Agent and Gatekeeper have been installed to, among other things, enforce blacklisting of certain image registries. Alter the existing constraint and/or template to also blacklist images from very-bad-registry.com.

Test it by creating a single Pod using image very-bad-registry.com/image in Namespace default, it shouldn't work.

You can also verify your changes by looking at the existing Deployment untrusted in Namespace default, it uses an image from the new untrusted source. The OPA contraint should throw violation messages for this one.

 

Answer:

We look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:

So we can do:

and then look at the one that is probably about blacklisting images:

It looks like this constraint simply applies the template to all Pods, no arguments passed. So we edit the template:

We simply have to add another line. After editing we try to create a Pod of the bad image:

Nice! After some time we can also see that Pods of the existing Deployment "untrusted" will be listed as violators:

Great, OPA fights bad registries !

 

 

Question 8 | Secure Kubernetes Dashboard

 

Use context: kubectl config use-context workload-prod

 

The Kubernetes Dashboard is installed in Namespace kubernetes-dashboard and is configured to:

  1. Allow users to "skip login"

  2. Allow insecure access (HTTP without authentication)

  3. Allow basic authentication

  4. Allow access from outside the cluster

You are asked to make it more secure by:

  1. Deny users to "skip login"

  2. Deny insecure access, enforce HTTPS (self signed certificates are ok for now)

  3. Add the --auto-generate-certificates argument

  4. Enforce authentication using a token (with possibility to use RBAC)

  5. Allow only cluster internal access

 

Answer:

Head to https://github.com/kubernetes/dashboard/tree/master/docs to find documentation about the dashboard. This link is not on the allowed list of urls during the real exam. This means you should be provided will all information necessary in case of a task like this.

First we have a look in Namespace kubernetes-dashboard:

We can see one running Pod and a NodePort Service exposing it. Let's try to connect to it via a NodePort, we can use IP of any Node:

(your port might be a different)

The dashboard is not secured because it allows unsecure HTTP access without authentication and is exposed externally. It's is loaded with a few parameter making it insecure, let's fix this.

First we create a backup in case we need to undo something:

Then:

The changes to make are :

Next, we'll have to deal with the NodePort Service:

And make the following changes:

Let's confirm the changes, we can do that even without having a browser:

We see that insecure access is disabled and HTTPS works (using a self signed certificate for now). Let's also check the remote access:

(your port might be a different)

Much better.

 

 

Question 9 | AppArmor Profile

 

Use context: kubectl config use-context workload-prod

 

Some containers need to run more secure and restricted. There is an existing AppArmor profile located at /opt/course/9/profile for this.

  1. Install the AppArmor profile on Node cluster1-node1. Connect using ssh cluster1-node1

  2. Add label security=apparmor to the Node

  3. Create a Deployment named apparmor in Namespace default with:

    • One replica of image nginx:1.19.2

    • NodeSelector for security=apparmor

    • Single container named c1 with the AppArmor profile enabled only for this container

    The Pod might not run properly with the profile enabled. Write the logs of the Pod into /opt/course/9/logs so another team can work on getting the application running.

 

Answer:

https://kubernetes.io/docs/tutorials/clusters/apparmor

 

Part 1

First we have a look at the provided profile:

Very simple profile named very-secure which denies all file writes. Next we copy it onto the Node:

And install it:

Verify it has been installed:

There we see among many others the very-secure one, which is the name of the profile specified in /opt/course/9/profile.

 

Part 2

We label the Node:

 

Part 3

Now we can go ahead and create the Deployment which uses the profile.

What's the damage?

This looks alright, the Pod is running on cluster1-node1 because of the nodeSelector. The AppArmor profile simply denies all filesystem writes, but Nginx needs to write into some locations to run, hence the errors.

It looks like our profile is running but we can confirm this as well by inspecting the container:

First we find the Pod by it's name and get the pod-id. Next we use crictl ps -a to also show stopped containers. Then crictl inspect shows that the container is using our AppArmor profile. Notice to be fast between ps and inspect as K8s will restart the Pod periodically when in error state.

To complete the task we write the logs into the required location:

Fixing the errors is the job of another team, lucky us.

 

 

Question 10 | Container Runtime Sandbox gVisor

 

Use context: kubectl config use-context workload-prod

 

Team purple wants to run some of their workloads more secure. Worker node cluster1-node2 has container engine containerd already installed and it's configured to support the runsc/gvisor runtime.

Create a RuntimeClass named gvisor with handler runsc.

Create a Pod that uses the RuntimeClass. The Pod should be in Namespace team-purple, named gvisor-test and of image nginx:1.19.2. Make sure the Pod runs on cluster1-node2.

Write the dmesg output of the successfully started Pod into /opt/course/10/gvisor-test-dmesg.

 

Answer:

We check the nodes and we can see that all are using containerd:

But just one has containerd configured to work with runsc/gvisor runtime which is cluster1-node2.

(Optionally) we ssh into the worker node and check if containerd+runsc is configured:

Now we best head to the k8s docs for RuntimeClasses https://kubernetes.io/docs/concepts/containers/runtime-class, steal an example and create the gvisor one:

And the required Pod:

After creating the pod we should check if it's running and if it uses the gvisor sandbox:

Looking good. And as required we finally write the dmesg output into the file:

 

 

Question 11 | Secrets in ETCD

 

Use context: kubectl config use-context workload-prod

 

There is an existing Secret called database-access in Namespace team-green.

Read the complete Secret content directly from ETCD (using etcdctl) and store it into /opt/course/11/etcd-secret-content. Write the plain and decoded Secret's value of key "pass" into /opt/course/11/database-password.

 

Answer:

Let's try to get the Secret value directly from ETCD, which will work since it isn't encrypted.

First, we ssh into the master node where ETCD is running in this setup and check if etcdctl is installed and list it's options:

Among others we see arguments to identify ourselves. The apiserver connects to ETCD, so we can run the following command to get the path of the necessary .crt and .key files:

The output is as follows :

With this information we query ETCD for the secret value:

ETCD in Kubernetes stores data under /registry/{type}/{namespace}/{name}. This is how we came to look for /registry/secrets/team-green/database-access. There is also an example on a page in the k8s documentation which you could save as a bookmark to access fast during the exam.

The tasks requires us to store the output on our terminal. For this we can simply copy&paste the content into a new file on our terminal:

We're also required to store the plain and "decrypted" database password. For this we can copy the base64-encoded value from the ETCD output and run on our terminal:

 

 

Question 12 | Hack Secrets

 

Use context: kubectl config use-context restricted@infra-prod

 

You're asked to investigate a possible permission escape in Namespace restricted. The context authenticates as user restricted which has only limited permissions and shouldn't be able to read Secret values.

Try to find the password-key values of the Secrets secret1, secret2 and secret3 in Namespace restricted. Write the decoded plaintext values into files /opt/course/12/secret1, /opt/course/12/secret2 and /opt/course/12/secret3.

 

Answer:

First we should explore the boundaries, we can try:

But no permissions to view RBAC resources. So we try the obvious:

We're not allowed to get or list any Secrets. What can we see though?

There are some Pods, lets check these out regarding Secret access:

This output provides us with enough information to do:

And for the second Secret:

None of the Pods seem to mount secret3 though. Can we create or edit existing Pods to mount secret3?

Doesn't look like it.

But the Pods seem to be able to access the Secrets, we can try to use a Pod's ServiceAccount to access the third Secret. We can actually see (like using k -n restricted get pod -o yaml | grep automountServiceAccountToken) that only Pod pod3-* has the ServiceAccount token mounted:

 

NOTE: You should have knowledge about ServiceAccounts and how they work with Pods like described in the docs

 

We can see all necessary information to contact the apiserver manually:

Let's encode it and write it into the requested location:

This will give us:

We hacked all Secrets! It can be tricky to get RBAC right and secure.

 

NOTE: One thing to consider is that giving the permission to "list" Secrets, will also allow the user to read the Secret values like using kubectl get secrets -o yaml even without the "get" permission set.

 

 

Question 13 | Restrict access to Metadata Server

 

Use context: kubectl config use-context infra-prod

 

There is a metadata service available at http://192.168.100.21:32000 on which Nodes can reach sensitive data, like cloud credentials for initialisation. By default, all Pods in the cluster also have access to this endpoint. The DevSecOps team has asked you to restrict access to this metadata server.

In Namespace metadata-access:

  • Create a NetworkPolicy named metadata-deny which prevents egress to 192.168.100.21 for all Pods but still allows access to everything else

  • Create a NetworkPolicy named metadata-allow which allows Pods having label role: metadata-accessor to access endpoint 192.168.100.21

There are existing Pods in the target Namespace with which you can test your policies, but don't change their labels.

 

Answer:

There was a famous hack at Shopify which was based on revealed information via metadata for nodes.

 

Check the Pods in the Namespace metadata-access and their labels:

There are three Pods in the Namespace and one of them has the label role=metadata-accessor.

Check access to the metadata server from the Pods:

All three are able to access the metadata server.

To restrict the access, we create a NetworkPolicy to deny access to the specific IP.

 

NOTE: You should know about general default-deny K8s NetworkPolcies.

 

Verify that access to the metadata server has been blocked, but other endpoints are still accessible:

Similarly, verify for the other two Pods.

Now create another NetworkPolicy that allows access to the metadata server from Pods with label role=metadata-accessor.

Verify that required Pod has access to metadata endpoint and others do not:

It only works for the Pod having the label. With this we implemented the required security restrictions.

If a Pod doesn't have a matching NetworkPolicy then all traffic is allowed from and to it. Once a Pod has a matching NP then the contained rules are additive. This means that for Pods having label metadata-accessor the rules will be combined to:

We can see that the merged NP contains two separate rules with one condition each. We could read it as:

Hence it allows Pods with label metadata-accessor to access everything.

 

 

Question 14 | Syscall Activity

 

Use context: kubectl config use-context workload-prod

 

There are Pods in Namespace team-yellow. A security investigation noticed that some processes running in these Pods are using the Syscall kill, which is forbidden by a Team Yellow internal policy.

Find the offending Pod(s) and remove these by reducing the replicas of the parent Deployment to 0.

 

Answer:

Syscalls are used by processes running in Userspace to communicate with the Linux Kernel. There are many available syscalls: https://man7.org/linux/man-pages/man2/syscalls.2.html. It makes sense to restrict these for container processes and Docker/Containerd already restrict some by default, like the reboot Syscall. Restricting even more is possible for example using Seccomp or AppArmor.

But for this task we should simply find out which binary process executes a specific Syscall. Processes in containers are simply run on the same Linux operating system, but isolated. That's why we first check on which nodes the Pods are running:

All on cluster1-node1, hence we ssh into it and find the processes for the first Deployment collector1 .

  1. Using crictl pods we first searched for the Pods of Deployment collector1, which has two replicas

  2. We then took one pod-id to find it's containers using crictl ps

  3. And finally we used crictl inspect to find the process name, which is collector1-process

We can find the process PIDs (two because there are two Pods):

Using the PIDs we can call strace to find Sycalls:

First try and already a catch! We see it uses the forbidden Syscall by calling kill(666, SIGTERM).

Next let's check the Deployment collector2 processes:

Looks alright. What about collector3:

Also nothing about the forbidden Syscall. So we finalise the task:

And the world is a bit safer again.

 

 

Question 15 | Configure TLS on Ingress

 

Use context: kubectl config use-context workload-prod

 

In Namespace team-pink there is an existing Nginx Ingress resources named secure which accepts two paths /app and /api which point to different ClusterIP Services.

From your main terminal you can connect to it using for example:

  • HTTP: curl -v http://secure-ingress.test:31080/app

  • HTTPS: curl -kv https://secure-ingress.test:31443/app

Right now it uses a default generated TLS certificate by the Nginx Ingress Controller.

You're asked to instead use the key and certificate provided at /opt/course/15/tls.key and /opt/course/15/tls.crt. As it's a self-signed certificate you need to use curl -k when connecting to it.

 

Answer:
Investigate

We can get the IP address of the Ingress and we see it's the same one to which secure-ingress.test is pointing to:

Now, let's try to access the paths /app and /api via HTTP:

What about HTTPS?

HTTPS seems to be already working if we accept self-signed certificated using -k. But what kind of certificate is used by the server?

It seems to be "Kubernetes Ingress Controller Fake Certificate".

 

Implement own TLS certificate

First, let us generate a Secret using the provided key and certificate:

Now, we configure the Ingress to make use of this Secret:

After adding the changes we check the Ingress resource again:

It now actually lists port 443 for HTTPS. To verify:

We can see that the provided certificate is now being used by the Ingress for TLS termination.

 

 

Question 16 | Docker Image Attack Surface

 

Use context: kubectl config use-context workload-prod

 

There is a Deployment image-verify in Namespace team-blue which runs image registry.killer.sh:5000/image-verify:v1. DevSecOps has asked you to improve this image by:

  1. Changing the base image to alpine:3.12

  2. Not installing curl

  3. Updating nginx to use the version constraint >=1.18.0

  4. Running the main process as user myuser

Do not add any new lines to the Dockerfile, just edit existing ones. The file is located at /opt/course/16/image/Dockerfile.

Tag your version as v2. You can build, tag and push using:

Make the Deployment use your updated image tag v2.

 

Answer:

We should have a look at the Docker Image at first:

Very simple Dockerfile which seems to execute a script run.sh :

So it only outputs current date and credential information in a loop. We can see that output in the existing Deployment image-verify:

We see it's running as root.

Next we update the Dockerfile according to the requirements:

Then we build the new image:

We can then test our changes by running the container locally:

Looking good, so we push:

And we update the Deployment to use the new image:

And afterwards we can verify our changes by looking at the Pod logs:

Also to verify our changes even further:

Another task solved.

 

 

Question 17 | Audit Log Policy

 

Use context: kubectl config use-context infra-prod

 

Audit Logging has been enabled in the cluster with an Audit Policy located at /etc/kubernetes/audit/policy.yaml on cluster2-controlplane1.

Change the configuration so that only one backup of the logs is stored.

Alter the Policy in a way that it only stores logs:

  1. From Secret resources, level Metadata

  2. From "system:nodes" userGroups, level RequestResponse

After you altered the Policy make sure to empty the log file so it only contains entries according to your changes, like using truncate -s 0 /etc/kubernetes/audit/logs/audit.log.

 

NOTE: You can use jq to render json more readable. cat data.json | jq

 

 

Answer:

First we check the apiserver configuration and change as requested:

 

NOTE: You should know how to enable Audit Logging completely yourself as described in the docs. Feel free to try this in another cluster in this environment.

 

Now we look at the existing Policy:

We can see that this simple Policy logs everything on Metadata level. So we change it to the requirements:

After saving the changes we have to restart the apiserver:

Once the apiserver is running again we can check the new logs and scroll through some entries:

Above we logged a watch action by OPA Gatekeeper for Secrets, level Metadata.

And in the one above we logged a list action by system:nodes for a ConfigMaps, level RequestResponse.

Because all JSON entries are written in a single line in the file we could also run some simple verifications on our Policy:

Looks like our job is done.

 

 

Question 18 | Investigate Break-in via Audit Log

 

Use context: kubectl config use-context infra-prod

 

Namespace security contains five Secrets of type Opaque which can be considered highly confidential. The latest Incident-Prevention-Investigation revealed that ServiceAccount p.auster had too broad access to the cluster for some time. This SA should've never had access to any Secrets in that Namespace.

Find out which Secrets in Namespace security this SA did access by looking at the Audit Logs under /opt/course/18/audit.log.

Change the password to any new string of only those Secrets that were accessed by this SA.

 

NOTE: You can use jq to render json more readable. cat data.json | jq

 

 

Answer:

First we look at the Secrets this is about:

Next we investigate the Audit Log file:

Audit Logs can be huge and it's common to limit the amount by creating an Audit Policy and to transfer the data in systems like Elasticsearch. In this case we have a simple JSON export, but it already contains 4451 lines.

We should try to filter the file down to relevant information:

Not too bad, only 28 logs for ServiceAccount p.auster.

And only 2 logs related to Secrets...

No list actions, which is good, but 2 get actions, so we check these out:

There we see that Secrets vault-token and mysql-admin were accessed by p.auster. Hence we change the passwords for those.

Audit Logs ftw.

By running cat audit.log | grep "p.auster" | grep Secret | grep password we can see that passwords are stored in the Audit Logs, because they store the complete content of Secrets. It's never a good idea to reveal passwords in logs. In this case it would probably be sufficient to only store Metadata level information of Secrets which can be controlled via a Audit Policy.

 

 

Question 19 | Immutable Root FileSystem

 

Use context: kubectl config use-context workload-prod

 

The Deployment immutable-deployment in Namespace team-purple should run immutable, it's created from file /opt/course/19/immutable-deployment.yaml. Even after a successful break-in, it shouldn't be possible for an attacker to modify the filesystem of the running container.

Modify the Deployment in a way that no processes inside the container can modify the local filesystem, only /tmp directory should be writeable. Don't modify the Docker image.

Save the updated YAML under /opt/course/19/immutable-deployment-new.yaml and update the running Deployment.

 

Answer:

Processes in containers can write to the local filesystem by default. This increases the attack surface when a non-malicious process gets hijacked. Preventing applications to write to disk or only allowing to certain directories can mitigate the risk. If there is for example a bug in Nginx which allows an attacker to override any file inside the container, then this only works if the Nginx process itself can write to the filesystem in the first place.

Making the root filesystem readonly can be done in the Docker image itself or in a Pod declaration.

Let us first check the Deployment immutable-deployment in Namespace team-purple:

The container has write access to the Root File System, as there are no restrictions defined for the Pods or containers by an existing SecurityContext. And based on the task we're not allowed to alter the Docker image.

So we modify the YAML manifest to include the required changes:

SecurityContexts can be set on Pod or container level, here the latter was asked. Enforcing readOnlyRootFilesystem: true will render the root filesystem readonly. We can then allow some directories to be writable by using an emptyDir volume.

Once the changes are made, let us update the Deployment:

We can verify if the required changes are propagated:

The Deployment has been updated so that the container's file system is read-only, and the updated YAML has been placed under the required location. Sweet!

 

 

Question 20 | Update Kubernetes

 

Use context: kubectl config use-context workload-stage

 

The cluster is running Kubernetes 1.29.5, update it to 1.30.1.

Use apt package manager and kubeadm for this.

Use ssh cluster3-controlplane1 and ssh cluster3-node1 to connect to the instances.

 

Answer:

Let's have a look at the current versions:

 

Control Plane Components

First we should update the control plane components running on the master node, so we drain it:

Next we ssh into it and check versions:

We see above that kubeadm is already installed in the required version. Else we would need to install it:

Check what kubeadm has available as an upgrade plan:

And we apply to the required version:

Next we can check if our required version was installed correctly:

 

Control Plane kubelet and kubectl

Now we have to upgrade kubelet and kubectl:

Done, only uncordon missing:

 

Data Plane

Our data plane consist of one single worker node, so let's update it. First thing is we should drain it:

Next we ssh into it and upgrade kubeadm to the wanted version, or check if already done:

Now we follow that kubeadm told us in the last line and upgrade kubelet (and kubectl):

Looking good, what does the node status say?

Beautiful, let's make it schedulable again:

We're up to date.

 

 

Question 21 | Image Vulnerability Scanning

 

(can be solved in any kubectl context)

 

The Vulnerability Scanner trivy is installed on your main terminal. Use it to scan the following images for known CVEs:

  • nginx:1.16.1-alpine

  • k8s.gcr.io/kube-apiserver:v1.18.0

  • k8s.gcr.io/kube-controller-manager:v1.18.0

  • docker.io/weaveworks/weave-kube:2.7.0

Write all images that don't contain the vulnerabilities CVE-2020-10878 or CVE-2020-1967 into /opt/course/21/good-images.

 

Answer:

 

The tool trivy is very simple to use, it compares images against public databases.

To solve the task we can run:

The only image without the any of the two CVEs is docker.io/weaveworks/weave-kube:2.7.0, hence our answer will be:

 

 

Question 22 | Manual Static Security Analysis

 

(can be solved in any kubectl context)

 

The Release Engineering Team has shared some YAML manifests and Dockerfiles with you to review. The files are located under /opt/course/22/files.

As a container security expert, you are asked to perform a manual static analysis and find out possible security issues with respect to unwanted credential exposure. Running processes as root is of no concern in this task.

Write the filenames which have issues into /opt/course/22/security-issues.

 

NOTE: In the Dockerfile and YAML manifests, assume that the referred files, folders, secrets and volume mounts are present. Disregard syntax or logic errors.

 

Answer:

We check location /opt/course/22/files and list the files.

We have 3 Dockerfiles and 7 Kubernetes Resource YAML manifests. Next we should go over each to find security issues with the way credentials have been used.

 

NOTE: You should be comfortable with Docker Best Practices and the Kubernetes Configuration Best Practices.

 

While navigating through the files we might notice:

 

Number 1

File Dockerfile-mysql might look innocent on first look. It copies a file secret-token over, uses it and deletes it afterwards. But because of the way Docker works, every RUN, COPY and ADD command creates a new layer and every layer is persistet in the image.

This means even if the file secret-token get's deleted in layer Z, it's still included with the image in layer X and Y. In this case it would be better to use for example variables passed to Docker.

So we do:

 

Number 2

The file deployment-redis.yaml is fetching credentials from a Secret named mysecret and writes these into environment variables. So far so good, but in the command of the container it's echoing these which can be directly read by any user having access to the logs.

Credentials in logs is never a good idea, hence we do:

 

Number 3

In file statefulset-nginx.yaml, the password is directly exposed in the environment variable definition of the container.

This should better be injected via a Secret. So we do:

 

preview

CKS Simulator Preview Kubernetes 1.30

https://killer.sh

This is a preview of the full CKS Simulator course content.

The full course contains 22 questions and scenarios which cover all the CKS areas. The course also provides a browser terminal which is a very close replica of the original one. This is great to get used and comfortable before the real exam. After the test session (120 minutes), or if you stop it early, you'll get access to all questions and their detailed solutions. You'll have 36 hours cluster access in total which means even after the session, once you have the solutions, you can still play around.

The following preview will give you an idea of what the full course will provide. These preview questions are not part of the 22 in the full course but in addition to it. But the preview questions are part of the same CKS simulation environment which we setup for you, so with access to the full course you can solve these too.

The answers provided here assume that you did run the initial terminal setup suggestions as provided in the tips section, but especially:

 

These questions can be solved in the test environment provided through the CKS Simulator

 

 

Preview Question 1

Use context: kubectl config use-context infra-prod

 

You have admin access to cluster2. There is also context gianna@infra-prod which authenticates as user gianna with the same cluster.

There are existing cluster-level RBAC resources in place to, among other things, ensure that user gianna can never read Secret contents cluster-wide. Confirm this is correct or restrict the existing RBAC resources to ensure this.

I addition, create more RBAC resources to allow user gianna to create Pods and Deployments in Namespaces security, restricted and internal. It's likely the user will receive these exact permissions as well for other Namespaces in the future.

 

Answer:
Part 1 - check existing RBAC rules

We should probably first have a look at the existing RBAC resources for user gianna. We don't know the resource names but we know these are cluster-level so we can search for a ClusterRoleBinding:

From this we see the binding is also called gianna:

It links user gianna to same named ClusterRole:

According to the task the user should never be able to read Secrets content. They verb list might indicate on first look that this is correct. We can also check using K8s User Impersonation:

But let's have a closer look:

Still all expected, but being able to list resources also allows to specify the format:

The user gianna is actually able to read Secret content. To prevent this we should remove the ability to list these:

 

Part 2 - create additional RBAC rules

Let's talk a little about RBAC resources:

A ClusterRole|Role defines a set of permissions and where it is available, in the whole cluster or just a single Namespace.

A ClusterRoleBinding|RoleBinding connects a set of permissions with an account and defines where it is applied, in the whole cluster or just a single Namespace.

Because of this there are 4 different RBAC combinations and 3 valid ones:

  1. Role + RoleBinding (available in single Namespace, applied in single Namespace)

  2. ClusterRole + ClusterRoleBinding (available cluster-wide, applied cluster-wide)

  3. ClusterRole + RoleBinding (available cluster-wide, applied in single Namespace)

  4. Role + ClusterRoleBinding (NOT POSSIBLE: available in single Namespace, applied cluster-wide)

 

The user gianna should be able to create Pods and Deployments in three Namespaces. We can use number 1 or 3 from the list above. But because the task says: "The user might receive these exact permissions as well for other Namespaces in the future", we choose number 3 as it requires to only create one ClusterRole instead of three Roles.

This will create a ClusterRole like:

Next the three bindings:

Which will create RoleBindings like:

And we test:

Feel free to verify this as well by actually creating Pods and Deployments as user gianna through context gianna@infra-prod.

 

 

Preview Question 2

Use context: kubectl config use-context infra-prod

 

There is an existing Open Policy Agent + Gatekeeper policy to enforce that all Namespaces need to have label security-level set. Extend the policy constraint and template so that all Namespaces also need to set label management-team. Any new Namespace creation without these two labels should be prevented.

Write the names of all existing Namespaces which violate the updated policy into /opt/course/p2/fix-namespaces.

 

Answer:

We look at existing OPA constraints, these are implemeted using CRDs by Gatekeeper:

So we can do:

And check violations for the namespace-mandatory-label one, which we can do in the resource status:

We see one violation for Namespace "sidecar-injector". Let's get an overview over all Namespaces:

When we try to create a Namespace without the required label we get an OPA error:

Next we edit the constraint to add another required label:

As we can see the constraint is using kind: RequiredLabels as template, which is a CRD created by Gatekeeper. Let's apply the change and see what happens (give OPA a minute to apply the changes internally):

After the changes we can see that now another Namespace jeffs-playground is in trouble. Because that one only specifies one required label. But what about the earlier violation of Namespace sidecar-injector?

Namespace sidecar-injector should also be in trouble, but it isn't any longer. This doesn't seem right, it means we could still create Namespaces without any labels just like using k create ns test.

So we check the template:

In the rego script we need to change count(missing) == 1 to count(missing) > 0 . If we don't do this then the policy only complains if there is one missing label, but there can be multiple missing ones.

After waiting a bit we check the constraint again:

This looks better. Finally we write the Namespace names with violations into the required location:

 

 

Preview Question 3

Use context: kubectl config use-context workload-stage

 

A security scan result shows that there is an unknown miner process running on one of the Nodes in cluster3. The report states that the process is listening on port 6666. Kill the process and delete the binary.

 

Answer:

We have a look at existing Nodes:

First we check the master:

Doesn't look like any process listening on this port. So we check the worker:

There we go! We could also use lsof:

Before we kill the process we can check the magic /proc directory for the full process path:

So we finish it:

Done.

tips

CKS Tips Kubernetes 1.30

In this section we'll provide some tips on how to handle the CKS exam and browser terminal.

 

Knowledge

 

Pre-Knowledge

You should have your CKA knowledge up to date and be fast with kubectl, so we suggest to do:

Knowledge

Approach

  • Do 1 or 2 test session with this CKS Simulator. Understand the solutions and maybe try out other ways to achieve the same thing.

  • Setup your aliases, be fast and breath kubectl

Content

 

CKS Exam Info

Read the Curriculum

https://github.com/cncf/curriculum

Read the Handbook

https://docs.linuxfoundation.org/tc-docs/certification/lf-handbook2

Read the important tips

https://docs.linuxfoundation.org/tc-docs/certification/important-instructions-cks

Read the FAQ

https://docs.linuxfoundation.org/tc-docs/certification/faq-cka-ckad-cks

 

Kubernetes documentation

Get familiar with the Kubernetes documentation and be able to use the search. Allowed links are:

NOTE: Verify the list here

 

CKS clusters

In the CKS exam you'll get access to as many clusters as you have questions, each will be solved in its own cluster. This is great because you cannot interfere with other tasks by breaking one. Every cluster will have one master and one worker node.

 

The Test Environment / Browser Terminal

You'll be provided with a browser terminal which uses Ubuntu 20. The standard shells included with a minimal install of Ubuntu 20 will be available, including bash.

Laggin

There could be some lagging, definitely make sure you are using a good internet connection because your webcam and screen are uploading all the time.

Kubectl autocompletion and commands

Autocompletion is configured by default, as well as the k alias source and others:

kubectl with k alias and Bash autocompletion

yq and jqfor YAML/JSON processing

tmux for terminal multiplexing

curl and wget for testing web services

man and man pages for further documentation

Copy & Paste

There could be issues copying text (like pod names) from the left task information into the terminal. Some suggested to "hard" hit or long hold Cmd/Ctrl+C a few times to take action. Apart from that copy and paste should just work like in normal terminals.

Score

There are 15-20 questions in the exam. Your results will be automatically checked according to the handbook. If you don't agree with the results you can request a review by contacting the Linux Foundation Support.

Notepad & Skipping Questions

You have access to a simple notepad in the browser which can be used for storing any kind of plain text. It might makes sense to use this for saving skipped question numbers. This way it's possible to move some questions to the end.

Contexts

You'll receive access to various different clusters and resources in each. They provide you the exact command you need to run to connect to another cluster/context. But you should be comfortable working in different namespaces with kubectl.

 

PSI Bridge

Starting with PSI Bridge:

  • The exam will now be taken using the PSI Secure Browser, which can be downloaded using the newest versions of Microsoft Edge, Safari, Chrome, or Firefox

  • Multiple monitors will no longer be permitted

  • Use of personal bookmarks will no longer be permitted

The new ExamUI includes improved features such as:

  • A remote desktop configured with the tools and software needed to complete the tasks

  • A timer that displays the actual time remaining (in minutes) and provides an alert with 30, 15, or 5 minute remaining

  • The content panel remains the same (presented on the Left Hand Side of the ExamUI)

Read more here.

 

Browser Terminal Setup

It should be considered to spend ~1 minute in the beginning to setup your terminal. In the real exam the vast majority of questions will be done from the main terminal. For few you might need to ssh into another machine. Just be aware that configurations to your shell will not be transferred in this case.

Minimal Setup

Alias

The alias k for kubectl will already be configured together with autocompletion. In case not you can configure it using this link.

Vim

The following settings will already be configured in your real exam environment in ~/.vimrc. But it can never hurt to be able to type these down:

The expandtab make sure to use spaces for tabs. Memorize these and just type them down. You can't have any written notes with commands on your desktop etc.

Optional Setup

Fast dry-run output

This way you can just run k run pod1 --image=nginx $do. Short for "dry output", but use whatever name you like.

Fast pod delete

This way you can run k delete pod1 $now and don't have to wait for ~30 seconds termination time.

Persist bash settings

You can store aliases and other setup in ~/.bashrc if you're planning on using different shells or tmux.

Alias Namespace

In addition you could define an alias like:

Which allows you to define the default namespace of the current context. Then once you switch a context or namespace you can just run:

But only do this if you used it before and are comfortable doing so. Else you need to specify the namespace for every call, which is also fine:

 

Be fast

Use the history command to reuse already entered commands or use even faster history search through Ctrl r .

If a command takes some time to execute, like sometimes kubectl delete pod x. You can put a task in the background using Ctrl z and pull it back into foreground running command fg.

You can delete pods fast with:

 

Vim

Be great with vim.

toggle vim line numbers

When in vim you can press Esc and type :set number or :set nonumber followed by Enter to toggle line numbers. This can be useful when finding syntax errors based on line - but can be bad when wanting to mark&copy by mouse. You can also just jump to a line number with Esc :22 + Enter.

copy&paste

Get used to copy/paste/cut with vim:

Indent multiple lines

To indent multiple lines press Esc and type :set shiftwidth=2. First mark multiple lines using Shift v and the up/down keys. Then to indent the marked lines press > or <. You can then press . to repeat the action.

 

Split terminal screen

By default tmux is installed and can be used to split your one terminal into multiple. But just do this if you know your shit, because scrolling is different and copy&pasting might be weird.

https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux