All content in this post provided by SUSE

How to Secure Your Kubernetes Pipeline

The Importance of Securing Container Deployments
Containers and tools like Kubernetes enable enterprises to automate many aspects of application deployment, providing tremendous business benefits. But these new deployments are just as vulnerable to attacks and exploits from hackers and insiders as traditional environments. Ransomware extortion, cryptojacking, data theft and service disruption will continue to be used against new, container-based virtualized environments in both private and public clouds.

The hyper-dynamic nature of containers creates the following security challenges

1. Vulnerabilities introduced in the CI/CD pipeline. The heavy use of open source components and the continual discovery of critical vulnerabilities affect container images in the build phase, in registries, and in production.

2. Explosion of east-west traffic. While monolithic applications could be secured by traditional firewalls and host security tools, containers can be dynamically increasing the east-west, or internal traffic that must be monitored for attacks.

3. Increased attack surface. Each container may have a different attack surface and vulnerabilities which can be exploited. Also, the additional attack surface introduced by container orchestration tools such as Kubernetes and Docker must be considered.

4. Automating security to keep pace. Old models and tools for security will not be able to keep up in a constantly changing container environment. Given the automated nature of Kubernetes, containers and pods can appear and disappear in minutes or seconds. Application behaviors which can include new network connections must be instantly factored into enforced security policies. Next-generation automated security tools are needed to secure containers, declaring security policies early in the pipeline and managed as code.

How Kubernetes Works: The Basics

Kubernetes is an orchestration tool which automates the deployment, updating, and monitoring of containers. Kubernetes is supported by all major container management and cloud platforms such as Red Hat OpenShift, Docker EE, Rancher, IBM Cloud, AWS EKS, Azure, SUSE CaaS, and Google Cloud. Following are some of the key things you need to know about Kubernetes:

• Master node. The server which manages the Kubernetes worker node cluster and the deployment of pods on nodes. Nodes can be physical or virtual machines.
• Worker node. Also known as slaves or minions, these servers typically run the application containers and other Kubernetes components such as agents and proxies.
• Pods. The unit of deployment and addressability in Kubernetes. A pod has its own IP address and can contain one or more containers (typically one).
• Services. A service functions as a proxy to its underlying pods and requests can be load-balanced across replicated pods. A service can also provide an externally accessible endpoint for access to one-or-more-pods by defining an external IP or NodePort. Kubernetes also provides a DNS service, router, and load balancer.

Kubernetes role-based access controls

Kubernetes role-based access controls (RBACs) provide granular management of resources. They can enable access to application workloads as well as Kubernetes system resources. Management tools such as OpenShift may add additional capabilities but rely on or use native Kubernetes basic security controls. It is critical to properly configure access controls to prevent unauthorized access to Kubernetes components such as the API Server or application workloads.

Kubernetes networking basics

The key networking concept in Kubernetes is that every pod is assigned its own routable IP address. Kubernetes (actually, its network plug-in) takes care of routing all requests internally between hosts to the appropriate pod. External access to Kubernetes pods can be provided through a service, load balancer, or ingress controller, which Kubernetes routes to the appropriate pod. Kubernetes uses iptables to control the network connections between pods (and between nodes), handling many of the networking and port forwarding rules.

This way, clients do not need to keep track of IP addresses to connect to Kubernetes services. Also, port mapping is greatly simplified (and mostly eliminated) since each pod has its own IP address and its container can listen on its native port. With all of this overlay networking being handled dynamically by Kubernetes, it is extremely difficult to monitor network traffic, much less secure it.

Kubernetes Vulnerabilities and Attack Vectors

Attacks on Kubernetes containers running in pods can originate externally through the network or internally by insiders, including victims of phishing attacks whose systems become conduits for insider attacks. Here are a few examples:

1. Container compromise. An application misconfiguration or vulnerability enables the attacker to get into a container to start probing for weaknesses in the network, process controls, or file system.

2. Unauthorized connections between pods. Compromised containers can attempt to connect with other running pods on the same or other hosts to probe or launch an attack. Although Layer 3 network controls whitelisting pod IP addresses can offer some protection, attacks over trusted IP addresses can only be detected with Layer 7 network filtering.

3. Data exfiltration from a pod. Data stealing is often done using a combination of techniques, which can include a reverse shell in a pod connecting to a command and control server and network tunneling to hide confidential data.

4. Compromised container running malicious process. Containers generally have a limited and well-defined set of processes running, but a compromised container can start malware such as crypto mining or suspicious processes such as network port scanning, or inject a binary (process exploit) that has not been seen before.

5. Container file system compromised. An attacker can install vulnerable libraries/packages to exploit the container. Sensitive files can also be changed. Once exploited, a privilege escalation to root or other breakout can be attempted.

6. Compromised worker node. The host itself can be compromised, the same as any active container. For example, the Dirty Cow Linux kernel vulnerability enabled a user to escalate to root privilege

Securing Kubernetes System and Resources

If not protected, orchestration tools such as Kubernetes and the management platforms built on top of them can be vulnerable to attacks. These expose potentially new attack surfaces for container deployments which previously did not exist, and thus are vulnerable to hacker incursion. The Tesla hack and Kubelet exploit are among the first of what is expected to be a continuing cycle of exploits and patching for new technologies. To protect Kubernetes and management platforms from attacks, it’s critical to properly configure the RBACs for system resources. Following are areas to review and configure for proper access controls:

1. Protect the api server. Configure RBAC for the API Server or manually create firewall rules to prevent unauthorized access.

2. Restrict kubelet permissions. Configure RBAC for Kubelets and manage certificate rotation to secure the Kubelet.

3. Require authentication for all external ports. Review all ports externally accessible and remove unnecessary ports. Require authentication for those external ports needed. For non-authenticated services, restrict access to a whitelist source.

4. Limit or remove console access. Don’t allow console/proxy access unless properly configured for user login with strong passwords or two-factor authentication.

Summary Checklist for Run-Time Kubernetes Security

Following is a convenient checklist summary of the security protections to review for securing Kubernetes deployments in the CI/CD pipeline and during run-time.

CI/CD pipeline

• Scan images in build phase and fail/reject those with critical vulnerabilities with fixes available, or compliance violations.
• Continuously scan approved images in registries to alert if new vulnerabilities have been discovered.
• Scan images for compliance violations of CIS Benchmarks as well as embedded secrets, running as root, file permissions and other security issues.
• Implement declarative security as code practices to involve developers and/or DevOps teams to create or review allowed (whitelisted) application behavior for their application workloads.

Pre-production checklist for devops and security teams

• Use namespaces
• Restrict Linux capabilities
• Enable SELinux
• Utilize Seccomp
• Configure Cgroups
• Use R/O mounts

Run-time checklist for operations and security teams

• Prevent unauthorized and vulnerable deployments by using admission control policies
• Enforce isolation by application/service using network segmentation and namespaces
• Inspect network connections for application attacks
• Monitor containers for suspicious process or file system activity
• Protect worker nodes from host privilege escalations, suspicious processes or file system activity
• Capture packets for security events
• Quarantine or remediate compromised containers
• Scan containers and hosts for vulnerabilities
• Alert, log, and respond in real-time to security incidents
• Conduct security auditing and compliance checks with CIS Benchmarks

Kubernetes system protections

• Review all RBACs
• Protect the API Server
• Restrict Kubelet permissions
• Secure external ports
• Whitelist non-authenticated services
• Limit/restrict console access
• Monitor system container connections and processes in production
• Run the CIS Benchmarks for Kubernetes to audit configurations
• Keep the orchestrator version updated to remediate critical vulnerabilities

Next Steps: Want to learn more?

Visit NeuVector.com for additional container security articles on our blog or to schedule a demo of the SUSE NeuVector Kubernetes security platform.

Connect with our SUSE Brand Sales Specialist!