Kubernetes DevSecOps with Sumo Logic

Containers and Kubernetes have revolutionized the way many teams deploy applications. But with the many benefits that these technologies provide come new challenges.

Key among those challenges is security. By adding more layers and complexity to application environments, containers and Kubernetes create new opportunities for attackers and new security threats for Kubernetes admins to address. And although Kubernetes provides certain built-in security features, they are hardly enough to stop all attacks on their own.

The following is an overview of Kubernetes security essentials, including the main types of security risks that exist in a Kubernetes-based environment, why securing Kubernetes is harder than securing non-containerized environments, and best practices that teams can follow for maximizing Kubernetes security.

[Watch: Kubernetes Observability]

Kubernetes Vulnerabilities

The main reason why securing Kubernetes is challenging is that Kubernetes is a sprawling platform composed of many different parts. Each of those components carries its own security risks.

Here’s a rundown of the key parts of a Kubernetes environment and the most common security risks that affect them:

• Containers: Containers can contain malicious code that was included in their container images. They can also be subject to misconfigurations that allow attackers to gain unauthorized access under certain conditions.
• Host operating systems: Vulnerabilities or malicious code within the operating systems installed on Kubernetes nodes can provide attackers a path into Kubernetes clusters.
• Container runtimes: Kubernetes supports a variety of container runtimes. All of them could potentially contain vulnerabilities that allow attackers to take control of individual containers, escalate attacks from one container to another, and even gain control of the Kubernetes environment itself.
• Network layer: Kubernetes relies on internal networks to facilitate communication between nodes, pods, and containers. It also typically exposes applications to public networks so that they can be accessed over the Internet. Both network layers could allow attackers to gain access to the cluster or to escalate attacks from one part of the cluster into others.
• API: The Kubernetes API, which plays a central role in allowing components to communicate and applying configurations, could contain vulnerabilities or misconfigurations that enable attacks.
• Kubectl (and other management tools): Kubectl, Dashboard, and other Kubernetes management tools might be subject to vulnerabilities that allow abuse on a Kubernetes cluster.

Built-in Kubernetes Security Features

Kubernetes offers native security features to protect against some of the threats described above, or at least to mitigate the potential impact of a breach. The main security features offered by Kubernetes include:

• Role-based access control (RBAC): Kubernetes allows admins to define what it calls Roles and ClusterRoles, which specify which users can access which resources within a namespace or an entire cluster. RBAC provides one way of regulating access to resources.
• Pod security policies and network policies: Admins can configure pod security policies and network policies, which place restrictions on how containers and pods can behave. For example, pod security policies can be used to prevent containers from running as the root user, and network policies can restrict communication between pods.
• Network encryption: Kubernetes uses TLS to encrypt network traffic, providing a safeguard against eavesdropping.

While these built-in Kubernetes security features provide layers of defense against certain types of attacks, they do not cover all threats. Kubernetes offers no native protections against the following types of attacks:

• Malicious code or misconfigurations inside containers or container images: To scan for these, you would have to use a third-party container scanning tool.
• Security vulnerabilities on host operating systems: Again, you would have to scan for these using other tools. And although some Kubernetes distributions (like OpenShift) integrate SELinux or similar kernel-hardening frameworks to provide more security at the host level, this is not a feature of Kubernetes itself.
• Container runtime vulnerabilities: Here again, Kubernetes has no way of knowing or alerting you if a vulnerability exists within your runtime, or if an attacker is trying to exploit a vulnerability in the runtime.
• Abuse of the Kubernetes API: Beyond following any RBAC and security policy settings that you define, Kubernetes does nothing to detect or respond to API abuse.
• Management tool vulnerabilities or misconfigurations: Kubernetes cannot guarantee that management tools (like Kubectl) are free of security problems.

[Learn More: Kubernetes Cloud SIEM]

Kubernetes Hardening Best Practices

Because the built-in security features of Kubernetes are limited in scope, it’s critical for teams to take extra steps to secure their clusters. The following are some best practices for getting the most out of the security features that Kubernetes offers, as well as for leveraging external tools and strategies to provide more security.

Configure Pod Security and Network Policies

As noted above, pod security policies and network policies can be used to enforce security restrictions. However, it’s important to understand that these policies are not configured and enabled in most Kubernetes distributions by default (and even if they are turned on by default in your distribution, they likely need to be tailored to your needs).

Therefore, a critical first step in hardening Kubernetes is to make sure that you set up and enforce these policies in a way that reflects your team’s security needs. The level of strictness that you apply in these policies will vary depending on how secure your cluster needs to be; for example, a production cluster is more likely to have more restrictive policies (such as policies that prevent write-access to resources and prevent all non-essential network traffic) than a cluster that is used internally for development or testing purposes (in which case having very strict security policies is typically not as important, because the cluster will not be running mission-critical apps connected to the public Internet).

Kubernetes Host Security

Kubernetes is only as secure as the operating systems that power its nodes. Because Kubernetes has no way of monitoring or hardening host operating systems, admins need to cover that ground themselves.

It’s a best practice to choose a host Linux distribution that has a minimal footprint, since extraneous operating system apps or services that are not necessary for Kubernetes increase your attack surface needlessly. It’s also a best practice to enable SELinux, AppArmor, or a similar security framework on the host system; these tools add another layer of protection against certain types of exploits against the host. Finally, user, group, and filesystem permissions should be properly configured on the host to ensure that only user accounts that should be able to access the Kubernetes installation have the ability to do so.

Keep Your Runtime Secure and Up-to-Date

No container runtime used in conjunction with Kubernetes is immune to security vulnerabilities. Therefore, you can never be certain that your runtime is safe. However, you can mitigate the risk by keeping the runtime up-to-date.

Leverage logging and auditing to improve security

Log data provides crucial insights into potential security breaches. It’s also critical for investigating past security events. However, while Kubernetes provides facilities for generating log data, it provides no features for auditing or interpreting that data for any purpose, least of all for security. You therefore need to adopt third-party tools to leverage Kubernetes log data as a basis for security operations.

Sumo Logic helps with this process by making it easy to aggregate and interpret Kubernetes logs. By installing the Sumo Logic Kubernetes App, teams can put Kubernetes logs to work to detect anomalous activity on Kubernetes nodes and networks, and thus gain critical visibility into their Kubernetes environments.

With Sumo Logic, we can put all of these pieces together to build end-to-end Observability in Kubernetes.

1. Setup and Collection - The entire collection process can be set up with a single Helm chart. Fluentbit, Fluentd, Prometheus, and Falco are deployed throughout the cluster in order to collect log, metric, event and security data.
2. Enrichment - Once collected, the data flows into a centralized Fluntd pipeline for metadata enrichment. Data is enriched— tagged— with the details about where in the cluster it originated; the service, deployment, namespace, node, pod, container, and their labels.
3. Sumo Logic - Finally, the data is sent to Sumo Logic via HTTP for storage, access, and most importantly analytics.

Note: Labels — When you create objects in Kubernetes, you can assign custom key-value pairs to each of those objects, called labels. These labels can help you organize and track additional information about each object. For example, you might have a label that represents the application name, the environment the pod is running in or perhaps what team owns this resource. These labels are entirely flexible and can be defined as you need. Our FluentD plugin ensures those labels get captured along with the logs giving you continuity between the resources you have created and the log files they are producing.

Metadata Enrichment

Unified metadata enrichment is critical to building context about the data in your cluster, and the hierarchy of the components present. Standalone prometheus or fluentd deployments give some context about the data — node, container, and pod level information — but not valuable insight to the service, deployment or namespace. Sumo Logic uses Fluentd as a centralized metadata pipeline to ping the API server and gain rich context about the data getting pass into Sumo Logic.

By centralizing metadata enrichment, the Sumo Logic solution reduces the load on the Kubernetes API server and ensures consistent metadata tagging across logs, metrics and events without which it would be impossible to correlate data when troubleshooting. You can use this metadata when searching through your logs and your metrics and use them together to have a unified experience when navigating your machine data.

Ingestion into Sumo Logic

There is tremendous value in having this data come to a single place. With metrics serving as the smoke detector, and logs enabling us to drill down to the root cause, unifying these data sources around a common metadata language enables us to easily correlate these signals. We can pivot from the metrics data about a cluster to the events data about a cluster to the logs data about an application.

Metadata enables us to build a hierarchical view of a cluster. By connecting pods to their services or group nodes by cluster, it becomes easier to explore the Kubernetes stack. By tapping into the Auto-discovery capabilities inherent in Prometheus, we can ensure that the hierarchy visualized in Sumo Logic is accurate and up to date.

Tying together DevOps and SecOps

We can take this further by providing data about security relevant events in the context of the Kubernetes mental model. Below we can see top security rules triggered in the cluster overview. Zoom in and we see this same data for the service or namespace and so on.

Displaying security information within the natural hierarchies of Kubernetes, we can enable a consistent view across DevOps and SecOp to build closer and more efficient DevSecOps cooperation.

Kubernetes security, application security, and network security

Zooming out, we can also take out Kubernetes security data and insert it in our high-level security dashboards. Combining infrastructure security, network security, full-stack security, and Kubernetes security gives us comprehensive visibility into the entire security story.