Implementing Policies in Kubernetes

Implementing Policies in Kubernetes

Kubernetes, as we know, coordinates a highly available cluster of computers that are connected to work as a single unit.

Kubernetes contains a number of abstractions that allows deployment of containerized applications to the cluster without attaching them to individual machines.

In short, Kubernetes is -

• Portable: public, private, hybrid, multi-cloud
• Extensible: modular, pluggable, hook able, composable
• Self-healing: auto-placement, auto-restart, auto-replication, auto-scaling

In this article, I’ll explain what Kubernetes policies are, and how they can help you manage and secure the Kubernetes cluster. We will also look at why we need a policy engine to author and manage the policies.

Kubernetes Policies - Introduction

In simplest terms, policies define what end-users can do on the cluster and possible ways to ensure that clusters are in compliance with organization policies. Policies could be of type Governance for example, to meet some of organizational conventions or it could be to meet Legal requirements or to enforce best practices.

Policy-enablement empowers organizations to take control of Kubernetes operation and ensure that clusters are in compliance with organization policies.  

Key benefits of Policy Enablement

By default, containers run with unbounded compute resources on a Kubernetes cluster. To limit or restrict you have to implement appropriate policies in the following ways:

1. Network Policies - To define how groups of pods are allowed to communicate with each other and other network endpoints - Use NetworkPolicy resources labels to select pods and define rules that specify what traffic is allowed to the selected pods.
2. Volume Policies - Kubernetes scheduler has default limits on the number of volumes that can be attached to a Node. To define the maximum number of volumes that can be attached to a Node for various cloud providers, use Node-specific Volume Limits.
3. Resource Usage Policies - To enforce constraints on resource usage, use Limit Range option for appropriate resource in the namespace
   1. Compute resources usage per Pod or Container.
   2. Storage request per PersistentVolumeClaim.
   3. Ratio between request and limit.
   4. Set default request/limit for compute resources and automatically inject them to Containers at runtime.
4. Resource consumption Policies - To limit aggregate resource consumption per namespace, use below Resource Quotas
   1. Compute Resource Quota
   2. Storage Resource Quota
   3. Object Count Quota
   4. Limits the number of resources based on scope defined in Quota Scopes option
   5. Requests vs Limits - Each container can specify a request and a limit value for either CPU or memory.
   6. Quota and cluster capacity – Expressed in absolute units
   7. Limit Priority Class consumption by default – For example, restrict usage of certain high priority pods
5. Access Control Policies - To allow/deny fine-grained permissions, use RBAC (Role Based Access Control) and rules can be defined to allow/deny fine-grained permissions. For example, an "autoscaler" role may have permission to "update" deployments in a specific namespace for changing their number of replicas.
6. Security Policies - To define & control security aspects of Pods, use Pod Security Policy (available on v1.15). According Kubernetes Documentation, it would enable fine-grained authorization of pod creation and updates. Defines a set of conditions that a pod must run with in order to be accepted into the system, as well as defaults for the related fields. They allow an administrator to control the following:
   • Running of privileged containers
   • Usage of host namespaces
   • Usage of host networking and ports
   • Usage of volume types
   • Usage of the host filesystem
   • Restricting escalation to root privileges
   • The user and group IDs of the container
   • AppArmor or seccomp or sysctl profile used by containers

Limitations of Kubernetes Policies

As you can see from above, there is NO single security configuration for Kubernetes. For example, to define what a specific user can do, the groups they belong to, the actions they can perform on various Kubernetes resources (pods, deployments, services, etc), the network and pod security policies that apply to the objects they create, etc. cannot be expressed as rules across different policy components.

Due to the lack of single point security solution, ensuring compliance manually can be error-prone and frustrating. There is a need for a lightweight general-purpose policy engine that allows developers to operate independently without sacrificing compliance and also ensures ease of policy enforcement, automated discovery of violations, conflicts. Policy Authors would be also able to author and deploy custom policies that control the behaviour of service’s policy-enabled features.

Introducing Open Policy Agent Gatekeeper

Open Policy Agent Gatekeeper enforces policies and strengthen governance on Kubernetes cluster. Following are the key functionalities it provides:

• Extensible, parameterized policy library.
• High-level declarative language (Rego) to author fine-grained policies in system.
• Native Kubernetes CRDs for instantiating the policy library – Allows the definition of "constraints" where in you want a system to meet a given set of requirements.
• Native Kubernetes CRDs for extending the policy library – Allows definition of "constraint templates" that allows users to declare new Constraints.
• Audit functionality - Allows periodic evaluations of replicated resources against the Constraints enforced in the cluster to detect any mismatches.
• Test framework that you can use to write tests for policies. By writing tests for policies, development process of new rules is accelerated and time saved.

 
Image – Open Policy Agent Gatekeeper Components / Source – Open Policy Agent Gatekeeper Documentation

Kubernetes provides Admission controller webhooks (HTTP Callbacks) to intercept admission requests before they are persisted as objects in Kubernetes, OPA Gatekeeper uses the same for making policy decisions from the API Server. Once all object modifications are complete, and incoming object is validated by the API server, validating admission webhooks are invoked and they can either reject or accept requests to enforce policies.

Gatekeeper enforces CRD-based policies executed by Open Policy Agent and thus enables users to customize admission control via configuration.

Key Concepts

• Validation of Controls - Once all the Gatekeeper is installed in the cluster, API server will then trigger the Gatekeeper admission webhook to process the admission request whenever a resource in the cluster is created, updated, or deleted. During the validation process, Gatekeeper acts as a bridge between the API server and OPA. API Server will enforce all policies executed by OPA.
• Policies / Constraints - Constraint is a declaration that wants a system to meet a given set of requirements. Each Constraint is written with Rego, a declarative query language to enumerate instances of data that violate the expected state of the system. All Constraints are evaluated as a logical AND. If one Constraint is not satisfied, then the whole request is rejected.
• Audit Functionality - Enables periodic evaluations of replicated resources against the Constraints enforced in the cluster to detect pre-existing misconfigurations.
• Data replication is required by Constraints that need access to objects in the cluster other than the object under evaluation. For example, a Constraint that enforces uniqueness of ingress hostname must have access to all other ingresses in the cluster.

Implementing Simple Constraint/ConstraintTemplate with OPA Gatekeeper

In this example, we would be defining new constraint template and constraint that requires all labels to be present and valid. Here, I’m going to use the samples that comes up with OPA Gatekeeper installation.

Installation

To deploy a released version of Gatekeeper on the cluster with a prebuilt image, run following command.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Image – Kubernetes Cluster is ready

We have our Kubernetes cluster ready, lets install Gatekeeper with a prebuilt image.

Image – Open Policy Agent Gatekeeper Installation completed

Gatekeeper Role, CRDs are now installed. Next step is to create new constraint template to enforce labels on the namespace to be present and valid.

Define constraint template(s)

ConstraintTemplate defines what needs to be enforced and the schema of the constraint. Here if you notice the openAPIV3Schema and targets constraint field allows users to fine-tune the behaviour of a constraint.

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
        listKind: K8sRequiredLabelsList
        plural: k8srequiredlabels
        singular: k8srequiredlabels
      validation:
        # Schema for the `parameters` field
        openAPIV3Schema:
          properties:
            labels:
              type: array
              items: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

Install ConstraintTemplate with the following command

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/basic/templates/k8srequiredlabels_template.yaml

Image – OPA Gatekeeper ConstraintTemplate created

ConstratintTemplate is created, the next step is to define constraints and apply it to Namespace.

Define constraints

Following constraint uses the K8sRequiredLabels constraint template defined in the previous step. Next step is to use constraints to make sure the gatekeeper label is defined on all namespaces.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: ns-must-have-gk
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    labels: ["gatekeeper"]

match field, which defines the scope of objects to which a given constraint will be applied.

• kinds accept a list of objects with apiGroups and kinds fields that list the groups/kinds of objects to which the constraint will apply.
• namespaces is a list of namespace names. If defined, a constraint will only apply to resources in a listed namespace.
• labelSelector, namespaceSelector is a standard Kubernetes label and namespace selector.

Install above Constraint with the following command

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/basic/constraints/all_ns_must_have_gatekeeper.yaml

Image – OPA Gatekeeper Constraints created

Now that ConstraintTemplate & Constraint is enabled, let’s try out to create new namespace without label.

Image – OPA Gatekeeper Test Simple Constraint

As you can see, OPA Gatekeeper has prevented namespace creation without labels. Next, we can look at example on how to set container limits policy.

Implementing Container Limits Constraint/ConstraintTemplate with OPA Gatekeeper

In this example, we would be defining new constraint template and constraint that requires container limits to be specified during definition of Pod.

We are going to reuse Kubernetes cluster with Gatekeeper components installed in the previous demo. Our first step is to define the constraint template.

Define constraint template(s)

ConstraintTemplate defines what needs to be enforced and the schema of the constraint. Here limits are defined in k8scontainterlimits_template.yaml. Install ConstraintTemplate with the following command

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/templates/k8scontainterlimits_template.yaml

Image – OPA Gatekeeper Container Limits Constraint Template Created

Define constraint

Next step is to define constraints to make sure that cpu and memory should be equal or less than 200m and 1Gi limits.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: container-must-have-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "200m"
    memory: "1Gi"

 
Image – OPA Gatekeeper Container Limits Constraint Created

Now that we have ConstraintTemplate & Constraint created, let’s try out creating new resource without limits.

apiVersion: v1
kind: Pod
metadata:
  name: opa
  namespace: production
  labels:
    owner: me.agilebank.demo
spec:
  containers:
    - name: opa
      image: openpolicyagent/opa:0.9.2
      args:
        - "run"
        - "--server"
        - "--addr=localhost:8080"

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/demo/agilebank/bad_resources/opa_no_limits.yaml

 
Image – OPA Gatekeeper Container Limits Constraint Testing

As you can see, ConstraintTemplate & Constraint restricts the pod creation without limits.

Congrats! We have successfully enforced policies with OPA Gatekeeper Policy engine.

To uninstall Gatekeeper policy engine, first clean up old Constraints, ConstraintTemplates, and the Config resource in the gatekeeper-system namespace and then uninstall Gatekeeper. Currently the uninstall action only removes the Gatekeeper system. This will make sure all finalizers are removed by Gatekeeper. Otherwise the finalizers will need to be removed manually.

Conclusion

Though Open Policy Agent Gatekeeper enables Kubernetes administrators to have fine-grained policy-based control across the stack, but applying policies are not without challenges because of the complex nature of deploying applications.

References

1. Kubernetes Admission Webhooks
2. Kubernetes Limit Ranges
3. Kubernetes Resource Quotas
4. Kubernetes Pod Security Policies
5. Kubernetes Storage Limits
6. Open Policy Agent
7. Gatekeeper Github
8. Gatekeeper Samples

About the Author

Karthikeyan Shanmugam (Karthik) is an experienced Solutions Architect professional with about 18+ years of experience in design & development of enterprise applications across Banking, Financial Services, Healthcare and Aviation domains. Currently engaged in Technical consulting & providing solutions in the Application Transformation space.