Usage⚓︎

Understand how to plan, install and operate Semgr8s.

Considerations⚓︎

Before integrating Semgr8s, it is important to bear a few considerations in mind:

Semgr8s is still in an early stage of development with exciting ideas for improvement
There is only limited operational experience so far and there might be breaking changes. We are happy for any feedback, bug reports, feature requests, and contributions via GitHub discussions, issues and PRs
Semgrep's yaml support is currently experimental.
Semgr8s (like any other Kubernetes admission controller) can break a cluster when misconfigured. Therefore, testing should be rigorous and happen on a dedicated test cluster.
Semgr8s can be used with remote rules. Those introduce an external dependence for validation which can affect performance and availability.
Kubernetes admission controllers have maximum timeout of 30s which can require special attention for large deployments.

Setup⚓︎

Semgr8s is installed via Helm, but instructions can be adapted for usage with kubectl apply or your method of choice.

Requirements⚓︎

git
Kubernetes cluster for testing (e.g. kind, microk8s, or minikube)
kubectl
Helm
(optional) make (e.g. via build-essential)
(optional) yq v4.x
(optional) docker

Get Code⚓︎

The Helm charts are contained within the Semgr8s repository:

git clone https://github.com/semgr8ns/semgr8s.git
cd semgr8s

Configuration⚓︎

Semgr8s comes preconfigured with some basic rules. However, configuration can be adjusted to your needs. Central configuration is maintained in charts/semgr8s/values.yaml:

values.yaml chart

charts/semgr8s/values.yaml

deployment:
  image:
    repository: ghcr.io/semgr8ns/semgr8s
    pullPolicy: IfNotPresent
    tag: ""
  imagePullSecrets: []
  replicas: 2
  containerPort: 5000
  podAnnotations: {}
  podSecurityContext: {}
  resources:
    limits:
      cpu: 1000m
      memory: 128Mi
    requests:
      cpu: 100m
      memory: 64Mi
  securityContext:
    allowPrivilegeEscalation: false
    capabilities:
      drop:
        - ALL
    privileged: false
    readOnlyRootFilesystem: true
    runAsNonRoot: true
    runAsUser: 10001 # remove when using openshift or OKD 4
    runAsGroup: 20001 # remove when using openshift or OKD 4
    seccompProfile:
      type: RuntimeDefault

service:
  type: ClusterIP
  port: 443

webhooks:  # configuration options for webhooks described under https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#webhook-configuration
  validating: # main webhook
    failurePolicy: Fail
    sideEffects: None
    timeoutSeconds: 30
    admissionReviewVersions: ["v1","v1beta1"]
    namespaceSelector:
      matchLabels:
        semgr8s/validation: enabled
    rules:
      - scope: "Namespaced"
        apiGroups: ["", "apps", "batch", "networking.k8s.io", "rbac.authorization.k8s.io"]
        resources: ["*/*"]
        apiVersions: ["*"]
        operations: ["CREATE", "UPDATE"]
  mutating: # autofix webhook, only used when enabled
    failurePolicy: Fail
    sideEffects: None
    timeoutSeconds: 30
    admissionReviewVersions: ["v1","v1beta1"]
    namespaceSelector:
      matchLabels:
        semgr8s/validation: enabled
    rules:
      - scope: "Namespaced"
        apiGroups: ["", "apps", "batch", "networking.k8s.io", "rbac.authorization.k8s.io"]
        resources: ["*/*"]
        apiVersions: ["*"]
        operations: ["CREATE", "UPDATE"]

application:
  # Configure the log level. Either one of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. Defaults to `INFO`
  logLevel: INFO
  # fail on rule violation (true/false)
  enforce: true
  # remoteRules: Apply remote rules from e.g.
  # * semgrep registry: https://semgrep.dev/r
  # * semgrep-rules github repo: https://github.com/semgrep/semgrep-rules
  # common choices: p/kubernetes, r/yaml.kubernetes
  remoteRules: ["p/kubernetes"]
  # apply semgrep fixes before validation (see https://semgrep.dev/docs/writing-rules/autofix)
  autofix: false
  # requires generic secret with name 'semgrep-app-token' and key 'token' in semgr8ns namespace
  semgrepLogin: false

Configuration aims to provide the most native integration of Semgrep's functionality into Kubernetes. Working knowledge of Kubernetes and the Semgrep documentation should be sufficient to understand the concepts and options being used. In charts/semgr8s/values.yaml:

.deployment and .service only affect the Semgr8s deployment itself.
.webhooks allows configuration of validation scope (resources, operations, ...).
.application provides application-specific configuration (features, remote rules).

Tip

Use .webhooks to configure the overall Semgr8s scope. In production, this should be a careful trade-off between desired policy scope, availability, performance, and security.

Rules form the basis to construct your policy. It is possible to either reference remote rules or add custom rules locally. Remote Semgrep rules, rulesets, or repository rules are configured via .application.remoteRules in charts/semgr8s/values.yaml, e.g. set to "r/yaml.kubernetes.security.allow-privilege-escalation.allow-privilege-escalation" or "p/kubernetes", or "r/yaml.kubernetes" respectively. Custom Semgrep rules can be placed in charts/semgr8s/rules/ and will be auto-mounted into the admission controller or added later as configmaps. Additional information on rule creation and management is shared below. Some useful example rules are provided under ./rules/.

At present, Semgr8s is shipped with Semgrep's p/kubernetes ruleset and one local test rule detecting a unique label.

Installation⚓︎

To deploy the preconfigured admission controller simply run:

helm install semgr8s charts/semgr8s --create-namespace --namespace semgr8ns

output

NAME: semgr8s
LAST DEPLOYED: Tue Apr 25 00:16:04 2023
NAMESPACE: semgr8ns
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Successfully installed semgr8s!

You can check successful deployment of semgr8s via:

kubectl get all -n semgr8ns

output

NAME                           READY   STATUS    RESTARTS   AGE
pod/semgr8s-665dbb8756-qhqv6   1/1     Running   0          7s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/semgr8s-service   ClusterIP   10.96.135.157   <none>        443/TCP   7s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/semgr8s   1/1     1            1           7s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/semgr8s-665dbb8756   1         1         1       7s

Once all resources are in READY state, you have successfully installed Semgr8s.

Enable namespaces⚓︎

To activate Semgr8s admission control for a namespace in default configuration, it is required to set the label semgr8s/validation=enabled. Either update the labels for existing namespaces directly via kubectl:

kubectl label namespace test-semgr8s semgr8s/validation=enabled --overwrite

Or extend the Kubernetes yaml files for target namespaces:

namespace.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: test-semgr8s
  labels:
    semgr8s/validation: enabled

It is recommended to exclude cluster operation critical namespaces such as kube-system and semgr8ns to avoid interruptions.

Testing⚓︎

Several test resources are provided under tests/demo/. Semgr8s only validates resources in namespaces with label semgr8s/validation=enabled:

kubectl apply -f tests/demo/00_test-namespace.yaml

input

tests/demo/00_test-namespace.yaml

---
apiVersion: v1
kind: Namespace
metadata:
  name: test-semgr8s
  labels:
    semgr8s/validation: enabled

output

namespace/test-semgr8s created

It denies creating pods with non-compliant configuration according to the local rules in charts/semgr8s/rules and remote rules configured under .application.remoteRules in charts/semgr8s/values.yaml:

kubectl apply -f tests/demo/40_failing-deployment.yaml

input

tests/demo/40_failing-deployment.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: forbiddenlabel-pod
  namespace: test-semgr8s
  labels:
    foo: bar
    semgr8s-test: forbidden-test-label-e3b0c44298fc1c
spec:
  containers:
  - image: busybox
    name: forbiddenlabel-container
    command: ["/bin/sh", "-ec", "sleep 1000"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 10001  # remove when using openshift or OKD 4
      runAsGroup: 20001  # remove when using openshift or OKD 4
      seccompProfile:  # remove when using Kubernetes prior v1.19, openshift or OKD 4
        type: RuntimeDefault  # remove when using Kubernetes prior v1.19, openshift or OKD 4
---
apiVersion: v1
kind: Pod
metadata:
  name: failing-testpod-1
  namespace: test-semgr8s
spec:
  containers:
  - image: busybox
    name: failing-testpod-1
    command: ["/bin/sh", "-ec", "sleep 1000"]
---
apiVersion: v1
kind: Pod
metadata:
  name: failing-testpod-2
  namespace: test-semgr8s
spec:
  containers:
  - image: busybox
    name: failing-testpod-2
    command: ["/bin/sh", "-ec", "sleep 1000"]
    securityContext:
      allowPrivilegeEscalation: true
      capabilities:
        drop:
          - ALL
      privileged: true
      readOnlyRootFilesystem: true
      runAsNonRoot: true
---
apiVersion: v1
kind: Pod
metadata:
  name: failing-testpod-3
  namespace: test-semgr8s
spec:
  containers:
  - image: busybox
    name: failing-testpod-3
    command: ["/bin/sh", "-ec", "sleep 1000"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
  hostNetwork: true

output

Error from server: error when creating "tests/demo/40_failing-deployment.yaml": admission webhook "semgr8s-svc.semgr8ns.svc" denied the request: Found 1 violation(s) of the following policies: 
* rules.test-semgr8s-forbidden-label
Error from server: error when creating "tests/demo/40_failing-deployment.yaml": admission webhook "semgr8s-svc.semgr8ns.svc" denied the request: Found 1 violation(s) of the following policies: 
* yaml.kubernetes.security.writable-filesystem-container.writable-filesystem-container
Error from server: error when creating "tests/demo/40_failing-deployment.yaml": admission webhook "semgr8s-svc.semgr8ns.svc" denied the request: Found 1 violation(s) of the following policies: 
* yaml.kubernetes.security.privileged-container.privileged-container
Error from server: error when creating "tests/demo/40_failing-deployment.yaml": admission webhook "semgr8s-svc.semgr8ns.svc" denied the request: Found 1 violation(s) of the following policies: 
* yaml.kubernetes.security.hostnetwork-pod.hostnetwork-pod

Compliantly configured resources on the other hand are permitted to the cluster:

kubectl apply -f tests/demo/20_passing-deployment.yaml

input

tests/demo/20_passing-deployment.yaml

---
apiVersion: v1
kind: Pod
metadata:
  name: passing-testpod-1
  namespace: test-semgr8s
spec:
  containers:
  - image: busybox
    name: passing-testpod-1
    command: ["/bin/sh", "-ec", "sleep 1000"]
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
      privileged: false
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 10001  # remove when using openshift or OKD 4
      runAsGroup: 20001  # remove when using openshift or OKD 4
      seccompProfile:  # remove when using Kubernetes prior v1.19, openshift or OKD 4
        type: RuntimeDefault  # remove when using Kubernetes prior v1.19, openshift or OKD 4

output

pod/passing-testpod-1 created

Cleanup⚓︎

To remove all resources of the admission controller run:

helm uninstall semgr8s -n semgr8ns
kubectl delete ns semgr8ns

output

release "semgr8s" uninstalled

Test resources are deleted via:

kubectl delete -f tests/demo/

output

namespace "test-semgr8s" deleted
pod "passing-testpod-1" deleted
Error from server (NotFound): error when deleting "tests/demo/40_failing-deployment.yaml": pods "forbiddenlabel-pod" not found
Error from server (NotFound): error when deleting "tests/demo/40_failing-deployment.yaml": pods "failing-testpod-1" not found
Error from server (NotFound): error when deleting "tests/demo/40_failing-deployment.yaml": pods "failing-testpod-2" not found
Error from server (NotFound): error when deleting "tests/demo/40_failing-deployment.yaml": pods "failing-testpod-3" not found

Features⚓︎

Audit mode⚓︎

It is possible to run Semgr8s in audit mode by which it will not block but only warn on non-compliant resources. As the logs contain an explicit error code, it is possible to alert on admission responses with warnings via typical monitoring solutions such as Prometheus or DataDog. This is particularly useful during rollout of Semgr8s and for enforcement of new policies.

To activate audit mode, set .application.enforce=false in the charts/semgr8s/values.yaml.

With the Semgrep login feature, you can connect Semgr8s to your Semgrep AppSec Platform account and use platform features such as private remote rules. To use the login feature, set .application.semgrepLogin=true in the charts/semgr8s/values.yaml and provide a Kubernetes generic secret semgrep-app-token containing a Semgrep agent token as token key:

kubectl create secret generic -n semgr8ns --from-literal=token=iamsupersecret semgrep-app-token

To generate a new token go to the API token settings on web platform and create a new token with Agent (CI) scope. Proceed with the Semgr8s installation as normal.

Autofix⚓︎

Semgr8s supports the Semgrep autofix feature. To use autofix, simply set .application.autofix=true in the charts/semgr8s/values.yaml and provide fix instructions for your rules. Semgr8s will attempt to fix resources before validation.

Technically, this is implemented via an additional mutating admission controller that is called before the validating admission controller.

Rules⚓︎

Rules form the core of Semgr8s functionality. They follow Semgrep syntax that provides an extensive pattern language. As admission requests resemble Kubernetes manifests, standard Kubernetes patterns can be directly used for Semgr8s. It is however important to keep some differences in mind. Rules can be provided in two different ways: remote rules and local rules. Remote rules are requested from their external sources such as the Semgrep registry upon validation. Local rules are provided as configmaps directly to Semgr8s.

Remote rules⚓︎

Remote rules are directly used from external sources. They can be individual rules (e.g. r/yaml.kubernetes.security.allow-privilege-escalation.allow-privilege-escalation) or entire rulesets (e.g. p/kubernetes). Common sources are:

For inspiration checkout Semgreps Kubernetes ruleset or the Kubernetes repository folder. They are added as a list under .application.remoteRules in charts/semgr8s/values.yaml. Simply reference the respective rule(set) as you would for a local installation, e.g. p/kubernetes. Remote rules can currently only be configured prior to deployment and changes require re-installation of Semgr8s. However, it is possible to use private rules via Semgrep login feature. Adding, changing, or modifying private rules and even rulesets propagates to the running Semgr8s installation.

Warning

Remote rules require requests to external resources. This introduces delays, may lead to unexpected denial for rules modified by the external authority, and can cause failures if these resources become unavailable.

Local rules⚓︎

Local rules are your custom written rules and added as configmaps with label semgr8s/rule=true to Semgr8s's namespace semgr8ns. They can either be provided prior to installation as files under charts/semgr8s/rules/ or added after deployment. Templates and selected rules are available under ./rules/.

Share your own rules

We hope to continuously extend the list of selected rules to facilitate policy creation. So, please contribute your own favorite rules via PR

Adding rules⚓︎

Local rules are provided as configmaps that are automatically written to local rule files in the semgr8s pod by the update job. Therefore, adding, modifying, or deleting local rules does not require an update of the deployment.

To add a new rule, simply create a configmap from a standard semgrep rule yaml file and add the label semgr8s/rule=true:

kubectl create configmap -n semgr8ns my-local-rule --from-file=path/to/rule.yaml
kubectl label configmap -n semgr8ns my-local-rule semgr8s/rule=true

Info

Semgr8s only updates its rules every 1min. Consequently, it can take up to 1min until the changes take effect. This accounts for adding, modifying, and deleting local rules.

Removing rules⚓︎

To delete a local rule, run:

kubectl delete configmap -n semgr8ns my-local-rule

It is also possible to delete all local rules via:

kubectl delete -n semgr8ns cm -l semgr8s/rule=true

Warning

Semgrep fails if no rules are provided and consequently deleting all local rules in absence of remote rules causes Semgr8s to fail.

Writing rules⚓︎

Semgr8s rules follow Semgrep syntax and, therefore, must comply with the Semgrep rule requirements. For convenience, admission requests are converted to yaml (just like manifest files) and consequently all rules should define yaml as language. A basic rule takes the form:

rules/template-rule.yaml

rules:
- id: template-rule
  message: Rule template.
  languages: [yaml]
  severity: INFO
  patterns:
    - pattern: |
        semgr8s-test: forbidden-test-label-e3b0c44298fc1c

In order to use the autofix feature, a fix value must be specified additionally:

rules/template-autofix-rule.yaml

rules:
- id: template-autofix-rule
  message: Rule template with autofix.
  languages: [yaml]
  severity: INFO
  patterns:
    - pattern: |
        semgr8s-test: forbidden-test-label-e3b0c44298fc1c
  fix: "semgr8s-test: allowed-test-label"

While admission requests purposely share similarities with Kubernetes manifest files, there is critical differences and additional information to consider when writing Semgr8s rules. More details on admission requests are provided in the respective conceptual section. In order to develop a rule for the actual admission object locally, create the admission object for your target resource resource.yaml via:

kubectl apply -f resource.yaml --dry-run=server -o yaml

Rules should be carefully tested before rollout to production.

Tip

Semgrep provides provides extensive learning resources for writing your own rules. Interactive development without local setup is supported via the online playground.

Testing rules⚓︎

Semgrep provides some convenient testing mechanisms out of the box. These can be leveraged to test rules locally before deployment. To validate rule syntax, Semgrep offers linting patterns:

semgrep scan --metrics-off --validate ./rules/

Testing functionality of rules requires tests. To assess the example rules, run:

semgrep scan --metrics=off --test ./rules/

Typical patterns⚓︎

Below, we share a few common and helpful patterns.

Restrict to resource type⚓︎

In order to restrict a rule to only certain resource types, simply prepend a pattern-inside expression for the desired resource type. In case of Pod resources, this takes the following form:

rules/forbidden-pod-label.yaml

rules:
- id: forbidden-pod-label
  message: Kubernetes pod with forbidden label. Any pod with label "semgr8s-test=forbidden-test-label-e3b0c44298fc1c" is denied. This label carries no meaning beyond testing and demonstration purposes.
  languages: [yaml]
  severity: INFO
  patterns:
    - pattern-inside: |
        ...
        kind: Pod
        ...
    # remaining pattern as normal
    - pattern-inside: |
        metadata:
          ...
    - pattern-inside: |
        labels:
          ...
    - pattern: |
        semgr8s-test: forbidden-test-label-e3b0c44298fc1c
  fix: "semgr8s-test: allowed-test-label"

It is also possible to instead use pattern-not-inside in order to exclude a rule for one specific resource type. Multiple in-scope resource types can be defined via metavariable regular expressions:

rules/forbidden-workload-label.yaml

rules:
- id: forbidden-workload-label
  message: Kubernetes workload with forbidden label. Any workload resource with label "semgr8s-test=forbidden-test-label-e3b0c44298fc1c" is denied. This label carries no meaning beyond testing and demonstration purposes.
  languages: [yaml]
  severity: INFO
  patterns:
    - pattern-inside: |
        ...
        kind: $KIND
        ...
    - metavariable-regex:
        metavariable: $KIND
        regex: (Pod|Deployment|ReplicaSet|DaemonSet|StatefulSet)
    # remaining pattern as normal
    - pattern-inside: |
        metadata:
          ...
    - pattern-inside: |
        labels:
          ...
    - pattern: |
        semgr8s-test: forbidden-test-label-e3b0c44298fc1c
  fix: "semgr8s-test: allowed-test-label"

Restrict to specific namespaces⚓︎

By prepending a pattern-inside, it is possible to restrict a rule to selected namespaces:

rules/forbidden-namespaced-label.yaml

rules:
- id: forbidden-namespaced-label
  message: Kubernetes resource with label forbidden in designated namespace. Any resource with label "semgr8s-test=forbidden-test-label-e3b0c44298fc1c" is denied for this namespace. This label carries no meaning beyond testing and demonstration purposes.
  languages: [yaml]
  severity: INFO
  patterns:
    - pattern-inside: |
        metadata:
          ...
          namespace: $NS
        ...
    - metavariable-regex:
        metavariable: $NS
        regex: (test-semgr8s|audit-semgr8s)
    # remaining pattern as normal
    - pattern-inside: |
        metadata:
          ...
    - pattern-inside: |
        labels:
          ...
    - pattern: |
        semgr8s-test: forbidden-test-label-e3b0c44298fc1c
  fix: "semgr8s-test: allowed-test-label"

This cannot extend beyond the enabled namespaces, but allows for more granular control on a per rule basis.

Usage⚓︎

Considerations⚓︎

Setup⚓︎

Requirements⚓︎

Get Code⚓︎

Configuration⚓︎

Installation⚓︎

Enable namespaces⚓︎

Testing⚓︎

Cleanup⚓︎

Features⚓︎

Audit mode⚓︎

Semgrep login⚓︎

Autofix⚓︎

Rules⚓︎

Remote rules⚓︎

Local rules⚓︎

Adding rules⚓︎

Removing rules⚓︎

Writing rules⚓︎

Testing rules⚓︎

Typical patterns⚓︎

Restrict to resource type⚓︎

Restrict to specific namespaces⚓︎