`k8s` — Kubernetes¶

This backend deploys challenges to a Kubernetes cluster. Each challenge is deployed under its own namespace, and exposed via either a NodePort service or an Ingress object, depending on the protocol specified by the challenge. No accommodations are currently being made in case of NodePort conflicts—it is recommended that challenges are deployed to an isolated cluster (you should be doing this anyways since Kubernetes currently does not have hard multi-tenancy). A NetworkPolicy is also created to prevent network traffic from outside a challenge’s namespace reaching any containers which are not explicitly exposed.

Like with rCDS’s Docker integration, the Kubernetes backend does not have a dependency on any system commands (e.g. kubectl); having a kubeconfig in the standard location is all that is necessary.

Configuration¶

The only required option is domain—NodePort services must be reachable on this domain, and the cluster’s ingress controller must be reachable on its subdomains. For example, if domain is set as example.com, then example.com must accept incoming TCP connections to NodePort services, and chall.example.com must be routed through the cluster’s ingress controller. It is your responsibility to set up the ingress controller.

Additional annotations on ingress and service objects can be specified through the annotations key, and affinity and tolerations on pods can be set through affinity and tolerations, respectively.

See the Options Reference for more details.

Recommended Cluster Configuration¶

RBAC¶

As always, we recommend running rCDS from a CI service; thus, rCDS will need to authorize with your Kubernetes cluster. We have provided a ClusterRole which grants the minimum privileges required by the Kubernetes backend (also accessible here):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: rcds
rules:
- apiGroups: [""]
  resources: ["services", "namespaces"]
  verbs: ["list", "get", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["list", "get", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["list", "get", "watch", "create", "update", "patch", "delete"]

Cluster Operator¶

We recommend Google Kubernetes Engine, because it supports restricting of the metadata server by Kubernetes service account, a feature called Workload Identity. This prevents SSRF from escalating into takeover of compute resources.

Ingress¶

Traefik is recommended as an ingress controller, since it is very configurable and supports JSON access logging for easy visibility into your challenges with your cluster operator’s logging solution. Consider manually issuing a wildcard LetsEncrypt certificate and setting it as the default. Then, set annotations on ingresses to use TLS, and configure Traefik to upgrade HTTP to HTTPS for full HTTPS on challenges.

Note

By default, Traefik will attempt to auto-detect Content-Type; apply the contentType middleware to disable this behavior if it breaks your challenges.

Options Reference¶

http://rcds.redpwn.com/schemas/backends/k8s/options.yaml
type	object
properties
kubeContext	The name of the context in the kubeconfig file to use. If unset, defaults to current context in kubeconfig file.
	type	string
namespaceTemplate	Template for the namespace to create for each challenge. Evaluated using Jinja. The challenge’s config is available under the variable challenge
	type	string
	default	rcds-{{ challenge.id }}
domain	Domain that the cluster is accessible at. NodePorts must be accessible on this domain, and ingresses will be created on subdomains.
	type	string
annotations	Annotations for Kubernetes objects that are created.
	type	object
	properties
	ingress	#/definitions/annotation
	service	#/definitions/annotation
tolerations	Kubernetes tolerations applied to challenge pods.
	type	array
affinity	Kubernetes affinities applied to challenge pods.
	type	object
definitions
annotation	type	object
	additionalProperties	type	string

Raw schema:

$schema: http://json-schema.org/draft-07/schema#
$id: http://rcds.redpwn.com/schemas/backends/k8s/options.yaml
type: object
definitions:
  annotation:
    type: object
    additionalProperties:
      type: string
properties:
  kubeContext:
    type: string
    description: >-
      The name of the context in the kubeconfig file to use. If unset, defaults
      to current context in kubeconfig file.
  namespaceTemplate:
    type: string
    description: >-
      Template for the namespace to create for each challenge. Evaluated using
      Jinja. The challenge's config is available under the variable `challenge`
    default: "rcds-{{ challenge.id }}"
  domain:
    type: string
    description: >-
      Domain that the cluster is accessible at. NodePorts must be accessible on
      this domain, and ingresses will be created on subdomains.
  annotations:
    type: object
    description: >-
      Annotations for Kubernetes objects that are created.
    properties:
      ingress:
        $ref: "#/definitions/annotation"
      service:
        $ref: "#/definitions/annotation"
  tolerations:
    type: array
    description: >-
      Kubernetes tolerations applied to challenge pods.
  affinity:
    type: object
    description: >-
      Kubernetes affinities applied to challenge pods.
required:
  - domain

k8s — Kubernetes¶