No Go? No Problem! Use Bash, Python, or Kubectl

Piotr

·

Follow

Published in

ITNEXT

·

8 min read

·

Jul 7, 2024

--

3

Listen

Share

Introduction

Developing Kubernetes operators traditionally require a deep understanding of Go and Kubernetes internals. This complexity often limits operator development to seasoned developers, leaving system administrators and platform engineers reliant on existing tools and solutions. Enter Shell Operator — a powerful open-source project that simplifies creating Kubernetes operators using familiar scripting languages like Bash, Python, and Perl.

By democratizing the development process, Shell Operator enables a broader range of professionals to build custom controllers and automate Kubernetes workflows. In this blog, we will explore how Shell Operator works, its integration with Crossplane and Terraform, and how it can streamline real-time pod monitoring as a practical example.

The companion video shows in detail how the setup process is done and how to recreate it in your environement. Repository with steps to reproduce.

Who Is This Blog For?

This blog is for Kubernetes enthusiasts, system administrators, DevOps engineers, and platform builders who are interested in adding one more valuable tool to their Kubernetes toolbelts.

In this blog, I will provide practical insights and step-by-step instructions how to use shell operator and other open source cloud-native projects to accomplish a realistic task.

Project Goal: Real-time Pod Monitoring

Our objective is to design and implement a real-time monitoring system for Kubernetes pods. This system will track the lifecycle events of pods, specifically focusing on detecting their creation and deletion in real-time. Upon detecting these events, the system will send immediate notifications to a designated Slack channel.

By leveraging open-source tools such as Shell Operator, Crossplane, and Terraform, we aim to create a robust and extensible solution that enhances cluster observability and ensures prompt responses to critical changes within the Kubernetes environment.

Here is what we want to build using shell-operator:

Shell Operator: Kubernetes Event Watcher

Shell Operator is an open-source project designed to simplify the automation of Kubernetes workflows. It enables users to create custom operators and controllers by writing scripts in familiar languages like Bash, Python, or Perl.

Shell Operator functions by watching for specific Kubernetes events, such as the creation, deletion, or update of resources, and then executing the corresponding scripts in response. This approach democratizes the development of Kubernetes operators, allowing a broader range of professionals to automate complex tasks without needing to write extensive Go code.

Key Features of Shell Operator

Shell Operator offers several key features that make it an invaluable tool for Kubernetes automation:

• Event Handling: Shell Operator can watch for various Kubernetes events, such as the creation, deletion, and updates of resources like pods, deployments, secrets, and custom resource definitions (CRDs).
• Script Execution: Upon detecting an event, Shell Operator executes user-defined scripts. These scripts can be written in any language, making it accessible to system administrators and developers who may be more comfortable with scripting languages rather than Go.
• Ease of Use: Shell Operator simplifies the process of creating Kubernetes operators by allowing users to define their automation logic using scripts. This makes it easier for those who may not be familiar with the complexities of Kubernetes internals to build custom operators and controllers.
• Flexibility: It can run any binary or script placed in the hooks directory, providing flexibility in how you respond to Kubernetes events.

By leveraging these features, Shell Operator empowers users to automate and streamline their Kubernetes workflows efficiently and effectively.

Setting Up Our Environment

In the video linked at the start of the blog, you can see how the environment is set up. To quickly recap:

• terraform is used to create a local kind cluster
• crossplane with crossplane http provider are used to send Slack notification to the slack webhook

Developing Shell Operator Hooks

Hooks in Shell Operator are executable scripts that are triggered by specific events in the Kubernetes cluster. These events can include the creation, update, or deletion of Kubernetes resources such as pods, services, or custom resources. The hooks are registered with Shell Operator and are executed when the specified events occur, allowing you to automate responses to changes in the cluster.

One of the key advantages of Shell Operator is the ability to write hooks using simple shell scripts. This makes it accessible for system administrators and DevOps engineers who may not have extensive programming knowledge but are proficient with scripting languages like Bash.

Here’s a basic example of how to write a shell script hook that reacts to the creation of a pod and sends a notification to Slack.

#!/usr/bin/env bash

# Define the secret details
SECRET_NAMESPACE="default"
SECRET_NAME="webhook-secret"
SECRET_KEY="webhook-url"

if [[ $1 == "--config" ]]; then
 cat <<EOF
configVersion: v1
kubernetes:
- apiVersion: v1
  kind: Pod
  executeHookOnEvent: ["Added"]
EOF
else
 # Retrieve the webhook URL from the secret
 WEBHOOK_URL=$(kubectl get secret $SECRET_NAME -n $SECRET_NAMESPACE -o jsonpath="{.data.$SECRET_KEY}" | base64 -d)

 if [[ -z "$WEBHOOK_URL" ]]; then
  echo "Error: WEBHOOK_URL is empty or the secret does not exist."
  exit 1
 fi

 # Iterate over all events in the binding context
 for i in $(seq 0 $(($(jq length $BINDING_CONTEXT_PATH) - 1))); do
  podName=$(jq -r .[$i].object.metadata.name $BINDING_CONTEXT_PATH)
  podNamespace=$(jq -r .[$i].object.metadata.namespace $BINDING_CONTEXT_PATH)
  creationTimestamp=$(jq -r .[$i].object.metadata.creationTimestamp $BINDING_CONTEXT_PATH)
  image=$(jq -r .[$i].object.spec.containers[0].image $BINDING_CONTEXT_PATH)
  # Create a YAML for the CRD
  cat <<EOF | kubectl apply -f -
apiVersion: http.crossplane.io/v1alpha2
kind: DisposableRequest
metadata:
  name: slack-webhook-creation-$podName
spec:
  deletionPolicy: Orphan
  forProvider:
    url: $WEBHOOK_URL
    method: POST
    body: '{
      "channel": "#app-notify",
      "username": "webhookbot",
      "text": "A new pod has been created.\n\nPod Name: $podName\nNamespace: $podNamespace\nCreation Timestamp: $creationTimestamp\nImage: $image",
      "icon_url": "https://example.com/path/to/icon.png"
    }'
EOF

  echo "CRD created for created pod '${podName}' with webhook URL."
 done
fi

A very similar script can be created for the deletion event.

Create http request to send notification to Slack

In our project, we use Crossplane’s HTTP provider to send notifications to Slack when pods are created or deleted in our Kubernetes cluster.

Crossplane is an infrastructure management tool that allows us to define and manage cloud infrastructure using Kubernetes Custom Resource Definitions (CRDs). By leveraging Crossplane, we can create an HTTP request to send a notification to Slack in a more structured and manageable way.

Read this blog if you are interested in learning more about Crossplane:

First, ensure that the Crossplane HTTP provider is installed and configured in your Kubernetes cluster. This provider allows us to create HTTP requests as Kubernetes resources.

Check out the companion repository to see how everything is plugged together.

To send a notification to Slack, we define a DisposableRequest resource. This resource specifies the details of the HTTP request, including the URL, method, and body. Here's how you can set up the HTTP request to send a notification to Slack:

apiVersion: http.crossplane.io/v1alpha2
kind: DisposableRequest
metadata:
  name: slack-webhook-creation-$podName
spec:
  deletionPolicy: Orphan
  forProvider:
    url: $WEBHOOK_URL
    method: POST
    body: '{
      "channel": "#app-notify",
      "username": "webhookbot",
      "text": "...",
      "icon_emoji": ":ghost:"
    }'

Building and Pushing the Shell Operator Docker Image

To deploy your custom hooks in a Kubernetes environment using Shell Operator, you need to package them into a Docker image. This Docker image will contain the Shell Operator binary and your custom hook scripts. Here’s how to create, build, and push the Docker image.

The Dockerfile is a blueprint that defines the environment and the steps required to set up the Shell Operator along with your custom hooks. Here’s an example Dockerfile:

# Use the official Shell Operator image as the base image
FROM flant/shell-operator:latest
COPY hooks /hooks
RUN chmod +x /hooks/*

In this Dockerfile:

• Base Image: We start from the official Shell Operator image provided by Flant.
• Copy Hooks: We copy our custom hook scripts into the /hooks directory inside the container.
• Set Executable Permissions: We ensure that the hook scripts are executable by setting the appropriate permissions.

Build and Push the Docker Image

With the Dockerfile in place, the next step is to build the Docker image. Use the following command to build the image:

docker build -t your-dockerhub-username/shell-operator-hooks:v1 .
docker push your-dockerhub-username/shell-operator-hooks:v1

This command builds the Docker image using the Dockerfile in the current directory (.) and tags it as your-dockerhub-username/shell-operator-hooks:v1.

Deploying Shell Operator

To ensure the Shell Operator has the necessary permissions to interact with Kubernetes resources, we need to set up Role-Based Access Control (RBAC) rules. These rules define what actions the Shell Operator can perform on specific resources within the cluster. Let’s walk through the required RBAC configurations.

Service Account

First, we create a Service Account for the Shell Operator. The Service Account allows the Shell Operator to authenticate and interact with the Kubernetes API securely.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: monitor-pods-acc
  namespace: default

ClusterRole

Next, we define a ClusterRole that specifies the permissions required by the Shell Operator. This includes permissions to get, list, watch, create, update, patch, and delete pods, namespaces, secrets, and Crossplane DisposableRequests CRD.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: pod-manager-clusterrole
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["http.crossplane.io"]
    resources: ["disposablerequests"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

ClusterRoleBinding

To bind the ClusterRole to the Service Account, we create a ClusterRoleBinding. This binding grants the Service Account the permissions defined in the ClusterRole.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: monitor-pods-acc-clusterrolebinding
subjects:
  - kind: ServiceAccount
    name: monitor-pods-acc
    namespace: default
roleRef:
  kind: ClusterRole
  name: pod-manager-clusterrole
  apiGroup: rbac.authorization.k8s.io

Shell Operator Pod Manifest

With the RBAC rules in place, we can now deploy the Shell Operator pod using the defined Service Account. This manifest configures the Shell Operator to run with the necessary permissions to manage pod lifecycle events and send notifications to Slack.

apiVersion: v1
kind: Pod
metadata:
  name: shell-operator
  namespace: default
spec:
  serviceAccountName: monitor-pods-acc
  containers:
    - name: shell-operator
      image: your-dockerhub-username/shell-operator-hooks:latest
      imagePullPolicy: Always

In this manifest:

• Service Account: The serviceAccountName is set to the Service Account we created (monitor-pods-acc), ensuring the pod has the required permissions.
• Container: The container uses the Docker image we built and pushed (your-dockerhub-username/shell-operator-hooks:latest), which includes the Shell Operator and our custom hooks.

Applying the Manifests

To apply these manifests to your Kubernetes cluster, use the following kubectl apply commands:

kubectl apply -f service-account.yaml
kubectl apply -f cluster-role.yaml
kubectl apply -f cluster-role-binding.yaml
kubectl apply -f shell-operator-pod.yaml

Check Slack for notifications

Now we are ready to test, creating a new pod, for example kubectl run test-pod-nginx image:nginx --restart=Always should trigger the following slack notification

A new pod has been created.

Pod Name: test-pod-nginx
Namespace: default
Creation Timestamp: 2024-07-06T13:32:18Z
Image: nginx

whereas deleting the same pod should result in:

A pod has been deleted.
Pod Name: test-pod-nginx
Namespace: default
Deletion Timestamp: 2024-07-06T13:32:40Z

Closing Thoughts

In this blog, we successfully built a real-time Kubernetes pod monitoring solution using Shell Operator, Crossplane, and Terraform. By leveraging Shell Operator, we created custom hooks to handle Kubernetes events and send notifications to Slack, enhancing our cluster’s observability.

We combined the power of Crossplane and Terraform for infrastructure management, demonstrating a seamless integration of multiple tools within the Kubernetes ecosystem. We didn’t need to create our own controller from scratch; instead, utilized existing skills with shell scripting to design the automation.

Thanks for taking the time to read this post. I hope you found it interesting and informative.

🔗 Connect with me on LinkedIn

🌐 Visit my Website

📺 Subscribe to my YouTube Channel