devOops.blog

Exposing an app on GKE using Ingress and TLS Termination

Shay Ulmer — Tue, 17 Jan 2023 09:24:53 GMT

I recently needed to expose an application that runs on GKE to the outside world. To do that, I had to learn and use GKE's way of working with Ingress and Load Balancers. Although it's not too complicated (compared to other Ingress implementations), there are some technicalities that we need to resolve first.

Overview

Today we will expose a simple "hello world" app (shayulmer/loadbalancer-tester container image) to the world, using TLS on GKE. This app serves as a web server that prints out the hostname of the running container, so we would be able to see the GCP Load Balancer in action.

The application itself runs on HTTP, but we will expose it over HTTPS using a GCP Managed Certificate and TLS Termination on our Ingress Load Balancer. We will also use Google Cloud Armor to limit traffic to the Load Balancer.

Prerequisites

A running GKE Cluster
Basic Kubernetes knowledge - understanding of Deployments, Services, Ingress.
gcloud CLI configured and logged in to the relevant GCP Project.

The Helm Chart

The following repository contains a helm chart for today's experiment: https://github.com/shay-ul/gke-ingress-example

Note: If you are going to deploy this helm chart on your environment, make sure to override the values.yaml file with your specific environment's details.

Since we want to see what's being deployed to GKE, we will not go over the helm chart itself. Instead, we will discuss the Kubernetes Manifests which are being generated from this chart. We will generate the manifests using this command:

helm template hello-world . -f values.yaml

Step 1: Prepare the GCP Infrastructure

Since we want to issue a public Google-managed certificate, we have to use an external HTTP(S) Load Balancer[1]. When working with External Load Balancers, we have to consider two things:

Public Static IP - External GCP Load balancers (unlike their AWS counterparts) will not receive a dynamic DNS that points to a dynamically allocated IP address[2]. Instead, you have to manually point a DNS Record to the IP address assigned to the Load Balancer. To verify the IP address will never be changed (and the DNS will always point to the correct address), we have to request a Global Static Public IP. This address will be attached to our Load Balancer.
Manage IP Whitelist - Our application will be widely accessible over the internet unless we limit who can access our application. Since this is just a test run, we will utilize Google's Cloud Armor to block traffic which is not originated from IP addresses we specifically allow.

Step 1.a - Request a Global Static IP Address

To do this, we will run the following command:
gcloud compute addresses create gke-ingress-example-address --global

After the request is completed, we can observe the allocated IP using this command:
gcloud compute addresses describe gke-ingress-example-address --global

Step 1.b - Configure Cloud Armor Policy

Here we will create a new network security policy:

gcloud compute security-policies create gke-ingress-example-security-policy`

Next we will create a default rule to block all traffic, with the lowest priority allowed (2147483647):

gcloud compute security-policies rules update 2147483647 --security-policy gke-ingress-example-security-policy --action "deny-502"

And finally, we will add our "Allow" rule to allow traffic from known IPs:

gcloud compute security-policies rules create 1000 
    --security-policy gke-ingress-example-security-policy 
    --description "allow traffic" 
    --src-ip-ranges "192.0.2.0/24"  
    --action "allow"

Make sure to insert your specific allowed IP Addresses or subnets.

Step 2: The Deployment

Let's dig into our Kubernetes Manifests which were generated by the Helm Template. This is our deployment, which is very basic:

# Source: gke-ingress-example/templates/deployments/deployment.yamlapiVersion: apps/v1kind: Deploymentmetadata:  name: hello-world-deployment  labels:    app: hello-worldspec:  replicas: 3  selector:    matchLabels:      app: hello-world  template:    metadata:      labels:        app: hello-world    spec:      containers:      - name: hello-world-deployment        image: shayulmer/loadbalancer-tester        ports:        - containerPort: 80

This is the Kubernetes Service which exposes the deployment. Notice how it references a "backend-config", which we will deploy and discuss in a moment:

---# Source: gke-ingress-example/templates/services/service.yamlapiVersion: v1kind: Servicemetadata:  name: hello-world-service  labels:    app: hello-world  annotations:    cloud.google.com/backend-config: '{"default": "hello-world-gke-backend-config"}'spec:  type: NodePort  selector:    app: hello-world  ports:    - protocol: TCP      port: 80      targetPort: 80

Note: External GKE Load Balancers will not work with ClusterIP Services [4], therefore our service type is "NodePort".

Now it's time for the Backend Config which is referenced by the Service. This config specifies the desired Cloud Armor Security Policy (which we created earlier):

---# Source: gke-ingress-example/templates/crds/gke-backend-config.yamlapiVersion: cloud.google.com/v1kind: BackendConfigmetadata:  name: hello-world-gke-backend-configspec:  securityPolicy:    name: gke-ingress-example-security-policy

BackendConfigs can be very important if your application does not return HTTP 200 response code on the "/" path. The Load Balancer health check defaults to this path, so if your app will not return 200 OK on "/", the Load Balancer will show "unhealthy backends" and will not work. To fix this, you will need to directly configure a different health check on the BackendConfig: https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#direct_hc

Since we want to redirect all traffic to HTTPS, we also need a Kubernetes Custom Resource named FrontendConfig. We will later reference this config in our Ingress object:

---# Source: gke-ingress-example/templates/crds/gke-frontend-config.yamlapiVersion: networking.gke.io/v1beta1kind: FrontendConfigmetadata:  name: hello-world-gke-frontend-configspec:  redirectToHttps:    enabled: true

We also need to create a GCP Managed Certificate. This certificate will be used on the Ingress object. The most crucial part of this manifest is specifying the correct domain, since the certificate will be verified and signed against the specified domain name.

---# Source: gke-ingress-example/templates/crds/gke-managed-certificate.yamlapiVersion: networking.gke.io/v1kind: ManagedCertificatemetadata:  name: hello-world-managed-certspec:  domains:    - gke-ingress-example.devoops.blog

And lastly, our Ingress which puts it all together:

---# Source: gke-ingress-example/templates/ingress/ingress.yamlapiVersion: networking.k8s.io/v1kind: Ingressmetadata:  name: hello-world-ingress  annotations:    kubernetes.io/ingress.class: "gce"    networking.gke.io/managed-certificates: hello-world-managed-cert    kubernetes.io/ingress.global-static-ip-name: gke-ingress-example-address    networking.gke.io/v1beta1.FrontendConfig: hello-world-gke-frontend-configspec:  rules:  - host: gke-ingress-example.devoops.blog    http:      paths:      - path: /*        pathType: ImplementationSpecific        backend:          service:            name: hello-world-service            port:              number: 80

See how under "Annotations", the Ingress references the names of the following:

Our Ingress class ("gce", which is the External Load Balancer ingress class)
The managed certificate which will be used to initiate TLS Sessions
The Global Static IP address which should be attached to the Load Balancer.
The FrontendConfig which redirects HTTP traffic to HTTPS.

After reviewing the generated manifests, we may now install our Helm Chart. Verify you are in the desired Namespace and run:

helm install hello-world . -f values.yaml

Step 3: Signing the certificate

Google signs certificate requests with DNS validation[5]. All we have to do is go to our DNS provider and make sure our ingress domain name (in our case - gke-ingress-example.devoops.blog) points to the global static IP address we requested earlier. After this is done, we will wait for the ManagedCertificate to become "Active". This can take some time due to DNS propagation delay.

You can check the ManagedCertificate status by running:

kubectl get ManagedCertificate

Once the certificate is signed (Active), it's going to take a few more minutes for GCP to configure the certificate on the load balancer (this will happen automatically).

Step 4: Successful Deployment!

we can finally access our application securely (from Cloud-Armor whitelisted IP addresses only):

Pro tip: Refresh the application (or browse from multiple tabs) to see the Load Balancer in action. The hostname of the pod changes as the Load Balancer distributes traffic to different pods.

Mission Accomplished! It was quite a challenge to figure all of this out, but once the concepts are laid out - I believe it's not too hard to understand and deploy.

The best part of this experiment is that now we have a valid Helm chart template for exposing an application on GKE. To take this to the next level, we can even sync this chart to our cluster with a GitOps tool like ArgoCD, but this is a topic for another post :)

]]>

First post: Creating a simple web app to list kubernetes resources

Shay Ulmer — Mon, 12 Sep 2022 13:26:03 GMT

Today we're going to overview a fun experiment that involves many aspects of working with Kubernetes. I had a very interesting use case where I needed a simple and easy way for my developers to see some Kubernetes resources. To do that, I created a quick solution which is basically a very simple flask app that does one thing: it lists resources on the Kubernetes cluster it runs on.

I personally thought that this is a specifically interesting experiment since it involved learning how to utilize kubernetes Service Accounts, RBAC, python client, Ingress and more.

All the resources in this article are already hosted on github - Click here to view.

TL;DR

to deploy and check out the application, run:

git clone https://github.com/shay-ul/kubernetes-pods-extractor.gitcd kubernetes-pods-extractorkubectl apply -f manifests/

Prerequisites

A running Kubernetes cluster.
Basic Kubernetes knowledge (i.e - How to apply manifests).

Step 1: Prepare the Kubernetes resources

First, let's create our namespace:

kind: NamespaceapiVersion: v1metadata:  name: kubernetes-pods-extractor

Since our application will run inside the cluster, we will need to create a dedicated Service Account that the application will use:

apiVersion: v1kind: ServiceAccountmetadata:  name: kubernetes-pods-extractor-sa  namespace: kubernetes-pods-extractor

Now we need to create a ClusterRole that will enable our ServiceAcccount to get the information needed on pods:

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  # "namespace" omitted since ClusterRoles are not namespaced  name: kubernetes-pods-extractor-readerrules:- apiGroups: [""]  resources: ["pods"]  verbs: ["get", "watch", "list"]

Since we now have a ServiceAccount and a ClusterRole, we need to couple those two together, we will do so using ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: kubernetes-pods-extractor-clusterrolebindingsubjects:- kind: ServiceAccount  name: kubernetes-pods-extractor-sa  namespace: kubernetes-pods-extractor roleRef:  kind: ClusterRole  name: kubernetes-pods-extractor-reader   apiGroup: rbac.authorization.k8s.io

We are now ready to create our application.

Step 2: Working with Flask and the Kubernetes python client

Let's dive deep into our main script (src/extract_pods.py).Our first function is "get_kube_config":

def get_kube_config():    try:        config.load_incluster_config()    except config.ConfigException:        try:            config.load_kube_config()        except config.ConfigException:            raise Exception("Could not configure kubernetes python client")    v1 = client.CoreV1Api()    return v1

This function will basically load the ServiceAccount token if it runs inside the cluster, or your local kubectl context if you run the script locally.

How will the Kubernetes client know which one to choose? Well, our script just tries both methods (try-except). If it can't load the in-cluster config, it will then go and look for the kubernetes context (config.load_incluster_config vs config.load_kube_config).

When we ask the Kubernetes client to load the in-cluster config, it basically looks for the ServiceAccount token which should be loaded to "/var/run/secrets/kubernetes.io/serviceaccount/token" file. If the ServiceAccount was configured correctly on the pod level, we should have no problems querying the Kubernetes API from within our pod.

Our next function is the main function, which will call our previous explained get_kube_config function. It will then iterate over all pods in the cluster and create a dedicated list (pods_details_list). This list will be comprised from dictionaries, each one will hold the specific details of a specific pod: name, namespace, ip, node, status.

def main():    v1 = get_kube_config()    pods = v1.list_pod_for_all_namespaces(watch=False)    pods_details_list = []    for pod in pods.items:        status = get_pod_status(pod)        dict = {            "name": pod.metadata.name,            "namespace": pod.metadata.namespace,            "ip": pod.status.pod_ip,            "node": pod.spec.node_name,            "status": status        }        pods_details_list.append(dict)    return (pods_details_list)

The main function calls for one more function called get_pod_status:

def get_pod_status(pod):    status = pod.status.phase    for container_status in pod.status.container_statuses:        if container_status.started is False or container_status.ready is False:            waiting_state = container_status.state.waiting            if waiting_state is not None:                status = waiting_state.reason    return status

This function is basically a workaround for fetching the pod status. See, when kubectl get pod fetches a pod status, it does a similar thing. It looks for containers failing inside the specific pod, then reports back the specific problematic container status as the entire pod status. This is what we're doing here, since the python kubernetes client won't give us a simpler method to get this information.

The next part will be the entry function for the entire script. Since our script will be ran upon a HTTP request to Flask, we need to initialize our Flask app:

@app.route('/')def hello_world():    return render_template('index.html', pods_list=main())if __name__ == '__main__':    app.run(host="0.0.0.0", port=8080)

The most important line in this function is return render_template('index.html', pods_list=main()).

This line will be executed when an HTTP request hits our flask application, and it will render a JINJA2 template file (src/templates/index.html) and forward a list called pods_list as a variable to this template. The list contains the return values from our "main" function, described above.

Now that we understand our main script, we can have a look at our template (src/templates/index.html):

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.1.3/dist/css/bootstrap.min.css" integrity="sha384-MCw98/SFnGE8fJT3GXwEOngsV7Zt27NXFoaoApmYm81iuXoPkFOJwJ8ERdknLPMO" crossorigin="anonymous"><table class="table table-striped">    <thead>      <tr>        <th scope="col">Name</th>        <th scope="col">Namespace</th>        <th scope="col">IP</th>        <th scope="col">Node</th>        <th scope="col">Status</th>      </tr>    </thead>    <tbody>        {% for pod in pods_list %}        <tr>        <td style="width: 10%;">{{ pod['name'] }}</td>        <td style="width: 10%;">{{ pod['namespace'] }}</td>        <td style="width: 10%;">{{ pod['ip'] }}</td>        <td style="width: 10%;">{{ pod['node'] }}</td>        <td style="width: 10%;">{{ pod['status'] }}</td>        </tr>    {% endfor %}    </tbody>  </table>

In this template, we are using basic bootstrap so our table will look pretty.Our pod data is populated since we are iterating over our pods list which is recieved as an input to this template ({% for pod in pods_list %}).

Step 3: Build and package our application

our application is ready, but we now need to package it to a container image. To do so, we need to create a new Dockerfile:

FROM python:3.7-slim-busterCOPY src /app/COPY requirements.txt /app/requirements.txtRUN apt-get update \    && pip install --upgrade pip \    && pip install --no-cache-dir -r app/requirements.txtWORKDIR /app/CMD ["python3", "extract_pods.py"]

As you can see, we will build our image on top of python:3.7 image and our application will run as a simple python script when our container starts.

We will also need a requirements.txt file that will hold our pip dependencies:

kubernetes==24.2.0flask==2.2.2

We are only using two packages, which makes our requirements file kind of small :)

You can now build and push your container image to any registry. If you wish to avoid building and pushing your own image, you can use the image published from my source repository:

shayulmer/kubernetes-pods-extractor:latest

This image is automatically built and pushed using Github Actions on any push to the main branch on my source repostory (some very basic CI, you can check it out here)

Step 4: Deploy our application

Finally, we can now apply our application to Kubernetes, using this Deployment:

apiVersion: apps/v1kind: Deploymentmetadata:  name: kubernetes-pods-extractor  namespace: kubernetes-pods-extractor  labels:    app: kubernetes-pods-extractorspec:  replicas: 1  selector:    matchLabels:      app: kubernetes-pods-extractor  template:    metadata:      labels:        app: kubernetes-pods-extractor    spec:      containers:      - name: kubernetes-pods-extractor        image: shayulmer/kubernetes-pods-extractor:latest      serviceAccountName: kubernetes-pods-extractor-sa

Our ServiceAccount is mounted to our pod in the last line of this file.

And now we can expose the deployment using a service:

apiVersion: v1kind: Servicemetadata:  annotations:  name: kubernetes-pods-extractor-service   namespace: kubernetes-pods-extractorspec:  ports:  - name: http    port: 80    protocol: TCP    targetPort: 8080  selector:    app: kubernetes-pods-extractor  type: ClusterIP

Step 5: We are done!

Everthing is ready for us to access our application. All we need is to run:

kubectl port-forward svc/kubernetes-pods-extractor-service 8081:80 -n kubernetes-pods-extractor

Access http://localhost:8081 and behold!

If we wish to expose this service to (potentially) the entire world, we can do so using an Ingress object, but this is a topic for our next post :)

]]>

What is this?

Shay Ulmer — Tue, 06 Sep 2022 09:13:43 GMT

The most interesting aspect of DevOps is the fact that if you ask 3 different DevOps engineers what DevOps is, you will probably receive 5 different answers.There are more philosophies around DevOps than I could count, and most of them are correct because DevOps is actually a large mix of tools, methodologies, ideas and views.

Some say its a form of Software Engineering, but I personally believe that DevOps is a lot about bodging. You have to learn how to pry, twist and bend new tools and technologies for your needs. As the name of the blog suggests this involves trying (and failing) a lot until things work out your way.

You have to have the agility to adapt to new technologies and tools fairly fast, and if youre a busy DevOps engineer, youll soon learn that diving deep into different stuff is going to be a lot harder once you hold an entire set of tools and infrastructures. You have to be very knowledgeable in every aspect of your work, and in other peoples work as well which can be very challenging.

Your main ambition as a DevOps engineer should be helping your Dev team achieve better results faster while making sure your workload runs reliably. Theres an entire ecosystem you must design, build and maintain to achieve this goal (Virtualization\ Containerization\Orchestration, Monitoring\Alerting\Logging, CI/CD and more).

This blog will contain my personal experience with various technical challenges and how I solved them. My bodge solutions may not work for every case, but they may help understanding some concepts.

]]>