Deployment Strategies

The Kubernetes Deployment object is probably the resource we use most often, at least until we start building more advanced clusters that leverage service meshes. It’s a rock-solid native Kubernetes object that helps us declare what a set of Pods should look like that run a workload. It provides the famous Kubernetes control loop that keeps the Pods we’ve declared running, and it also offers some basic functionality for safely updating a workload.

Recall that a Deployment, under the hood, is managing ReplicaSets, which you can think of as versions of our workload configuration. When we make a change to that configuration, a new ReplicaSet is created, and a previous ReplicaSet is ultimately deprecated. Deployments attempt to do this safely using one of two methods:

• RollingUpdate (the default): The new ReplicaSet is gradually scaled up to the desired number of Pods, as the old ReplicaSet is gradually scaled down. We can also influence this type of update by limiting how many Pods we’ll tolerate as unavailable (with MaxUnavailable) and how many additional Pods we will allow during the update (with MaxSurge).

• Recreate: With this strategy, the entire existing ReplicaSet is scaled down and terminated before the new one is created. This is sometimes helpful if you want a clean cut off of traffic between different versions of your workload.

Often when we’re first trying Kubernetes, we learn how to implement versions of the canary and blue-green patterns by combining multiple Deployments with a Service object.

For example, we can run a blue Deployment and a green Deployment, and switch between them easily with a Service selector. Or we can run a canary Deployment with a smaller number of Pods, and let the Service object select this along with a larger production Deployment. But these techniques don’t scale well, and they require constant manual intervention to manage. This is where Argo’s automation can help.

The Rollout Object

Argo provides us with a new custom resource definition (CRD): the Rollout.

Essentially this object combines everything we can declare in a Deployment object with a much more advanced strategy definition. Within the strategy we can now describe the steps required to successfully rollout updates using the canary or blue-green patterns, including traffic splitting and approval steps. Let’s walk through a basic example to see how this works!

Prerequisites

To follow along, you’ll need access to a Kubernetes cluster. I’m normally a fan of Kind, or even Minikube, but when writing this post I struggled to get local forwarding of the LoadBalancer to work reliably enough to actually demonstrate traffic splitting. You might have more success than me and you’re welcome to try! But full disclosure, I spun up a GKE cluster in the end.

Installing Argo Rollouts

To set up Argo Rollouts we’ll create a namespace for the Argo controller, and we’ll install the other CRDs we need:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

We’ll also install the Rollouts plugin for kubectl, which will give us access to the kubectl argo rollouts sub-commands. You can obtain this from the releases page, or if you’re using Homebrew just run:

brew install argoproj/tap/kubectl-argo-rollouts

Creating a Rollout

We’re going to create a Rollout object that uses the rather excellent Argo Rollouts web app. This app gives us a really nice visualisation of what’s happening as we release or rollback updates. We’ll also create a LoadBalancer object so we can access the app in a browser. Let’s start by creating the rollout.yaml file below:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m

As you can see, most of this spec looks very much like a Deployment object. The big difference is the strategy section, which is specific to the Rollout CRD. In this section we specify the canary pattern, and then define the steps that we want for a successful rollout as a list. These are basically the automation instructions the controller will follow when we want to rollout an update.

• First we set the weight of the canary to 20. In other words, we ask for 20% of the available Pod replicas to match the canary definition. Elsewhere in the spec we can see there are 5 Pod replicas, so 1 of them will match the canary. Next we have an empty pause definition, which means an indefinite pause; in other words, manual intervention will be required here to promote the rollout and continue with the next steps.

• We then proceed with the rest of the steps in the canary. We set the weight to 40% (2 of 5 Pods), and wait for 10 seconds. Then we set the weight to 60% (3 of 5 Pods) and wait for 10 seconds. Then 80% and another 10 seconds, and finally the canary process will complete and all Pods in the Rollout will match the new definition.

A cognitive hurdle I had to get over here is to figure out why we only have a single Pod spec. After all, if we’re defining a canary pattern, shouldn’t there be separate production and canary deployments? And of course, this is the beauty and simplicity of the Argo Rollout.

Every rollout starts as a canary, and eventually becomes production.

We’ll see this in a moment, when the first time we create this object we just skip to having all of our Pods running the rollouts-demo:blue container, but when we perform the first change, we’ll see the canary logic in action.

Okay, next we need a Service object so we can access the workload. This is just a plain old regular LoadBalancer we’ll save as service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
  type: LoadBalancer

Once we’ve applied both of these objects to our cluster, we can watch the status of our Rollout object with this command:

kubectl argo rollouts get rollout rollouts-demo --watch

Because this is the initial creation of the object, we immediately scale up to 100% of the replicas running the rollouts-demo:blue container. Remember - the canary logic is only applied to updates, not to the initial creation.

I mentioned earlier that the demo wep app supplied by Argo Rollouts is actually very good, and that’s because it provides a very nice visualisation of the requests being made by a web browser and the version of the Pod that’s serving them.

Grab the external IP of the rollouts-demo service with kubectl get svc and hopefully, you’ll see something like this:

Updating a Rollout

Now it’s time to do our first update! Just like with a Deployment object, a Rollout is managing versions of our Pods using a ReplicaSet object. Right now we just have a single ReplicaSet, and if we make a change, a new ReplicaSet will be created. So let’s patch our Rollout object and change the container image:

kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow

This is where the Rollout logic comes in. The update will be progressively applied based on the logic we specified earlier. So first we’ll get a new ReplicaSet that will represent 20% of the total Pods. And we’ll pause there, requiring some manual intervention to proceed.

If you’re still running the previous watch command, you can see the updated state of the Rollout:

From this detail we can also see that our Rollout is at step 1 of 8, and is currently paused.

Jump back into your web browser, and you should eventually start to see the occasional request being served by a yellow Pod instead of a blue one. (Note, you may need to reload the page if it gets “stuck” making requests to the same Pods over and over again)

Like I said, a pause step with no duration defined will just remain paused indefinitely, so we must promote the rollout for it to continue to the next step:

kubectl argo rollouts promote rollouts-demo

Now we can observe the Rollout continue through the rest of its defined canary steps, slowly increasing the weight of the update until finally all Pods are running the new version. You can observe this in the output of kubectl argo rollouts get, but it’s much prettier to watch it on the demo web app:

Aborting a Rollout

The canary pattern is of course about letting us try an update with a small subset of production traffic. So when we’re at the manual intervention stage, we can abort the rollout instead of promoting it, which will return the Rollout to its previous state.

Give this a try yourself, by first updating from the yellow container to the red one:

kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:red

At this point you’ll have a canary running the red version (weighted at about 20%). Run the following command to abort, rather than promote, this rollout:

kubectl argo rollouts abort rollouts-demo

Now you can watch everything rollback to the previous version.

This, however, puts our Rollout in a degraded state. This is the definition of an abort as opposed to a rollback, and we can see this detail in the watch view:

To fix this we need to “re-declare” the state we want to match the state we currently have. If our code specified the rollouts-demo:yellow container we could simply re-apply the object. In our case, it’s quicker to patch the object again:

kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:yellow

No actual changes to Pods are required because we’re already running 100% yellow containers, we’re just reconciling the current state of the cluster with what should be running. This means the state of the Rollout will immediately change to healthy.

Summary

This has been a very short tour of Argo Rollouts, where really we’ve just demonstrated how the Rollout object serves as a more advanced drop-in replacement for a Deployment. But by doing this, hopefully I’ve helped demystify how this project works, and you can start to appreciate how useful it can be.

Stay tuned for the final stop on our journey through the Argo project - Argo Events!

What is a Workflow?

Before we jump into Argo specifically, I think it will help to understand what we mean by a Workflow.

Many folks may be used to setting up resources in Kubernetes that are designed to keep running - such as Deployments for example. They may change and mutate, or scale up and down, but generally once we’ve declared that we want a service or workload to exist, we’re used to it being there until its eventual end of life. These are basically “long running services”, but not everything we need to accomplish fits that model.

Sometimes we want to run a workload that does something, or maybe even a series of somethings and then stops. It executes successfully, and exits successfully, content in the knowledge of a job well done. This is the workflow pattern, and its used extensively for things like data processing and infrastructure automation.

Kind of like a Job?

A Kubernetes Job is an implementation of this pattern, designed for various batch tasks, but with a very simplistic approach. Kubernetes Jobs typically execute a single task or batch job to completion, but they have very basic error handling and limited scope for dependency management.

By comparison, Argo can handle complex multi-step workflows, with built-in support for dependencies, retry logic and even handling artifacts between workflow steps. Now we’ve set the scene for the problem we’re tying to solve, let’s get hands on and try them out for ourselves!

Prerequisites

To follow along, you’ll need access to a Kubernetes cluster. A simple local dev cluster will do, such as you might run with Kind or Minikube.

Installing Argo Workflows

To get started, we’ll run the following commands to create a dedicated namespace called argo and install the necessary CRDs (just like we did in our previous post!). In this command we’re specifying Argo version 3.7.0, but there may be a newer version available (check their releases page here).

kubectl create ns argo
kubectl apply -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/v3.7.0/quick-start-minimal.yaml"

Along with the CRDs we need to define Argo objects, we’ve also now installed a few components into the argo namespace. These are:

• argo-server: This is the central component that provides an API for interacting with Argo workflows. It also provides a web UI if you like that sort of thing.

• workflow-controller: This component is responsible for managing the execution of workflows in Argo. It does things like watch for new workflow submissions, orchestrates their execution and manages their state.

• httpbin: This is a simple HTTP request and response service. It’s often used for testing workflows.

• minio: A high-performance, distributed object storage that should probably get its own blog post in this series some day! Argo uses minio to handle artifact storage during workflow execution.

Argo also has its own CLI for Workflows. Whereas the CLI for ArgoCD was called argocd, the CLI for Argo Workflows is just called argo. This doesn’t bother me at all 😬 so let’s go ahead and install it.

You can grab a binary from the releases page, or if you’re using Homebrew run:

brew install argo

The argo CLI only has a handful of simple commands, and this is all we need to interact with the Argo server:

• To submit a workflow, we use argo submit

• To list current workflows, we use argo list

• To see info about a specific workflow, we use argo get

• To print logs from a workflow, we use argo logs

• Finally, to delete a workflow, we use argo delete

An example workflow

The Argo docs provide an example “Hello World” workflow to get you started. Before we submit it though, let’s take a look at the YAML so we understand what it’s actually doing:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: hello-world-
  labels:
    workflows.argoproj.io/archive-strategy: "false"
  annotations:
    workflows.argoproj.io/description: |
      This is a simple hello world example.
spec:
  entrypoint: hello-world
  templates:
  - name: hello-world
    container:
      image: busybox
      command: [echo]
      args: ["hello world"]

As you can see, this is a Workflow type of object, as defined by one of the CRDs we just installed.

In the metadata section, we’re using generateName instead of name. This is actually a native Kubernetes feature that you don’t see too often in the wild. Using generateName here means that this object, when created, will use hello-world- as a prefix, and the server will add a unique generated suffix to its name.

Like any other Kubernetes object, the spec then defines the core structure of our workflow. In Argo, we provide this as a list of templates and an entrypoint. Templates are re-usable definitions of some sort of workflow logic which we’ll explore in more detail below, and the entrypoint is just which template we run first when the workflow starts.

Templates

So as we’ve just stated, templates are just re-usable sets of instructions. There are many different types of templates, but they all belong to one of these categories:

• Template definitions are the types of templates that define work to be done, usually in some sort of container.

• Template invocators are ways to call other templates, and define execution control (such as ordering and dependencies).

Coming back to our “Hello World” example, we are defining a single template called hello-world, and this template is a container type. This is the most commonly used template type in the definition category, and it simply lets you define a container to run in exactly the same way as you would anywhere else in Kubernetes (such as in a Pod spec).

The expectation is that the container will execute some command, optionally with some arguments, and then exit successfully. The output of the container is stored in an Argo variable, so that you could use it later if you wanted to (we’re not doing that here of course, because we only have a single template in this workflow).

Submitting a Workflow

Let’s submit this example workflow to our Argo service. We’ll add the optional --watch flag to this command, so that we can watch the workflow complete:

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml

Watching the workflow shows us neatly what’s happening: the workflow executes successfully, and the watch exits when it’s finished.

We can see that it took about 20 seconds for the workflow to finish successfully, and that a Pod called hello-world-qnk9t existed for about 9 seconds. Pretty cool!

Other Template Types

While containers are most commonly used for templates, there are other types too within the definition category:

• script is basically a convenience wrapper around container, which adds a source field to allow you to define a script in-place (for example, to run an in-line Python script on a python:alpine container)

• resource allows you to perform resource operations on a Kubernetes cluster, for example to create a ConfigMap. This is super useful when you realise you can pass variables and parameters between templates in a workflow.

• suspend allows you to suspend the execution of a workflow until it is resumed manually.

• plugin allows you to reference external plugins.

• containerset lets you use multiple containers within a single Pod.

• http lets you execute HTTP requests and store the results as a variable to use elsewhere.

Invocators and execution control

Now we’re getting to the good stuff, and hopefully things will start to make more sense!

In the “Hello World” example, we have a single template acting as a single step in a workflow. This is a great starting point, but it can make the terminology confusing. Why is a step called a template?

The answer is that a template is designed to be re-used. We should use templates to create functions that can be referenced multiple times (applying the principle of Don’t Repeat Yourself).

We can then arrange calls to these functions using one of the invocator categories of templates.

So, within this category, you have 2 options:

steps is the most straightforward invocator type, and lets you define your tasks in a series of steps. The steps will run one by one, but you can nest them - outer lists will run sequentially, and inner lists will run in parallel. There are also advanced synchronisation and conditional options for steps, but they’re a bit beyond our scope for today.

Here’s an example of using steps in a workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: steps-example-
spec:
  entrypoint: data-workflow
  templates:
  - name: data-workflow
    steps:
    - - name: step1
        template: prepare-data
    - - name: step2a
        template: process-data
      - name: step2b
        template: notify-data-team

In this snippet, we create a workflow object with an entrypoint of data-workflow. The data-workflow template is a steps type, so it defines a set of steps to run.

• step1 runs first, and it calls a template called prepare-data (we would declare later on in this YAML file what prepare-data actually is and what it does, but we’re omitting it for now to keep this simple).

• Once that step is completed, step2a and step2b run in parallel as they’re in a nested list. So the process-data and notifiy-data-team templates would both be called at the same time.

If you have a more complex set of dependencies for steps in a workflow, you can specify these using a directed acyclic graph, or DAG, with the dag invocator template type. A DAG is basically a data structure that consists of nodes connected by directed edges, where the edges have a specific direction, and there are no cycles (you cannot return to a node once you leave it). They’re used a lot in workflow orchestration, and pop up in other tools like Apache Airflow.

Let’s look at this example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-example-
spec:
  entrypoint: data-workflow
  templates:
  - name: data-workflow
    dag:
      tasks:
      - name: A
        template: echo
      - name: B
        dependencies: [A]
        template: echo
      - name: C
        dependencies: [A]
        template: echo
      - name: D
        dependencies: [B, C]
        template: echo

For the sake of simplicity, every step here is referencing the same dummy template called echo - the important thing is the order of how these steps are run, and how they depend upon each other.

• A runs first

• B and C will run in parallel, as this is the default, but only if A has completed successfully because of the dependency we’ve defined.

• D can then run, but again only if its dependency on the successful completion of B and C is met.

Templates and parameters

Passing data or artifacts between steps is super useful too. Let’s take a look at another steps example:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: messages-
spec:
  entrypoint: my-workflow
  templates:
  - name: my-workflow
    steps:
    - - name: hi-bob
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Bob"
    - - name: hi-emily
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "Emily"
    - - name: hi-again
        template: print-message
        arguments:
          parameters:
          - name: message
            value: "{{steps.hi-emily.outputs.result}}"

  - name: print-message
    inputs:
      parameters:
      - name: message
    container:
      image: busybox
      command: [echo]
      args: ["Hi {{inputs.parameters.message}}"]

In this YAML our workflow object will get a name starting with messages- and we invoke the workflow with the my-workflow template (specified as our entrypoint). Also, my-workflow is of the steps type. Each step will run in order, as we don’t have any nested lists here. This should all be starting to make sense now 😄

Each step references the same template, called print-message. So we’re re-using that function multiple times. You can see it defined at the bottom of the YAML file, where it just uses the busybox container to echo a string to standard output. A nice convention here is to leave a blank line in the YAML before the print-message template, reminding us that it’s a function that gets called by steps in the workflow, but it’s not a part of the workflow on its own.

Let’s walkthrough my-workflow:

• In each step we’re demonstrating how we can parameterise our templates. In the hi-bob and hi-emily steps, we’re using arguments to pass a parameter to the template. The parameter is called message, and as you can see we can use any value we want in each step.

• In the final step, rather than directly specify the value of message, we actually reference the output of the previous step!

So what does this look like when we run it? Let’s give it a try. We’ll write this to a file called messages.yaml and submit it to Argo:

argo submit -n argo --watch messages.yaml

Here’s what the watch looks like:

We can also see the logs of a workflow like this:

argo logs -n argo @latest

This output is helpfully colour-coded! We can see the standard output of each step, saying “Hi Bob”, “Hi Emily” and of course, “Hi Hi Emily” (because we passed “Hi Emily” as the name to say Hi to!) 😄

Please tell me there’s a UI

Of course there is! We can port-forward the UI and then connect to https://localhost:2746. You will get some scary self-signed TLS errors however, and if you go on to use Argo in production you should definitely configure TLS properly.

kubectl -n argo port-forward service/argo-server 2746:2746

Summary

We’ve now had a quick tour of Argo Workflows, and hopefully you’ve got a good idea of its basic functionality as well as its potential use cases. To get to grips with the advanced features of Argo, I’d definitely recommend browsing the user guide and playing with some more examples. Stay tuned, as next time we’ll be tackling Argo Rollouts!

1. Learning new stuff is hip and cool 😎

2. The CNCF landscape looks scary! But learning the graduated projects one by one seems manageable and useful.

3. The whole CNCF project is truly inspiring. It’s a real community effort contributed to by thousands of smart people that builds actually useful things together!

Graduated projects are considered stable and are used successfully in production environments (so the CNCF website proudly declares), so I’m going to explore each one in turn in this series. The first one on this list is:

Argo

And Argo is actually more than one thing! Argo defines itself as “Kubernetes-native tools to run workflows, manage clusters, and do GitOps right”. Argo was accepted to CNCF on March 26, 2020 at the Incubating maturity level and then moved to the Graduated maturity level on December 6, 2022. Here we are some years later, and the Argo project has thousands of users and some impressive production case studies.

Argo now comprises 4 different tools:

• Argo Workflows: Kubernetes-native workflow engine supporting DAG and step-based workflows.

• Argo CD: Declarative continuous delivery with a fully-loaded UI.

• Argo Rollouts: Advanced Kubernetes deployment strategies such as Canary and Blue-Green made easy.

• Argo Events: Event based dependency management for Kubernetes.

Taking things slightly out of order, I’m going to start by looking at Argo CD, as this is the project’s most popular tool, and it’ll give us a good understanding of the project’s conventions we can then take forward to learn about the other tools. Let’s go!

Argo CD

So what is Argo CD? In a nutshell, Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. The idea is that while native Kubernetes brings the declarative model to its foundational building blocks, Argo CD extends that approach so that everything you need to deploy applications including configurations and environments are also declarative and version controlled. Treating everything in the deployment lifecycle as software makes it more reliable and easy to collaborate on.

Argo introduces its own opinionated concepts about how to get stuff done in Kubernetes. There are quite a few of these, but the core ones you need to understand are:

• An Application is a group of Kubernetes resources defined by a manifest. Following best practices, we tend to break down things we run in Kubernetes into their components, such as an individual microservice, data store or messaging queue. But even these things require multiple native Kubernetes objects. An Argo Application then groups all the necessary resource objects together to form a single deployable component.

• Applications have an Application Source Type, which is determined by the Tool used to build them. Essentially the Tool is just something you use to process Kubernetes manifests and make any necessary changes. Commonly used tools are Kustomize and Helm.

As a declarative system, ArgoCD cares about the state of your applications. So when dealing with Argo lifecycles we need to understand these state definitions:

• The Target State is the desired state of an Application, declared in code in a Git repo

• The Live State is the current state of the Application (eg. what Pods are currently deployed)

• A Sync is the process of moving an Application to its Target State (ie. by making changes to a Kubernetes cluster)

• A Sync Status represents whether or not the Live State matches the Target State.

With those concepts in mind, let’s walk through Argo CD’s Getting Started Guide. You can of course follow the guide on Argo’s website, but it’s fairly bare-bones! In this post I want to try to add some more detail and explanation to what we’re doing, so we really understand the purpose of using ArgoCD.

Prerequisites

To follow along, you’ll need access to a Kubernetes cluster. A simple local dev cluster will do, such as you might run with Kind or Minikube.

Installing Argo CD

To get started, we’ll run the following commands to create a dedicated namespace called argocd and install the necessary CRDs:

kubectl create ns argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

It might not feel like a good practice to put everything in a single namespace right now, but you can consider argocd to be your Control Plane namespace. While your Application CRDs may get deployed here, the actual Kubernetes resources that they create and manage can be directed to other Data Plane namespaces, which can be designed using the usual methods for application isolation.

In one hopefully trustworthy command, we’ve now installed around 60 new resources to our Kubernetes cluster. This includes CRDs to support Argo Applications, along with Roles, ClusterRoles, RoleBindings, ClusterRoleBindings, and the necessary Deployments, Services and ConfigMaps to support the key moving parts of ArgoCD:

• argocd-server: This is the API server. It exposes the gRPC/REST API that the Argo CD CLI and UI consume. It's responsible for managing applications, projects, and authentication.

• argocd-repo-server: This server is responsible for cloning your Git repositories, caching them locally, and generating the Kubernetes manifests from the source (e.g., by running kustomize or hydrating a Helm chart).

• argocd-application-controller: This is the core controller that continuously monitors the live state of your applications against the desired state defined in Git. When a difference is detected (OutOfSync), this controller is what executes the sync operations.

• argocd-dex-server: Dex is an identity service that uses OpenID Connect (OIDC). Argo CD includes it by default to handle authentication by federating identity from other providers like SAML, LDAP, or social logins (e.g., GitHub, Google).

• argocd-redis: This deployment runs a Redis cache. Argo CD uses it extensively for storing the application state cache, OIDC tokens, and other temporary data to improve performance and reduce requests to the Kubernetes API server.

• argo-notifications-controller: An optional controller that monitors the Application resources and, based on triggers you configure, sends notifications about application health and sync status to services like Slack, Email, and (if you’re unlucky) Microsoft Teams.

Next we’ll install the argocd CLI. You can download the latest release from https://github.com/argoproj/argo-cd/releases/latest or if you’re using Homebrew, go ahead and run:

brew install argocd

We can now use the CLI to log into our ArgoCD service:

argocd login --core

Finally, let’s set the default namespace for our current Kubernetes context. This just saves us some time because for the purposes of this demo, we’re running Argo and our applications on the same cluster. (Note: Using other clusters is super easy and can be managed with the argocd cluster command)

kubectl config set-context --current --namespace=argocd

Creating a sample Application

The Argo project provides a number of sample applications for us to play with when we’re learning how ArgoCD works. We’ll start with the basic guestbook example. You can view the components at https://github.com/argoproj/argocd-example-apps/tree/master/guestbook

As you can see, it’s just a simple Deployment and Service.

To create an Argo application using this repo, we’ll run this command:

argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace default

Let’s break that down:

• argocd app create is the command to create a new ArgoCD Application. We specify guestbook as the application name, and then provide some flags:

• --repo sets the repository source where the resource definitions can be found. We could have alternatively used local files.

• --path identifies the location within that repo for our resources, and in this case that’s the guestbook directory within the repo.

• --dest-server and --dest-namespace identify the destination Kubernetes cluster and namespace respectively (you probably figured that one out yourself 😁)

When this command completes, we have created the declarative definition of our new ArgoCD Application. Let’s check on its status with this command:

argocd app get guestbook

Hmm… the Health Status is Missing and the Sync Status is OutOfSync!

That’s to be expected, because we need to perform the first Sync. Recall from earlier that the Sync process should align our Live State with our Target State. Right now, our Live State is that nothing has been deployed, so the Sync should fix that!

argocd app sync guestbook

If you check the status again after running this command, you should see that the Sync operation has succeeded, and the Health Status is Progressing. In other words, ArgoCD is happy in what it needs to do to perform the Sync, and it is currently underway.

After a few moments, the Health Status should change to Healthy because all the actions will have been completed, and the new guestbook Application is running as expected.

Making and syncing changes

The job of declarative systems like ArgoCD is to make sure that the live state matches the state you are declaring in code. So let’s say we make a change to our git repo, what happens next? (Note: You can try this for yourself by forking the Argo examples repo and then making some changes).

The argocd-application-controller performs a reconciliation loop to ensure that the live state still matches the state declared in code. It relies on the argocd-repo-server to monitor the git repo associated with an Application for any changes (by default it checks around every 3 minutes). A diff is performed, and any kind of changes (such as a new container image, or change in replica count) will mean that the Application’s Health Status gets changed to OutOfSync.

By default, it’s then up to the operator to perform another sync command, which will allow ArgoCD to reconcile the live status with the changes.

Optionally, you can configure an automated syncPolicy in your Application definition. In this policy you can also enable selfHeal, which will allow ArgoCD to automatically revert back to the state defined in git if anyone makes any manual changes directly on the cluster. Neat!

Using Helm

One of the primary benefits of an automated CD solution like this is to automate changes across different application environments (such as dev to production). Let’s walk through a super simple example that leverages Helm and manages different versions of an application. We’ll use the sample code from the ArgoCD repo here: https://github.com/argoproj/argocd-example-apps/tree/master/helm-guestbook

Note that Helm is only used here to inflate a chart using helm template under the hood. When coupled with ArgoCD, we’re letting Argo handle the lifecycle of the component itself, not Helm. We’re basically taking the nice and easy templating stuff from Helm (because Kustomize is weird!) but making our deployments continuous with ArgoCD.

So first we’ll create 2 different namespaces, one for development and one for production:

kubectl create ns guestbook-dev
kubectl create ns guestbook-prod

Now we can create an ArgoCD Application for development:

argocd app create guestbook-dev \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path helm-guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace guestbook-dev \
  --sync-policy automated --auto-prune --self-heal \
  --values values.yaml

Notice here we are enabling automated syncs, and automated healing. Our development environment will now stay dynamically up to date with the latest version of our resources committed to our git repo.

Let’s create the production version of this Application, but this time we’ll default to manual synchronisation and we’ll specify the values-production.yaml file, providing some environment-specific overrides:

argocd app create guestbook-prod \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path helm-guestbook \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace guestbook-prod \
  --values values-production.yaml

Now if we run argocd app get on each app, we’ll see that both have been created, and the development app has automatically synced. We can go ahead and sync the production app manualy with argocd app sync like we did earlier.

Is there a fancy UI?

Yes, there is! The first time you access it, you’ll need to retrieve the default admin password. You can do this with this command:

argocd admin initial-password -n argocd

Now we’ll use port-forwarding to connect to the web UI (other options, like ingress, are available):

kubectl port-forward svc/argocd-server -n argocd 8080:443

Once you’ve logged in you can explore all of your Applications, states, syncs and revisions through the UI. You should also change the admin password, and delete the argocd-initial-admin-secret from the Argo CD namespace once you have done so.

Summary

That’s the end of our quick tour of ArgoCD, but hopefully I gave you a bit more to go on that the official docs, which are very good if a little overwhelming! Hopefully I can keep the momentum of this series going, and in the next post I’ll be tacking Argo Workflows. See you then!

Learning how complex systems work is often an exploratory procedure. We need to slowly and methodically explore new features, learn their dependencies, and understand how they operate holistically with other components. This ultimately provides us with the knowledge to fully understand a system and to design architectures that use it. And that’s basically what I’ve been trying to do in this blog series!

But working manually is not a recommended way to manage production systems once they have been deployed. When organisations rely on critical infrastructure, we need build and deployment methods that are reliable, repeatable and auditable, and none of these things apply to iterative manual work. And that’s why the concept of Infrastructure as Code has long been established in IT to achieve these goals.

We can define infrastructure in code and rely on that code to be deterministically repeatable. We can even audit that code, as it provides a living documentation of what’s being built. Extending those principles to “Everything as Code”, the way we deploy applications can also be automated using many of the same approaches and tools. Giving us faster and safer deployment methods means we can iterate on software rapidly but reliably.

Software lifecycles are now managed almost exclusively through CI/CD pipelines. This approach combines Continuous Integration (CI), where developers synchronize code changes in a central repository as frequently as possible, and Continuous Delivery (CD), where new updates are released into a production environment. We’ve come a long way from the 6-month release cycles of a decade ago, to many cloud-native organisations deploying small code changes hundreds of times a day.

So in this post, we’re going to learn about the GKE tooling that we can use to achieve this sort of developer agility. We’ll be looking at:

• An overview of DevOps and GitOps practices

• Building a basic CI/CD architecture with Cloud Build and Cloud Deploy

• Some considerations for private build infrastructure

By the end of this post, you should feel comfortable automating the complete lifecycle of your workloads on GKE!

Just one quick note before we get started: This post covers native CI/CD tools for GKE to help you build a software delivery framework – but we’re talking about your workloads here, not your clusters. It is also recommended that your GKE infrastructure is automated through Infrastructure as Code, as I touched on at the start of this series. We can’t get side-tracked into a large Terraform tutorial here though, so if you need some further reading on this topic, take a look at: https://cloud.google.com/docs/terraform

An overview of DevOps and GitOps practices

There’s a lot of debate about the true definition of the term “DevOps” (which comes from the contraction of Developer and Operations). In a nutshell, DevOps is a methodology. It’s a change to the way we work – the processes we follow and the tools we use, to introduce elements of the software engineering world into how we build and maintain infrastructure. GitOps is simply an extension of DevOps that implements automation based around a git repository as a single source of truth.

But why do we need DevOps? Why are automation and speed so important? While there’s definitely a business case for improving developer agility and getting new features to market faster than your competitors, speed is really just a benefit of doing DevOps properly – it's not a requirement for doing it in the first place. If you can trust your platform to allow you to iterate quickly, this means that it must be a reliable platform, and a reliable platform is much easier to fix when things go wrong.

So how does automation give us a reliable platform? Once again, it’s about removing the human variable from the equation. Historically, production computer systems could be considered to be in a fairly fragile state. Operators would dread having to update applications, not least because the updated code had probably been thrown over a wall by a development team with little knowledge of the actual server where it was intended to end up. Installing an update meant changing the state of a system that had been established by a hundred different manual interactions over the previous years. Even worse, in the event of a failure or disaster recovery scenario, how do you get that server back to that unknowable state?

As mentioned way back at the start of this series, the move to declarative configuration was a paradigm shift in the way we manage systems. If we trust that our declarative tools can maintain a deterministic state of a system, suddenly we’re not afraid to change that state. Rolling back a failed update or recovering from a failure simply means referring to an earlier documented state. And now we have modern tooling that achieves this model across infrastructure and application deployments.

A software delivery framework

While real-life implementations will vary considerably based on team sizes, workloads and even organisation policies, we can describe an ideal architecture for a software delivery framework as illustrated below. Later on we’ll walk through actually building this reference architecture.

The foundation for the architecture is using git as a source of truth and basing all actions within the architecture (deploying or updating infrastructure or applications) as actions that are triggered by events in those git repositories. Manual changes are expressly forbidden by design, and this is achieved by restricting permissions in production environments to service accounts operated by the pipelines only. It’s worth noting that within those git repositories there are still many different options for how to collaborate and construct your workflows. That level of detail is outside our scope here, but if you’re interested in comparing git workflows such as Feature Branch and Forking, a good resource is Atlassian’s documentation here: https://www.atlassian.com/git/tutorials/comparing-workflows

Our architecture is separated by two distinct roles: Developer and Operator (these days, Operators might be called Platform Engineers). The Developer is responsible for developing and building applications and maintaining them in the Application Repo. The Operator is responsible for building infrastructure and maintaining this code in the Infrastructure Repo. Of course, depending on the size and layout of your teams, these could also be the same people in a single SRE team, but the important thing is the separation of duties in the respective pipelines. Let’s walk through them!

• The Developer commits code that represents a new version of an application to a git repo (each application or microservice is likely to have its own repo).

  • This commit triggers a CI build pipeline.

  • The CI system will build a new artifact from the updated code (usually a Docker container).

  • Tests can and should be run at various stages of this process as well, from tests on the uncompiled code to tests on the final container image. If tests pass successfully, the image can be pushed into an artifact registry.

  • This in turn triggers the CD system, which deploys the application to a development cluster, or updates an existing deployment in place with the new container.

  • Using the CD system, Developers or Operators can monitor the state of deployments and choose to promote them through environments – from development to staging, and from staging to production.

  • Each environment uses its own specific configuration, also retrieved from the git source repository.

• Meanwhile, the Operator is responsible for building and maintaining the cluster infrastructure itself, rather than the applications that run on it.

  • Infrastructure code belongs in its own git repo, and changes to infrastructure are handled through a different CI process.

  • Normally it's sufficient to use a CI tool such as Cloud Build to manage changes to infrastructure code (like Terraform).

Now we’ve discussed all of the theory, let’s go ahead and put all of this infrastructure in place!

Building a basic CI/CD architecture with Cloud Build and Cloud Deploy

We'll now walk through building a simple reference architecture that uses a GitOps approach to building an application (or updating an existing build), and then automatically deploying that application to a staging environment, where it can be manually promoted to a production environment. All the sample code below can be found in my repo here: https://github.com/timhberry/gke-cicd-demo

We’re going to concentrate on the Application Repo part of the diagram we looked at earlier. So to follow along you’ll need to have two GKE clusters set up and ready to go. In our code we refer to them as staging-cluster and prod-cluster, and both run in us-central1-c. Keep an eye out for these references in the code if you need to change them for your own environment. You will also need a git repo to host your application code. In the walkthrough we’ll use Github, but Bitbucket is also supported.

Before we get started, let’s look at an overview of the layout we want to build for our application repo. For demonstration purposes, our app will be a simple “Hello World” web server. To provide the CI/CD automation we need, we’ll bundle the following files along with our code in our repo:

• cloudbuild.yaml: The configuration for Cloud Build. This will define how the CI pipeline builds our application and triggers the next stage.

• clouddeploy.yaml: The configuration for Cloud Deploy. This file defines how our CD pipelines should work, managing deployments to our staging and production clusters.

• skaffold.yaml: The configuration for Skaffold, an automation framework for Kubernetes used by Cloud Deploy (don’t worry, we’ll explain this in a moment!)

• k8s/deployment.yaml and k8s/service.yaml: Definitions for the Kubernetes objects our application needs.

• Dockerfile: The container manifest for building our image.

We’ll build up the repo piece by piece in the following sections.

Setting up permissions for Cloud Build

Before we can start using our repo, we need to provide some permissions to the Cloud Build service account. In the following commands, you’ll need your 10-digit project number, as it forms part of the server account name. You can find your project number on the dashboard of the Cloud Console, or by using the following command (replacing <PROJECT-NAME> with your project name):

gcloud projects describe <PROJECT-NAME> \
    --format="value(projectNumber)"

In the following commands, we’ll use the fake project number 5556667778, so just make sure to substitute yours (along with your <PROJECT-NAME>). First, we give Cloud Build permission to create releases in Cloud Deploy:

gcloud projects add-iam-policy-binding <PROJECT-NAME> \
    --member=5556667778@cloudbuild.gserviceaccount.com \
    --role="roles/clouddeploy.releaser"

Next, we need to make sure that Cloud Build can act as the default Compute Engine service account. This requires roles to act as a service account and create service account tokens:

gcloud projects add-iam-policy-binding <PROJECT-NAME> \
  --member serviceAccount:5556667778@cloudbuild.gserviceaccount.com \
  --role roles/iam.serviceAccountTokenCreator

gcloud iam service-accounts add-iam-policy-binding \
     5556667778-compute@developer.gserviceaccount.com \
     --member serviceAccount: 5556667778@cloudbuild.gserviceaccount.com  \
     --role roles/iam.serviceAccountUser

Note that this isn’t exactly a best practice – ideally, we would create IAM policy bindings with the least privileges required for our pipelines, and not use the default service account. However, we’re trying to learn Cloud Deploy, and pages and pages of IAM instructions would probably distract from that!

With IAM out of the way, we can now start to set up our repo.

Creating a sample app with automated builds

To demonstrate our CI/CD pipeline, we’ll build a very simple application that will be easy to update so we can experiment with the update process. All the cool kids are using Golang these days, so I’ve built a small app in Python which you can find in the repo as server.py. In addition, we have a Dockerfile that specifies how to build the container image, a simple HTML template.html, and a requirements.txt file that Python uses to install the libraries it needs.

If you’ve checked out the repo locally, you can test the image on your own machine by building it and running it with Docker (or Podman):

docker build -t sample-app .
docker run -d -p 8080:8080 sample-app

When the container is running, try to load http://localhost:8080 in your browser and you should see a “Hello World!” message.

In our pipeline, Cloud Build will be responsible for building our image, tagging it, and storing it in the registry. Let’s create a basic cloudbuild.yaml file to do this for us:

steps:
- id: Build container image
  name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/sample-app:$COMMIT_SHA', '.']

- id: Push to Artifact Registry
  name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/sample-app:$COMMIT_SHA']

Once you have this file and your app code committed to your repo, it’s time to set up a Cloud Build trigger. This enables Cloud Build to run automatically whenever a new commit is made to your repo.

By default, Cloud Build will watch the main branch of your repo. In a real-world situation, you may want to build from feature branches or use PRs to merge to the main branch before a build is triggered. This will depend on your chosen Git workflow, and once again, I’m trying to keep this walkthrough as simple as possible!

From the Cloud Build page in the Cloud Console, select Triggers and Create Trigger. You’ll need to give your trigger a name and then choose a repository. You'll have the option to connect a new repository here, which will forward you to Github to provide permission to connect Cloud Build. Once your repository is selected, we can leave everything else about the trigger at a default and click Create.

You should now see your trigger configured, as shown below. Note that the event is “Push to branch”, meaning that a push to the specified branch will trigger Cloud Build. The Build configuration is set to “Auto-detected”, which means that Cloud Build will look for a cloudbuild.yaml file, but if it can’t find one it will also look for a Dockerfile or a buildpacks configuration. A cloudbuild.yaml file will always take priority.

You can now go ahead and commit your code to your repo. Remember this should include server.py, Dockerfile, template.html, requirements.txt and cloudbuild.yaml. If you go back to your Cloud Build dashboard, you should now see that a build has been triggered. A short time later, you’ll see that Cloud Build has completed the steps that we defined: it has built your container image and stored it in the registry. You can select the build to see the completed build steps, as shown below:

We’ve now achieved the CI in CI/CD, as we have an automated workflow that continuously integrates changes to our code and automatically generates a new container image every time we do so. Next, we’ll add our Kubernetes objects and configure a pipeline to continuously deploy our changes as well.

Adding Kubernetes manifests

Our application will ultimately get deployed to our GKE clusters, so we need to add some Kubernetes objects to our repo. In a sub-directory called k8s/ we’ll create a deployment.yaml and service.yaml file. The service.yaml file is very straightforward and simply creates a LoadBalancer service for our app, exposing HTTP port 80:

apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  selector:
    app: sample-app
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer

The deployment.yaml is also very straightforward at first glance:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: sample-app
  name: sample-app
spec:
  replicas: 1 
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - image: _IMAGE_NAME
        name: sample-app
        imagePullPolicy: Always

Note the container image name however: _IMAGE_NAME. This is a placeholder! In a moment, we’ll update our Cloud Build steps so that the deployment YAML is overwritten with the latest build reference. Add these manifest files to your repo.

Let’s keep this simple! There's lots of moving parts here and we’re trying to keep the focus on Cloud Deploy. Overwriting a deployment YAML in our CI process is a very simplistic approach, but there’s nothing wrong with a bit of simplicity! Cloud Deploy also supports complex approaches with tools like Kustomize and Helm, if you’re already using these to build and deploy your applications. For more information see https://cloud.google.com/deploy/docs/using-skaffold/managing-manifests

Configuring Cloud Deploy pipelines and releases

Now we can add the configuration required for Cloud Deploy. There are two distinct concepts in Cloud Deploy we need to understand:

• A pipeline is the configuration of how our CD process should run. It defines the targets we deploy to, what parameters they should use and other variables such as approval stages.

• A release specifies a single run through the pipeline CD process that releases artifacts to targets. This is usually triggered by the end of a CI process.

We’ll create the pipeline manually first (although technically you could do this with infrastructure as code too!), and in a moment we’ll rely on Cloud Build to run a command that triggers a release. Here’s our clouddeploy.yaml file:

apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: sample-app
description: sample app pipeline
serialPipeline:
  stages:
  - targetId: staging
    profiles: []
  - targetId: prod
    profiles: []
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: staging
description: staging cluster
gke:
  cluster: projects/PROJECT_ID/locations/us-central1-c/clusters/staging-cluster
---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
description: production cluster
gke:
  cluster: projects/PROJECT_ID/locations/us-central1-c/clusters/prod-cluster

In this file we define a pipeline with two stages: staging and production. We then create and associate targets with each stage, and within each target we reference our GKE clusters (make sure you update your zones and references to PROJECT_ID). This is a simple example, but we could expand on it to provide differing values or parameters to be applied to each target environment.

We can now create the pipeline with this command:

gcloud deploy apply --file=clouddeploy.yaml \
  --region=us-central1

In the Cloud Deploy section of the Cloud Console, we should now see an empty delivery pipeline, as shown below. At this stage we’ve configured the pipeline, but nothing has run through it yet. But we’re nearly there!

Under the hood, Cloud Deploy leverages an automation framework called Skaffold. This framework manages Kubernetes manifests for Cloud Deploy, rendering them into their required state for deployment. So, we need to add a very simple skaffold.yaml to our repo:

apiVersion: skaffold/v4beta7
kind: Config
metadata:
  name: sample-app
manifests:
  rawYaml:
  - k8s/*
deploy:
  kubectl: {}

This file simply configures Skaffold to look in the k8s/ directory in our repo for our manifest files and processes them with kubectl. Once again there are many other things Skaffold can do for you, and it even functions as its own automation system outside of Cloud Deploy, so if you’d like to learn more then check out https://skaffold.dev/

Finally, we’ll add two more steps to our cloudbuild.yaml file, after the existing steps. Here’s the first one:

- id: Update deployment.yaml
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    sed -i "s/_IMAGE_NAME/gcr.io\/$PROJECT_ID\/sample-app:$COMMIT_SHA/g" k8s/deployment.yaml

This is the step we use to update the placeholder in deployment.yaml. When our application is deployed, we want to make sure we’re using the latest build of our container image, so we use sed to update the file in place with the full image name including the SHA hash from the commit.

The final step we add to cloudbuild.yaml actually triggers a release in Cloud Deploy:

- id: Trigger Cloud Deploy
  name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    SHORT_SHA=$(echo $COMMIT_SHA | cut -c1-7)
    gcloud deploy releases create "release-$SHORT_SHA" \
      --delivery-pipeline=sample-app \
      --region=us-central1 \
      --images=sample-app=gcr.io/$PROJECT_ID/sample-app:$COMMIT_SHA

Note that we reference the name of the pipeline we created earlier with --delivery-pipeline. We also provide a name for the release, which we construct by creating a shorter version of the commit SHA.

With all of these pieces now in place, push your changes to your repo and the following magic will happen:

• Your Cloud Build pipeline will be triggered by your git repo changes

• Cloud Build will build your application, tag it and push it to the registry

• Cloud Build will also update your deployment.yaml with the new container image tag

• It will then trigger a release in your Cloud Deploy pipeline

• The Cloud Deploy pipeline will deploy your new application and the Kubernetes manifests to your staging cluster

Observing the CI/CD pipeline

First, your Cloud Build pipeline will run, only now it will contain the extra stages you defined. When it is complete, you should see that Cloud Deploy has been triggered, as shown below:

Now if you hop over to the Cloud Deploy dashboard in the Cloud Console, you should see that your pipeline has a new release. A successful release also creates a roll-out, which is simply when the desired artifacts get deployed to their target. In other words, your application has now been deployed to your staging cluster!

You can see this from the pipeline view in Cloud Deploy, as shown below:

From the Kubernetes section in the Cloud Console, you can also look at the deployed workloads and see that your sample app is running on the staging cluster:

In the details of the deployment, you should be able to locate the external IP of the LoadBalancer service and load that in your browser. You should be greeted with the friendly blue page of a very boring Hello World web app:

In our sample developer workflow, tests could now be run on the latest version of the application in our staging environment. Once we are happy with the updated application, we can promote it to production directly from Cloud Deploy.

From the Cloud Deploy pipeline dashboard, we simply select Promote. This prompts us to confirm the promotion target (which in this case is our production cluster) and provides us with a summary of the actions that the system will take once we start the promotion. Go ahead and click Promote again and you can watch what happens:

• The current release creates a new roll-out, this time targeting the production cluster.

• The application artifact and Kubernetes manifests are rendered to this new target.

You’ll now have 2 workloads running – your sample app in each cluster:

Updating apps through the CI/CD workflow

Now we’ve finished the initial roll-out of the application, you can simulate the process for changes as well. Everything is now in place so that next time code is pushed to the main branch of the repo, the CI/CD process runs again. The new version of the app will be built and automatically pushed to the staging cluster, ready for you to test and approve for promotion to production.

You can try this for yourself by making a simple change – I would suggest altering the background colour of the app to something more appealing! You can find the bgcolor variable defined in server.py.

In this walkthrough, we’ve built a reference CI/CD architecture for Cloud Build and Cloud Deploy. But of course, we’ve only just scratched the surface of what these tools can do. For example:

• Cloud Deploy can automatically promote rollouts for you, based on specific rules and targets.

• Cloud Deploy also supports complex deployment patterns like canary testing and deploying to multiple targets at once.

• Skaffold configuration can further customise each target, for example by defining a specific number of Pod replicas per target.

Now you understand the basics of these systems, you may be ready for some more advanced approaches. An even more in-depth tutorial that includes frameworks for different programming languages can be found at https://cloud.google.com/kubernetes-engine/docs/tutorials/modern-cicd-gke-reference-architecture

Some considerations for private build infrastructure

Cloud Build and Cloud Deploy are fantastic tools for automating the various build and deploy stages of your CI and CD pipelines. Each step of your Cloud Build pipeline runs as an isolated and containerised process with access to a shared workspace for the pipeline's duration. Under the hood, when Cloud Deploy executes steps in your CD pipelines it also uses workers in Cloud Build.

However, by default these workers are running in a shared pool as part of their respective services. In many cases this might not be desirable: you may have regulatory requirements that mean all compute processes have to stay within certain network boundaries, or you may have components that only accept private connections within your VPC that are required for steps in your pipelines.

Thankfully, Cloud Build allows you to configure private pools, which are pools of worker resources that run within a dedicated private VPC. This service producer VPC is still Google-managed and does not exist within your project, however you can peer with it directly for a private connection that only requires internal IP addresses.

Alternatively, if you don’t want to configure VPC peering, you can use a dedicated private pool but allow it to use public endpoints to communicate with resources in your VPC. You might choose this option if your VPC already has public endpoints, and you just want a private pool to give you more options over the configuration of the Cloud Build workers.

You can create a private pool using the following command. There are lots of parameter placeholders here, which I’ll explain in moment:

gcloud builds worker-pools create <PRIVATEPOOL_ID> \
    --project=<PROJECT_ID> \
    --region=<REGION> \
    --peered-network=<PEERED_NETWORK> \
    --worker-machine-type=<MACHINE_TYPE> \
    --worker-disk-size=<DISK_SIZE> \
    --no-public-egress

Let’s walk through those parameters:

• PRIVATEPOOL_ID is the unique name that you assign to your private pool.

• PROJECT_ID is the name of the project that you wish to use with this pool.

• REGION is your chosen region where the private pool should run.

• PEERED_NETWORK is the VPC network in your project that should be peered with Google’s private service producer network. Note that you’ll need to specify the long resource name for this parameter, for example: projects/my-project/global/networks/my-vpc.

• MACHINE_TYPE allows you to choose the machine type for your workers. Several types of Compute Engine instance type are supported, and if you don’t specify a preference, an e2-standard-2 will be used. Note that this parameter can also be overridden at runtime by passing the --machine-type option to gcloud builds submit.

• DISK_SIZE lets you adjust the disk size of the worker instances, which must be between 100 and 4000 GB. Once again this is optional; Cloud Build will default to a size of 100GB. This can also be overridden at runtime with the --disk-size option.

• The --no-public-egress option ensures that workers are created without an external IP address.

Note that these parameters can also be passed in a configuration file which defines a private pool schema. The layout of this schema is documented at https://cloud.google.com/build/docs/private-pools/private-pool-config-file-schema

Once the private pool has been created, you can choose to use it any time Cloud Build executes a pipeline. If you’re submitting a build job manually, you can specify the name of the worker pool on the command line. For example:

gcloud builds submit --config=cloudbuild.yaml \
  --worker-pool=projects/my-project/locations/us-central1/workerPools/my-privatepool

You can also specify the private pool directly in your cloudbuild.yaml, which helps if your builds are being triggered by a git commit. Simply add the pool configuration to an options section at the bottom of your cloudbuild.yaml file like this:

steps:
- name: 'bash'
  args: ['echo', 'Previous steps up to this point']
options:
  pool:
    name: 'projects/my-project/locations/us-central1/workerPools/my-privatepool'

As we mentioned earlier, all Cloud Deploy steps also run within Cloud Build workers, so we can also specify a Cloud Build private pool to be used in our Cloud Deploy pipelines. Cloud Build is used for rendering manifests (using Skaffold) as well as deploying to clusters (using kubectl) and we can specify which worker pools to use as well as for which stages.

We do this by updating clouddeploy.yaml and adding an executionConfig to the object definition for each Target (ie. each cluster). Here’s an example using the production cluster from our earlier walkthrough:

apiVersion: deploy.cloud.google.com/v1 
kind: Target 
metadata: 
  name: prod 
description: production cluster 
gke: 
  cluster: projects/PROJECT_ID/locations/us-central1-c/clusters/prod-cluster
executionConfigs:
  - privatePool:
      workerPool: "projects/my-project/locations/us-central1/workerPools/my-privatepool"
    usages:
    - RENDER
    - DEPLOY

Using private pool configurations like this means that you can still leverage Cloud Build and Cloud Deploy, even if you have completely private infrastructure. This is helpful if you’re using hybrid connections between your on-premises networks or other clouds via your VPC, or if you’re just using private GKE clusters.

Summary

In this post I’ve hopefully established a compelling argument for using automation tools and a CI/CD architecture for your software development lifecycle. As I stated earlier, while these tools and practices can enable rapid iteration and development, that’s normally not the goal in and of itself. In order to support fast but safe deployments, all of these additional mechanisms work together to make your deployments more reliable. This means you’ll have much more confidence changing the state of your workloads or fixing them if something goes wrong.

And that’s a wrap! (for now…)

Ten posts on a single subject seems like a good milestone! I’ve found it fascinating digging deep into the world of GKE Enterprise, and I hope through these posts I’ve made it a bit more accessible for anyone out there trying to learn more about these topics.

Stay tuned to my blog for more general Kubernetes and cloud writing, and maybe some other completely different stuff too! 😄

Regardless of where our clusters run or how our applications are configured, security must be a fundamental consideration in any design. The power and flexibility of Kubernetes can bring an increased level of complexity, enlarging attack surfaces and making security considerations all the more important. In my last post, I explored how Cloud Service Mesh can help us to identify our workloads and then encrypt and control their communication. Even though we’ve completed our Service Mesh journey in this series, we’ll now continue the topic of security in this post and learn some of the most useful Kubernetes features that can help us keep our workloads secure. Heads up that some of these are specific to GKE, and some of are just generally available in Kubernetes.

The one thing we cannot afford to do is be complacent about security! A long time ago people used to make terrible mistakes like assuming an application was secure if it was “behind the firewall”, but as we know, the Internet is a zero-trust environment and should be treated as such. The Cloud Native Computing Foundation (CNCF) summarises this with its “4 Cs” that we should care about when it comes to security: Code, Container, Cluster and Cloud. It’s our job to secure each of these things to the best of our ability!

To achieve this, we’re going to learn the following things in this post:

• Leveraging Workload Identity for your apps

• Trusting container deployments with Binary Authorization

• Integrating other Google Cloud security tools

• Securing Traffic with Kubernetes Network Policies

Of course, we can’t cover every Kubernetes security topic in this post! We’re going to investigate some of these specialised areas, but you should already have a solid grounding in Kubernetes security concepts such as Role-Based Access Control (RBAC) and namespaces. If you need a quick brush up, I’d recommend reviewing the documentation here: https://kubernetes.io/docs/concepts/security/

Leveraging Workload Identity for your apps

I’ve mentioned Workload Identity a few times already in this series of posts, and chances are that you have it enabled by default in your fleet settings. Like most Google products, Workload Identity has a self-explanatory name, and provides your workloads with an identity. But why is that important?

One of the most fundamental security concepts you should embed in your system designs is the principle of least privilege. Simply put: this principle states that any component of a system should be granted only the permissions it requires to perform its function, and nothing greater. Okay, but why is that important?

You can consider any component in your system as a potential attack surface, vulnerable to being compromised and used as a jumping off point into other parts of your system. The permissions that a single component is granted will determine what other parts of your system it can access and to what extent (for example, read only or read and write). By reducing the permissions granted to each component you are therefore reducing the size of that attack surface.

Historically, when Google’s Cloud IAM was in its infancy, the principle of least privilege was difficult to achieve. All Google Cloud services run with a service account as an identity, and it was common for compute services (including GKE nodes) to run with the default Compute Engine service account. Some early Google Cloud engineer in their naivety, figured that this service account should have the “Project Editor” role. The upshot of this meant that any compromised service suddenly had complete access to everything in a project – all the other services, all the data, all the APIs.

Fast forward to today when we have an extremely well structured and powerful Cloud IAM service, and a set of carefully crafted predefined roles. Now when we need a workload to access something, we can define an IAM binding that grants a specific set of permissions only, therefore achieving the principle of least privilege. All we have to do is identify ourselves as the service account to which the IAM roles have been bound.

This used to mean downloading and managing service account keys, but this created an additional attack surface! If you accidentally left your keys lying around, someone could use them to essentially steal the identity of a service account and access any systems for which it had permissions. However, recently all of the compute platforms in Google Cloud have been improved so that manual keys are no longer required. Instead, the platform itself will manage a short-lived token to provide access to a service account.

Workload Identity Federation

This capability has now been extended to Kubernetes service accounts within GKE (note that these are Kubernetes objects inside your cluster, not Cloud IAM service accounts – although the two can be used together, more on that in a moment!)

When you run a Pod workload with a Kubernetes service account identity, you can now create an IAM binding for that identity to specific IAM roles and permissions. For example, if your Pod workload needs to access the Cloud Storage API to write to a specific bucket, you can grant the specific granular permission to allow this with an IAM binding, rather than using the default service account of the cluster node.

To do this, you reference the Kubernetes service account as a principal in the membership of the IAM policy binding. The membership identifier is a bit cumbersome however, as it contains your project number, the name of your workload identity pool, a namespace and the name of the Kubernetes service account.

Here’s an example where the project number is 123456123456 and the project name is my-project, which makes the workload identity pool my-project.svc.id.goog. Assuming we have a namespace called frontend and a Kubernetes service account called fe-web-sa, this would make the identifier:

principal://iam.googleapis.com/projects/123456123456/locations/global/workloadIdentityPools/my-project.svc.id.goog/subject/ns/frontend/sa/fe-web-sa

Federated identity works in all GKE clusters, but clusters in fleets have an additional advantage. Rather than an identity pool spanning a single cluster, all clusters attached to a fleet share the same fleet-wide pool. This means that you can create an IAM binding just once, and it will be applied to all clusters in the fleet where the specified namespace and service account name match (bonus points if you remember the concept of sameness that I talked about very early on in this series!)

Let’s walk through an example of creating a workload that access the Cloud Storage API to demonstrate some best practices. We’ll assume we have a storage bucket that contains some files, and a GKE cluster with Workload Identity enabled. In this example, we’ll add the IAM policy binding at the level of an individual bucket, not at the project level.

Following the examples we used earlier to explain the IAM principal membership, let’s go ahead and create a namespace called frontend and a Kubernetes service account called fe-web-sa:

kubectl create namespace frontend
kubectl -n frontend create serviceaccount fe-web-sa

Now that we’ve created a Kubernetes service account, we can simply reference it as the member (or principal) in an IAM policy binding. In this example, we’ll grant the Storage Object User role to a bucket called frontend-files. This predefined IAM role grants access to create, view, list, update and delete objects and their metadata, but it doesn’t grant the user any further permissions to manage ACLs or IAM policies. This is an example of just the right level of privilege for a use-case where a workload needs read-write access to objects in a bucket, without being able to change the bucket itself.

We’ll create the binding with this command:

gcloud storage buckets add-iam-policy-binding gs://frontend-files \
  --member=serviceAccount:my-project.svc.id.goog/subject/ns/frontend/sa/fe-web-sa \
  --role=roles/storage.objectUser

Now, let’s create a workload that actually uses these permissions. In a real-world use case, we can perhaps imagine a front-end web server Pod that grabs its files from Cloud Storage on startup. For testing purposes, we’ll just spin up a Pod that contains the gcloud tool so we can test if our permissions work. We’ll define the Pod as follows:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
  namespace: frontend
spec:
  serviceAccountName: fe-web-sa
  containers:
  - name: test-pod
    image: google/cloud-sdk:slim
    command: ["sleep","infinity"]

Once the container is up and running, we can test permissions by running some gcloud commands that should only be possible with the correct IAM binding. For example, we could list the contents of the bucket:

kubectl -n frontend exec -it test-pod -- gcloud storage ls gs://frontend-files

Or try to create a new file by copying one that already exists inside the bucket:

kubectl -n frontend exec -it test-pod -- gcloud storage cp gs://frontend-files/test1.txt gs://frontend-files/test2.txt

Of course, this isn’t the sort of thing your workloads are likely to be doing, but the key thing is that our Pod uses its service identity to get the permissions it needs, and only those permissions. Thanks to identity federation, we were able to treat our Kubernetes service account as a principal in an IAM binding just like we would with a Cloud IAM service account.

However, at the time of writing there were some limitations with specific APIs and the extent to which federated identities could be used with them. You can find these details documented here: https://cloud.google.com/iam/docs/federated-identity-supported-services

Identity with unsupported APIs

If you need to control access to a specific Google Cloud API in a way that is not supported by federated identity, you can still fall back to a legacy method that achieves the same thing. In this method, we create a Cloud IAM service account with the correct IAM bindings and permissions, then we allow the Kubernetes service account to impersonate the IAM service account.

There are a few more moving parts to this process. Assuming we have already created our Kubernetes service account fe-web-sa and our workload that uses that service account, we now need to create a matching Cloud IAM service account, which in this case we’ll call iam-fe-web-sa:

gcloud iam service-accounts create iam-fe-web-sa

In this scenario, we add the policy bindings to the Cloud IAM service account. So, following our previous example, we’ll grant it the Storage Object User role on the same bucket as before. Remember that Cloud IAM service accounts are identified by an email address in the format of <service-account-name>@<project-name>.iam.gserviceaccount.com which makes our command:

gcloud storage buckets add-iam-policy-binding gs://frontend-files \
  --member=serviceAccount: iam-fe-web-sa@my-project.iam.gserviceaccount.com \
  --role=roles/storage.objectUser

Next, we need to create an IAM policy that allows the Kubernetes service account to impersonate the IAM service account:

gcloud iam service-accounts add-iam-policy-binding iam-fe-web-sa@my-project.iam.gserviceaccount.com \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:my-project.svc.id.goog/subject/ns/frontend/sa/fe-web-sa"

And finally, we annotate the Kubernetes service account so that GKE understands there is a link between the two service accounts:

kubectl annotate serviceaccount fe-web-sa \
  --namespace frontend \
  iam.gke.io/gcp-service-account=iam-fe-web-sa@my-project.iam.gserviceaccount.com

This essentially triggers the same process as identity federation and allows the cluster to obtain and use short-lived credentials automatically. These approaches may seem like extra effort, but once again: it cannot be stressed enough how important least privilege is as a security principle. In the unfortunate event of an attacker gaining access to a component of your system, these practices drastically reduce the further damage they can do.

So, we’ve covered regulating the API access that our workloads have when running in our clusters, but what about regulating what workloads will run in the first place?

Trusting container deployments with Binary Authorization

If you recall the 4 Cs we discussed earlier, the first and most important is “Code”. We can put all kinds of structural security in place, but we’re still trusting the code that runs inside our containers. It’s helpful then to have a way to guarantee that only trusted container images should be allowed to run, and all other containers images should be blocked. This is the purpose of Binary Authorization.

Binary Authorization works with two main concepts: attestations and policies:

• An attestation is essentially an electronic signature, attached to a container image, that confirms that the image has been signed by an attestor. An attestor can run at any stage in your container image build pipeline to provide important safeguards in the build process. For example, you may want to leverage Google’s Artifact Analysis service to scan the metadata of your images and check their vulnerability severity. Or you can use third-party scanning services like Kritis Signer or Voucher or use your own services to check containers as they go through the build process. The important thing is that at each stage, if the attestor is happy that the container is compliant, it creates a signed attestation.

• A Binary Authorization policy then determines which attestations are required for a container image to be deployed. Optional rules can be added to a policy to control which clusters and service identities can deploy an image, and these can be scoped to a namespace if you wish. You can also choose if policies should just evaluate or enforce the rules you have set.

A typical CI/CD pipeline that embeds Binary Authorization would leverage an attestation at each environment stage, as shown below:

Here we can see the usual process of building and pushing an image on a commit. The build pipeline then triggers the vulnerability scan, and we determine if the output of the scan deems the image to be safe or not. For example, an image that contains no CVEs with a severity score greater than five could possibly be considered safe (but your mileage may vary!) At this point, a safe image can generate an attestation that can be checked later.

The pipeline can go on to deploy to a staging environment, and then run quality assurance tests to make sure the updated deployment works as expected. These tests could be automated or manually run by a member of a QA team, and if they are passed, we generate another attestation to say so.

Finally, when the image is due to be deployed to the production cluster, a Binary Authorization policy can ensure that both positive attestations are in place, and check that they have been signed by the relevant attesters. If they are not, the deployment will be denied, protecting the production environment.

Unfortunately, configuring Binary Authorization in a CI/CI pipeline is a complex process which would warrant an entire blog post series of its own, so I can’t cover it here! If you want to try this out for yourself, Google’s own guide is recommended: https://cloud.google.com/binary-authorization/docs/cloud-build

Let’s jump ahead to the final C in the 4 we discussed at the start of this post. Even if our code, container and cluster are air-tight, surely there are some platform tools in Google Cloud that can help us?

Integrating other Google Cloud security tools

You may already be familiar with some of the tools available in Google Cloud that can generally help with platform security. In this section we’ll discuss two of them specifically – Cloud Armor and Cloud IAP – and how they can be integrated into the GKE Enterprise configurations we’ve been discussing so far. Let’s start with Cloud Armor!

Cloud Armor

Google’s Cloud Armor is essentially a Web Application Firewall (WAF), applied as a set of configuration polices which combine with Google’s different load balancing options to filter incoming Layer 7 network traffic and protect your workloads and backends. Cloud Armor policies can comprise multiple rules, and there are several types of rules you can use:

• IP allowlist and denylist rules can filter traffic based on IP or CIDR, and can return a variety of HTTP response codes

• Source geography rules can exclude traffic from specific geolocations

• Preconfigured WAF rules can protect your workloads from common attacks maintained by the Open Worldwide Application Security Project (OWASP)’s Core Ruleset (CSR) list. These include attacks like SQL injection, cross scripting, PHP injection and many more.

• Bot Management rules help you manage requests that may be coming from automated clients or bots. For example, you can force requests to identify themselves through a reCAPTCHA challenge.

• Rate limiting rules can throttle requests or temporarily ban clients that exceed a predefined rate threshold.

You can also write custom rules using Google’s Common Expression Language (CEL). Cloud Armor works with any Google Cloud Load Balancer and is not limited to backends on GKE. However, if you’ve configured your cluster ingress using the GKE Gateway Controller you can easily associate a Cloud Armor policy with your Gateway to protect the services it exposes.

We do this by creating a GCPBackendPolicy object. This object contains details of the additional functionality that should be added to the load balancer in its spec, such as referencing a Cloud Armor policy. The GCPBackendPolicy then targets a specific Service, or in the case of a multi-cluster service, a ServiceImport.

Here’s an example where we have already created a Cloud Armor policy called web-security-policy, and we want to use it to protect a backend Service called store:

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: webstore-backend-policy
  namespace: store
spec:
  default:
    securityPolicy: web-security-policy
  targetRef:
    group: ""
    kind: Service
    name: store

Note that the GCPBackendPolicy must exist in the same namespace as the Gateway that it is attached to, and only a single GCPBackendPolicy object may be used per service. The targetRef points to our store Service; the group parameter is blank because Services belong to the core API group. If we had set up store as a multi-cluster service, we would simply change the targetRef section to refer to the ServiceImport object instead, like this:

  targetRef:
    group: net.gke.io
    kind: ServiceImport
    name: store

At the time of writing, some limited aspects of the GCPBackendPolicy could be applied to the Gateway object itself, not just the service, which could potentially save you a lot of time if you want to apply the same security policy to lots of different services. It’s worth checking the documentation at https://cloud.google.com/kubernetes-engine/docs/how-to/configure-gateway-resources to see if this support has been expanded.

Identity Aware Proxy

Google’s Identity Aware Proxy (IAP) is an additional layer of protection that can be enabled for applications that are exposed via load balancers, in use cases where you only want to grant access to your own organization’s users. As part of Google’s “BeyondCorp” methodology, it provides a simple way to prove a user’s identity rather than having them connect to an internal application using a VPN.

IAP works by intercepting requests that travel through a load balancer and redirecting users to the Google sign-in page. Here they must successfully authenticate themselves with Google’s identity services. This completes the authentication stage of IAP.

Next comes authorization. For a user’s request to be passed to the backend service, IAP checks that the correct IAM policy is in place to allow this. A user requires the IAP-Secured Web App User role assigned to them in the same Google Cloud project as the backend resource.

The benefit of this functionality is that IAP is easy to drop in to provide a highly secure method of authentication and authorization without having to modify any of your backend workloads. The downside is that it only works for users who actually have accounts within your Google organization (ie. your Google domain), because you’ll need to be able to assign IAM roles to them within your project. Just in case this scenario is relevant to you, let’s discuss how we can add IAP to one of our GKE backend services.

Configuring the Consent Screen

The Consent Screen is the user interaction that appears when a user is asked to sign into their Google account to access your service, as shown below.

We’ve all seen this a hundred times; we’re logging in with our Google credentials, but a third-party application wants to know who we are. When you enable IAP for your Google Cloud project, you’ll need to configure a consent screen with the name of your application, some contact details and optionally an image for a logo. Then you’ll need to decide what scopes your consent screen is asking for.

The scopes in your consent screen determine what information you want Google to pass onto your application if authentication is successful. For example, if you’re trying to access a third-party application that needs to write files to your Google Drive, you would be presented with this information at the consent screen. An application can’t access scopes unless they are specifically authorized by the end-user in this way. It might be that all you need is a user’s email address to identify them in your own data, but there are dozens of other scopes you can include if they are necessary.

Once you’re completed the Consent Screen setup, IAP is enabled on your project, but there’s still one more step to take. You’ll need to generate an OAuth 2.0 Client ID so that your IAP integration has access to the correct APIs. You can do this from the Credentials screen in the APIs section of the Cloud Console. Once you’ve created the OAuth 2.0 Client ID, download the credentials file and extract the client ID and secret. We’ll use these in a moment.

Configuring IAP for GKE backends

Any services running in your cluster should now be available as backends that can be secured by IAP (along with any services in your project from other Compute options, such as Compute Engine, App Engine or Cloud Run). You can see a list of these backends to confirm your GKE services are included with this command:

gcloud compute backend-services list

Don’t be concerned with the compute sub-command! Services exposed by the GKE Gateway controller will show up in that list. You can also see this list on the IAP page in the Cloud Console, although at this stage it will show that OAuth is not properly configured. We’ll fix that next!

First, we need to store the OAuth 2.0 secret in a Kubernetes Secret object. Take the secret that you extracted from the credentials file a moment ago and write it into a text file called secret.txt. Then we’ll store this in a Secret called oauth-secret:

kubectl -n store create secret generic oauth-secret --from-file=key=secret.txt

Now we can create the GCPBackendPolicy to attach IAP to our service. In this example, you’ll need to replace <CLIENT_ID> with the OAuth 2.0 client ID from your credentials file:

apiVersion: networking.gke.io/v1
kind: GCPBackendPolicy
metadata:
  name: iap-backend-policy
  namespace: store
spec:
  default:
    iap:
      enabled: true
      oauth2ClientSecret:
        name: oauth-secret
      clientID: <CLIENT_ID>
  targetRef:
    group: ""
    kind: Service
    name: store

Once again, we can change the targetRef like before if we’re using a ServiceImport rather than a Service. It will take a few minutes for the configuration to synchronize, but you can check on its status with this command:

kubectl -n store describe gcpbackendpolicy

This should show you a message like: Application of GCPGatewayPolicy "default/backend-policy" was a success. You should now also see that everything is okay in the IAP page of the Cloud Console:

External users who now try to access your application through the load balancer will be redirected to the Google sign-in process using the consent screen that you configured. That user will need to successfully authenticate with their Google credentials, and they’ll need the IAP-Secured Web App User role in your project’s IAM bindings to be granted access.

Other policies are available to help you configure your Gateway resources, but these two are the most useful regarding application security. For full details you can check out the documentation here: https://cloud.google.com/kubernetes-engine/docs/how-to/configure-gateway-resources

Securing Traffic with Kubernetes Network Policies

In the previous post we described using a collection of AuthorizationPolicy objects with our service mesh to control the flow of traffic between our various workloads. As we’ve already learned, the service mesh approach is very powerful and quite comprehensive. However, it can be difficult to apply control logic between your workloads and other points on the network that exist outside of your mesh. And after reading the last few posts in this series, you may have actually decided that you don’t want a mesh after all!

Thankfully, we still have a Kubernetes-native form of traffic control in the form of NetworkPolicies. While these objects do not require a service mesh, the network plugin for GKE must be updated to support them. You can use the --enable-network-policy argument when creating a new cluster, or enable network policy enforcement on an existing cluster with this command:

gcloud container clusters update my-cluster \
  --update-addons=NetworkPolicy=ENABLED

NetworkPolicies can look a little confusing at first, but once you understand their logic, they are quite easy to read. The primary components of the object are:

• A podSelector: This element chooses which Pods should be affected by the policy

• Policy types: You can include Ingress and Egress rules in a policy, for traffic entering and leaving a Pod respectively.

• The rules themselves: Inside your Ingress and Egress elements, you can specify rules that determine whether traffic is allowed. These rules are based on additional Pod selectors, namespace selectors or IP blocks to match the source of traffic for Ingress rules, or the destination of traffic for Egress rules.

Let’s take a look at the example NetworkPolicy from the Kubernetes documentation to walk through an example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

At the start of the spec is a podSelector, which determines which Pods will be affected by this policy by “selecting” them. You may see podSelectors in other parts of the policy, and this is why indentation levels in YAML are so important! In this example, all Pods that contain the label role with the value db in their metadata will be selected and affected.

If you don’t want to select specific Pod labels, you can instead affect an entire namespace. You can do this by specifying a namespace (either in the YAML or at the time of applying a manifest), and using an empty podSelector:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: all-pods-policy
  metadata: secure-namespace
spec:
  podSelector: {}

Going back to the example from the Kubernetes documentation, once we know which Pods are being affected, we then specify which policyTypes we’re going to include in our policy. Once again, we specify Ingress rules to control how traffic should be allowed into selected Pods, and Egress rules to control how traffic should be allowed out. Each set of rules then gets its own section in our YAML manifest. Incidentally, if you just want to create Ingress rules on their own, you can leave out the policyTypes section entirely. By default, Kubernetes will then expect you only to define the Ingress section.

Next is our ingress section, where we specify the sources from which we will accept network traffic. Put another way, the Pods we selected a moment ago will only receive traffic from sources that match these rules. In our example, we have three different types of source:

• The ipBlock specifies a network range of 172.17.0.0/16, but excludes the specific sub-range of 172.17.1.0/24.

• The namespaceSelector uses labels to match the originating namespace of any Pod. In this case, Pods must come from a namespace that contains the metadata label project with a value of myproject for this rule to evaluate to true.

• The podSelector, used as a source, matches Pods that contain the label role with the value of frontend.

It’s important to note that any of these sources can match for this rule to evaluate to true and for the ingress traffic to be allowed. However, we also have a ports section in our ingress rule. This specifies that only TCP traffic on port 6379 will be accepted. As you can see, this is a fairly restrictive policy.

Finally, we have an egress section. We could use the same variety of sources and selectors as we used for ingress, but in this example, we simply specify an ipBlock and ports parameter. This means that Pods selected by this policy will only be allowed to connect to IP addresses within the specified CIDR range on TCP port 5978.

Designing network policies

So how should we design our network policies? They can quickly become complex and cumbersome, so it's important to understand some basic logic of how they work. As we’ve already seen, network policies can apply two types of restrictions: Ingress for incoming connections and Egress for outgoing connections.

In both cases, if a Pod is not selected by any policy, a connection is allowed (in other words, it is not isolated or restricted). Remember that a Pod can be selected in several different ways as we saw in the previous example, including just by being part of a namespace that has been selected. But if no Ingress policies select a Pod, all inbound connections for that Pod will be allowed. Likewise, if no Egress policies select a Pod, all outbound connections from that Pod will be allowed.

If a Pod is selected by a policy, then only the traffic allowed by the rules of that policy will be allowed. Ingress and Egress are evaluated separately, however. Also, return traffic for any allowed connection is also implicitly allowed.

When deciding how to design our policies, remember that they are additive. Once a policy applies to a Pod, we’re starting with nothing being allowed and adding exceptions with our rules. This means that rules can’t conflict with each other, and the ordering of rules doesn’t matter.

We can create multiple policies that may apply to a single Pod or group of Pods. For this reason, it's recommended to create a policy per single intention. A connection from a frontend to a backend would qualify as a single intention. A backend may also accept connections from elsewhere, but these could be separated into additional policies.

Some sensible defaults

In most environments it's likely that you may select specific groups of Pods for specific connection types, but you may still wish to apply defaults to all other Pods that don’t have their own unique rules. We can use the logic of the NetworkPolicy object to apply some sensible defaults quite easily.

For example, when you’ve finished creating specific ingress rules for some workloads, you might want to make sure that all other Pods are denied Ingress traffic, even if they aren’t specifically selected. This policy will do that:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes:
  - Ingress

This works because we don’t specify a namespace, and we use an empty podSelector. Then we apply an empty ingress policy – essentially specifying that we want to control ingress, but without supplying any rules that allow it. This policy will have no effect on Egress traffic.

Conversely, maybe you want to explicitly allow all ingress connections. The policy is similar, but now we supply an empty ingress rule, thereby matching all incoming traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all-ingress
spec:
  podSelector: {}
  ingress:
  - {}
  policyTypes:
  - Ingress

Remember what we said earlier about policy addition and conflict? What do you think would happen if we added an additional ingress policy to select some Pods and try to control their inbound traffic? The answer is: it would have no effect. No incoming network traffic can now be denied with this blanket policy in place.

Once again, controlling ingress like this has no effect on egress traffic. However, you can create the egress equivalent of these rules simply by changing the policyType, and swapping the empty ingress parameter for an empty egress parameter on the allow-all rule.

Finally, if you want a blanket rule to deny all traffic except for what is specified in other policies, you can deny everything in one go with this policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

NetworkPolicies may seem like an additional chore, but you should simply consider them as an extension of any other configuration you create for your workload, alongside such standard objects as Deployment and Service. Doing the hard work up front means that in the unfortunate event that a workload is compromised or starts misbehaving, its potential to cause further damage inside your system is considerably reduced.

Summary

In this post I’ve tried to cover some of the most essential security topics you need to know in addition to the security aspects of service mesh we already learned about. We’ve revisited Workload Identity, introduced Binary Authorization and discussed integrations with Cloud Armor and Cloud IAP. Finally, we’ve looked at NetworkPolicies and tried to normalize them as part of the standard configuration for any workload. In doing so, we’ve tried to emphasize that steps to improve security are a fundamental part of the way you design and operate your systems and applications. They’re not a “nice to have”; you ignore them at your peril. Do the work now and your future self will thank you!

But of course, we’ve barely touched the surface. Hopefully I’ve convinced you why this topic is important, but the whole world of Kubernetes security is too big to squeeze into a blog series that is ostensibly about operating GKE. I would strongly recommend researching the world of Pod Security Admission and Admission Controllers in general, Role Based Access Control (RBAC), and the various hardening guides available from the Kubernetes project and others.

The next post will be the last in this series (for now)! We may have learned a lot about GKE Enterprise and how to deploy its many features, but so far, we’ve been doing the work manually for the most part. Manual work doesn’t scale and is liable to human error, so to finish the series we’ll learn about automation and how modern DevOps tools and GitOps patterns can help us deploy securely again and again.

See you next time!

Cover image by Tung Lam from Pixabay.

In the last 2 posts we looked at how GKE Service Mesh can help us with things like multi-cluster networking and fleet ingresses, and in the final part of our tour of this particular feature we’ll look at the additional security features a mesh provides; this can be one of the most compelling reasons to implement a mesh in the first place. Historically, many security design decisions ended at the boundary of a single server or a single monolithic application. This was also true for networking components, where entities within a specific network boundary (such as a firewall or VPN) were trusted and not subject to further rigorous security measures.

Cloud native designs change these approaches completely. Organizations including Google, the Cloud Native Computing Foundation and even the US Department of Defense all recommend zero-trust approaches to security, where service identity must always be proven, and permissions granted on a least-privilege basis. This is important when migrating monolithic applications to microservices, as this increases the number of moving parts and thus the attack vector size. Individual services may be compromised and impersonate other services on a network, either disrupting applications or exposing sensitive information.

So, we need a way to make sure that all service interactions are authenticated, authorized and controlled. Luckily, this is something the Service Mesh excels at. In this post we’re going to cover the following main topics:

• Implementing mutual transport layer security to encrypt all traffic

• Adding service account identities to our workloads

• Controlling which services and namespaces can access each other

As we’ve covered the bulk of how Service Mesh works in previous posts, this will be a lighter tour through these topics as they are engineered well into Service Mesh and easy to implement. At the end of the post, we’ll summarize what we’ve learned about Service Mesh, and use these topics to springboard into further security concepts.

Service identity and end-to-end encryption

Let’s start by discussing why securing individual workloads such as microservices is so important. Many people think that performing security up to the boundary of their cloud infrastructure is sufficient. Perhaps you’re terminating TLS connections at your load balancer, or even using a Web Application Firewall to provide additional protections. Once the traffic is inside your network, you may be assuming it is safe. But as our design for applications becomes more abstract and granular, we are introducing some more complexity which in turn leads to a wider attack vector. If a bad actor can infiltrate some vulnerable code in one part of your application stack, what’s to stop them impersonating any other part to steal or damage your data?

A Service Mesh allows us to secure an environment composed of multiple microservices by adding a secure identity to each of our workloads. In this scenario, a single workload may be a Pod, or a collection of replica Pods managed by some kind of controller such as a Deployment or StatefulSet. The workload may represent some part of an overall system, such as the checkout, cart or frontend components we saw in the Online Boutique demo earlier in this series of posts. In a typical cluster, service discovery means that any of these workloads can find and connect to any other workload. Typically, this internal traffic is not encrypted either. That means that if just one component becomes compromised, an attacker may be able to intercept data intended for other components.

To illustrate this, let’s imagine that our frontend workload has been compromised. As it uses the same CIDR as other workloads, it could potentially intercept their network traffic. Attack code could even be run on this frontend to simulate a checkout service and infiltrate the payment service backend.

The first part of remediating this is to encrypt the data sent between individual microservices. Historically this would have introduced a significant management burden, creating, assigning and distributing TLS certificates to each microservice, and making sure that each service trusts the certificate chain used to issue them. But now, Service Mesh can automate this entirely by simply enabling the policy and leveraging Google’s Mesh CA service. We’ll see how to do this in a moment.

The second part of our approach is to be descriptive with which services should be allowed to communicate with which other services, and in what ways. Now that your microservices are encrypted with their own client certificate, they will all have their own unique, provable identity. That means we can create rules to lock down communication between microservices to only the interactions that we know should be happening, therefore reducing the attack surface for bad actors.

Best of all, the implementation of these new security endeavors is all handled by the Envoy sidecar container, so in most cases no changes to actual application containers are required. Let’s look at how we set these features up in our clusters.

Enabling secure two-way communication

Mutual TLS (or mTLS) is implemented by Istio in Cloud Service Mesh using the Envoy proxy sidecar containers running alongside your workloads. mTLS has two operating modes:

• Permissive: Sidecar proxies will use TLS to connect with other sidecar proxies but will also allow non-TLS communication for incoming connections or connections to other workloads without sidecars.

• Strict: Only TLS connections will be allowed.

By default, Cloud Service Mesh will enable permissive mTLS across your cluster out-of-the-box. However, it’s recommended to switch to strict mode to secure your mesh. This configuration is done with a PeerAuthentication object in the istio-system namespace. Istio considers this the root of your mesh and will apply the configuration to every namespace where Istio injection is enabled.

To secure your entire mesh, you could apply the object with the following YAML:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "mesh-wide-mtls"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT

Alternatively, you could use the above YAML to apply a strict mTLS policy to a specific namespace by simply changing the namespace referenced, or removing it entirely from the YAML and adding it dynamically with kubectl -n, which can be useful for scripting.

In some scenarios, we may want to enable strict mTLS only for specific workloads rather than an entire namespace. We can do this by adding selectors to the PeerAuthentication object so that it only targets specific Pods.

For example, if our Pods have the metadata label app=frontend, we could use the following YAML definition:

apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "frontend-mtls"
  namespace: "frontend"
spec:
  selector:
    matchLabels:
      app: frontend
  mtls:
    mode: STRICT

However, Cloud Service Mesh can’t aggregate workload-level policies for outbound mTLS traffic to a service, so we also need to add a matching DestinationRule:

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "frontend-dr-mtls"
spec:
  host: "frontend.demo.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

With these objects in place, incoming requests to the frontend service will now require TLS. We’ll need to be mindful of what incoming connections we expect as unencrypted requests will now be rejected. Usually, we rely on an Ingress gateway to accept a request from an external user (either directly or via a Cloud Load Balancer). The Ingress Gateway Pods would then require mTLS to communicate securely with the frontend.

Using policies to authorize connections

As we’ve already discussed, encrypting network traffic between workloads is just one part of the solution. However, in forcing the use of certificates for mTLS, we can now also trust that the originator of a request is who they say they are. For example, a proxy accepting requests from the frontend workload over TLS can trust that the request really is from frontend because of the certificate being used.

Once we’ve authenticated a request with its certificate, Cloud Service Mesh also allows us to create rules regarding authorization. In other words, we trust the identity of this workload, but do we want to allow them to make the connection?

In a microservices architecture, authorization policies can be useful to control which workloads are allowed to communicate with which other workloads. Remember, in the ephemeral world of Kubernetes, our Pods will have constantly changing IP addresses, so traditional firewall rules simply no longer make sense. Instead, we can use an AuthorizationPolicy to define a set of rules for traffic that we will permit, based on either a workload or namespace identity.

The structure of an AuthorizationPolicy

The AuthorizationPolicy object is a custom resource which, when applied to our cluster, will create rules on all of the affected sidecar proxies in scope. An individual policy object can target the entire mesh, a namespace or an individual workload using selectors in a similar fashion to the PeerAuthentication object we saw earlier.

Policies are then comprised of an action and optionally (although usually!) some rules:

• The action is usually set to ALLOW or DENY the matching request. As we’ll see in a moment, there’s a slightly counter-intuitive way to build sets of rules using multiple ALLOWs without any DENYs for most use cases. In advanced use cases, the action can be CUSTOM which allows you to delegate the access control to an external authorization system. Finally, you can also specify an action of AUDIT, which causes the request to simply be logged and has no bearing on whether the connection is allowed or not. Normally, audit rules are applied in addition to allow and deny rules to assist with troubleshooting.

• The rules define which requests will match and should be affected by the action. Within the object’s rules, we can specify the source of the request in the from section. The to section allows us to specify which operations should be permitted (such as HTTP GET, POST etc.). It may feel like the to section should specify the target of our policy, but don’t forget that we specify this in a selector, if we’re trying to target a specific workload. Finally, we can apply some additional conditions in a when field, including request headers that must match for us to apply the policy.

Let’s see some example AuthorizationPolicy objects to illustrate how these conventions work. Here’s a basic policy to allow all requests to Pods that match the selector app=frontend:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "frontend-ap"
  namespace: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  action: ALLOW

This is obviously a very permissive policy, so we could consider locking it down a bit by adding some approved operations. In the updated version below, we’ll specify that we only accept HTTP GET requests for paths that start with /public or /test:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: tester
  namespace: default
spec:
  selector:
    matchLabels:
      app: products
  action: ALLOW
  rules:
  - to:
    - operation:
        methods: ["GET"]
        paths: ["/public/*", "/test/*"]

Matching keywords like paths can also be used with their negative condition versions, which in this example would be notPaths. Including a negative condition in a policy would mean that the policy applies as normal but will not be applied to paths specified in the notPaths parameter.

For a full list of supported operations, see https://istio.io/latest/docs/reference/config/security/authorization-policy/#Operation

Using service accounts with mesh identities

A common pattern used is to specify policies that grant access based on the identity of the caller. As we’ve already mentioned, Cloud Service Mesh provides each service with a secure identity, so we know who is calling. We can then create policies that only allow requests from the identified callers we choose. However, we don’t use the details of a TLS certificate in our AuthorizationPolicy objects, we instead reference a Service Account.

Service Accounts in Kubernetes give a distinct identity to a workload and work with Role Based Access Control to provide a way to control which objects a workload may access. You’re probably already using them in your own Kubernetes projects; if you’re not, I definitely recommend looking them up in the Kubernetes documentation! So how do they relate to AuthorizationPolicies?

Let’s imagine a typical scenario where we have 2 workloads: frontend and backend, and we want to only allow requests to the backend service if they come from the frontend workload. In other words, we shouldn’t allow direct connections to backend from anywhere else.

Here’s some example YAML we could use to create a unique service account identity for the frontend, and make sure it’s being used in the frontend deployment:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: frontend-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: frontend-image
      serviceAccountName: frontend-sa

We can now create an AuthorizationPolicy for our backend service, which will only allow access from the frontend service account as a principal. In Istio, principals are identified by their certificate authority, which in Cloud Service Mesh on Google Cloud is the name of your project. With a project ID of my-project-id, your policy YAML could look like this:

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "backend-access"
spec:
  selector:
    matchLabels:
      app: backend
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["my-project-id.svc.id.goog/ns/default/sa/frontend-sa"]

When a request comes from the identity of the frontend-sa service account, it will be allowed by this policy, as illustrated below. Requests can also be allowed from entire namespaces rather than individual principles using the namespaces source, and just like other conditions, we can construct rules with negative conditions such as notPrincipals and notNamespaces.

For a full list of supported options for the source of a request, see https://istio.io/latest/docs/reference/config/security/authorization-policy/#Source

Assembling and layering policies

As we mentioned earlier, you can stack ALLOW and DENY policies however you want to, but the way they are processed can be a little counterintuitive. The most important thing to remember is that DENY policies are evaluated first. If a single DENY policy matches a request, that request will be denied before any ALLOW policies have been evaluated. In some systems, it’s customary to use something akin to a DENY ALL rule and then build up specific ALLOW rules based on desired behaviors. With AuthorizationPolicies, that won’t work as the ALLOW rules will never be seen.

The recommended approach for this pattern is instead to use an ALLOW rule that matches nothing. In the absence of other ALLOW rules, this will cause all requests to be denied (in the same spirit as a DENY ALL rule). However, we can now add additional ALLOW rules to grant the policies we want to, and they will still be evaluated. The “Allow nothing” rule looks like this:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-nothing
spec:
  action: ALLOW

Conversely, we might choose to set up an “Allow all” rule. This is similar but contains an empty rules block that matches all workloads:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all
spec:
  action: ALLOW
  rules:
  - {}

With an “Allow all” rule in place, we would have to build multiple DENY policies as well to stop undesirable connections. These would be evaluated first and could still block traffic, before any unmatched request is handled by the “Allow all” rule. As you can imagine, this approach leads to a lot of management overhead.

For this reason, the “Allow nothing” approach is definitley recommended. I just wanted to show you both approaches to help you understand the concepts involved.

Testing policies before enforcing them

Cloud Service Mesh also supports an Istio annotation that applies your policies in a “dry-run” mode. If a policy matches, it will always be allowed but the enforcement result will be written to Cloud Logging. You can enable “dry-run” mode on any AuthorizationPolicy by adding the following annotation to the object’s metadata:

metadata:
  annotations:
    "istio.io/dry-run": "true"

This feature can be useful if you need to test policies on live production traffic without actually enforcing it straight away.

Using policies to define what network connections count as legal and valid is a good way to document your system and enforce an important security principle. If you reduce the permitted connections between proxies to only the traffic that you know is required for your application stack, you are reducing the potential attack surface that could be exploited by a bad actor who has gained access to any component of your system. Policies can also protect components from services that are incorrectly configured or have been recently changed without proper testing.

Summary

We’ve now completed our journey through Cloud Service Mesh, a powerful and fully managed implementation of the popular Istio stack that is a key offering in GKE Enterprise. Hopefully the concepts of Service Mesh have now been demystified for you, and you’ll be able to determine if using Service Mesh is the right choice for your environments and workloads in the future.

Some argue against Service Mesh due to the additional complexity it can introduce. It’s true that using it effectively will require you to think about additional layers of configuration for your workloads. However, with Cloud Service Mesh it's never been easier to get started and leverage the benefits of Istio. Ultimately the level of increased complexity you want to add to your environment may depend on the original complexity of your application stack. Environments that may not warrant a mesh include single stateless services, or long-running applications with consistent levels of demand. Such environments are rare these days however, and if you’re already considering GKE Enterprise, chances are you have a complex microservices stack to deploy. Despite the extra work, a mesh will reward you by making traffic management, observability and security easier in the long run.

While we’ll be moving away from discussing Service Mesh, in my next post I’ll continue with the theme of security. Traffic control is just one part of our security toolbelt in Kubernetes, so stay tuned for more guidance on workload identity, binary authorization, network policies and how we integrate Google Cloud’s native security tools into our GKE workloads.

Cover image by Tung Lam from Pixabay

In this post we’ll continue our journey into the wonderful world of Cloud Service Mesh by exploring how we extend a mesh across multiple clusters, including clusters in different subnets, projects and even clouds! So far, we’ve discussed several scenarios where running multiple clusters can be beneficial, above and beyond simple scalability and high availability. Often there are requirements for elements of a workload stack to run in a specific geographic location, and even on specific hardware on-premises. GKE Enterprise allows us to manage clusters that run on our own hardware, and combining this approach with a service mesh allows us to also leverage things like service discovery and secure microservices in a true hybrid cloud environment.

Cloud Service Mesh provides multi-cluster capabilities and endpoint discovery across GKE clusters in Google Cloud automatically, but extending this approach to managed GKE clusters in other clouds and attached Kubernetes clusters on-premises presents some new challenges. Building on the foundational service mesh topics we covered in the last post, we’ll add these multi-cluster scenarios to our tool belt, before approaching service mesh security in my next post. After that, you’ll have all the knowledge you need to know if service mesh is right for your workloads.

So, in this post, we’ll cover the following topics:

• Understanding the different operating modes of Cloud Service Mesh

• Setting up service mesh discovery across multiple clusters

• Extending service mesh outside of Google Cloud

Just like in previous posts, I’ll also walk through a working example so you can see how all the pieces fit together.

Types of multi-cluster mesh

Google’s Cloud Service Mesh supports several modes of installation and operation, with varying levels of complexity:

• Managed Multi-Cluster Mesh on Google Cloud: This is the most straightforward option and works almost completely “out of the box”, leveraging Cloud Service Mesh as a fully managed service and providing endpoint discovery between all GKE clusters inside Google Cloud. Clusters can exist in the same project or in different projects, providing they run on the same VPC network and in the same fleet. Shared VPC can be used to manage multi-project networking.

• In-Cluster Mesh inside Google Cloud: This option uses the asmcli tool to install the Istio control plane directly into your clusters. It gives you complete control over all service mesh components, but this means you can no longer rely on Cloud Service Mesh as a managed service; you will need to maintain and update the Istio components yourself. In-Cluster Mesh is no longer recommended for clusters running inside Google Cloud, except for in outlier use cases.

• In-Cluster Mesh outside Google Cloud: Using the asmcli tool you can also install Cloud Service Mesh in its non-managed form on GKE clusters running in VMWare, Azure, AWS and even bare metal. Rather than install Istio from scratch, using the In-Cluster mesh service still allows you to observe your mesh through the Google Cloud console, and provides some additional assistance with things like east-west gateways.

The limitation of each of these approaches is that the multi-cluster mesh will only extend to additional clusters in the same environment. A true hybrid mesh would allow the same service mesh installation to extend across your Google Cloud projects, other clouds and even into your own datacenter. At the time of writing, hybrid mesh is still in preview and is not supported by Google. However, we will briefly cover this approach at the end of the post.

Meshing clusters within Google Cloud

Meshing two or more GKE clusters within Google Cloud is supported by default with Cloud Service Mesh. The main prerequisites for service mesh are that all clusters belong to the same fleet, and that connectivity between all Pods in all clusters is allowed. This can be achieved in one of two ways:

• Clusters in the same project on the same network

• Clusters in different project sharing the same network via Shared VPC

There are some additional considerations for clusters in different subnets in a VPC, and private clusters, but we’ll cover those later on. For now, let’s walk through a basic multi-cluster mesh example.

To try out a multi-cluster mesh, we just need to ensure that Cloud Service Mesh is enabled for our fleet, and then create 2 GKE clusters, ensuring that they are registered to the fleet at the time of creation (we covered how to set this up in the last post).

In the following steps we’ll refer to these clusters as cluster-1 and cluster-2. If you’re following along, make sure the output of gcloud container fleet mesh describe shows that everything in your mesh is ready to go.

Testing the mesh with a sample app

To test the cross-cluster mesh, we’ll deploy a very basic Hello World application to both clusters. The app will expose a service that replies with “Hello World”, but with a different version number for each cluster so we can trace which cluster serves the request. Because we now have multi-cluster service discovery, the Service object we call will automatically include Pods from both clusters in its endpoints!

Because we’re going to be running commands on two clusters in this walkthrough, we’re going to employ some handy tricks:

• We’ll write YAML manifest files that contain multiple objects, so we’ll use -l to specify that only the objects that match specific labels should be created.

• We’ll be using the --context option with kubectl to specify which cluster to apply to. You should already have two contexts set up in your kubeconfig for your clusters, but you can rename them to something more convenient like cluster-1 and cluster-2 using kubectl config rename-context.

So, the first thing to do is create a namespace for our app and label it for Istio’s automatic sidecar injection. Using the tricks above, we’ll do this for both clusters:

kubectl --context=cluster-1 create ns sample
kubectl --context=cluster-1 label namespace sample istio-injection=enabled
kubectl --context=cluster-2 create ns sample
kubectl --context=cluster-2 label namespace sample istio-injection=enabled

Now let’s create the helloworld.yaml file that contains the Deployment and Service definitions we need:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
    service: helloworld
spec:
  ports:
  - port: 5000
    name: http
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  labels:
    app: helloworld
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1:1.0
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v2
  labels:
    app: helloworld
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v2
  template:
    metadata:
      labels:
        app: helloworld
        version: v2
    spec:
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v2:1.0
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000

Using this file and our labels trick, we’ll first create the Service object we need on both clusters:

kubectl create --context=cluster-1 -l service=helloworld -n sample -f helloworld.yaml
kubectl create --context=cluster-2 -l service=helloworld -n sample -f helloworld.yaml

And now the Deployment objects. Note that we create a different version on each cluster:

kubectl create --context=cluster-1 -l version=v1 -n sample -f helloworld.yaml
kubectl create --context=cluster-2 -l version=v2 -n sample -f helloworld.yaml

Try using the --context method to check the Service and Deployment objects on both clusters, as well as listing the running Pods. You should see that each Pod has two containers, as the sidecar proxies have been injected.

At this point we’ve got our sample app deployed, but how do we test it? In the next step we’ll create a simple Deployment that we can use to make some in-cluster requests to our newly deployed service. If we make repetitive requests, we should hopefully see the responses getting balanced across both clusters.

In sleep.yaml we essentially create a dummy application that just gives us an in-cluster environment for us to run curl commands and other network tests:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: sleep
---
apiVersion: v1
kind: Service
metadata:
  name: sleep
  labels:
    app: sleep
    service: sleep
spec:
  ports:
  - port: 80
    name: http
  selector:
    app: sleep
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sleep
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sleep
  template:
    metadata:
      labels:
        app: sleep
    spec:
      terminationGracePeriodSeconds: 0
      serviceAccountName: sleep
      containers:
      - name: sleep
        image: curlimages/curl
        command: ["/bin/sleep", "infinity"]
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - mountPath: /etc/sleep/tls
          name: secret-volume
      volumes:
      - name: secret-volume
        secret:
          secretName: sleep-secret
          optional: true

Now deploy the app to both clusters:

kubectl apply --context=cluster-1 -n sample -f sleep.yaml
kubectl apply --context=cluster-2 -n sample -f sleep.yaml

We can now execute a command within the sleep container to test the Hello World application. You can run this test from either cluster, but for simplicity we won’t specify a context in the command examples below.

First, we get the name of a sleep Pod:

kubectl get pod -n sample -l app=sleep

Now we can use that Pod to execute a loop of curl commands to access the Hello World service. In the following example, replace <POD_NAME> with the Pod name you grabbed from the previous command:

kubectl exec -n sample -c sleep <POD_NAME> \
  -- /bin/sh -c 'for i in $(seq 1 20); do curl -sS helloworld.sample:5000/hello; done'

You should see 20 responses, roughly distributed across both clusters. Once again, Pods from either cluster (represented by versions 1 and 2) will match endpoints for the service. Multi-cluster mesh and service discovery in action!

Exposing a meshed service to a load balancer

We can take what we learned in our previous post and expose a multi-cluster service via a service mesh VirtualService and a Gateway. However, bear in mind that in this scenario, the Pods running the ingress itself will still only run in a single cluster (we’ll sum up multi-cluster load-balancing options later on).

First, we need to install the Istio ingress gateway Deployment. Recall from my previous post that these are the standalone envoy proxies running at the edge of our cluster. The easiest way to do this is to use the manifests provided by Google in this repo: https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git

Let’s create a dedicated namespace for the ingress gateway, and label it for sidecar injection. We only need to do this on cluster-1 for this example, so set your kubectl context appropriately.

kubectl create ns gateway-ns
kubectl label namespace gateway-ns istio-injection=enabled

From the git repo directory, navigate to samples/gateways, where you should find a directory called istio-ingressgateway. This contains manifests for the Deployment, along with the other supporting objects we need. Deploy it to your cluster with this command:

kubectl -n gateway-ns apply -f istio-ingressgateway/

With our ingress Pods running, we can now configure the Gateway and VirtualService objects. Remember that the Gateway object defines how the ingress gateway Pods should be configured, including information such as hosts, protocols and ports. The VirtualService object will then define the routes that our gateway should use, mapping traffic from our gateway to our backend services.

First we’ll create the sample-gateway.yaml object and apply it to cluster-1:

apiVersion: networking.istio.io/v1alpha3 
kind: Gateway 
metadata: 
  name: frontend-gateway 
  namespace: sample 
spec: 
  selector: 
    istio: ingressgateway 
  servers: 
  - port: 
      number: 80 
      name: http 
      protocol: HTTP 
    hosts: 
    - "*"

Then we create the sample-virtualservice.yaml object and also apply it to cluster-1:

apiVersion: networking.istio.io/v1alpha3 
kind: VirtualService 
metadata: 
  name: frontend-ingress 
  namespace: sample 
spec: 
  hosts: 
  - "*" 
  gateways: 
  - frontend-gateway 
  http: 
  - route: 
    - destination: 
        host: helloworld 
        port: 
          number: 5000

Our multi-cluster meshed service is now exposed to the world via a Cloud Load Balancer! To test requests, you can find the external IP by listing the services in the gateway-ns namespace:

kubectl -n gateway-ns get svc

Then just hit http://<EXTERNAL-IP>/hello in your browser, substituting <EXTERNAL-IP> for the IP address you noted from the previous command. You should see the Hello World app respond, and if you keep reloading the page you’ll get responses from version 1 (on cluster 1) and version 2 (on cluster 2).

Considering other options for multi-cluster load balancing

So far, we’ve walked through a simple demonstration to help learn the concepts of service mesh in a multi-cluster environment, but we’ve still arrived at an environment where ingress is dependent on a single cluster. This is not going to be suitable for many production use-cases, so where do we go from here?

There are a few different options to choose from for a multi-cluster approach to load balancing that does not require a single point of failure. For extensive details on each you can refer to the documentation here: https://cloud.google.com/kubernetes-engine/docs/concepts/choose-mc-lb-api

The recommended option is to use a Multi-Cluster Gateway. In the example we’ve just set up, our single-cluster gateway simply exposes an external service, connecting itself to a Cloud Load Balancer, but the gateway is an on-cluster resource.

However, as I covered in a previous post, we can leverage the GKE Gateway Controller to provide a fully managed off-cluster controller than interacts with our Gateway classes, VirtualServices and HTTPRoutes, while providing reliable and highly available cross-cluster load balancing. The GKE Controller works with both Standard and Autopilot clusters, and it simply provides a management layer over standard open-source Kubernetes objects, rather than requiring any proprietary implementation.

Another alternative is the GKE Multi Cluster Ingress controller. This controller also runs off-cluster as a managed service, but it requires less configuration by simply leveraging the MultiClusterService object. However, that means it doesn’t support all the advanced features of the Gateway API we’ve learned about.

The final option is to configure your own Network Endpoint Groups (NEGs) for Cloud Load Balancers. This is an advanced option as it will require the use of Terraform and the Config Connector to automate the assignment of Pod IPs to Load Balancer backends. This might be a useful option for outlier use cases such as combining backends from GKE clusters with serverless workloads, private endpoints or hybrid cloud endpoints in the same load balancer. As always, the best way forward is to consider exactly what your use case is and make your design choices based on exactly what your use case needs to achieve. The simplest choice may often be the best but try to consider if you may need additional features in the future!

Using clusters in different subnets

In the example we walked through, all nodes in our clusters existed on the same VPC subnet. By default, GKE will create firewall rules that allow traffic between nodes on the same subnet, so in this scenario no additional firewall configuration is required. However, in many cases you will be using different subnets, and you must create these rules yourself.

It’s recommended to allow all ports between your cluster nodes on their internal IP addresses. We can achieve this by simply adding a firewall rule to the VPC. However, we will need lists of all subnet CIDRs and network tags used by our nodes. We can do some Bash trickery to obtain these lists and store them in variables, which we’ll use in a moment:

function join_by { local IFS="$1"; shift; echo "$*"; }
ALL_CLUSTER_CIDRS=$(gcloud container clusters list --project $PROJECT_1 --format='value(clusterIpv4Cidr)' | sort | uniq)
ALL_CLUSTER_CIDRS=$(join_by , $(echo "${ALL_CLUSTER_CIDRS}"))
ALL_CLUSTER_NETTAGS=$(gcloud compute instances list --project $PROJECT_1 --format='value(tags.items.[0])' | sort | uniq)
ALL_CLUSTER_NETTAGS=$(join_by , $(echo "${ALL_CLUSTER_NETTAGS}"))

Now we have the ALL_CLUSTER_CIDRS and ALL_CLUSTER_NETTAGS variables populated, we can run the following command to create a firewall rule called isitio-multicluster-pods. You’ll just need to substitute YOUR_VPC with the name of your VPC network:

gcloud compute firewall-rules create istio-multicluster-pods \
    --allow=tcp,udp,icmp,esp,ah,sctp \
    --direction=INGRESS \
    --priority=900 \
    --source-ranges="${ALL_CLUSTER_CIDRS}" \
    --target-tags="${ALL_CLUSTER_NETTAGS}" --quiet \
    --network=YOUR_VPC

All your Pods will now have free-flowing communication between clusters, providing the underlying VPC routes support the connections.

Considerations for private clusters

If you are using private GKE clusters, you will not have public endpoints that can be accessed by the service mesh control plane. This means you have quite a few more hoops to jump through to get a multi-cluster mesh working. The instructions are quite complex, so rather than list them all here I will simply discuss them so you understand what they accomplish, then point you at the Google Cloud documentation page that will build the necessary commands for you.

At a high level:

• You will first need to configure endpoint discovery. This involves manually creating remote secrets to represent the private IPs of the cluster (because public IPs are not available). You will then need to apply each cluster’s secrets to the other clusters.

• Then you will need to configure authorised networks for the private clusters. This will involve getting the CIDR ranges used for Pods from each cluster and adding them as authorized networks to other clusters.

Just to make this process even trickier, these instructions are detailed across multiple pages of documentation! The instructions for endpoint discovery can be found here: https://cloud.google.com/service-mesh/docs/operate-and-maintain/multi-cluster#endpoint-discovery-declarative-api and the instructions for opening ports on private clusters can be found here: https://cloud.google.com/service-mesh/docs/operate-and-maintain/private-cluster-open-port

A final consideration for private clusters is that they will not, by default, have access to the Internet to download the container images from Docker Hub that we have used in this post. To solve this, you can either manually download and add container images to Google’s Artifact Registry or configure a Cloud NAT connection for Internet access.

Using Cloud Service Mesh Outside Google Cloud

As I mentioned earlier, Cloud Service Mesh can be installed on your GKE clusters running outside of Google Cloud, on VMWare, in AWS and Azure, and even on bare metal. This is achieved by using Google’s asmcli tool to install an in-cluster control plane that is managed via Cloud Service Mesh. Your clusters must still belong to your GKE Enterprise fleet and will require connectivity to Google Cloud APIs.

The in-cluster service mesh supports either using Google’s Mesh CA service or the Istio CA service as a certificate authority. This is used when creating mutual TLS (mTLS) certificates, which I’ll explore in a future post. Mesh CA is a reliable and scalable managed service specifically designed for managing workload mTLS certificates, and it's recommended to use the Mesh CA for your service mesh. You can optionally choose to use an existing Istio CA if you already have one set up, if you require a custom CA, or if there’s another reason not to use Google’s CA service, but this is a complex outlier use case.

With this in mind, let’s look at the steps required to enroll a cluster that exists outside of Google Cloud. These instructions should work for any of the supported platforms I’ve talked about already.

Readying your cluster for service mesh

First, we’ll need to download Google’s asmcli tool. You should do this wherever you connect to your Kubernetes clusters (for example, your local computer). You can download the tool and make it executable with these commands:

curl https://storage.googleapis.com/csm-artifacts/asm/asmcli_1.20 > asmcli
chmod +x asmcli

Next, we need to create a cluster role binding so that we have the necessary permissions to create RBAC roles for service mesh. We’ll do this with the kubectl command, so it's important that you have authentication set up for your cluster in your local .kube/config file. How you achieve this will depend on where your cluster is running.

For each cluster context, run the following command to create the necessary permissions, swapping out you@youremail.com for the user account that you identify with when connecting to Kubernetes:

kubectl create clusterrolebinding cluster-admin-binding \ 
  --clusterrole=cluster-admin \ 
  --user=you@youremail.com

Now we can run the asmcli validate command. This will check that your project and cluster will support the service mesh’s minimum requirements, and that the necessary permissions and APIs are set up. This command will also download and extract some useful sample manifest files:

./asmcli validate \ 
  --kubeconfig KUBECONFIG_FILE \
  --fleet_id FLEET_PROJECT_ID \
  --output_dir DIR_PATH \
  --platform multicloud

To replace the placeholders:

• KUBECONFIG_FILE should point to your local .kube/config file for authentication.

• FLEET_PROJECT_ID refers to the project that owns your GKE fleet.

• DIR_PATH specifies a local directory where asmcli will write installation files and sample manifests.

If asmcli detects any errors with your environment or clusters that might prevent service mesh from operating, it will report this to you and hopefully provided some suggested fixes. You will most likely see a few warnings that certain things need to be enabled (such as namespaces, cluster labels etc.), but we can actually get the asmcli command to do that for us in a moment.

Installing service mesh components

The next step is to install the service mesh control plane using the asmcli command. This process will register our clusters to the fleet (if they don’t already belong to it), install the service mesh components for the control and data planes, configure the mesh to trust the fleet’s workload identity, and finally set up remote secrets so that multiple clusters in the same fleet can trust each other.

Now that we’ve validated our environment, we can run the following command to install everything we need, substituting the same placeholders as last time:

./asmcli install \
  --fleet_id FLEET_PROJECT_ID \
  --kubeconfig KUBECONFIG_FILE \
  --output_dir DIR_PATH \
  --platform multicloud \
  --enable_all \
  --ca mesh_ca

Note the --enable_all option, which will correct any missing configuration that was identified during the validation stage. The --ca option specifies that we want to use Google’s managed Mesh CA service. When the command completes successfully, you will see a helpful note on how to enable sidecar injection.

The service mesh should now be deployed and running! The last thing to do before we attempt any deployments is to enable the sidecar auto-injection. You should be able to use the command that you captured previously from the output of the asmcli installation. If you don’t have this command to hand, all you need to do is find out the revision label for the isiotd deployment. We can find this out with this command:

kubectl -n istio-system get pods -l app=istiod --show-labels

Just look for the istio.io/rev label in the output. You should see something like istio.io/rev=asm-1206-0. You can then use this label to apply to a namespace like this:

kubectl label namespace my-namespace istio.io/rev=asm-1206-0 --overwrite

At the time of writing there was one final quirk that you may need to deal with: application monitoring and logging are not enabled by default for clusters outside of Google Cloud. We definitely want to see all the helpful visualisations in the Cloud Service Mesh dashboard, so we need to take steps to switch this functionality on.

An object called stackdriver in the kube-system namespace manages the integration of application logging. We can edit this object in situ and set the value of enableCloudLoggingForApplications to true. This can be done with the kubectl edit command:

kubectl –n kube-system edit stackdriver stackdriver

Look for the enableCloudLoggingForApplications option in the spec and set it to true. Hopefully this option will be automated in future releases!

Meshing multiple clusters externally

Now we understand how to deploy Cloud Service Mesh to clusters outside of Google Cloud, we can start to build up an external multi-cluster mesh. This solution is technically supported by Google provided that all clusters exist within the same single environment, whether that is in GKE on VMware, bare metal, Azure or AWS.

At a high level, we can mesh multiple clusters together by following these steps:

• Creating an east-west gateway between the clusters

• Exposing local services through the gateways

• Enabling endpoint discovery

Google’s guide on how to set up this external multi-cluster mesh has a few different dependencies. Be careful, because you’re now at the cutting edge of multi-cluster technology, where things break easily!

Assuming we have two clusters in one of the supported environments, and that we’ve already installed Cloud Service Mesh, let’s walk through how to action these steps and get our clusters talking to each other. We’ll need a handful of files from the Anthos Service Mesh packages git repo we obtained earlier (available from https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/tree/).

You will also need the istioctl tool installed, which you can obtain from https://istio.io/latest/docs/setup/getting-started/

Inside the git repo, navigate to the asm/istio/expansion folder, where you’ll find a shell script called gen-eastwest-gateway.sh. This script takes in options about your environment and dynamically generates an IstioOperator object, which is then piped via the istioctl command into your cluster.

Assuming the kubectl context for your first cluster is called cluster-1, run the following command for GKE on VMware or GDVC Bare Metal clusters:

gen-eastwest-gateway.sh \
    --revision asm-1213-3 | istioctl --context cluster-1 \
    install -y --set spec.values.global.pilotCertProvider=kubernetes -f -

Or this version of the command if the clusters are running in Azure or AWS:

gen-eastwest-gateway.sh \
    --revision asm-1213-3 | istioctl --context cluster-1 \
    install -y --set spec.values.global.pilotCertProvider= istiod -f -

Repeat the instructions for your other cluster, using the appropriate kubectl context.

Next, we’ll create the Gateway object that exposes services across both clusters. In the same git repo directory, you’ll find a script called expose-services.yaml. Run this for each of your cluster contexts, for example:

kubectl --context cluster-1 apply -n istio-system -f expose-services.yaml

Finally, we enable endpoint discovery by asking the asmcli tool to create the mesh. Sadly, at the time of writing, the tooling is still quite immature. Outside of Google Cloud, asmcli doesn’t have an easy way to identify clusters. To create the mesh, you must pass in individual kubeconfig files that contain the authentication credentials for each individual cluster. It’s likely that these credentials are all in one big .kube/config file right now, but you’ll need to manually split them out into individual files. Then you can run the command like this:

./asmcli create-mesh FLEET_PROJECT_ID \
  KUBECONFIG_FILE_1 KUBECONFIG_FILE_2

Replace FLEET_PROJECT_ID with the Google Cloud project ID that hosts your GKE fleet and substitute the other parameters with the filenames of the individual cluster credentials files you just created, and finally, you should have a multi-cluster mesh set up and ready to go! Feel free to return to the Hello World example from earlier in this post and test it out on your new non-Google fleet. Services should now be able to communicate across clusters via the gateways you configured.

Configuring a true hybrid mesh

A true hybrid mesh would involve creating a single service mesh, including full service discovery, across multiple clusters that exist in different environments. At the time of writing, this capability was in preview for clusters that span GKE on Google Cloud and GKE on VMWare or bare metal. If you want to give this a try, see the documentation at https://cloud.google.com/service-mesh/docs/operate-and-maintain/hybrid-mesh. Needless to say, unless the offering and tooling has evolved considerably by the time you read this, the solution is not recommended for production environments!

Summary

We are now two-thirds of the way through our exploration of Cloud Service Mesh and service mesh principles in general. This post may have you thinking that the technologies involved are simply too immature and unstable to risk in production, despite the extra functionality that a service mesh provides. This is often the side effect of a cloud provider like Google Cloud taking a dynamic open-source project (in this case: Istio) and attempting to turn it into a managed service. The technology doesn’t slow down, and while we are usually willing to do some work “under the hood” on our own projects, we usually have high expectations of fully managed services. It’s likely that Google will continue to refine the offering of Cloud Service Mesh so that one day soon the experience in all environments, or across hybrid deployments, is just as smooth as when you mesh clusters inside Google Cloud.

In the meantime, Cloud Service Mesh inside Google Cloud can be considered quite stable, and running the service on a single external cluster is well supported by the existing tooling. While Google continues to improve its tools, you could consider managing your own Istio deployments for hybrid or multi-cluster scenarios – after all, outside of Google’s customer base, this is how lots of other people are managing meshed container workloads.

In my next planned post I’ll finish our Service Mesh journey by discussing its potential for securing our workloads. Thankfully, this is an area where the technology is mature and works well across any environment once you’ve done the hard work of installing the mesh!

In the first part of this series, we’ve focused on the new concepts that GKE Enterprise introduces for managing your clusters such as fleets and configuring features and policies. We’ll now pivot to look more closely at the advanced features that GKE Enterprise provides for managing our workloads. Since the early days of Kubernetes, workloads have typically been deployed following a microservices architecture pattern, deconstructing monoliths into individual services, and then managing and scaling these components separately. This provided benefits of scale and flexibility, but soon enough issues developed around service discovery, security and observability. The concept of a Service Mesh was introduced to address all of these concerns.

But what is a Service Mesh? How do you use one and do you even need one? Once people have mastered the complexity of Kubernetes, introducing a Service Mesh seems like yet another complicated challenge to master. There are arguments on both sides for the necessity of a Service Mesh and the benefits it provides versus the resources it consumes and the complexity that it either solves or introduces. By the end of this post, hopefully you’ll have a good understanding of these issues and know when and where using a Service Mesh is right for you.

So here’s what we’re going to cover today:

• Understanding the concept and design of a Service Mesh

• How to enable Service Mesh components in your GKE cluster and fleet

• Creating Ingress and Egress gateways with Service Mesh

• Using Service Mesh to provide network resilience

We’ll focus on the fundamentals of Service Mesh in this post, and then build on that knowledge in the next two posts as we leverage Service Mesh for multi-cluster networking and security.

What is a Service Mesh?

To fully explain the concept of a Service Mesh, it helps to consider the interconnected parts of a Kubernetes cluster from the point of view of the control plane and the data plane.

• A typical cluster may be running many different containers and services at any time, and these workloads, along with the nodes that are hosting them, can be considered the data plane.

• The control plane on the other hand is the brain of the cluster. It comprises the scheduler and controllers that make decisions about what to run and where. Essentially, the control plane tells the data plane what to do.

But the control plane in Kubernetes has limitations. For example, most common patterns for network ingress only provide flexibility at the ingress layer, without leaving much control over traffic or observability inside the cluster. A Service Mesh is an attempt to extend these capabilities by providing an additional control plane, specifically for service networking logic. This new Service Mesh control plane works alongside the traditional control plane, but it can now provide advanced logic to control how workloads operate and how their services connect to each other and the outside world.

Some of the most useful features of a Service Mesh are:

• Controlling inter-service routing

• Setting up failure recovery and circuit breaking patterns

• Microservice-level traffic observability

• Mutual end-to-end service encryption and service identity

Once you’ve learned how to implement these new features, you should have a better idea of where they can be useful and if you want to use them or not for your own workloads.

Istio and the Sidecar pattern

The Service Mesh in GKE Enterprise is powered by Istio, one of the most popular open-source Service Mesh projects. Istio provides advanced traffic management, observability and security benefits, and can be applied to a Kubernetes cluster without requiring any manual changes to existing deployments. For more information about Istio, see the website at https://istio.io/

Istio deploys controllers into the control plane of your clusters, and gains access to the data plane using a sidecar pattern. This involves deploying an Envoy proxy container that is run as a sidecar container into each workload Pod. This proxy takes over all network communication for that Pod, providing an extension of the data plane. The proxy is then configured by the new Istio control plane. Hence the terminology of a mesh: Istio is overlaying the services in your cluster with its own web of proxies. Here’s a basic illustration of this concept:

Deploying Service Mesh components

Service Mesh can be installed in a few different ways when using GKE Enterprise. The recommended approach is to enable the managed Service Mesh service, which provides a fully managed control plane for Istio. You can enable Service Mesh from the Feature Manager page in the GKE Enterprise section of the Google Cloud console, as shown below. Once Service Mesh is enabled for your fleet, new clusters registered to it will automatically have Service Mesh installed and managed. You can optionally sync your fleet settings to any existing clusters; however, you will need to enable Workload Identity on these clusters first if you haven’t already.

Using the managed option for Service Mesh reduces the load on your clusters because most of the control plane work is offloaded to the managed service. Enrolled clusters will still need to run the mdp-controller deployment, which requires 50 millicores of CPU and 128Mi of RAM, and the istio-cni-node daemonset on each node, which requires 100 millicores of CPU and 100Mi of RAM per node.

If you have a specific use case outside of the managed service, you can use Google’s asmcli tool to install the Istio Service Mesh directly to your clusters and bypass the managed service. You may choose to do this for example if you need a specific Istio release channel that isn’t aligned to your GKE release channel, or if you need to integrate a private certificate authority. Generally, the managed service is recommended to reduce the complexity involved.

Enabling sidecar injection

At this stage we’ve just set up the control plane for our Service Mesh, and the next thing to do is to set up the data plane. As we mentioned earlier, the data plane comprises an Envoy proxy sidecar container running inside our application Pods. This proxy takes over all network communication for our applications and communicates with the Service Mesh control plane.

Conveniently, we can enable the automatic injection of sidecars into our workloads. This is done at the namespace level and can be achieved simply by adding a metadata label to a namespace. For example, if we have a namespace called frontend, we can label it for sidecar injection with this command:

kubectl label namespace frontend istio-injection=enabled

Any new Pod now created in this namespace will also run the Envoy proxy sidecar. This won’t affect existing Pods unless they are terminated, and then potentially recreated by their controller object (such as a Deployment). A quick way to tell if sidecar injection has worked is to simply look at the output from kubectl get pods and note that you should have one more ready container than you used to (for example, 2/2 containers ready in a Pod).

Alternatively, you can manually inject the sidecar container by modifying the object configuration, such as a Deployment YAML file; the istioctl command line tool can do this for you. For more information see https://istio.io/latest/docs/setup/additional-setup/sidecar-injection/#manual-sidecar-injection

Ingress and Egress gateways

The next stage in setting up our Service Mesh is to consider gateways, which manage inbound and outbound traffic. We can optionally configure both Ingress gateways to manage incoming traffic, and Egress gateways to control outbound traffic. Ingress and Egress gateways comprise standalone Envoy proxies that are deployed at the edge of the mesh, rather than attached to workloads.

The benefit of using Service Mesh gateways is that they give us much more control than using low level objects like Service or even the Kubernetes Ingress object. These previous attempts have bundled all configuration logic into a single API object, whereas our Service Mesh lets us separate configuration into a load balancing object – the Gateway – and an application-level object – the VirtualService.

In the OSI network model, load balancing takes place at layers 4-6 and involves things like port configurations and transport layer security. Traffic routing is a layer 7 issue and can now be handled separately by the Service Mesh.

We’ll talk more about VirtualServices later in this post, but for now let’s just set up an Ingress Gateway. The minimum requirement for this will be a deployment of the istio-proxy container, a matching service and the necessary service accounts and RBAC role assignments. For scalability it’s also a good idea to attach a Horizontal Pod Autoscaler to the deployment. Thankfully, Google have done the hard work for us and provided a git repo here: https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git that contains all the necessary object definitions.

Before we go ahead and apply those manifests though, we quickly need to think about namespaces again. A gateway should be considered a user workload, so it should run inside a namespace to which your users, or developers, have access. Depending on the way your teams are set up in your organization you may wish to have a central dedicated namespace just for the Ingress Gateway, for example gateway-ns, or you may wish to create gateways in the same namespace as the workloads they serve. Either pattern is acceptable, you just need to choose the one that works for the way you manage user access to your clusters.

For now, let’s go ahead and create a dedicated namespace for the gateway:

kubectl create ns gateway-ns

Just like we demonstrated earlier, we need to apply a metadata label to this namespace that will enable Istio’s auto-injection:

kubectl label namespace gateway-ns istio-injection=enabled

Now we can go ahead and apply the manifests from the git repo to create the Ingress Gateway. Inside the git repo in your local filesystem, change into the samples/gateways directory, and then apply the manifests to our chosen namespace with this command:

kubectl apply –n gateway-ns –f istio-ingressgateway

This will apply all of the objects in that directory, including the deployment, the autoscaler, RBAC configuration and even a Pod disruption budget. At this stage this might feel eerily familiar to just deploying a plain old Ingress controller, but we have a few more moving pieces to deploy before it will start to make sense.

Deploying sample microservices

Continuing to use Google Cloud’s demo repo, let’s go ahead and deploy the Online Boutique application stack. This is a neat microservices demonstration that will deploy multiple workloads and give us great visualisations in the Service Mesh dashboard later.

From the samples/online-boutique/kubernetes-manifests directory of the git repo in your local filesystem, create all the required namespaces with this command:

kubectl apply –f namespaces

We’ll also need to enable sidecar injection on each namespace. We can do this with a handy bash for loop:

for ns in ad cart checkout currency email frontend loadgenerator payment product-catalog recommendation shipping; do 
  kubectl label namespace $ns istio-injection=enabled 
done;

Then we’ll create the service accounts and deployments:

kubectl apply –f deployments

And finally, the services:

kubectl apply –f services

You should now be able to see that all Pods in the namespaces we’ve created have 2 containers running in them, because the Envoy proxy has been successfully injected. But so far, these are still all the regular Kubernetes objects that we already know about. When do we get to the Istio CRDs?

Service Entries

The demo Online Boutique application makes use of a few external services, such as Google APIs and the Metadata server. When your workloads need to access external network connections, they can of course do this directly. Making a network request to a service that is unknown to the mesh will simply pass through the proxy layer. But what if you could treat external connections as just another hop in your mesh, and benefit from the observability this could give you? That’s the job of the ServiceEntry object, which allows you to define external connections and treat them as services registered to your mesh.

Back in our git repo, move to the samples/online-boutique/istio-manifests directory and take a look at the allow-egress-googleapis.yaml file. In this manifest, service entries are created for allow-egress-googleapis and allow-egress-google-metadata, and both entries specify the hosts and ports required for the connections.

ServiceEntry objects can also be used to add sets of virtual machines to the Istio service registry and can be combined with other Istio objects (which we’ll learn about soon) to control TLS connections, retries, timeouts and more.

We’ll apply this manifest to our cluster, followed by the frontend-gateway.yaml file in the same directory. This will create two more Istio CRD objects, a Gateway and a VirtualService.

Now at this stage we’ve deployed all kinds of workloads and objects, so we need to slow down and take a step back to explain how they all work together. Here’s a rough diagram of the request from a user getting all the way to Pods that serve the frontend workload – this is basically the website element of the Online Boutique demo stack:

Firstly, our Google Cloud Load Balancer provides our actual endpoint on the Internet with an anycast IP address. Requests received by the load balancer are routed through the istio-ingressgateway Service object to Pods in the istio-ingressgateway Deployment. Recall that this is the Ingress Gateway we deployed right at the start.

At the bottom of the diagram is our actual frontend Deployment, comprising Pods that will receive traffic from the frontend Service. So far, these are standard Kubernetes objects. So, let’s explore the new CRDs in between.

Gateway

Although we already created the istio-ingressgateway Deployment that actually provides the Ingress Gateway, we haven’t yet configured it. The Gateway CRD object describes the gateway, including a set of ports to expose over which protocols and any other necessary configuration. In the definition used by our example, we specify that port 80 should be exposed for HTTP traffic, and that we accept traffic for all hosts (denoted by the asterisk in this section of the manifest). The istio: ingressgateway selector in the Gateway CRD object tells Istio to apply this configuration to the Pods in our istio-ingressgateway Deployment, because if you take a look at that manifest, you’ll see a matching label in its metadata.

For more detail about what can be accomplished with the Gateway object see https://istio.io/latest/docs/reference/config/networking/gateway

Virtual Services

The VirtualService object configures the routing of traffic once it has arrived through our Gateway. In our example manifest we specify a single HTTP route so it will capture all traffic, and we specify that the “backend” for this traffic is a host called frontend. Hosts simply represent where traffic should be sent, and could be IP addresses, DNS names or Kubernetes service names. In this case, we can get away with using a short-name like frontend because the Service its referring to exists in the same namespace as the VirtualService.

The VirtualService object also references the frontend-gateway Gateway object we created earlier. This is a pattern of Istio, where by referencing the Gateway like this, we apply our VirtualService configurations to it. This is similar to how our Gateway object applied configurations to our ingress gateway Pods.

The VirtualService CRD allows for some very flexible configuration. Although this example routes all traffic to a single backend, we could choose to route traffic to different hosts based on different routing rules which would be evaluated in sequential order. For example, routing rules can specify criteria such as HTTP headers, URL paths, specific ports or source labels. We can match routing rules based on exact strings, prefixes or regular expressions. And we can even specify policies for HTTP mirroring and fault injection.

For more detail about the VirtualService CRD, see https://istio.io/latest/docs/reference/config/networking/virtual-service/

Service Mesh dashboards

With the Online Boutique sample stack running, we can access the external endpoint provided by the load balancer and browse around a few pages. Just use kubectl to get the service in the gateway-ns namespace and copy its external IP.

Back in the Kubernetes section of the Google Cloud console, we can choose Service Mesh from the sub-menu, and see a list of services that our mesh knows about, as shown in the screenshot below.

Straight away you should be able to see some of the benefits of using Istio! Because the Envoy proxies capture all network information, from the moment a request enters the ingress gateway and even while traffic traverses between microservices, we suddenly have a much higher level of observability than we would when using traditional Kubernetes objects. We can view the traffic rate for each microservice, along with its error rate and latency. Selecting individual services will also allow us to create Service Level Objectives (SLOs) and alerts for when they are not met. This is a powerful level of observability!

The dashboard also provides a graphical topology view, breaking down how each microservice is connected through its Pods, service and Istio service, as shown below. Part of the magic of the Service Mesh is that we don’t need to tell Istio how all of our microservices are connected together. The mesh topology is automatically generated by Istio simply observing network connections between services, thanks to the Envoy proxies.

Advanced Configuration

The Online Boutique example, while it does contain multiple microservices, is still quite a simple example of deploying a default mesh over existing services. However, using the new CRDs we have available from Istio we could add some advanced configuration that might prove useful. Let’s explore some more of Istio’s capabilities with some theoretical scenarios.

Advanced routing patterns

As we mentioned earlier, the VirtualService object can be used to implement some advanced routing patterns. A useful example of this is inspecting the user-agent HTTP header and routing the request accordingly. We can create conditional routes like this simply by adding a match to the http section of the object:

http:
  - match:
    - headers:
        user-agent:
          regex: ^(.*?;)?(iPhone)(;.*)?$
      route:
        destination:
          host: frontend-iphone
  - route:
    - destination:
        host: frontend
        port:
          number: 80

In this example, a regular expression finds user agents from iPhone users and directs their requests to a different service: frontend-iphone. Any other requests will continue to be handled by the frontend service. Any standard or custom HTTP header can be used.

Complex matching rules can be based on other parameters such URI and URI schemes, HTTP methods, ports and even query parameters. Once you have defined matching criteria for the different types of requests you want to capture, you specify a destination host. In the Online Boutique example, these are simple internal hostnames matching the Service objects that have been created. However, we can also provide more granular instructions for routing by combing VirtualServices with a DestinationRule.

Subsets and Destination Rules

The DestinationRule object applies logic to how a request should be routed once the VirtualService has determined where it should be routed. These objects let us configure advanced load balancing and traffic policy configurations, as well as implement patterns for canary and A/B testing.

A common pattern is to define the exact load balancing behaviour you require. For example, to apply the LEAST_REQUEST behaviour to requests for the frontend service, we could create the following object:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: frontend-destination
spec:
  host: frontend
  trafficPolicy:
    loadBalancer:
      simple: LEAST_REQUEST

With this object applied, endpoints handling the lowest current number of outstanding requests will be favoured by the load balancer. An alternative policy would be ROUND_ROBIN that will simple round-robin across all endpoints, although this is generally considered unsafe for many scenarios. These configurations can also be applied to a subset of endpoints, and in general, subsets can be used to implement traffic weighting and test patterns.

A common approach to testing changes with production traffic is the canary pattern, where a small subset of traffic is routed to a newer version of a workload to test for any issues. If the service is reliable, the new version of the workload can be promoted. Previously, this pattern could be achieved simply by creating Deployments that were sized to a particular ratio; for example, Deployment A containing 8 Pods and Deployment B containing 2 Pods, then using Deployment B for a canary workload. A Service routing traffic to all Pods would logically hit the canary workload 20% of the time.

However, this low-level approach is not scalable. With Istio CRDs we can define multiple routes in our VirtualService and simply assign a weighting to them. We do this by adding a subset to each destination, along with a weight. In our DestinationRule we then define these subsets, by adding additional label selectors that will match the appropriate Pods.

Here’s an example of an updated VirtualService definition for our frontend. We will now route 5% of all traffic to the canary version of the frontend:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-ingress
  namespace: frontend
spec:
  hosts:
  - "*"
  gateways:
  - frontend-gateway
  http:
  - route:
    - destination:
        host: frontend
        subset: prod-frontend
      weight: 95
    - destination:
        host: frontend
        subset: canary-frontend
      weight: 5

We now need to create a matching DestinationRule, which defines additional configuration for the frontend host we’re referencing in our route. Inside the DestinationRule, we create the definitions for the subsets that are referenced in the VirtualService. Those subsets simply specify the extra label selectors that should be used.

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: frontend-destination
spec:
  host: frontend
  trafficPolicy:
    loadBalancer:
      subsets:
      - name: prod-frontend
        labels:
          version: v1
      - name: canary-frontend
        labels:
          version: v2

Putting the objects together, we can see that 95% of the time requests will be routed to Pods that are part of the frontend service, but also have the label version:v1 in their metadata, while 5% of the time requests will be routed to Pods in that service with version:v2 instead.

Timeouts and Retries

Our new Service Mesh objects also provide some configuration options that can help with network resilience. For example, within the route section of our VirtualService object, we can specify a maximum timeout time to prevent a request hanging excessively waiting for a response. Returning a timeout error within a more appropriate timeframe can help us to spot errors more quickly, particularly when they are set up with appropriate SLOs and monitoring.

By default, failed requests will be retried by the Envoy proxy. Failures are sometimes due to transient network problems or simply an overloaded service, so with luck a retry may succeed. However, you may wish to tune this behaviour to enhance your overall application performance.

In this example, we add configuration to the route section of our VirtualService to specify that only 3 retries should be attempted, and that each retry should have an individual timeout of 3 seconds:

http:
  - route:
    - destination:
        host: frontend
        subset: prod-frontend
      retries:
        attempts: 3
        perTryTimeout: 3
        retryOn: connect-failure,409

Here we’re also using retryOn to tell the Envoy proxy specifically that retries should only be attempted if we received a connection timeout or the HTTP 409 Too Busy response.

Circuit Breaking

Circuit breaking is an interesting design pattern for distributed systems that is designed to prevent overloading or cascading failures across services. Much like a circuit breaker in your home will trip and stop electricity flowing to a faulty circuit, this pattern is designed to isolate a problematic system so that requests are no longer sent to it if we know for certain that the service has failed.

In Istio and Envoy, circuit breaking is achieved through the concept of “outlier detection”, in other words, detecting network behaviour that is abnormal. In a DestinationRule, we can define what an outlier behaviour would look like, which will trigger the ejection of Pods from a backend if that behaviour definition is met. Envoy continues to monitor ejected Pods and can automatically add them back into a load balancing pool if they return to normal behaviour.

Here’s an example:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: frontend-destination
spec:
  hosts: frontend
  subsets:
  - name: prod-frontend 
  - labels:
      version: v1
    trafficPolicy:
      connectionPool:
        tcp:
          maxConnections: 100
      outlierDetection:
        consecutive5xxErrors: 3
        interval: 1s
        baseEjectionTime: 2m
        maxEjectionPercent: 50

In this configuration we have now limited the total number of concurrent connections to version 1 of our frontend to 100 by defining a connection pool within our traffic policy. Then we’ve created some parameters for outlier detection. If the service returns three consecutive HTTP 500 errors within a 1 second interval, the behaviour is considered abnormal, and the circuit breaker will trip. The Pod responsible for the behaviour will be ejected from the load balancing pool; this essentially means Istio will not consider it a candidate to receive requests, even though the Pod itself is not changed. We have specified that the base ejection should be 2 minutes, but the actual ejection time will be multiplied by the number of ejections, providing a kind of exponential backoff. Finally, we have also stated that we can only ever eject half of the Pods in this service.

Injecting faults

Fault injection is another interesting tool used to help test the resilience of a microservice architecture stack. If you’ve never done this before, you may think it’s odd to deliberately make services fail some of the time, but it’s better to know how your interdependent services will cope during a testing phase than to find out once everything has gone live! Historically, fault injection has required complex instrumentation software, but once again we can set this up using the all-powerful VirtualService object.

There are two different ways to simulate problems with the VirtualService object, which are defined in the fault subsection of the http section. The first is delay, which will create a delay before a request is forwarded from the preceding Envoy proxy, which can be useful for simulating things like network failures. The second option is abort, which will return error codes downstream, simulating a faulty service.

http:
  - route:
    - destination:
        host: frontend
        subset: frontend-production
    fault:
      delay:
        percentage:
          value: 2.0
        fixedDelay: 5s
      abort:
        percentage:
          value: 1.0
        httpStatus: 503

In the snippet above, we apply both techniques. A 5 second delay will be added to 2% of requests, while 1 in 100 requests will return an HTTP 503 error. We could apply this sort of configuration to a staging environment to test how well our other microservices deal with such faults. For example, you might be able to determine how well frontend services cope if some backend services fail or are degraded a percentage of the time.

Mirroring traffic

A final outstanding feature of Istio is the ability to create a complete mirror of all service traffic, for the purposes of capture and analysis. In the VirtualService object, the mirror section can define a secondary service which will be sent a complete copy of all traffic. This allows you to perform traffic analysis on traffic without interrupting any live traffic, and your mirror service will never interact with the original requestor.

http:
  - route:
    - destination:
        host: frontend
        subset: frontend-production
    mirror:
      host: perf-capture
    mirrorPercent: 100

In the above snippet, a copy of 100% of production traffic is also sent to a service called perf-capture. This could typically be an instance of a network or performance analysis tool such as Zipkin or Jaegar. Alternatively, traffic mirroring can also be used for testing and canary deployments. This way you can send real traffic to a canary service, without ever exposing it to your end-users.

Summary

In this post I’ve attempted to demystify the complex world of Service Mesh. As you’ll now appreciate, Istio and its CRDs add a whole new level of complexity to the control plane of our clusters, however they also provide us with some incredible new powers and abilities in terms of service management and observability. We’ve learned how to define the logic of application and network load balancing through Service Mesh, how to route requests to backend services and subsets of those services, and how to set up advanced routing and load balancing configurations. We’ve also looked at some features that would be extremely difficult to obtain without a Service Mesh, such as circuit breaking and traffic mirroring.

You might now be considering whether Service Mesh is right for your application, your cluster or even your fleet. But don’t decide just yet! In this post we’ve just laid down the foundations of Service Mesh. Over the next couple of posts I’ve got planned I’ll continue to explore its capabilities, first in multi-cluster scenarios, and finally in how it can help us with microservice application security. Then you’ll either be ready to fully embrace the mesh for all your projects, or never switch it on again!

So far in this series we’ve been focussing on how to build various types of clusters, and of course we’ve lightly touched on the concept of fleets while doing so. As we know, fleets are collections of GKE clusters, and we can use this top-level organisational concept to enable features and assign configurations across all clusters in a fleet.

In this post we’ll expand our knowledge of fleets and learn how they are used for advanced patterns of network and cluster isolation, and how we manage ingress across multi-cluster fleets. Fleets are an advanced grouping concept that allow us to manage collections of clusters based on different ideas of separation, such as deployment environment or even business unit. Fleets also allow us to distribute workloads and services easily across multiple clusters for high availability and failover.

In my previous post, we looked at an example of enabling configuration and policy management across all clusters in a fleet. But fleets also enable us to scale to multiple clusters while encouraging the normalisation of configuration and resources across clusters inside the same fleet. Google recommends the concept of sameness across all clusters in a fleet, using configuration management to create the same namespace and identity objects. This makes it much easier to deploy workloads that can span clusters in different ways, helping to provide advantages like geo-redundancy and workload isolation. By the end of this post, you’ll be able to identify which of Google’s recommended fleet design patterns would be most appropriate for your organisation!

Specifically, we’re going to learn:

• Examples of Google-recommended fleet designs

• The concepts of north-south and east-west routing

• Use-cases for deploying workloads to multiple clusters

• How to set up multi-cluster services and gateways

Let’s jump in!

Multi-cluster Design Patterns

We've already seen fleets in action in previous posts in this series, but let’s take a step back and truly familiarise ourselves with the fundamentals of fleets on GKE Enterprise before continuing any further.

Fleets, as we’ve said, are logical grouping of clusters and other configuration artifacts that belong to those clusters. There is a one-to-one mapping of projects and fleets: a single host project may hold a single fleet. However, as we’ll see in a moment, it is common to maintain multiple host projects and fleets, and clusters in different projects can belong to a fleet in another project.

Features such as configuration management and policy controllers are enabled at the fleet level. While it is possible to be selective about which clusters in the fleet are affected by these features, it’s a recommended best practice to normalise your fleet and apply configurations to all clusters. Clusters can only belong to a single fleet, but by grouping and managing multiple clusters into a fleet, we move the isolation boundary of trust from individual clusters to the level of the fleet itself. This means that there is an implied level of trust between individual clusters, and we must now consider the trust boundary from the point of view of different fleets and projects. If you have followed Google’s best practices for the organizational hierarchy of your cloud resources already, it’s likely that you already have similar project-based trust boundaries already established.

Let’s look at a multi-cluster example to explore this approach further.

A multi-cluster example

For this fictional example, we will consider an organisation with users in Europe and Asia, with a development team in the USA. The organisation currently deploys production workloads to GKE clusters in the europe-west1 and asia-southeast1 regions within the same Google Cloud project. Additionally, within that project, a staging GKE cluster is maintained in the us-central1 region. Development work takes place on an on-premises GKE cluster in the organisation’s USA office.

Within each production cluster we can imagine several namespaces for different applications, or frontend and backend services. When deciding how these clusters should be set up in fleets, we need to consider the concepts of isolation and consistency. If clusters are consistent, they may belong to the same fleet, however a degree of isolation is usually required to keep a production environment safe from unwanted changes. Right now, all the clusters belong to the same project and fleet, which may not be the best approach. So let’s look at a few different design patterns to see how we can achieve different levels of isolation and consistency, which should help you choose which one is right for your team.

Maximum isolation with multiple fleets

Let’s rearrange our clusters across different projects which are separated based on their operating environment: production, staging or development. As I mentioned earlier, this is a common approach to managing Google Cloud projects themselves, not just GKE clusters.

In a high-security environment this gives us the strongest possible isolation between clusters. However, we have now increased our operational complexity because we are managing three different fleets, which means three different sets of configuration management and feature settings. For example, some extra effort will be required to maintain consistency across namespaces and policies in the three sources of truth used. Additionally, development teams will need permissions configured across both development and staging fleets.

Project isolation with a shared fleet

To simplify management while retaining some resource isolation, let’s modify our design pattern to use a single fleet even if our clusters remain in different projects:

Using a single fleet achieves the maximum level of “sameness” across our clusters. In other words, we are back to using a single source of truth to define configurations, policies and namespaces. However, we haven’t eradicated the complexity completely, we’ve merely shifted it to a different set of tools. Now we’ll need to manage different namespaces for different environments (such as frontend-staging and frontend-prod), and potentially rely on a service mesh to decide which services may communicate across cluster and environment boundaries. This arrangement does however guarantee consistency between our development and production environments, which is important when testing application changes before promoting them to production.

Production and non-production fleets

Our final approach achieves a compromise, combining staging and development clusters into a single fleet to reduce some management complexity, while keeping the hard isolation of a separate fleet for production. This approach additionally allows you to test changes to fleet features and policies before promoting them to production.

These examples represent just some basic examples of multi-cluster configurations. You may choose one of these or a different variation based on your own team or business requirements. Generally, Google’s recommended best practices suggest that you minimise the differences between production and non-production environments. If developers cannot build and test applications in an environment with parity to production, the chances of successful changes and deployments are reduced. This can be achieved through enforcing configuration management and using GitOps practices to manage the differences between a production and non-production source of truth.

There are other reasons to run multiple clusters despite separating environments. Often, an organization will want to run clusters in multiple locations to bring services closer to their users. Running multiple clusters in different regions also significantly increases availability in the event of downtime in a particular region, which may be a requirement of your organization. Sometimes, data must be held in a specific locality, such as data on European consumers which may be subject to EU data legislation.

Now, to help us map out use cases for multiple clusters, it's important to understand the conceptual terms of north-south and east-west routing in GKE Enterprise.

North-South routing

If you visualise your system in a diagram, think of the outside world at the top of the drawing and all your clusters next to each other below that. North-south routing in this pattern represents traffic or communications coming into your system from outside. Typically, this would be traffic from the Internet received via a Load Balancer, but it could also be communications inside Google Cloud that are still considered outside of your fleet of clusters. The important thing is the communication enters your system via a north-south route.

East-West routing

East-west routing by comparison represents communication between your clusters. The traffic has already entered your system, or it may have originated there, but it may be routed to different clusters inside the system based on different variables such as availability and latency. In a moment we’ll look at some basic patterns to help us understand these concepts, and this will lay the foundation for us to explore more advanced east-west patterns when we get to service mesh later in the series.

Multi-cluster Services

As we’ve already established, there are many use cases for running multiple clusters such as increased availability, capacity or data locality. Now that we’re running multiple clusters within a single fleet, we can start treating the fleet like one big virtual cluster. But how do we enable services and workloads to communicate across the cluster boundaries of our fleet?

We’re of course aware of the humble Kubernetes Service object, which creates a ClusterIP and a logical routing to a set of Pods defined by some label selectors. However, a Service only operates within the confines of a single cluster. To access a Service running on a different cluster within the same fleet in GKE Enterprise, we can configure something called a Multi-cluster Service (MCS). This configuration object creates a fleet-wide discovery mechanism, allowing any workload to access a Service anywhere in the fleet. Additionally, MCS objects can reference services running on more than one cluster for high availability.

To create an MCS, you export an existing service from a specific cluster. The MCS controller will then create an import of that service on all other clusters in the fleet. The controller will configure firewall rules between your clusters to allow Pod to Pod communication, set up Cloud DNS records for exported services, and use Google’s Traffic Director as a control plane to keep track of service endpoints.

At the time of writing, MCS objects were only supported on VPC-native GKE clusters running in Google Cloud, where the clusters can communicate either on the same VPC network (including a Shared VPC network) or a peered VPC network. MCS services cannot span clusters outside of a single project. Depending on your use case, these may be significant constraints. However, the MCS object is designed to be a simplistic approach to extending a standard Service object. As we’ll see later in future posts, more advanced patterns can be achieved using a service mesh.

An example multi-cluster service

Let’s walk through a basic example where we have a fleet of clusters providing a middleware service for our application stack, in this case, an imaginary service called checkfoo. As illustrated below, our clusters are distributed geographically so that frontend services run as close to our users as possible, but right now the checkfoo service only runs on a single cluster in the europe-west2 region. For simplicity, we’ll exclude the actual Pods from these diagrams. All clusters belong to the same fleet.

Because the checkfoo service runs on a ClusterIP on the prod-eu-1 cluster, it is currently only accessible from that cluster. So let’s fix that! The first thing we’ll need to do is enable multi-cluster services on the fleet. This can be done from the Cloud Console, or via the command line:

gcloud container fleet multi-cluster-services enable

Next, we’ll need to enable Workload Identity Federation for our clusters. When we set up multi-cluster services, GKE deploys a component to our clusters called the gke-mcs-importer, which needs permissions to read information about your VPC network and incoming traffic.

Side note: Workload Identity Federation is a very useful feature, which allows you to map Kubernetes service accounts to IAM service accounts and leverage those service accounts’ permissions for your Kubernetes workloads. You can read more about it here: https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity

If you’re using Autopilot clusters, Workload Identity Federation will already be enabled by default. I think it may soon be the default for standard clusters too, but if you’re working with existing clusters, you may have to enable it manually. Here’s an example for our prod-eu-1 cluster:

gcloud container fleet memberships register prod-eu-1 \
   --gke-cluster europe-west2/prod-eu-1 \
   --enable-workload-identity

Once Workload Identity Federation is enabled, we need to grant the required roles to the gke-mcs-importer service account (don’t forget to replace my-project-id in these examples with your own project ID):

gcloud projects add-iam-policy-binding my-project-id \
    --member "serviceAccount:my-project-id.svc.id.goog[gke-mcs/gke-mcs-importer]" \
    --role "roles/compute.networkViewer"

All that’s left to do now is to create a ServiceExport object on the cluster that is hosting our service. This simple object essentially just flags the target Service object for export and will be picked up by the MCS controller. Note in the example below, our service is running a namespace called foo:

kind: ServiceExport
apiVersion: net.gke.io/v1
metadata:
  namespace: foo
  name: checkfoo

There is quite a bit of scaffolding for the controller to set up the first time you create a ServiceExport, so it may take up to 20 minutes to synchronise changes to all other clusters in the fleet. Further exports and updates should only take a minute or so.

The process includes creating the matching ServiceImport object on every cluster, allowing the original service to be discovered, as well as setting up Traffic Director endpoints and health checks, plus the necessary firewall rules. Finally, MCS will register a cross-cluster FQDN in Cloud DNS that will resolve from anywhere in the fleet. In this example, the FQDN would be: checkfoo.foo.svc.clusterset.local (combining the service name checkfoo, with the namespace foo).

Note that at this stage, we could deploy a further checkfoo service to the prod-eu-2 cluster as well. Unlike a regular Service, an MCS can also use clusters as selectors, selecting any matching Pod on any cluster where those Pods may be running. The MCS provides a simple east-west internal routing for multi-cluster services, but as we’ll learn next, it provides a backbone for north-south incoming requests as well.

Multi-cluster Gateways

Controlling incoming traffic or network ingress for our Kubernetes clusters has taken many forms over the years as it has evolved. We started by integrating Load Balancer support into the humble Service object, then moved on to more advanced implementations of ingress with Ingress and IngressController objects. The current way forward is based on the Kubernetes project’s Gateway API, a more role-oriented, portable and expressive way to define service networking for Kubernetes workloads.

Inside Google Cloud, the GKE Gateway Controller implements the Gateway API and provides integration with Google’s cloud Load Balancing services. The Gateway controller is a managed service that runs in your project, but not inside your clusters themselves. It watches for changes in managed clusters, and then deploys and manages the necessary Load Balancing and other network services for you. The Gateway controller has an extensive list of features:

• Support for internal and external load balancing, with HTTP, HTTPS and HTTP/2

• Traffic splitting and mirroring

• Geography and capacity-based load balancing

• Host, path and header-based routing

• Support for enabling and managing supporting features such as Google Cloud Armor, Identity-Aware Proxy and Cloud CDN

The Gateway Controller is available as a single-cluster or multi-cluster implementation. Conceptually the two approaches are the same, but we’ll explore the multi-cluster implementation in this post on the assumption that your fleets will be running more than one cluster.

Configuring Multi-cluster Gateways

If you’re already familiar with the Kubernetes Gateway API, it will help you understand how Google has implemented this API to support a multi-cluster gateway. If this is completely new to you, you might want to review the documentation here first.

The GKE Gateway Controller provides you with multiple GatewayClasses that can be used to define a gateway. These are:

• gke-l7-global-external-managed-mc for global external multi-cluster Gateways

• gke-l7-regional-external-managed-mc for regional external multi-cluster Gateways

• gke-l7-rilb-mc for regional internal multi-cluster Gateways

• gke-l7-gxlb-mc for global external Classic multi-cluster Gateways

You choose a class when you create the Gateway object, as we’ll see in a moment. Just like in our previous example, a typical scenario for a multi-cluster gateway would involve multiple clusters, but these must all be in the same fleet and have Workload Identity Federation enabled.

You’ll need to choose one of your clusters to act as the config cluster. This cluster will host the Gateway API resources (Gateway, Routes and Policies) and control routing across all your clusters. This does however make the config cluster a potential single point of failure, because if its API is unavailable, gateway resources cannot be created or updated. For this reason, it's recommended to use regional rather than zonal clusters for high availability. It doesn’t have to be a dedicated cluster – it may also host other workloads, but it does need to have all of the namespaces that will be used by target clusters set up, and any user who will need to create ingress services across the fleet will need access to the config cluster (although potentially only for their own namespace).

Assuming we already have multi-cluster services enabled in our fleet (as described earlier), we can now enable the multi-cluster gateway by nominating a config cluster, such as the prod-us cluster from our previous example:

gcloud container fleet ingress enable \
    --config-membership=projects/my-project-id/locations/us-central1/memberships/prod-us

Next, we need to grant IAM permissions required by the gateway controller. Note that in this command, you will need your project ID and your project number, which you can find in the Cloud Console. In this example, we’re using the fake project number of 555123456555 just so you can see how this references the relevant service account:

gcloud projects add-iam-policy-binding my-project-id \ 
    --member "serviceAccount:service-555123456555@gcp-sa-multiclusteringress.iam.gserviceaccount.com" \ 
    --role "roles/container.admin"

We can now confirm that the GKE Gateway Controller is enabled for our fleet, and that all the different GatewayClasses exist in our config cluster, with the following two commands:

gcloud container fleet ingress describe
kubectl get gatewayclasses

We’re now ready to configure a gateway! For this example, we’ll deploy a sample app to both of our European clusters, then configure the services to be exported as a multi-cluster service, and finally set up an external multi-cluster Gateway.

We’ll use the gke-l7-global-external-managed-mc gateway class, which provides us with an external application load balancer. When we’ve finished, external requests should be routed to either cluster based on the cluster health and capacity, and the proximity of the cluster to the user making the request. This provides the lowest latency and best possible service for the end-user. The app itself is just a basic web server that tells us where it’s running to help us test this.

Another quick aside: This stuff is cutting edge! At the time of writing, the command to enable ingress and set up the controller didn’t exactly have a 100% success rate. If at this stage you don’t see any GatewayClasses in your config cluster, you can force the controller to install with this command:

gcloud container clusters update --gateway-api=standard --zone=<cluster zone>

Substitute --region=<cluster-region> for regional clusters. Also, if you see some classes, but not the multi-cluster ones (denoted with the suffix mc), try disabling the fleet ingress feature and re-enabling it sometime later (after you have forced the controller to install). Hopefully some of these bugs will get ironed out as the features mature!

Deploying the app and setting up the multi-cluster service

Using our existing fleet, we’ll now create a namespace across all our clusters, and deploy a test app to just our EU clusters. Then we’ll set up the Gateway and HTTPRoute on our config cluster, and test ingress traffic from different network locations. Let’s imagine we’re creating something like an e-commerce store, but again we’ll just be using an app that shows us the Pod it’s running from.

First, we'll create the store namespace on all the clusters in the fleet:

kubectl create ns store

You need to repeat this for every cluster; you could use different contexts in kubectl or a tool like kubectx.

Next, we’ll create the store deployment. This is just a basic 2-replica deployment that uses Google’s whereami container to answer requests with details about where it is running. We'll use the following YAML, and we’ll create this deployment on the prod-eu-1 and prod-eu-2 clusters. (Just write the YAML to a file and use kubectl apply):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: store
  namespace: store
spec:
  replicas: 2
  selector:
    matchLabels:
      app: store
      version: v1
  template:
    metadata:
      labels:
        app: store
        version: v1
    spec:
      containers:
      - name: whereami
        image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.20
        ports:
        - containerPort: 8080

We’ll also create a Service object for each deployment, as well as a ServiceExport, so that the MCS controller can pick it up and create a ServiceImport on every cluster in the fleet. Once again, the YAML required is very simple:

apiVersion: v1
kind: Service
metadata:
  name: store
  namespace: store
spec:
  selector:
    app: store
  ports:
  - port: 8080
    targetPort: 8080
---
kind: ServiceExport
apiVersion: net.gke.io/v1
metadata:
  name: store
  namespace: store

These objects are applied to the prod-eu-1 and prod-eu-2 clusters, where the previous deployments were also applied. Unlike our intentions in the first part of this post, we’re not doing this for cluster-to-cluster communication, but to help our Gateway find our services on either cluster. A corresponding ServiceImport will be created on any cluster that does not host the ServiceExport (in this case, prod-us), and this object will be used by our Gateway as a logical identifier for a backend service, pointing it at the other cluster endpoints. Using a ServiceImport as a backend instead of a Service means our target Pods could run on any cluster in the fleet!

Here’s what it all looks like now:

With our multi-cluster workload running happily in our fleet, we can now set up the necessary objects to expose it to the outside world. As we’ve already enabled the multi-cluster gateway controller, we can now create a Gateway object which defines how we would like to create and leverage a Load Balancer.

We’ll use the gke-l7-global-external-managed-mc class for our Gateway, which specifies that we want to use the global HTTPs load balancer with multi-cluster support. The separation of duties enabled by the Kubernetes Gateway API means that we create the Load Balancer now with this object, but we can configure HTTP routes (in other words, its URL map) later.

We’ll deploy this configuration object to our config cluster (in our example, prod-us):

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: external-http
  namespace: store
spec:
  gatewayClassName: gke-l7-global-external-managed-mc
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      kinds:
      - kind: HTTPRoute

Finally, we’ll add an HTTPRoute to the config cluster to configure how we would like the load balancer to route requests to our backend services (or service imports in this case):

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: public-store-route
  namespace: store
  labels:
    gateway: external-http
spec:
  hostnames:
  - "store.example.com"
  parentRefs:
  - name: external-http
  rules:
  - backendRefs:
    - group: net.gke.io
      kind: ServiceImport
      name: store
      port: 8080

This is a very basic example, where all requests to the load balancer that contain the hostname store.example.com in the header will be routed to the ServiceImport called store. Of course, you have the full power of the Kubernetes Gateway API at your disposal, and you could configure your HTTPRoute object to route to multiple different backends based on URL path, HTTP headers or even query parameters. For more details see https://gateway-api.sigs.k8s.io/api-types/httproute/

After a few minutes, the Gateway should be ready. You can confirm its status with this command:

kubectl -n store describe gateways.gateway.networking.k8s.io external-http

Now we just need to grab the external IP address to test a connection. We get this with a modified kubectl command:

kubectl get gateways.gateway.networking.k8s.io external-http -o=jsonpath="{.status.addresses[0].value}" --context prod-us --namespace store

We can use curl from the command line to test the external IP, passing in a header to request the hostname we specified in the HTTPRoute. In this example, replace the sample IP address with the one you got from the previous command:

curl -H "host: store.example.com" http://142.250.200.14

You should see some output similar to the screenshot below, showing a response from the Pod that includes its cluster name, Pod name, and even the Compute Engine instance ID its running from. Which region serves your request will depend on your current location (or if you’re using the Cloud Shell terminal, the location of your Cloud Shell VM).

We can experiment with the locality-based routing by creating test VMs in different regions. For example, if we create a test VM in europe-west4, we should see our request served by the prod-eu-2 cluster:

We have now successfully configured a multi-cluster service running across different regions in our fleet, with a single global HTTPS load balancer directing incoming traffic to the backend that is closest to our users and in a healthy state. Additionally, we know from the power of the HTTPRoute object that we could introduce URL maps and more complex route matching for multiple backend services and service imports. In a later post we’ll look at how to integrate additional Google Cloud services into gateways, such as Cloud Armor for web-application firewall protection, and the Identity-Aware-Proxy (IAP).

The final state of our example fleet, including the flow of requests, is shown below. Note that despite the seemingly complex logic we’ve set up, the actual flow of traffic goes directly from the load balancer frontend to the Pods themselves via Network Endpoint Groups (NEGs):

Summary

In the first few posts in this series we introduced some of the new fundamental concepts that set GKE Enterprise apart from a standard Kubernetes infrastructure deployment. Along the way we’ve gradually built up our knowledge of fleets, and you should now have a good idea of how you might design your own systems based on multiple clusters and potentially multiple fleets.

Of course there are many ways to achieve a desired outcome, and each will have their own tradeoffs. It’s worth sketching out the possibilities before committing to any design, and if you commit to reading future posts in this series (thank you!) you should still feel free to revisit the ideas in this post on how to achieve different levels of isolation or management complexity.

Hopefully you’ve finished this post with an understanding of how to create a truly multi-cluster deployment. While the fully managed Gateway controller dynamically provisions and operates load balancers for us, you may be wondering about its limitation to back-ends that run inside Google Cloud clusters. There are two scenarios where this limitation may impact you:

1. If your back-end service runs in an on-premises cluster, but you still want to use a Google Cloud load balancer. According to Google Cloud, this isn’t exactly a preferred design pattern. However, it is still achievable outside of GKE Enterprise services using Network Endpoint Groups with hybrid connectivity. For more information see https://cloud.google.com/load-balancing/docs/negs/hybrid-neg-concepts.

2. If your front-end workloads run in Google Cloud, but they need to communicate with workloads on external clusters.

While this arrangement isn’t supported by a standard MCS, we’ll soon learn that this can be achieved, along with so much more, in the amazing world of service mesh. We’ve got the fundamentals out of the way, so in the next post I can really dig into the nuts and bolts of Istio and use service mesh for traffic routing, service identity, service security and more!

In this series we have now covered some of the basics of building clusters in GKE Enterprise, and we can begin to explore some of its more useful features, starting in this post with configuration management. Before you drift off to sleep, let me tell you why this is important stuff.

For those of us who have worked in IT for a long time, the concept of configuration management may bring back memories of tools like Puppet and Chef, which were attempts at maintaining the state of systems by declaring their configuration with some sort of code. Their overall success was mixed, but they inspired new ways of thinking about automation and systems management, such as declarative infrastructure and GitOps – both topics we will explore in this post. There are many different reasons to consider implementing a configuration management system, but probably the two most relevant concepts are automation and drift.

As systems engineers, we always strive to automate manual tasks. If you successfully introduce Kubernetes to your organisation, you may soon find yourself managing multiple clusters for multiple teams, which could mean lots of repetitive work installing, upgrading, fixing, and securing. Wouldn’t it be better to automate this based on a standard template of what clusters should look like? And whether you complete this work manually or via automation, what happens next? Configurations are likely to drift over time due to small or accidental changes or even failures in resources. If the state of your clusters has drifted from what it should be, you’re going to have a much harder time maintaining it.

So, let’s learn about some of the configuration management tools GKE Enterprise gives us to solve these problems. By the end of this post, you will hopefully understand:

• The concepts of declarative infrastructure and GitOps

• How to use GKE’s Config Sync to manage cluster state with Git

• How the Policy Controller can be used to create guardrails against unwanted behaviour in your clusters

• How you can extend configuration management to other Google Cloud resources with the Config Controller

Let’s get started!

Understanding Declarative Infrastructure

The tools and technologies we will learn about in this post are all based on the concept of a declarative model of configuration management, so it’s important to understand this fundamental concept. In the history of DevOps, there have generally been two schools of thought about which model should be used in configuration management: declarative or imperative.

In a nutshell, the declarative model focuses on what you want, but the imperative model focuses more on how you should get it. To clarify that further, a declarative model is based on you declaring a desired state of configuration. If you’re familiar with Kubernetes but came to it years after systems like Puppet and Chef fell out of fashion, then you may never have considered that people did this any other way. Declarative models make the most sense because if the desired state of a system is declared in code, the code itself becomes living documentation of the state of a system. Further, if the software that applies this code is clever enough, it can maintain the state – in other words, guarantee that it remains true to what is declared, and thereby solve the problem of configuration drift.

The imperative model, by focusing on the steps required to reach a desired state, is a weaker approach. The administrator of an imperative system must be sure that each step is taken in the correct order and manage the dependencies and potential failure of each step. Because an imperative model does not declare the end-state of a system, it is harder to use this model to maintain a state once it is built, so it doesn’t help solve the drift problem.

A quick side note: You may absolutely disagree with me here 😀 My descriptions of declarative and imperative models are based on my own experience and opinion. You are perfectly entitled to disagree and may have experiences that are very different to my own! For example, some imperative systems claim to have solved the drift problem. For this post’s purposes however, all the tools we will discuss implement a declarative model.

Infrastructure as Code

While configuration management tools have existed in some form or another for over 20 years now, the concept of Infrastructure as Code emerged as we began to use these tools to manage the lifecycle of virtual machines and other virtualised resources in addition to software and services. The most successful of these tools is arguably Terraform, which has broad support for multiple providers allowing you to take a declarative approach to defining what virtual infrastructure you would like, what other services you need to support the infrastructure, how they should operate together, and ultimately what applications they should run.

GitOps

While Terraform can be used to declaratively manage almost anything, these days it is common to pick the best tool for each individual job and combine them with automation. This might mean using Terraform to declare Kubernetes infrastructure resources, then applying Kubernetes objects with kubectl. While we may use different tools to manage different parts of our systems, we still need a common way to automate them together, which brings us to GitOps.

The purpose of GitOps is to standardise a git repository as the single source of truth for all infrastructure as code and configuration management, and then automate the application of different tools as part of a continuous process. When code is committed to a repo, it can be checked and tested as part of a continuous integration (CI) process. When changes have passed tests and potentially a manual approval, a continuous delivery (CD) service will then invoke the necessary tooling to make the changes. The living documentation of our system now lives in this git repo, which means that all its changes are tracked and can be put through this rigorous testing and approval cycle, improving collaboration, security, and reliability.

The GKE Enterprise approach to configuration management means that artifacts like configuration and security policies now live in a git repo and are automatically synchronised across multiple clusters. This gives you a simplified, centralised approach to managing the configuration and policies of different clusters in different environments without having to worry about configuration drift.

But it’s important to understand where these tools sit within an overall GitOps approach:

• The GKE Enterprise config management tools focus on configuration only (deploying shared configuration objects and policy controllers that we’ll learn about in this post)

• They do not automate the creation of clusters (Terraform is recommended for this instead)

• They are not recommended for the automation of workloads on clusters. Workloads are expected to be frequently changed and updated, so they lend themselves to different tooling and processes. I’ve got plans to write about CI/CD practices for workloads later in this series.

Now that we’ve clarified this position, let’s start learning about the individual tools themselves, starting with Config Sync.

Using GKE Config Sync

The purpose of the Config Sync tool is to deploy consistent configurations and policies across multiple clusters, directly from a git repository. This gives us the benefits of using git such as accountability, collaboration, and traceability for changes, but it also guards against configuration drift with options for self-healing and periodic re-syncs. Clusters are enrolled in Config Sync via fleet management, and it is recommended to enroll all clusters in a fleet to ensure a consistent approach across every cluster.

To set up Config Sync, we must first create a git repo containing the desired state of our cluster configurations. When we enable Config Sync, a Kubernetes operator is used to deploy a Config Sync controller to our clusters. The controller will pull the configuration from the git rep, apply it to the cluster, and then continue to watch for any changes so that it can reconcile its current state with the desired state declared in the repo.

By default, these checks happen every hour. If we optionally enable drift prevention, every single request to the Kubernetes API server to modify an object that is managed by the Config Sync controller will be intercepted. The controller will validate that the change will not conflict with the state defined in the repo before allowing the change. Even with drift prevention enabled, the controller will continue its regular re-syncs and self-healing activity.

Config repos

Config Sync supports two different approaches to laying out files in a git repo for configuration management. Hierarchical repositories use the filesystem-like structure of the repo itself to determine which namespaces and clusters to apply a configuration to. If you choose the hierarchical option, you must structure your repo this way. An example structure is shown below, where we have configurations that apply to all clusters, and some that are specific to namespaces owned by two different teams.

.
├── README.md
├── cluster
│   ├── clusterrole-ns-reader.yaml
│   └── clusterrolebinding-ns-reader.yaml
├── namespaces
│   ├── blue-team
│   │   ├── namespace.yaml
│   │   └── network-policy.yaml
│   ├── limits.yaml
│   └── red-team
│       ├── namespace.yaml
│       └── network-policy.yaml
└── system
    └── repo.yaml

The cluster/ directory contains configurations that will apply to entire clusters and are not namespaced. In this example, we create a ClusterRole and a ClusterRoleBinding. We could also optionally use a ClusterSelector object to limit the scope of these to specific clusters.

The namespaces/ directory contains configurations that will apply to specific namespaces – although again these could be on any cluster or a scoped subset of clusters. In this example, we define the namespace itself in the namespace.yaml file and then apply a specific network policy with the network-policy.yaml file. We could include any other additional namespace-specific configuration we want, in each namespace’s own sub-directory. Note the name of the sub-directory must match the name of the namespace object that is defined.

Also in this example, the namespaces/ directory also contains a standalone file, limits.yaml, that will be applied to all namespaces. Because this repo is hierarchical, we can take advantage of this sort of inheritance.

Finally, the system/ directory contains the configuration for the Config Sync operator itself. A valid structured repository must contain the cluster, namespaces and system directories.

The alternative to a hierarchical repo is unstructured, and these are recommended for most users, giving you the most flexibility. In an unstructured repo, you are free to organise your files as you see fit. When you configure Config Sync, you tell it where to look in your repo, which means your repo could theoretically contain other data such as Helm charts. Which configurations are applied where can be determined by the use of ClusterSelector objects, NamespaceSelector objects, or just namespace annotations in metadata.

Enabling Config Sync on a cluster

Let’s walk through a quick demonstration of adding Config Sync to a GKE cluster. We’ll assume that we have already built a GKE Enterprise autopilot cluster running in Google Cloud, and that the cluster is registered to your project’s fleet (take a look at previous posts in this series if you need help getting set up - and always be mindful of the costs of playing with these features!)

We’ll set up the Config Sync components and get them to apply the demo configurations from Google’s quick start repo, which you can find here: https://github.com/GoogleCloudPlatform/anthos-config-management-samples/tree/main/config-sync-quickstart

If we take a quick look through the multirepo/ directory of this repo, we’ll find a further namespaces/gamestore directory containing two Kubernetes objects to deploy to the gamestore namespace. There is also a root/ directory containing objects that will be deployed to the entire cluster, including cluster roles, role bindings and custom resource definitions. Take a look at the YAML code and familiarize yourself with these objects.

In the GKE Enterprise section of the Google Cloud console, you should be able to find the Config Sync section in the left-hand menu, under Features.

On this page, we can select Install Config Sync which will bring up an installation dialog box. Here we can choose which clusters to target for the installation, and which version of Config Sync to use. The recommended course of action is to install Config Sync across all clusters in the fleet. For this demonstration, we’ll just select the cluster we have available and the latest version.

While the dashboard won’t update straight away, in the background GKE is now prepared to initialize the installation of Config Sync. After a few minutes, the Config Sync status should show that you have one cluster in a pending state, and after a few more minutes this state should change to Enabled. At this point, Config Sync has installed its operator on your cluster, but it still doesn’t have any actual configuration to synchronise.

Config Sync requires a source of truth that it can use to obtain configurations and other objects that it should deploy and manage to your clusters. The most common approach to creating a source of truth is to use a git repository as we have already discussed, but it’s also possible to use OCI container repositories for synchronising container images to your clusters.

Config Sync refers to these sources as Packages, so to set one up we will go to the Packages section of the Config Sync page and select Deploy Package. We will choose the git repository option and select our cluster. Now we need to enter a few details about the package:

• Package name: This will be the name of the configuration object created to match this package. We’ll use sample-repository for now, as we’re using Google’s sample repo.

• Repository URL: The git repo URL we mentioned earlier, which is: https://github.com/GoogleCloudPlatform/anthos-config-management-samples

• Path: We need to point to the correct part of our repo that contains the objects we wish to synchronise, which is config-sync-quickstart/multirepo/root

We can now click Deploy Package and let Config Sync do its work. Specifically, Config Sync will now look at the objects declared in the repo and make sure that they exist on our cluster. In its current state, these objects have never been created, so Config Sync will reconcile that by creating them. The overall package is defined as a RootSync object, one of the Config Sync custom resource definitions, which holds all of the setup information you just entered. This object gets created in the config-management-system namespace.

You’ll see from the dashboard that your cluster has entered a reconciling synchronisation status. Once everything has been created, and the cluster state matches your declared state in the repo, the status should change to synced. The job of Config Sync then becomes to maintain these objects, so that the actual state of the cluster never drifts from what is declared in the repo.

You may have noticed at this point that two packages have synchronised when we only created one. How did that happen? The root part of the package that we synchronised contained 28 resources, and one of these was a RepoSync object which contained a reference to the multirepo/namespaces/gamestore path in our repo, creating the gamestore namespaced resources. This object is represented as an additional package, even though technically the source of truth is the same git repo. We could optionally have used a RepoSync object to point to a completely different repo.

Examining synchronised objects

There are two useful ways we can check on the work that Config Sync has carried out. Assuming we have configured authentication for kubectl, we can query for objects that contain the app.kubernetes.io/managed-by=configmanagement.gke.io label annotation. This will tell us which objects are managed by Config Sync. For example, we can list all namespaces that contain this label:

kubectl get ns -l app.kubernetes.io/managed-by=configmanagement.gke.io

You should see the gamestore and monitoring namespaces. Google also provides a command line tool called nomos as part of the Google Cloud CLI. We can use the nomos status command to view the state of synchronisation. For more information about this command, see https://cloud.google.com/kubernetes-engine/enterprise/config-sync/docs/how-to/nomos-command

Let’s recap what we’ve just achieved and why it would be useful in a real environment. We’ve set up Config Sync for our cluster, but as we previously stated, it would be normal to enable Config Sync across an entire fleet of clusters. We’ve then deployed a Config Sync package, creating a code-based deployment of all the configuration we want across our clusters.

In this example repository, objects are defined for role-based access control, custom resource definitions, service accounts, namespaces, and monitoring services. These are the sort of supporting services that are normally configured as part of “Day 2” operations, but using Config Sync we’ve entirely automated them. Using git as our source of truth, we can now maintain a single code base that contains all the structural support our cluster needs, and as our fleet grows, Config Sync will ensure that these objects remain deployed and in their desired state across all our clusters no matter where they are running. We’ve seen how Config Sync can make sure that certain desirable objects are deployed to every cluster in your fleet. What other workloads and objects are then deployed on top of this baseline will be up to your developers and other teams who have access to your clusters. So how do you control what gets deployed?

Using the GKE Policy Controller

The Policy Controller is another tool that falls under the banner of configuration management for GKE Enterprise, and its job is to apply programmable policies that can act as guardrails. This allows you to set your own best practices for security and compliance and ensure that those policies are being applied uniformly across your entire fleet.

Under the hood, Policy Controller is built on the open-source Open Policy Agent (OPA) Gatekeeper project, which you can read more about at https://open-policy-agent.github.io/gatekeeper/website/docs/. The Policy Controller in GKE Enterprise extends the OPA Gatekeeper project by also providing an observability dashboard, audit logging, custom validation, and a library of pre-built policies for best practice security and compliance. In this way, you can view the GKE Policy Controller as a managed and opinionated version of the OPA Gatekeeper service. If you prefer, you can simply install and configure OPA Gatekeeper on its own. This will give you greater flexibility, but of course it will require more work on your part. If you wish to do this, you could consider automating its installation via Config Sync.

Constraints

Policy Controller works at a low level by leveraging the validating admission controller provided by the OPA Gatekeeper project. This means that all requests to the Kubernetes API on a given cluster are intercepted by this admission controller and validated against predefined policies.

These policies are made of constraint objects. Each constraint can either actively block non-compliant API requests, or it can audit a request and report a violation. By default, constraints will audit or mutate requests (more on that later), and report violations which you can view in the Policy dashboard.

GKE provides dozens of constraint templates in a library which you can use to build policies that suit your own security and compliance requirements. To give you a feel for the type of constraints these templates can implement, here are just a few examples:

• The AsmSidecarInjection template ensures that the Istio proxy sidecar is always injected into workload Pods.

• The K8sAllowedRepos template will only allow container images to be used where the source URL starts with an approved string.

• K8sContainerLimits and K8sContainerRequests will only allow containers to be deployed that have resource limits and requests set, and only if those limits are requests are within a specified threshold.

• The K8sRequiredProbes template requires Pods to have readiness and/or liveness probes defined, or they cannot be scheduled.

These are just a small sample from the available list, but they should give you an idea of the sort of compliance you can enforce on your cluster. The full list is available at: https://cloud.google.com/kubernetes-engine/enterprise/policy-controller/docs/latest/reference/constraint-template-library

If you have a requirement to constrain an object that is not covered by one of these templates, you can create your own custom constraint template. This should be an outlier case however, as templates to cover most common situations are provided in the Policy Controller library, and writing custom templates requires an advanced knowledge of the OPA Constraint Framework and the Rego language that it uses.

Policy Bundles

While you can easily apply constraint templates individually, GKE Enterprise also gives you the option of applying pre-prepared policy bundles. These are sets of policies prepared by Google Cloud to apply best practices or to meet specific compliance or regulatory requirements.

For example, the Center for Internet Security (CIS) maintains a benchmark for hardening and securing Kubernetes. This benchmark contains dozens of requirements to ensure the security of the control plane, workers nodes, policies and the implementation of GKE by managed services. A CIS GKE Benchmark Policy Bundle is maintained by Google Cloud and made available in GKE Enterprise to allow you to apply these requirements across clusters in your fleet.

Alternatively, other bundles are available that are based on requirements from the (NIST), PCI-DSS requirements the National Security Agency National Institute of Standards and Technology from the PCI Security Standards Council, and even a set of hardening requirements from (NSA).

Applying Policy Control to a Cluster

Let’s walk through a simple demonstration of setting up the Policy Controller for a GKE cluster and then applying some constraints from the template library. Just like before, we’ll assume that we have already built a GKE Enterprise autopilot cluster running in Google Cloud, and that the cluster is registered to your project’s fleet.

To install the Policy controller, go to the Policy page in the GKE section of the Google Cloud console. This page provides a dashboard for compliance with industry standards and best practices. With no clusters yet configured, you’ll just see some greyed-out example insights.

Select Configure Policy Controller. You will now see an overview of the Policy Controller feature for your fleet. This page will also inform you that some essential policy settings will now be applied to all clusters in your fleet.

You can choose to view or customise these settings by scrolling down and selecting Customize Fleet Settings, as shown below. The Policy Bundle pop-up will give you a choice of policy bundles to apply, but by default, the template library is enabled and so is the latest version of the Policy Essentials bundle. This bundle of best practices enforces policies for role-based access control, service accounts, and pod security policies.

You can cancel this customisation pop-up and select Configure at the bottom of the page. You will then have to confirm that you are happy to apply your new Policy settings to your fleet.

It will take several minutes for GKE to install all necessary components to enable the Policy Controller. This includes the gatekeeper-controller-manager deployment in a dedicated namespace which performs the key actions of the OPA Gatekeeper admission controller, such as managing the webhook that intercepts requests to the Kubernetes API. The controller also evaluates policies and enforcement decisions.

Our Policy dashboard should now show that our cluster is compliant with the Policy Essentials policy bundle that we enabled across our fleet, as shown below:

Next, let’s test how the Policy Controller audits and reports on best practices. The Policy Essentials bundle that we already applied is surprisingly permissive, so let’s add a new policy bundle to our fleet that will flag up some more risky behaviour:

• From the Policy page, select the Settings tab.

• Select Edit Fleet Settings.

• Scroll down and select Customize Fleet Settings.

• Look for NSA CISA Kubernetes hardening guide and select Enable.

• Click Save Changes.

• At the bottom of the page, click Configure, then Confirm.

It can take a few minutes for new fleet settings to be synchronised with your cluster, but if you don’t want to wait for you can click the Sync with Fleet Settings button to start the process now. After a few minutes, the additional policy bundle will be synchronised with your cluster, and your dashboard should show that you are now compliant with the NSA Kubernetes standards.

Just like we have done many times now, let’s create a deployment for a Hello World application:

kubectl create deployment hello-server --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0

You’ll notice that even though the deployment is created successfully, the admission controller has mutated our request. Admission controllers, in addition to validating and potentially declining a request, can also modify them to suit a set of policies. In this case, we did not specify CPU and memory resources for our container, but we have a policy that states that they are required. So, in this case the admission controller modified the request for us and added them.

Next, let’s create a Cluster IP service and an Ingress to map to it. Don’t worry that we don’t actually have an Ingress Controller to use right now, as we’re not really trying to set up a proper service. We just want to see what the Policy Controller will do.

kubectl expose deployment hello-server --type ClusterIP --port 80 --target-port 8080
kubectl create ingress hello-ingress --rule="example.com/=hello-server:80"

If you investigated the details of the policy bundle before we applied it, you may be surprised that these commands returned successfully. The NSA policy bundle specifically contains a constraint called nsa-cisa-k8s-v1.2-block-all-ingress, so shouldn’t that previous command have failed?

As we mentioned earlier, policies are applied, by default, in a dry-run mode. If we go back to our Policy dashboard, we can see that the policies are active because we are now showing multiple violations, as shown in the screenshot below. Google’s recommended best practice is to review all policy violations in the dashboard before taking any automated corrective action. This makes sense from the point of view of changing security requirements in existing production environments; you don’t want your security team to make an arbitrary decision, update the fleet policy, and suddenly disable a bunch of production workloads!

If you really want to, it is possible to change the enforcement mode of specific policies and policy bundles by patching the constraint objects directly in Kubernetes. Just look for the enforcementAction parameter and change it from dryrun to warn or deny. However, this approach is not recommended.

We’ve now seen how the Policy Controller can help us monitor the compliance of the clusters in our fleet to a set of security and best practices principles. Automating and visualizing this cluster intelligence makes the job of maintaining our fleet much easier. We can work with developers and teams to ensure that any violating workloads are fixed and rest easy knowing that our infrastructure is meeting some rigorous standards. These tools leverage the power of Kubernetes’ declarative approach to configuration and make the task of setting up clusters for developers easy and repeatable. Wouldn’t it be great if we could apply the same tooling to managing other resources in Google Cloud?

Leveraging the Config Controller

The final tool that falls under the umbrella of GKE Config Management is the Config Controller. This service extends the declarative GitOps approach of GKE Enterprise and Config Sync to create and manage objects that are Google Cloud resources outside of your GKE clusters, such as storage buckets, Pub/Sub topics, or any other Google Cloud resource you need. Using Config Controller allows you to continue using your familiar Kubernetes-based approach to declaring your resources and gives you a consistent approach if you are already using Config Sync and GitOps processes. You can combine these tools to apply policy guardrails, auditing, and automatic drift detection and repair for all your deployments, inside of Kubernetes or anywhere else in Google Cloud.

Choosing the right tools

Before we jump in to look at an example of the Config Controller in action, let’s just take a step back and consider this tool in the context of other infrastructure as code tools. It’s fair to say that Terraform is the industry’s leading platform-agnostic tool for infrastructure as code. Most organisations with at least a moderate number of developers and a good understanding of DevOps will be using Terraform to deploy infrastructure, including virtualised infrastructure on cloud platforms. For Kubernetes users, this of course will include Kubernetes clusters, either configured manually on virtual machines or by using managed services like GKE, but again, declared and maintained via Terraform.

Once clusters are built, we move on to the right tooling to deploy workloads to those clusters. In this post, we’ve looked at some of the tools that are unique to GKE Enterprise, and later in the series I’m planning to look at other Google Cloud recommended CI/CD best practices for deploying and maintaining workloads. Many teams will continue to use Terraform to manage Kubernetes objects, but many may adopt a templating approach such as using Helm or Kustomize.

Given that these are industry norms (if not standards), how would you choose to deploy additional cloud resources external to Kubernetes at this point? Again, most teams use Terraform if this is their primary tool for declaring the state of their infrastructure. However, it might be that you don’t want your teams to have to context-switch between the Terraform configuration language and Kubernetes APIs. You may have additional tools in the mix, and any increase in complexity normally means a reduction in developer velocity or an increased chance of making mistakes.

So, in answer to this, Google Cloud created an open-source project called the GCP Config Connector, which declares a set of custom resource definitions for Google Cloud. Using this add-on for Kubernetes, anyone can now declare resources in Google Cloud using Kubernetes objects.

Now if you don’t want to create or maintain a Kubernetes cluster just to use the Config Connector, you can instead choose to use the Config Controller, which is a fully managed and hosted service for the Config Connector. For the rest of this post, we’ll continue to explore the Config Controller as a feature of GKE Enterprise, but bear in mind that similar outcomes can be achieved simply by adding the open-source Config Connector to an existing Kubernetes cluster. The names are very similar and can be confusing, but hopefully this has clarified things for you!

Setting up the Config Controller

So for our final walkthrough/demo in this post, let’s create a Config Controller instance and use it to deploy a non-Kubernetes Google Cloud resource!

A Config Controller instance is a fully managed service, so we don’t need to configure or create the infrastructure required to run it, we just request the service with this command:

gcloud anthos config controller create cc-example --location=us-central1 --full-management

This command can take up to 15 minutes to complete because in the background a new GKE cluster is being provisioned just to run the Config Controller components for you. When the command completes it will also generate a kubeconfig entry for you, just like when you create a GKE cluster from the command line. This will allow you to use kubectl to create your Google Cloud resources.

Wait a minute, why are we running an Anthos command? That name got deprecated surely! Well at the time of writing, the commands we use for Config Controller still reference Anthos. While the product name is now officially deprecated, it can take a long time to rewrite command line tools to reflect this. If the commands don’t work in this section when you try them, please refer to the documentation here: https://cloud.google.com/sdk/gcloud/reference/anthos/config/controller

Back to our demo, and when the instance has been created, it will show up as a GKE cluster in the cloud console cluster list. You can also confirm its state from the command line:

gcloud anthos config controller list --location=us-central1

Before we can ask our new Config Controller to create resources for us, we still need to give it the necessary IAM permissions in our project. As part of setting up the Config Controller instance, a service account for Config Controller has also been created. We will grant this service account the Project Owner IAM role so that it can have full control over project resources for us. Note that this is a rather “broad brush” approach. If you only want to use Config Controller for a subset of your project resources, you may want to consider a different IAM role.

First we’ll get the value of the service account email address and store it in an environment variable that we’ll use in a moment:

export SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"

Now we’ll use that service account email to add an IAM policy binding. In the following command, replace my-project-id with the ID of your own Google Cloud project:

gcloud projects add-iam-policy-binding my-project-id \
  --member "serviceAccount:${SA_EMAIL}" \
  --role "roles/owner" \
  --project my-project-id

Our Config Controller instance is now authorised to manage resources in our project, and we can go ahead and start using it! Google Cloud resources are created by declaring them as Kubernetes objects, in much the same way as if we were creating Kubernetes resources. We can do this simply by writing the necessary YAML files, or of course using Config Sync or other GitOps-based options.

As a simple test, let’s say we want to enable the Pub/Sub API in our project and create a Pub/Sub topic. If you’re not familiar with Pub/Sub, it’s a serverless distributed messaging service, often used in big data pipelines. We’re not going to do anything with Pub/Sub right now, however, other than to demonstrate how we can set it up with the Config Controller.

So first, we’ll create a YAML declaration that states that the Pub/Sub API should be enabled. We’ll create a file called enable-pubsub.yaml and copy in the code below (make sure to once again replace any occurrences of my-project-id with your own Google Cloud project ID if you’re following along):

apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
kind: Service
metadata:
  name: pubsub.googleapis.com
  namespace: config-control
spec:
  projectRef:
    external: projects/my-project-id

You can now apply the YAML declaration with kubectl apply, just as you would with any other Kubernetes object. This creates an object of the type serviceusage.cnrm.cloud.google.com in the config-control namespace on the Config Connector instance. This object represents the resource configuration you have requested in your project; namely, that the Pub/Sub API should be enabled. If you query the object, you should see that the object’s state represents whether the desired resource has been created, as shown below:

Next, let’s create a Pub/Sub topic by writing a new YAML file called pubsub- topic.yaml. Don’t forget to substitute your own project ID again:

apiVersion: pubsub.cnrm.cloud.google.com/v1beta1
kind: PubSubTopic
metadata:
  annotations:
    cnrm.cloud.google.com/project-id: my-project-id
  name: example-topic
  namespace: config-control

Once again you can query the configuration object with kubectl. To confirm the resource has been created, we can also use gcloud to list all Pub/Sub topics:

gcloud pubsub topics list

You’ll notice that your topic contains a label identifying that it is managed by the Config Controller.

Combining Policy Controller and Config Controller

The Config Controller instance also provides us with a built-in Policy Controller, although at the time of writing only a handful of non-Kubernetes, general Google Cloud resource constraints were available. One example is the GCPStorageLocationConstraintV1 constraint, which can be used to restrict the allowed locations for Cloud Storage buckets. Outside of the supported templates, other non-Kubernetes constraints could theoretically be created by applying constraint logic to the configuration objects used by the Config Controller, but this isn’t a solution that’s officially supported by Google.

We can demonstrate a constraint that would restrict all Cloud Storage buckets to being created in the us-central1 region by creating a bucket-constraint.yaml file like this:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: GCPStorageLocationConstraintV1
metadata:
  name: storage-only-in-us-central1
spec:
  match:
    kinds:
    - apiGroups:
    - storage.cnrm.cloud.google.com
      kinds:
      - StorageBucket
  parameters:
    locations:
    - us-central1

Once we apply this object to the cluster, we can try to create a definition for a bucket in a location that should not be allowed:

apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: my-bucket
  namespace: config-control
spec:
  location: asia-southeast1

When we try to apply this configuration, we should hopefully receive an error from the API server. The Gatekeeper admission controller has denied the request, because the location was disallowed!

To fully leverage the Config Controller, you may also wish to combine it with Config Sync, allowing you to benefit from all the features of Config Sync we discussed earlier in this post. Conceptually, in addition to applying Config Sync to clusters in our fleet, we also apply it to the Config Controller instance, allowing us to use centralised repositories for configuration and policy for Kubernetes objects and Google Cloud resources.

To set this up, all we need to do is manually create a RootSync object and apply it to the Config Controller instance. A sample YAML declaration might look like this:

apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: root-sync
  namespace: config-management-system
spec:
  sourceFormat: unstructured
  git:
    repo: https://github.com/mygithub/myrepo
    branch: main
    dir: config-dir
    auth: none

It’s worth keeping an eye on how Google develops the constraints available for other cloud resources. If there is broad enough support, it could help reduce the overall number of tools that you end up using to secure your environments. Fewer tools means fewer things to go wrong!

Summary

In this post, I’ve introduced the three main tools used by GKE Enterprise for configuration management: Config Sync, Policy Controller and Config Controller. I started by making the case for declarative configuration management and learned how this could be applied with the modern approach of GitOps. Each of these tools performs a specific function, and I would recommend that you consider which of them is relevant to you and your organisation, and in what combination.

You don’t have to use all of them, or any of them if they don’t solve a relevant problem for you. Of the three tools, Config Controller is probably the least well-known because of the prevalence of Terraform for managing infrastructure, and Helm for managing Kubernetes-based workloads. As always, you should experiment to find the right fit for your team based on your own skillsets and other tools you may be using. Whichever tools you choose, declarative infrastructure as code, a single source of truth, and the automation of drift-reconciliation are recommended best practices.

So what’s next in this series? Exploring some of the fundamentals of GKE Enterprise has meant discussing some of the most niche topics and products I’ve ever dabbled in! But next time I’ll get into some more practical uses cases, looking at design patterns for multiple clusters and concepts like north-south and east-west routing. After that, we’ll be ready to embark on an epic Service Mesh quest!

In my last post of this series I looked at the most common use cases for deploying GKE - running it in the cloud (probably Google Cloud most of the time). Going all in with a single cloud provider was popular in the early boom days of the cloud, but it's now quite common for large enterprises to diversify their IT infrastructure. You may have a requirement to use multiple clouds or, as I’ll cover in this post, your own dedicated physical infrastructure. Use cases for keeping application deployments on-premises or your own managed infrastructure vary but can often be tied to regulatory compliance or the need to guarantee data locality. By using GKE Enterprise, you can keep the same approach to modernisation, the same processes and tools for deployment, and the same level of management and visibility, but have your choice of deployment targets. In this post, I’ll cover how GKE Enterprise enables this by deploying to bare metal, and other customer-managed infrastructure services that may be virtualised (GKE still considers them bare metal if there isn’t a specific cloud solution for them).

I’m going to walk through a demonstration of some principles of bare metal GKE deployments, but I’m going to simulate a physical datacenter using virtual machines in Google’s Compute Engine. There are simply too many different configurations for physical infrastructure for me to try to cover accurately in a blog post, but the concepts should translate to whatever you need to build in your own environments. As I pointed out in previous posts, if you want to try any of this out for yourself, please be mindful of the costs!

Building GKE on Bare Metal Environments

GKE Enterprise refers to bare metal as anything running outside of Google Cloud or other supported cloud vendors, but it’s probably not the best way to describe these non-cloud environments, and indeed Google’s own terminology may be updated in due course to reflect this.

Running GKE Enterprise in this way relies on a solution called Google Distributed Cloud Virtual (GDCV), which is part of the broader Google Distributed Cloud (GDC) service that extends Google Cloud services into alternative data centres or edge appliances. There are even options for GDC that allow you to run a completely isolated and “air-gapped” instance of Google Cloud services on your own hardware! For the purposes of running GKE on bare metal though, we’ll use GDCV to provide connectivity between Google Cloud and other locations such as on-premises data centres.

While many organisations will adopt a cloud-only approach, even if that involves multiple cloud vendors, some organisations may not be ready to fully move to the cloud, or they may have compelling reasons to not do so. This could be due to long-term infrastructure investments in their own data centres, or requirements to work with other types of physical infrastructure. For example, consider industries that work with factories, stadiums, or even oil rigs. Specialised physical infrastructure that is simply not available in the cloud such as bespoke appliances or even mainframes may be in use by some enterprises. An interesting outlier is gaming and gambling companies that are required to generate random numbers in a location that is mandated by law.

Hybrid use-cases

Hybrid computing opens up a world of possibilities for organisations with existing on-premises application deployments, or investments in on-premises infrastructure. Some of these use-cases might include:

• Developing and testing applications in a cost-controlled environment on-premises before deploying to a production environment in the cloud

• Building front-end and new services in the cloud while keeping enterprise software such as ERP systems on-premises

• Storing or processing data in a specific location or under specific controls due to working in a regulated industry

• Deploying services to customers in specific areas where no cloud resources are available

Even if you’re deploying to different environments, keeping the process of deploying and managing applications the same will provide developer efficiency, centralised control, and better security. Thankfully, GKE clusters that run in bare metal environments still have most of the capabilities of cloud-based clusters, including integrating with GKE’s management and operations tools, configuration management, and service mesh.

Running GKE on your own servers

Using your own servers – physical or virtual – gives you complete control over the design of your infrastructure, from how your nodes are built to your networking and security configuration. GKE on Bare Metal installs GKE clusters on your own Linux servers and registers them as managed clusters to your GKE Enterprise fleet. And remember, GKE doesn’t care if these really are bare metal, or Linux servers running in a different virtualisation environment.

So, how does this work? With bare metal, there are a few more moving parts than with cloud-based GKE clusters:

• User clusters: A Kubernetes cluster where workloads are deployed is now called a user cluster. A user cluster consists of at least one control plane node and one worker node. You can have multiple user clusters, which can operate on physical or virtual machines running the Linux operating system.

• Admin cluster: For GKE Bare Metal we also use something called an admin cluster. The admin cluster is a separate Kubernetes control plane that is responsible for the lifecycle of one or more user clusters, and it can create, upgrade, update or delete user clusters by controlling the software that runs on those machines. The admin cluster can also operate on physical or virtual machines running Linux.

• Admin workstation: It is recommended to create a separate machine (virtual or physical) to act as an admin workstation, which will contain all the necessary tools and configuration files to manage the clusters. Just like with regular Kubernetes clusters, you’ll use kubectl as one of these tools to manage workloads on your admin and user clusters. In addition, you’ll use a GKE-specific tool called bmctl to create, update and administer your clusters.

These are the components that differentiate GKE on Bare Metal from its cloud-based counterparts, but there are many different configurations of clusters to choose from based on your requirements for high availability and resource isolation.

Choosing a deployment pattern

High availability (HA) is recommended for production environments, and it can be achieved by using at least three control plane nodes per cluster (including the admin and user clusters), which we can see illustrated below. This configuration is also recommended when you want to manage multiple clusters in the same datacenter from a single centralised place. Using multiple user clusters allows you to isolate workloads between different environments and teams while keeping a centralised control plane to manage all clusters.

An alternative to this configuration is a hybrid cluster deployment as shown below. In this pattern, admin clusters serve a dual purpose and may also host user workloads in addition to managing user clusters. This can help to reclaim unused capacity and resources from admin clusters that may otherwise be over-provisioned (particularly as in a physical world, as you may have less flexibility around machine size). However, there are risks to this approach, as the admin cluster holds sensitive data such as SSH and service account keys. If a user workload exploits a vulnerability to access other nodes in the cluster, this data could be compromised.

Finally, it is actually possible to run everything you need on a single cluster, essentially taking the hybrid admin cluster from above and removing the separate admin clusters. This can still be HA by having multiple nodes, although everything now runs within a single cluster context which can have the same security risks as the hybrid cluster. The advantage of the standalone cluster, as illustrated below, is that there are significantly fewer resource requirements, which can be helpful in constrained environments or other deployments at the edge.

Network requirements

The networking and connectivity requirements for GKE on Bare Metal can be quite complex and extensive, depending on how your local physical networks are configured, or how you design networks in your own datacenter. I can’t cover all the possible different ways to provide networking for GKE in this post - it would never end! But at a high level you need to consider 3 primary requirements:

• Layer 2 and layer 3 networking for cluster nodes

• Connectivity to your Google Cloud VPC

• How external users will access your workloads

As we know, GKE on Bare Metal runs on physical or virtual machines that you provide; in other words, these machines already exist and will not be provisioned by GKE. They will therefore already be connected to your own network. GKE on Bare Metal supports layer 2 or layer 3 connectivity between cluster nodes. Remember from the OSI model that layer 2 refers to the data link layer, with physical network cards communicating across a single network segment, usually via ethernet connections. Layer 3 is the network layer, which uses the Internet Protocol (IP) to traverse multiple network segments.

There are lots of pros and cons for both approaches, but your local network design will usually determine which option you use. At a high level, layer 2 is sometimes more suitable for smaller clusters with a focus on simplicity or a requirement to bundle the MetalLB load balancer (more on that in a moment!), while layer 3 is recommended for larger deployments where scalability and security are paramount.

Your Kubernetes nodes will each require their own CIDR block so that they can assign unique IP addresses to Pods, and the size of that block will affect the number of Pods that can be scheduled. The minimum size of the CIDR block is a /26 which allows for 64 IP addresses and 32 Pods per node. The maximum CIDR block size is a /23 which allows for 512 IP addresses and the maximum of 250 Pods per node. Again, your local network design will impact this configuration.

Your local nodes will require connectivity to Google Cloud so that they can communicate with the Connect Agent that will facilitate communication between the GKE service and your local Kubernetes API server. You can set up this connectivity either across the public Internet using HTTPS, using a VPN connection or a Dedicated Interconnect. Google has a helpful article on choosing a network connectivity product here: https://cloud.google.com/network-connectivity/docs/how-to/choose-product

Finally, you will need to decide how your external users will connect to your applications hosted on bare metal nodes, if indeed that is a requirement of your clusters. You have a few different options to achieve this, which once again may depend on your local datacenter configuration and your individual requirements.

GKE on Bare Metal can bundle two types of load balancers as part of its deployment. The popular MetalLB bare metal load balancer can be completely managed by GKE for you, providing your cluster nodes run on a layer 2 network. MetalLB load balancers will run on either the control plane nodes of your cluster, or a subset of your work nodes, and provide virtual IPs that are routable through your layer 2 subnet gateway. Services configured with the LoadBalancer type will then be provided with a virtual IP.

Alternatively, GKE on Bare Metal supports a bundled load balancer that uses Border Gateway Protocol (BGP) for nodes on layer 3 networks. This approach provides much more flexibility, but it can be complex to configure. Once again Google provides a complete guide here: https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/how-to/lb-bundled-bgp but I recommend seeking the help of some experienced network engineers to set this up.

Bundled load balancers are just one way to go, however! If your datacenter already runs a load balancing service, such as those provided by F5, Citrix or Cisco, these can also be leveraged. It’s even possible to use cloud load balancers with bare metal servers, however for this to work you must run the ingress service on a cloud-hosted cluster, which then redirects traffic to the on-premises cluster using a multi-cluster ingress. And yes, I’m going to write a post about ingress very soon!

Building some bare metal infrastructure

With the network design out of the way, it’s time to start building our clusters. As mentioned, we’ll be simulating a bare metal environment using Compute Engine just so we can demonstrate the capabilities of GKE on Bare Metal. The tasks we’ll carry out to set up GKE will be almost identical in a real on-premises environment, you’d just be building the servers for real, or using some other kind of infrastructure service. Just like in my last post, you can find the commands below collected into scripts in my Github repo at https://github.com/timhberry/gke-enterprise/tree/main/bare-metal-demo-scripts

For reference, the scripts are:

• bm-vpc.sh - Creates a simulated bare metal network using a Google Cloud VPC

• create-servers.sh - Creates and configures multiple virtual machines to act as our bare metal infrastructure

• admin-ws-setup.sh - Configures the admin workstation

• admin-cluster.sh - Configures the admin cluster (and must be run from the admin workstation)

• user-cluster.sh - Configures the user cluster (and must be run from the admin workstation)

One final caveat! I just want to reiterate, that below I’m showing you how to build a demo environment using virtual machines in Compute Engine just so we can understand the tooling and the necessary steps to do this on bare metal. So again you’ll need a Google Cloud project with billing enabled, plus the gcloud tool installed and configured. Please be mindful of the costs you’re about to run up, and don’t run GKE for Bare Metal on Compute Engine in production!

Okay, let’s start by creating a VPC for our simulated bare metal environment. We’ll call it baremetal and specify that we’re going to use custom, not auto-assigned subnets. These commands can all be found in the bm-vpc.sh script:

gcloud compute networks create baremetal \
  --subnet-mode=custom \
  --mtu=1460 \
  --bgp-routing-mode=regional

Next we’ll create a subnetwork in the us-central1 region with a CIDR block of 10.1.0.0/24:

gcloud compute networks subnets create us-central1-subnet \
  --range=10.1.0.0/24 \
  --stack-type=IPV4_ONLY \
  --network=baremetal \
  --region=us-central1

We’ll create some firewall rules now. First, a rule to allow incoming SSH connections from Google’s Identity Aware Proxy (IAP):

gcloud compute firewall-rules create iap \
  --direction=INGRESS \
  --priority=1000 \
  --network=baremetal \
  --action=ALLOW \
  --rules=tcp:22 \
  --source-ranges=35.235.240.0/20

And then a rule to enable VXLAN traffic for our nodes:

gcloud compute firewall-rules create vxlan \
  --direction=INGRESS \
  --priority=1000 \
  --network=baremetal \
  --action=ALLOW \
  --rules=udp:4789 \
  --source-tags=vxlan

What is this mysterious VXLAN we speak of? Well, even though Google Cloud uses a software-defined network (SDN), we can still simulate a layer 2 network for our bare metal demonstration by using Virtual Extensible LAN (VXLAN) which encapsulates ethernet frames on the layer 3 network. If we were deploying servers on a real layer 2 network, we would not need to perform this step.

A quick side note: In my tests, the VXLAN package and Google Cloud’s SDN did not always play nice. Sometimes these firewall rules were sufficient, sometimes they weren’t. If you get stuck, you can create an “allow all” rule for your pretend bare metal VPC. This is obviously not a secure solution, but this is just for demonstration purposes anyway!

Getting back to our demo, we need to create a service account that we’ll use later for our admin workstation:

PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create bm-owner
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member=serviceAccount:bm-owner@${PROJECT_ID}.iam.gserviceaccount.com \
  --role=roles/owner

Time to create the nodes for our clusters! In this demonstration, we’ll build an admin cluster with a control plane but no worker nodes, and a user cluster with a control plane and one worker node. We won’t go for high availability as this is just a demo, but you can theorise how we could add high availability to this environment by increasing the number of nodes at each level. How you ensure there are no single failure points between multiple nodes depends heavily on how your physical infrastructure is built. These commands can all be found in the create-servers.sh script.

First we’ll create a bash array to hold our server names, and an empty array that will store IP address for us:

declare -a VMs=("admin-workstation" "admin-control" "user-control" "user-worker")
declare -a IPs=()

Now we’ll run a for loop to actually create these servers. To keep things simply, each server is identical. Every time we create a VM, we grab its internal IP and add it to the array:

for vm in "${VMs[@]}"
 do
     gcloud compute instances create $vm \
         --image-family=ubuntu-2004-lts \
         --image-project=ubuntu-os-cloud \
         --zone=us-central1-a \
         --boot-disk-size 128G \
         --boot-disk-type pd-standard \
         --can-ip-forward \
         --network baremetal \
         --subnet us-central1-subnet \
         --scopes cloud-platform \
         --machine-type e2-standard-4 \
         --metadata=os-login=FALSE \
         --verbosity=error
     IP=$(gcloud compute instances describe $vm --zone us-central1-a \
         --format='get(networkInterfaces[0].networkIP)')
     IPs+=("$IP")
done

Now we need to add some network tags to each instance which will be used by firewall rules we will set up in a moment. Tags are also used to identify which cluster a node belongs to, whether it provides a control plane or a worker, and additionally whether it will double up as a load balancer (which the control planes do). All servers are tagged with the vxlan tag, as we’re about to set up the VXLAN functionality:

gcloud compute instances add-tags admin-control \
  --zone us-central1-a \
  --tags="cp,admin,lb,vxlan"
gcloud compute instances add-tags user-control \
  --zone us-central1-a \
  --tags="cp,user,lb,vxlan"
gcloud compute instances add-tags user-worker \
  --zone us-central1-a \
  --tags="worker,user,vxlan"

For VXLAN to work, we also need to disable the default Ubuntu firewall on each server:

for vm in "${VMs[@]}"
do
    echo "Disabling UFW on $vm"
    gcloud compute ssh root@$vm --zone us-central1-a --tunnel-through-iap  << EOF
        sudo ufw disable
EOF
done

That for loop looked really simple didn’t it? Great, because the next one is a lot more complicated 😩 We’ll need to loop through all the VMs and create a VXLAN configuration locally. Doing this assigns an IP address in the 10.200.0.x range for the encapsulated layer 2 network:

i=2
for vm in "${VMs[@]}"
do
    gcloud compute ssh root@$vm --zone us-central1-a --tunnel-through-iap << EOF
        # update package list on VM
        apt-get -qq update > /dev/null
        apt-get -qq install -y jq > /dev/null

        # print executed commands to terminal
        set -x

        # create new vxlan configuration
        ip link add vxlan0 type vxlan id 42 dev ens4 dstport 4789
        current_ip=\$(ip --json a show dev ens4 | jq '.[0].addr_info[0].local' -r)
        echo "VM IP address is: \$current_ip"
        for ip in ${IPs[@]}; do
            if [ "\$ip" != "\$current_ip" ]; then
                bridge fdb append to 00:00:00:00:00:00 dst \$ip dev vxlan0
            fi
        done
        ip addr add 10.200.0.$i/24 dev vxlan0
        ip link set up dev vxlan0
EOF
    i=$((i+1))
done

Once that’s done, we’ll loop through the VMs again and make sure the VXLAN IPs are working:

i=2
for vm in "${VMs[@]}";
do
    echo $vm;
    gcloud compute ssh root@$vm --zone us-central1-a --tunnel-through-iap --command="hostname -I"; 
    i=$((i+1));
done

The final part of setting up our simulated physical environment is to create the necessary firewall rules that enable traffic to our control planes and worker nodes, as well as inbound traffic to the load balancer nodes and traffic between clusters:

# Add firewall rule to allow traffic to the control plane
gcloud compute firewall-rules create bm-allow-cp \
    --network="baremetal" \
    --allow="UDP:6081,TCP:22,TCP:6444,TCP:2379-2380,TCP:10250-10252,TCP:4240" \
    --source-ranges="10.0.0.0/8" \
    --target-tags="cp"

# Add firewal rule to allow inbound traffic to worker nodes
gcloud compute firewall-rules create bm-allow-worker \
    --network="baremetal" \
    --allow="UDP:6081,TCP:22,TCP:10250,TCP:30000-32767,TCP:4240" \
    --source-ranges="10.0.0.0/8" \
    --target-tags="worker"

# Add firewall rule to allow inbound traffic to load balancer nodes
gcloud compute firewall-rules create bm-allow-lb \
    --network="baremetal" \
    --allow="UDP:6081,TCP:22,TCP:443,TCP:7946,UDP:7496,TCP:4240" \
    --source-ranges="10.0.0.0/8" \
    --target-tags="lb"

gcloud compute firewall-rules create allow-gfe-to-lb \
    --network="baremetal" \
    --allow="TCP:443" \
    --source-ranges="10.0.0.0/8,130.211.0.0/22,35.191.0.0/16" \
    --target-tags="lb"

# Add firewall rule to allow traffic between admin and user clusters
gcloud compute firewall-rules create bm-allow-multi \
    --network="baremetal" \
    --allow="TCP:22,TCP:443" \
    --source-tags="admin" \
    --target-tags="user"

We’ve now got a completely simulated physical environment, ready for us to set up.

Configuring the admin workstation

The purpose of the admin workstation is to hold the tools and configuration we need to set up and manage our admin and user clusters. We have already created the server itself, but now it’s time to set it up. These commands can be found in the admin-ws-setup.sh script, but if you’re following along, make sure you run all of the commands on the admin workstation itself once you have connected to it.

So first, let’s connect to the workstation using IAP:

eval `ssh-agent`
ssh-add ~/.ssh/google_compute_engine
gcloud compute ssh --ssh-flag="-A" root@admin-workstation \
  --zone us-central1-a \
  --tunnel-through-iap

Like I said above, everything else for this section should run on the workstation VM itself (not your local machine, or the Cloud Shell terminal for example). Make sure your prompt looks like this:

root@admin-workstation:~#

Okay, first we need to remove the preinstalled snap version of the gcloud SDK, and then install the latest version:

snap remove google-cloud-cli
curl https://sdk.cloud.google.com | bash
exec -l $SHELL

Now we’ll use gcloud to install kubectl:

gcloud components install kubectl

We download Google’s bmctl tool, which simplifies cluster management for bare metal servers, and we install it into /usr/local/sbin:

gsutil cp gs://anthos-baremetal-release/bmctl/1.16.0/linux-amd64/bmctl .
chmod a+x bmctl
mv bmctl /usr/local/sbin/
bmctl version

We also download Docker and install it. Docker will be used to run local containers as part of the installation process for admin and user clusters:

curl -fsSL https://get.docker.com -o get-docker.sh
sh get-docker.sh
docker version

We need to set up an SSH key so that the admin workstation will be able to connect without passwords to each of our nodes. Once again we’ll leverage a bash loops and arrays to copy our SSH public key to each server:

ssh-keygen -t rsa

# Create a bash array containing our server names
declare -a VMs=("admin-control" "user-control" "user-worker")

# Copy our SSH public key to enable password-less access
for vm in "${VMs[@]}"
do
    ssh-copy-id -o StrictHostKeyChecking=no -i ~/.ssh/id_rsa.pub root@$vm
done

Finally, we install kubectx. This tool allows us to quickly switch contexts between different Kubernetes clusters:

git clone https://github.com/ahmetb/kubectx /opt/kubectx
ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx
ln -s /opt/kubectx/kubens /usr/local/bin/kubens

That’s it! Our admin workstation is now ready to help us build the rest of our bare metal infrastructure.

Creating the admin cluster

We will now create the admin cluster. In our demo environment, our admin cluster will contain a single control plane node, but no worker nodes, and this single node will also double up as a load balancer. To create the cluster, we’ll need to set up some services and service accounts, then create a configuration file that our admin workstation will use to configure our admin cluster node and turn it into a functioning Kubernetes cluster (albeit a cluster of one!). These commands can be found in the admin-cluster.sh script.

First, we set some environment variables for our project, zone, SSH key and load balancer IP addresses:

export PROJECT_ID=$(gcloud config get-value project)
export ZONE=us-central1-a
export SSH_PRIVATE_KEY=/root/.ssh/id_rsa
export LB_CONTROLL_PLANE_NODE=10.200.0.3
export LB_CONTROLL_PLANE_VIP=10.200.0.98

Next, we create a key for the service account we created earlier, and make sure we’re using that to authenticate with Google Cloud APIs:

gcloud iam service-accounts keys create installer.json \
  --iam-account=bm-owner@$PROJECT_ID.iam.gserviceaccount.com
export GOOGLE_APPLICATION_CREDENTIALS=~/installer.json

Then we create a config file with the bmctl command. This creates a draft configuration file which we’ll edit in a moment, but it also enables all the APIs we need in our project. The config file is a simple YAML file that tells bmctl how to set up our admin cluster:

bmctl create config -c admin-cluster \
  --enable-apis \
  --create-service-accounts \
  --project-id=$PROJECT_ID

Take a look at the file that’s been created for yourself at ~/bmctl-workspace/admin-cluster/admin-cluster.yaml

Now we’ll use the power of sed to change a bunch of things in the draft configuration file. First, we’ll replace the SSH key placeholder with our actual SSH key:

sed -r -i "s|sshPrivateKeyPath: <path to SSH private key, used for node access>|sshPrivateKeyPath: $(echo $SSH_PRIVATE_KEY)|g" bmctl-workspace/admin-cluster/admin-cluster.yaml

Then we’ll change the node type to admin:

sed -r -i "s|type: hybrid|type: admin|g" bmctl-workspace/admin-cluster/admin-cluster.yaml

And finally update the IP addresses for the load balancer and control plane:

sed -r -i "s|- address: <Machine 1 IP>|- address: $(echo $LB_CONTROLL_PLANE_NODE)|g" bmctl-workspace/admin-cluster/admin-cluster.yaml
sed -r -i "s|controlPlaneVIP: 10.0.0.8|controlPlaneVIP: $(echo $LB_CONTROLL_PLANE_VIP)|g" bmctl-workspace/admin-cluster/admin-cluster.yaml

The draft configuration contains a complete NodePool section, but we don’t want that because in the design pattern we’ve chosen, our admin cluster doesn’t have any worker nodes. We can remove this section with the head command:

head -n -11 bmctl-workspace/admin-cluster/admin-cluster.yaml > temp_file && mv temp_file bmctl-workspace/admin-cluster/admin-cluster.yaml

Now we can actually ask bmctl to create the admin cluster (that is, to configure the existing VM):

bmctl create cluster -c admin-cluster

bmctl will use the config file we have edited, then connect to the control plane node and set it up. Bootstrapping, creating the cluster and performing post-flight checks can take up to 20 minutes, so now is a good time to grab a drink!

When the cluster is finally ready, we’ll export the configuration for the admin cluster to kubectx, and then run kubectl get nodes just to make sure we can see the control plane node:

export KUBECONFIG=$KUBECONFIG:~/bmctl-workspace/admin-cluster/admin-cluster-kubeconfig
kubectx admin=.
kubectl get nodes

Finally, we create a Kubernetes service account that we’ll use to connect to the cluster from the Cloud Console. The last line in the script prints the token out:

kubectl create serviceaccount -n kube-system admin-user
kubectl create clusterrolebinding admin-user-binding \
    --clusterrole cluster-admin --serviceaccount kube-system:admin-user
kubectl create token admin-user -n kube-system

In the Kubernetes clusters page of the GKE section in the Google Cloud Console, you should now see your “bare metal” admin cluster. Click Connect from the 3 buttons action menu, and select Token as the method of authentication. Then paste in the token that was provided. This cluster can now be managed completely by GKE!

Back on our workstation, we can run kubectl get pods --all-namespaces within the admin cluster context and list all of the different Pods it is running:

As you can see, the admin cluster is doing quite a bit of heavy lifting. In addition to the usual kube-system Pods, we can see Pods for Stackdriver (the original name of Google’s Operations Suite), and several operators for Anthos (the original name of GKE Enterprise). The Anthos Cluster Operator helps to provision and manage other Kubernetes clusters on physical servers, and the Anthos Multinet Controller is an admission controller for these clusters. You’ll also see the GKE Connect Agent running in its own gke-connect namespace, which facilitates communication between your bare metal clusters and your Google Cloud projects.

Creating a user cluster

Now the admin cluster is built, we go through a similar process for the user cluster, which is where our workloads will run. These steps can be found in the user-cluster.sh script, and once again they should be run from the admin workstation.

Just like before, we use bmctl to create a template configuration file:

bmctl create config -c user-cluster \
  --project-id=$PROJECT_ID

The default file references a credentials file we don’t use, so we’ll get rid of that section, and add the path to our private SSH key instead (once again using sed):

tail -n +11 bmctl-workspace/user-cluster/user-cluster.yaml > temp_file && mv temp_file bmctl-workspace/user-cluster/user-cluster.yaml
sed -i '1 i\sshPrivateKeyPath: /root/.ssh/id_rsa' bmctl-workspace/user-cluster/user-cluster.yaml

This is a user cluster, so we’ll change the cluster type:

sed -r -i "s|type: hybrid|type: user|g" bmctl-workspace/user-cluster/user-cluster.yaml

Now we need to set the IP addresses for the control plane node and the API server, a well as the Ingress and LoadBalancer services. Once again sed is our friend:

sed -r -i "s|- address: <Machine 1 IP>|- address: 10.200.0.4|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|controlPlaneVIP: 10.0.0.8|controlPlaneVIP: 10.200.0.99|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|# ingressVIP: 10.0.0.2|ingressVIP: 10.200.0.100|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|# addressPools:|addressPools:|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|# - name: pool1|- name: pool1|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|#   addresses:|  addresses:|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|#   - 10.0.0.1-10.0.0.4|  - 10.200.0.100-10.200.0.200|g" bmctl-workspace/user-cluster/user-cluster.yaml

We’ll also enable infrastructure and application logging for the cluster:

sed -r -i "s|# disableCloudAuditLogging: false|disableCloudAuditLogging: false|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|# enableApplication: false|enableApplication: true|g" bmctl-workspace/user-cluster/user-cluster.yaml

If you recall, we didn’t create a worker node for the admin cluster, we just removed the entire NodePool section. But our user cluster will have a node pool, albeit of just a single additional VM:

sed -r -i "s|name: node-pool-1|name: user-cluster-central-pool-1|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|- address: <Machine 2 IP>|- address: 10.200.0.5|g" bmctl-workspace/user-cluster/user-cluster.yaml
sed -r -i "s|- address: <Machine 3 IP>|# - address: <Machine 3 IP>|g" bmctl-workspace/user-cluster/user-cluster.yaml

Then we use bmctl create cluster to create the cluster using the updated configuration file. Note that we pass in our existing kubeconfig file so that we can reuse our certificate credentials. It should be slightly quicker for the user cluster to build, but it can still take 10-15 minutes:

bmctl create cluster -c user-cluster --kubeconfig bmctl-workspace/admin-cluster/admin-cluster-kubeconfig

When the cluster is up and running, we’ll create a new kubectx context for it, and use kubectl to make sure both nodes in the cluster are running:

export KUBECONFIG=~/bmctl-workspace/user-cluster/user-cluster-kubeconfig
kubectx user=.
kubectl get nodes

Just like before we will now create a Kubernetes service account that we’ll use in the Cloud Console to log into the cluster:

kubectl create serviceaccount -n kube-system admin-user
kubectl create clusterrolebinding admin-user-binding \
    --clusterrole cluster-admin --serviceaccount kube-system:admin-user

And finally, the script will output the token we need to log in via the Console:

kubectl create token admin-user -n kube-system

Back in the Cloud Console you should now see your user cluster. Once again you can click Connect from the 3 buttons action menu, select Token as the method of authentication and paste in the token that was provided. You should now see both of your bare metal clusters - but note that only the user cluster shows that it has any available resources for workloads, which makes sense because it’s the only cluster with worker nodes! (okay, 1 node!)

Deploying a test workload

We should now be able to create a test workload and expose it via a load balancer. We’ll use the same demo container as before to create a Hello World deployment. If you’ve been working through the scripts, you should still be logged into the admin workstation and kubectl should be authenticated against your user cluster.

We’ll use kubectl create to create a simple test deployment, followed by kubectl expose to create a service exposed by a load balancer:

kubectl create deployment hello-server --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
kubectl expose deployment hello-server --type LoadBalancer --port 80 --target-port 8080

Within a few moments your deployment and service should be up and running. You can get the external IP of the service the usual way:

kubectl get service hello-server

However, this external IP exists only on the overlay network we created with VXLAN. To route to it from the public Internet for example, we’d also have to simulate some sort of external load balancer, which is outside the scope of this post! This is as far as we can go with simulating bare metal, so let's review what we have built:

1. We first set up the admin workstation, which we used as a base to provide the tools and other configuration artifacts we needed.

2. Then we deployed the admin cluster. This is a special Kubernetes cluster that acts as a management hub for our environment. Even though we issued the bmctl command from our workstation, the admin cluster was instrumental in the creation of the user cluster. The admin cluster is also responsible for communication with your Google Cloud project.

3. Next, we created the user cluster. User clusters are more like the traditional Kubernetes clusters that we know and love, as they are the places that deployed workloads will run.

4. Just like in our previous demonstrations, we deployed a “Hello World” app to test our new environment.

Even though we've been simulating bare metal, this section has hopefully provided a good overview of the requirements for working with bare metal clusters in your own environments. As we’ve already discussed, there are many scenarios where this hybrid approach is appropriate, and the extra complexity involved should be justified by your use case.

Building GKE on VMWare

If you have existing investments in VMWare, such as a large VMWare estate or significant VMWare skills in your team, it can make sense to continue using these instead of or alongside cloud deployments. Just like with bare metal servers, GKE Enterprise can leverage VMWare to deploy fully managed GKE clusters. Your workloads can be deployed to GKE clusters on VMWare, bare metal or in the cloud with a single developer experience and a centralised approach to management.

GKE Enterprise works with VMWare in a very similar way to the bare metal approach we’ve already described and demonstrated, but with an added integration into the infrastructure automation provided by the VSphere software in VMWare. At a very high level this means that VSphere can automate some of the infrastructure tasks required, such as creating virtual machines, whereas in bare metal scenarios this is normally done manually.

So just like with bare metal, GKE on VMWare comprises the following components:

• User clusters to run your containerised workloads. User clusters have one or more control plane nodes and one or more worker nodes, depending on your requirement for high availability.

• An admin cluster to manage the user clusters. Again, this can be configured in a high availability pattern if your underlying infrastructure offers redundant points of failure.

• An admin workstation to provide the configuration and tooling for building clusters. Our bare metal clusters used the bmctl tool for configuration and cluster building, but we’ll now use the GKE on VMWare tool called gkectl. This tool uses credentials for VCenter to automate the creation of virtual machines for all clusters, so you don’t have to build them manually.

Let’s walk through a hypothetical build of a GKE environment on VMWare so we can examine some design considerations. Because the overall process is quite similar to GKE on Bare Metal, we won’t be detailing the full installation steps with scripts and commands, we’ll just focus on illustrating what’s different with VMWare. For a full installation guide, see: https://cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/overview

Preparing your VMWare environment

At a bare minimum, GKE on VMWare requires at least one physical host running the ESXi hypervisor with 8 CPUs, 80GB or RAM and around half a terabyte of storage. At the time of writing, GKE on VMWare supports ESXi version 7.0u2 or higher and Center Server 7.0u2 or higher. Your vSphere environment requires:

• A vSphere virtual datacenter – this is essentially a virtual environment that contains all other configuration objects

• A vSphere cluster – although a single node will work, it still needs to be configured as a cluster

• A vSphere datastore – this will store the virtual machine files used to provision admin and user clusters for GKE

• A vSphere network – essentially a virtual network. There are quite a few network requirements for VMWare, so we’ll get into them below.

Just like GKE on Bare Metal, your VMWare environment will require IP address assignments for all nodes, virtual IPs for control plane components, and dedicated CIDR ranges for Pods and Services. It’s important to plan your IP allocations carefully, and how you do this will vary based on your existing VMWare deployment and other network infrastructure. Planning can also include the use of dynamically assigned IP addresses for nodes using DHCP, as vSphere builds the virtual machines for the clusters for you and can request IP addresses as it does so.

Your admin workstation and your clusters are likely to use IP addresses on your vSphere network, and so are the virtual IPs for the API server and cluster ingress. When you choose CIDR ranges for Pods and Services, it's recommended to use private IP ranges as specified in RFC1918. Typically, you spin up more Pods than Services, so for each cluster it’s recommended to create a Pod CIDR range larger than the Service CIDR range. For example, a user cluster could use the 192.168.0.0/16 block for Pods, and the 10.96.0.0/20 block for Services.

Finally, your clusters will need access to DNS and NTP services, which you may already have running within your VMWare environment.

Creating the admin workstation

Once again, the admin workstation is the first thing we need before building any clusters. This time we don’t need to build a server (or create a VM) manually, because the tooling will do it for us.

First, we download the gkeadm tool. Instructions for downloading the latest version can be found here: https://cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/download-gkeadm

Then we’ll need to create a credentials.yaml file that contains our vCenter login information. Here’s an example:

apiVersion: v1
kind: CredentialFile
items:
- name: vCenter
username: myusername
password: mypassword

Finally, we create a configuration file for the admin workstation. This will contain the details of the vSphere objects we discussed earlier (datacenter, datastore, cluster and network), and the virtual machine specifications used to build the workstation. You can find an example configuration file here: https://cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/minimal-create-clusters#create_your_admin_workstation_configuration_file

Once you’ve populated the configuration file with the details of your own vSphere environment, you can run this command:

gkeadm create admin-workstation --auto-create-service-accounts

The gkeadm tool then accesses vCenter and builds the admin workstation according to the specified configuration. It also sets up any necessary Google Cloud service accounts and creates some template configuration files for admin and user clusters.

Preparing vSphere to build clusters

Once the admin workstation is built, you can log into it and see that it has created some template cluster configurations, as well as service-account key files that will be used for connecting your clusters back to Google Cloud. The admin-cluster.yaml and user-cluster.yaml files contain the details of your vSphere environment, the sizing for cluster nodes, load balancing information and paths to the service accounts that will be used for GKE Connect as well as logging and monitoring.

Once you have edited or reviewed the configuration files, you’ll use the gkectl prepare command, which will check through the config and import the required images to vSphere, marking them as VM templates. Then you can run gkectl create for each cluster, and all of the required virtual machines and other resources will be built for you in your vSphere environment, thanks to gkectl communicating with your vCenter server. Your clusters are automatically enrolled in your GKE Enterprise fleet thanks to the GKE On-Prem API, which means you can manage them in the Google Cloud console or with gcloud commands just like any other GKE cluster.

Summary

In this post, I’ve tried to demonstrate that GKE Enterprise can bring the benefits of a single developer experience and centralised management to Kubernetes clusters running in any environment, not just the cloud. At this point, your GKE fleet might contain clusters running on bare metal, VMWare, Google and AWS all at the same time, and all managed through a single interface! We also looked at some of the networking and other environmental design considerations required for bare metal and walked through a demonstration of a simulated bare metal environment, once again deploying our trusty “Hello World” application.

Thank you for reading this post! I know it’s a long one, but we’ve still barely scratched the surface of what GKE Enterprise can do. In the next post, we’ll get into some of the fun stuff - automation and configuration management.

In this post, I'll begin to introduce the fundamental components of GKE Enterprise and highlight what makes it different from running a standard GKE cluster. To ease us into the hybrid world of GKE Enterprise, we’ll start just by focusing on cloud-based deployments in this post and explore VMWare and bare metal options next time around. This should give us a solid foundation to work from before we explore any further complex topics.

The primary benefit of using GKE Enterprise is the ability it gives you to run Kubernetes clusters in any cloud while maintaining a single developer experience and a centralised approach to management. You may have an organisational need to deploy to multiple clouds, perhaps for additional redundancy and availability purposes, or maybe just so you’re not putting all your eggs in a single vendor’s basket. GKE Enterprise allows you to deploy to Google Cloud, AWS and even Azure with a simplified workflow, so you have complete freedom to choose the target for your workload deployments.

If you want to follow along with anything in this post, you'll need a Google Cloud account (and an AWS account if you follow that part as well). Google used to offer an introductory cloud credit of $300, but that seems to have been changed into an unfathomable number of Gemini token credits, which aren't so useful for building infrastructure. Still, there are free usage quotas for most products. Meanwhile, AWS has a free tier that's equally difficult to decipher. In any case, if you are trying to learn this stuff to help with your job, by all means ask your employer to pay for it. Things can get expensive quickly!

I'm going to repeat this one more time: 😬 Using any cloud vendor means you are liable to be charged for services – that's how they make their money after all. Even with free trials and special offers, you should expect to pay something, and that something can quickly turn into a big something if you forget to delete resources. Please be careful!

With that disclaimer out of the way, let's dive back into GKE Enterprise.

GKE Enterprise Components

Fundamentally, GKE Enterprise is a system for running Kubernetes anywhere but managing it from a single place. Many different components are combined to achieve this which operate at different layers, from infrastructure to the network layer to service and policy management. At the lowest level, Kubernetes clusters are used to manage the deployment of workloads, and these can take several forms. In Google Cloud, you obviously deploy GKE clusters. When you use GKE Enterprise to deploy to environments outside of Google Cloud, the service will create GKE-managed clusters which it will operate for you (these were formally referred to as "GKE On-Prem clusters" even if they ran in other clouds). Finally, you can even choose to bring the management of an existing vanilla Kubernetes cluster into GKE Enterprise.

At the network layer, GKE Enterprise will leverage various connectivity options to communicate with your other environments. You may choose to set up VPN connections for example, or configure a dedicated Interconnect if you expect high traffic between your Google Cloud projects and other locations. GKE Enterprise communicates with clusters in other environments using a component called the Connect Gateway. This gateway provides a communication bridge between a remote cluster’s API and the control plane of your GKE Enterprise environment. Using the gateway allows GKE Enterprise to issue commands and control additional clusters on other clouds or on-premises. We’ll see how this is used to set up clusters in AWS later in this post. Above this layer are tools to control our workloads, such as service mesh, configuration management, and policy control, which we’ll discuss much later on in this series.

As you can imagine, when operating and managing multiple clusters, you will need some additional help to organise all of your resources, which is why GKE Enterprise uses a concept called fleets. A fleet is simply a logical grouping of Kubernetes clusters, allowing you to manage all clusters in that fleet rather than dealing with them individually. A single fleet can contain clusters that all run within Google Cloud or a combination of clusters running across a variety of environments. Fleets offer many different design patterns for workload, environment, and tenant isolation and also provide other powerful features beyond simple logical cluster groupings. I'll cover these options in more detail later in the series, but for now, you just need to know that all clusters in GKE Enterprise must belong to a fleet.

Building your first Enterprise cluster

With enough of the basic concepts covered, it’s now time to create your first GKE Enterprise cluster. To do this, we’ll first enable GKE Enterprise and set up a fleet associated with our project. Then we’ll create a GKE cluster and register it to that fleet. This post is about deploying GKE in the cloud – not just in Google Cloud, so once our fleet is running, we’ll set up a similar cluster in AWS and observe how GKE’s fleet management allows us to centrally manage both clusters.

Enabling GKE Enterprise

As I mentioned in the first post, GKE Enterprise is an add-on subscription service. To enable it, you will need a Google Cloud project with a valid billing account, and while it does come with its own free trial, the switch from GKE Standard to GKE Enterprise is likely to take you out of any free tier or free credits you may have. Once GKE Enterprise is enabled, all registered GKE Enterprise clusters will incur per-vCPU charges. Please read the pricing guide for more information at: https://cloud.google.com/kubernetes-engine/pricing.

Have I scared you off yet? 😂 If you're still here, let's do this!

1. From the Google Cloud console page, select Kubernetes Engine in the navigation menu.

2. In the Overview section of Kubernetes Engine, you will see a button that says Learn About GKE Enterprise. Click this button.

3. A pop-over box will appear containing details of the benefits of GKE Enterprise. At the bottom of this box is a button labeled Enable GKE Enterprise. You can optionally tick the 90-day free trial option (if it is still on offer when you’re reading this) which should waive the per-vCPU charges for 90 days.

4. Click the Enable GKE Enterprise button. A confirmation pop-over will appear next. This shows you the APIs that will be enabled for your project and it will also tell you the name of the default fleet that will be created for GKE Enterprise. You can edit, create, and delete this and other fleets later.

5. Click Confirm to proceed with enabling GKE Enterprise.

After a few moments, you will see a confirmation message. It will also prompt you to create your first cluster, but we’ll do that separately in a moment. You can now close this pop-over box and return to the Kubernetes Engine section. You should see that you’re back on the overview page with a dashboard showing you the current state of your fleet.

Creating the cluster

Creating a cluster inside a GKE Enterprise fleet is very similar to how you create a GKE Standard edition cluster, although you’ll see a few extra options are now available to you.

1. From the Kubernetes Engine section, select Clusters in the left-hand menu.

2. Click the Create button at the top of the page. The cluster configuration dialog box that pops up contains options for creating standard or autopilot clusters on Google Cloud, as well as GKE clusters on other clouds and even on-premises. At the time of writing, not all of these were supported as options in the UI and some required using the command line instead. For now, we’ll stick with the recommended Autopilot cluster type on Google Cloud. Click Configure on this option.

3. We could click Create at this stage and accept all of the defaults, but let’s walk through the setup so we understand what we’re creating: On the first page, Cluster basics, give your cluster a name of your choice and choose a region. This could be a region closest to you, or if you prefer you can use the default region which may be cheaper. Then click Next: Fleet Registration.

4. On the next page, you can choose to register your cluster to your fleet. As you can see, there’s an option to skip this, but the whole point of GKE Enterprise is to manage clusters across different environments, so go ahead and check the box. You’ll see a pop-up notifying you that no additional fleet features have been configured so far, but we’ll come back to those in a later post. Now you can click Next: Networking.

5. Just like a GKE Standard cluster, we have several networking options that you are probably already familiar with, such as which network and subnet your GKE nodes should run in, and whether the cluster should run with public or private IP addresses. Let’s leave this at the defaults for now and click Next: Advanced Settings.

6. In the Advanced Settings section, we can choose the release channel, which again is the same principle we would use for a GKE Standard cluster. Let’s stick to the regular channel for now. Notice all the drop-down boxes for the different advanced features. Have a look through and see what GKE Enterprise can offer you, but for now, don’t select anything other than the defaults. We’ll explore a lot of these options later in the blog. Now click Next: Review and Create.

7. Finally, review the options you’ve chosen for your cluster, and click Create Cluster.

After 5-10 minutes, your GKE Autopilot cluster will be built, ready to use, and registered to your GKE Enterprise fleet.

Testing a Deployment

Now we'll use the built-in deployment tool in the cloud console to test a deployment to our new cluster. Of course, you’re welcome to use any other deployment method you may be familiar with (for example, command-line kubectl), because this is just a regular Kubernetes cluster behind the scenes, even if Google manages the nodes for us in autopilot mode.

1. From the Kubernetes Engine section, select Workloads in the left-hand menu.

2. Click the Deploy button at the top of the page. Enter the following container image path as a single line: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0

3. Click Continue then change the deployment name to hello-world. Then click Continue again (don’t deploy it just yet!)

4. In the optional Expose section, check the box to also create a matching service for the deployment. In the Port Mapping section that appears, make sure you change the Item 1: Port 1 port number to 8080 – this is the port the container listens on. Accept the defaults and click Deploy.

After a few minutes, your deployment should be up and running and your service should be exposed. If you see errors at first (about Pods not being able to be scheduled, or the deployment not meeting minimum availability), just wait a few more minutes and refresh the page. Autopilot is scaling up the necessary infrastructure for your workload behind the scenes.

When your workload shows a green tick, you can click the link at the bottom of the page under Endpoints to view your new application. In a moment, we’ll repeat these steps for a completely different cloud just to see how easy it is to perform multi-cloud deployments from GKE Enterprise.

Automating with Terraform

Just like everything else in Google Cloud, we can automate the creation of GKE Enterprise components (and Kubernetes components) too. Let's walk through a basic example of using Terraform to automate everything we’ve just built so far. I don't want to go off on a tangent into how to learn Terraform from scratch, so to follow along below I'll assume that you already have the Terraform command line tools set up and ready to go for your Google Cloud project. You’ll need a service account with the necessary project permissions and a JSON key file which we’ll refer to as serviceaccount.json.

You can also find the code below in my Github repo here: https://github.com/timhberry/gke-enterprise/tree/main/gke-cluster-tf

First, we’ll create a file called terraform.tfvars. This file will store the name of our project ID and the default Google Cloud region we want to use. Make sure you update these values to match your own project and preferred region:

project_id = "my-googlecloud-project"
region     = "us-east1"

Next, we’ll create a new VPC and subnetwork to use for our cluster. In the example we went through a moment ago we used the default VPC network that is configured by default in all Google Cloud projects. However, when using infrastructure as code, it is better to create and manage all resources rather than to rely on the assumption that they exist and are accessible.

Create the following vpc.tf file:

variable "project_id" {
  description = "project id"
}

variable "region" {
  description = "region"
}

provider "google" {
  credentials = file("serviceaccount.json")
  project     = var.project_id
  region      = var.region
}

resource "google_compute_network" "gke-vpc" {
  name                    = "gke-vpc"
  auto_create_subnetworks = "false"
}

resource "google_compute_subnetwork" "gke-subnet" {
  name          = "gke-subnet"
  region        = var.region
  network       = google_compute_network.gke-vpc.name
  ip_cidr_range = "10.10.0.0/24"
}

Let's walk through this code.

• First, we create two variables that we can use later: project_id and region. This allows us to read the variables from the terraform.tfvars file that we created first. This makes our code somewhat portable and reusable, so it's a good practice to follow.

• Next, we set up the Google provider. Here we pass in the serviceaccount.json credentials file that we mentioned earlier and configure our project ID and default region.

• Finally, we create two new resources, a google_compute_network (ie., a VPC), and a google_compute_subnetwork inside it. As you can see, we switch off the automatic creation of subnetworks, so this VPC will only contain the one subnetwork that we have chosen to create. This subnetwork gets set up in the region we chose, with an IP address range of 10.10.0.0/24.

Now we’ll define the cluster itself. Create the following cluster.tf file:

resource "google_gke_hub_fleet" "dev-fleet" {
  display_name = "My Dev Fleet"
}

resource "google_container_cluster" "gcp-cluster" {
  name             = "gcp-cluster"
  location         = var.region
  enable_autopilot = true
  network          = google_compute_network.gke-vpc.name
  subnetwork       = google_compute_subnetwork.gke-subnet.name
}

resource "google_gke_hub_membership" "membership" {
  membership_id = "basic"
  location      = var.region
  endpoint {
    gke_cluster {
      resource_link = "//container.googleapis.com/${google_container_cluster.gcp-cluster.id}"
    }
  }
}

data "google_client_config" "provider" {}

data "google_container_cluster" "gcp-cluster" {
  name     = "gcp-cluster"
  location = var.region
}

provider "kubernetes" {
  host  = "https://${data.google_container_cluster.gcp-cluster.endpoint}"
  token = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(
    data.google_container_cluster.gcp-cluster.master_auth[0].cluster_ca_certificate,
  )
}

There’s quite a lot to unpack here, so let’s go through the code block by block.

• The first resource we create is a google_gke_hub_fleet, which we call dev-fleet. This is our GKE Enterprise fleet, which as we’ve discussed can manage multiple clusters for us.

• Next, we create the google_container_cluster. We give the cluster a name, specify its location and network settings, and make sure we turn on Autopilot mode. There are of course all kinds of other options we could specify here for the configuration of our cluster, but to keep it simple we’re just specifying the required settings, and we’ll accept the defaults for everything else.

• Now we have a cluster, we can join it to the fleet. We do this with a google_gke_hub_membership resource. This simply joins the cluster we just created by using Terraform’s own resource link.

• Finally, we want to be able to use Terraform to actually configure some Kubernetes objects, not just the cluster and fleet themselves. So we need to configure a Kubernetes provider. We do this by configuring our new cluster as a data source and then referencing it to extract the cluster certificate in the configuration of our Kubernetes provider. Doing this means that Terraform can send authenticated requests directly to the Kubernetes API to create and manage objects for us.

At this point, we have declared our fleet and our cluster, and we’ve configured the Kubernetes provider. Now we can move away from infrastructure and onto our workloads themselves.

The last file we’ll create is a deployment.tf that contains our Kubernetes deployment and service:

resource "kubernetes_deployment" "hello-world" {
  metadata {
    name = "hello-world"
    labels = {
      app = "hello-world"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "hello-world"
      }
    }
    template {
      metadata {
        labels = {
          app = "hello-world"
        }
      }
      spec {
        container {
          image = "us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0"
          name  = "hello-world"
        }
      }
    }
  }
}

resource "kubernetes_service" "hello-world" {
  metadata {
    name = "hello-world"
  }
  spec {
    selector = {
      app = kubernetes_deployment.hello-world.spec.0.template.0.metadata.0.labels.app
    }
    port {
      port        = 80
      target_port = 8080
    }
    type = "LoadBalancer"
  }
}

You should already be familiar with these basic Kubernetes building blocks, so we don’t need to examine this code too carefully. As you can probably tell, the configuration is very similar to what we would write if we were just sending a YAML file directly into kubectl. In Terraform of course, we use Hashicorp’s configuration language (HCL) which looks a little bit like JSON, just more verbose. All the usual things are there for a deployment and a service, including the container image name and target port. You can see in the service where we link the selector to the Terraform resource for the deployment.

If your Terraform environment is configured correctly and your service account has the correct permissions, you should now be able to spin up these resources by following the normal Terraform workload of init, plan, and apply. If you have any issues with your code, you can check it against the GitHub repo I mentioned earlier.

One final note on fleets – although again, we’ll be exploring these in much more detail later in the series. If you already created a fleet by hand following the steps earlier in this post, and then you tried to create a new fleet in the same project using Terraform, you may have seen an error. That’s because a single project can only contain a single fleet. If you still want to proceed, you can simply remove the google_gke_hub_fleet resource from the Terraform code, so it doesn’t try to create a new fleet. Leave in the google_gke_hub_membership resource for your cluster, and it will join the existing fleet in the project.

Setting up our first cluster inside Google Cloud probably seemed quite easy, and that’s because it is most definitely designed that way. Next, we’ll look at how we can set up GKE Enterprise to build a cluster in AWS, which will involve laying down a few more foundations first. As you read through these sections you may find that the steps involved are quite long and complicated, but bear in mind that much of the groundwork only needs to be done once, and of course, it too can be automated by tools like Terraform.

Building clusters on AWS

Getting GKE Enterprise ready to deploy into AWS for you requires many different moving parts. At a high level, we need to perform the following steps:

1. Create an AWS VPC with the correct subnets, routes and gateways.

2. Configure keys for the encryption of EC2 instance data, EBS volumes in AWS and ETCD state data in our clusters.

3. Create IAM roles and SSH key pairs GKE runs within its own VPC inside your AWS account, which will include private and public subnets.

The private subnets will host the EC2 virtual machines that provide control plane nodes as well as worker nodes (organised into node pools) for your cluster. An internal network load balancer will provide the front end for the control plane. Finally, NAT gateways will operate in the public subnets to provide connectivity for the private subnets.

The Anthos Multi-Cloud API (as it was still named at the time of writing) communicates with the AWS API to manage AWS resources, while the Connect API communicates with the Kubernetes API of the cluster running in AWS. This allows you to create and manage your AWS clusters entirely from within the Google Cloud console, provided the foundations have been configured first.

There is a lot of groundwork to cover in setting up AWS for GKE. Many complex commands are required over hundreds and hundreds of lines. You definitely want to automate this! I'll walk through everything below, but I've also provided the commands in a set of scripts in my GitHub repo here: https://github.com/timhberry/gke-enterprise/tree/main/aws-cluster-scripts

For reference, the scripts are:

• aws-keys.sh - Creating KMS and SSH key pairs

• aws-iam.sh - Creating the necessary IAM roles, policies and profiles

• aws-vpc.sh - Creating the VPC network, subnets, routes and gateways

• aws-cluster.sh - Finally creating the cluster and node pool

The commands in these scripts rely on you having the gcloud and aws command-line tools installed and configured, and you'll also need the jq JSON tool. Finally, I've only tested these commands on Linux and MacOS. If you are using Windows, it’s recommended to use the Windows Subsystem for Linux to provide a proper Linux shell experience. Alternatively, you can use the Cloud Shell terminal in the Google Cloud console. Also, note that the scripts use the us-east-1 region, but you can change that if you prefer. I also have a convention of prefixing names with gke-cluster which you can also change if you like.

Oh, and don't just run the scripts as-is! They're supposed to helpfully compile the commands together, but you should definitely run each command one at a time to help you spot any errors. Some commands create environment variables that are used in later commands. So much preamble!

Encryption keys first

GKE Enterprise uses the AWS Key Management Service (KMS) to create symmetric encryption keys. These are then used for encryption of Kubernetes state data in etcd, EC2 instance user data, and at-rest encryption of data on EBS volumes including control plane and node pool data.

In a production environment, you should use different keys for different components, for example by encrypting configuration and volumes separately. It’s also recommended to further secure the key policy by using a minimum set of KMS permissions. However, this is not an AWS Security post, so for now, we’ll be creating a single KMS key for our purposes.

Side note: everything you create in AWS gets a unique resource name, or ARN, and often we need to reference the ARNs of resources we've already created. When we need to do this, we'll pipe the output to jq and store the result in an environment variable.

KMS_KEY_ARN=$(aws --region us-east-1 kms create-key \
    --description "gke-key" \
    --output json| jq -r '.KeyMetadata.Arn')

Next, we'll create an EC2 SSH key pair for our EC2 instances, just in case we ever need to connect to them for troubleshooting purposes:

ssh-keygen -t rsa -m PEM -b 4096 -C "GKE key pair" \
      -f gke-key -N "" 1>/dev/null
aws ec2 import-key-pair --key-name gke-key \
      --public-key-material fileb://gke-key.pub

IAM permissions

Several sets of IAM permissions will need to be set up in your AWS account for GKE Enterprise to work. First, let's set some environment variables that reference the Google Cloud project where our fleet exists:

PROJECT_ID="$(gcloud config get-value project)"
PROJECT_NUMBER=$(gcloud projects describe "$PROJECT_ID" \
    --format "value(projectNumber)")

We'll now create an AWS IAM role for the GKE Multi-Cloud API. Remember, this is the API that will communicate directly with the AWS API when we ask GKE to run commands or control components within AWS:

aws iam create-role --role-name gke-api-role \
    --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
            "Federated": "accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
            "StringEquals": {
            "accounts.google.com:sub": "service-'$PROJECT_NUMBER'@gcp-sa-gkemulticloud.iam.gserviceaccount.com"
            }
      }
    }
  ]
}'

Almost forgot! We actually need the ARN for this role, so let's grab it:

API_ROLE_ARN=$(aws iam list-roles \
  --query 'Roles[?RoleName==`gke-api-role`].Arn' \
  --output text)

Now, the GKE IAM role will needs lots of different permissions, so we'll create a new AWS IAM policy for this:

API_POLICY_ARN=$(aws iam create-policy --policy-name gke-api-policy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateLaunchTemplate",
        "ec2:CreateNetworkInterface",
        "ec2:CreateSecurityGroup",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteLaunchTemplate",
        "ec2:DeleteNetworkInterface",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteTags",
        "ec2:DeleteVolume",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeInstances",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeKeyPairs",
        "ec2:DescribeLaunchTemplates",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DescribeSecurityGroupRules",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeVpcs",
        "ec2:GetConsoleOutput",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RunInstances",
        "iam:AWSServiceName",
        "iam:CreateServiceLinkedRole",
        "iam:GetInstanceProfile",
        "iam:PassRole",
        "autoscaling:CreateAutoScalingGroup",
        "autoscaling:CreateOrUpdateTags",
        "autoscaling:DeleteAutoScalingGroup",
        "autoscaling:DeleteTags",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DisableMetricsCollection",
        "autoscaling:EnableMetricsCollection",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "autoscaling:UpdateAutoScalingGroup",
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:CreateListener",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateTargetGroup",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTargetGroups",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticloadbalancing:ModifyTargetGroupAttributes",
        "elasticloadbalancing:RemoveTags",
        "kms:DescribeKey",
        "kms:Encrypt",
        "kms:GenerateDataKeyWithoutPlaintext"
      ],
      "Resource": "*"
    }
  ]
}' --output json | jq -r ".Policy.Arn")

Now we'll attach the policy to the role:

aws iam attach-role-policy \
    --policy-arn $API_POLICY_ARN \
    --role-name gke-api-role

Next, we create another IAM role, this time for the nodes that will make up our control plan:

aws iam create-role --role-name control-plane-role \
    --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
             "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }
  ]
}'

We'll also need to create a new policy for this role:

CONTROL_PLANE_POLICY_ARN=$(aws iam create-policy --policy-name control-plane-policy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:AttachNetworkInterface",
        "ec2:AttachVolume",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateRoute",
        "ec2:CreateSecurityGroup",
        "ec2:CreateSnapshot",
        "ec2:CreateTags",
        "ec2:CreateVolume",
        "ec2:DeleteRoute",
        "ec2:DeleteSecurityGroup",
        "ec2:DeleteSnapshot",
        "ec2:DeleteTags",
        "ec2:DeleteVolume",
        "ec2:DescribeAccountAttributes",
        "ec2:DescribeAvailabilityZones",
        "ec2:DescribeDhcpOptions",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInternetGateways",
        "ec2:DescribeLaunchTemplateVersions",
        "ec2:DescribeRegions",
        "ec2:DescribeRouteTables",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSnapshots",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeVolumes",
        "ec2:DescribeVolumesModifications",
        "ec2:DescribeVpcs",
        "ec2:DetachVolume",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyVolume",
        "ec2:RevokeSecurityGroupIngress",
        "autoscaling:DescribeAutoScalingGroups",
        "autoscaling:DescribeAutoScalingInstances",
        "autoscaling:DescribeLaunchConfigurations",
        "autoscaling:DescribeTags",
        "autoscaling:SetDesiredCapacity",
        "autoscaling:TerminateInstanceInAutoScalingGroup",
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:ApplySecurityGroupsToLoadBalancer",
        "elasticloadbalancing:AttachLoadBalancerToSubnets",
        "elasticloadbalancing:ConfigureHealthCheck",
        "elasticloadbalancing:CreateListener",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateLoadBalancerListeners",
        "elasticloadbalancing:CreateLoadBalancerPolicy",
        "elasticloadbalancing:CreateTargetGroup",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteLoadBalancerListeners",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
        "elasticloadbalancing:DeregisterTargets",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DescribeLoadBalancerPolicies",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeTargetGroups",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticloadbalancing:DetachLoadBalancerFromSubnets",
        "elasticloadbalancing:ModifyListener",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:ModifyTargetGroup",
        "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
        "elasticloadbalancing:RegisterTargets",
        "elasticloadbalancing:SetLoadBalancerPoliciesForBackendServer",
        "elasticloadbalancing:SetLoadBalancerPoliciesOfListener",
        "elasticfilesystem:CreateAccessPoint",
        "elasticfilesystem:DeleteAccessPoint",
        "elasticfilesystem:DescribeAccessPoints",
        "elasticfilesystem:DescribeFileSystems",
        "elasticfilesystem:DescribeMountTargets",
        "kms:CreateGrant",
        "kms:Decrypt",
        "kms:Encrypt",
        "kms:GrantIsForAWSResource"
      ],
      "Resource": "*"
    }
  ]
}' --output json | jq -r ".Policy.Arn")

And then attach that policy to the role:

aws iam attach-role-policy \
    --policy-arn $CONTROL_PLANE_POLICY_ARN \
    --role-name control-plane-role

Now, because we're using this role and policy with EC2 instances, we'll also create an instance profile, and add the role to it:

CONTROL_PLANE_PROFILE=control-plane-profile
aws iam create-instance-profile \
    --instance-profile-name $CONTROL_PLANE_PROFILE
aws iam add-role-to-instance-profile \
    --instance-profile-name $CONTROL_PLANE_PROFILE \
    --role-name control-plane-role

Finally, we'll go through a similar process for the node pools: first creating an IAM policy, then attaching it to a role, creating an instance profile, then adding the role to the instance profile:

aws iam create-role --role-name node-pool-role \
    --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Sid": "",
        "Effect": "Allow",
        "Principal": {
        "Service": "ec2.amazonaws.com"
        },
        "Action": "sts:AssumeRole"
    }
  ]
}'

NODE_POOL_POLICY_ARN=$(aws iam create-policy --policy-name node-pool-policy_kms \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "'$KMS_KEY_ARN'"
    }
  ]
}' --output json | jq -r ".Policy.Arn")

aws iam attach-role-policy --role-name node-pool-role \
    --policy-arn $NODE_POOL_POLICY_ARN

NODE_POOL_PROFILE=node-pool-profile
aws iam create-instance-profile \
    --instance-profile-name $NODE_POOL_PROFILE

aws iam add-role-to-instance-profile \
    --instance-profile-name $NODE_POOL_PROFILE \
    --role-name node-pool-role

Now we finally have our encryption keys, IAM profiles and roles set up, and we can move onto networking!

AWS Networking

Our GKE clusters will require a dedicated VPC, subnets, internet gateways, routing tables, elastic IPs and NAT gateways to work. There's a lot to get through, so let's get started!

First, the easy bit. We'll create a new VPC with a CIDR of 10.0.0.0/16. Feel free to adjust the region to suit your requirements:

aws --region us-east-1 ec2 create-vpc \
    --cidr-block 10.0.0.0/16 \
    --tag-specifications 'ResourceType=vpc, Tags=[{Key=Name,Value=gke-cluster-VPC}]'

Next, we'll capture the ID of the VPC, and use it to enable support for DNS and hostnames:

VPC_ID=$(aws ec2 describe-vpcs \
  --filters 'Name=tag:Name,Values=gke-cluster-VPC' \
  --query "Vpcs[].VpcId" --output text)
aws ec2 modify-vpc-attribute --enable-dns-hostnames --vpc-id $VPC_ID
aws ec2 modify-vpc-attribute --enable-dns-support --vpc-id $VPC_ID

The next step is to create private subnets in our VPC for our control plane nodes. By default, a GKE cluster in AWS is the equivalent of a private cluster, with no direct access to the API permissable via the Internet. We will instead connect to it via GKE’s Connect API, as we’ll see later on. You can create fewer than three subnets in your VPC if you wish, but three are recommended as GKE will create three control plane nodes regardless. Spreading them across three different availability zones gives you better redundancy in the event of any zonal outages.

In this example I'm using us-east-1a, us-east-1b and us-east-1c availability zones and assigning each of them a /24 CIDR block. If you’re using a different region you’ll want to change these availability zones accordingly.

aws ec2 create-subnet \
  --availability-zone us-east-1a \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.1.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PrivateSubnet1}]'
aws ec2 create-subnet \
  --availability-zone us-east-1b \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.2.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PrivateSubnet2}]'
aws ec2 create-subnet \
  --availability-zone us-east-1c \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.3.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PrivateSubnet3}]'

Next we need to create three public subnets. These will be used to provide outbound internet access for the private subnets (once again, you may need to change the availability zones if you’re using a different region).

aws ec2 create-subnet \
  --availability-zone us-east-1a \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.101.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PublicSubnet1}]'
aws ec2 create-subnet \
  --availability-zone us-east-1b \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.102.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PublicSubnet2}]'
aws ec2 create-subnet \
  --availability-zone us-east-1c \
  --vpc-id $VPC_ID \
  --cidr-block 10.0.103.0/24 \
  --tag-specifications 'ResourceType=subnet, Tags=[{Key=Name,Value=gke-cluster-PublicSubnet3}]'

PUBLIC_SUBNET_ID_1=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PublicSubnet1' \
  --query "Subnets[].SubnetId" --output text)
PUBLIC_SUBNET_ID_2=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PublicSubnet2' \
  --query "Subnets[].SubnetId" --output text)
PUBLIC_SUBNET_ID_3=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PublicSubnet3' \
  --query "Subnets[].SubnetId" --output text)

At this point we’ve just created the subnets, but we haven’t made them public. To do this we need to modify the map-public-ip-on-launch attribute of each subnet:

aws ec2 modify-subnet-attribute \
  --map-public-ip-on-launch \
  --subnet-id $PUBLIC_SUBNET_ID_1
aws ec2 modify-subnet-attribute \
  --map-public-ip-on-launch \
  --subnet-id $PUBLIC_SUBNET_ID_2
aws ec2 modify-subnet-attribute \
  --map-public-ip-on-launch \
  --subnet-id $PUBLIC_SUBNET_ID_3

Next step is to create an internet gateway for the VPC:

aws --region us-east-1  ec2 create-internet-gateway \
  --tag-specifications 'ResourceType=internet-gateway, Tags=[{Key=Name,Value=gke-cluster-InternetGateway}]'
INTERNET_GW_ID=$(aws ec2 describe-internet-gateways \
  --filters 'Name=tag:Name,Values=gke-cluster-InternetGateway' \
  --query "InternetGateways[].InternetGatewayId" --output text)
aws ec2 attach-internet-gateway \
  --internet-gateway-id $INTERNET_GW_ID \
  --vpc-id $VPC_ID

You're probably wondering at this point why AWS networking is so much harder than Google Cloud! Well guess what, we're not even closed to being done 😂

We now have a ton of groundwork to lay in the form of route tables. We start with our new public subnets, creating routing tables for each an then associating them with the corresponding subnet:

aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PublicRouteTbl1}]'
aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PublicRouteTbl2}]'
aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PublicRouteTbl3}]'

PUBLIC_ROUTE_TABLE_ID_1=$(aws ec2 describe-route-tables \
    --filters 'Name=tag:Name,Values=gke-cluster-PublicRouteTbl1' \
    --query "RouteTables[].RouteTableId" --output text)
PUBLIC_ROUTE_TABLE_ID_2=$(aws ec2 describe-route-tables \
    --filters 'Name=tag:Name,Values=gke-cluster-PublicRouteTbl2' \
    --query "RouteTables[].RouteTableId" --output text)
PUBLIC_ROUTE_TABLE_ID_3=$(aws ec2 describe-route-tables \
    --filters 'Name=tag:Name,Values=gke-cluster-PublicRouteTbl3' \
    --query "RouteTables[].RouteTableId" --output text)

aws ec2 associate-route-table \
  --route-table-id $PUBLIC_ROUTE_TABLE_ID_1 \
  --subnet-id $PUBLIC_SUBNET_ID_1
aws ec2 associate-route-table \
  --route-table-id $PUBLIC_ROUTE_TABLE_ID_2 \
  --subnet-id $PUBLIC_SUBNET_ID_2
aws ec2 associate-route-table \
  --route-table-id $PUBLIC_ROUTE_TABLE_ID_3 \
  --subnet-id $PUBLIC_SUBNET_ID_3

Public subnets also need default routes for the internet gateway:

aws ec2 create-route --route-table-id $PUBLIC_ROUTE_TABLE_ID_1 \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $INTERNET_GW_ID
aws ec2 create-route --route-table-id $PUBLIC_ROUTE_TABLE_ID_2 \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $INTERNET_GW_ID
aws ec2 create-route --route-table-id $PUBLIC_ROUTE_TABLE_ID_3 \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $INTERNET_GW_ID

The final component for the public subnets is a NAT gateway, that will NAT traffic to the private subnets. These gateways require public IP addresses, so first we assign some elastic IPs, one for each gateway, then we create the NAT gateway for each public subnet:

aws ec2 allocate-address \
  --tag-specifications 'ResourceType=elastic-ip, Tags=[{Key=Name,Value=gke-cluster-NatEip1}]'
aws ec2 allocate-address \
  --tag-specifications 'ResourceType=elastic-ip, Tags=[{Key=Name,Value=gke-cluster-NatEip2}]'
aws ec2 allocate-address \
  --tag-specifications 'ResourceType=elastic-ip, Tags=[{Key=Name,Value=gke-cluster-NatEip3}]'

NAT_EIP_ALLOCATION_ID_1=$(aws ec2 describe-addresses \
  --filters 'Name=tag:Name,Values=gke-cluster-NatEip1' \
  --query "Addresses[].AllocationId" --output text)
NAT_EIP_ALLOCATION_ID_2=$(aws ec2 describe-addresses \
  --filters 'Name=tag:Name,Values=gke-cluster-NatEip2' \
  --query "Addresses[].AllocationId" --output text)
NAT_EIP_ALLOCATION_ID_3=$(aws ec2 describe-addresses \
  --filters 'Name=tag:Name,Values=gke-cluster-NatEip3' \
  --query "Addresses[].AllocationId" --output text)

aws ec2 create-nat-gateway \
  --allocation-id $NAT_EIP_ALLOCATION_ID_1 \
  --subnet-id $PUBLIC_SUBNET_ID_1 \
  --tag-specifications 'ResourceType=natgateway, Tags=[{Key=Name,Value=gke-cluster-NatGateway1}]'
aws ec2 create-nat-gateway \
  --allocation-id $NAT_EIP_ALLOCATION_ID_2 \
  --subnet-id $PUBLIC_SUBNET_ID_2 \
  --tag-specifications 'ResourceType=natgateway, Tags=[{Key=Name,Value=gke-cluster-NatGateway2}]'
aws ec2 create-nat-gateway \
  --allocation-id $NAT_EIP_ALLOCATION_ID_3 \
  --subnet-id $PUBLIC_SUBNET_ID_3 \
  --tag-specifications 'ResourceType=natgateway, Tags=[{Key=Name,Value=gke-cluster-NatGateway3}]'

Moving onto the private subnets, each of these will also need a route table. We'll create them, assign the ID to an environment variable, and associate each table with its corresponding subnet:

aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PrivateRouteTbl1}]'
aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PrivateRouteTbl2}]'
aws ec2 create-route-table --vpc-id $VPC_ID \
  --tag-specifications 'ResourceType=route-table, Tags=[{Key=Name,Value=gke-cluster-PrivateRouteTbl3}]'

PRIVATE_SUBNET_ID_1=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateSubnet1' \
  --query "Subnets[].SubnetId" --output text)
PRIVATE_SUBNET_ID_2=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateSubnet2' \
  --query "Subnets[].SubnetId" --output text)
PRIVATE_SUBNET_ID_3=$(aws ec2 describe-subnets \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateSubnet3' \
  --query "Subnets[].SubnetId" --output text)
PRIVATE_ROUTE_TABLE_ID_1=$(aws ec2 describe-route-tables \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateRouteTbl1' \
  --query "RouteTables[].RouteTableId" --output text)
PRIVATE_ROUTE_TABLE_ID_2=$(aws ec2 describe-route-tables \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateRouteTbl2' \
  --query "RouteTables[].RouteTableId" --output text)
PRIVATE_ROUTE_TABLE_ID_3=$(aws ec2 describe-route-tables \
  --filters 'Name=tag:Name,Values=gke-cluster-PrivateRouteTbl3' \
  --query "RouteTables[].RouteTableId" --output text)

aws ec2 associate-route-table --route-table-id $PRIVATE_ROUTE_TABLE_ID_1 \
  --subnet-id $PRIVATE_SUBNET_ID_1
aws ec2 associate-route-table --route-table-id $PRIVATE_ROUTE_TABLE_ID_2 \
  --subnet-id $PRIVATE_SUBNET_ID_2
aws ec2 associate-route-table --route-table-id $PRIVATE_ROUTE_TABLE_ID_3 \
  --subnet-id $PRIVATE_SUBNET_ID_3

We're so close! The final stage is to create default routes to the NAT gateways for our private subnets:

NAT_GW_ID_1=$(aws ec2 describe-nat-gateways \
 --filter 'Name=tag:Name,Values=gke-cluster-NatGateway1' \
 --query "NatGateways[].NatGatewayId" --output text)
NAT_GW_ID_2=$(aws ec2 describe-nat-gateways \
 --filter 'Name=tag:Name,Values=gke-cluster-NatGateway2' \
 --query "NatGateways[].NatGatewayId" --output text)
NAT_GW_ID_3=$(aws ec2 describe-nat-gateways \
 --filter 'Name=tag:Name,Values=gke-cluster-NatGateway3' \
 --query "NatGateways[].NatGatewayId" --output text)

aws ec2 create-route --route-table-id $PRIVATE_ROUTE_TABLE_ID_1  \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_GW_ID_1
aws ec2 create-route --route-table-id $PRIVATE_ROUTE_TABLE_ID_2  \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_GW_ID_2
aws ec2 create-route --route-table-id $PRIVATE_ROUTE_TABLE_ID_3 \
  --destination-cidr-block 0.0.0.0/0 --gateway-id $NAT_GW_ID_3

Again, hats off to you if you work with AWS networking on a daily basis. I hope you get plenty of paid leave! But seriously, even though there's a ton of foundations to lay here, hopefully you only have to do it once.

Actually, finally, building a cluster in AWS

This is the moment we've been waiting for! Just note that the commands below will reference a lot of the environment variables that were created in the stages above, so make sure these still exist in your terminal session before you proceed. Building the cluster is a two step process - control plane first, then worker nodes.

Let's start with the cluster and its control plane:

gcloud container aws clusters create aws-cluster \
  --cluster-version 1.26.2-gke.1001 \
  --aws-region us-east-1 \
  --location us-east4 \
  --fleet-project $PROJECT_ID \
  --vpc-id $VPC_ID \
  --subnet-ids $PRIVATE_SUBNET_ID_1,$PRIVATE_SUBNET_ID_2,$PRIVATE_SUBNET_ID_3 \
  --pod-address-cidr-blocks 10.2.0.0/16 \
  --service-address-cidr-blocks 10.1.0.0/16 \
  --role-arn $API_ROLE_ARN \
  --iam-instance-profile $CONTROL_PLANE_PROFILE \
  --database-encryption-kms-key-arn $KMS_KEY_ARN \
  --config-encryption-kms-key-arn $KMS_KEY_ARN \
  --tags google:gkemulticloud:cluster=aws-cluster

As you can see, there are many different options that we pass into the command. Notably, we can specify a GKE version (1.26.2-gke.1001 in this example). We also need to provide the AWS region to build the cluster in with the --aws-region option, as well as the Google Cloud region where the Connect API should run. This isn’t available in all Google Cloud locations, so in this example, us-east4 is the closest location to our AWS region where the Connect API is available.

GKE will leverage the Multi-cloud API along with the IAM and KMS configuration you have set up in AWS to connect to the AWS API and build the control plane of your cluster. This should take about 5 minutes, after which your cluster should be shown in the GKE Enterprise cluster list in the Google Cloud console. Note that at this point you may see a warning next to the cluster – don't worry, we’ll fix that in a moment.

Next, we build a node pool for our new cluster – this doesn’t happen automatically like when we build a cluster on Google Cloud. To do this we specify a minimum and a maximum number of nodes along with other options such as the root volume size for each virtual machine, and the subnet ID where they should be hosted:

gcloud container aws node-pools create pool-0 \
  --location us-east4 \
  --cluster aws-cluster \
  --node-version 1.26.2-gke.1001 \
  --min-nodes 1 \
  --max-nodes 5 \
  --max-pods-per-node 110 \
  --root-volume-size 50 \
  --subnet-id $PRIVATE_SUBNET_ID_1 \
  --iam-instance-profile $NODE_POOL_PROFILE \
  --config-encryption-kms-key-arn $KMS_KEY_ARN \
  --ssh-ec2-key-pair gke-key \
  --tags google:gkemulticloud:cluster=aws-cluster

If you don't see a green tick next to your cluster on the GKE clusters page, scroll all the way to the right of your new AWS cluster and click the three dots (Google calls this an actions menu). Select Log in and use the option to authenticate with your Google credentials. This authenticates your console session and creates the bridge from GKE to the Kubernetes API and AWS API. After a few moments, you should see a green tick which means your AWS cluster is now being managed by GKE!

Testing the AWS cluster

Interacting with your new AWS cluster works the same way as if you were working with any other Kubernetes cluster. Even though we've built this cluster in AWS, we're managing it via GKE Enterprise, so we can use gcloud to get credentials for the Kubernetes API and store them in our local kubeconfig file:

gcloud container aws clusters get-credentials aws-cluster --location us-east4

Once you’ve done this, you can issue normal kubectl commands, for example, to see the state of the cluster’s nodes: kubectl get nodes.

Let’s run the following commands to run a test deployment and expose it via a Load Balancer, just like we did with our Google GKE cluster earlier:

kubectl create deployment hello-server --image=us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
kubectl expose deployment hello-server --type LoadBalancer --port 80 --target-port 8080

It will take a few minutes for the Load Balancer to get set up. Unlike exposing a Load Balancer service in Google Cloud, our AWS Kubernetes cluster will assign the service a hostname rather than an external IP. You can get the hostname with this command:

kubectl get service hello-server

You should now be able to load the hostname in your browser and see that your deployment is working on AWS! If it doesn’t work straight away, wait a few minutes while AWS configures the Load Balancer and try again.

Deleting the AWS cluster

While deleting a cluster hosted in Google Cloud is quite straightforward, deleting an AWS cluster requires separate steps for the node pools and control plane. Assuming your names and locations match the instructions we followed previously to create the cluster, first delete the node pool with this command:

gcloud container aws node-pools delete pool-0 --cluster aws-cluster --location us-east4

When that command has been completed, we can finally delete the control plane with this command:

gcloud container aws clusters delete aws-cluster --location us-east4

This will delete the EC2 virtual machines that were used to provide the cluster. However...

Deleting other AWS resources

Even though you’ve deleted the EC2 virtual machines, your AWS account will still contain the other resources we created for the VPC and IAM configurations, some of which may continue to incur fees (such as the elastic IP assignments). There’s not enough room in this post to detail how to delete everything, but if you want to be sure that you don’t incur any further charges, walk back through the commands we used to create the resources and remove each resource individually, either using the aws command line or the web console.

Summary

We’ve demonstrated how to build a Kubernetes cluster in AWS that can be managed by GKE, and hopefully you now have a good understanding of the requirements for using GKE in AWS. Google Cloud and AWS are often used together by organisations to provide complementary services, extra redundancy, or availability in different geographic regions, so understanding how to leverage GKE across both vendors should prove very useful.

We're hopefully starting to gain an understanding of the purpose of GKE Enterprise, and what makes it different from working with standard GKE clusters. We learned how the Connect API allows GKE to communicate with other clouds and external Kubernetes clusters, and we started to look at the requirements for building GKE in environments that are external to Google Cloud. GKE guarantees that we are using conformant versions of Kubernetes, so under the hood we always have the same technologies and components to build from, even if a different vendor’s approach to cloud networking requires us to make a few changes. And this is the beauty of using open standards and an open-source project like Kubernetes.

Congratulations if you made it to the end of this very, very long post! 🎉 (Thanks, designers of AWS networking concepts). In the next post, we'll look at how to build GKE Enterprise on bare metal and VMWare environments.

Introduction

Kubernetes is one of the most successful open-source software stories in the history of computing. Its rapid development and adoption can be attributed to the fact that it solves numerous problems that have faced developers and systems administrators for many years. If you have worked in technology for over 10 years, you may recall previous attempts to solve the fundamental problems of deploying, scaling, and managing software, which took various forms from simple scripting to package and configuration management systems, each with its own quirks and compromises. As these attempted solutions were developed, the problems themselves got more complex, as demand for software and services increased by orders of magnitude, complicating the way that we build and deploy software over distributed systems, often in other people’s datacenters.

As a way to package software, container technology was quickly adopted thanks to the developer-friendly toolset developed by Docker, but a logical and consistent way to orchestrate containers was still missing until Kubernetes. Based on Google’s extensive experience running software at an enormous scale over thousands of commodity servers, Kubernetes finally gave us a way to logically deploy, scale, and manage containerized software.

But progress never stops, and with it, complexity always increases. We’ve moved past the early days of mass cloud adoption and we live in a world where complex distributed systems need to run reliably on any combination of cloud or on-premises platforms. These new problems have mandated new solutions and new technologies such as service mesh and a distributed approach to authorization and authentication. Fully managed Kubernetes services evolved to support these new challenges with the launch of Google Cloud’s Anthos, which has now matured into GKE Enterprise.

Of course, not all deployments need to be this complex, and if you’re lucky enough to be working on small-scale projects with moderately sized clusters, you may not need all of these extra features. You certainly should not try to apply over-engineered solutions to every problem, just for the sake of playing with the latest cool new technologies. One of the approaches I want to take in this series is to identify the use case for every new tool, so you can appreciate where – and where not – to use it.

This brings us to our overall objective. Despite the increased complexity I’ve described, thankfully these new technologies are easy enough to integrate into your projects because they operate within Kubernetes’ fundamentally logical model: a clear set of APIs and a declarative approach to using them.

In this series, we’ll explore each advanced feature in a sensible order, explaining what it's for and how you can use it in an approachable, no-nonsense way. We’ll start by revisiting some core Kubernetes topics to set the scene for what we’re going to learn, but we’re not going to teach Kubernetes from scratch. If you don’t already have Kubernetes experience under your belt, you might want to get some more practice in before exploring GKE Enterprise!

A quick recap: Kubernetes Architecture

Like I said, you should already be reasonably familiar with Kubernetes, so we’re not going to deep dive into its architectural nuts and bolts here. However, the topic is worth revisiting to explain the limits of the architecture that the core Kubernetes project cares about, where those sit within managed environments, and how the advanced features of GKE Enterprise extend them to other architectures.

To recap, Kubernetes generally runs on clusters of computers. You can of course run all of its components on a single computer if you wish, but this would not be considered a production environment.

Computers in a Kubernetes cluster are referred to as nodes. These can be provided by physical (bare metal) machines or virtual machines. Nodes provide the raw computing power and the container runtimes to actually run our containerized workloads. Nodes that only run our workloads are often referred to as worker nodes, and there are mechanisms that exist to group worker nodes of the same configuration together. However, we also need some computers to run the Kubernetes control plane.

The control plane can be thought of as Kubernetes’ brain to a certain extent. The control plane’s components make decisions about the cluster, store its state, and detect and respond to events. For example, the control plane will receive instructions describing a new workload and decide where the workload should be run. In production environments, it's common to isolate the control plane to its own set of computers, or even run multiple copies of a control plane for high availability.

As a reminder, the control plane runs these components:

• kube-apiserver: The front-end of the control plane, exposing the Kubernetes API.

• etcd: The consistent key-value store for all cluster data such as configuration and state.

• kube-scheduler: A watch loop that keeps an eye out for workloads without an assigned node, so that it can assign one to them. Influencing the scheduler is an important but complex topic.

• kube-controller-manager: The component that runs controller processes. These are additional watch loops for tasks like monitoring nodes and managing workloads.

• cloud-controller-manager: If you’re running your cluster in the cloud, this component links Kubernetes to your cloud provider’s APIs.

Each worker node will run these components:

• kubelet: Kubernetes’ agent on each node and the point of contact for the API server. The kubelet agent also manages the container runtime to ensure the workloads it knows about are actually running.

• kube-proxy: A network proxy to implement Kubernetes service concepts (more on those in a moment).

• Container runtime: The underlying container runtime responsible for actually running containers (such as containerd, CRI-O, or other supported runtime).

Remember that Kubernetes is software that runs on your nodes. Kubernetes itself does not manage the actual nodes, whether they are physical or virtual. If you build Kubernetes for yourself, you will need to provision those computers and operating systems, and then install the Kubernetes software, although the Kubernetes project does provide tools like kubeadm to help you do this. When Kubernetes is running, it can only communicate with its other components and can’t directly manage a node, although the node controller component does at least notice if a node goes down.

It’s for this reason that managed services for Kubernetes have gained so much popularity. Services such as Google Kubernetes Engine (GKE) provide management of the infrastructure required to run Kubernetes as well as the software. For example, GKE can deploy nodes, group them together, manage and update the Kubernetes software on them, auto-scale them and even heal them if they become unhealthy. But this is an important distinction – GKE is managing your infrastructure, while Kubernetes is running on that infrastructure and managing your application workloads.

Within standard Kubernetes and most managed services, your Kubernetes cluster is the perimeter of your service, although of course you can have many clusters, all separately managed. As we’ll learn soon, GKE Enterprise allows us to extend those boundaries and manage multiple clusters in any number of different platforms, all with a single management view and development experience. But before we get into those topics, let’s refresh our memory on some more basic Kubernetes principles. If you’re a Kubernetes expert and you’re just here for the GKE Enterprise specifics, feel free to skip this section!

Fundamental Kubernetes Objects

Let's revisit some fundamental Kubernetes objects, because we need a solid understanding of these before we learn how to extend them with GKE Enterprise’s advanced features in future posts. If you live and breathe Kubernetes on a daily basis, feel free to skip this section! Alternatively, if you're not comfortable with any of them, a good place to start is https://kubernetes.io/docs/concepts/overview/working-with-objects/

Pods

Pods are the smallest deployable object on Kubernetes, representing an application-centric logical host for one or more containers. Arguably, their advanced patterns for colocating sidecar, init, and other patterns of containers provided the flexibility that pushed Kubernetes ahead of other orchestration platforms. The key design principle of Pods is that they are the atomic unit of scale. In general, you increase the number of Pods (not the number of containers inside a Pod) to increase capacity. Aside from some specific circumstances, it’s very rare to deploy an individual Pod on its own. Logical controllers such as Deployments and StatefulSets are much more useful.

ReplicaSets

ReplicaSets are sets of identical Pods, where each Pod in the set shares exactly the same configuration. ReplicaSets pre-date other controller objects, and even though we never explicitly create a ReplicaSet on its own, it is worth understanding their purpose within those other logical objects. You can think of a ReplicaSet as a version of a configuration. If you change a Deployment for example, a new ReplicaSet is created to represent that new version. Because ReplicaSets are just versions of configuration, we can keep a history of them in the Kubernetes data store, and easily roll back versions.

Deployments

Deployments are one of the most commonly used controller objects in Kubernetes. Controller objects are useful because the control plane runs a watch loop to ensure that the things you are asking for in your object actually exist. This is the fundamental concept of declarative configuration in Kubernetes. Individual Pods, for example, may come and go, and to a certain extent, the control plane doesn’t care. But if your Deployment object states that you should have 5 Pods of a specific configuration, the Deployment controller which make sure that this remains true.

Deployments are designed specifically for stateless workloads. This means that the container itself must not attempt to store or change any state locally and that each Pod in the Deployment must be an anonymous identical replica. Of course, you can still affect the state as long as it lives somewhere else, such as in a networked storage service or database. Stateless deployments offer you the most flexibility because your Pods can be scaled – effectively, deleted, and recreated – at will.

StatefulSets

For workloads that are required to manage state, a StatefulSet provides most of the logic of the Deployment object but also introduces some guarantees. Pods are scaled up and down in a logical order, do not have to be identical, and are guaranteed a unique identity. A common design pattern for stateful deployments is to have the first Pod serve as a controller and additional Pods act as workers. Each Pod can determine its own purpose based on its identity and local hostname. Additionally, storage volumes can be attached to Pods on a one-to-one basis.

Services

With Pods coming and going all the time, and workloads being served by multiple Pods at any given time, we need a way to provide a fixed network endpoint. This is the purpose of the Service object. Services come in a few different forms, but all of them provide a fixed internal IP available to the cluster that can route traffic in a round-robin fashion to a group of Pods. The group of Pods is determined by a label selector. For example, if your Service is configured to route traffic to Pods that match the labels app=web and env=prod, all Pods that contain these labels in their metadata will be included in the round-robin group. It’s a very low-level object that we rarely use on its own, but it’s important to understand.

Ingress

Service objects have been used historically to provide external access to workloads running in a Kubernetes cluster, often through the LoadBalancer type of Service, which attaches an external IP address or an external service, depending on where you’re running your cluster. However, there are limitations to using a low-level object that provides such a direct mapping of traffic, so the Ingress object was created.

The Ingress object is designed for HTTP and HTTPS traffic and offers more features and flexibility than the simple Service, such as the ability to route traffic to different back-end services based on request paths. Ingress objects themselves are used in conjunction with an Ingress Controller, which is typically a Deployment of some kind of proxy software, such as the NGINX web server. Depending on which server you use as your Ingress Controller, you can also add annotations to your Ingress objects, for example, to invoke NGINX rewrite rules.

The Ingress pattern has been extremely useful and popular, but due to its limited focus on HTTP the feature has now been frozen, meaning it will not receive any further development. Instead, it has inspired a new approach that takes the same flexible design ideas from Ingress but handles all kinds of traffic.

Gateway API

The Gateway API is the project that enables this approach. Notably, this is an add-on for Kubernetes, not part of its default supported API groups. Gateway API provides three new custom API resources:

• Gateway: An object representing how an external request should be translated within the Kubernetes cluster. A Gateway resource specifies protocols and ports and can also offer granular control over allowed routes and namespaces.

• HTTPRoute: An object representing how traffic coming through a gateway should be directed to a back-end service, including paths and rules to match. Other protocol-specific objects (such as GRPCRoute) are also available.

• GatewayClass: An object representing the controller that provides the actual functionality for gateways, similar to the Ingress Controller we described earlier. Gateways must reference a GatewayClass.

So we've refreshed our memory on some of the most commonly used Kubernetes objects. Of course, there are many more that we haven’t touched on, however, it was important to go back to basics and build up to the Gateway API as it’s a key component of many GKE Enterprise features such as service mesh. Next, to understand where GKE Enterprise fits in with the portfolio of Google Cloud services, let’s discuss the different editions of GKE available.

Anthos, Multi-cloud and GKE Editions

Google Kubernetes Engine (GKE) has evolved through many iterations over the years. As we previously mentioned, Google’s own internal container orchestration work was what originally inspired the open-source project Kubernetes, which was released to the world in June 2014. Google’s cloud platform at the time had a handful of core services, but fast forward to 2017, and something called Google Container Engine was launched at Google’s annual Next conference.

This new service promised to take all of the convenience and power of the Kubernetes project and apply a fully managed service for the infrastructure required to run it, for example by auto-scaling and auto-healing the virtual machines required to run a cluster. It started with a fantastic success story – the global phenomenon of Pokemon Go had recently launched with all of its backend service running on Google Container Engine (with lots of help from Google engineers). Later the same year, the service was renamed more accurately as Google Kubernetes Engine and then certified by the Cloud Native Computing Foundation as a certified Kubernetes platform, guaranteeing its compatibility with open-source Kubernetes implementations.

Over the following years, Google continued to improve GKE by offering the latest versions of Kubernetes as part of their managed service. During this time Google Cloud moved to position itself as a vendor that supported hybrid and multi-cloud portability and began to develop extensions to GKE that allowed it to run nodes on computers that ran in locations other than their own cloud. This new service launched under the product name of Anthos in 2019.

Anthos, as it turns out, was more of a name for a subscription service that you had to pay for rather than an individual product. Paying for Anthos gave you access to new features including GKE On-Prem and Anthos Migrate. GKE On-Prem allowed you to run Kubernetes clusters on bare-metal and VMWare servers which were managed by GKE. Anthos Migrate would (in theory) allow you to migrate virtual machines to stateful Pods on a GKE cluster. Anthos was rough around the edges on launch, but soon added even more useful features such as a managed Istio service mesh, a managed KNative implementation (known as Cloud Run for Anthos), policy agents, and more.

Over the years more features were developed for Anthos, and occasionally they would drift into the core GKE service – and back out again. This occurred for the service mesh features as well as a new service for binary authorization of containers. Support for running clusters in AWS and Azure was added, causing some confusion with the “On-Prem” product name. Anthos rapidly grew into an extremely powerful platform for running GKE anywhere, but its original product name now seemed somewhat incongruous.

From Anthos to GKE Enterprise

So towards the end of 2023, Google announced the launch of GKE Enterprise, which essentially means two things. Firstly, it’s a normalization of everything Anthos was but under the proper GKE product banner. Secondly, it’s a quiet deprecation of the Anthos brand. For most intents and purposes, particularly when it comes to features, technology, or documentation, you can consider the two terms as interchangeable. The main difference is that now Anthos is no longer a separate thing adjacent to GKE, it is an edition of GKE. GKE Enterprise to be exact!

Now, when choosing how to use GKE as a platform, you now have the two options of GKE Standard and GKE Enterprise. At a very high level, GKE Standard provides an advanced managed Kubernetes offering within Google Cloud, while GKE Enterprise extends the offering across multiple clouds and bare metal and adds additional features.

The most important features are presented for comparison in the table below:

Pricing changes all the time in the competitive world of cloud computing, but at the time of writing GKE Standard charges a fee per cluster hour, while GKE Enterprise charges a fee per CPU hour.

What about Autopilot?

A separate development in the world of GKE was the launch of Autopilot in 2021, an attempt to bring some of the convenience of the serverless world to Kubernetes. When creating a normal GKE cluster you must make decisions about the type, size, and number of your nodes. You may or may not enable node-autoscaling, but you normally have to continue to manage the overall capacity of your cluster during its lifetime.

With Autopilot, a cluster is fully managed for you behind the scenes. You no longer have the need to worry about nodes, as they are automatically scaled to the correct size for you. All you have to do is supply your workload configurations (for example, Deployments), and Autopilot will make all the necessary decisions to run them for you.

Confusingly, this means that GKE Standard is now an edition of GKE, as well as a type of cluster. A GKE Standard cluster is a cluster where you manage the nodes as well as the workloads, and these clusters can run in either edition of GKE, Standard, or Enterprise. Autopilot clusters are also supported in both the Standard and Enterprise editions of GKE. However, at the time of writing, Autopilot clusters are only supported inside Google Cloud.

Summary

In this first post, we have refreshed our memories on the basic architecture of a Kubernetes cluster and some of the fundamental component objects that are used to build workloads and services. Having these concepts fresh in our memory will be useful as we introduce the advanced features of GKE Enterprise throughout the rest of the series.

We have also clarified the different versions and editions of GKE, and where Anthos fits into the equation. Hopefully, this all makes sense now, despite the best efforts of Google’s product managers! Importantly, we’ve established what makes GKE Enterprise different, and now we’re ready to start learning how to use it. In the next post, we’ll begin by enabling GKE Enterprise and configuring our first GKE clusters.

That tutorial set out a very basic template for a FastAPI app that used API keys, but to keep it simple it used hard-coded database functions that simply checked for API keys in a local Dictionary. But my plan was always to extrapolate from that first step however, and slowly improve the template. In this post, we'll actually implement a database lookup function using SQLite3. If you want to follow along, just grab the code from the previous post.

The first thing we'll do is actually create a database. SQLite3 is rarely used in production systems (although sometimes it is!) but hopefully you can see from this example how easy it would be to start building something similar using Postgres for example. Within the app directory (remember we're using code from the previous post), create a new database file with this command:

sqlite3 db.sqlite

We can now populate the database. We'll keep it simple by creating a users table to store the API keys for Bob and Alice, our test users from before:

CREATE TABLE users(userid text, name text, apikey text);
INSERT INTO users VALUES('7oDYjo3d9r58EJKYi5x4E8','Bob','e54d4431-5dab-474e-b71a-0db1fcb9e659');
INSERT INTO users VALUES('mUP7PpTHmFAkxcQLWKMY8t','Alice','5f0c7127-3be9-4488-b801-c7b6415b45e9');

db.py

If you recall from the previous post, our db.py file contained hard-coded user IDs and API keys, and then two functions. check_api_key would check for the existence of the API key in the "database" (which was really just a Dictionary), and get_user_from_api_key would then retrieve the user object itself. I used this two-step approach to demonstrate the logic of things, but seeing as we're upgrading to using a real database, let's also make this more efficient.

We'll replace db.py entirely with the following:

import sqlite3

def check_api_key(api_key: str):
    with sqlite3.connect('db.sqlite') as conn:
        cur = conn.cursor()
        cur.execute('select name from users where apikey = ?', [api_key])
        row = cur.fetchone()
        if row:
            return(row)
    return False

Now we just have a check_api_key function. It makes the connection to the database, and searches for a row based on the provided API key. If an API key matches, it returns that row (in other words, the user object).

If no API key matches, we return False. Now we need to update how we call this function from auth.py.

auth.py.

We've changed the logic of how these two parts of the program talk to each other. Here's our updated auth.py:

from fastapi import Security, HTTPException, status
from fastapi.security import APIKeyHeader
from db import check_api_key

api_key_header = APIKeyHeader(name="X-API-Key")

def get_user(api_key_header: str = Security(api_key_header)):
    user = check_api_key(api_key_header)
    if user:
        return user
    raise HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Missing or invalid API key"
    )

Comparing this to the previous version, you can see that we now just define the get_user function. This is the only function that gets called by our secure route, and we've removed the need for the separate get_user_from_api_key function. (Note that we've also removed it from the import line).

All that get_user has to do now is call the check_api_key function from our database and check the response. As long as it's not false, it will return the user object. If the response is false however, we know that the API key was missing or invalid, so we return the correct HTTP code just like before.

What's next for our little FastAPI app? I was already thinking along these lines in the previous post, but if we've handled authentication, we should start thinking about authorisation. What are Bob and Alice permitted to do within our app, how do we store that in a database and apply the logic in our app?

Maybe next time I migrate my blog, I'll write the follow up :)

Clearly they matter to a lot of people, myself included. Passing a certification is a way of displaying that you have learned the skills to pass the exam at the very least, and depending on the exam this might be a very close approximation of the skills required to do a specific job. Certification badges are a kind of reputational capital. We acquire them to show that we have accumulated skills, and they demonstrate that we have taken an interest in our own continuing professional development. Often they are a requirement, or at least an easy filtering mechanism for recruiters.

But are there other forms of reputational capital? Absolutely there are! Let’s not forget, not everyone copes well under exam conditions - and more importantly, most exam conditions are completely detached from what is required to actually do a job (with some exceptions, more on that in a moment!). Open-source contributions are another great way to be demonstrable about your skills and interests, as are blog posts and other forms of content creation. Conversely, you may be a highly skilled software developer but your job prevents you from sharing this with the world, so certs might be your only choice.

So, if certs are right for you, where do you start? My background and experience are mostly in the clouds, so here’s a quick overview of where to get started with the most relevant cloud certifications (in my humble opinion). In alphabetical order:

Amazon Web Services (AWS)

The AWS entry level certification is Certified Cloud Practitioner (CCP). This exam proves you have a foundational knowledge of cloud concepts, as well as AWS services and technologies. In theory, you don’t need to have technical skills to pass this exam, although you will need to memorise lots of different AWS services which tend to have obscure names! For technical certifications, AWS offers Associate level certifications, which you must have before you progress onto Professional level certifications. These are all offered in the roles of Solutions Architect, Data Engineer, Developer and SysOps Administrator. Most cloud generalists take the Solutions Architect route, starting with the Solutions Architect: Associate exam. These are all multiple choice “closed book” exams.

Azure

Similar to AWS, Microsoft offers different levels of certifications, from Fundamentals to Associate, then Expert level with some Speciality exams too, and these are also generally based on roles such as Data Engineer and Administrator. If you don’t know where to specialise yet, you can start with Microsoft Certified: Azure Fundamentals, before moving onto one of the associate level certs like Microsoft Certified: Azure Administrator Associate. Depending on the level and topic of your exam, you may have a combination of multiple choice and other question types. Microsoft is also introducing some hands-on elements to their exams to test your practical skills. Make sure to check the exam guide in depth for your chosen certification!

Google Cloud

Almost all of Google Cloud’s certification exams are aimed at the Professional practitioner level, and based on various job roles including Cloud Architect, Data Engineer, Machine Learning Engineer and Developer. The exception to this rule is the Associate-level Cloud Engineer exam (ACE) and the Cloud Digital Leader exam. Either of these might be a good place to start for you. The Cloud Digital Leader certification is pitched as a non-technical exam for business leaders and other “tech-adjacent” stakeholders, however you will still need a good understanding of Google Cloud concepts and products, similar to how AWS sets up its CCP exam. If you’re looking to take your first technical step into Google Cloud certification, the ACE may be more appropriate for you, but it will require a good working knowledge of operating various Google Cloud solutions. Currently, all of these exams are multiple choice.

That’s enough vendor certs, what about cloud-native, cloud-adjacent, or just good old plain open source?

The Cloud Native Computing Foundation (CNCF) offers several exams in the realms of Kubernetes and other open source technologies. For example, the famous Certified Kubernetes Administrator (CKA) exam is a completely hands-on exam, testing your ability to operate, manage and fix Kubernetes clusters and their workloads (with not a single multiple-choice question in sight). This style of exam may seem more daunting than a simple question paper, but in reality it is a much better representation of what doing an actual job is like, which is why these credentials are held in such high regard. The CKA exam environment has you working with real problems on real clusters, with access to the Kubernetes project website and documentation. If you’re just getting started with Kubernetes, you may want to look at the Kubernetes and Cloud Native Associate (KNCA) exam first, which is not hands-on; it’s a multiple choice exam aimed at a more foundational level.

You might have a variety of motivations for getting certified. Maybe it’s to increase your reputational capital, maybe it’s to brush up your CV, or maybe it’s just because you love learning, and each new cert is another milestone. If you’ve never thought of trying for a certification before, don’t be afraid to give one a try. There’s no harm in patting yourself on the back once in a while!

Most of my time these days is spent teaching and developing teaching material for some advanced technical topics such as Kubernetes, serverless and machine learning, and most of the people I’m teaching have already been working in tech for a number of years. The sheer volume of things to learn about can be daunting for someone new coming into tech, especially if they’re attempting a career change or have no formal computer science education.

A career change is an enormous undertaking. It will require massive commitment, a great deal of time and a lot of faith. But I fundamentally believe anyone can do it, and start working in tech! So where do you start?

In recent years, the lack of skills in the market has prompted companies like IBM and Meta to produce some very slick online courses that can teach you everything you need to know to get started in some specific tech areas. Often they will assume no prior knowledge and will guide you through everything you need to know at a foundational level, with guided lessons and activities. These include Google’s Cybersecurity Professional Certificate, Meta’s Frontend Developer Professional Certificate, and IBM’s Full Stack Software Developer and DevOps programmes. You can find links to these and more at Coursera’s Find Your New Career page.

And don’t forget that you can “audit” any Coursera class for free if you don’t want to pay right now (or at all). That just means you won’t get the certification or the feedback, but if you’re just there to learn you’ll still have access to everything you need.

While these courses are great, they do require you to have at least an inkling of the career direction you want to take up front. Would you want to spend 6 months learning full stack development, only to discover a yearning to be a data analyst? These courses also take a very vocational approach, focusing on specific skills for a specific role in the industry.

For me, the joy of learning comes from a life-long personal interest in tech. How is it put together, and why does it work the way it does? Exploring some fundamentals of computer science might help you to decide if tech really is for you, and you might discover which aspects of it inspire your own joy along the way.

One of the best online courses for doing this is Harvard’s CS50, taught by the incredible David J. Malan. This course takes you through foundational building blocks but also throws you in at the deep end with C, explores data structures and algorithms and even covers some basic web and cybersecurity concepts. The course has been taught to thousands of people around the world through Harvard’s OpenCourseware or Edx (both have free options). It’s a great way to find out what excites you about tech, and if you want to continue your journey afterwards, it even offers specialisms in web development, Python, game development, and of course AI.

What was your path into tech? Did you switch from another career, and if so, how? I’d love to hear from you if you have other amazing educational resources to share.

Project layout

We’ll start with a simple Python project layout for a FastAPI app.

.
├── app
│   ├── __init__.py
│   ├── auth.py
│   ├── db.py
│   ├── main.py
│   └── routers
│       ├── __init__.py
│       ├── public.py
│       ├── secure.py
├── venv

Hopefully this makes sense if you’re fairly comfortable with Python. app is our Python package in this case, main is our main module, while auth and db are utility modules in the package. routers is a subpackage containing the public and secure submodules. There’s empty __init__.py files aplenty to tell Python this is a proper package, and in my case I also have a venv directory for my virtual environment, although that isn’t required. Now let’s go into what each of these files actually does.

main.py

Here’s our main module:

from fastapi import FastAPI, Depends
from routers import secure, public
from auth import get_user

app = FastAPI()

app.include_router(
    public.router,
    prefix="/api/v1/public"
)
app.include_router(
    secure.router,
    prefix="/api/v1/secure",
    dependencies=[Depends(get_user)]
)

We start by importing the modules we’ll need from FastAPI, namely itself and the Depends module. From our own auth module we import a function called get_user which we’ll explain in a moment. From our routers subpackage we import secure and public. And we create our app with app = FastAPI().

So what are these routers? Well, in a simple FastAPI app we could just start defining paths and responses in our main program. Here’s a super simple example:

@app.get("/")
async def root():
    return {"message": "Hello World"}

In 3 lines we can specify that we want to return “Hello World” for every GET request to / (asynchronously to boot!). But this doesn’t scale very well.

A router is a separate set of paths and functions grouped together. They’re useful for grouping features of a much larger API together into logical pieces. In this tutorial, we’ll create two groups of paths: secure for paths that require an API key, and public for those that don’t.

Let’s look at this piece again:

app.include_router(
    public.router,
    prefix="/api/v1/public"
)

Here we call the include_router function on our app, and pass in 2 paramters. The first, public.router is the router object from the public module. The second prefix, defines where these paths sit in the context of our overall API app. As you can see, routers make it easy to change URL prefixes and even introduce API versioning.

The next one is a bit more complex:

app.include_router(
    secure.router,
    prefix="/api/v1/secure",
    dependencies=[Depends(get_user)]
)

Now we’re using the router object from the secure module, and we’ve added a dependency using FastAPI’s Depends function. The dependency is on the get_user function, that we’ve defined in the auth module. As you can probably guess, that means that all paths in this router depend on that function. So let’s take a look at it.

auth.py

from fastapi import Security, HTTPException, status
from fastapi.security import APIKeyHeader
from db import check_api_key, get_user_from_api_key

api_key_header = APIKeyHeader(name="X-API-Key")

def get_user(api_key_header: str = Security(api_key_header)):
    if check_api_key(api_key_header):
        user = get_user_from_api_key(api_key_header)
        return user
    raise HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Missing or invalid API key"
    )

First we import some more useful stuff from FastAPI, including methods for dealing with headers and HTTP exceptions. They’ll make more sense when we get to the part that actually uses them in a moment. And yes, I do appreciate that this is a bit circular, because next we’re importing functions from db which we haven’t seen yet. But before we get into that file, just know that check_api_key returns True if an API key is valid, and get_user_from_api_key returns a user object from a valid API key.

The main purpose of the auth module is to provide us with the get_user function, which as we saw in the previous file, our secure routes depend on. This function accepts one parameter - the value of X-API-Key that has been sent in the request. Using the APIKeyHeader and Security functions in FastAPI allows us to define the header name for our API key (and therefore populate this automatically in our OpenAPI documentation) and extract it from the header.

With the API key stored in api_key_header, we next call our check_api_key function (again, we’ll see how that works in a moment). If the response is True, we proceed to get some user data by calling get_user_from_api_key and then return it.

If check_api_key returned False, then the API key we checked wasn’t valid. So we use FastAPI’s HTTPException to return a suitable response, in this case a 401 error.

Now we know what has to happen for every path that depends on get_user. Before we look at those paths and routes, let’s quickly explore db.py, which is where we defined check_api_key and get_user_from_api_key.

db.py

This file is basically a placeholder for whatever backend database integration you want to use. Right now, it implements the methods we needs, and contains a hard-coded set of API keys and users.

api_keys = {
    "e54d4431-5dab-474e-b71a-0db1fcb9e659": "7oDYjo3d9r58EJKYi5x4E8",
    "5f0c7127-3be9-4488-b801-c7b6415b45e9": "mUP7PpTHmFAkxcQLWKMY8t"
}

users = {
    "7oDYjo3d9r58EJKYi5x4E8": {
        "name": "Bob"
    },
    "mUP7PpTHmFAkxcQLWKMY8t": {
        "name": "Alice"
    },
}

def check_api_key(api_key: str):
    return api_key in api_keys

def get_user_from_api_key(api_key: str):
    return users[api_keys[api_key]]

I’m sure I don’t need to tell you - don’t do this in production! Instead, you should incorporate whatever database connections and query methods you need to to acquire API keys and user details from your database (how you actually create those keys is another discussion). The actual data structures may be very different as well.

For now, we have a simple dict of key:value pairs for api_keys where the key is the actual API key, and the value is the user ID. I’m using long UUIDs for API keys, and short UUIDs for user IDs, but again this could be something very different. For the users we have another dict where the key is the user ID, and the value is another dict to hold all the important user data. Right now we’re just storing a name.

check_api_key simply checks that the API key exists in the list of keys it knows about. Again, you’d probably need to change this for some sort of DB lookup in production.

get_user_from_api_key then uses the API key to get a corresponding user ID, and returns the nested user dict.

So far we have all of our basic functionality in terms of API authentication. Now we just need some routes!

routers/public.py

from fastapi import APIRouter

router = APIRouter()

@router.get("/")
async def get_testroute():
    return "OK"

This is about as simple as it gets. In this module, all we do is define a router object, with a single path. When a request is made to /. it will return "OK". Don’t forget, the actual path is relative, and if you refer back to main.py you’ll see we set up this router to run under /api/v1/public.

routers/secure.py

from fastapi import APIRouter, Depends
from auth import get_user

router = APIRouter()

@router.get("/")
async def get_testroute(user: dict = Depends(get_user)):
    return user

The secure routes are only slightly more complex. Just like in main.py we’re using FastAPI’s Depends to specify that we are dependent on a function, in this case: get_user. So we know this route is guaranteed secure, and can only be accessed by someone with a valid API key, because without one you won’t get through the get_user function without triggering the HTTP Not Authorized error we set up earlier. If you have a valid key, for now we just return the user object.

Setup

This may vary based on your own development preferences, but the quickest way I find to set up a FastAPI app is to create a virtualenv, install the FastAPI package, and then use Uvicorn to run the app. You may have noticed the venv directory earlier - I created this with:

python -mvenv venv

I then activated the virtual environment and installed FastAPI:

source venv/bin/active
pip install "fastapi[standard]"

Finally, to run my FastAPI app, I can now run this command from the app directory:

uvicorn main:app --reload

The --reload option will reload my code automatically if I make any changes. If everything has worked for you, you should be able to access your app at http://127.0.0.1:8000/

Testing

First we sent a GET request with no API key to the public endpoint /api/v1/public/:

Now let’s try the secure endpoint at /api/v1/secure/:

The API returns a 403 error. Note, this isn’t the 401 error we coded for when an API key isn’t found in the database. FastAPI can’t even find an API key, because we haven’t specified one, so immediately returns the 403 Forbidden error.

Let’s try a random API key that is not in the database:

This time, we’ve supplied an X-API-Key header, but the key we’re supplying isn’t in the database. So we return a 401 Unauthorized error (technically, we’re unauthenticated but these error codes were written a long time ago!)

Finally, let’s try a working API key, for the user Bob:

Success! The API key is found, checked and proven valid, and the user details are retrieved and used in the response.

What’s next?

This is hopefully some helpful project boilerplate to get you started on your own FastAPI app. Don’t forget, we’ve hardcoded a dummy database in db.py, so connecting the authentication up to a real database should be the next thing we do. I’ll try to tackle this in the next post!

From there, we could continue to implement different routes and paths, which brings us onto the topics that come after authentication - authorisation and/or admission control.

Let me know what you think in the comments!

In the meantime, several people (about 230k in the last few days) have decided to seek out an alternative community micro-blogging platform, such as Mastodon, which is part of the Fediverse. It’s always difficult to turn your back on a platform after so many years, and learn new terminology and new ways of doing things. As someone who has now been using these things for just a few short days, I’d like to try to help anyone else who wants to make the leap!

The first thing to realise is that Mastodon is not a replacement for a Twitter experience. It’s a different, and I think most people will agree, better experience. To explain how it all works, let’s look at some of the individual components.

The Fediverse

The Fediverse is the name given to a collection of connected servers that can share things with each other. The Fediverse gets its name from being a universe of federation. Federation in this context means lots of connected servers, that can run different types of platforms, and that are not owned or managed by a single entity (see how it’s different already?). These servers might share videos, or posts, or something else depending on the software they are running, but they mostly all share information with each other using an open standard called ActivityPub.

Mastodon

Mastodon is one of the most popular Fediverse platforms because it provides a very feature-rich micro-blogging social networking platform not entirely unlike Twitter (but again - better! I’ll explain more on that in a moment). In Mastodon:

• You don’t tweet, you post

• You don’t “like” a post, you favourite it

• You don’t retweet, you boost a post

From that basic point of view, it’s a similar experience to Twitter. Notably, the ability to do something akin to “quote tweeting” is disabled on most Mastodon sites. This is to try to limit the negative behaviour that, it turns out, quote tweeting has been promoting all this time.

Instances

Anyone with a bit of know-how and some server resources can run their own Mastodon instance. But that doesn’t mean that you’re joining a micro-blogging site with just a handful of people, because all Mastodon instances can share their posts and users with the rest of the Fediverse. You can follow anyone you like (unless they have opted to prevent people from following them), and it doesn’t matter which instance they are using, because all of the information is federated. People can even move their accounts from one instance to another if they want to.

So if everything is decentralised and federated, where do these individual server instances come into it?

The Experience

This is the key thing that helps you build your own experience in Mastodon and the Fediverse. You can choose an instance run by and populated by like-minded people; and that doesn’t have to mean people who have exactly the same hobbies as you, they might just share the same sensibilities. Your instance admins will work hard to maintain the server, and provide rules on what sort of behaviour they expect. This allows you to join a community with rules you respect. You can easily discover people local to your server or of course explore the rest of the Fediverse, but it’s likely your server admin may restrict the sharing of information with instances that they feel go against their own rules.

So your experience is now based on like-minded communities, and exploring and building connections to other interesting people, while at the same time being protected from some of the more extreme elements of the Internet. And of course there are no adverts or corporate interests, so you’re no longer being manipulated by algorithms to keep you addicted to doom-scrolling. For those of us who’ve been around since the dawn of the Internet, it’s like a welcome return to the more community-focused, decentralised places we used to visit before we all became beholden to social media giants and their need for ad clicks.

How to Get Started

Even though Mastodon has been around for a while now, it’s still early days for its mass adoption, although I expect this to grow quickly with the exodus from Twitter. Right now though there aren’t that many user-friendly newbie guides for folks who aren’t already technically inclined.

The first thing you’ll need to do is choose an instance to join, and you should take your time to find the right base community for yourself. Don’t sweat it too much though, because you can move between instances in the future. Here’s some links to help get you started:

• instances.social is a curated list of currently available Mastodon instances, that also has a “wizard” to help you choose an instance.

• fedi.directory is another curated list, this time of interesting people to follow.

• fedi.tips contains lots of informal and unofficial guides for all aspects of using Mastodon. There’s a lot of information there, so it can be a bit overwhelming at first.

Once you’ve found your community, just start to explore and take your time. When you feel settled, you should consider supporting your instance admin in whatever way they deem appropriate.

For me, Twitter was always a way to keep up with tech news (and sci-fi, books, RPGs and all things Trek). But it was always hard work to remove the doom, gloom, hate and deliberately triggering click-bait. I’ve only been using Mastodon for a week or so, but it’s already the complete opposite of that experience, and as more people make the migration, I don’t feel like I’m missing any tech news either! (or sci-fi, books, RPGs etc.)

I hope this post has helped you! Feel free to follow me at @timberry@hachyderm.io

Once your app is containered up, deploying it to Google’s Cloud Run is really easy and gives you out-of-the-box auto-scaling and a secure HTTPS front-end. To follow along below, make sure you have the Google Cloud SDK set up on your local system.

First we’ll use a multi-stage docker build. In the directory of your Go web app, simply add the following Dockerfile:

FROM golang:1.13 as build
WORKDIR /go/src/app
COPY . .
RUN go build -v -o app .

FROM gcr.io/distroless/base
COPY --from=build /go/src/app/. /
CMD ["/app"]

In the first stage of this file, we use the golang:1.13 image to give us a build environment for our app. We copy everything from our local filesystem into the /go/src/app directory inside the container environment. Then we run go build to compile everything. Simple!

The next stage is the clever part. We start a new image from gcr.io/distroless/base and copy over just the files from our build stage (including our compiled runtime). In this example, we’re assuming there are supporting files to copy as well (for example, HTML and other static content), but we could refine this even more by just copying the binary application. Google’s distroless project contains just enough Linux to run our compiled binary. There’s no package manager, no shell, so it makes for a very efficient image.

You can build this image locally with Docker and push it to Google Container Registry, or just use Google’s Cloud Build to do the work for you:

gcloud builds submit --tag gcr.io//<your-project-id>/<your-image-name> .

Now you can deploy your app with just one more command:

gcloud run deploy <deployment-name> gcr.io/<your-project-id>/<your-image-name>

(Don’t forget to replace <your-project-id>, <your-image-name> and <deployment-name> in these examples)

In the Cloud Run console you should now see your new deployed service, complete with its URL.

Cloud Run offers some great features like traffic splitting and migration and custom domains. Deploying container based applications has never been easier! Except when things go wrong…

Missing libraries

You may have successfully tested your Docker image locally, but it’s failing when you deploy it. If you’re really unlucky, you may receive this error message in your docker logs:

exec user process caused "no such file or directory"

This error is almost completely useless, and will probably send you down the path of debugging the contents of your image looking for missing files or directories. What’s actually at fault is that there are system libraries missing that Go needs, because they are not part of your base image. This happens when Go’s statically compiled binaries aren’t quite as static as we’d like them to be.

To fix this, we need to send some extra parameters to the go build command so that it also compiles in any libraries it needs:

CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o app .

This should fix the problem most of the time.

But I need CGO!

The above fix works great as long as switching off CGO isn’t a problem. But some Go libraries require CGO (for example, the rather handy go-sqlite3). So to create a completely static binary while still allowing CGO, we have to update our build line again.

RUN go get -d -v ./...
RUN CGO_ENABLED=1 GOOS=linux go build -a -ldflags '-linkmode external -extldflags "-static"' -o app .

First we run go get to include any external dependencies. Then we send extra parameters to go build to make sure it really includes everything. This results in a much longer build process, but hopefully a static binary that works.

Hooray for write once, run almost everywhere! 😊

We’ve come a long way, from a simple Hello World to a fully functional dialog search engine for Star Trek: The Next Generation. This is quite an achievement! But how can we improve on it further? Wouldn’t it be nice to see some pictures with our search results? Picard’s shiny head, or Riker’s magical beard for example.

Another API

Unfortunately, the API we’re querying doesn’t contain any image data or URLs. However, it does contain the IMDB ID of the episode, which we can use to query a further API for some image data. To do this, we’re going to use The Movie Database (TMDb), a community driven movie database with a fantastic public API.

To use this API, you will need to sign up for a free TMDb developer account and create your own API key. Full instructions can be found here. Make a note of the key - we’ll use it in a moment.

More structs

We’re going to be passing multiple sources of data to our template, so we’ll need to change the way we’re using our structs. First, we’ll change the Dialog []struct. Rather than keep this struct as a slice, we’ll make it a singular struct (you’ll see why in a moment). To do this, just change its first line to:

type Dialog struct {

Now we’ll create a new type of struct, which will contain all of the results we want to send to our template. Enter this after the complete defintion for the Dialog struct:

type Results struct {
    SearchKey string
    Lines     []Dialog
    Images    map[string]string
}

As you can see, we’re now creating as field called Lines, which will be the slice of Dialogs. Inside this new struct, we also store our original SearchKey, and a new field called Images. This is a map, a series of key+value pairs. You may be familiar with maps from Java, or dictionaries from Python, which are essentially the same thing.

Next we need to create the struct for the TMDb API. Enter the following:

type TMDBQuery struct {
    MovieResults     []string `json:"-"`
    PersonResults    []string `json:"-"`
    TvResults        []string `json:"-"`
    TvEpisodeResults []struct {
        AirDate        string  `json:"air_date"`
        EpisodeNumber  int     `json:"episode_number"`
        ID             int     `json:"id"`
        Name           string  `json:"name"`
        Overview       string  `json:"overview"`
        ProductionCode string  `json:"production_code"`
        SeasonNumber   int     `json:"season_number"`
        ShowID         int     `json:"show_id"`
        StillPath      string  `json:"still_path"`
        VoteAverage    float64 `json:"vote_average"`
        VoteCount      int     `json:"vote_count"`
    } `json:"tv_episode_results"`
    TvSeasonResults string `json:"-"`
}

There’s quite a lot here but it’s easy to break down. First of all - ignore MovieResults, PersonResults and TvSeasonResults. The API we’re querying could potentially return results in these 3 keys, but we don’t want to use them - we know the IDs we are sending in our query will only return TvEpisodeResults. That’s why we are tagging them with json:"-". This tells the JSON Decoder to completely ignore them, and anything they might contain. Inside TvEpisodeResults we can see all the data we will scrape from this API, the most important part being StillPath, which will lead us to a public URL for a screenshot from the episode.

TvEpisodeResults is also a slice, as denoted by []struct, because the API could potentially return mutliple results in this field. However, as we’re sending the IMDB ID in our query, we can be sure that this slice will only contain a single entry.

Using the API key

Remember that API key you got from TMBb? Store it in a string variable now. Only joking! Of course, hard-coding API keys and other sensitive information inside your program is a terrible idea. Instead, just above the var tpl defintion, add this line:

var API_KEY string

We’re declaring the variable, but we haven’t stored anything in it yet. Jump down into your main() function and add this at the start of it:

API_KEY = os.Getenv("API_KEY")
if API_KEY == "" {
    fmt.Println("No API_KEY in environment")
    os.Exit(1)
}

os.Getenv grabs the API_KEY from the runtime environment. Then we check to make sure the variable isn’t empty - if it is, we quit with an error message.

On Linux and Mac systems, you can set an environment variable like this:

export API_KEY=EJ5RkgowmPfwSIra9EDqelQirMOSoyd6

(Obviously, replace the key with the one you’ve obtained. That’s not a real key!) I’ve never tried this with Windows, but I googled this for you :)

Updating the search handler

Next, we’ll make several changes to our searchHandler function. To start with, replace:

dialog = &Dialog{}

with:

results := &Results{}
results.Images = make(map[string]string)
results.SearchKey = searchKey

We’re now creating an instance of our new Results struct, which can store the search key, lines of dialog, and a map of images. We call make to set up our empty map, specifying that the keys and values will both be strings. Then we store the searchKey we retreived from the HTTP request as results.SearchKey.

The next few parts of our function stay the same. We still conctact the original API and check for errors or return codes that are not OK. But we’ll change the call to json.NewDecoder to:

err = json.NewDecoder(resp.Body).Decode(&results.Lines)

We’re now storing the results in Lines, a field of our results object. Leave the next piece of error checking in place. The next line should be spew.Dump(dialog). We’ll delete that because we no longer need this level of debug, and we no longer have a variable called dialog!

Now comes a rather large chunk of code to enter:

for _, d := range results.Lines {
    tmdbquery := &TMDBQuery{}
    imdbid := d.Imdbid
    _, ok := results.Images[imdbid]
    if !ok {
        tmdbep := fmt.Sprintf("https://api.themoviedb.org/3/find/%s?api_key=%s&language=en-US&external_source=imdb_id", imdbid, API_KEY)
        tmdbresp, err := http.Get(tmdbep)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        defer tmdbresp.Body.Close()
        if tmdbresp.StatusCode != 200 {
            w.WriteHeader(http.StatusInternalServerError)
            return
        }

        err = json.NewDecoder(tmdbresp.Body).Decode(&tmdbquery)

        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            return
        }
        if len(tmdbquery.TvEpisodeResults) > 0 {
            results.Images[imdbid] = tmdbquery.TvEpisodeResults[0].StillPath
        } else {
            results.Images[imdbid] = "8do6gZErem4wfdPwALiT8agtJfb.jpg"
        }
    }
}

Phew! Let’s go through some of that to clarify what it’s doing, although none of it is particularly complicated. This is our first for loop in Go. This is actually the only looping construct in Go - if you’re used to whiles or untils you’ll have to get used to doing everything with for.

Using range results.Lines will iterate through all the objects in the Lines slice, providing us with 2 variables at a time: the index of the slice, and the value itself. We don’t need the index, so we use the dummy variable _, and we store the value as d.

Next, we create an instance of TMDBQuery to store the results we’re about to get, and we grab the IMDB ID of our current result and store it as imdbid. In a moment we’ll start storing image URLs in the map we created earlier.

The next line looks a little odd: _, ok := results.Images[imdbid], but all this does is check to see if we already have an image in our map for this IMDB ID. If we do, we skip the next section entirely. This saves time and API calls. For example, if we queried the dialog “Darmok”, we’ll get a few dozen results, but they’ll all have the same IMDB ID (they’re all from the same episode!). So we only need to get the image URL once.

Inside the next block, we’re only proceeding if there’s no match in the map (if !ok). We prep the API URL as tmdbep using the IMDB ID and our API key. The next few chunks of code should look very familiar. In the same way as we queried the original API, we send a request, do some error checking on the response, then use the JSON Decoder to store the results in a struct.

Finally, we check to see how many results are in TvEpisodeResults. If there’s more than zero, we know we have a match (and only one match). So we can grab that index and its image URL: tmbdquery.TvEpisodeResults[0].StillPath.

If for some reason there was no match, we’ve failed to find an image for this episode. So instead, we fall back to a hard-coded URL, which is a lovely full cast shot of our starship crew.

One last line of code to change! We need to send the new struct to our template. So replace:

err = tpl.Execute(w, dialog)

with:

err = tpl.Execute(w, results)

Updating the template

We’re so close! Because we’ve changed the struct that’s being passed to our template, we need to update the template itself. First, add the following to the end of style.css:

.episode-image {
  width: 200px;
  flex-grow: 0;
  flex-shrink: 0;
  margin-left: 20px;
}

Now update the <section class="container"> part of the index.html file:

<section class="container">
  <ul class="search-results">
    {{ range .Lines }}
    <li class="dialog">
      <div>
        <h3 class="title"><a href="https://www.imdb.com/title/{{ .Imdbid }}/">{{ .Episodename }}</a></h3>
        {{ if eq .Texttype "speech" }}
        <p class="description">{{ .Who }}: <i>"{{ .Text }}"</i></p>
        {{ else }}
        <p class="description">{{ .Text }}</p>
        {{ end }}            
        <p>Act {{ .Act }} Scene {{ .Scenenumber }}</p>
        <p>Season {{ .Season }} Episode {{ .Episode }}</p>
      </div>
      <img class="episode-image" src="https://image.tmdb.org/t/p/w454_and_h254_bestv2/{{ index $.Images .Imdbid }}">
    </li>
    {{ end }}
  </ul>
</section>

The changes here are quite self-explanatory. Rather than range through a single object (as .) we use the .Lines field of the struct. To make things look a bit nicer, we’ve also removed the word “Episode” and again used the Imbdid to link to the actual IMDB page for an episode.

Then we add an image tag, using the start of the TMDB media URL, but appending a string we grab from the Images map using Imdbid as a key. Pretty neat, huh?

Now we’re finally done! Save everything and restart your Go server. I’m sure you know how by now :)

Results!

Conclusion

We’ve successfully built a reasonably complex web application in Go, that queries 2 different HTTP APIs and performs some pretty advanced template rendering.

Does this mean that we’re now at least basically competent in Go? For me, the answer is No. There are still plenty of concepts I’ve yet to grasp. But the difference for me is that now I’ve actually built something.

Before, I would look at a Go program and shudder. I would attempt the Tour of Go or Go by Example and just fall asleep mystified, as I had no context to apply to either of those rather verbose tutorials. Now I’ve seen a program work in the wild, and it all makes just a little bit more sense. It’s a solid place to start the continuing voyage of learning to Go!

For some next steps, I’d encourage you to try a couple of things on your own:

• Can you add a count to the number of results?

• Can you pre-populate the search box with the search key when you show results? Don’t forget it’s already in the struct being passed to the template.

If you’d like to try another step-by-step Go tutorial before jumping straight back into the comprehensive stuff, I’d also highly recommend Daniela Patruzalek’s Pac Go (a Pac Man clone written in Go). Dani was a huge inspiration for me in writing this series!

I really hope you’ve enjoyed following along. Let me know your thoughts, and share your own Go learning experience :)

If you need to check your code, here are the full gists: main.go, index.html and style.css.