Installing Gemfire on TKG

2021-01-29 · 11 min read

Gemfire

The k8s version of Gemfire was released, so I tried installing it.

Environment

Tested in the following environment:

TKG v1.18.10+vmware.1-tkg.1.3a6cd48 (via TMC)
Gemfire for k8s : 1.0.0

Steps

Here's a quick walkthrough. Always check the latest manual:

https://tgf.docs.pivotal.io/tgf/1-0/work-with-cluster.html

Download the binaries

Download from Tanzu Network. Only the Tanzu Gemfire Operator seems to be needed.

https://network.pivotal.io/products/tanzu-gemfire-for-kubernetes

Install Cert Manager

As an undocumented prerequisite, Cert Manager must be installed.

First, when using TKGs and the like, the following RoleBindings are required beforehand:

kubectl create rolebinding -n cert-manager cert-manager-role-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated

Download VMware Tanzu Kubernetes Grid Extensions Manifest 1.2.0 from:

https://www.vmware.com/go/get-tkg

After extracting, install cert manager with:

cd tkg-extensions-v1.2.0+vmware.1/cert-manager
kubectl apply -f ./

Then confirm cert-manager is installed:

kubectl api-resources | grep cert-manager.io
challenges                                     acme.cert-manager.io                         true         Challenge
orders                                         acme.cert-manager.io                         true         Order
certificaterequests               cr,crs       cert-manager.io                              true         CertificateRequest
certificates                      cert,certs   cert-manager.io                              true         Certificate
clusterissuers                                 cert-manager.io                              false        ClusterIssuer
issuers                                        cert-manager.io                              true         Issuer

Note that this release supports API versions up to cert-manager.io/v1beta1:

kubectl api-versions | grep cert-manager.io
acme.cert-manager.io/v1alpha2
acme.cert-manager.io/v1alpha3
acme.cert-manager.io/v1beta1
cert-manager.io/v1alpha2
cert-manager.io/v1alpha3
cert-manager.io/v1beta1

Install the Gemfire Operator

First install the Gemfire Operator. When using TKGs and the like, first create a Role Binding like this:

kubectl create namespace gemfire-system
kubectl create rolebinding -n gemfire-system gemfire-system-role-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated

Next, set the registry password Gemfire uses:

kubectl create secret docker-registry image-pull-secret --namespace=gemfire-system --docker-server=registry.pivotal.io --docker-username='USERNAME' --docker-password='PASSWD'

Now, on to installing the Operator... or so we'd like, but at the time of writing the Helm chart uses cert-manager/v1, while — as mentioned above — installing from the TKG Extensions only supports API versions up to cert-manager/v1beta1. So we have no choice but to install while patching part of the helm chart.

We use these tools:

YTT
kapp

After installing them on your workstation, prepare an update-cert-manager.yaml like this:

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.subset({"apiVersion":"cert-manager.io/v1"}), expects="1+"
---
apiVersion: cert-manager.io/v1beta1

Then run helm template > patch the values > deploy with kapp, all in one go:

helm template geode-cluster-operator ~/Downloads/gemfire-operator-1.0.0.tgz --namespace gemfire-system | ytt --ignore-unknown-comments -f- -f update-cert-manager.yaml | kapp deploy -f- -n gemfire-system -a geode-cluster-operator -y

If it works, it installs like this:

kubectl get po -n gemfire-system
NAME                                                         READY   STATUS    RESTARTS   AGE
geode-cluster-operator-controller-manager-7c8f54f4c5-nl2cz   2/2     Running   0          78m

Install the Gemfire cluster

With the operator installed, build the cluster. Once again, for TKGs, create these RoleBindings first:

kubectl create namespace gemfire-cluster
kubectl create rolebinding -n gemfire-cluster gemfire-cluster-role-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated

And register the registry password again:

kubectl create secret docker-registry image-pull-secret --namespace=gemfire-cluster --docker-server=registry.pivotal.io --docker-username='USERNAME' --docker-password='PASSWD'

Then just try creating a cluster:

cat <<EOF | kubectl -n gemfire-cluster apply -f -
apiVersion: gemfire.tanzu.vmware.com/v1
kind: GemFireCluster
metadata:
  name: gemfire1
spec:
  image: registry.pivotal.io/tanzu-gemfire-for-kubernetes/gemfire-k8s:1.0.0
EOF

After a while, the service is installed like this:

kubectl -n gemfire-cluster get GemFireCluster
NAME       LOCATORS   SERVERS
gemfire1   1/1        2/2

Load balancer for the API

Create the load balancer used for API access:

cat <<EOF | kubectl apply -f-
apiVersion: v1
kind: Service
metadata:
 name: gemfire-mgm-api
 namespace: gemfire-cluster
spec:
 selector:
   app: gemfire1-locator
 ports:
   - name: management
     port: 7070
     targetPort: 7070
 sessionAffinity: ClientIP
 sessionAffinityConfig:
     clientIP:
       timeoutSeconds: 10800
 type: LoadBalancer
EOF

Load balancer for DEV

Create the DEV load balancer. First, edit the GemFireCluster:

kubectl edit gemfirecluster gemfire1 -n gemfire-cluster

Then update the following under spec:

servers:
  overrides:
    gemfireProperties:
      start-dev-rest-api: "true"

Then set up the load balancer as below:

cat <<EOF | kubectl apply -f-
apiVersion: v1
kind: Service
metadata:
  name: gemfire-dev-api
  namespace: gemfire-cluster
spec:
  selector:
    app: gemfire1-server
  ports:
    - name: rest-api
      port: 7070
      targetPort: 7070
  type: LoadBalancer
EOF

Verify the installation

Finally, verify the installation. First, use kubectl to confirm gfsh is accessible:

kubectl -n gemfire-cluster exec -it gemfire1-locator-0 -- gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  /
 / /__/ / ____/  _____/ / /    / /
/______/_/      /______/_/    /_/    1.13.1

Monitor and Manage Apache Geode
gfsh>
gfsh>
gfsh>
gfsh>exit

No problem. Next, confirm access through the load balancer. Change --url for your environment.

    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  /
 / /__/ / ____/  _____/ / /    / /
/______/_/      /______/_/    /_/    9.10.5

Monitor and Manage VMware Tanzu GemFire
gfsh>connect --locator --use-http=true --url=http://10.195.35.80:7070/gemfire/v1
Successfully connected to: GemFire Manager HTTP service @ http://10.195.35.80:7070/gemfire/v1

You are connected to a cluster of version: 1.13.1

No problem. Finally, confirm information can be retrieved via gfsh:

gfsh>list members
Member Count : 3

       Name        | Id
------------------ | ---------------------------------------------------------------------
gemfire1-locator-0 | 172.20.1.49(gemfire1-locator-0:1:locator)<ec><v0>:41000 [Coordinator]
gemfire1-server-1  | 172.20.1.55(gemfire1-server-1:1)<v4>:41000
gemfire1-server-0  | 172.20.1.56(gemfire1-server-0:1)<v6>:41000

gfsh>

OK.

Summary

Gemfire on K8S was very easy to set up.