JGroups clustering in AWS EKS using KUBE_PING with a Spring boot App

This article discusses how to setup a Spring boot application using JGroups clusters for pod-pod messaging in AWS EKS using KUBE-PING

JGroups (http://www.jgroups.org/) is a method by which nodes in a cluster (regular cluster) running java applications can discover and talk to each other. Notable features include node-node messaging and discovery of nodes joining or leaving the cluster.

In a normal setup, the nodes in the cluster can discover each other using a variety of methods such as TCP, UDP multicast and others. Whereas in Kubernetes, such systems might not work.

A bit of terminology — In a clustered application (in non-kubernetes world), a node will mean an instance of the application, whereas in Kubernetes world, the synonym would be a pod, where your application is running

In Kubernetes (K8s), each pod of your application will work on worker-nodes and the master node (or control-plane) in your Kubernetes cluster will know when each of the pods are starting up and this presents us with an opportunity to discover other pods. The project “Kubernetes discovery protocol for JGroups” (https://github.com/jgroups-extras/jgroups-kubernetes) makes it easy for application pods to discover each other.

Assumptions:

This article assumes that you have your AWS CLI (https://aws.amazon.com/cli/) working (i.e. your credentials are working). It also assumes that you have the following tools are installed and are accessible over your command prompt

1. Kubectl (https://kubernetes.io/docs/tasks/tools/) — a CLI tool to talk to your Kubernetes cluster
2. Helm package manager for Kubernetes (https://helm.sh/docs/intro/install/)
3. Helmfile (https://github.com/roboll/helmfile) — A wrapper over Helm Charts that enables easier management
4. Helm diff plugin (https://github.com/databus23/helm-diff)

This article also assumes that you are quite familiar creating Spring boot applications.

Minimal instructions are given herein to clean up the resources created and some of these resources will cost you money in your AWS account.

Setting up a working AWS EKS cluster

We will use a managed Kubernetes offering from Amazon called as the EKS (Elastic Kubernetes Service) (https://aws.amazon.com/eks/). The first step in setting up an EKS cluster is to setup a VPC (Virtual Private Cloud). You can follow the document at https://docs.aws.amazon.com/eks/latest/userguide/create-public-private-vpc.html to create a VPC that has both public and private subnets. Assume that this resulted in creation of the VPC with:

1. private subnets as subnet-ddddd, subnet-eeeee and
2. public subnets as subnet-fffff and subnet-ggggg

To create the EKS cluster itself, we will use the eksctl utility (https://eksctl.io/introduction/#installation). For this article, I have created a file called cluster.yml with the following content

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: jiju-test-eks-cluster
  region: us-west-1

vpc:
  subnets:
    private:
      us-west-1a: { id: subnet-ddddd }
      us-west-1c: { id: subnet-eeeee }
    public:
      us-west-1a: { id: subnet-fffff }
      us-west-1c: { id: subnet-ggggg }

nodeGroups:
  - name: ng-1
    instanceType: m5.large
    desiredCapacity: 3
    volumeSize: 80

Then to create the cluster using eksctl we used the command:

eksctl create cluster -f cluster.yml

Now we used the AWS CLI to configure our KUBECONFIG file.

aws eks update-kubeconfig --region us-west-1 --name jiju-test-eks-cluster

This ensures that our kubectl now points properly to our Kubernetes cluster.

Creating our JGroups Spring boot application

Consider the POJO “Chat” that represents a chat

import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;

import java.util.List;

@Getter
@Setter
@NoArgsConstructor
public class Chat {
    private String currentMessage;
    private String messageStatus;

    private List<String> previousMessages;
}

Next, lets write a simple REST controller for our project

import lombok.AllArgsConstructor;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.*;

import java.util.List;
import java.util.stream.Collectors;

@RequestMapping("/")
@Slf4j
@Controller
@AllArgsConstructor
public class ChatController {
    private Chatter chatter;

    @GetMapping("/chat")
    public String getChats(Model model){
        List<String> chats=chatter.getChats();

        Chat c= new Chat();
        c.setCurrentMessage("");
        c.setPreviousMessages(chats);
        c.setMessageStatus("");
        model.addAttribute("chat",c);
        return "chat";
    }

    @PostMapping("/chat")
    public String sendChat(@ModelAttribute Chat chat, Model model){
        int returnValue=chatter.sendMessage(chat.getCurrentMessage());
        String output = returnValue==0?"Message sent":"Failed to send message";
        chat.setMessageStatus(output);
        chat.setPreviousMessages(chatter.getChats());
        model.addAttribute("chat",chat);
        return "chat";
    }
}

The referenced class Chatter.java is as below

import lombok.extern.slf4j.Slf4j;
import org.jgroups.JChannel;
import org.jgroups.Message;
import org.jgroups.ReceiverAdapter;
import org.jgroups.View;
import org.jgroups.util.Util;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Component;

import javax.annotation.PostConstruct;
import java.io.*;
import java.util.LinkedList;
import java.util.List;

@Slf4j
@Component
public class Chatter extends ReceiverAdapter {
    JChannel channel;
    String userName=System.getProperty("user.name", "n/a");
    private final List<String> state = new LinkedList<>();

    @Value("${app.jgroups.config:jgroups-config.xml}")
    private String jGroupsConfig;

    @Value("${app.jgroups.cluster:chat-cluster}")
    private String clusterName;

    @PostConstruct
    public void init()  {
       try {
            channel = new JChannel(jGroupsConfig);
            channel.setReceiver(this);
            channel.connect(clusterName);
            channel.getState(null, 10000);
        } catch (Exception ex) {
            log.error("registering the channel in JMX failed: {}", ex);
        }
    }

    public void close() {
        channel.close();
    }

    public int sendMessage(String message) {
        Message msg = new Message(null, message);
        try {
           channel.send(msg);
        } catch (Exception e) {
            log.error("Failed: {}", e);
            return -1;
        }
        return 0;
    }

    public void viewAccepted(View newView) {
        log.info("** view: " + newView);
    }


    public void receive(Message msg) {
        String line = msg.getSrc() + ": " + msg.getObject();
        log.info("Received: " + line);
        synchronized (state) {
            state.add(line);
        }
    }

    public List<String> getChats(){
        return List.copyOf(state);
    }

    public void getState(OutputStream output) throws Exception {
        synchronized (state) {
            Util.objectToStream(state, new DataOutputStream(output));
        }
    }

    public void setState(InputStream input) throws Exception {
        List<String> list;
        list = Util.objectFromStream(new DataInputStream(input));

        synchronized (state) {
            state.clear();
            state.addAll(list);
        }        
    }
}

And the spring boot startup class is as follows:

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class ChatMain {
    public static void main(String[] args) {
        SpringApplication.run(ChatMain.class,args);
    }
}

The application.yml file for the application is as below. The location of this file is under resources folder.

app:
  jgroups:
    config: jgroups-config.xml
    cluster: simple-chat

The meat of this application is in the resources/jgroups-config.yml. The contents of this file is given below:

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="urn:org:jgroups"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
    <TCP
            external_addr="${JGROUPS_EXTERNAL_ADDR:match-interface:eth0}"
            bind_addr="site_local,match-interface:eth0"
            bind_port="${TCP_PORT:7800}"

            recv_buf_size="5M"
            send_buf_size="1M"
            max_bundle_size="64K"
            enable_diagnostics="true"
            thread_naming_pattern="cl"

            thread_pool.min_threads="0"
            thread_pool.max_threads="500"
            thread_pool.keep_alive_time="30000" />

    <org.jgroups.protocols.kubernetes.KUBE_PING
                          namespace="${NAMESPACE}" />

    <MERGE3 max_interval="30000"
            min_interval="10000"/>
    <FD_SOCK external_addr="${JGROUPS_EXTERNAL_ADDR}"
             start_port="${FD_SOCK_PORT:9000}"/>
    <FD_ALL timeout="30000" interval="5000"/>
    <VERIFY_SUSPECT timeout="1500"  />
    <BARRIER />
    <pbcast.NAKACK2 xmit_interval="500"
                    xmit_table_num_rows="100"
                    xmit_table_msgs_per_row="2000"
                    xmit_table_max_compaction_time="30000"
                    use_mcast_xmit="false"
                    discard_delivered_msgs="true" />
    <UNICAST3
            xmit_table_num_rows="100"
            xmit_table_msgs_per_row="1000"
            xmit_table_max_compaction_time="30000"/>
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="8m"/>
    <pbcast.GMS print_local_addr="true" join_timeout="3000"
                view_bundling="true"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60K"  />
    <pbcast.STATE_TRANSFER  />
    <CENTRAL_LOCK />
    <COUNTER/>
</config>

The most important part in the above jgroups config file is the configuration of the KUBE_PING as shown below:

<org.jgroups.protocols.kubernetes.KUBE_PING
                          namespace="${NAMESPACE}" />

The namespace of the application is controlled by the environment variable NAMESPACE. The portion of the TCP only exists if you want to test your application in a local environment without using Kubernetes first.

We have also thrown a thymeleaf based simple UI over this application. The contents of they resources/templates/chat.html is as below.

<!DOCTYPE HTML>
<html xmlns:th="https://www.thymeleaf.org">
<head>
    <title>Chat Application</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <link th:href="@{/styles/main.css}" rel="stylesheet" />
</head>
<body>
<h2>Chat over Jgroups</h2>
<br/>
<form action="#" th:action="@{/chat}" th:object="${chat}" method="post">
    <p class="center">Type your message: <input type="text" th:field="*{currentMessage}" /></p>
    <p class="center"><input type="submit" value="Send Chat" /> </p>
    <br/>
    <br/>
    <label class="statuslabel" th:text="'Previous Message Status : '+*{messageStatus}"></label>
    <p>Previous chats: </p>
    <div th:if="*{previousMessages != null}">
        <ul>
            <li th:each="m : *{previousMessages}" th:text="${m}"/>
        </ul>
    </div>
</form>
</body>
</html>

A simplistic style sheet at resources/static/styles/main.css is as

h2 {
    font-family: sans-serif;
    font-size: 1.5em;
    text-transform: uppercase;
    text-align: center;
}

p {
    font-family: sans-serif;
}

p.center {
    font-family: sans-serif;
}

body {
    background-color: papayawhip;
}

label.statuslabel {
    font-size: 80%;
    color: crimson;
}

The application UI looks like below (Its a very simple UI to demonstrate the concept, nothing fancy)

You can compile your application with either Maven or Gradle to create a Spring bootable jar. A sample gradle dependencies section is as shown below

dependencies {
    implementation 'org.jgroups:jgroups:4.0.15.Final'
    implementation 'org.jgroups.kubernetes:jgroups-kubernetes:1.0.16.Final'

    implementation 'org.springframework.boot:spring-boot-starter-actuator:2.6.0'
    implementation 'org.springframework.boot:spring-boot-starter-web:2.6.0'
    implementation 'org.springframework.boot:spring-boot-starter-logging:2.6.0'
    implementation 'org.springframework.boot:spring-boot-starter-thymeleaf:2.6.0'
    compileOnly 'org.projectlombok:lombok:1.18.22'
    annotationProcessor 'org.projectlombok:lombok:1.18.22'
}

Ship your application as a docker image and host the compiled image on any image repository such as Dockerhub or AWS ECR Repository.

Creating the Helm chart and Helmfile

We will now create a helm chart that will be deployed in our Kubernetes cluster

The helm chart contains of several templates. Lets start with deployment.yaml that will deploy the application

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jgroups-poc
  labels:
    app: jgroups-poc
spec:
  replicas: 3
  selector:
    matchLabels:
      app: jgroups-poc
  template:
    metadata:
      labels:
        app: jgroups-poc
    spec:
      serviceAccountName: jgroups-kubeping-service-account
      containers:
      - name: jgroups-poc-container
        image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        env:
          - name: JGROUPS_EXTERNAL_ADDR
            value: "match-interface:eth0"
          - name: TCP_PORT
            value: "7800"
          - name: NAMESPACE
            value: {{ .Values.namespace }}
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - containerPort: 7800
          name: k8sping-port

You will notice that the NAMESPACE value is being fed into the container as an environment variable from the chart. The chart also refers to a service-account. This service account is used to run the container and should have permissions to list and view pods. You will also notice that we have 3 replicas of this deployment, i.e. we will have 3 pods that are launched.

The service account is created by the following template file

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jgroups-kubeping-service-account
  namespace: {{ .Values.namespace }}

The cluster role and the cluster-rolebinding are created using the following template files for the helm chart

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jgroups-kubeping-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

and

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: jgroups-kubeping-api-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: jgroups-kubeping-pod-reader
subjects:
  - kind: ServiceAccount
    name: jgroups-kubeping-service-account
    namespace: {{ .Values.namespace }}

The Chart.yaml and the default values.yaml for the chart are as follows:

apiVersion: v2
name: jgroups-kubernetes
description: A Helm chart for Jgroups demo on Kubernetes
type: application
version: 1.0.0
appVersion: "1.0.0"

and

image:
  repository: public.ecr.aws/xxxx/yyyy
  pullPolicy: IfNotPresent
  tag: "latest"
namespace: jgroups

It assumes that the image is already published to public.ecr.aws/xxxx/yyyy with the latest tag.

We will use a helmfile to wrap this helm chart and deploy this to Kubernetes. The helmfile.yaml file is as below.

repositories:
  - name: stable
    url: https://charts.helm.sh/stable

helmDefaults:
  tillerless: true
  verify: false 
  wait: true
  timeout: 300

releases:
  - name: jgroups
    chart: ./charts/jgroups-kubernetes
    version: 1.0.0
    namespace: jgroups
    installed: true
    values:
    - image:
        tag: "latest"
        repository: "public.ecr.aws/xxxx/yyyy"
      namespace: jgroups

You can deploy the chart using helmfile command to the Kubernetes cluster by using

helmfile -f helmfile.yaml apply

This will ensure that chart is deployed and 3 pods are created in the jgroups namespace.

Verification

We will use port forwarding to point to different pods and issue chat messages to one pod and ensure that the other pod is now able to get the message. To list the pods

kubectl get pods -n jgroups

Lets say this lists pods with name pod1, pod2, pod3. Then we can use the following to setup port-forwarding from our local machine to the pods.

kubectl port-forward -n jgroups pod1 9090:8080

In the above command, we are mapping local port 9090 to the pod1’s 8080 port.

Similarly, we map the local port 9091 to pod2’s port 8080

kubectl port-forward -n jgroups 9091:8080

Now point your browser window1 to “http://localhost:9090/chat” and window2 to “http://localhost:9091/chat”. In the window1, type in a message and click on “Send Chat” button. On window2, click refresh. And you will find the message typed on window1 appearing on window2. Window1 was pointing to pod1. The message sent to pod1 was replicated to pod2 via Jgroups. Hurray!!

Cleaning up

The fastest way to clean up the cluster is to use the eksctl command to delete the EKS cluster.

Acknowledgements

I would like to acknowledge the following articles with help in my journey to get jgroups working.

Run JGroups on Kubernetes

The JGroups tutorial