Bravery Family

Setup ETCD Cluster

Written by

GoldKeeperSnr

in

Follow-up

So, I’ll try to write up some notes, but the long and short of this is – Registrator doesn’t work. Having managed to setup the ETCD cluster I began to expand it with Registrator – and immediately hit various problems. FWIW, Registrator does function and can write to an ETCD cluster.

Registrator isn’t being developed properly anymore. This means it’s has drifted away from supporting etcd properly – and etcd has changed quite a bit recently. There is an updated version of the API that should be used to interact with it and guess what v2 and v3 are incompatible. The API stores the data in two places. And wouldn’t you know it, Registrator talks v2 and Traefik talks v3. So, it’s now back to the drawing board.

Looks like consul is the way to go, tbh.

Pre-amble

Continuing the setup of traefik, I want to setup an ETCD Cluster to allow the two Docker instances to be able to register changes and have traefik pick those up and push to the internet/intranet as appropriate.

This has been driven by a discovery – traefik will only work with a Docker provider once. My original thought process was to setup a Docker ssh connection from Traefik. So, I’d setup a user on the remote Docker instance, configured it to allow ssh, and fought with a weird location for the docker command. I could set a DOCKER_HOST environment variable, run docker info and see the remote machine setup. However, after added the duplicate provider into the traefic.yaml I discovered the fatal flaw – no duplicate providers.

Having no real desire to hard code everything all the time, it was back to a drawing board.

The original source of this idea was this page:

https://technologyconversations.com/2015/09/08/service-discovery-zookeeper-vs-etcd-vs-consul

I had toyed with Consul before – but ran into an issue with DNS, which resulted in my dumping it. I had been hunting around how to maintain the etcd repository – there didn’t seem to be many formally documented software solutions that I could see. However, this page also talks about using Registrator – which solves the other side of the problem I was seeing too.

Setup ETC Cluster

That means I have a potential solution, and if I get this working properly, I can think about Consul again in the future.

To make this work, I need etcd clustering, which lead to here:

https://gist.github.com/kanwar-saad/b54a728fa872a767c037a5b8dc7f6e75

Although some of the cluster setup has also come from here in the end:

https://etcd.io/docs/v3.5/tutorials/how-to-setup-cluster/

This is mainly because I wanted to setup the etcd services in Portainer, and I wanted them to be stacks – which means formulating a compose file.

Couple to that the fact that I didn’t want to use a HELM image (I couldn’t be bothered to make another account). Which meant making a Bitnami image work.

So, this is my docker compose file. Obviously, alter IP address and names as needed:

version: '3'

services:
  etcd:
    image: bitnami/etcd:latest
    restart: unless-stopped
    container_name: etcd
    ports:
      - "2379:2379"
      - "2380:2380"
    volumes:
      - etcd-data:/bitnami/etcd
    environment:
      ETCD_ADVERTISE_CLIENT_URLS: http://<host ip address>:2379
      ETCD_INITIAL_ADVERTISE_PEER_URLS: http://<host ip address>:2380
      ETCD_INITIAL_CLUSTER: etcd1=http://<host ip address>:2380,etcd2=http://<other host ip address>:2380
      ETCD_INITIAL_CLUSTER_STATE: new
      ETCD_INITIAL_CLUSTER_TOKEN: token-01
      ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
      ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
      ETCD_NAME: etcd<id number of cluster member>
      ALLOW_NONE_AUTHENTICATION: yes
    labels:
      - "traefik.enable=false"

volumes:
  etcd-data:

Starting this up on both docker nodes has resulted in a stable cluster, that I can connect to from my external PC using etcdctl – and seems quite stable when I take a node away (i.e. it still answers questions).

More posts