Tag: Traefik

  • Install Traefik Proxy on Debian.

    This post is intended to cover how to install the Traefik Proxy on a Debian server. The process is a little complex as, unfortunately, there isn’t a .deb file that I could find.

    This means a bit more manual fiddling about.

    Logo for Traefik Proxy

    First off, some of the steps here come from this blog:
    https://blog.emka.web.id/2022/09/how-to-install-traefik-in-debian.html
    and the Traefik site:
    https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-binary-distribution

    I’ve adjusted the steps to cover my own needs.

    Preparation

    The installation of Traefik proxy on Debian requires some setup task.

    Create a group for Traefik – obviously check the group ID is unique.

    sudo groupadd -g 321 traefik

    Then a user – obviously check the group ID is unique:

    sudo useradd \
     -g traefik --no-user-group \
     --home-dir /var/www --no-create-home \
     --shell /usr/sbin/nologin \
     --system --uid 321 traefik

    Create necessary directories for the software.

    mkdir /opt/traefik
    mkdir /etc/traefik
    mkdir /var/opt/traefik

    I’m not fiddling about with directory permissions yet – as I am trying to avoid having to run everything through sudo.

    Now download a copy of Traefik Proxy software. That’s done by getting an appropriate file from here:
    https://github.com/traefik/traefik/releases

    Finally fetch a copy of the service file from here:
    https://github.com/traefik/traefik/raw/master/contrib/systemd/traefik.service

    Install

    Install Traefik Proxy Service

    Update the traefik.service file:

    vi traefik.service

    Removed the comments from:

    • Within [Unit]
      • After
      • AssertFileIsExecutable
        Fixing the path to the executable.
      • AssertPathExists
        Fixing the file to be traefik.yaml
    • Within [Service]
      • User
      • AmbientCapabilities
      • ExecStart
        Fixing the executable path and making the configFile parameter point to traefik.yaml file.

    Then transfer the service file to the correct location:

    sudo mv traefik.service /etc/system/system/
    sudo chown root: root /etc/systemd/system/traefik.service
    sudo chmod 644 /etc/systemd/system/traefik.service

    Then update systemd:

    systemctl daemon-reload

    This will not be use under a functional configuration is in place.

    Starter Configuration

    Static Configuration

    Now copy the below starter configuration and place it in /etc/traefik/traefik.yaml

    ################################################################
    #
    # Configuration sample for Traefik v2.
    #
    # For Traefik v1: 
    # https://github.com/traefik/traefik/blob/v1.7/traefik.sample.toml
    #
    ################################################################
    
    ################################################################
    # Global configuration
    ################################################################
    global:
      checkNewVersion: true
      sendAnonymousUsage: true
    ################################################################
    # EntryPoints configuration
    ################################################################
    entryPoints:
      web:
        address: :85
      websecure:
        address: :448
    ################################################################
    # Traefik logs configuration
    ################################################################
    log:
      level: DEBUG
    ################################################################
    # API and dashboard configuration
    ################################################################
    # Enable API and dashboard
    api:
      dashboard: true
    
    providers:
      file:
        filename: /etc/traefik/dynamic.yaml

    In short, this file will:

    • Make traefik listen on port 85 and 448
    • Increase logging to debug level
      So, we can see what is going on
    • Enable the API and the Dashboard
    • Configure a file provider call dynamic.yaml

    Dynamic Configuration

    Now create an authentication user (replace the content of the angle brackets) – you will need the output from this in the next step.

    htpasswd -nb <user> <password>

    Follow this up with placing this in /etc/traefik/dynamic.yaml replacing <string from htpasswd> with your string from htpasswd (yes, put it inside the double quotes).

    # dynamic.yaml
    
    http:
      routers:
        api:
          rule: PathPrefix(`/api`) || PathPrefix(`/dashboard`)
          service: api@internal
          entryPoints:
            - "web"
          middlewares:
            - auth
        catchall:
          # attached only to web entryPoint
          entryPoints:
            - "web"
          # catchall rule
          rule: "PathPrefix(`/`)"
          service: unavailable
          # lowest possible priority
          # evaluated when no other router is matched
          priority: 1
      middlewares:
        auth:
          basicAuth:
            users:
              - "<string from htpasswd>"
      services:
        # Service that will always answer a 503 Service Unavailable response
        unavailable:
          loadBalancer:
            servers: {}

    Yes, the back ticks are correct, single quote will cause a problem due to go.

    This file sets up the following:

    • Two routers
      1. API
        • catch traffic coming in from the “web” entry point (85)
        • protects using the auth middleware
        • looks for /api and /dashboard traffic
        • will then route trafic to api@internal service
      2. CatchAll
        • catch traffic coming in from the “web” entry point (85)
        • looks for anything under /
        • will route traffic to unavailable service.
    • Middleware
      To protect the api and dashboard endpoints from prying eyes.
    • Service
      To show when things are broken.

    Install Traefik Proxy

    The next step of the install of Traefik Proxy on Debian is to extract the downloaded zip file:

    cd /opt/traefik
    tar -zxvf <download location>/traefik_v<version>_<platform>.tar.gz

    Since traefik proxy will be using privileged ports, it therefore needs a permission setting:

    sudo setcap 'cap_net_bind_service=+ep' traefik

    Testing Traefik Proxy

    This is where things begin to get fun. I had some fun and games with getting the Dashboard to work initially, hence why I am writing this and posting it for posterity.

    You MUST have the PathPrefix rule to make Dashboard and API endpoints work right.

    On the plus side, its presence in dynamic.yaml means we can fiddle about with the file and Traefik Proxy will just reload it.

    Now, simply run the Traefik Proxy executable – it looks in standard locations for configuration and /etc/traefik/traefik.yaml is one of those.

    ./traefik

    If it works you should see something like this:

    
    
    
    
    

    You should also be able to access various URLs.

    api/endpoints

    dashboard

    anything else

    Assuming everything is working we can now go about locking the software installation, enabling the services configuration that was setup, and linking it to the Docker.

    Troubleshooting

    Hopefully, Traefic Proxy is working for you. If it isn’t the terminal that Traefic is running in should give you some indication of what has gone wrong.

    It is likely to be related to listening on the selected ports. The simplest option would then be to move to a different port by altering the values for address in /etc/traefik/traefik.yaml.

    Having done this, stop traefik [if needed] and re-run it.

  • Renew LetsEncrypt in Traefik

    I’d setup Traefik to use LetsEncrypt, but due to the scarey words around rate limits etc, I felt it sensible to use the staging LetsEncrypt servers, rather than the production ones – basically I wasn’t sure how many changes I’d have to make as it has been a while since I played with any of this in anger. I assumed that Traefik would renew the certificates with LetsEncrypt when I needed production ones.

    I surprisingly got this working quite straightforwardly – I had more issues with getting Apache and WordPress (mainly Apache) playing nicely with everything.

    However, the staging certificates result in a warning which I didn’t want any longer than necessary. Since it now was working, I altered my certificate resolver in Traefik to point to the production LetsEncrypt server. I then performed a bounce to pick up the changes, expecting to see a new certificate and no more errors.

    Anxious pause…

    I waited a bit (pressing F5) and was disappointed to see that it hadn’t changed. Same certificate was in use, and I could see in the log that no attempt had been made to fetch a new certificate.

    Frantic prodding ensued as I attempted to work out what I had done wrong. Nothing was the answer. The problem, such as it is, is that Traefik already has a certificate that it can use. Traefik doesn’t feel it needs to change it the certificate.

    Solution?

    A quick google later found me a couple of links:

    Whilst the last link on the face of it initially seemed less relevant (it related to fixing a security flaw), on reading it gave the simplest answer for my situation.

    In my situation, it looked like I could simply delete the storage acme.json file. No certificates needed to be kept – I could just start again from scratch.

    I stopped Traefik, moved acme.json out of the way (belts and braces right), and then restarted it.

    Traefik Renew from LetsEncypt!

    Hey, Presto! I could see in the log file the lack of certificate being identified and Traefik calls to renew certificates from the production LetsEncrypt server.

    Padlock look right, everything is gravy! Let’s move on to the next thing.

  • Traefik Log Rotation

    An explanation of how to set up Traefik Log Rotation. This covers both the access log AND the traefik log.

    I initially setup simple log rotation using the standard logrotate package i have installed for everything else. I implemented this by following this page (there are others about) but this covered my scenario the best (as I have traefik running natively, not in a container.

    The main change I made was in the directory (mine is /var/log/traefik/) and then the files (I have an access.log and a traefik.log, so I altered the pattern to be *.log. I also added size – more on that in a minute.

    This resulted in a file /etc/logrotate.d/traefik below:

    <directory location>/*.log {
      compress
      create 0640 <user> <group>
      daily
      delaycompress
      missingok
      notifempty
      rotate 5
      size 10M
    
      postrotate
        kill -USR1 `pgrep traefik`
      endscript
    }

    However, I began to spot that it did not working as anticipated.

    • access.log wouldn’t rotate
    • traefik.log rotated but still wrote to the rotated our file

    It turns out access.log not rotating was because I misunderstood what size would do. It’s not an either/or with the daily setting – size supersedes daily if it is in the file. Simply removing size made access.log rotate as expect.

    However, traefik.log still won’t rotate on a USR1 – which seems contrary to their own documentation (link).

    I decided that something weird must be going on, so I dug about a bit. If you check the code in git (link) you’ll see that the code specifically mentions rotation of the access log, and their tests also only check the access log.

    This means that USR1 signal only rotates the access log. That was annoying.

    I wondered if I was missing something, so I went in search of rotate in git (link) and found something that whilst documented at source level seems to be missing in the Traefik log documentation (link).

    It turns out that Traefik can do some quite clever stuff with rotating its own log if you configure it right.

    log:
      # Log level
      #
      # Optional
      # Default: "ERROR"
      #
      level: <whatever level you want>
    
      # Sets the filepath for the traefik log. If not specified, stdout will be used.
      # Intermediate directories are created if necessary.
      #
      # Optional
      # Default: os.Stdout
      #
      filePath: <directory location>/<file>.log
    
      # Format is either "json" or "common".
      #
      # Optional
      # Default: "common"
      #
      #format: json
      maxSize:    5
      maxBackups: 50
      maxAge:     10
      compress:   true

    This will rotate at 5Mb, compress the old copies, and keep for 10 days and keep a maximum of 50 files. There are other options in the document – I have not used them.

    This then means I can make the logrotate configuration focus just on the access log:

    <directory location>/access.log {
      compress
      create 0640 <user> <group>
      daily
      delaycompress
      missingok
      notifempty
      rotate 5
    
      postrotate
        kill -USR1 `pgrep traefik`
      endscript
    }

    Combining these two pieces of information (access log rotation and traefik log rotation) together gives me a functional solution.

    -rw-r----- 1 traefik traefik   706383 Sep 15 13:46 access.log
    -rw-r----- 1 traefik traefik  2071725 Sep 14 23:59 access.log.1
    -rw-r----- 1 traefik traefik   101616 Sep 13 23:59 access.log.2.gz
    -rw-r----- 1 traefik traefik    43272 Sep 12 23:59 access.log.3.gz
    -rw-rw-r-- 1 traefik traefik   574683 Sep 12 12:29 access.log.4.gz
    -rw-r----- 1 traefik traefik       23 Sep 10 02:11 traefik-2023-09-10T01-11-42.862.log.gz
    -rw-r----- 1 traefik traefik    62635 Sep 12 20:48 traefik-2023-09-12T19-48-36.125.log.gz
    -rw-r----- 1 traefik traefik    45843 Sep 12 23:48 traefik-2023-09-12T22-48-51.210.log.gz
    -rw-r----- 1 traefik traefik    47196 Sep 13 02:50 traefik-2023-09-13T01-50-47.804.log.gz
    -rw-r----- 1 traefik traefik    46280 Sep 13 05:51 traefik-2023-09-13T04-51-02.691.log.gz
    -rw-r----- 1 traefik traefik    59591 Sep 13 08:44 traefik-2023-09-13T07-44-32.782.log.gz
    -rw-r----- 1 traefik traefik    51714 Sep 13 11:35 traefik-2023-09-13T10-35-02.705.log.gz
    -rw-r----- 1 traefik traefik    62231 Sep 13 14:31 traefik-2023-09-13T13-31-29.208.log.gz
    -rw-r----- 1 traefik traefik    47698 Sep 13 17:30 traefik-2023-09-13T16-30-44.233.log.gz
    -rw-r----- 1 traefik traefik   114460 Sep 13 20:00 traefik-2023-09-13T19-00-49.560.log.gz
    -rw-r----- 1 traefik traefik    88101 Sep 13 22:45 traefik-2023-09-13T21-45-27.953.log.gz
    -rw-r----- 1 traefik traefik    45687 Sep 14 01:48 traefik-2023-09-14T00-48-21.418.log.gz
    -rw-r----- 1 traefik traefik    48091 Sep 14 04:48 traefik-2023-09-14T03-48-36.486.log.gz
    -rw-r----- 1 traefik traefik    45339 Sep 14 07:49 traefik-2023-09-14T06-49-32.199.log.gz
    -rw-r----- 1 traefik traefik    58615 Sep 14 10:29 traefik-2023-09-14T09-29-32.183.log.gz
    -rw-r----- 1 traefik traefik    50523 Sep 14 13:24 traefik-2023-09-14T12-24-47.288.log.gz
    -rw-r----- 1 traefik traefik   111852 Sep 14 15:48 traefik-2023-09-14T14-48-17.251.log.gz
    -rw-r----- 1 traefik traefik   102566 Sep 14 18:19 traefik-2023-09-14T17-19-54.417.log.gz
    -rw-r----- 1 traefik traefik    49430 Sep 14 21:08 traefik-2023-09-14T20-08-39.426.log.gz
    -rw-r----- 1 traefik traefik    52971 Sep 14 23:53 traefik-2023-09-14T22-53-09.416.log.gz
    -rw-r----- 1 traefik traefik    48757 Sep 15 02:45 traefik-2023-09-15T01-45-41.245.log.gz
    -rw-r----- 1 traefik traefik    47160 Sep 15 05:37 traefik-2023-09-15T04-37-56.255.log.gz
    -rw-r----- 1 traefik traefik    48261 Sep 15 08:29 traefik-2023-09-15T07-29-56.249.log.gz
    -rw-r----- 1 traefik traefik    48879 Sep 15 11:18 traefik-2023-09-15T10-18-41.241.log.gz
    -rw-r----- 1 traefik traefik  4548970 Sep 15 13:46 traefik.log
  • Setup ETCD Cluster

    Follow-up

    So, I’ll try to write up some notes, but the long and short of this is – Registrator doesn’t work. Having managed to setup the ETCD cluster I began to expand it with Registrator – and immediately hit various problems. FWIW, Registrator does function and can write to an ETCD cluster.

    Registrator isn’t being developed properly anymore. This means it’s has drifted away from supporting etcd properly – and etcd has changed quite a bit recently. There is an updated version of the API that should be used to interact with it and guess what v2 and v3 are incompatible. The API stores the data in two places. And wouldn’t you know it, Registrator talks v2 and Traefik talks v3. So, it’s now back to the drawing board.

    Looks like consul is the way to go, tbh.

    Pre-amble

    Continuing the setup of traefik, I want to setup an ETCD Cluster to allow the two Docker instances to be able to register changes and have traefik pick those up and push to the internet/intranet as appropriate.

    This has been driven by a discovery – traefik will only work with a Docker provider once. My original thought process was to setup a Docker ssh connection from Traefik. So, I’d setup a user on the remote Docker instance, configured it to allow ssh, and fought with a weird location for the docker command. I could set a DOCKER_HOST environment variable, run docker info and see the remote machine setup. However, after added the duplicate provider into the traefic.yaml I discovered the fatal flaw – no duplicate providers.

    Having no real desire to hard code everything all the time, it was back to a drawing board.

    The original source of this idea was this page:

    https://technologyconversations.com/2015/09/08/service-discovery-zookeeper-vs-etcd-vs-consul

    I had toyed with Consul before – but ran into an issue with DNS, which resulted in my dumping it. I had been hunting around how to maintain the etcd repository – there didn’t seem to be many formally documented software solutions that I could see. However, this page also talks about using Registrator – which solves the other side of the problem I was seeing too.

    Setup ETC Cluster

    That means I have a potential solution, and if I get this working properly, I can think about Consul again in the future.

    To make this work, I need etcd clustering, which lead to here:

    https://gist.github.com/kanwar-saad/b54a728fa872a767c037a5b8dc7f6e75

    Although some of the cluster setup has also come from here in the end:

    https://etcd.io/docs/v3.5/tutorials/how-to-setup-cluster/

    This is mainly because I wanted to setup the etcd services in Portainer, and I wanted them to be stacks – which means formulating a compose file.

    Couple to that the fact that I didn’t want to use a HELM image (I couldn’t be bothered to make another account). Which meant making a Bitnami image work.

    So, this is my docker compose file. Obviously, alter IP address and names as needed:

    version: '3'
    
    services:
      etcd:
        image: bitnami/etcd:latest
        restart: unless-stopped
        container_name: etcd
        ports:
          - "2379:2379"
          - "2380:2380"
        volumes:
          - etcd-data:/bitnami/etcd
        environment:
          ETCD_ADVERTISE_CLIENT_URLS: http://<host ip address>:2379
          ETCD_INITIAL_ADVERTISE_PEER_URLS: http://<host ip address>:2380
          ETCD_INITIAL_CLUSTER: etcd1=http://<host ip address>:2380,etcd2=http://<other host ip address>:2380
          ETCD_INITIAL_CLUSTER_STATE: new
          ETCD_INITIAL_CLUSTER_TOKEN: token-01
          ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
          ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
          ETCD_NAME: etcd<id number of cluster member>
          ALLOW_NONE_AUTHENTICATION: yes
        labels:
          - "traefik.enable=false"
    
    volumes:
      etcd-data:

    Starting this up on both docker nodes has resulted in a stable cluster, that I can connect to from my external PC using etcdctl – and seems quite stable when I take a node away (i.e. it still answers questions).