I’d setup Traefik to use LetsEncrypt, but due to the scarey words around rate limits etc, I felt it sensible to use the staging LetsEncrypt servers, rather than the production ones – basically I wasn’t sure how many changes I’d have to make as it has been a while since I played with any of this in anger. I assumed that Traefik would renew the certificates with LetsEncrypt when I needed production ones.
I surprisingly got this working quite straightforwardly – I had more issues with getting Apache and WordPress (mainly Apache) playing nicely with everything.
However, the staging certificates result in a warning which I didn’t want any longer than necessary. Since it now was working, I altered my certificate resolver in Traefik to point to the production LetsEncrypt server. I then performed a bounce to pick up the changes, expecting to see a new certificate and no more errors.
Anxious pause…
I waited a bit (pressing F5) and was disappointed to see that it hadn’t changed. Same certificate was in use, and I could see in the log that no attempt had been made to fetch a new certificate.
Frantic prodding ensued as I attempted to work out what I had done wrong. Nothing was the answer. The problem, such as it is, is that Traefik already has a certificate that it can use. Traefik doesn’t feel it needs to change it the certificate.
Whilst the last link on the face of it initially seemed less relevant (it related to fixing a security flaw), on reading it gave the simplest answer for my situation.
In my situation, it looked like I could simply delete the storage acme.json file. No certificates needed to be kept – I could just start again from scratch.
I stopped Traefik, moved acme.json out of the way (belts and braces right), and then restarted it.
Traefik Renew from LetsEncypt!
Hey, Presto! I could see in the log file the lack of certificate being identified and Traefik calls to renew certificates from the production LetsEncrypt server.
Padlock look right, everything is gravy! Let’s move on to the next thing.
An explanation of how to set up Traefik Log Rotation. This covers both the access log AND the traefik log.
I initially setup simple log rotation using the standard logrotate package i have installed for everything else. I implemented this by following this page (there are others about) but this covered my scenario the best (as I have traefik running natively, not in a container.
The main change I made was in the directory (mine is /var/log/traefik/) and then the files (I have an access.log and a traefik.log, so I altered the pattern to be *.log. I also added size – more on that in a minute.
This resulted in a file /etc/logrotate.d/traefik below:
However, I began to spot that it did not working as anticipated.
access.log wouldn’t rotate
traefik.log rotated but still wrote to the rotated our file
It turns out access.log not rotating was because I misunderstood what size would do. It’s not an either/or with the daily setting – size supersedes daily if it is in the file. Simply removing size made access.log rotate as expect.
However, traefik.log still won’t rotate on a USR1 – which seems contrary to their own documentation (link).
I decided that something weird must be going on, so I dug about a bit. If you check the code in git (link) you’ll see that the code specifically mentions rotation of the access log, and their tests also only check the access log.
This means that USR1 signal only rotates the access log. That was annoying.
I wondered if I was missing something, so I went in search of rotate in git (link) and found something that whilst documented at source level seems to be missing in the Traefik log documentation (link).
It turns out that Traefik can do some quite clever stuff with rotating its own log if you configure it right.
log:
# Log level
#
# Optional
# Default: "ERROR"
#
level: <whatever level you want>
# Sets the filepath for the traefik log. If not specified, stdout will be used.
# Intermediate directories are created if necessary.
#
# Optional
# Default: os.Stdout
#
filePath: <directory location>/<file>.log
# Format is either "json" or "common".
#
# Optional
# Default: "common"
#
#format: json
maxSize: 5
maxBackups: 50
maxAge: 10
compress: true
This will rotate at 5Mb, compress the old copies, and keep for 10 days and keep a maximum of 50 files. There are other options in the document – I have not used them.
This then means I can make the logrotate configuration focus just on the access log:
