posts/traefik-3-docker-certificates/

[giscus[bot]]

posts/traefik-3-docker-certificates/

In today’s Traefik tutorial we’ll get FREE Wildcard certificates to use in our HomeLab and with all of our internal self-hosted services. We’re going to set up Traefik 3 in Docker and get Let’s Encrypt certificates using Cloudflare as our DNS Provider (we’ll cover how to set up others too). Then we’ll configure local DNS using PiHole (or any other local DNS) to route to our services that are now protected with secure certificates!

https://technotim.live/posts/traefik-3-docker-certificates/

[P1tt187]

I really like your approach. I'm using it for more than a year now and it just works.

I've one improvement for you. In the video you mentiond that you have to register a cname for every service.
If you're using a wildcard certificate why not using a wildcard cname?
Pihole is using dnsmasq. The webinterface of pihole does not allow to set all possible values. But if you configure it from the terminal pihole will recognize rules and respect them.

To achieve this you simply have to modify the pihole volumes.
Assuming you created a folder named pihole-dnsmasq.d and mounted the path /etc/dnsmasq.d
Create file named 02-local-wildcard-dns.conf
simply adding the line cname=*.example.com,default.example.com should do the trick
source

In my case i use an ip address approach.
Therefor can set the following:

address=/local.example.com/192.168.0.10
address=/local.example.com/:::1

It is importand to set the ipv4 and ipv6 entry because windows will work with just the ipv4 rule but linux will try ipv6 first.

[mrsatyp]

Do you know if something like this is doable with pfsense using resolver.. TIA

[TDX44]

Easily the biggest time saver in the comments. Thanks!

[Bictonbandit]

Excellent tutorial . Thanks.

I have followed everyt.hing in the video and everything appears to be working fine, BUT, I am not getting ANY SSL certs on the internal browser pages.

Being a relative novice to running a server, I am at a loss to figure out where to start looking to get letsencrypt to send me some certs.

I look forward to any pointers you can send me, so I can fix my almost working traefik system.

Rgds Ken

[timothystewart6]

Check the troubleshooting section in the video! That may help!

[AlphaInfamous]

I was having the same issue, I thought since udm-pro points to AdguardHome for DNS & having example.domain.com pointing to Traefik ip as a dns rewrite in adguard will do the trick without port forwarding but its a dead end.

Need to Port Forward in order to get SSL Certs

[bozhikovstanislav]

I have some troubles.

I install traefik on rosberiPy. but I still get

ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains...

I have acme.sh . and it generates serts from the terminal . but traefik did not do it ?

Any Help

[bozhikovstanislav]

I made it . It was environment variables - I have to use .env instead of secrets

[fenixthebuilder]

I followed the tutorial. Docker doesn't show any logs, which is good, but it doesn't show me the traefik dashboard. I had written the secrets and the .env file, but no luck. What do I do?

[ariewtf]

i have the same issue,
the password prompt appears but it won't authenticate me. the credentials show inside of the container tho.

[ariewtf]

Found the answer here:
https://community.traefik.io/t/basic-auth-not-working-from-env-docker-compose-setup-only-if-hardcoded/20207

Example:
echo $(htpasswd -nbB USER "PASS")

[drapermache]

Unfortunately this did not work for me as I think my htpasswd was using some incompatible algorithm to make the hash or something, nothing seemed to work until I found this alternative method to make the password WITHOUT htpasswd. You can use open SSL to specify the algorithm to create the hash. I used the following command to store in the .env file:

openssl passwd -apr1

it will ask for the password which will then create the hash. When adding to your .env file you'll have to add your user and the colon. So mine looked like this:

TRAEFIK_DASHBOARD_CREDENTIALS='user:[insert your hash that was created]' (include the single quotes, but don't include square brackets)

so while you have to type in user:, you're guaranteed to use something that is compatible with apache.

Source: https://blog.roberthallam.org/2020/05/generating-a-traefik-nginx-password-hash-without-htpasswd/

Forget htpasswd, long live openssl! Took me TWO days to get the simple authentication right!

[jrs-anderson]

I had the same problem. The follow commend generated a hash that worked.

htpasswd -nb admin password

[lordcheeto]

With the redirection on the entrypoint, you don't need the redirect rules in the labels or config.

entryPoints:
  http:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: https
          scheme: https
          permanent: true

[hodgkiss]

I have notice two things:
example.com is not a valid domain for techno Tim links at the end of the article.

I am getting a 404 error still, I have resolved pihole issue but this remains at issue.
I get the redirect visiting the ip address 192.168.. from port 80 to via 301, to port 443 but get 404 error: 404 page not found.

For clarity I am using my own domain and change exmaple.com to my my-site. Computer domain.

Does this have a limit using the tld. .computer as in getting confused with com domains.
I don't have .com domains to play with.

I have repeated the process with the video 3 times to ensure I did not miss a step; the error remains the same.
Each time used a different pi (pi2, pi3, pi4) with 3 different micro SD cards each time.

[AlphaInfamous]

are you seeing certs ? and can you share the traefik log

[mrsatyp]

Now playing around with Authentik, trying to get it to work with traefik...

[hodgkiss]

My logs are empty - the cert is the default traefik one lasting a day.

[boomxi]

Did you solve this one? I'm getting same error message and I've ran across discussions saying you can't plug port into host line. That didn't help either.

[mrsatyp]

Firstlly,

Desptie getting errors ONLY on NOT getting any letscrypt certs.. I had lo in say this in absolutely one the best well rounded , well thought, in depth traefik install walk-through I have come accross thus far, just had to had to say it , thanks and well done Tim..

On the off chance anyone has the same issue:

ERR Unable to obtain ACME certificate for domains error="unable to generate a certificate for the domains [xx.xx.io .xx.xx.io]: error: one or more domains had a problem:\n[.xx.xx.xx] propagation: time limit exceeded: last error: NS xx.ns.cloudflare.com. returned REFUSED for _acme-challenge.xx.xx.io.\n[xx.xx.io] propagation: time limit exceeded: last error: NS xx.ns.cloudflare.com. returned REFUSED for _acme-challenge.xx.xx.io.\n

• I can see the TXT record come through in cloudflare for 2 mins before disappearing and and getting the above message, I know it can't be any port 80 in pfsense as I have tried a rule to specifically allow this with no change.
• I even subscribed to the to $10 pm cloudflare advanced certificates in edge certificates to include the 2nd and 3rd levels domains thinking it might resolve - no joy

Hopefully someone can point out the obvious if they have come across the same message.

[AlphaInfamous]

token issue is what I'm thinking if cd refused

[mrsatyp]

Thanks Alphaine, this worked when I un-commented the 2 long lines at the end of the compose file for whatever reasons, probably needed more time...

I should have updated here.... 😉🤘🙏

[mrsatyp]

Basically I get the traefiic certs and NOT the Letsenscrypt beauties sadly..

[DanB12]

I'm getting the same, how did you resolve this?

[mrsatyp]

Sorry didn't see post until now.. been between job apps etc.. Going to resurrect this hopefully soon again, I hope this worked for you in the end there...

[mrsatyp]

Anyone got a decent links showing how to add multiple external links.

I've got my proxmox added as per the video no issues.. Looking to add more starting with Authentik first.

Is it just a case of just adding more ip along with the respective traffic labels in the Config.yml file?

Don't fancy breaking anything just yet 😊🤗

[timothystewart6]

Sure, in this post there's a link to a config file where many have contributed to https://technotim.live/posts/traefik-portainer-ssl/

[mrsatyp]

Will check it out in due course, thanks much appreciated

[ryanbarrett]

This helped me remember to change the router/service name... and as a side effect, reduced some redundancy.

    labels:
      - traefik.enable=true
      - traefik.http.routers.${TRAEFIK_SERVICE_NAME}.rule=Host(${TRAEFIK_HOST})
      - traefik.http.routers.${TRAEFIK_SERVICE_NAME}.entrypoints=https
      - traefik.http.routers.${TRAEFIK_SERVICE_NAME}.tls=true
      - traefik.http.services.${TRAEFIK_SERVICE_NAME}.loadbalancer.server.port=${TRAEFIK_SERVICE_PORT}
    networks:
      - proxy
networks:
  proxy:
    external: true

.env file:

TRAEFIK_HOST=`pihole.local.mydomain.com`
TRAEFIK_SERVICE_NAME=pihole
TRAEFIK_SERVICE_PORT=80

[AFCM1]

Thanks for the tutorial!
Just wanted to let you know there's a small typo in your table content, you wrote "Acme Endpoings"

[DanB12]

Thanks Tim, followed all but getting below error. Any ideas?
2024-05-26T13:54:28+01:00 ERR Command error | error=command traefik error: yaml: line 16: did not find expected key
2024-05-26T13:55:20+01:00 ERR Command error | error=command traefik error: yaml: line 16: did not find expected key
2024-05-26T13:56:20+01:00 ERR Command error | error=command traefik error: yaml: line 16: did not find expected key
2024-05-26T13:57:21+01:00 ERR Command error | error=command traefik error: yaml: line 16: did not find expected key

[TDX44]

I can't remember the exact solution for this but I threw my docker-config.yml and traefik.yml into chatgpt and if I remember right it was a formatting or other syntax error. Double check your spacing and indentations.

[DanB12]

Hi TDX44, all checked with chatgpt and in vs code, no errors. Anymore suggestions?

[TDX44]

I'm no expert but I got lost in some testing and realized that not deleting my associated volumes caused a lot of headaches. Try deleting your volumes and recreating. If that doens't work post your docker-config.yaml and traefik.yaml.

[boomxi]

I ran into same issue and pasted my config files in here https://www.yamllint.com/ one at the time and it was formatting error.

[TDX44]

I ran into same issue and pasted my config files in here https://www.yamllint.com/ one at the time and it was formatting error.

I've never seen that site. Thanks for sharing.

[arlandzatania]

Everything works except I am not able to make any links for proxmox LXC´s. I tried making one for jellyfin and other services but no luck at all. Can anyone offer a word of advice?

[info-path]

Tim, thank you very much for this video, it helped me a lot. It must have taken a lot of work, I appreciate that! Thanks to you, all my local domains have a valid certificate.

But, how to obtain the certificate for a public address?

I have a registered A record (name) and would like to use it for a reverse proxy. How to force a certificate for this record? Maybe a stupid question, but I only have experience with SWAG. This doesn't work, it keeps returning me a certificate for *.local.example.com only:

http:
 #regionrouters 
  routers:
    some-public-server:
      entryPoints:
        - "https"
      rule: "host(name.example.com`)"
      middlewares:
        - default-headers
        - https-redirectscheme
      tls: {}
      service: some-public-server
    pihole:
#endregion
#region services
  services:
    some-public-server:
      loadBalancer:
        servers:
          - url: "https://192.168.1.10:5000"
        passHostHeader: true

[boomxi]

Tim, thank you for the info. I have two questions. Do you expect an update on CloudFlare that is visible in the DNS list? And do you run firewall did you have to enable any new rules? (I have FortiGate and I suspect it is causing me headache because I can't access urls from outside of the network).

[camdenyoung]

Obviously the above tutorial is based around running traefik in docker. I run traefik 'natively' inside a debian LXC, but I can't workout how you inform traefik of the cloudflare API in that scenario. Surely there is a way of defining it directly inside your static traefik yaml, but for the life of me I cannot find that documented.

Thoughts?

[rightfittech]

Thank you for the tutorial. In your design, you show multiple Docker nodes; however, I don't believe that you're using a Docker Swarm. I found that I had to run other containers on the same Docker node as Traefik so that it would work even though the "proxy" network is shared amongst the nodes. Have you run into a case where Traefik is used in a Docker Swarm? Would I just add the other containers to the config.yml?

[Palgogo]

Great video, thanks. Followed all the above, receive traeifik cert in the end and 404 when trying to access dashboard. Think to start with more easy setup and access traefik unsecure, and then configure it with secure properties. It's kinda demotivates receive 404 on the last stage of connection.

[pne137]

I had the issue where despite all the settings appearing correct within the traefik container I was getting the default traefik certificate when accessing the dashboard. As usual Techno Tim was right when he said "It's always DNS".

I had overlooked that when I set up my Firewall I had created a NAT port forwarding rule that redirected all DNS traffic to my local DNS resolver PiHole. So this was preventing Cloudflare from preforming its check that the site covered by the certificate resolved.

Solution was to simply add another NAT port forwarding rule (with a higher priority) that allowed DNS requests from the Traefik host to external DNS services. After that my certificates could be validated and used by traefik.

[pne137]

To help in troubleshooting this I used the 'dig' command in a terminal session on the host running the traefik container. I was able to see in the Pi-Hole audit log that all 'dig' requests were being re-directed to PiHole regardless of any server specified in the dig command eg. dig @1.1.1.1 bbc.co.uk was being resolved by PiHole.
Was then just a matter of adjusting the rule in OPNsense that I'd created to redirect all external DNS queries to PiHole to allow the host running the traefik container to make external DNS requests.
Testing the updated firewall rule was simply done by making further dig requests using external DNS servers and monitoring that they no longer appeared in the PiHole log. When I was certain those external DNS requests were no longer being redirected to PiHole I updated the traefik.yml to use the production letsencrypt certificate server. Restarted the traefik container and all was well with the world as it now was able to pull down and verify the certificate.

So in summary if you're having problems check out your DNS and make sure that those resolver queries are working as you would expect.

[jorge-gbs]

Maybe you can shed some light, I haves apps.local.domain.com. local.domain.com is on the local DNS server no records are on the public dns. The acme challenge TXT record and Traefik adds the info into acme.json but it still serves the DEFAULT CERT. When I dig @1.1.1.1 apps.local.domain.com I get the external Name Servers not the internal which is what I would expect.

[pne137]

Check your docker logs for the container, LetsEncrypt are ruthless in blocking you if you exceed more than just a couple of certificate pull requests in 24hrs. I experienced this and had to wait 24 hours before I could pull a certificate from production. Being blocked by LetsEncrypt, even when all other settings are correct will result in the default cert being served.

Patience is definitely a must have when trouble shooting, use the Let'sEncrypt staging until you are absolutely sure that everything is working, especially the DNS resolution from the container. Good luck.

[jorge-gbs]

Legit, check DNS per dig above, check logs periodically see if there are errors, fix them and wait.

Would be interest in knowing if there is a command to force it to recheck. Especially while its set to staging.

[drsa23]

I followed the tutorial to the core and most things are working fine. However, I need help with certain other things.

First, I'd like to describe my setup.

1. First, I have a main Proxmox PC on which I have a few LXCs, including docker portainer and dockage, a TrueNAS Scale which has Nextcloud installed on it, and then I have a Ubuntu VM which hosts my Traefik docker.
2. I then have another proxmox test PC running some test LXCs and including docker portainer.
   3 Lastly, I have an NUC running docker portainer in Dietpi.

I am new to all this so keep trying a few things.

Now, I can get certs for my urls of proxmox, my docker portainers on proxmox, and my Truenas. However, I am not sure how to get certs and DNS for services running in dockers and LXCs. Also, even though I get cert for Truenas, I am not able to get cert for Nexcloud, even though my internal DNS for Nexcloud is resolving fine using PiHole.

Any guide to how to accomplish my tasjks?

[jorge-gbs]

So, that's a complicated setup. Here's my two cents. You gotta remember that Traefik is not getting a certificate for each sub-domain and container. Its using a wildcard cert and then routing the connection to the correct container / vm. So this is a network routing issue not a certificate issue. You have to make sure that your traffic to those endpoints is being routed through traefik. This will probably require going into Firewall rules and NAT in both docker proxmox and potentially your firewall...

[drsa23]

Any guide on resource on how to achieve that. I am not an IT guy, I just do this for fun, just as a hobby.

[mrsatyp]

Try cloudflare tunnels theres video that touches it with traefik for containers by Christian Lempa.. If it helps.. Ta

…

On Sat, 7 Sept 2024, 19:06 drsa23, ***@***.***> wrote: Any guide on resource on how to achieve that. I am not an IT guy, I just do this for fun, just as a hobby. — Reply to this email directly, view it on GitHub <#319 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEKC7ZOUYPD6JX2FY6XWTBLZVM6DHAVCNFSM6AAAAABHA6E6BWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANJXHAZDSOA> . You are receiving this because you commented.Message ID: <techno-tim/techno-tim.github.io/repo-discussions/319/comments/10578298@ github.com>

[61karadeniz61]

Hello Tim,

Thank you for your introduction. It took me a while to get it running on a test environment. I’m someone who tries these instructions at home as a hobby. I managed to get it working without restarting the docker-compose file by adding a dynamic entry in the docker-compose.yaml and traefik.yml files, and I was able to obtain a certificate for the client devices by creating additional configuration.yml files.

However, I’m having trouble creating a configuration for setting up a Remote Desktop Server. I added the entrypoints in both docker-compose.yml and traefik.yml, but so far without success. Could you check my configuration file or suggest how I could solve this issue? Or maybe you could add an adjusted rdp.yml to the tutorial?

Perhaps you could extend the tutorial with Crowdsec if that's feasible.

Thank you very much for your work, and best regards

tcp:
routers:
rdp-router:
entryPoints:
- "rdp" # EntryPoint for RDP connections on port 3389
rule: "HostSNI(rdp.example.com)" # SNI rule to match the hostname
service: rdp-service # Name of the service to be used
tls:
passthrough: true # Passes through the TLS connection to the backend server

services:
rdp-service:
loadBalancer:
servers:
- address: "192.168.50.104:3389" # ddress of the Remote Desktop (RDP) server

[drsa23]

So, I have something weird going on lately and would like to get some help. I had my basic setup up and running fine with about 11 hostnames defines within the respective dockers and I had another 10 which were configured using middleware in config.yml. Yesterday I tried to add a new host to config.yml and for some reason the hostnames for the 10 services configured via config.yml stopped working. As I am not a very knowledgeable person in all this and I just follow tutorials and guides, I thought it is best to restore my last backup for my Ubuntu VM (running Traefik) in Proxmox. After backup everything started working fine. This morning I tried to add another service (different that the one yesterday) to my config.yml. Again had some issue and did the restore of a working setup. However, this time it did not help. I am still not able to get those 10 services to route. Someone, please help me on how to get this issue resolved.

[drsa23]

Just checked the logs and this is what I am getting:

ERR Error while building configuration (for the first time) error="field not found, node: services" providerName=file

[andrewdietzel]

I am experiencing this same thing. Perfectly working config.yml file until just recently. Now, as soon as I enable the use of config.yml I'm getting: {"level":"error","error":"command traefik error: field not found, node: file","time":"2024-10-09T01:43:22Z","message":"Command error"}

[andrewdietzel]

I was eventually able to resolve my error by double checking and adjusting my indentation and spacing in my config.yml file.

[Spinkman]

I'm getting "exec /entrypoint.sh: operation not permitted" over and over in the logs after starting things up. not sure how to fix that.

[mudgw2]

Was working for me, then I rebooted my proxmox environment and fired up Traefik to find an error:
2024-10-16T16:59:33Z ERR Command error | error=command traefik error: field not found, node: filename

Assuming its the traefik.yml file with the filename field /config.yml. Should this be ./config.yml instead? Any suggestions as to what generates this error?

[mudgw2]

SOLVED: The spacing in the traefik.yml is very specific and filename: needs to be positioned as per the documentation. Once I did this, I was able to move on to another error :)

[erdemoney]

Awesome tutorial! I've found this topic very overwhelming so thanks for walking me through it.
One suggestion though is for testing traefik entries, you assume that we have a local DNS server which I do not and I don't think most people do. I just configure my DNS entries via Cloudflare. So I would suggest instead showing how to test by adding entries to the /etc/hosts file as that is available to everyone.

[poulsen75]

Hi Tim Thanks for the guide
I have an issue, when I go to website, i'll get an "401 Unauthorized", the certificate is the staging one.. I tried to change it to production, same error.
Do have any Ideas?

TIA Søren

[andrewdietzel]

Assuming you're using Tim's starter files, did you comment out the default-whitelist entries and test without those enabled? Also, have you tried accessing your sites in an incognito window or a different browser?

[poulsen75]

Yes I tried in 3 different browser.
this is from the log.
"Authentication failed middlewareName=traefik-auth@docker middlewareType=BasicAuth"
I ran the curl test from Clouldflare and that work.

[Eucal-IT]

Good day,
Thangs for the informitive video. It worked well till I changed the certs from stagging to production, then I keep getting this error: 2024-12-13T10:31:41Z ERR Unable to obtain ACME certificate for domains error="cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/': Get "https://acme-v02.api.letsencrypt.org/\": dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: server misbehaving" ACME CA=https://acme-v02.api.letsencrypt.org/ acmeCA=https://acme-v02.api.letsencrypt.org/ domains=["local.van-zyl.org","*.local.van-zyl.org"] providerName=cloudflare.acme routerName=traefik-secure@docker rule=Host(traefik-dashboard.local.van-zyl.org)

[jme1483]

Thanks so much for this! I got it working finally (fyi for those that block preprogrammed dns on individual devices, you need to allow the system/container this is installed on to access ports 80 & 444 to get the certificates; this tripped me up for awhile).

Silly question but how do I then add additional services to the config file? Do I need separate files?

[jme1483]

Sorry, had the wrong ports. Ports you need to allow are 53, 853, and 5053