HTTP 409 "Conflict" when the certificate identified by 'replaces' has already been replaced

[vmihailenco]

What version of the package are you using?

v0.25.0

What are you trying to do?

I've been hit by ietf-wg-acme/acme-ari#56 . I see the comment on the thread indicating that nothing can be done on the client, but the error message indicates otherwise and there PRs like this.

What steps did you take?

I had to manually remove existing certificates from the storage to let the renewal proceed.

What did you expect to happen, and what actually happened instead?

The certificate must be renewed. Instead, the certificate expired and could not renewed.

How do you think this should be fixed?

Either https://github.com/go-acme/lego/pull/2475/files or just deleting broken certificates from the storage.

Please link to any related issues, pull requests, and/or discussion

ietf-wg-acme/acme-ari#56
https://github.com/go-acme/lego/pull

Bonus: What do you use CertMagic for, and do you find it useful?

👍

[zllovesuki]

Can confirm that we are hitting the same issue

[mholt]

Interesting... here's our logic regarding that:

	// Notify the ACME server we are replacing a certificate (if the caller says we are),
	// only if the following conditions are met:
	// - The caller has set a Replaces value in the context, indicating this is a renewal.
	// - Not using test CA. This should be obvious, but a test CA should be in a separate
	// environment from production, and thus not have knowledge of the cert being replaced.
	// - Not a certain attempt number. We skip setting Replaces once early on in the retries
	// in case the reason the order is failing is only because there is a state inconsistency
	// between client and server or some sort of bookkeeping error with regards to the certID
	// and the server is rejecting the ARI certID. In any case, an invalid certID may cause
	// orders to fail. So try once without setting it.
	if !am.config.DisableARI && !usingTestCA && attempts != 2 {
		if replacing, ok := ctx.Value(ctxKeyARIReplaces).(*x509.Certificate); ok {
			params.Replaces = replacing
		}
	}

In other words, on the 3rd attempt we try to get the certificate without "replaces" specified. Do you have the debug logs I can examine to verify that this is actually the problem, and not something else?

[vmihailenco]

I don’t have the logs anymore, but as far as I remember there were no indications that CertMagic was performing retries — for example, there were no messages like “retry #123” in the logs.

And I can easily imagine why a single retry without replaces might fail...

[mholt]

That seems very unlikely, CertMagic has retries built in if there's any errors. I would need to see the (full) logs in order to fix this if there's a bug.

[vatsaljrathod]

We got the same issue. I am sharing the logs(i updated the actual certificate host with abc.xyz), might be useful.

info maintenance started background certificate maintenance {"cache": "0xc0001c0180"}
info certificate needs renewal based on ARI window {"subjects": [".abc.xyz"], "expiration": 1768505981, "ari_cert_id": "jw0TovYuftFQbDMYOF1ZjiNykco.Bb2fOSS1ZG9wIviLW3hG60vQ", "next_ari_update": 1765972364.6836352, "renew_check_interval": 600, "window_start": 1765837425, "window_end": 1765992874, "selected_time": 1765844188, "renewal_cutoff": 1765843588}
info renew acquiring lock {"identifier": ".abc.xyz"}
info renew lock acquired {"identifier": ".abc.xyz"}
info renew renewing certificate {"identifier": ".abc.xyz", "remaining": 2537398.605025773}
info waiting on internal rate limiter {"identifiers": [".abc.xyz"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "support@abc.xyz"}
info done waiting on internal rate limiter {"identifiers": [".abc.xyz"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "support@abc.xyz"}
info using ACME account {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/2355426277", "account_contact": ["mailto:support@abc.xyz"]}
error renew could not get certificate from issuer {"identifier": ".abc.xyz", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 409 urn:ietf:params:acme:error:alreadyReplaced - Could not validate ARI 'replaces' field :: cannot indicate an order replaces certificate with serial "05bd9f3924b5646f7022f88b5b7846eb4bd0", which already has a replacement order"}
info renew releasing lock {"identifier": ".abc.xyz"}

[mholt]

@vatsaljrathod Thanks, but can you please share the full logs? The thing is I believe you when you say it happens, but I need the full logs, before and after, for context and diagnosis.

[adityapatadia]

The issue happened because we used ManageSync and we wrote code to crash if there is any error using Fatalf. We changed it to Errorf and also switched to ManageAsync. Hopefully it will not cause issue in future.

[vmihailenco]

Having the full logs (before and after) would certainly help with context and diagnosis. However, it’s also possible that there are no relevant logs at all.

From my understanding, this can happen when the renewal itself succeeds, but the storage layer fails to persist the newly renewed certificate—or when the certificate is later overwritten by another host. In these cases, no logs are emitted.

The current assumption seems to be that a single retry is sufficient to handle this scenario, but a single retry is not a reliable failure-handling strategy. Moreover, retries aren’t reflected in the logs, which makes it difficult to confirm whether this mechanism is actually being triggered or working as intended.

[oliverpool]

I just got the same issue. In my case attempt is always 1, because I directly call Config.GetCertificate.

Looking at the code, this attempt counter is only increased when calling RenewCertAsync or ObtainCertAsync (which are properly called by ManageAsync only).

So as a workaround, I will have to manually implement doWithRetry or find a way to keep track of the failed attempts by some other mean.

certmagic/async.go

Line 84 in d2a7286

func doWithRetry(ctx context.Context, log *zap.Logger, f func(context.Context) error) error {

Suggested solutions

• Ideally Config.GetCertificate should manage this automatically (since it already stores the certificate on disk, why not store the attempt as well?)
• Or if this management must be managed by the caller, improve the doc of GetCertificate to mention this footgun.

[oliverpool]

I think that #364 would help in such situation.

Regarding the root-cause, I see the following logs (I was running v0.23.0 at that time) before the alreadyReplaced failure occurs (somewhat redacted, since I am not sure what is sensitive - I can send details per e-mail):

Oct 23 09:17:15 pages pages[917]: 1.7612038353294635e+09        info        certificate needs renewal based on ARI window        {"subjects": ["log.pfad.fr"], "expiration": 1763829777, "ari_cert_id": "jw0TovYuftFQbDMYOF1ZjiNykco.BRmf4juu-lIn5ev9GfgbBgoV", "next_ari_update": 1761220705.071686, "renew_check_interval": 600, "window_start": 1761161221, "window_end": 1761316671, "selected_time": 1761204295, "renewal_cutoff": 1761203695}
Oct 23 09:17:15 pages pages[917]: 1.7612038353299868e+09        info        on_demand        attempting certificate renewal        {"remote_ip": "64:ff9b:1::7cf3:bfac", "remote_port": "41803", "server_name": "log.pfad.fr", "subjects": ["log.pfad.fr"], "expiration": 1763829777, "remaining": 2625941.670353895, "revoked": false}
Oct 23 09:17:15 pages pages[917]: 1.7612038353320372e+09        info        renew        acquiring lock        {"identifier": "log.pfad.fr"}
Oct 23 09:17:15 pages pages[917]: 1.7612038353336687e+09        info        renew        lock acquired        {"identifier": "log.pfad.fr"}
Oct 23 09:17:15 pages pages[917]: 1.761203835334023e+09        info        renew        renewing certificate        {"identifier": "log.pfad.fr", "remaining": 2625941.665981135}
Oct 23 09:17:15 pages pages[917]: 1.76120383533481e+09        info        waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:17:15 pages pages[917]: 1.761203835334842e+09        info        done waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:17:15 pages pages[917]: 1.761203835334856e+09        info        using ACME account        {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/1897480***", "account_contact": ["mailto:*redacted*"]}
Oct 23 09:17:16 pages pages[917]: 1.7612038367776399e+09        error        validating authorization        {"problem": {"type": "urn:ietf:params:acme:error:malformed", "title": "", "detail": "No such authorization", "instance": "", "subproblems": null}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1897480***/440843816431", "attempt": 1, "max_attempts": 3}
Oct 23 09:17:16 pages pages[917]: github.com/mholt/acmez/v3.(*Client).ObtainCertificate
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/mholt/acmez/v3@v3.1.2/client.go:152
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*ACMEIssuer).Issue
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:382
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*Config).renewCert.func2
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:906
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.doWithRetry
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/async.go:104
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*Config).renewCert
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:982
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*Config).RenewCertAsync
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/config.go:768
Oct 23 09:17:16 pages pages[917]: github.com/caddyserver/certmagic.(*Config).renewDynamicCertificate.func2
Oct 23 09:17:16 pages pages[917]:         /home/olivier/.cache/go/pkg/mod/github.com/caddyserver/certmagic@v0.23.0/handshake.go:767
Oct 23 09:17:16 pages pages[917]: 1.761203836777781e+09        error        renew        could not get certificate from issuer        {"identifier": "log.pfad.fr", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization"}
Oct 23 09:17:16 pages pages[917]: 1.761203836777811e+09        error        renew        will retry        {"error": "[log.pfad.fr] Renew: [log.pfad.fr] solving challenge: getting authorization at https://acme-v02.api.letsencrypt.org/acme/authz/1897480***/601833951861: attempt 1: https://acme-v02.api.letsencrypt.org/acme/authz/1897480***/601833951861: HTTP 404 urn:ietf:params:acme:error:malformed - No such authorization (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 1.444107144, "max_duration": 2592000}
Oct 23 09:18:02 pages pages[917]: 1.761203882761257e+09        info        certificate needs renewal based on ARI window        {"subjects": ["log.pfad.fr"], "expiration": 1763829777, "ari_cert_id": "jw0TovYuftFQbDMYOF1ZjiNykco.BRmf4juu-lIn5ev9GfgbBgoV", "next_ari_update": 1761220705.071686, "renew_check_interval": 600, "window_start": 1761161221, "window_end": 1761316671, "selected_time": 1761204295, "renewal_cutoff": 1761203695}
Oct 23 09:18:16 pages pages[917]: 1.761203896778619e+09        info        renew        renewing certificate        {"identifier": "log.pfad.fr", "remaining": 2625880.221387401}
Oct 23 09:18:16 pages pages[917]: 1.7612038967796943e+09        info        creating new account because no account for configured email is known to us        {"email": "*redacted*", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "error": "open /var/lib/pages/certmagic/acme/acme-staging-v02.api.letsencrypt.org-directory/users/*redacted*/git.json: no such file or directory"}
Oct 23 09:18:16 pages pages[917]: 1.7612038967797682e+09        info        ACME account has empty status; registering account with ACME server        {"contact": ["mailto:*redacted*"], "location": ""}
Oct 23 09:18:16 pages pages[917]: 1.761203896781499e+09        info        creating new account because no account for configured email is known to us        {"email": "*redacted*", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory", "error": "open /var/lib/pages/certmagic/acme/acme-staging-v02.api.letsencrypt.org-directory/users/*redacted*/git.json: no such file or directory"}
Oct 23 09:18:17 pages pages[917]: 1.7612038975917425e+09        info        new ACME account registered        {"contact": ["mailto:*redacted*"], "status": "valid"}
Oct 23 09:18:17 pages pages[917]: 1.7612038975953891e+09        info        using ACME account        {"account_id": "https://acme-staging-v02.api.letsencrypt.org/acme/acct/237181***", "account_contact": ["mailto:*redacted*"]}
Oct 23 09:18:17 pages pages[917]: 1.7612038979033508e+09        info        trying to solve challenge        {"identifier": "log.pfad.fr", "challenge_type": "tls-alpn-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
Oct 23 09:18:17 pages pages[917]: 2025/10/23 09:18:17 http: TLS handshake error from [2a01:4f8:c17:f026::1]:60408: EOF
Oct 23 09:18:19 pages pages[917]: 1.7612038991453004e+09        info        served key authentication certificate        {"server_name": "log.pfad.fr", "challenge": "tls-alpn-01", "remote": "[2600:3000:2710:300::82]:56835", "distributed": false}
Oct 23 09:18:19 pages pages[917]: 1.7612038995008953e+09        info        served key authentication certificate        {"server_name": "log.pfad.fr", "challenge": "tls-alpn-01", "remote": "[2a05:d016:dcc:9102:bab1:2876:172:882d]:12460", "distributed": false}
Oct 23 09:18:19 pages pages[917]: 1.7612038997615201e+09        info        served key authentication certificate        {"server_name": "log.pfad.fr", "challenge": "tls-alpn-01", "remote": "[2600:1f16:13c:c400:9caf:6c8a:941c:1856]:65284", "distributed": false}
Oct 23 09:18:20 pages pages[917]: 1.7612039000121236e+09        info        served key authentication certificate        {"server_name": "log.pfad.fr", "challenge": "tls-alpn-01", "remote": "[2600:1f14:a8b:500:db19:34:6620:e75d]:44002", "distributed": false}
Oct 23 09:18:20 pages pages[917]: 1.7612039000539253e+09        info        served key authentication certificate        {"server_name": "log.pfad.fr", "challenge": "tls-alpn-01", "remote": "[2406:da18:611:5701:defc:5dcf:9dfc:d927]:20174", "distributed": false}
Oct 23 09:18:21 pages pages[917]: 1.761203901258178e+09        info        authorization finalized        {"identifier": "log.pfad.fr", "authz_status": "valid"}
Oct 23 09:18:21 pages pages[917]: 1.7612039012582302e+09        info        validations succeeded; finalizing order        {"order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/237181***/28093365083"}
Oct 23 09:18:24 pages pages[917]: 1.761203904862836e+09        info        got renewal info        {"names": ["log.pfad.fr"], "window_start": 1766307835, "window_end": 1766463284, "selected_time": 1766337872, "recheck_after": 1761225504.8628223, "explanation_url": ""}
Oct 23 09:18:25 pages pages[917]: 1.7612039051628788e+09        info        got renewal info        {"names": ["log.pfad.fr"], "window_start": 1766307835, "window_end": 1766463284, "selected_time": 1766371314, "recheck_after": 1761225505.1628644, "explanation_url": ""}
Oct 23 09:18:25 pages pages[917]: 1.761203905162978e+09        info        successfully downloaded available certificate chains        {"count": 2, "first_url": "https://acme-staging-v02.api.letsencrypt.org/acme/cert/2c8de496e26c8b5efaa376fa5bf36511f22a"}
Oct 23 09:18:25 pages pages[917]: 1.7612039051634145e+09        info        waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:18:25 pages pages[917]: 1.7612039051635897e+09        info        done waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:18:25 pages pages[917]: 1.761203905163629e+09        info        using ACME account        {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/1897480***", "account_contact": ["mailto:*redacted*"]}
Oct 23 09:18:25 pages pages[917]: 1.7612039057189689e+09        error        renew        could not get certificate from issuer        {"identifier": "log.pfad.fr", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 409 urn:ietf:params:acme:error:alreadyReplaced - Could not validate ARI 'replaces' field :: cannot indicate an order replaces certificate with serial \"05199fe23baefa5227e5ebfd19f81b060a15\", which already has a replacement order"}
Oct 23 09:18:25 pages pages[917]: 1.761203905719257e+09        info        renew        releasing lock        {"identifier": "log.pfad.fr"}
Oct 23 09:18:25 pages pages[917]: 1.7612039057194345e+09        error        on_demand        renewing and reloading certificate        {"remote_ip": "64:ff9b:1::7cf3:bfac", "remote_port": "41803", "server_name": "log.pfad.fr", "subjects": ["log.pfad.fr"], "expiration": 1763829777, "remaining": 2625941.670353895, "revoked": false, "server_name": "log.pfad.fr", "error": "[log.pfad.fr] Renew: [log.pfad.fr] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 409 urn:ietf:params:acme:error:alreadyReplaced - Could not validate ARI 'replaces' field :: cannot indicate an order replaces certificate with serial \"05199fe23baefa5227e5ebfd19f81b060a15\", which already has a replacement order (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Oct 23 09:18:56 pages pages[917]: 1.7612039362580962e+09        info        certificate needs renewal based on ARI window        {"subjects": ["log.pfad.fr"], "expiration": 1763829777, "ari_cert_id": "jw0TovYuftFQbDMYOF1ZjiNykco.BRmf4juu-lIn5ev9GfgbBgoV", "next_ari_update": 1761220705.071686, "renew_check_interval": 600, "window_start": 1761161221, "window_end": 1761316671, "selected_time": 1761204295, "renewal_cutoff": 1761203695}
Oct 23 09:18:56 pages pages[917]: 1.7612039362586539e+09        info        on_demand        attempting certificate renewal        {"remote_ip": "64:ff9b:1::652e:11", "remote_port": "59907", "server_name": "log.pfad.fr", "subjects": ["log.pfad.fr"], "expiration": 1763829777, "remaining": 2625840.74174667, "revoked": false}
Oct 23 09:18:56 pages pages[917]: 1.761203936260575e+09        info        renew        acquiring lock        {"identifier": "log.pfad.fr"}
Oct 23 09:18:56 pages pages[917]: 1.76120393626192e+09        info        renew        lock acquired        {"identifier": "log.pfad.fr"}
Oct 23 09:18:56 pages pages[917]: 1.7612039362622173e+09        info        renew        renewing certificate        {"identifier": "log.pfad.fr", "remaining": 2625840.737786318}
Oct 23 09:18:56 pages pages[917]: 1.761203936262882e+09        info        waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:18:56 pages pages[917]: 1.7612039362629125e+09        info        done waiting on internal rate limiter        {"identifiers": ["log.pfad.fr"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "*redacted*"}
Oct 23 09:18:56 pages pages[917]: 1.7612039362629251e+09        info        using ACME account        {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/1897480***", "account_contact": ["mailto:*redacted*"]}
Oct 23 09:18:56 pages pages[917]: 1.7612039365652416e+09        error        renew        could not get certificate from issuer        {"identifier": "log.pfad.fr", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 409 urn:ietf:params:acme:error:alreadyReplaced - Could not validate ARI 'replaces' field :: cannot indicate an order replaces certificate with serial \"05199fe23baefa5227e5ebfd19f81b060a15\", which already has a replacement order"}
Oct 23 09:18:56 pages pages[917]: 1.7612039365653067e+09        error        renew        will retry        {"error": "[log.pfad.fr] Renew: [log.pfad.fr] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 409 urn:ietf:params:acme:error:alreadyReplaced - Could not validate ARI 'replaces' field :: cannot indicate an order replaces certificate with serial \"05199fe23baefa5227e5ebfd19f81b060a15\", which already has a replacement order (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 0.303361245, "max_duration": 2592000}

The creating new account because no account for configured email is known to us looks suspicious, because this domain has been running for quite a while. The email is the same, but apparently 2 accounts got registered...

Edit: the only related logs before that read got renewal info {"names": ["log.pfad.fr"]... every 6 hours

[mholt]

@vmihailenco

The current assumption seems to be that a single retry is sufficient to handle this scenario, but a single retry is not a reliable failure-handling strategy. Moreover, retries aren’t reflected in the logs, which makes it difficult to confirm whether this mechanism is actually being triggered or working as intended.

I must be misunderstanding you, because I don't think any of that is correct. To clarify, there are multiple retries, one of them doesn't include replaces, which should resolve the issue at hand. And, retries are logged, (look for "rwill retry" messages, as well as "attempt" fields in other logs.

From my understanding, this can happen when the renewal itself succeeds, but the storage layer fails to persist the newly renewed certificate—or when the certificate is later overwritten by another host. In these cases, no logs are emitted.

The storage module should return an error, which is then logged; if that doesn't happen, then it would be a bug in the storage module.

@oliverpool

I just got the same issue. In my case attempt is always 1, because I directly call Config.GetCertificate.

Ah, so you're calling GetCertificate directly, which implies that on-demand TLS is enabled. Because on-demand TLS happens in real-time during a handshake, synchronous mode is used, which disables retries, because if it retried it could take hours or days to get a certificate, in theory. 🫠 Obviously we need to complete the handshake in milliseconds... so it makes sense that a retry without replaces isn't used in your case.

I'll take a look at the PR next. Thanks!

[oliverpool]

on-demand TLS is enabled

It’s true that only ARI refresh happens in the background for on-demand TLS:

certmagic/handshake.go

Line 626 in d2a7286

go func(hello *tls.ClientHelloInfo, cert Certificate, logger *zap.Logger) {

Maybe something like this could be added for ARI: if the time has come for a renewal (but not expired yet), start a goroutine which might perform multiple renewal attempts (thinking more about it: this might get very complex very fast. Probably not a very good idea).

---

Regarding the root cause, do you have any idea what a new account was created? (Was there any change since v0.23.0?)