Support for custom url request headers in .chezmoiexternal

[NavyD]

Is your feature request related to a problem? Please describe.

Unable to configure the url request header in .chezmoiexternal causes the download to fail, as some web services check the access request headers such as User-Agent, Host

The following curl command will download the 8.7MB maven archive, if you remove the curl's default headers(Customize headers) the download will fail.

# success
$ curl -vsSfL https://mirrors.ustc.edu.cn/apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz | sha512sum --check <(echo "7d171def9b85846bf757a2cec94b7529371068a0670df14682447224e57983528e97a6d1b850327e4ca02b139abaab7fcb93c4315119e6f0ffb3f0cbc0d0b9a2 -")      
*   Trying 198.18.1.141:443...
* Connected to mirrors.ustc.edu.cn (198.18.1.141) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2063 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=mirrors.ustc.edu.cn
*  start date: Jun 23 07:17:37 2024 GMT
*  expire date: Sep 21 07:17:36 2024 GMT
*  subjectAltName: host "mirrors.ustc.edu.cn" matched cert's "mirrors.ustc.edu.cn"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz]
* h2h3 [:scheme: https]
* h2h3 [:authority: mirrors.ustc.edu.cn]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5640bbbabce0)
} [5 bytes data]
> GET /apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz HTTP/2
> Host: mirrors.ustc.edu.cn
> user-agent: curl/7.88.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 200 
< server: openresty
< date: Tue, 13 Aug 2024 07:01:06 GMT
< content-type: application/octet-stream
< content-length: 9083702
< last-modified: Mon, 17 Jun 2024 07:44:26 GMT
< etag: "666fe95a-8a9b36"
< accept-ranges: bytes
< 
{ [16236 bytes data]
* Connection #0 to host mirrors.ustc.edu.cn left intact
-: OK

# failed
$ curl -vsSfL -H 'Host:' https://mirrors.ustc.edu.cn/apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz | sha512sum --check <(echo "7d171def9b85846bf757a2cec94b7529371068a0670df14682447224e57983528e97a6d1b850327e4ca02b139abaab7fcb93c4315119e6f0ffb3f0cbc0d0b9a2 -")
*   Trying 198.18.1.141:443...
* Connected to mirrors.ustc.edu.cn (198.18.1.141) port 443 (#0)
* ALPN: offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2063 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [78 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=mirrors.ustc.edu.cn
*  start date: Jun 23 07:17:37 2024 GMT
*  expire date: Sep 21 07:17:36 2024 GMT
*  subjectAltName: host "mirrors.ustc.edu.cn" matched cert's "mirrors.ustc.edu.cn"
*  issuer: C=US; O=Let's Encrypt; CN=E6
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz]
* h2h3 [:scheme: https]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x558107414ce0)
} [5 bytes data]
> GET /apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz HTTP/2
> user-agent: curl/7.88.1
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 400 
< server: openresty
< date: Tue, 13 Aug 2024 06:59:34 GMT
< content-type: text/html; charset=UTF-8
< content-length: 154
* The requested URL returned error: 400
* Connection #0 to host mirrors.ustc.edu.cn left intact
curl: (22) The requested URL returned error: 400
-: FAILED
sha512sum: WARNING: 1 computed checksum did NOT match

Downloading this url in .chezmoiexternal will result in a 403 forbidden error, i think it's the lack of Host header when chezmoi accesses the url that causes the download to fail.

But after I try to modify the source code to set the User-Agent=curl/7.88.1 can be downloaded normally, so should need to provide a way to modify the default request header of the url

chezmoi/internal/chezmoi/sourcestate.go

Line 1586 in 64f40d9

req, err := http.NewRequestWithContext(ctx, http.MethodGet, external.URL, http.NoBody)

req.Header.Set("User-Agent", "curl/7.88.1")

Describe the solution you'd like

Add the configuration headers to .chezmoiexternal as follows

['path/to']
url = 'https://...'
headers."User-Agent" = "curl/7.88.1"
headers.Host = "example.com"

Describe alternatives you've considered

There is currently no way for chezmoi to successfully download this type of url.

[twpayne]

Thanks for reporting this! I think a simpler solution is sufficient:

• mirrors.ustc.edu.cn only requires a User-Agent: header, not Host: header.
• Host: is special anyway as it is part of the HTTP/1.1 protocol and the Go standard library prevents it from being overridden.

So, #3903 sets the User-Agent: header on all HTTP requests that chezmoi makes. The string is chezmoi.io/version to avoid revealing any personal information while still allowing web server operators to know which service is requesting their data.

Would you be able to test this?

[NavyD]

Sure, happy to help with that!

I pulled the latest code from compilation #3903 and apply .chezmoiexternal.toml using the compiled chezmoi binary, and here's a simple test using docker

docker build --load --no-cache --progress plain -f - . <<'DOCK_EOF'
FROM debian:12
ENV HOME=/root
COPY ./chezmoi /usr/local/bin/chezmoi
RUN chmod +x /usr/local/bin/chezmoi && \
    chezmoi init && \
    cat >"$(chezmoi source-path)/.chezmoiexternal.toml" <<'EOF'
[".local/lib/maven"]
url = 'https://mirrors.ustc.edu.cn/apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz'
type = "archive"
stripComponents = 1
exact = true
checksum.sha512 = "7d171def9b85846bf757a2cec94b7529371068a0670df14682447224e57983528e97a6d1b850327e4ca02b139abaab7fcb93c4315119e6f0ffb3f0cbc0d0b9a2"
EOF
RUN [ ! -d ~/.local/lib/maven ] && chezmoi apply --debug && [ -d ~/.local/lib/maven ]
DOCK_EOF

#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 646B done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/debian:12
#2 DONE 0.9s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [1/4] FROM docker.io/library/debian:12@sha256:aadf411dc9ed5199bc7dab48b3e6ce18f8bbee4f170127f5ff1b75cd8035eb36
#4 CACHED

#5 [internal] load build context
#5 transferring context: 31B done
#5 DONE 0.0s

#6 [2/4] COPY ./chezmoi /usr/local/bin/chezmoi
#6 DONE 0.2s

#7 [3/4] RUN chmod +x /usr/local/bin/chezmoi &&     chezmoi init &&     cat >"$(chezmoi source-path)/.chezmoiexternal.toml" <<'EOF'
#7 DONE 0.7s

#8 [4/4] RUN [ ! -d ~/.local/lib/maven ] && chezmoi apply --debug && [ -d ~/.local/lib/maven ]
#8 0.616 time=2024-08-15T10:38:31.467Z level=INFO msg=persistentPreRunRootE version.version="" version.commit=2f7e4306b2588602c63daecf4362caa5f5e4464f-dirty version.date=2024-08-15T00:07:37Z version.builtBy="" args="[chezmoi apply --debug]" goVersion=go1.22.6
#8 0.616 time=2024-08-15T10:38:31.467Z level=ERROR msg=Stat component=system err="stat /root/.config/chezmoi/chezmoistate.boltdb: no such file or directory" name=/root/.config/chezmoi/chezmoistate.boltdb
#8 0.616 time=2024-08-15T10:38:31.467Z level=INFO msg=Stat component=system name=/root/.local/share/chezmoi
#8 0.616 time=2024-08-15T10:38:31.467Z level=INFO msg=Stat component=system name=/root/.local/share/chezmoi/.git
#8 0.616 time=2024-08-15T10:38:31.468Z level=ERROR msg=ReadFile component=system err="open /root/.local/share/chezmoi/.chezmoiroot: no such file or directory" name=/root/.local/share/chezmoi/.chezmoiroot size=0 data=""
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=ReadDir component=system name=/root/.local/share/chezmoi
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=Get component=persistentState bucket=configState key=configState value=""
#8 0.617 time=2024-08-15T10:38:31.469Z level=ERROR msg=ReadFile component=system err="open /root/.local/share/chezmoi/.chezmoiversion: no such file or directory" name=/root/.local/share/chezmoi/.chezmoiversion size=0 data=""
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=Stat component=system name=/root/.local/share/chezmoi
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=Stat component=system name=/root/.local/share/chezmoi
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=ReadDir component=system name=/root/.local/share/chezmoi
#8 0.617 time=2024-08-15T10:38:31.469Z level=INFO msg=ReadFile component=system name=/root/.local/share/chezmoi/.chezmoiexternal.toml size=322 data="[\".local/lib/maven\"]\nurl = 'https://mirrors.ustc.edu.cn/apache/m..."
#8 0.618 time=2024-08-15T10:38:31.470Z level=ERROR msg=Stat component=system err="stat /root/.cache/chezmoi/external/93e8892000f7c2b1b41c1694c349fdd1f19545d4ac806418340fbc66d84429f2: no such file or directory" name=/root/.cache/chezmoi/external/93e8892000f7c2b1b41c1694c349fdd1f19545d4ac806418340fbc66d84429f2
#8 0.820 time=2024-08-15T10:38:31.672Z level=INFO msg=HTTPRequest component=sourceState duration=201.939305ms method=GET url=https://mirrors.ustc.edu.cn/apache/maven/maven-3/3.9.8/binaries/apache-maven-3.9.8-bin.tar.gz statusCode=200 status="200 OK" contentLength=9083702
#8 4.267 time=2024-08-15T10:38:35.119Z level=INFO msg=Mkdir component=system name=/root/.cache/chezmoi/external perm=448
#8 4.285 time=2024-08-15T10:38:35.136Z level=INFO msg=WriteFile component=system name=/root/.cache/chezmoi/external/93e8892000f7c2b1b41c1694c349fdd1f19545d4ac806418340fbc66d84429f2 size=9083702 data="\x1f\x8b\b\x00\x00\x00\x00\x00\x00\xff\xec\xbd\xe9r\x1bI\xb2.\xd8\x7f'\x9f\"\x0f\xef\xdc\x119\x03\x82\x9b\x96*\xa9\xbb\xaeA$$\xa1\x0f\xb7\x06\xc0Ri\xda\xda\xec&\x80\x04\x99%\x00\x89\x93\t\x88\xc2\x19\x9b\xe7\x99..." perm=384
#8 4.285 time=2024-08-15T10:38:35.136Z level=INFO msg=Chtimes component=system name=/root/.cache/chezmoi/external/93e8892000f7c2b1b41c1694c349fdd1f19545d4ac806418340fbc66d84429f2 atime=2024-08-15T10:38:31.470Z mtime=2024-08-15T10:38:31.470Z
#8 5.200 time=2024-08-15T10:38:36.051Z level=INFO msg=WriteFile component=system name=/root/.local/lib/maven/lib/wagon-provider-api-3.5.3.jar size=55101 data="PK\x03\x04\x14\x00\x00\b\b\x005\xa7\x92UN9G\xbf\xad\x00\x00\x00[\x01\x00\x00\x14\x00\x00\x00META-INF/MANIFEST.MF\x8d\x8f\xd1\n\x820\x14\x86\xef\a{\x87\xbd\xc0..." perm=420
# .......
#8 5.203 time=2024-08-15T10:38:36.054Z level=INFO msg=Set component=persistentState bucket=entryState key=/root/.local/lib/maven/lib/wagon-provider-api-3.5.3.jar value="{\n  \"type\": \"file\",\n  \"mode\": 420,\n  \"contentsSHA256\": \"5e72000338945ed3e96f8e4f578d1d0672e1af7e19c0e9014197ae5b31af3ef4\"\n}\n"
#8 5.203 time=2024-08-15T10:38:36.055Z level=INFO msg=Close component=persistentState
#8 DONE 5.2s

#9 exporting to image
#9 exporting layers
#9 exporting layers 0.4s done
#9 writing image sha256:5b625f7cab17ec42ab30a1cd2335a37fe76179847d6fc4d4c44263c6245d7201 done
#9 DONE 0.4s

Thank you very much, it looks like the issue has been resolved!

[twpayne]

Great, thanks for reporting and testing!