TLS cert caching error

[cuchaz]

Hello again,

I diagnosed an issue with Ferron today where the cached TLS certificate files became corrupted somehow. All of the certificate_xxxx files in folders with similar random-looking names appear to be ASCII-armored certificates serialized in a JSON format. But today, after doing a server reboot, Ferron failed to start because the JSON files had "trailing characters". The error log for Ferron had a message like this:

Error while obtaining a TLS certificate: trailing characters at line 1 column 3200

I went and inspected the files and indeed, they did have extra characters after the closure of the last JSON object. Those extra characters looked like more JSON syntax, but it was somehow out of sync with the rest of the contents of the file, which was otherwise valid JSON. Removing the trailing characters and restarting Ferron fixed the issue.

I don't know exactly how to reproduce the file corruption, but I suspect somewhere the JSON file was overwritten by Ferron without truncating the old one. Maybe a certificate refresh caused Ferron to update the cached file, and by luck, the new serialized data was smaller than the old data, so not all of the file was overwritten. I think changing the file open flags for cache file writes might fix the issue?

Thanks!

[DorianNiemiecSVRJS]

Thank you for reporting the bug!

I have experienced exactly the same issue related to corrupted ACME cache. This led me to continuing even when the cache is corrupted in the 887f91b commit, and released the change in Ferron 2.3.0. Although this seems to only partially address the issue...

I have created some programs with code with similar open flags to the Ferron's ACME cache writing function. The programs depend Tokio (can be added using cargo add tokio -F full).

Here's the source code for the first program:

use tokio::io::AsyncWriteExt;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut open_options = tokio::fs::OpenOptions::new();
    open_options.write(true).create(true);

    let contents = if tokio::fs::try_exists("/tmp/something.txt").await? {
      b"A bug.\n".to_vec()
    } else {
      b"Something\n".to_vec()
    };

    let mut file = open_options.open("/tmp/something.txt").await?;
    file.write_all(&contents).await?;
    file.flush().await?;
    Ok(())
}

And the execution result:

❯ cargo build -r
    Finished `release` profile [optimized] target(s) in 0.01s
❯ ./target/release/wrongwrite
❯ cat /tmp/something.txt
Something
❯ ./target/release/wrongwrite
❯ cat /tmp/something.txt
A bug.
ng

There's extraneous "ng", because the file wasn't truncated.

And here's the source code for the second program:

use tokio::io::AsyncWriteExt;

#[tokio::main]
async fn main() -> std::io::Result<()> {
    let mut open_options = tokio::fs::OpenOptions::new();
    open_options.write(true).create(true).truncate(true);

    let contents = if tokio::fs::try_exists("/tmp/something.txt").await? {
      b"A bug.\n".to_vec()
    } else {
      b"Something\n".to_vec()
    };

    let mut file = open_options.open("/tmp/something.txt").await?;
    file.write_all(&contents).await?;
    file.flush().await?;
    Ok(())
}

And the execution result:

❯ cargo build -r
    Finished `release` profile [optimized] target(s) in 0.01s
❯ ./target/release/correctwrite
❯ cat /tmp/something.txt
Something
❯ ./target/release/correctwrite
❯
❯ cat /tmp/something.txt
A bug.

There's no extraneous "ng".

I think the ACME cache files got corrupted, because they weren't truncated before writing new data. This resulted in extraneous JSON data (just like the first program I have shown).

I think I have fixed it in the ac10f2a commit, so it truncates the ACME cache files to not corrupt them (just like the second program I have shown). The next version of Ferron 2 is going to have this change.

I don't know exactly how to reproduce the file corruption, but I suspect somewhere the JSON file was overwritten by Ferron without truncating the old one. Maybe a certificate refresh caused Ferron to update the cached file, and by luck, the new serialized data was smaller than the old data, so not all of the file was overwritten. I think changing the file open flags for cache file writes might fix the issue?

I think this is an issue. Yes, I have changed the open flags, so that it behaves more like the second program I have shown rather than the first program.