Specify log directory path of structured logs

[weihanglo]

Problem

Currently, when -Zbuild-analysis is enabled, Cargo writes log files, to ~/.cargo/log directly. This is good enough for a prototype but not ideal for stabilization.

We would like to have a way to configure the location of log files, and perhaps also have a way to write to stdout. Here are some use cases:

• feat(log): emit build-started JSON message with run_id  #16632 (comment) — read-only file system that can't write to CARGO_HOME. For example, crater runs Rust crates in a Docker container where CARGO_HOME is mounted as read-only.
• Allow to specify the output file for --timings #12898 — Some build orchestrators are opinionated and restricted in where to store and find (log) files, as they want to avoid host side effect. Customizing log directory gives the flexibility. (this is related to Associate --message-format=json with session ID from -Zbuild-analysis #16576)
• IDE tools may want to have a dedicated log directory under their control, rather than overriding CARGO_HOME entirely.

Proposed Solution

The easiest way to implement it is adding a config file to specify log directory location to the [build.analysis] table. For example,

[build.analysis]
enabled = true
log-dir = "<directory-path>"

However, we have discussed that the structured logging might be more general that build analysis, and might be renamed to the [log] table. The field name log-path feels redundant when under log table. (Another issue of using [log] table is that it might conflict with CARGO_LOG env var, though that is a bit out of scope here).

We also need to consider if we want to support streaming to stdout, for example, a special - to stream to stdout.

Notes

No response

[epage]

Could you summarize the use cases, rather than deferring them out to other discussions? Overall, I'm skeptical of this being in an MVP. Feels like it would be helpful to stabilize the minimum so we have flexibility to evolve this.

We also need to consider if we want to support streaming to stdout, for example, a special - to stream to stdout.

imo that is what --message-format json is.

[weihanglo]

Expanded the PR description a bit. Thanks!

We also need to consider if we want to support streaming to stdout, for example, a special - to stream to stdout.

imo that is what --message-format json is.

Thanks for reminding me that. Yeah I feel like every piece is related. We'll need to consider situations like log.log-dir and --message-format=json are both set. Having a log-dir config might be weird if it is silently ignored.

[epage]

I view log and --message-format as orthogonal sinks and not one replacing the other.

[epage]

read-only file system that can't write to CARGO_HOME. For example, crater runs Rust crates in a Docker container where CARGO_HOME is mounted as read-only.

I worry that this motivation is playing whack-a-mole

Some build orchestrators are opinionated and restricted in where to store and find (log) files, as they want to avoid host side effect. Customizing log directory gives the flexibility. (this is related to #16576)
IDE tools may want to have a dedicated log directory under their control, rather than overriding CARGO_HOME entirely.

In these cases, cargo then won't know about the logs unless you always specify them.

Is an export feature more appropriate?

Is solving these cases part of the MVP?

[weihanglo]

I view log and --message-format as orthogonal sinks and not one replacing the other.

Totally agree, though they are still related in terms of the config we exposed. For example, right now build.analysis.enabled = true emits logs, and we have plan to emit log message via --message-format=json-something. However, we lack design of only emitting to standard stream bug not files (prior effort #16418). I think two sinks should still be designed together, at least figure out how to separate the concerns.

[weihanglo]

Some build orchestrators are opinionated and restricted in where to store and find (log) files, as they want to avoid host side effect. Customizing log directory gives the flexibility. (this is related to #16576)
IDE tools may want to have a dedicated log directory under their control, rather than overriding CARGO_HOME entirely.

In these cases, cargo then won't know about the logs unless you always specify them.

Is an export feature more appropriate?

Is solving these cases part of the MVP?

I don't think this needs to be part of the MVP. Part of my motivation was to fill in the blank of the removed --timings=json, especially when seeing crater is involved. For rust-lang projects we usually have a fast track with a temporary solution. Anyway, I don't think we need to push forward for a half baked solution now. Maybe the "everything-goes-to-stdout" fulfill most of the needs here.

[epage]

However, we lack design of only emitting to standard stream bug not files (prior effort #16418).

Wouldn't that be --message-format=json --config build.analyzis.enabled = false

[weihanglo]

However, we lack design of only emitting to standard stream bug not files (prior effort #16418).

Wouldn't that be --message-format=json --config build.analyzis.enabled = false

yes but we'll need to talk about --message-format=jsonv2 or different options of message format, and whether we want to emit everything by default or just essential one. And whether enabled=false should disable all analysis and build timing collection.

[epage]

yes but we'll need to talk about --message-format=jsonv2 or different options of message format, and whether we want to emit everything by default or just essential one. And whether enabled=false should disable all analysis and build timing collection.

I feel like the first question to address is: are we ok with --message-format and log emitting everything all the time? If not, we need to understand that to understand how and where we might want to split this up.