Publishing symbol graph bson

[wti]

The symbolgraph binary bson files captured during swiftinit builds could be quite useful for third-party analysis.

With the unidoc tool, folks can compile them independently, but making the binaries publicly available would relieve said third parties from that hard work.

If/since the binaries are stored per-package and referenced on each package page, each could be linked from there for manual download.

My own goal is scaling analysis for many packages, so it would be nice to have an officially permitted way to get e.g., 100 or 1000+ packages at a time. I believe the consumer would be responsible for aligning both package and bson versions and comply with size/frequency throttling to avoid egress costs. Because batching could be a bit of work with a different cost/benefit profile, it can be split off as another issue. My own foreseeable need is only for one big batch.

Batch and package users might both need some way to avoid comparing across versions. Each file includes the symbol graph ABI version as a bson field, and the metadata for each package dependency seems to include a version. For the package itself, direct users are typically clicking on a versioned link, but batch users might need a version for the package represented in the bson file (perhaps encoded in the file name?).

For context, please see
https://forums.swift.org/t/documentation-workgroup-meeting-september-9th-2024/74420/7

[tayloraswift]

we recently pushed an update to the site that adds a Download link under the “…” menu next to each symbol graph entry on the tags pages.

please note that these files are zlib-compressed, so you will need to unarchive them using something like this:

swift-unidoc/Sources/UnidocLinkerPlugin/AWS.S3.GraphLoader.swift

Lines 32 to 36 in 9229838

	case .bson_zz:
	// https://github.com/apple/swift/issues/71605
	var inflator:LZ77.Inflator = .init(format: .zlib)
	try inflator.push((/* consume */ bytes)[...])
	bytes = inflator.pull()

[wti]

Thanks! I can download the bson (menu ... visible only after login) and load the SymbolGraph...

let source = try Data(contentsOf: graphUrl) 
try source.withUnsafeBytes { sourcePtr in
  var inflator: LZ77.Inflator = .init(format: .zlib)
  sourcePtr.withMemoryRebound(to: UInt8.self) { sourceBytes in
    let slice: ArraySlice<UInt8> = .init(sourceBytes)
    try inflator.push(slice)
  }
  let bytes = inflator.pull()
  let bson: BSON.Document = .init(bytes: bytes[...])
  return try UDSymbolGraph(bson: bson)
}

Using packages swift-mongodb (bson), swift-unidoc (SymbolGraph), and swift-png (LZ77)