imdl torrent link creates invalid magnet

[angrynode]

The trackers in the generated magnet link are not url-encoded and so cannot be parsed properly by other software.

We've had the question of xt params in this upstream issue but i think so far i've only seen unencoded ":" in urn:btih:... so that should be fine.

But the tracker URLs are usually url-encoded. I don't know of a counter-example.

[angrynode]

I think #554 is on good track for merging, but before closing this issue we should take time and analyze other magnet URIs to make sure other fields should not be url-encoded.

For example should the : in peer addresses be encoded as %3A?

[casey]

I'm actually a bit confused about this. From my understanding of this wikipedia article, reserved characters only have to be percent encoded in URL contexts where they have a special meaning.

So, / must be percent-encoded in a path component, but does not need to be percent-encoded in a query parameter or fragment.

The whatwg URL Standard seems to agree with this. They list the characters which must be percent encoded in a query parameter as the control characters, bytes above 0x7E (I.e. non-ascii), space, ", #, <, and >.

So, I don't think that :, /, or other characters not included in the above need to be encoded.

[casey]

You mentioned in the original comment that magnet links generated by intermodal were incompatible with some software. Which software/versions did you run into problems with?

[angrynode]

The problem is what spec are we using? I have the same reading of that wikipedia page as you do, but the specs say otherwise.

Quoting RFC 3986 Section 2.2, as referenced by HTML5 spec:

URI producing applications should percent-encode data octets that correspond to characters in the reserved set unless these characters are specifically allowed by the URI scheme to represent data in that component. If a reserved character is found in a URI component and no delimiting role is known for that character, then it must be interpreted as representing the data octet corresponding to that character's encoding in US-ASCII.

So : and / must be parsed as such (by a valid implementation), but should not be produced.

You mentioned in the original comment that magnet links generated by intermodal were incompatible with some software. Which software/versions did you run into problems with?

I'm sorry i'm not so sure anymore. I'm testing so many clients and weird edge-cased torrents. I do remember i first encountered this issue when parsing a magnet with an IPv6 literal tracker (tr=http://[::1]:3000/announce) which drew my attention that all magnet links i saw online were percent-encoded (except for the urn:btih part). (then i added tests in my own magnet parser which also failed)

QBittorrent and Transmission "copy magnet" feature is also percent-encoded. I just think that's the correct thing to do, but i'm no URL or torrent expert so feel free to have an opposite opinion :)

[casey]

I think RFC 9686 is conditional:

If data for a URI component would conflict with a reserved character's purpose as a delimiter, then the conflicting data must be percent-encoded before the URI is formed.

So only if, in that component, the reserved character is used as a delimiter does it need to be percent encoded.

Digging around the spec, these are the relevant parts of the grammar for a query:

query = *( pchar / "/" / "?" )
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
unreserved  = ALPHA / DIGIT / "-" / "." / "_" / "~"
pct-encoded = "%" HEXDIG HEXDIG
sub-delims  = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "="

Which seems to confirm that, since, for example, / and : are explicitly allowed, even though they are reserved characters in other parts of a URL. (& and = are both allowed, but have to be percent encoded, since they're used to delimit query items and values within the query string.)

[casey]

I opened #556 (tests till failing, still a draft) with percent encoding based on RFC 9686. I think this is correct, but if there's a popular piece of software out there which can't handle magnet links with the more permissive percent encoding, then that would definitely be cause to do something different, so specific examples are very welcome.

[angrynode]

After some thought i don't think it was actually a torrent client that failed to parse. I think it was fluent_uri crate failing to parse the IPv6 literal which prompted me to investigate. That's probably a bug on their side so i may have overreacted in my initial bug report, sorry for that.

Still, magnet links i find around are all percent-encoded so it doesn't seem wrong to do this. (i still haven't found a magnet link where the trackers were not percent-encoded)

Your branch on #556 looks good from my perspective.

[casey]

I just merged #556, which tries to be as lenient as possible, and only percent encode things which absolutely must be percent encoded. It probably doesn't do any harm to percent encode more, but it produces less readable links, so it's nice to encode as little as possible.

No worries at all! This was definitely a bug, since there are tons of things that we should have been percent encoding all along.