Why do control characters disappear?

[chris-martin]

I noticed this while attempting to write a round-trip property test for an HTML representation of a markup datatype, which is using Lucid for the rendering portion. I find that when text contains a control character (like �) Lucid's output omits the character entirely, whereas I would expect it to print the escape sequence (like ).

ghci> import Lucid

ghci> import Data.Text.Lazy qualified as TL

ghci> TL.length (renderText (toHtml ("\ACK" :: TL.Text)))
0

Is this considered a bug, or are my expectations wrong?

[chrisdone]

Consulting ChatGPT and an HTML3 spec say that control characters are to be avoided due to having no visible representation and parsing issues. Presumably the blaze-builder which lucid is based on made the choice to filter them out. Not a bad choice, think. I believe even ampersand encoding would be considered sketchy due to having no visible representation. Quote from spec:

In SGML applications, the use of control characters is limited in order to maximize the chance of sucessful interchange over heterogenous networks and operating systems. In HTML, there are only three control characters which are used. The remaining 55 control characters are shunned and should not appear in an HTML document.

I don’t consider GPT an authority, but it echoes what I found on w3c.org, so that’s something.

Source, c>=' ':

condB (\c -> c >= ' ' || c == '\t' || c == '\n' || c == '\r')
          (P.liftFixedToBounded P.char7) $
    …

[chris-martin]

Thanks Chris, I appreciate it.