Support for PixelFormats::Gray32Float and PixelFormats::Rgba128Float formats?

[lovettchris]

Description

I'm wondering if you plan to add support for more floating point image formats, along the lines of the UAP PixelFormats::Gray32Float and PixelFormats::Rgba128Float ? I see you have HalfSingle already. How hard would it be to add these missing formats?

Steps to Reproduce

Try converting a PixelFormats::Gray32Float depth map from WriteableBitmap to SixLabors.ImageSharp.Image and back.

System Configuration

• ImageSharp version: 1.0.1
• Other ImageSharp packages and versions: N/A
• Environment (Operating system, version and so on): Windows 10, .netcore3.1
• .NET Framework version: core 3.1
• Additional information: N/A

[lovettchris]

Ah, actually now I see the Rgba128Float is called "RgbaVector"... so I think it's just the Gra32Float that I'm missing?

[JimBobSquarePants]

Thanks @lovettchris

I'm assuming you mean these pixel formats yeah?

PixelFormats.Gray32Float

PixelFormats.Rgba128Float

I don't think our RgbaVector type is an accurate representation actually. It looks like the Rgba128Float format uses the ScRGB gamut which represents a different range -0.5F .. 0.74999F when expressed as a Vector4. Our
RgbaVector type represent a normalized 0F..1F range.

I'm not sure what model to use to translate Gray32Float to/from a Vector4. We use ITU-R Recommendation BT.709 for our 16 and 8bit types.

It's not an insignificant amount of work and we'd need assistance to get specifications mocked out for the individual formats matching our IPixel contract.

One thing we'd have to also add would be some sort of Gamut enumeration property to our PixelTypeInfo type so we don't do things like sRGB companding when operating on ScRGB pixel formats. @antonfirsov @saucecontrol I'd love your input there.

[saucecontrol]

I'm not sure what model to use to translate Gray32Float to/from a Vector4. We use ITU-R Recommendation BT.709 for our 16 and 8bit types.

There's no one correct answer for that, but there's a bit of background here. The approach I take is to use Rec.601 coefficients for luma calculation on companded data, mainly for compatibility with JPEG but also because it's more likely to match user expectations. For linear RGB data, I use Rec.709, which is more correct since its goal was to more closely match luma with luminance, and luminance can actually only be calculated from linear values.

One thing we'd have to also add would be some sort of Gamut enumeration property to our PixelTypeInfo type so we don't do things like sRGB companding when operating on ScRGB pixel formats.

I describe this as PixelValueEncoding in my implementation because it has more to do with the actual encoding of the numerical data. Gamut is defined by the color space, which exists independent of the value encoding. For example, Adobe RGB has a wider gamut than sRGB, but the colors they represent can be encoded as either linear or companded values. They could also be encoded using the scRGB range (or something even wider) in the intermediate to represent out of gamut colors that may be brought back into range with a later filter/adjustment such as HDR tone mapping.

[JimBobSquarePants]

Just had a look at your enum. Yeah, I really like that!

internal enum PixelValueEncoding
{
	Unspecified,
	Companded,
	Linear,
	scRgb
}

We went with 709 because that's what the PNG spec references (and because our jpeg decoder couldn't grayscale then)

[saucecontrol]

~~Actually, I'm not entirely sure I have that right. Really, scRGB is linear, but it's a specific encoding of HDR values in a linear space, optimized for 16-bit storage. Any time you've got 32-bit values you have the headroom to do both linear and HDR encoding, so maybe the correct value for that enum is LinearHdr instead of scRGB. ~~ There's a reason I haven't made those public on my side. ;) It's good to chat about this stuff with you and @antonfirsov to firm up a few of these ideas.

[saucecontrol]

I never noticed PNG specified Rec.709. I think those tables of colors on the wiki page show pretty well how they're both flawed. If you look at the bottom row of the Rec.709 table in particular, I think those are all darker than the reference blue primary even though that grey appears to be closer than the other. I personally prefer the Rec.601 conversions overall (with gamma-corrected data), but I can certainly see how some would prefer the other.

Most of the argument against using Rec.709 coefficients are around the idea it's not really a clear improvement, but it hurts compatibility with anything that uses Rec.601 when round-tripping through a YUV-like encoding since luma adjustments can shift colors when moving between those encodings.

[saucecontrol]

Ignore that earlier comment about scRGB... I confused myself thinking about the fact Windows defines floating-point color data as belonging to the scRGB color space by default (i.e. in the absence of an explicit color profile). PixelValueEncoding is really just about describing how to interpret the numbers that define the pixels. For fixed-point, the default is normally 0=0...2^nbits-1=1. That's almost universal for 8-bit data, with the exceptions being things like YCbCr where the chroma channels have fixed-point values offset to encode the range -0.5..0.5. I use the Unspecified value for those currently, but they might warrant their own entry.

For floating-point data, even though WIC defines that as being scRGB by default, the encoding isn't any different; it's always IEEE 754 as far as I've seen. It's only a question of whether or not values are allowed outside the 0..1 range. And it's most likely linear.

Fixed-point in the 10-bit plus range can be more tricky, with scRGB being one of the more wacky definitions. Windows Photo was the only thing that ever used it AFAIK, so any data you encounter in the wild is much more likely straight 0=0...2^nbits-1=1. Even newer HDR formats use things like HLG to keep values in the 0..1 range.