Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce MLNumber for specifying numeric inputs of any type #647

Merged
Merged
Changes from 11 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
f83ecfd
Introduce MLNumber for specifying numeric inputs of any type
inexorabletash Apr 9, 2024
29d5191
Merge remote-tracking branch 'origin/main' into bigint-numeric-union
inexorabletash Apr 19, 2024
67b5a68
Add note about fused activations
inexorabletash Apr 19, 2024
adb5742
Merge remote-tracking branch 'origin/main' into bigint-numeric-union
inexorabletash Apr 24, 2024
a36d97f
Merge remote-tracking branch 'origin/main' into bigint-numeric-union
inexorabletash Apr 25, 2024
8a8b3d8
Merge remote-tracking branch 'origin/main' into bigint-numeric-union
inexorabletash Apr 25, 2024
dc655a6
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Apr 27, 2024
c47b31d
Merge remote-tracking branch 'origin/main' into bigint-numeric-union
inexorabletash Apr 28, 2024
88a29b1
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Apr 29, 2024
bf32b3f
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Apr 30, 2024
6072331
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 1, 2024
c873c35
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 2, 2024
07aaedc
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 3, 2024
84b4cfc
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 6, 2024
4c2a041
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 7, 2024
3389f87
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 7, 2024
35453d6
Capitalize WebIDL ref
inexorabletash May 7, 2024
fc0a090
Added more explanatory text around processing
inexorabletash May 7, 2024
a3ff823
Add MLFiniteNumber
inexorabletash May 7, 2024
aa5d31f
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 9, 2024
28707fe
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 9, 2024
5061a8c
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 10, 2024
0452b10
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 13, 2024
ab54710
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 14, 2024
9f0dd89
WIP for https://github.com/webmachinelearning/webnn/issues/678
inexorabletash May 8, 2024
588f7ac
Settle on using 'double' instead of 'float' per #325
inexorabletash May 14, 2024
6278afc
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 15, 2024
a357c10
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 24, 2024
98a6263
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash May 30, 2024
77816a3
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 3, 2024
4f1ebc9
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 4, 2024
c972a88
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 5, 2024
f2a7de3
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 5, 2024
4a58736
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 8, 2024
83a8c46
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 10, 2024
7c9fc64
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 13, 2024
47d8b6c
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 17, 2024
f0e6f85
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 18, 2024
6c7215a
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 25, 2024
5f313b0
Update example MLActivation
inexorabletash Jun 25, 2024
2cce26e
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 26, 2024
d7f8da4
Remove obsolete note
inexorabletash Jun 26, 2024
3120474
resolve lint errors
inexorabletash Jun 26, 2024
024f576
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 27, 2024
e5345fb
Merge branch 'refs/heads/draft' into bigint-numeric-union
inexorabletash Jun 27, 2024
18dd7e3
Fix upperBound and lowerBound values in ConvertToFloat
inexorabletash Jun 27, 2024
6f75f2b
Always clamp, unrestricted FP, and no failures.
inexorabletash Jun 26, 2024
b5c7d4b
Remove MLActivation text for MLNumber as it is unused
inexorabletash Jul 1, 2024
1934d25
Parenthesize expression
inexorabletash Jul 3, 2024
c353079
Restore MLNumber reference lost in merge
inexorabletash Jul 3, 2024
a5f299d
Align pad() with constant(), remove MLFiniteNumber
inexorabletash Jul 3, 2024
2e06299
Revert change re: scalar
inexorabletash Jul 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 19 additions & 5 deletions index.bs
Original file line number Diff line number Diff line change
Expand Up @@ -1074,6 +1074,8 @@ interface MLOperand {
MLOperandDataType dataType();
sequence<unsigned long> shape();
};

typedef (bigint or unrestricted double) MLNumber;
</script>

<div class=internal-slots>
Expand Down Expand Up @@ -1133,6 +1135,18 @@ The {{MLOperand}} objects are created by the methods of {{MLGraphBuilder}}, inte
To <dfn for="MLGraphBuilder">validate operand</dfn> given {{MLGraphBuilder}} |builder| and {{MLOperand}} |operand|, return true if |operand|.{{MLOperand/[[builder]]}} is |builder|, and false otherwise.
</p>

#### {{MLNumber}} #### {#api-mlnumber-typedef}

<dfn typedef>MLNumber</dfn> is used when specifying the type of a numeric option for an {{MLOperand}} which can be of any {{MLOperandDataType}}, including both 64-bit integer types ({{MLOperandDataType/"uint64"}} and {{MLOperandDataType/"int64"}}) and 32-bit floating point ({{MLOperandDataType/"float32"}}). Implementations must process the value according to the corresponding {{MLOperandDataType}}.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementations must process the value according to the corresponding {{MLOperandDataType}}.

What if there is no corresponding MLOperandDataType? For example, clamp() can be built as an MLActivation without needing to specify any particular dtype. What is the dtype or the activation's associated operator?

The concept of an activation's operator is a bit hand-wavy in the spec, but it's very concrete in the Chromium implementation and the data type must be known when we pass the activation as an input to some other builder method anyways (we need to check that the types match, assuming we don't want to allow mixed precision #283 (comment))

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably the data type of the eventual operator's input? i.e. for the impl we'd need to hold onto the union in the MLActivation until the operator is constructed?

We need to improve the spec text, but I want to understand the implementation first.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably the data type of the eventual operator's input?

Fused activations should use operator's output as its own input, although the output's data type of operators that support fusion is usually the same as the input's.

i.e. for the impl we'd need to hold onto the union in the MLActivation until the operator is constructed?

I think so.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a short note in 67b5a68 but we can probably improve it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i.e. for the impl we'd need to hold onto the union in the MLActivation until the operator is constructed?

I think so.

Does this make sense? What happens if an MLActivation is passed to multiple operators with different data types?

Copy link
Member Author

@inexorabletash inexorabletash Apr 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minValue <= maxValue is particularly interesting depending on the conversions. If you pass {minValue: -1, maxValue: 128} this looks valid, but if the data type ends up being "uint8", does this turn into {minValue: 0xFF, maxValue: 0x80} ? Ooof.

with the idea that an MLActivation has one operator slot

Agreed that "the whole "operator" concept is pretty hand-wavy anyways". An MLActivation's internal [[operator]] slot is not the specific operator instance of the MLOperand that the activation is fused with, and is probably more like an operator type than a specific instance. So I don't think there's a conflict, but we could definitely improve the text - and explicitly mention that MLActivations can be re-used when creating multiple MLOperands and even with different data types.

Copy link
Member Author

@inexorabletash inexorabletash Apr 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be explicit, here's where I'm envisioning the conversion/validation happens:

  • for constant() - immediate conversion based on explicitly passed MLOperandDataType
  • for MLOperand clamp() - immediate conversion based on input's data type
  • for MLActivation clamp() - when the activation is passed to an operand-vending method
  • for pad() - immediate conversion based on input's data type

See #649 (comment) for a framework where activation validation could be plugged in.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question:
Does this mean for example for gemm, if the input a and b are passed as float16, and if you pass
{alpha: MLNumber(3.14123452435)}, it will automatically convert that to float16 as 3.141?

Copy link
Member Author

@inexorabletash inexorabletash May 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess under the hood - yes (although it'd be just {alpha: 3.14123452435})

Presumably that's what the spec/implementation implicitly does today, even if we didn't introduce MLNumber?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under the hood it preserves the fp32 precision right now (as what you are suggesting).
But I'd prefer we explicitly set the expectation that it will be casted to the precision to be the same as the input type.
In the gemm example, since it's emulated as alpha * matmul(a,b) + beta * c in CoreML, and CoreML requires all these binary ops to have the same type. so if a, b are fp16, then it's better to cast alpha to fp16 to multiply them. Otherwise we need to cast input to fp32 (which doesn't get executed on ANE).

So right now we just don't support fp16 inputs for gemm on CoreML until this is sorted out.


When used with an {{MLActivation}}, the data type is not known until the activation function is fused into a specific operation. Implementations must process the value according to the corresponding {{MLOperandDataType}} of the output of the fused operation.

<div class="note">
Specifying the option as {{double}} would lose accuracy when passing values over 2<sup>53</sup>, and specifying {{long long}} would disallow values over 2<sup>63</sup>.
</div>

Issue(whatwg/webidl#1388): Support for unions of {{bigint}} and [=numeric types=] new in [[WebIDL]], and implementation support is also limited. Prototype implementations are encouraged to provide feedback for this approach.

### {{MLOperand/dataType()}} ### {#api-mloperand-datatype}
Return a data type of the {{MLOperand}}.

Expand Down Expand Up @@ -1232,7 +1246,7 @@ interface MLGraphBuilder {
MLOperand constant(MLOperandDescriptor descriptor, ArrayBufferView bufferView);

// Create a single-value operand from the specified number of the specified type.
inexorabletash marked this conversation as resolved.
Show resolved Hide resolved
MLOperand constant(double value, optional MLOperandDataType type = "float32");
MLOperand constant(MLNumber value, optional MLOperandDataType type = "float32");

// Compile the graph up to the specified output operands asynchronously.
Promise<MLGraph> build(MLNamedOperands outputs);
Expand Down Expand Up @@ -1325,7 +1339,7 @@ Data truncation will occur when the specified value exceeds the range of the spe

<div>
**Arguments:**
- *value*: a {{float}} number. The value of the constant.
- *value*: an {{MLNumber}}. The value of the constant.
- *type*: an optional {{MLOperandDataType}}. If not specified, it is assumed to be {{MLOperandDataType/"float32"}}.
**Returns:** an {{MLOperand}}. The constant output.
</div>
Expand Down Expand Up @@ -1610,8 +1624,8 @@ partial interface MLGraphBuilder {
Clamp the input tensor element-wise within a range specified by the minimum and maximum values.
<script type=idl>
dictionary MLClampOptions {
float minValue;
float maxValue;
MLNumber minValue;
MLNumber maxValue;
inexorabletash marked this conversation as resolved.
Show resolved Hide resolved
};

partial interface MLGraphBuilder {
Expand Down Expand Up @@ -4442,7 +4456,7 @@ enum MLPaddingMode {

dictionary MLPadOptions {
MLPaddingMode mode = "constant";
float value = 0;
MLNumber value = 0;
inexorabletash marked this conversation as resolved.
Show resolved Hide resolved
};

partial interface MLGraphBuilder {
Expand Down