add checks to prevent negative outputs of map fst's that will be set to zero by the ReLU activation

[William-Baker]

Currently there is no validation check to prevent negative values being produced by a Map operation, these will get set to 0 by the ReLU in MLP layers and cause craft models to behave unexpectedly.

The simplest example that would cause this is a program like this:

def test_prog():
    SO1 = rasp.Map(lambda x: x - 1, rasp.tokens).annotated(name="SO1", encoding=rasp.Encoding.NUMERICAL)
    return SO1
vocab = [0, 1]
sequence_length = 1

you would expect the model to output -1 and 0 for inputs 0 and 1, but instead the craft model will output 0 and 0.

I have added a validation check within the dynamic program validator and added an exception into the compiler that will throw an error if any negative values are found at the output of the first layer of a Map MLP.

[langosco]

I think the checks should only check for negative values on numerical Maps. Currently it also raises an error on categorical Maps which we don't want, since categorical maps deal fine with negative values.

[langosco]

On another note - I wonder if this is fixable in the compiler? Naively I would think it should be not hard to fix, since rasp.LinearSequenceMap is able to handle negative output values, but I haven't looked at what's going on at the compiler level here.

[William-Baker]

I've now added a check to verify that the map is numerical.
The Sequence Map operation at the very least does require the non-linearity of the ReLU in the dense layers to function. But perhaps a bias could be added in fst before the activation and subtracted in snd?

[david-lindner]

As discussed offline: let's make the check in the basis inference step print a warning instead of raising an exception, and let's add a test case for the validator check.