from-151-Ternary-dense-try-different-normalization-to-optimize-outcome

[david-thrower]

Kind of issue: Feature development

Issue described: We have a successful implementation of a Ternary replacement for Dense layers. The metrics are not quite what we want on some problems.

One possible issue is the default batch normalization behavior. Perhaps Dropout will work better or layer normalization.

[david-thrower]

Tried thus far:

• No normalization nor dropout: Results: Basically no good RMSE on tabular = Mean value
• Layer Normalization: Some improvement over no normalization but worse than default batch normalization on tabular. However, exceptional results on Ham / Spam problem. The disparity between the results on tabular and text classification is strange. It may be that the hyperparameter range for tabular are just way off for the Ternary neural network. It could also be that we should make a different node for Ternary and Dense and continue using Dense with batch normalization for tabular and use Ternary with Layer Normalization for NLP tasks. There is probably something that will make both work.

To try:

• Dropout .10
• Dropout 0.5
• Dropout 0.8