Symbolic Transformers

[LukasZahradnik]

Additions:

• Transformer modules - Transformer, TransformerEncoder, and TransformerDecoder.
• Attention modules - Attention, MultiheadAttention.
• PositionalEncoding module.
• Softmax as a (splittable) aggregation function.
• Concatenation as an aggregation function.
• Reshape transformation function.
• Slice transformation function.
• Learning rate decay as an optimizer argument.
• Batching support.
• Progress bar for tracking dataset-building progress.
• Special arithmetic predicates for addition (add), subtraction (sub), and modulo (mod).

Changes:

• Concatenation can now be parametrized with a dimension.
• Refactored MLP module - the current version is more compact.
• Changed the behavior of Variable/V factory - only the first letter is capitalized.
• Performance improvements.

---

This discussion was created from the release Symbolic Transformers.

[GustikS]

this will be huge, we really need to find ways to propagate it...

[joaquincabezas]

I was just checking that nothing had changed from October and then I found this! Amazing :)

do you have any result/benchmark on this new feature?

[LukasZahradnik]

@joaquincabezas We only have benchmarks for GNN models, although those need to be updated, as we made a lot of performance improvements. But I'm considering adding benchmarks on transformers.

[joaquincabezas]

I was not aware of this trend! In fact the approach proposed in that paper is completely novel for me. I am still reading it but I think this can work for my PhD also, so instead of a light contribution I can do some heavy work.

[GustikS]

Well this was just put on arxiv a few days ago, not sure it will survive into a longer-term trend, but it is a line of thinking closely related to this framework, the way I see it. But we surely need to start with some concrete examples and use-cases first while improving the UX, I guess it will be a bit clearer once Lukas comes with the tutorial...

[LukasZahradnik]

@joaquincabezas I was thinking about replicating the tutorial that PyTorch has. But that might be too large a dataset for PyNeuraLogic (I'm not sure), so maybe something smaller, like a translation of roman numerals, etc., would be sufficient.

[joaquincabezas]

You're right. But these days to publish something on ML you either need to go into the unknown or assume the risk of becoming obsolete before publishing.

@LukasZahradnik how can I help you with the tutorial?

[joaquincabezas]

Oh my previous answer was for the message #37 (reply in thread) but I clicked the Reply button after a while. So you already answered my question in the past!

I like the roman numbers idea, I saw a sequence2sequence example in Matlab and it also is a clear one for a tutorial