Parsing whitespace to demarcate blocks

[DVaughan]

I'm working on parsing a language that uses a two space indent to denote blocks, rather like Python or YAML. I'm wondering if someone can point me to an example of parsing this kind of thing with Superpower.

Thanks,
Daniel

[nblumhardt]

Hey Daniel, thanks for dropping by!

It's a bit of a tough question, I'm not sure just how well this fits into Superpower's model, but it seems like it's worth a shot.

I haven't attempted it myself, but this is how I'd first try tackling it. Whipping this up in a hurry - apologies if it's a bit half-baked, would really love to hear how it goes.

1. Custom token kinds

Superpower parsers are almost always written around enums, but (unless we've broken anything in the last few versions 😅 ) it's possible to use any arbitrary token type. I'd try classes:

[Token(Description = "number")]
class Number: YourToken { }

[Token(Description = "indent")]
class Indent: YourToken
{
    public int Depth { get; }
    public Indent(int depth) { Depth = depth }
}

2. Custom tokenizer

There's an example of one of these in the repo (test cases perhaps)? The idea is to discard insignificant whitespace, while keeping track of what whitespace is significant (the good old imperative way :-)) and yielding up new Indent(depth) as appropriate.

Imagining this is a fragment of your source text:

  loop:
    print 'hi!'

The tokenized representation will be something like [new Indent(2), new LoopKeyword(), new Colon(), new Indent(4), new PrintKeyword(), new String("hi!")].

3. Context-sensitive parser

Here's where things get a bit awkward; expression-y things are fine and you'd write them like any other Superpower parser. Imagine you've created those rules as static members in an ExpressionParsers class (ExpressionParsers.Number, ExpressionParsers.String, operators, etc.)

StatementParsers is different, it's parameterized by requiredIndent. Things get started as:

var parsers = new StatementParsers(0);

// StatementParsers ctor:
public StatementParser(int requiredIndent)
{
    _requiredIndent = new Indent(requiredIndent);
}

In StatementParsers, rules like Statement, Loop and Block (implicitly) depend on the _requiredIndent field:

public TokenListParser<YourToken, Statement> Statement { get; } = Token.EqualTo(_requiredIndent).IgnoreThen(Loop.Or(...));

public TokenListParser<YourToken, Statement[]> Block { get; } = Statement.AtLeastOnce();

public TokenListParser<YourToken, Loop> = 
    from loop in Token.EqualTo(new LoopKeyword())
    from colon in Token.EqualTo(new ColonToken())
    from body in new StatementParsers(_requiredIndent.Indent + 2).Block
    return new Loop(body);

Lots of mechanics here to figure out - probably much nicer making stateless tokens like LoopKeyword into constants somewhere, etc., - but should work ---- as far as I can tell :-)

Hope this helps, good luck with it!
Nick

[DVaughan]

Thanks Nicholas! I hadn't considered using a custom class for the token.
In my first draft I took to using the Position property of the token. I was hacking away at it last night and came up with something like this:

from definition in ClassDeclaration			
	let requiredColumn = definition.Position.Column + 2
	from foo in 
		(from nl in Token.EqualTo(TokenKind.NewLine).AtLeastOnce()
		 from pe in PropertyEquals.Where(x => x.IdentifierSyntax.Position.Column >= requiredColumn).Many()
		 where (pe == null || !pe.Any()) && pe.First().IdentifierSyntax.Position.Column == requiredColumn select pe)
		.ManyDelimitedBy(Token.EqualTo(TokenKind.NewLine))
                ...

Sorry that this snippet is a little convoluted and half baked.
I'm ignoring spaces in the tokenizer.

Btw thanks for Superpower, it's awesome.

[nblumhardt]

Oh, that's a promising approach! I had forgotten that Token tracks columns. Fingers crossed it'll all come together that way 🤞