WIP: Add SPPF Visitor #96

stevefan1999-personal · 2023-10-15T09:15:20Z

This PR implements part of the idea expressed here: #88 (comment)

I have another obscure idea I can't express clearly (I'm not sure if that's novel or not) that if we want to implement semantic checker to multiple "trees"/"branches" of the SPPF at the same time, at the very least, the data structure state holding the information about the current AST walker must be immutable -- which means data is cloned between each forest transition. Then we will need to use persistent data structure/Copy-On-Write mechanism/transactional memory to manage the state data. While I'm pretty sure functional programming people may have enjoyed writing parsers this way because their way to handle data structures are immutable by design, so this is trivial for all kinds of program, but this idea came back to the time when I was still using rust-peg (kevinmehall/rust-peg#301)

TODO:

Figure out how to emit the divergent visitors on different versions
1. This can be done using a vector, but I think iterator would be okay too
Figure out how to do tree pruning
1. In case if semantic checker determined this tree is invalid, so that we can do an early exit for this tree anyway (feels like branch and bound/backtrack to me)

…ions

…999-personal/hime into patch-sppf-visitor-gen

stevefan1999-personal · 2023-10-15T13:45:25Z

Seems like there is a bit of a trouble on the cloning part. I'm not sure why the visitors are not following...

ASTWalker { test: Vector { root: Leaf(["function_definition"]), bits: 5, length: 1 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement"]), bits: 5, length: 2 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list"]), bits: 5, length: 3 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list"]), bits: 5, length: 4 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list"]), bits: 5, length: 5 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "declaration"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "expression_statement"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "expression_statement", "expression"]), bits: 5, length: 9 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "declaration"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "expression_statement"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "expression_statement", "expression"]), bits: 5, length: 9 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "jump_statement"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "block_item_list", "block_item_list", "block_item_list", "block_item", "statement", "jump_statement", "expression"]), bits: 5, length: 9 } }

#[derive(Clone, Debug)]
struct ASTWalker {
    test: VectorSync<&'static str>
}

impl c::AstVisitor for ASTWalker {
    fn on_variable_declaration(&mut self, node: &AstNode) {
        self.test = self.test.push_back("declaration");
        println!("{:?}", self);
    }

    fn on_variable_expression(&mut self, node: &AstNode) {
        self.test = self.test.push_back("expression");
        println!("{:?}", self);
    }

    fn on_variable_block_item(&mut self, node: &AstNode) {
        self.test = self.test.push_back("block_item");
        println!("{:?}", self);
    }

    fn on_variable_compound_statement(&mut self, node: &AstNode) {
        self.test = self.test.push_back("compound_statement");
        println!("{:?}", self);
    }

    fn on_variable_block_item_list(&mut self, node: &AstNode) {
        self.test = self.test.push_back("block_item_list");
        println!("{:?}", self);
    }

    fn on_variable_function_definition(&mut self, node: &AstNode) {
        self.test = self.test.push_back("function_definition");
        println!("{:?}", self);
    }

    fn on_variable_statement(&mut self, node: &AstNode) {
        self.test = self.test.push_back("statement");
        println!("{:?}", self);
    }

    fn on_variable_expression_statement(&mut self, node: &AstNode) {
        self.test = self.test.push_back("expression_statement");
        println!("{:?}", self);
    }

    fn on_variable_jump_statement(&mut self, _node: &AstNode) {
        self.test = self.test.push_back("jump_statement");
        println!("{:?}", self);
    }

}

impl c::SppfVisitor for ASTWalker {
    fn on_variable_declaration(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("declaration");
        println!("{:?}", self);
    }

    fn on_variable_expression(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("expression");
        println!("{:?}", self);
    }

    fn on_variable_block_item(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("block_item");
        println!("{:?}", self);
    }

    fn on_variable_compound_statement(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("compound_statement");
        println!("{:?}", self);
    }

    fn on_variable_block_item_list(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("block_item_list");
        println!("{:?}", self);
    }

    fn on_variable_function_definition(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("function_definition");
        println!("{:?}", self);
    }

    fn on_variable_statement(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("statement");
        println!("{:?}", self);
    }

    fn on_variable_expression_statement(&mut self, node: &SppfNodeVersion) {
        self.test = self.test.push_back("expression_statement");
        println!("{:?}", self);
    }

    fn on_variable_jump_statement(&mut self, _node: &SppfNodeVersion) {
        self.test = self.test.push_back("jump_statement");
        println!("{:?}", self);
    }

}

fn main() {
    let result = c::parse_str_to_sppf(
        r#"

int main() {
    T * a;
    U * b;
    return a * b;
}
    "#);

    let mut walker: Box<dyn SppfVisitor> = Box::new(ASTWalker { test: Vector::new_sync() });
    c::visit_sppf(&result, &mut walker);
}

I'm using the C grammar here: #88 (comment)

I'm also using rpds for VectorSync (a persistent data structure that supports O(1) clone)

stevefan1999-personal · 2023-10-15T14:08:34Z

Seems like the backlink to the next AST node on divergent version was not present.

I also find it curious that I can get an expected result using the optimized grammar here: #88 (comment)

ASTWalker { test: Vector { root: Leaf(["function_definition"]), bits: 5, length: 1 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement"]), bits: 5, length: 2 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration"]), bits: 5, length: 3 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "declaration"]), bits: 5, length: 4 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "declaration", "statement"]), bits: 5, length: 5 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "declaration", "statement", "jump_statement"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "declaration", "statement", "jump_statement", "expression"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement"]), bits: 5, length: 2 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement"]), bits: 5, length: 3 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement"]), bits: 5, length: 4 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression"]), bits: 5, length: 5 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "declaration"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "declaration", "statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "declaration", "statement", "jump_statement"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "declaration", "statement", "jump_statement", "expression"]), bits: 5, length: 9 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement"]), bits: 5, length: 2 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration"]), bits: 5, length: 3 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement"]), bits: 5, length: 4 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement", "expression_statement"]), bits: 5, length: 5 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement", "expression_statement", "expression"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement", "expression_statement", "expression", "statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement", "expression_statement", "expression", "statement", "jump_statement"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "declaration", "statement", "expression_statement", "expression", "statement", "jump_statement", "expression"]), bits: 5, length: 9 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement"]), bits: 5, length: 2 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement"]), bits: 5, length: 3 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement"]), bits: 5, length: 4 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression"]), bits: 5, length: 5 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement"]), bits: 5, length: 6 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement", "expression_statement"]), bits: 5, length: 7 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement", "expression_statement", "expression"]), bits: 5, length: 8 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement", "expression_statement", "expression", "statement"]), bits: 5, length: 9 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement", "expression_statement", "expression", "statement", "jump_statement"]), bits: 5, length: 10 } }
ASTWalker { test: Vector { root: Leaf(["function_definition", "compound_statement", "statement", "expression_statement", "expression", "statement", "expression_statement", "expression", "statement", "jump_statement", "expression"]), bits: 5, length: 11 } }

…r-gen

stevefan1999-personal added 12 commits October 15, 2023 16:07

Add preliminary SPPF visitor code

8b0f7f0

disambiguate the visit function name

1cba0d5

fix include item for sppf

7c5c6a0

use dyn-clone to clone visitor on different versions

5c95aaf

fixup! use dyn-clone to clone visitor on different versions

d85ce86

fixup! fixup! use dyn-clone to clone visitor on different versions

5d0c482

fixup! fixup! fixup! use dyn-clone to clone visitor on different vers…

b313a09

…ions

use dyn-clone to clone visitor on different versions

0b92596

Merge branch 'patch-sppf-visitor-gen' of https://github.com/stevefan1…

2d1cdfc

…999-personal/hime into patch-sppf-visitor-gen

fix ownership problem for sppf visitor

cd17d53

fix indent

da1e260

add missing reference

73535fd

stevefan1999-personal mentioned this pull request Oct 15, 2023

Tree felling for SPPF #97

Open

stevefan1999-personal added 5 commits October 15, 2023 17:52

let visitor to be mutable

25a23f0

do pre-order traversal instead for sppf

1205f4b

support mut for ast visitor

03dfa8b

use clone_trait_object instead

4e88caf

fixup! use clone_trait_object instead

39dd2cd

stevefan1999-personal added 3 commits October 16, 2023 00:21

push tree to a buffer on divergent branch

13e227e

Merge remote-tracking branch 'upstream/master' into patch-sppf-visito…

bf87633

…r-gen

use generated constants for id matching

1685de8

stevefan1999-personal mentioned this pull request Oct 17, 2023

SPPF spanning tree iterator #100

Open

stevefan1999-personal marked this pull request as ready for review December 19, 2023 05:12

stevefan1999-personal marked this pull request as draft December 19, 2023 05:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add SPPF Visitor #96

WIP: Add SPPF Visitor #96

stevefan1999-personal commented Oct 15, 2023 •

edited

Loading

stevefan1999-personal commented Oct 15, 2023

stevefan1999-personal commented Oct 15, 2023 •

edited

Loading

WIP: Add SPPF Visitor #96

Are you sure you want to change the base?

WIP: Add SPPF Visitor #96

Conversation

stevefan1999-personal commented Oct 15, 2023 • edited Loading

stevefan1999-personal commented Oct 15, 2023

stevefan1999-personal commented Oct 15, 2023 • edited Loading

stevefan1999-personal commented Oct 15, 2023 •

edited

Loading

stevefan1999-personal commented Oct 15, 2023 •

edited

Loading