Add a rudimentary parser (#5)

* Add the parser struct

* Start the AST to get basic types for the parser

* Added clean command to Makefile

The update introduces a 'clean' command in the Makefile. This new command removes the '.make' directory and 'cover.profile' file, helping to maintain a cleaner workspace by deleting unnecessary files generated during build or test processes.

* Enhanced AST with new structures and methods

The Abstract Syntax Tree (AST) has been significantly expanded. New structures like File, TargetList, PreReqList, LiteralFileName, and Ident have been introduced to better represent the syntax of a Makefile. Each structure now includes Pos() and End() methods for determining their position within the file. The token package has also been replaced with a custom implementation to provide more flexibility in handling tokens.

* Enhanced parser functionality and added tests

Significant enhancements have been made to the parser.go file. The Parser struct now includes additional fields for error handling and token tracking. New methods have also been introduced to parse files, handle errors, and manage tokens. A new test file, parser_test.go, has been created to ensure the correct functioning of the updated Parser.

* Added new constant for token position

A new constant, NoPos, has been added to the token package. This will be used to represent a non-existent or undefined position in the code.

* Refactor parser and add error handling

The parser has been refactored to improve its readability and efficiency. The scanner is now initialized within the NewParser function, removing the need for an external fmt import. Error messages have also been made more descriptive, providing better feedback when a token does not match expectations. Additionally, unnecessary print statements have been removed to clean up the console output.

In terms of functionality, rules parsing has been enhanced to handle cases where identifiers are expected but not found. This includes situations where a rule starts with a colon instead of an identifier.

Corresponding tests have also been updated and expanded to cover these new scenarios, ensuring that errors are handled correctly and informative messages are returned.

* Writer coverage

* A few comment cases

* More coverage and a bunch of broken stuff

* Refactor AST tests and update parser test description

Significant changes include:
- Reorganized import statements in ast_test.go
- Updated the expected end position of targets and prerequisites to account for name length
- Added new tests for LiteralFileName, Recipe, and Ident classes in ast_test.go
- Modified the description of a test case in parser_test.go for better clarity

Makefile

@@ -24,6 +24,10 @@ test_all:
 cover: cover.profile
 	go tool cover -func=$<
 
+clean:
+	rm -rf .make
+	rm -f cover.profile
+
 cover.profile: $(shell $(DEVCTL) list --go) | bin/ginkgo bin/devctl
 	$(GINKGO) run --coverprofile=cover.profile -r ./
 

ast/ast.go

@@ -0,0 +1,160 @@
+package ast
+
+import (
+	"go/ast"
+
+	"github.com/unmango/go-make/token"
+)
+
+type Node = ast.Node
+
+var Walk = ast.Walk
+
+// A File represents text content interpreted as the make syntax.
+// Most commonly this is a Makefile, but could also be any file
+// understood by make, i.e. include-me.mk
+type File struct {
+	FileStart, FileEnd token.Pos
+	Comments           []*CommentGroup
+	Rules              []*Rule
+}
+
+// A CommentGroup represents a sequence of comments with no other tokens and no empty lines between.
+type CommentGroup struct {
+	List []*Comment
+}
+
+// Pos implements Node
+func (c *CommentGroup) Pos() token.Pos {
+	return c.List[0].Pos()
+}
+
+// End implements Node
+func (c *CommentGroup) End() token.Pos {
+	return c.List[len(c.List)-1].End()
+}
+
+// TODO: Handle multi-line comments with '\' escaped newlines
+
+// A comment represents a single comment starting with '#'
+type Comment struct {
+	Pound token.Pos // position of '#' starting the comment
+	Text  string    // comment text, excluding '\n'
+}
+
+// Pos implements Node
+func (c *Comment) Pos() token.Pos {
+	return c.Pound
+}
+
+// End implements Node
+func (c *Comment) End() token.Pos {
+	return token.Pos(int(c.Pound) + len(c.Text))
+}
+
+// A Rule represents the Recipes and PreRequisites required to build Targets. [Rule Syntax]
+//
+// [Rule Syntax]: https://www.gnu.org/software/make/manual/html_node/Rule-Syntax.html
+type Rule struct {
+	Colon   token.Pos // position of ':' delimiting targets and prerequisites
+	Pipe    token.Pos // position of '|' delimiting normal and order-only prerequisites
+	Semi    token.Pos // position of ';' delimiting prerequisites and recipes
+	Targets *TargetList
+	PreReqs *PreReqList
+	Recipes []*Recipe
+}
+
+// Pos implements Node
+func (r *Rule) Pos() token.Pos {
+	return r.Targets.Pos()
+}
+
+// End implements Node
+func (r *Rule) End() token.Pos {
+	return r.Recipes[len(r.Recipes)-1].End()
+}
+
+// A TargetList represents a list of Targets in a single Rule.
+type TargetList struct {
+	List []FileName
+}
+
+// Pos implements Node
+func (t *TargetList) Pos() token.Pos {
+	return t.List[0].Pos()
+}
+
+// End implements Node
+func (t *TargetList) End() token.Pos {
+	return t.List[len(t.List)-1].End()
+}
+
+// A PreReqList represents all normal and order-only prerequisites in a single Rule.
+type PreReqList struct {
+	Pipe token.Pos
+	List []FileName
+}
+
+// Pos implements Node
+func (p *PreReqList) Pos() token.Pos {
+	return p.List[0].Pos()
+}
+
+// End implements Node
+func (p *PreReqList) End() token.Pos {
+	return p.List[len(p.List)-1].End()
+}
+
+// A FileName represents any Node that can appear where a file name is expected.
+type FileName interface {
+	Node
+	fileNameNode()
+}
+
+// A LiteralFileName represents a name identifier with no additional syntax.
+type LiteralFileName struct {
+	Name *Ident
+}
+
+func (*LiteralFileName) fileNameNode() {}
+
+func (l *LiteralFileName) Pos() token.Pos {
+	return l.Name.Pos()
+}
+
+func (l *LiteralFileName) End() token.Pos {
+	return l.Name.End()
+}
+
+// A Recipe represents a line of text to be passed to the shell to build a Target.
+type Recipe struct {
+	Tok    token.Token // TAB or SEMI
+	TokPos token.Pos   // position of Tok
+	Text   string      // recipe text excluding '\n'
+}
+
+// Pos implements Node
+func (r *Recipe) Pos() token.Pos {
+	return r.TokPos
+}
+
+// End implements Node
+func (r *Recipe) End() token.Pos {
+	return token.Pos(int(r.TokPos) + len(r.Text))
+}
+
+// An Ident represents an identifier.
+type Ident struct {
+	Name    string
+	NamePos token.Pos
+}
+
+// Pos implements Node
+func (i *Ident) Pos() token.Pos {
+	return i.NamePos
+}
+
+// End implements Node
+func (i *Ident) End() token.Pos {
+	return token.Pos(int(i.NamePos) + len(i.Name))
+}

ast/ast_suite_test.go

@@ -0,0 +1,13 @@
+package ast_test
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestAst(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Ast Suite")
+}

ast/ast_test.go

@@ -0,0 +1,175 @@
+package ast_test
+
+import (
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/unmango/go-make/ast"
+	"github.com/unmango/go-make/token"
+)
+
+var _ = Describe("Ast", func() {
+	Describe("CommentGroup", func() {
+		It("should return the position of the first comment", func() {
+			c := &ast.CommentGroup{[]*ast.Comment{{
+				Pound: token.Pos(69),
+			}}}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position of the last comment", func() {
+			c := &ast.CommentGroup{[]*ast.Comment{
+				{Pound: token.Pos(69), Text: "foo"},
+				{Pound: token.Pos(420), Text: "Some comment text"},
+			}}
+
+			Expect(c.End()).To(Equal(token.Pos(437)))
+		})
+	})
+
+	Describe("Comment", func() {
+		It("should return the pound position", func() {
+			c := &ast.Comment{Pound: token.Pos(69)}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+			Expect(c.Pos()).To(Equal(c.Pound))
+		})
+
+		It("should return the position after the comment text", func() {
+			c := &ast.Comment{
+				Pound: token.Pos(420),
+				Text:  "Some comment text",
+			}
+
+			Expect(c.End()).To(Equal(token.Pos(437)))
+		})
+	})
+
+	Describe("Rule", func() {
+		It("should return the position of the first target", func() {
+			c := &ast.Rule{Targets: &ast.TargetList{
+				List: []ast.FileName{&ast.LiteralFileName{
+					Name: &ast.Ident{NamePos: token.Pos(69)},
+				}},
+			}}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position after the final recipe", func() {
+			c := &ast.Rule{Recipes: []*ast.Recipe{{
+				TokPos: token.Pos(420),
+			}}}
+
+			// TODO: This is wrong, should be position after text
+			Expect(c.End()).To(Equal(token.Pos(420)))
+		})
+	})
+
+	Describe("TargetList", func() {
+		It("should return the position of the first target", func() {
+			c := &ast.TargetList{
+				List: []ast.FileName{&ast.LiteralFileName{
+					Name: &ast.Ident{NamePos: token.Pos(69)},
+				}},
+			}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position of the last target", func() {
+			c := &ast.TargetList{List: []ast.FileName{
+				&ast.LiteralFileName{Name: &ast.Ident{NamePos: token.Pos(69)}},
+				&ast.LiteralFileName{Name: &ast.Ident{
+					NamePos: token.Pos(420),
+					Name:    "foo",
+				}},
+			}}
+
+			Expect(c.End()).To(Equal(token.Pos(423)))
+		})
+	})
+
+	Describe("PreReqList", func() {
+		It("should return the position of the first target", func() {
+			c := &ast.PreReqList{
+				List: []ast.FileName{&ast.LiteralFileName{
+					Name: &ast.Ident{NamePos: token.Pos(69)},
+				}},
+			}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position after the lat prereq", func() {
+			c := &ast.PreReqList{List: []ast.FileName{
+				&ast.LiteralFileName{Name: &ast.Ident{NamePos: token.Pos(69)}},
+				&ast.LiteralFileName{Name: &ast.Ident{
+					NamePos: token.Pos(420),
+					Name:    "baz",
+				}},
+			}}
+
+			Expect(c.End()).To(Equal(token.Pos(423)))
+		})
+	})
+
+	Describe("LiteralFileName", func() {
+		It("should return the position of the identifier", func() {
+			c := &ast.LiteralFileName{Name: &ast.Ident{
+				NamePos: token.Pos(69),
+			}}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position after the identifier", func() {
+			c := &ast.LiteralFileName{Name: &ast.Ident{
+				NamePos: token.Pos(420),
+				Name:    "bar",
+			}}
+
+			Expect(c.End()).To(Equal(token.Pos(423)))
+		})
+	})
+
+	Describe("Recipe", func() {
+		It("should return the position of the tab", func() {
+			c := &ast.Recipe{
+				TokPos: token.Pos(420),
+			}
+
+			Expect(c.Pos()).To(Equal(token.Pos(420)))
+		})
+
+		It("should return the position after the text", func() {
+			c := &ast.Recipe{
+				TokPos: token.Pos(420),
+				Tok:    token.TAB,
+				Text:   "foo",
+			}
+
+			Expect(c.End()).To(Equal(token.Pos(423)))
+		})
+	})
+
+	Describe("Ident", func() {
+		It("should return the position of the name", func() {
+			c := &ast.Ident{
+				NamePos: token.Pos(69),
+			}
+
+			Expect(c.Pos()).To(Equal(token.Pos(69)))
+		})
+
+		It("should return the position after the name", func() {
+			c := &ast.Ident{
+				NamePos: token.Pos(420),
+				Name:    "foo",
+			}
+
+			Expect(c.End()).To(Equal(token.Pos(423)))
+		})
+	})
+})

internal/testing/testing.go

@@ -1,6 +1,10 @@
 package testing
 
-import "errors"
+import (
+	"bytes"
+	"errors"
+	"fmt"
+)
 
 type ErrReader string
 
@@ -13,3 +17,24 @@ type ErrWriter string
 func (e ErrWriter) Write(p []byte) (int, error) {
 	return 0, errors.New(string(e))
 }
+
+type ErrAfterWriter struct {
+	After int
+	Buf   *bytes.Buffer
+	at    int
+}
+
+func NewErrAfterWriter(after int) *ErrAfterWriter {
+	return &ErrAfterWriter{
+		After: after,
+		Buf:   &bytes.Buffer{},
+	}
+}
+
+func (e *ErrAfterWriter) Write(p []byte) (int, error) {
+	if e.at++; e.at >= e.After {
+		return 0, fmt.Errorf("write err: %d", e.at)
+	} else {
+		return e.Buf.Write(p)
+	}
+}