NOTES.md

Development Notes

# Initialize this project
go mod init afreeorange/bock

# Remove unused mods
go mod tidy

# Remove a package
go get package@none

# Build
CGO_ENABLED=1 go build --tags "fts5" -o "dist/bock-$(uname)-$(uname -m)" .

git commands

This the rough outline of how you'd get the revision history of a file. git cat-file -p <something> is a "swiss army knife" of sorts that lets you examine all manner of things. The -p flag tells git to figure out what the <something> is.

# Get the commits. Note that a single commit can change many articles!
# This is something you need to try to avoid.
git log /path/to/article.md

# Now get the *contents* of a commit by reading the blob.
git show 27e239b8f52ceb29a14a5d4fdf8d957ce4f022f4:/path/to/article.md

And here's how you find deleted files:

# On the branch you're on
git log --diff-filter D --pretty="format:" --name-only | sed '/^$/d'

# Across all branches
git reflog --diff-filter D --pretty="format:" --name-only | sed '/^$/d'

git log slowness with go-git

This is a known issue. I thought I could just use the git command and don't appear to be the only one who's had this thought. Note that with the few articles I have in my wiki (~250) generating JSON, HTML, the SQLite Database, etc takes ~200ms (on an M1 Max with 24GB memory). With revisions this goes up to 6s :/

Here's another comment that compares go-git with git2go (libgit2 wrapper) and just the git command.

TODO

• Fix issue with apostrophes 🤦‍♀️

• Compare Page
• Gist of recursive tree generation!
• Recent Changes (Global)
• Revisions argument
• STATS : Average number of revisions
• STATS : Average words per article (length)
• STATS : Oldest and newest article
• STATS : Recent articles
• STATS : Total words
• Syntax Highlighting
• Table of Contents
• Template argument
• Use context in lieu of config struct? What are the dis/advantages?
• 404 Page
• Better, less buggy tree
• Breadcrumbs > Revision
• Fix builds on cimg/go:1.18
• FIX FOLDER generation
• FIX THE NAVIGATION FFS
• FIX THE PATH GENERATION PROBLEM FFS
• Fix timestamps; make them consistent
• JSON for Revision
• Markdown highlight in Raw view
• Filtering logs with filename is very slow in go-git```

---

rm -rf $HOME/Desktop/temp/*;time go run --tags "fts5" . -a $HOME/personal/wiki.nikhil.io.articles -o $HOME/Desktop/temp
pushd $HOME/Desktop/temp; find . -type f -exec gzip -9 '{}' \; -exec mv '{}.gz' '{}' \;; popd
aws s3 sync $HOME/Desktop/temp/ s3://wiki.nikhil.io/ --delete --content-encoding gzip --size-only --profile nikhil.io

Libraries

• A possible progress bar.
• Hugo uses afero as its filesystem abstraction layer. I have not needed it. Yet.
• Structured Logging with Logrus.
• This is Commander but for golang <3 Maybe not necessary here since the flag library in STDLIB has everything I need. But longopts are nice!
• Cobra is a full-featured CLI app framework
• gin for live-reloading
• Martini for a web framework
• Gore for a REPL
• Minifier for HTML, CSS, XML, etc
• Awesome Go
• go-flags for CLI opt parsing. A bit heavy but appears to get the job done.

References

• Go Proverbs - Based on a talk by Rob Pike at GopherFest 2015
• Go Maps in Action
• https://maelvls.dev/go111module-everywhere/
• flosch/pongo2#68
• Colors in fmt
• Versioning
• Strings
• getopts
• Concurrency and Parallelism "Crash Course"
• Templates and Embed
• Recursive copying. I love that you have to implement quite a few things by hand in Golang!
• Chroma/Pygment style reference
• Enabling FTS5 with go-sqlite
• Build Tags in Golang
• Go Routines Under the Hood
• A million files in git repo.
• A Crash Course on Concurrency & Parallelism in Go
• How to have an in-place string that updates on stdout
• Intro to Golang logging

Concurrency/Parallelism

If you understand this enough you can roll your own with WaitGroup and channels.

• https://github.com/remeh/sizedwaitgroup
• https://github.com/nozzle/throttler

Books and Other Media

• Learning Go
• Lexical Scanning in Go
• Head-First Go is awesome

Afero Experiment

package main

import (
	"fmt"
	"os"
	"path"
	"path/filepath"
	"strings"

	"github.com/spf13/afero"
)

var fpath = "/Users/nikhil/personal/wiki.nikhil.io.articles"
var fpsaf = path.Dir(fpath) + "/"
var articleFolder = strings.Replace(fpath, fpsaf, "", -1)

func main() {
	fmt.Println("Starting...")

	memFS := afero.NewMemMapFs()
	var _ = memFS

	fmt.Print("Transferring articles to memory: ")
	filepath.Walk(fpath, func(p string, f os.FileInfo, err error) error {
		if f.IsDir() {
			name := strings.Replace(p, fpsaf, "", -1)
			fmt.Print("F")
			memFS.MkdirAll(name, os.ModePerm)
		} else {
			name := strings.Replace(p, fpsaf, "", -1)
			fmt.Print(".")
			contents, _ := os.ReadFile(p)
			afero.WriteFile(memFS, name, contents, os.ModePerm)
		}

		return nil
	})

	fmt.Println()
	fmt.Println("Done")

	fmt.Println("Checking in-memory FS")
	c, _ := afero.ReadFile(memFS, articleFolder+"/"+"Food/Thai Curry Experiments/Thai Green Curry Chicken - Instant Pot.md")
	fmt.Println(">>>", string(c))
}

Rules for Article Repository

• Home.md will be the generated homepage 🏠
• Anything in __assets folder in your article repository will be copied over to the output folder as-is.
• Reserved paths
  • /archive
  • /{Article Path}/raw
  • /{Article Path}/revisions

Templating

Uses pongo2 for a Django/Nunjucks-style syntax since I don't yet like the Golang's text/template. There's a base template that's embedded in the built binary which isn't too bad-looking but I'll add a way to specify custom templates later. Here are all of Pongo2's filters.

The base template uses the Gruvbox palette.

Template Types

• archive
• article
• folder
• index
• not-found
• random
• raw
• revision
• revision-list
• revision-raw

Search

Powered by SQL.js.

Simple Progress Bar

package main

import (
	"fmt"
)

// Bar ...
type Bar struct {
	percent int64  // progress percentage
	cur     int64  // current progress
	total   int64  // total value for progress
	rate    string // the actual progress bar to be printed
	graph   string // the fill value for progress bar
}

func (bar *Bar) NewOption(start, total int64) {
	bar.cur = start
	bar.total = total
	if bar.graph == "" {
		bar.graph = "â–ˆ"
	}
	bar.percent = bar.getPercent()
	for i := 0; i < int(bar.percent); i += 2 {
		bar.rate += bar.graph // initial progress position
	}
}

func (bar *Bar) getPercent() int64 {
	return int64((float32(bar.cur) / float32(bar.total)) * 100)
}

func (bar *Bar) Play(cur int64) {
	bar.cur = cur
	last := bar.percent
	bar.percent = bar.getPercent()
	if bar.percent != last && bar.percent%2 == 0 {
		bar.rate += bar.graph
	}
	fmt.Printf("\r[%-50s]%3d%% %8d/%d", bar.rate, bar.percent, bar.cur, bar.total)
}

func (bar *Bar) Finish() {
	fmt.Println()
}

Tree Generating Reference

package main

import (
	"encoding/json"
	"fmt"
	"io/fs"
	"os"
	"path/filepath"
	"regexp"
	"strings"
)

var articleRoot = "/Users/nikhil/personal/wiki.nikhil.io.articles"

// var articleRoot = "/Users/nikhil/haha"

var IGNORED_PATHS_REGEX = regexp.MustCompile(
	strings.Join([]string{
		"__a",
		"__assets",
		"_a",
		"_assets",
		".circleci",
		".git",
		"css",
		"img",
		"js",
		"node_modules",
	}, "|"))

type Entity struct {
	IsDir        bool   `json:"isDir"`
	IsSymlink    bool   `json:"isSymlink"`
	LinksTo      string `json:"linksTo"`
	Name         string `json:"name"`
	Path         string `json:"path"`
	RelativePath string `json:"relativePath"`
	Size         int64  `json:"size"`
	URI          string `json:"uri"`

	Children *[]Entity `json:"children"`
}

/*
Recursively create a tree of entities (files and folders)

Inspired by an iterative version here: https://stackoverflow.com/a/32962550
*/
func makeTree(path string, tree *[]Entity, ignoredPaths *regexp.Regexp) {
	currentRoot, _ := os.Stat(path)
	entityInfo := getEntityInfo(currentRoot, path)

	// Make list of the child entities in the path and then filter out any
	// children on the ignored paths list. Note that it is less code to use
	// `ioutil.ReadDir` since it returns the `fs.FileInfo` type but it's
	// deprecated.
	_children, _ := os.ReadDir(path)
	var children []fs.FileInfo

	for _, de := range _children {
		child, _ := de.Info()

		if !ignoredPaths.MatchString(child.Name()) {
			children = append(children, child)
		}
	}

	for i, c := range children {
		child := getEntityInfo(c, filepath.Join(entityInfo.Path, c.Name()))
		*tree = append(*tree, *child)

		if c.IsDir() {
			makeTree(child.Path, (*tree)[i].Children, ignoredPaths)
		}
	}
}

func getEntityInfo(entityInfo fs.FileInfo, path string) *Entity {
	entity := Entity{
		IsDir:    entityInfo.IsDir(),
		Size:     entityInfo.Size(),
		Name:     entityInfo.Name(),
		Path:     path,
		Children: &[]Entity{},
	}

	// Follow symlinks
	if entityInfo.Mode()&os.ModeSymlink == os.ModeSymlink {
		entity.IsSymlink = true
		entity.LinksTo, _ = filepath.EvalSymlinks(filepath.Join(path, entityInfo.Name()))
	}

	return &entity
}

func main() {
	var tree []Entity
	makeTree(articleRoot, &tree, IGNORED_PATHS_REGEX)

	s, _ := json.MarshalIndent(tree, "", "   ")
	fmt.Println(string(s))
}

A Simple Progress Bar

package main

import (
	"fmt"
	"time"
)

// Bar ...
type Bar struct {
	percent int64  // progress percentage
	cur     int64  // current progress
	total   int64  // total value for progress
	rate    string // the actual progress bar to be printed
	graph   string // the fill value for progress bar
}

func (bar *Bar) New(start, total int64) {
	bar.cur = start
	bar.total = total

	if bar.graph == "" {
		bar.graph = "."
	}

	bar.percent = bar.getPercent()

	for i := 0; i < int(bar.percent); i += 2 {
		bar.rate += bar.graph
	}
}

func (bar *Bar) getPercent() int64 {
	return int64((float32(bar.cur) / float32(bar.total)) * 100)
}

func (bar *Bar) Play(cur int64) {
	bar.cur = cur
	last := bar.percent

	bar.percent = bar.getPercent()

	if bar.percent != last && bar.percent%2 == 0 {
		bar.rate += bar.graph
	}

	fmt.Printf(
		// This is the key thing here to 'redraw' things on screen...
		"\r[%-50s] %2d%% %8d of %d",
		bar.rate,
		bar.percent,
		bar.cur,
		bar.total,
	)
}

func (bar *Bar) Finish() {
	fmt.Println()
}

func main() {
	// Use the bar!
	var bar Bar
	bar.New(0, 100)

	for i := 0; i <= 100; i++ {
		time.Sleep(100 * time.Millisecond)
		bar.Play(int64(i))
	}
	bar.Finish()
}

Pongo2 Custom Filter Sample

var _ = pongo2.RegisterFilter("round", func(in, param *pongo2.Value) (out *pongo2.Value, err *pongo2.Error) {
	var rounded *pongo2.Value

	if s, err := strconv.ParseFloat(in.String(), 32); err == nil {
		rounded = pongo2.AsSafeValue(math.Round(s))
	} else {
		return pongo2.AsSafeValue("!_COULD_NOT_ROUND_VALUE"), &pongo2.Error{OrigError: err}
	}

	return rounded, nil
})

Local Builds

docker run -ti -v /Users/nikhil/personal/bock:/home/circleci/project cimg/go:1.18 /home/circleci/project/.scripts/build.sh

Update all Dependencies

go get -u ./...