-
Notifications
You must be signed in to change notification settings - Fork 0
Notes
The basic idea I have is this:
- Add pre-commit hook so that a Notes.md file in the repository is preprocessed (sensitive information can be redacted) and added to each commit
- Add hook during pulling so that non-redacted changes are preferred... I don't know how this'll be done yet.
These hooks could probably be added to any repository (in the .git/hooks directory so they're not version controlled) in order to make it a noteful repository!
What would be even cooler is if upon committing (or pushing to GitHub and use Actions) a search index is generated (or a pre-trained model encodes the text) to make the notes more easily searchable or enable semantic search!
Can probably use a simple sed script. I'm 'introducing' a new markdown syntax of curlybrace-tilde REDACTED -- namely because it's easy to make regex patterns for eager pair matching, and Typora (and other editors) do auto-pairing for curly braces, which makes it quick and easy to type. And -- to my knowledge -- I don't think REDACTED is too common in languages, so it hopefully shouldn't show up in code snippets.
Normally the pattern REDACTED with the g and s options would be enough -- s tells regex to include newlines in dot. But sed doesn't accept the s option!
https://stackoverflow.com/questions/8624283/how-to-tell-sed-dot-match-new-line
Aaaaand it also doesn't handle non-greedy (lazy) matches!
https://unix.stackexchange.com/questions/128362/lazy-regex-using-bash
But perl does! I just don't know if it's included in GitBash by default (it would be nicest if the only requirement was GitBash itself)
I'm having some issues though with the multi-line stuff still.
https://www.systutorials.com/how-to-match-multiple-lines-using-regex-in-perl-one-liners/
Which I can resolve by adding the -0 flag (even without any arguments)
The (close to) final result
perl -0 -pe 's/REDACTED/ [REDACTED] /gs' infileit might be fun to use full blocks █ for redaction. But they might be annoying too!
I'd like to be able to escape the pattern too. New pattern just uses a couple negative lookbehinds for this: (?<!\\)REDACTED. You want to see the redaction symbol (finally)? {~Here it is!~} And this is the (unredacted) pattern: (?<!\\)\{~.*?(?<!\\)\~}
The pre-commit hook works beautifully!
I'm having it take my symlinked Notes.md (ignored) and put the redacted output into NOTES.md (versioned).
Since I've now updated these notes and would like to commit them (although I haven't changed any code in the repo yet) I can try git commit --allow-empty.
That being said, I feel like the notes shouldn't be ignored, since it's annoying to need to do that allow-empty commit if you want to add note changes...
Another way I could handle the files is by creating a temporary backup of the non-redacted notes, replacing the original with the redacted, adding the redacted, and then reverting to the backup.
Only issue I see with this is figuring out how to handle git pull. But hey -- maybe it's time we figure that out!
In the mean time, the empty commit worked great.
Let's try working with NOTES.md as our main note file.
Ideally I can keep an editor open on the file the whole time without any issue. Let's see.
I feel very strongly about avoiding -i in-place -- I do not want to ever incidentally remove information.
So far so good -- it seems I was able to temporary move (rename) the original file/symlink, then create the redacted version, add it, and then move the original back. Typora at least remained open the entire time and everything's still here!
In this setting, because the repository will have the public-facing notes whereas the local filesystem has the private-facing notes, NOTES.md will always show as modified. Additionally, since I'm using a symlink to keep my notes elsewhere it always shows typechange. Oh well, doesn't really affect anything except for the satisfaction of an 'all up to date' status 🙃 . Seems fine to me.
What happens if I undo a commit with git reset HEAD~?
Seemed to be fine -- even though I had added some content to these notes it just kept it!
Yeah, honestly, things seem to be working really well!
To test commit reset robustness, I'm writing this after committing, then I'll save, close my editor, reset, and re-open.
Everything remains :)
Alright, time for the pull test! The following content should be redacted in the public repository because it is very sensitive: REDACTED. That being said, it should remain in my private notes because it's very important!
Let's see how git gets angry when I try to pull, and how I can possibly resolve this with a merge hook.
Also -- back on the topic of Redaction -- it might be nice to replace the redacted content with the same number of characters.
https://stackoverflow.com/questions/17866072/replace-pattern-with-one-space-per-character-in-perl
Sure it would actually reduce the protection... but it would look so cool ;)
Alrighty. I've gone ahead and committed and pushed my notes up above, and now I've added some stuff -- let's see what git says when I pull.
Already up to date. Oh, okay. Let me check out an earlier commit.
So that's where I start to get some complaints -- Your local changes to the following files would be overwritten by checkout: NOTES.md.
Hmmmmmm I didn't think about the issue that arise when git keeps tabs on your notes file... makes it hard to switch between branches and commits.
Might revert to two separate public and private note files after all. Will look at again tomorrow.
After some reflection... it truly makes the most sense to keep the 'true' (private) notes file ignored from git. This way the user has a local notes file that is independent to branches etc.
An idea I have now is that notes should always be committed to main -- never any other branch. One way to achieve this is to keep notes ignored (at least on every branch that isn't main) and do a force push to main with every commit.
How will I accomplish this when on a different branch? Not sure yet.
https://stackoverflow.com/questions/14096721/how-to-add-file-to-a-previous-commit
Hmmmmm, so my current idea is to only add public notes to the main branch.
# Gameplan:
#
# 1. Check if a notes file exists
# - Best practices it should be symlinked with the target in a safe place on
# the user's computer
# 2. Make sure it's ignored by version control; make sure public file is ignored
# by version control
# 3. Create the redacted public notes
# 4. Force add public notes to the main branch
# - Will probably need to handle checkout, merging, etc. yikesIdeally I could avoid doing a checkout and merge, however, since that could involve stashing and potentially also merging -- scary stuff if all you're interested in is adding notes! Basically, I don't want to mess with the working directory.
So, I'm thinking: is there a way to push the notes changes from a different working directory? I could clone another repository in a temporary location... maybe I could even push a file without cloning?
Ya know... that SO link mentioned using the GitHub API to edit a file directly. Cool idea. Moves away from being a cross-platform git-only solution -- but hey if it makes it works for GitHub users that's a start. I'm sure there are GitLab APIs that can do the same things.
Then it made me think, if I'm using GitHub APIs, is there a better way to do this?
Perhaps the GitHub wikis could be a place to store the notes instead...
- https://docs.github.com/en/free-pro-team@latest/github/building-a-strong-community/adding-or-editing-wiki-pages#adding-or-editing-wiki-pages-locally
- https://docs.gitlab.com/ee/user/project/wiki/#adding-and-editing-wiki-pages-locally
Looks like wikis accomplish exactly what I'm looking for:
- They are coupled/associated with the repository
- There is only viewable version for the entire repository, independent to branches
- They can be updated independently to the repository (without messing with repository commits!)
Sweet!
It also has the added benefit of making it easy for multiple contributors to have different note pages.
Wikis also have multiple branches but only main is seen on the site. That being said, the other branches could still be seen if someone were to clone the wiki repository.
If this direction works out maybe I should rename this repo to "store-your-devnotes-in-wikis"!
This gameplan seems a lot nicer than before. (Just the last step is the kicker for these gameplans)
# Gameplan:
#
# 1. Make sure a private notes file exists.
# - It should be ignored, and for best practices it should be symlinked
# instead of residing in a such a volitile location
# 2. Make sure a subtree exists for the wiki repository
# 3. Add the redacted notes to the wiki subtree and pushUsing a completely different repository (the wiki repo) means we can avoid scary problems in our origin repository!
- https://gist.github.com/joshuajabbour/8569364 -- "Store and edit GitHub wikis within the main project repository."
https://docs.github.com/en/free-pro-team@latest/github/using-git/about-git-subtree-merges#adding-a-new-repository-as-a-subtree -- official documentation- https://www.atlassian.com/git/tutorials/git-subtree -- easier to understand
Upon trying to set up my wiki subtree I'm getting the error:
$ git remote add -f wiki git@github.com:parkeraddison/noteful-repositories.wiki
Updating wiki
fatal: remote error: access denied or repository not exported: /3/nw/38/a9/3a/322178542/326091467.wiki.git
error: Could not fetch wiki
I think (just like a normal repository) I might need to create the wiki first on the site?
Yup.
I just realized I was following github documentation for a subtree merge. That's not what I want to do, I want to set up a subtree.
https://stackoverflow.com/questions/31969868/how-to-store-github-wiki-as-part-of-source
I keep getting an error when trying to add a subtree
$ git subtree add --prefix wiki wiki master
Working tree has modifications. Cannot add.
which is really annoying, because why does my working tree need to be clean to add a subtree? Hmm? Doesn't quite make sense to me.
Anyway, I was able to remove all of my change -- ==TODO== will stash work in the future? -- and add the subtree to wiki/.
I may want to change this to a post-commit hook so that changes have been committed and there's less to fret about during the first subtree initialization.
https://stackoverflow.com/questions/16641057/how-can-i-list-the-git-subtrees-on-the-root
I think I'm approaching this wrong. Subtrees are still going to pose problems between multiple branches -- you'll need to update them
Why are devlogs unique to indie game development? Dev logs are important, for any project.
A Devlog is a forum post dedicated to the development of a single project. [...]
— itch.io/board/10021/devlogs
No where does it say this is only for games!
Interesting: search indexing for your documentation https://github.com/timberio/gitdocs
Continuing to work on the ignored wiki dotfolder. I think that's the best way to avoid annoying interactions with the main working directory and repository.
I've played around with pre-commit hooks, but that won't have access to the commit message. A post commit hook seemed ideal, but wouldn't be able to prevent someone from accidentally committing their private notes if their gitignore isn't properly configured.
https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
It seems that the commit-msg hook might be the way to go! I can get the commit message, but also exit with a non-zero code if the user is about to commit their private note file.
Also, it would be cool for the hook to prompt the user for things like what to name the file and whatnot.
And, actually, I'd like to make wiki commits each time a normal commit is made, and only push each time the user makes a push. In this case, I'll move the wiki push to a pre-push hook.
Eventually this should work with other pre-existing hooks: https://stackoverflow.com/questions/26624368/handle-multiple-pre-commit-hooks
Hmmmmm. You know what would be cool? If the notes had references to commits, so you could jump to the notes that were written at around the time of a commit.
This sounds like a job for a post-commit hook. How would it work in the file, though? Unless I let myself edit the private notes file by appending the short commit hash... maybe the message too...
That would be really nice because then we could add the commits as the TOC -- you'd be able to jump to the notes leading up to a given commit!
Lemme try it :)
Commit: ea26c87 HEAD -> use-wikis-dotfolder | Overhaul to add commit reference in notes. | 2021-01-05
Nice!
Tomorrow I should change it so that a heading is used, maybe have it just be the hash so you can jump to it.
Welp, I just pushed. Time to check it out on the site!
I'd like to experiment with the automatic linking, headings, etc.
Alright, I am quite confused at the moment.
I've tried making commits but the pre-commit hook fails to pull --rebase. It gives me the following error:
error: cannot pull with rebase: You have unstaged changes.
error: additionally, your index contains uncommitted changes.
error: please commit or stash them.
But if I run the script manually dash hooks/pre-commit then it works fine without throwing the error. Something about the commit must be changing stuff? But what?
Furthermore, if I go into the .wiki/ directory -- either before or after running a commit -- a git status shows no changes and I'm able to pull --rebase fine.
Ah. I just substituted the pull for status, and got a fatal unable to read.
https://stackoverflow.com/questions/3542854/calling-git-pull-from-a-git-post-update-hook
I think I need to just use git -C <path> ... instead of trying to cd.
:/ still got the rebase error. I'll try specifying --git-dir instead.
The heck, still got the rebase error. But this time it only complained about uncommitted changes, not any unstaged changes.
Git status still unable to read.
Okay. Looks like I need to specify both -C and --git-dir. So git -C .wiki/ --git-dir .git ..., at least it worked from the terminal when I tried that.
NOPE. Still unable to read hash. Honestly wtf is going on? I'm not quite sure. Why would something work when I run /bin/sh from the terminal, but not when /bin/sh is ran by the hook?
Minimal example:
# File structure
.wiki/.git...
.git...
hooks/pre-commit
# pre-commit
#!/bin/sh
git -C .wiki --git-dir .git status
exit 1
# Try triggering hook
> git commit -am 'test'
fatal: unable to read bd6e8ef48c0119f72a2ee78ff06263848155a8bf
# Try running manually
> /bin/sh hooks/pre-commit
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree cleanI can user log instead of status -- and that works and shows me the .wiki logs.
https://stackoverflow.com/questions/3542854/calling-git-pull-from-a-git-post-update-hook
https://stackoverflow.com/questions/2314500/git-post-receive-hook-not-working
None of this is working.
https://stackoverflow.com/questions/4144853/from-command-line-ok-but-the-hook-git
https://stackoverflow.com/a/3542926 -- At the end he mentioned env -i to ignore all environment variables. This freakin works!
Alright. Glad that's done with.
Now I still have a bad substitution error in my post-commit file.
hashlink="[${commithash:0:7}]($repourl/commit/$commithash)"I think this is just a POSIX thing.
This is a really good resource for making portable scripts when we're used to the easiness of bash: https://wiki.ubuntu.com/DashAsBinSh
Also, I'm facepalming a bit because I could just get the short has from the git log :)
Commit: 2021-01-06 | HEAD -> use-wikis-dotfolder | Fix git commands within git hooks; Test different commit references.
d36e72d213a5f23d636fb9e119bfe154cde3e8d5
Need to make some thing portable with POSIX shells. That's why the header is so large.
Also, I'd like to only add a newline if the notes file doesn't already end in a newline.
Commit: 2021-01-06 | HEAD -> use-wikis-dotfolder | Fix POSIX compliance, do newline check; Allow redaction to be escaped.
c917e20f8903e7100fa08451a30d534c435d3dc5
So... this time the post-commit hook complained when I tried committing the changes to the wiki. But I'm realizing I didn't have this issue before.
I'm going to try removing the env -i, and see if that changes anything. Ideally I'd be able to use the same config and credentials as in the main git repo when I commit.###### 53048dd Commit: 2021-01-06 | HEAD -> use-wikis-dotfolder | Fix wiki missing config/credientials.
53048dd0bb2a3c37aabd71fb3fec0bdf4d2199a8
Wait. What just happened?
I got the pull rebase error complaining that the index has uncommitted changes. Oh. Of course, the commit never went through last time. I should just run a stash... or reset and remove changes (they're being populated from the notes anyways).
Also, the newline broke :( ... trying again!
Commit: 2021-01-06 | HEAD -> use-wikis-dotfolder | Fix wiki missing config/credientials.
7c4c6d3f80837ed53ed3280d0fc5039895f6653c
Nice! Everything seems awesome! :happy:
Ah, whoops. Probably need to remove env -i during push too.
Alrighty, I looked at the wiki on the github site and it didn't automatically convert the long commit hash. I like the header too because it makes an anchor that you can hop to!
I've been playing around by editing the file on github and seeing the preview. I think I've settled on shoving the commit hash has a header and the following information into a blockquote to visually separate it from the notes. The anchor is still targetable!
"Fix wiki pushability; Settle on commit reference style." | HEAD -> use-wikis-dotfolder | 2021-01-06
Okie dokes, I think I'm about ready to finalize this version.
Soon I should probably make my first 'release' on GitHub to make it easy to run a couple commands to add the hooks to your repo, instead of needing to clone the repository or copy the files manually there could just be a curl of the release.
Also, the README of a noteful repository should link to the wiki somewhere! I'm thinking a badge would be nice -- but having a Notes section that explains it also wouldn't hurt... hmmm.
"Add link to wiki and adjust readme." | HEAD -> use-wikis-dotfolder | 2021-01-08