Skip to content

Commit

Permalink
Add simple dump command
Browse files Browse the repository at this point in the history
  • Loading branch information
banks committed Mar 28, 2023
1 parent 66ffff9 commit 9f1b7ae
Show file tree
Hide file tree
Showing 3 changed files with 99 additions and 0 deletions.
1 change: 1 addition & 0 deletions cmd/waldump/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
waldump
36 changes: 36 additions & 0 deletions cmd/waldump/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# waldump

A simple command for dumping the contents of WAL segment files to JSON for
debugging.

## Usage

```
$ waldump /path/to/wal/dir [-after INDEX] [-before INDEX]
...
{"Index":227281,"Term":4,"Type":0,"Data":"hpGEpUNvb3JkhKpBZGp1c3RtZW50yz7pEPrkTc4tpUVycm9yyz/B4NJg87MZpkhlaWdodMs/ABkEWHeDZqNWZWOYyz8FyF63P/XOyz8Fe2fyqYpayz7eXgvdsOWVyz7xX/ARy9MByz7XZq0fmx5eyz7x8ic7zxhJy78EgvusSgKUy77xVfw2sEr5pE5vZGWiczGpUGFydGl0aW9uoKdTZWdtZW50oA==","Extensions":null,"AppendedAt":"2023-03-23T12:24:05.440317Z"}
...
```

Each `raft.Log` is written out as JSON followed by a newline. The `Data` and
`Extensions` fields are opaque byte strings that will be base64 encoded.
Decoding those requires knowledge of the encoding used by the writing
application.

## Limitations

This tool is designed for debugging only. It does _not_ inspect the wal-meta
database. This has the nice property that you can safely dump the contexts of
WAL files even while the application is still writing to the WAL since we don't
have to take a lock on the meta database.

The downside is that this tool might in some edge cases output logs that have
already been deleted from the WAL. It's possible although extremely unlikely
that the WAL could be in the process of truncating the tail which could result
in there being both pre-truncate and post-truncate segment files present. This
tool might possibly output duplicate and out-of-order log indexes from before
and after the truncation. Or if `before` and `after` are used, it's possible we
might skip records entirely because an older file that has already been removed
was read instead of the newer one. These are all very unlikely in practice and
if the application that writes the WAL is still up and running are likely to be
resolved by the time you run the tool again.
62 changes: 62 additions & 0 deletions cmd/waldump/waldump.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Copyright (c) HashiCorp, Inc.

package main

import (
"encoding/json"
"flag"
"fmt"
"os"

"github.com/hashicorp/raft"
wal "github.com/hashicorp/raft-wal"
"github.com/hashicorp/raft-wal/fs"
"github.com/hashicorp/raft-wal/segment"
"github.com/hashicorp/raft-wal/types"
)

type opts struct {
Dir string
After uint64
Before uint64
}

func main() {
var o opts
flag.Uint64Var(&o.After, "after", 0, "specified a raft index to use as an exclusive lower bound when dumping log entries.")
flag.Uint64Var(&o.Before, "before", 0, "specified a raft index to use as an exclusive upper bound when dumping log entries.")

flag.Parse()

// Accept dir as positional arg
o.Dir = flag.Arg(0)
if o.Dir == "" {
fmt.Println("Usage: waldump <path to WAL dir> [-after INDEX] [-before INDEX]")
os.Exit(1)
}

vfs := fs.New()
f := segment.NewFiler(o.Dir, vfs)

codec := &wal.BinaryCodec{}
var log raft.Log
enc := json.NewEncoder(os.Stdout)

err := f.DumpLogs(o.After, o.Before, func(info types.SegmentInfo, e types.LogEntry) (bool, error) {
if info.Codec != wal.CodecBinaryV1 {
return false, fmt.Errorf("unsupported codec %d in file %s", info.Codec, segment.FileName(info))
}
if err := codec.Decode(e.Data, &log); err != nil {
return false, err
}
// Output the raft Log struct as JSON
if err := enc.Encode(log); err != nil {
return false, err
}
return true, nil
})
if err != nil {
fmt.Printf("ERROR: %s\n", err)
os.Exit(1)
}
}

0 comments on commit 9f1b7ae

Please sign in to comment.