Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Implement a commitlog. #15

Open
wants to merge 56 commits into
base: master
Choose a base branch
from
Open

[WIP] Implement a commitlog. #15

wants to merge 56 commits into from

Conversation

jkozlowski
Copy link
Owner

@jkozlowski jkozlowski commented Aug 28, 2019 •
edited
Loading

TODO

  • Writing to segment buffer and returning a handle that indicates when that position of commit log is durable
  • Flushing to a position.
  • Closing a commit log safely.
  • Flush semaphore for concurrency control.
  • Commit log replay
  • End-to-end tests: start commit log, write concurrently for 1 minute, with flushes happening etc., then collect all writes, close cleanly, replay and check that state in-memory is the same.

Done

  • Flush queue
  • Gate
  • Cycling a buffer (writing part of segment to underlying file).
  • Maintaining an open pool of segments.

Functions

  • Commitlog#add

  • SegmentManager#allocate_when_possible

  • SegmentManager#active_segment

  • SegmentManager#current_segment

  • SegmentManager#new_segment

  • SegmentManager#finish_and_get_new

  • SegmentManager#flush_segments

  • SegmentManager#do_pending_deletes

  • SegmentManager#acquire_zeroed_buffer

  • Segment#replenish_reserve

  • Segment#cycle

  • Segment#do_flush

  • Segment#new_buffer

  • Segment#batch_cycle

  • Segment#flush

  • Segment#sync

  • Segment#reset_sync_time

@jkozlowski
Code drop
facb503
@jkozlowski
Implement internal mutability.
4e0fce7
@jkozlowski
Turn on the power of tokio
9be9400
@jkozlowski
Implement Filesystem on top of tokio.
c446f1b
@jkozlowski
Trying to get latest tokio
6c93c22
@jkozlowski
Using latest tokio
3ff2514
@jkozlowski
Parse descriptor from file names.
c1ea422
@jkozlowski
list_descriptors works.
79b6fd7
@jkozlowski
Replenishing loop
7b31a3a
@jkozlowski
Pull active segment or allocate new under lock.
df7b940
@jkozlowski
Start defining commitlog.
079c9ef
@jkozlowski
Tweak
f7c9328
@jkozlowski
Cargo fix
d6dd6c5
@jkozlowski
Turn off for now
8aa1f6d
@jkozlowski
Start working on taking mutations
432c8c8
@jkozlowski
Currently fighting cycle errors.
a3b40cd
@jkozlowski
Ok I think I kinda fixed it. At least it compiler
2bf2a36
@jkozlowski
Wow, this compiles?
ed326df
@jkozlowski
Start writing a test for FlushQueue.
04b4473
@jkozlowski
A mialo byc tak pieknie...
c8c6eca
@jkozlowski
Test runs, but I suspect there's no actual concurrency here.
fc35b9b
@jkozlowski
Cleanup.
b2563ac
@jkozlowski
Again
dfbd3ff
@jkozlowski
Bla
67592dd
@jkozlowski
Ok so this is completely wrong now, the delay helps.
320758d 
I need to figure out the ownership crap.
@jkozlowski
Fuck yea, it works!
6db8c9f
@jkozlowski
Cleanup
53ec7fe
@jkozlowski
Move code around to use the same pattern as flush_queue.
d9caa60
@jkozlowski
Fix little baguettes.
c3ac4ba 
This range API is powerful, but I completely miseunderstood what double iterator means.
@jkozlowski
Restructure segment_manager to use Shared properly.
cba86d4
@jkozlowski
Start writing the header.
5f526a6
@jkozlowski
Write headers
bf3e455
@jkozlowski
Finally got to writing the file.
c64cfc9 
This whole queue thing feels a bit weird,
given the position is maintained by the OS...
Might need to clone + seek, since we want to maintain this ourselves,
since we'll want to eventually use spdk/io_uring.
@jkozlowski
File writing.
7cec49b 
Keeping the interface so that it is compatible with uring,
so not taking buffers by reference, instead owning it.
Also pushing pos to each write, which means with current impl
I need to open the file everytime.
@jkozlowski
Ok file is being written, it's soo wrong but it compiles nicely.
b024032
@jkozlowski
Refactor Shared.
4f9ccf3 
Need to now open the file every time and I should be able to get what I want.
@jkozlowski
A bit of cleanup.
ad63d3b
@jkozlowski
Fixup logging.
b75dd41
@jkozlowski
Started flushing, rather quick sketching, likely will have borrowing …
87e43bc 
…issues.
@jkozlowski
Start defining gate.
02d68ea
@jkozlowski
Upgrade libs and add tonic.
c183071
@jkozlowski
Jump to beta channel.
2cdf0ed
@jkozlowski
Format
d456c84
@jkozlowski
Implement gate.
f1468d5
@jkozlowski
Format.
62d9912
@jkozlowski
Finish flush guard for now.
1dfcaf1 
It doesn't actually control concurrency, but code can use it.
@jkozlowski
Start defining and writing things up from add downwards.
07f6a83
@jkozlowski
Semaphore interface.
200ac71
@jkozlowski
Wire up flush semaphore.
67dc969
@jkozlowski
Wire up request semaphore and call into segment.
b4ff2fe
@jkozlowski
Small improvements in Segment#sync.
cea5e73
@jkozlowski
Start setting up flush timer.
95a8ea5
@jkozlowski
Flush segments on timer.
4920af4
@jkozlowski
Track metrics when cycling.
74e1495
@jkozlowski
Fixup
9b4f2cd
@jkozlowski
Fix current_segment to return if it is still allocating.
e5e1722
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jkozlowski