Document improvements: optimization, refactoring #6240

snej · 2023-05-11T21:28:00Z

CBG-0000

Describe your PR here...

Moved document, revision, attachment code out of db into new documents package
DocumentRevision aids in processing incoming document bodies, allowing efficient access to special properties without ever parsing the whole body.
Improved attachment types, using a DocAttachment struct that marshals to/from JSON. This gets rid of a lot of code that manually reads/writes attachment properties from a raw map.
Reimplemented RevTree, with a new more efficient JSON format
Made RevTree and some other SyncData fields unmarshal themselves lazily on first use

Pre-review checklist

Removed debug logging (fmt.Print, log.Print, ...)
Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Dependencies (if applicable)

Link upstream PRs
Update Go module dependencies when merged

Integration Tests

GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/000/

- JSONExtract now uses a callback to delegate the logic about which keys are special/reserved. - implemented JSONExtract for CE - DocumentRevision calls JSONExtract, handles special properties.

# Conflicts: # rest/replicatortest/replicator_test.go

Body changed to UnmarshalBody. Removed some usages of both methods, working with JSON data instead.

- RevisionCache.GetRevision no longer returns a Body - Removed body field from revCacheValue struct

# Conflicts: # rest/blip_api_delta_sync_test.go # rest/replicatortest/replicator_test.go # rest/replicatortest/replicator_test_helper.go # rest/user_api_test.go # rest/utilities_testing.go

…ESTS PASS

# Conflicts: # db/access_test.go # db/attachment_test.go # db/crud.go # db/crud_test.go # db/database_test.go # db/revision_cache_test.go

* CBG-2894: Reject user auth when channel threshold is over 500 in serverless mode * fix panic where authetciator was needed and it wasn't availible * linter issue * linter issue again * remove extra methods off interface * pass user into function * rebase * ensure 500 code is retruned for http error added * updates based off comments * fix panic * updates based off comments * updates based off dicussion yesterday * lint error * updates based of comments

* Add stats for transferred of bytes data ReplicationBytesReceived and ReplicationBytesSent * Stats are updated when a BlipSyncContext exits and at a threshold set by DatabaseContextOptions.BlipStatsReportingInterval (set to thirty seconds arbitrarily to see perf impact).

# Conflicts: # db/crud.go

Channels, Access, RoleAccess, Attachments, ChannelSet, ChannelSetHistory are now not unmarshaled until accessed.

- Missing licenses - An unnecessary assignment in util_ce.go - DocumentRevision should format expiration timestamps in UTC - Removed an unnecessary declaration from NameSet - Also moved NameSet to base package

snej added 30 commits February 23, 2023 13:53

Moved document, revision, attachment stuff into new 'document' package

d07305f

Moved validation.go from db to document

62d8e07

Moved RevisionCache from db to document

6755026

Merge branch 'master' into refactor-document

a078e59

rest has its own Body type, not using db.Body anymore

2ba5640

Moved document.ParseRevisions to rest

91039b6

Extracted document.SyncData into its own source file

9f3c599

Fixed remaining build errors by casting params to db.Body

cf97cf5

Implemented base.JSONExtractUnderscored() [EE only so far]

63d4064

Making DocumentRevision use JSONExtract fn

b70ec3f

- JSONExtract now uses a callback to delegate the logic about which keys are special/reserved. - implemented JSONExtract for CE - DocumentRevision calls JSONExtract, handles special properties.

Merge branch 'master' into refactor-document

1b40494

# Conflicts: # rest/replicatortest/replicator_test.go

tweak ParseDocumentRevision

073245f

DocumentRevision.BodyBytes is now a method not a field

3a6fd33

Removed DocumentRevision.MutableBody(), renamed Body()

5eeb9e9

Body changed to UnmarshalBody. Removed some usages of both methods, working with JSON data instead.

Removed Body from RevisionCache

ff3184e

- RevisionCache.GetRevision no longer returns a Body - Removed body field from revCacheValue struct

Removed includeBody parameters from RevisionCache interface

48eb314

Fixed some spots where BodyBytes didn't get "()" appended

57d4e33

Fixing test failures in db

f486410

More bug fixes & test fixes

eeae2a6

Cleaned up GetRev, getRev, Get1xBody methods

605c3fe

Avoid unmarshaling rev in blipHandler.processRev

4ec9bfe

rest.Body is the same as db.Body again (disabled earlier change)

2ee0774

Optimize creating DocumentRevision from pre-parsed merged delta

f112552

Some minor improvements

99291b0

Optimized Document.UnmarshalJSON; a few other fixes

028e55a

Removed several call sites of Document.Body()

127db6a

Stop using Document.Body in handlePutDocReplicator2

1866ccd

Document no longer caches Body. Renamed Body()->UnmarshalBody()

93f6be8

Merge branch 'master' into refactor-document

58bad1c

# Conflicts: # rest/blip_api_delta_sync_test.go # rest/replicatortest/replicator_test.go # rest/replicatortest/replicator_test_helper.go # rest/user_api_test.go # rest/utilities_testing.go

Cleanup & minor API changes

ffe0dd5

snej and others added 29 commits March 27, 2023 15:14

Slight change to attachment map types to fix delta compression; ALL T…

70b005b

…ESTS PASS

added missing license comments

0cfe123

add missing mod and run go mod tidy

7deef64

Merge branch 'master' into refactor-document

ed7a4af

# Conflicts: # db/access_test.go # db/attachment_test.go # db/crud.go # db/crud_test.go # db/database_test.go # db/revision_cache_test.go

Renamed package document --> documents

271f56d

Cleanup and doc-comments.

dfc5965

Merge branch 'master' into refactor-document

cb656e6

Report the correct error (#6232)

ae024a2

CBG-2916 Add database examples with scopes (#6233)

c776492

Update scopes documentation (#6231)

ddb447f

CBG-2895: Add static replication connection limit (#6226)

370a2bd

API Spec cleanup (scopes/collections) (#6236)

80d5eae

Merge branch 'master' into refactor-document

4b7c051

# Conflicts: # db/crud.go

Made RevTree a struct, to enable further refactoring

8044c9c

RevTree unmarshals itself lazily

955027e

More RevTree work. Optimize re-marshaling

5cb37e5

WIP: New RevTree JSON format

1c5b40a

Further optimization of new RevTree JSON format

84aa329

RevTree.Parent is now a pointer not a RevID

704a78b

Fixed some tests

2529959

Added NameSet utility (for channels)

02260f5

More cleanup & optimization of RevTree

c1e129a

Further RevTree JSON optimization, and some documentation

bfb514b

Fixed a missing tree.init()

9144924

Three RevTree bug fixes

f393eb7

Lazy loading of complex SyncData properties

63d0eac

Channels, Access, RoleAccess, Attachments, ChannelSet, ChannelSetHistory are now not unmarshaled until accessed.

Fix issues found by CI

a7f5b31

- Missing licenses - An unnecessary assignment in util_ce.go - DocumentRevision should format expiration timestamps in UTC - Removed an unnecessary declaration from NameSet - Also moved NameSet to base package

bbrks force-pushed the main branch from a9811cd to 1879730 Compare April 8, 2024 11:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document improvements: optimization, refactoring #6240

Document improvements: optimization, refactoring #6240

snej commented May 11, 2023

Document improvements: optimization, refactoring #6240

Are you sure you want to change the base?

Document improvements: optimization, refactoring #6240

Conversation

snej commented May 11, 2023

Pre-review checklist

Dependencies (if applicable)

Integration Tests