Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DO NOT MERGE. Intro serializer lib with json and protobuf. #107

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

DO NOT MERGE. Intro serializer lib with json and protobuf. #107

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
7358dbd
serialier lib
yb01 Jul 22, 2022
c9606ae
refactor to flat type folder for easy protobuf code gen
yb01 Jul 24, 2022
2546484
protobuf auto-gen files and hard-coded to protobuf serializer for pul…
yb01 Jul 25, 2022
ada6187
ensure correct length of the events set-- fix panic in auto-gen-ed pb.go
yb01 Jul 26, 2022
bbb7060
fmt change and remove unused commented code for now
yb01 Jul 27, 2022
17ed647
Merge branch 'cent-main' into seriliazer-works-2
yb01 Jul 27, 2022
8522950
remove uneeded rvmap filed for ResponseFromRRM struct
yb01 Jul 27, 2022
f1b4b3f
change back to localhost
yb01 Jul 27, 2022
6b00652
construct protobuf serializer in aggregator object level
yb01 Jul 27, 2022
5c8b908
add missing headers of files
yb01 Jul 27, 2022
7dbcb74
Merge branch 'cent-main' into seriliazer-works-2
yb01 Jul 27, 2022
11eb067
use protobuf in redis serialization of logical nodes
yb01 Jul 28, 2022
8620ce8
Merge branch 'cent-main' into seriliazer-works-2
yb01 Jul 29, 2022
6653e59
fix merge conflict
yb01 Jul 29, 2022
2c61791
Merge branch 'cent-main' into seriliazer-works-2
yb01 Jul 29, 2022
fb9b951
eventqueue optimization
yb01 Jul 30, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions resource-management/pkg/distributor/cache/eventqueue.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import (
)

// TODO - read from config
const LengthOfNodeEventQueue = 10000
const LengthOfNodeEventQueue = 100000

type nodeEventQueueByLoc struct {
circularEventQueue []*node.ManagedNodeEvent
Expand All @@ -40,6 +40,7 @@ func (qloc *nodeEventQueueByLoc) enqueueEvent(e *node.ManagedNodeEvent) {

if qloc.endPos == qloc.startPos+LengthOfNodeEventQueue {
// cache is full - remove the oldest element
klog.Warningf("cache is full")
qloc.startPos++
}

Expand Down Expand Up @@ -149,7 +150,7 @@ func (eq *NodeEventQueue) Watch(rvs types.InternalResourceVersionMap, clientWatc
return err
}

eq.watchChan = make(chan *types.NodeEvent)
eq.watchChan = make(chan *types.NodeEvent, 30)
// writing event to channel
go func(downstreamCh chan *types.NodeEvent, initEvents []*types.NodeEvent, stopCh chan struct{}, upstreamCh chan *types.NodeEvent) {
if downstreamCh == nil {
Expand Down Expand Up @@ -201,7 +202,7 @@ func (eq *NodeEventQueue) getAllEventsSinceResourceVersion(rvs types.InternalRes
}
}

nodeEvents := make([]*types.NodeEvent, 0)
nodeEvents := make([]*types.NodeEvent, 0, 1000)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By test, this change does not improve performance.

for loc, qByLoc := range eq.eventQueueByLoc {
startIndex, isOK := locStartPostitions[loc]
var events []*types.NodeEvent
Expand Down