-
Notifications
You must be signed in to change notification settings - Fork 0
Giulia Meeting Notes rolling log
Giulia edited this page May 28, 2021
·
11 revisions
We discussed about the early benchmarks for cassandra. Testing in localhost with a 5 node cluster, the difference between different consistency protocols is not that evident
ping to localhost is in the order of microseconds, but the total cost of the get put operations is in the order of 2-3 ms for 16-byte kv pair
todo:
- match number of replicas with number of nodes, makes more sense
- plot the distribution of the latency for all the iterations
- try to distribute the cassandra nodes on all the different machines
- tc on one machine to see if the latency adds up and how it affects multiple quorum levels
- e.g. tc with 5 ms latency on localhost
- compare python and c++ driver
- python can be justify for ml workloads and whatnot
- c++ for efficiency
- clone and build cassandra on my own, will have to do that soon for benchmarking purposes
- it makes sense to think about the project as an evaluation of the footprint of cassandra
read paper link
- basic traffic shaping single machine
- traffic shaping on multiple physical nodes so we dont have interference in the NIC
- try to understand performance breakdown of the get-put operation. with basically null RTT cost it takes 2-3 ms for the get-put of 16 bytes even in single-copy consistency -> goal is to have performance breakdown
- extend python script for benchmarking, support multithreading, different kv sizes, different load distributions for requests to the cluster...
- look into cassandra consistency algorithm -> paxos
- medium-term goal is to scope down the phd project to decide the angle, mobility vs programmabilty vs dynamic monitoring/replica placement
todo:
- deploy dockerized cassandra
- deploy native cassandra
target: deploy 5/7 nodes in cassandra → then add tc rules with different latencies also look at pathstore, necessary to benchmark as well as cassandra
- from last week: I had to evaluate if fogify is good to experiment with cassandra
results
- ran ping and iperf in fogify topology to validate the latency and bandwidth
- results seem reasonable within the constraints specified by the topology model
discussion:
- go on with fogify and go ahead with cassandra and pathstore deployment
- goal for SEC is an initial evaluation of Cassandra and Pathstore
- we need to find reasonable topologies
- understand a good topology for cassandra/pathstore experiments
- deploy cassandra on docker/fogify
- discussion about individual project for Tom, validate fogify and potentially port to kubernetes
what I'm doing now:
- stuck on cassandra multinode docker deployment
- stuck on fogify deployment of infra
- native cassandra benchmark
- fogify benchmark for cassandra
- docker benchmark on fogify
- update on fogify demo and network capabilities
- presentation on fogify capabilities. Looks promising to emulate all the characteristics that are needed for now
- potential master thesis ideas: support for mobility of end nodes, migration from docker to cgroups
- Q: how many containers does our cluster support?
- decision on rejected paper draft -> go over the reviews, need to substantiate the programmability aspect.
- docker deployment to test latency, bandwidth and drop rate compliance, does fogify match the expected behavior?
- make revision plan for hotstorage for position paper, address the comments from the reviewers
- update on fogify state of things
- solved missing image so now I can deploy barebone fogify controllers, but still not deploy an application scenario
- problem at the moment is that they treat and deploy the application and infrastructure in a coupled way, whereas I would find it more convenient to deploy them separately
roadmap:
- finish taxi demo
- identify network capabilities in fogify, can it support mobility scenarios, network trace emulations, degradation of quality of signal as user moves, nodes joining and leaving
- deploy empty docker images on fogify, decouple infrastructure from application
- cassandra on docker (off fogify)
- YCSB experiment on the cluster, plot the results and understand how the latency behaves and if it matches with the cluster specs
- cassandra on docker (on fogify)
- evaluate cassandra at the edge
- Create a couple of slides with fogify infrastructure overview and functionality
- run demo example for the taxi app
- (hopefully) run cassandra example on docker
With Lin:
- talked about joining the HPDC reading group Sys4ML (FYI)
- discussed BSc thesis topic on in-network computing
- missing edge platform, need to build a system to deploy nodes/sensors and emulate different network conditions
- need to do paper reading for background and foundational papers for storage and networking
- create wiki page for papers to dump
- final review and work to submit the edgesys paper
- skipped meeting due to ACE seminar