Mainnet release diff 09142019 #1

rlan35 · 2019-09-14T16:11:06Z

No description provided.

Signed-off-by: Leo Chen <leo@harmony.one>

Fix explorer node stuck issue

add rpc call to resend missing cross shard receipt

…enesis [cleanup] remove is_genesis flag

Signed-off-by: Leo Chen <leo@harmony.one>

[nodetype] add nodetype to replace is_explorer

…cking; use sleep instead of ticker

fix beacon sync channel blocking issue

Signed-off-by: Leo Chen <leo@harmony.one>

Fix the beacon committee check

-d download only Signed-off-by: Leo Chen <leo@harmony.one>

-T node_type support node type (validator/explorer) Signed-off-by: Leo Chen <leo@harmony.one>

Signed-off-by: Leo Chen <leo@harmony.one>

-i shard_id specify shard_id, this is applicable only to explorer node Signed-off-by: Leo Chen <leo@harmony.one>

…script_support Non validator node script support

chaosma · 2019-09-14T18:04:02Z

node/node_cross_shard.go

-		cxReceipts, err := node.Blockchain().ReadCXReceipts(uint32(i), newBlock.NumberU64(), newBlock.Hash(), false)
-		if err != nil || len(cxReceipts) == 0 {
-			utils.Logger().Warn().Err(err).Uint32("ToShardID", uint32(i)).Int("numCXReceipts", len(cxReceipts)).Msg("[BroadcastCXReceipts] No ReadCXReceipts found")
+		go node.BroadcastCXReceiptsWithShardID(newBlock, commitSig, commitBitmap, uint32(i))


add closure for i

after double check, no need to add closure or j:=i

rlan35 · 2019-09-14T18:25:48Z

cmd/harmony/main.go

-	if *isExplorer {
-		startMsg = "==== New Explorer Node ===="
-	}
+	startMsg := fmt.Sprintf("==== New %s Node ====", strings.Title(*nodeType))


TODO: keep the original log

add it back

rlan35 · 2019-09-14T18:33:34Z

hmy/api_backend.go

+		return 0, false
+	}
+	tx := txs[int(index)]
+	if tx.ShardID() == tx.ToShardID() || blk.Header().ShardID() != tx.ShardID() {


TODO(chao): add comment for this logic.

rlan35 · 2019-09-14T18:40:39Z

node/node_cross_shard.go

-		cxReceipts, err := node.Blockchain().ReadCXReceipts(uint32(i), newBlock.NumberU64(), newBlock.Hash(), false)
-		if err != nil || len(cxReceipts) == 0 {
-			utils.Logger().Warn().Err(err).Uint32("ToShardID", uint32(i)).Int("numCXReceipts", len(cxReceipts)).Msg("[BroadcastCXReceipts] No ReadCXReceipts found")
+		go node.BroadcastCXReceiptsWithShardID(newBlock, commitSig, commitBitmap, uint32(i))


TODO(chao): add closure or j := i

rlan35 · 2019-09-14T18:43:32Z

node/node_cross_shard.go

+	utils.Logger().Info().Uint32("ToShardID", toShardID).Msg("[BroadcastCXReceiptsWithShardID] ReadCXReceipts and MerkleProof Found")
+
+	groupID := p2p.ShardID(toShardID)
+	go node.host.SendMessageToGroups([]p2p.GroupID{p2p.NewGroupIDByShardID(groupID)}, host.ConstructP2pMessage(byte(0), proto_node.ConstructCXReceiptsProof(cxReceipts, merkleProof, block.Header(), commitSig, commitBitmap)))


TODO(EK): check whether SendMessageToGroups is blocking or not.

rlan35 · 2019-09-14T18:44:21Z

node/node_cross_shard.go

-		go node.host.SendMessageToGroups([]p2p.GroupID{p2p.NewGroupIDByShardID(groupID)}, host.ConstructP2pMessage(byte(0), proto_node.ConstructCXReceiptsProof(cxReceipts, merkleProof, newBlock.Header(), commitSig, commitBitmap)))
+		sig := nextHeader.LastCommitSignature()
+		bitmap := nextHeader.LastCommitBitmap()
+		go node.BroadcastCXReceiptsWithShardID(blk, sig[:], bitmap, toShardID)


TODO(chao): use only one "go" routine.

remove go keyword

rlan35 · 2019-09-14T18:44:43Z

node/node_cross_shard.go

+}
+
+// BroadcastCXReceiptsWithShardID broadcasts cross shard receipts to given ToShardID
+func (node *Node) BroadcastCXReceiptsWithShardID(block *types.Block, commitSig []byte, commitBitmap []byte, toShardID uint32) {


TODO(chao): should be called with thread-safety in mind.

rlan35 · 2019-09-14T18:48:40Z

node/node_handler.go

@@ -392,6 +390,8 @@ func (node *Node) PostConsensusProcessing(newBlock *types.Block, commitSigAndBit
 		}
 	}

+	node.BroadcastMissingCXReceipts()


TODO(chao): explain why this line.

rlan35 · 2019-09-14T18:51:34Z

node/node_syncing.go

+		}
+		if node.beaconSync.GetActivePeerNumber() == 0 {
+			utils.Logger().Info().Msg("no peers; bootstrapping beacon sync config")
+			peers, err := node.SyncingPeerProvider.SyncingPeers(0)


TODO(EK): explain the magic "0" number.

rlan35 · 2019-09-14T18:56:05Z

scripts/go_executable_build.sh

+      if [ -e $BINDIR/$bin ]; then
+         $AWSCLI s3 cp $BINDIR/$bin s3://${PUBBUCKET}/$FOLDER/$bin --acl public-read
+      else
+         echo "!! MISSGIN $bin !!"


TODO(Leo): fix the typo.

* [reward] Add Accumulator of rewards in DB * [reward] Update Accumulator after checking all block rewards to state * [reward] Address PR comments #1 * [reward] Make fresh base 18 block reward always * [reward] Remove dead method on fakeChainReader * [committee] Adjust logging of prev,next committee on each usage of updateconsensusinformation * [reward] Set block accumulator only once to 0 when flip to staking epoch * [reward] Separate out condition to write block reward to 0 once past staking epoch (fix mistake) * [reward] lint * [reward] Initial write of reward accumulator needs lower-level first usage * [reward] Give block reward stake adjustment in right denomination

…aconchain (harmony-one#2253) * [double-sign] Commit changes in consensus needed for double-sign * [double-sign] Leader captures when valdator double signs, broadcasts to beaconchain * [slash] Add quick iteration tool for testing double-signing * [slash] Add webhook example * [slash] Add http server for hook to trigger double sign behavior * [double-sign] Use bin/trigger-double-sign to cause a double-sign * [double-sign] Full feedback loop working * [slash] Thread through the slash records in the block proposal step * [slash] Compute the slashing rate * [double-sign] Generalize yaml malicious for many keys * [double-sign][slash] Modify data structures, verify via webhook handler * [slash][double-sign] Find one address of bls public key signer, seemingly settle on data structures * [slash] Apply to state slashing for double signing * [slash][double-sign] Checkpoint for working code that slashes on beaconchain * [slash] Keep track of the total slash and total reporters reward * [slash] Dump account state before and after the slash * [slash] Satisfy Travis * [slash][state] Apply slash to the snapshot at beginning of epoch, now need to capture also the new delegates * [slash] Capture the unique new delegations since snapshot as well * [slash] Filter undelegation by epoch of double sign * [slash] Add TODO of correctness needed in slash needs on off-chain data * [rpc] Fix closure issue on shardID * [slash] Add delegator to double-sign testing script * [slash] Expand crt-validator.sh with commenting printfs and make delegation * [slash] Finish track payment of leftover slash debt after undelegation runs out * [slash] Now be explicit about error wrt delegatorSlashApply * [slash] Capture specific sanity check on slash paidoff * [slash] Track slash from undelegation piecemeal * [slash][delegation] Named slice types, .String() * [slash] Do no RLP encode twice, once is enough * [slash] Remove special case of validators own delegation * [slash] Refactor approach to slash state application * [slash] Begin expanding out Verify * [slash] Slash on snapshot delegations, not current * [slash] Fix Epoch Cmp * [slash] Third iteration on slash logic * [slash] Use full slash amount * [slash] More log, whitespace * [slash] Remove Println, add log * [slash] Remove debug Println * [slash] Add record in unit test * [slash] Build Validator snapshot, current. Fill out slash record * [slash] Need to get RLP dump of a header to use in test * [slash] Factor out double sign test constants * [slash] Factor out common for validator, stub out slash application, finish out deserialization setup * [slash] Factor out data structure creation because of var lexical scoping * [slash] Seem to have pipeline of unit test e2e executing * [slash] Add expected snitch, slash amounts * [slash] Checkpoint * [slash] Unit test correctly checks case of validator own stake which could drop below 1 ONE in slashing * [config] add double-sign testnet config (#1) Signed-off-by: Leo Chen <leo@harmony.one> * [slash] Commit for as is code & data of current dump.json * [slash] Order of state operation not correct in test, hence bad results, thank you dlv * [slash] Add snapshot state dump * [slash] Pay off slash of validator own delegation correctly * [slash] Pay off slash debt with special case for min-self * [slash] Pass first scenario conclusively * [slash] 2% slash passes unit test for own delegation and external * [slash] Parameterize unit test to easily test .02 vs .80 slash * [slash] Handle own delegation correctly at 80% slash * [slash] Have 80% slash working with external delegator * [slash] Remove debug code from slash * [slash] Adjust Apply signature, test again for 2% slash * [slash] Factor out scenario in testing so can test 2% and 80% at same time * [slash] Correct balance deduction on plan delegation * [slash] Mock out ChainReader for TestVerify * [slash] Small surface area interface, now feedback loop for verify * [slash] Remove development json * [slash] trigger-double-sign consumes yaml * [slash] Remove dead code * [slash][test] Factor ValidatorWrapper into scenario * [slash][test] Add example from local-testing dump - caution might be off * [slash] Factor out mutation of slashDebt * [slash][test] Factor out tests so can easily load test-case from bytes * [slash] Fix payment mistake in validator own delegation wrt min-self-delgation respected * [slash] Satisfy Travis * [slash] Begin cleanup of PR * [slash] Apply slash from header to Finalize via state processor * [slash] Productionize code, Println => logs; adjust slash picked in newblock * [slash] Need pointer for rlp.Decode * [slash] ValidatorInformation use full wrapper * Fix median stake * [staking] Adjust MarshalJSON for Validator, Wrapper * Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279) * Refactor offchain data; Add epoch to ValidatorSnapshot * Make block onchain/offchain data commit atomically * [slash][committee] Set .Active to false on double sign, do not consider banned or inactive for committee assignment * [effective] VC eligible.go * [consensus] Redundant field in printf * [docker] import-ks for a dev account * [slash] Create BLS key for dockerfile and crt-validator.sh * [slash][docker] Easy deployment of double-sign testing * [docker] Have slash work as single docker command * [rpc] Fix median-stake RPC * [slash] Update webhook with default docker BLS key * [docker][slash] Fresh yaml copy for docker build, remove dev code in main.go * [slash] Remove helper binary, commented out code, change to local config * [params] Factor out test genesis value * Add shard checking to Tx-Pool & correct blacklist (harmony-one#2301) * [core] Fix blacklist & add shardID check * [staking + node + cmd] Fix blacklist & add shardID check * [slash] Adjust to PR comments part 1 * [docker] Use different throw away funded account * [docker] Create easier testing for delegation with private keys * [docker] Update yaml * [slash] Remove special case for slashing validator own delegation wrt min-self-delegate * [docker] Install nano as well * [slash] Early error if banned * [quorum] Expose earning account in decider marshal json * Revert "Refactor offchain data commit; Make block onchain/offchain commit atomic (harmony-one#2279)" This reverts commit 9ffbf68. * [slash] Add non-sanity check way to update validator * [reward] Increase percision on percentage in schedule * [slash] Adjust logs * [committee] Check eligibility of validator before doing sanity check * [slash] Update docker * [slash] Move create validator script to test * [slash] More log * [param] Make things faster * [slash][off-chain] Clear out slashes from pending in writeblockwithstate * [cross-link] Log is not error, just info * [blockchain] Not necessary to guard DeletePendingSlashingCandidates * [slash][consensus] Use plain []byte for signature b/c bls.Sign has private impl fields, rlp does not encode that * [slash][test] Use faucet as sender, assume user imported * [slash] Test setup * [slash] reserve error for real error in logs * [slash][availability] Apply availability correct, bump signing count each block * [slash][staking] Consider banned field in sanity check, pay snitch only half of what was actually slashed * [slash] Pay as much as can * [slash] use right nowAmt * [slash] Take away from rewards as well * [slash] iterate faster * [slash] Remove dev based timing * [slash] Add more log, sanity check incoming slash records, only count external for slash rate * [availability][state] Adjust signature of ValidatorWrapper wrt state, filter out for staked validators, correct availaibility measure on running counters * [availability] More log * [slash] Simply pre slash erra slashing * [slash] Remove development code * [slash] Use height from recvMsg, todo on epoch * [staking] Not necessary to touch LastEpochInCommittee in staking_verifier * [slash] Undo ds in endpoint pattern config * [slash] Add TODO and log when delegation becomes 0 b/c slash debt payment * [slash] Abstract staked validators from shard.State into type, set slash rate based BLSKey count Co-authored-by: Leo Chen <leo@harmony.one> Co-authored-by: flicker-harmony <52401354+flicker-harmony@users.noreply.github.com> Co-authored-by: Rongjian Lan <rongjian@harmony.one> Co-authored-by: Daniel Van Der Maden <daniel@harmony.one>

Leo Chen and others added 23 commits September 10, 2019 17:29

[cleanup] remove is_genesis flag

97880b2

Signed-off-by: Leo Chen <leo@harmony.one>

Fix explorer node stuck issue

5474494

Merge pull request harmony-one#1575 from rlan35/cross-link-fork

a59db3f

Fix explorer node stuck issue

add http rpc call to resend missing cross shard receipt

c78b8fa

add tx type check; improve several efficiency issues

00c2356

[Pangaea] Reset DNS Server and Bootnode (harmony-one#1578)

dcbdab5

add more comments to CxPool

a81544f

refactor BroadcastCXReceiptsWithShardID and clean up corresponding logs

ad10f19

Merge pull request harmony-one#1577 from chaosma/cx_rpc

e903486

add rpc call to resend missing cross shard receipt

Merge pull request harmony-one#1560 from LeoHChen/cleanup_remove_is_g…

3f428ce

…enesis [cleanup] remove is_genesis flag

[nodetype] add nodetype to replace is_explorer

f08fd32

Signed-off-by: Leo Chen <leo@harmony.one>

Merge pull request harmony-one#1582 from LeoHChen/node_type

6497634

[nodetype] add nodetype to replace is_explorer

fix beacon sync channel blocking issue

3420bbe

use lastMileMux to protect lastMileBlock queue to avoid potential blo…

1a3267e

…cking; use sleep instead of ticker

Merge pull request harmony-one#1586 from chaosma/beaconsync-fix

082ddc7

fix beacon sync channel blocking issue

Fix the beacon committee check

97cde76

[release] add release action to go_executable_build.sh

5ab5936

Signed-off-by: Leo Chen <leo@harmony.one>

Merge pull request harmony-one#1588 from harmony-ek/beaconchain_sync_oom

8ea1650

Fix the beacon committee check

[node.sh] add -d option

005c942

-d download only Signed-off-by: Leo Chen <leo@harmony.one>

[node.sh] add -T node_type option

ce3553b

-T node_type support node type (validator/explorer) Signed-off-by: Leo Chen <leo@harmony.one>

[node.sh] backward compatible with older harmony binary

7210e15

Signed-off-by: Leo Chen <leo@harmony.one>

[node.sh] support -i shard_id option

cb3b8a0

-i shard_id specify shard_id, this is applicable only to explorer node Signed-off-by: Leo Chen <leo@harmony.one>

Merge pull request harmony-one#1589 from LeoHChen/non_validator_node_…

9b79b7c

…script_support Non validator node script support

chaosma reviewed Sep 14, 2019

View reviewed changes

rlan35 commented Sep 14, 2019

View reviewed changes

rlan35 pushed a commit that referenced this pull request Nov 22, 2019

[PR] Address PR comments #1

0114330

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mainnet release diff 09142019 #1

Mainnet release diff 09142019 #1

rlan35 commented Sep 14, 2019

chaosma Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

rlan35 Sep 14, 2019

rlan35 Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

rlan35 Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

chaosma Sep 14, 2019

rlan35 Sep 14, 2019

Mainnet release diff 09142019 #1

Are you sure you want to change the base?

Mainnet release diff 09142019 #1

Conversation

rlan35 commented Sep 14, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment