Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

*: Make tiflash compatible with the cse branch #9765

Merged
merged 15 commits into from
Feb 7, 2025
Merged

*: Make tiflash compatible with the cse branch #9765

merged 15 commits into from
Feb 7, 2025

Conversation

JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Jan 6, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #9767

Problem Summary:

What is changed and how it works?

The following adoption of cse only enable when SERVERLESS_PROXY = 1 is defined

  • Length of short value in lock CF encode as varInt instead of u8
  • Blocklist feature
    • keyspace blocklist
    • region_id blocklist
    • store_id blocklist under disagg arch
  • mtls
  • A HTTP API for Acquiring the all the region ids created in this TiFlash node with given keyspace id
  • File-base large txn
  • Special threshold on parallel prehandle

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Side effects

  • Performance regression: Consumes more CPU
  • Performance regression: Consumes more Memory
  • Breaking backward compatibility

Documentation

  • Affects user behaviors
  • Contains syntax changes
  • Contains variable changes
  • Contains experimental features
  • Changes MySQL compatibility

Release note

None

@ti-chi-bot ti-chi-bot bot added do-not-merge/needs-linked-issue do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note-none Denotes a PR that doesn't merit a release note. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. do-not-merge/needs-linked-issue labels Jan 6, 2025
@JaySon-Huang JaySon-Huang force-pushed the align_with_cse branch 2 times, most recently from 2dbf453 to 236d14f Compare January 7, 2025 07:40
@JaySon-Huang JaySon-Huang changed the title [WIP] *: Make tiflash compatible with the cse branch *: Make tiflash compatible with the cse branch Jan 8, 2025
@ti-chi-bot ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 8, 2025
yongman
yongman approved these changes Jan 8, 2025
View reviewed changes
Copy link
Member

@yongman yongman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jan 8, 2025

@yongman: adding LGTM is restricted to approvers and reviewers in OWNERS files.

In response to this:

LGTM

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

CalvinNeo
CalvinNeo reviewed Jan 14, 2025
View reviewed changes
dbms/src/Server/MetricsPrometheus.cpp
Poco::Net::Context::VerificationMode::VERIFY_STRICT
#else
// mtls: metrics server allows anonymous pullers @iosmanthus
Poco::Net::Context::VerificationMode::VERIFY_RELAXED
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where does this line come from? Can not find on either tiflash master or cse

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has been merge months ago https://github.com/tidbcloud/tiflash-cse/pull/216

CalvinNeo
CalvinNeo reviewed Jan 14, 2025
View reviewed changes
dbms/src/TiDB/Schema/TiDBSchemaSyncer.cpp
if (version <= cur_version)
{
return false;
}
#else
if (version == cur_version)
Copy link
Member

@CalvinNeo CalvinNeo Jan 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

        // if the `version` is less than `cur_version`, it means that the schema version in TiKV has been rolled back by restore.
        // We should sync the schema again.

Why won't this happen in TiFlash master?

Copy link
Contributor Author

@JaySon-Huang JaySon-Huang Feb 6, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In TiFlash master, when executing  flashback cluster, tidb won't overwrite the schema_version.
But in cse branch, they may restore the cluster by the data from S3. And this will lead to schema version rollbacked to a smaller version.

@CalvinNeo
Copy link
Member

/cc @guo-shaoge Could you please also review the blocklist part?

JaySon-Huang
JaySon-Huang commented Jan 14, 2025
View reviewed changes
dbms/src/Storages/KVStore/MultiRaft/PrehandleSnapshot.cpp Outdated Show resolved Hide resolved
dbms/src/Storages/KVStore/FFI/ProxyFFI.cpp Show resolved Hide resolved
dbms/src/Server/MetricsPrometheus.cpp
Poco::Net::Context::VerificationMode::VERIFY_STRICT
#else
// mtls: metrics server allows anonymous pullers @iosmanthus
Poco::Net::Context::VerificationMode::VERIFY_RELAXED
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it has been merge months ago https://github.com/tidbcloud/tiflash-cse/pull/216

CalvinNeo
CalvinNeo approved these changes Feb 5, 2025
View reviewed changes
Copy link
Member

@CalvinNeo CalvinNeo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ti-chi-bot ti-chi-bot bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Feb 5, 2025
@JaySon-Huang
Align the diff of short value in lock cf under cse
ae5e930 
Signed-off-by: JaySon-Huang <tshent@qq.com>
JaySon-Huang and others added 12 commits February 6, 2025 16:27
@JaySon-Huang
Add ut
8c6b009 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add blocklist feature
eed5e09 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
fix blacklist-file args to proxy
3698062 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Add http api for cse
6d78f8e 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
mtls: metrics server allows anonymous pullers
f48ff24 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Enlarge the max parall prehandle size
363617a 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
txn-file: support txn file feature with cse
1cb2559 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Fix parallel prehandle limit
ef91aaa 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Fix lint
5091098 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Fix compile error
50a57e5 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Remove unstable settings
6233173 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang @CalvinNeo
Apply suggestions from code review
5b38f50 
Co-authored-by: Calvin Neo <CalvinNeo@users.noreply.github.com>
@JaySon-Huang
Fix logging of opening sst with serverless compile options
80502ba 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@JaySon-Huang
Refine logging
ff81918 
Signed-off-by: JaySon-Huang <tshent@qq.com>
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Feb 7, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: CalvinNeo, Lloyd-Pottiger, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [CalvinNeo,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Feb 7, 2025
@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Feb 7, 2025

[LGTM Timeline notifier]

Timeline:

  • 2025-02-05 02:28:46.628640999 +0000 UTC m=+238798.161052997: ☑️ agreed by CalvinNeo.
  • 2025-02-07 06:53:31.415565833 +0000 UTC m=+427482.947977832: ☑️ agreed by Lloyd-Pottiger.

@ti-chi-bot ti-chi-bot bot merged commit 6a4b04d into pingcap:master Feb 7, 2025
5 checks passed
@JaySon-Huang JaySon-Huang deleted the align_with_cse branch February 7, 2025 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yongman yongman yongman approved these changes

@CalvinNeo CalvinNeo CalvinNeo approved these changes

@Lloyd-Pottiger Lloyd-Pottiger Lloyd-Pottiger approved these changes

@JinheLin JinheLin Awaiting requested review from JinheLin

Assignees
No one assigned
Labels
approved lgtm release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Make tiflash compatible with the cse branch
4 participants
@JaySon-Huang @CalvinNeo @yongman @Lloyd-Pottiger