How to maintain the forward/backward compatibility between different versions

[git-hulk]

For example, we introduced the STREAM column family in 2.2.0 and this breaks the forward compatibility due to the missing column family in the old version, so it's impossible for users to rollback if there's something wrong with the new version. Another example is the RocksDB upgrade RocksDB from 6.29.5 to 7.x in 2.3.0 which makes kXXH3 checksum the new default value for the block, and this cause 2.3.0 can't rollback to 2.0.x(its RocksDB version is 6.19.3 and kXXH3 was introduced in 6.27.0).

For those two specific cases:

• We can create a patch for 2.1 and 2.0, and ask users to upgrade to the corresponding version before upgrading to the newer version
• Create a patch for 2.0 and upgrade RocksDB to 6.29.5

But for future compatibility, I think we can add more documents to guide developers to keep the compatibility and test cases to ensure that.

[git-hulk]

An intuitive way to test the compatibility is by testing if the new dataset can be loaded and read by previous binaries. But I don't know how to do that yet. To see if you guys have any thoughts on this topic. @apache/kvrocks-committers

[tisonkun]

An intuitive way to test the compatibility is by testing if the new dataset can be loaded and read by previous binaries. But I don't know how to do that yet.

Roughtly if we do not delete the data (folder), we can start a server in new version, shutting down, and start a server in old version with the same data dir and perform basic operations. I'm unsure if it's viable according to our bootstrap and teardown logics.

[git-hulk]

Yes, I think it's easier for testing backward compatibility and we can do it first.

[tisonkun]

As we are trying to follow the semver flavor, when we make breaking changes, it's necessary to bump major versions.

This should happen during the review phase and we can pay extra attention to known risks as listed above from now on.

[tisonkun]

BTW, we can prepare a breaking changes page like what I did for Curator - https://curator.apache.org/docs/breaking-changes

[enjoy-binbin]

The proposer of the PR (someone who is familiar with it, not the normal user) or the maintainer needs to be able to review breaking changes.

For breaking changes, we can label them and call people to review and reach consensus (but do not merge, breaking change is a major decision, ideally with PMC approval), delayed merging until a major version update (or merge into the unstable branch but do not release).
Unless it is a breaking change caused by a very serious bug, it can be merged into a minor version.
And then when releasing, release notes need to be specially added a breaking changes section.

My saying are a bit off topic, but we also need to pay attention to normal breaking changes

[PragmaTwice]

Firstly, I think the thing you mentioned is not backward compatibility, but forward compatibility, i.e. new data can still be read by old binaries.

I think the most important thing is backward compatibility, i.e. old data can still be read by new binaries. And then we can care about forward compatibility, which seems less important to me than code maintainability and feature implementation.

To me, if it does not hurt code maintainability, we can keep forward compatibility, and if we cannot, we can at least alleviate it by highlighting in release note and releasing patch versions for the old version.

BTW, I think it is rare to rollback from 2.3.0 to 2.0.6. Users can firstly try to rollback to more close versions like 2.2.0 or 2.1.0.

[tisonkun]

Yeah. Supporting arbitrary backward and forward rolling update is challenging even in the business world. Basically, operators should roll up between the adjacent versions.

However, it's a major concern to users that: After I upgrade to new versions and encounter bugs, I cannot roll back from where I come from. This requires, at least, an "IMPORTANT NOTICE" as we should also seldom change the on-disk data format.

[git-hulk]

Firstly, I think the thing you mentioned is not backward compatibility, but forward compatibility, i.e. new data can still be read by old binaries.

Yes, I'm wrongly using the forward and backward. What I want to discuss is forward compatibility indeed. I have fixed the title.

BTW, I think it is rare to rollback from 2.3.0 to 2.0.6. Users can firstly try to rollback to more close versions like 2.2.0 or 2.1.0.

Yes, but upgrade 2.0.6 to 2.1.0 still have the forward compatibility issue due to the missing column family.

[mapleFU]

I guess the tile is "forward-compatible"? Since it means a old-version kvrocks read latest data.

RocksDB has some flags about compatible: https://github.com/facebook/rocksdb/wiki/RocksDB-Compatibility-Between-Different-Releases#format-compatibility . It gurantee the forward compatible on a major version.

As for minor version, if schema change happens, we'd better add them in document, and maybe provide some flags to control them?

[git-hulk]

Yes, should be forward NOT backward. My intention is to discuss how to find or detect the compatible issue, for the previous two cases, we didn't notice that before users complains.

[mapleFU]

I guess:

1. Newer datatype doesn't matter. When user uses newer datatype, he/she should ensure the responsbility of cannot downgrade.
2. Bugfix and RocksDB upgrade might need to be backport?
3. Change/Evolution on existing data type should has some control flags

[git-hulk]

Newer datatype doesn't matter. When user uses newer datatype, he/she should ensure the responsbility of cannot downgrade.

Yes, that right

Bugfix and RocksDB upgrade might need to be backport?

Maybe we should backport if we know the newer RocksDB has the compatible issue.

[tisonkun]

Concretely,

• Cannot read streams data doesn't hurt, because old versions don't know those data and the server API doesn't answer on those read/write calls. As long as the server can gracefully ignore unknown data (instead of corrupted; that is, we can check the integrity, but with a version or type metadata indicates the server doesn't support), it's not a big issue.
• Change the default value of kXXH3 checksum can be a serious compatibility issues as data "corrupted" for old versions. But how can we merge that PR (can you share the link?)? I suggest it should be always backed by an option to choose the old behavior to support in-place upgrading.

[git-hulk]

Cannot read streams data doesn't hurt, because old versions don't know those data and the server API doesn't answer on those read/write calls. As long as the server can gracefully ignore unknown data (instead of corrupted; that is, we can check the integrity, but with a version or type metadata indicates the server doesn't support), it's not a big issue.

For adding the new column family, we can backport the column family to the previous versions and it won't cause any problem since users have no way to read the data.

Change the default value of kXXH3 checksum can be a serious compatibility issues as data "corrupted" for old versions. But how can we merge that PR (can you share the link?)? I suggest it should be always backed by an option to choose the old behavior to support in-place upgrading.

It's NOT changed by Kvrocks, rocksdb did that in v7.8.3

[tisonkun]

It's changed by Kvrocks, rocksdb did that in v7.8.3

Can you refer to the PR in Kvrocks?

[git-hulk]

Should be this PR: facebook/rocksdb#10778

[enjoy-binbin]

i suppose it is this one #1203? (bump rocksdb to 7.8.3)
We now seem to be a bit casual about the upgrade of libs.

Usually, the libs can be updated less frequently (of course, there seems to be nothing wrong with frequent updates).
Frequent updates may introduce changes we are not sure about since we are not so family with those libs.
Libs with minor version can be updated, but we need to be cared with the major versions.
Libs with major version can be released together with our major version, but we need to be aware of the changes, otherwise we don't upgrade.

[git-hulk]

For #1203, it's a minor version update(7.7.3->7.8.3) and it makes sense for RocksDB to do this change since they don't promise to be forward-compatible with 6.x versions. I think the key problem is we don't have enough test cases for confirming the compatibility.

[jihuayu]

I have an idea for handling modifications across multiple versions. We can provide an "Upgrade paths" similar to GitLab, where we only need to ensure compatibility between 2 nearly key versions, rather than between any two versions.

[caipengbo]

I also have a small suggestion, I think we can add a tag mark on CR, when we CR, we take the compatibility issue as a point of consideration.

[git-hulk]

The previously mentioned compatibility cases weren't aware before users report them, so it will be better to let test cases find that.

[xiaobiaozhao]

I agree with this suggestion

[tisonkun]

I think here is where we can introduce a PR template to materialize some check items the community agrees on.

I'll try to prepare a patch for review.