Initializing to use SkyWalking to replace Zipkin Server in v3

[mrproliu]

In the upcoming release of Zipkin v3, we are attempting to replace Zipkin Server with SkyWalking OAP. In this PR, I mainly removed the Storage and Collector involved in Zipkin and replaced the Zipkin Server with SkyWalking OAP for data collection and querying. This is a primary functionality replacement, and later on, I will submit more PRs to reintroduce all the functionalities of Zipkin.

[jcchavezs]

Instead of removing .github/workflows/test.yml I'd rather comment it out, we need guarantees that the new server will work as good as the old and adapt the tests to guarantee that.

[jcchavezs]

Also at some point we should make sure the quickstart.sh script works so we can integrate it in https://github.com/openzipkin/zipkin-php-example/blob/master/.circleci/config.yml which runs a nightly e2e test.

[mrproliu]

Instead of removing .github/workflows/test.yml I'd rather comment it out, we need guarantees that the new server will work as good as the old and adapt the tests to guarantee that.

Go it! However, since storage and receiver are both in the skywalking repo, I will write some IT specifically to complete the testing in another PR.

[mrproliu]

Also at some point we should make sure the quickstart.sh script works so we can integrate it in https://github.com/openzipkin/zipkin-php-example/blob/master/.circleci/config.yml which runs a nightly e2e test.

Of course, by reading quickstart.sh, I found that it will download the latest exec.jar from the Maven repository and run it. According to the current logic, it should be the same. It's just that it hasn't been released to the Maven repository yet (in v3 branch), so merging PRs won't trigger the behavior of uploading to the Maven repository.

[wu-sheng]

I want to confirm, as we are using a submodule that locks to skywaling upstream version. Actually, are we using the skywalking dev version mostly as the codebase?
I support this as it provides Zipkin side release more flexibly.

[mrproliu]

Yes, currently it is introducing the latest SHA of Skywalking as a submodule.

[marcingrzejszczak]

Hey, out of curiosity is there any backlog of why this is happening or other steps for Zipkin v3? Also, maybe this could be added to v3 #3561 in a better way?

[basvanbeek]

There have been discussions within the Zipkin admin group about future support for the existing server and the bandwidth that we have to keep v2 up and running. With SkyWalking fully supporting the Zipkin datamodel and API's (both ingestion and query) and working well with the Zipkin Lens UI, it is suitable as a drop-in replacement. It also is actively maintaining a multitude of storage solutions as well as having OTLP tracing data support since version 9.6.0.

Since we've collaborated closely with the SkyWalking community over the last few years always in good faith and lots of good energy, this seemed to us the most logical step forward. This is not at risk to be a PR and abandon effort. We actually have site owners and companies backing this v3 effort to provide Zipkin with a modern server component which will be actively maintained for the coming years.

The upstreaming effort is happening in a separate branch until ready for prime time. We encourage all Zipkin stakeholders to provide input on encountered issues, what needs to be supported, what future enhancements might be on the horizon. We will provide a timeline and call to action on the main site as well as the server's Github repo so it will not go unnoticed.

[marcingrzejszczak]

That sounds interesting! So that way my PR for OTLP ingress can be dropped unless we want it to be there for Zipkin 2.

So what will remain there in Zipkin 3 codebase if it gets replaced with SkyWalking?

[basvanbeek]

Excited to see this become ready for stable / production usage.

[basvanbeek]

That sounds interesting! So that way my PR for OTLP ingress can be dropped unless we want it to be there for Zipkin 2.

I think since we will not want to switch too quickly from marking v2 deprecated and v3 stable, it still makes sense to add this PR to v2. Since I lack bandwidth the next few weeks I do hope that one of the other maintainers can validate and review. FYA @openzipkin/core

So what will remain there in Zipkin 3 codebase if it gets replaced with SkyWalking?

The v3 distribution will combine core SkyWalking around tracing, the Zipkin v2 API's (both query and ingestion) and bundle the latest Zipkin Lens UI which will remain under active development. Internals will be quite new but user facing API's will remain the same.

The key thing to understand is that when upgrading from v2 to v3 you will not be able to automatically migrate your tracing data as the storage data schemas are not the same. Unless someone volunteers we have no plans to do schema migrations for any of the storage modules. Since Zipkin data typically has a short retention span we expect site owners to keep the old Zipkin deployment active until all data is purged in case they must keep the data during the retention period. It also means that any custom cold storage pipelines need adjustment on their inputs to the new schemas.

@wu-sheng we should publish the storage schemas for the various storage options we'll have with v3.

[wu-sheng]

we should publish the storage schemas for the various storage options we'll have with v3.

Make sense. Do you have previous doc about this for v2? We could follow that path. If there isn't, where do you suggest to put?

[wu-sheng]

Merging the PR to continue the further steps. We could continue the docs relative discussion here :)

[shakuzen]

Maybe I don't understand all the context and implications here. What is the difference between publishing a Zipkin Server v3 that embeds Skywalking server versus we stop publishing new versions of Zipkin Server and tell users to use Skywalking server (or whatever else they want) instead?