diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..3039e35 --- /dev/null +++ b/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2024 violetpay-org + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md new file mode 100644 index 0000000..461de7e --- /dev/null +++ b/README.md @@ -0,0 +1,75 @@ +# Queue Streamer + +Queue Streamer is a Go package that processes and transfers data between Kafka topics with exactly-once delivery guarantees. This package receives messages from Kafka brokers and transfers them to specified topics. This document explains how to install and use Queue Streamer. + +## Installation + +To install Queue Streamer, use the Go modules: + +```shell +go get github.com/violetpay-org/queue-streamer +``` + +## Usage + +Here is an example code to use Queue Streamer. + +### Example + +```go +package main + +import ( + "github.com/violetpay-org/queue-streamer" + "sync" +) + + +func main() { + wg := &sync.WaitGroup{} + + brokers := []string{"localhost:9092", "localhost:9093", "localhost:9094"} + origin := qstreamer.NewTopic("origin-topic", 3) // Topic name and partition + + // Serializer that converts the message to the message to be produced. + // In this case, the message is not converted, so it is a pass-through serializer. + serializer := qstreamer.NewPassThroughSerializer() + + destination1 := qstreamer.NewTopic("destination-topic-1", 5) // Topic name and partition + destination2 := qstreamer.NewTopic("destination-topic-2", 3) + + streamer := qstreamer.NewTopicStreamer(brokers, origin) + + cfg := qstreamer.NewStreamConfig(&TestSerializer{}, destination1) + streamer.AddConfig(cfg) + + cfg = qstreamer.NewStreamConfig(&TestSerializer{}, destination2) + streamer.AddConfig(cfg) + + streamer.Run() // Non-blocking + defer streamer.Stop() + wg.Add(1) + + wg.Wait() +} +``` + +### Explanation + +1. Set Topics: Use the NewTopic function to set the start and end topics. + +2. Use PassThroughSerializer: Create a pass-through serializer using NewPassThroughSerializer which does not alter the message. + * If you want to convert the message, you can create a custom serializer that implements the Serializer interface. +3. Set StreamConfig: Use the NewStreamConfig function to configure the stream settings. + +4. Create and Configure TopicStreamer: Use the NewTopicStreamer function to create the topic streamer and the AddConfig method to add the stream configuration. + +5. Run and Stop Streamer: Call the Run method to start the streamer and the Stop method to stop the streamer.Use WaitGroup: Use sync.WaitGroup to prevent the main goroutine from exiting. + +## Contribution + +Contributions are welcome! You can contribute to the project by reporting bugs, requesting features, and submitting pull requests.Run method to start the streamer and the Stop method to stop the streamer. + +## License + +Queue Streamer is distributed under the MIT License. See the LICENSE file for more details. \ No newline at end of file