From 34d4f871093b9640cdc31dbd284fa5b19884e9da Mon Sep 17 00:00:00 2001 From: Parker Selbert Date: Fri, 17 Aug 2018 15:09:46 -0500 Subject: [PATCH] Flesh out the README and add a LICENSE file --- LICENSE.txt | 21 +++++++ README.md | 158 ++++++++++++++++++++++++++++++++++++++++++++++++++-- lib/kiq.ex | 2 +- 3 files changed, 174 insertions(+), 7 deletions(-) create mode 100644 LICENSE.txt diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..43bf9ba --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2018 Parker Selbert + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index 4ed8aa4..8c23592 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,81 @@ # Kiq -**TODO: Add description** +[![Build Status](https://travis-ci.org/sorentwo/kiq.svg?branch=master)](https://travis-ci.org/sorentwo/kiq) +[![Hex.pm](https://img.shields.io/hexpm/v/kiq.svg)](https://hex.pm/packages/kiq) + +Kiq is a robust and extensible job processing queue that aims for compatibility +with [Sidekiq][sk], [Sidekiq Pro][skp] and [Sidekiq Enterprise][ske]. + +Job queuing, processing and reporting are all built on GenStage. That means +maximum parallelism with the safety of backpressure as jobs are processed. + +### Why Kiq? + +Many features and architectural choices in Kiq were drawn from existing Elixir +job processing packages like [Exq][exq], [Verk][verk] and [EctoJob][ej]. Each +of those packages are great and have varying strenghts, but they lacked seamless +interop with Sidekiq Pro or Sidekiq Enterprise jobs. + +Sidekiq Pro and Enterprise are amazing pieces of commercial software and worth +every cent—your organization should buy them! Kiq is intended as a _bridge_ for +your team to _interop_ between Ruby and Elixir. As an organization embraces +Elixir it becomes necessary to run some background jobs in Elixir, and it must +be just as reliable as when jobs were ran through Sidekiq. + +[sk]: https://sidekiq.org/ +[skp]: https://sidekiq.org/products/pro.html +[ske]: https://sidekiq.org/products/enterprise.html +[exq]: https://github.com/akira/exq +[verk]: https://github.com/edgurgel/verk +[ej]: https://github.com/mbuhot/ecto_job + +### Sidekiq Pro & Enterprise Comaptible Feature Set + +Kiq's feature set includes many marquee offerings from Sidekiq, Sidekiq Pro and +Sidekiq Enterprise—plus some additional niceties made possible by running on the +BEAM. + +These are the high +level marquee features that Kiq aims to support: + +* [x] Integration with Sidekiq's Web UI and support for metrics, processes and + in-progress jobs +* [x] Custom reporters for custom logging, stats, error reporting, etc. +* [x] Reliable job fetching to prevent ever losing jobs before they are + processed +* [ ] Reliable job pushing in the event of network disconnection +* [ ] Unique job support, prevent enqueuing duplicate jobs within a period of time +* [ ] Batch job support, monitor a collection of jobs as a group and execute + callbacks when all jobs are complete +* [ ] Periodic job support, schedule jobs to be enqueued automatically on a + recurring basis +* [ ] Expiring job support, prevent running jobs that have exceeded an + expiration period + +Not all of Sidekiq's features are supported. If a feature isn't supported or +planned it is _probably_ for one of these reasons: + +1. We get it for free on the BEAM and it isn't necessary, i.e. (leader election, + safe shutdown, multi-process, rolling restarts) +2. We lean on Sidekiq to do the work (Web UI) +3. We enable developers to use custom reporters to do it themselves (stats, + error reporting) + +### Design Decisions + +* Avoid global and compile time configuration. All configuration can be defined + programatically, eliminating the need for hacks like `{:system, "REDIS_URL"}`. +* Not an application, it is included in your application's supervision + tree +* Testing focused, provide helpers and modes to aid testing +* Extensible job handling via GenStage consumers +* Simplified worker definitions to ease job definition and pipelining + +[ent]: https://sidekiq.org/products/enterprise.html ## Installation -If [available in Hex](https://hex.pm/docs/publish), the package can be installed -by adding `kiq` to your list of dependencies in `mix.exs`: +Add `kiq` to your list of dependencies in `mix.exs`: ```elixir def deps do @@ -15,7 +85,83 @@ def deps do end ``` -Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc) -and published on [HexDocs](https://hexdocs.pm). Once published, the docs can -be found at [https://hexdocs.pm/kiq](https://hexdocs.pm/kiq). +Then run `mix deps.get` to install the dependency. + +Kiq itself is not an application and must be started within your application's +supervision tree. + +## Usage + +Kiq isn't an application that must be started. Similarly to Ecto, you define +one or more Kiq modules within your application. This allows multiple +supervision trees with entirely different configurations. + +Define a Kiq module for your application and define an `init/2` callback for +runtime configuration: + +```elixir +defmodule MyApp.Kiq do + use Kiq, queues: [default: 25, events: 50] + + @impl Kiq + def init(_reason, opts) do + for_env = Application.get_env(:my_app, :kiq, []) + + opts = + opts + |> Keyword.merge(for_env) + |> Keyword.put(:client_opts, [redis_url: System.get_env("REDIS_URL")]) + |> Keyword.put(:server?, start_server?()) + + {:ok, opts} + end + + defp start_server? do + testing? = Code.ensure_loaded?(Mix) && Mix.env() == :test + console? = Code.ensure_loaded?(IEx) && IEx.started? + + not testing? && not console? + end +end +``` + +Include the module in your application's supervision tree: + +```elixir +defmodule MyApp.Application do + @moduledoc false + + use Application + + alias MyApp.{Endpoint, Kiq, Repo} + + def start(_type, _args) do + children = [ + {Repo, []}, + {Endpoint, []}, + {Kiq, []} + ] + + Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor) + end +end +``` + +Check the [hexdocs][hd] for additional details, configuration options, how to +test, defining workers and custom reporters. + +[hd]: https://hexdocs.pm/kiq + +## Contributing + +Clone the repository and run `$ mix test` to make sure everything is working. For +tests to pass, you must have a Redis server running on `localhost`, port `6379`, +database `3`. You can configure a different host, port and database by setting +the `REDIS_URL` environment variable before testing. + +Note that tests will wipe the the configured database on the Redis server +multiple times while testing. By default database 3 is used for testing. + +## License +Kiq is released under the MIT license. See the [LICENSE](LICENSE.txt). diff --git a/lib/kiq.ex b/lib/kiq.ex index 3a30477..2b71b92 100644 --- a/lib/kiq.ex +++ b/lib/kiq.ex @@ -64,7 +64,7 @@ defmodule Kiq do @impl Kiq def init(_reason, opts) do - for_env = Application.get_env(:my_app, :kiq) + for_env = Application.get_env(:my_app, :kiq, []) opts = opts