Skip to content

Commit

Permalink
Flesh out the README and add a LICENSE file
Browse files Browse the repository at this point in the history
  • Loading branch information
sorentwo committed Aug 17, 2018
1 parent 8fa4174 commit 34d4f87
Show file tree
Hide file tree
Showing 3 changed files with 174 additions and 7 deletions.
21 changes: 21 additions & 0 deletions LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2018 Parker Selbert

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
158 changes: 152 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,81 @@
# Kiq

**TODO: Add description**
[![Build Status](https://travis-ci.org/sorentwo/kiq.svg?branch=master)](https://travis-ci.org/sorentwo/kiq)
[![Hex.pm](https://img.shields.io/hexpm/v/kiq.svg)](https://hex.pm/packages/kiq)

Kiq is a robust and extensible job processing queue that aims for compatibility
with [Sidekiq][sk], [Sidekiq Pro][skp] and [Sidekiq Enterprise][ske].

Job queuing, processing and reporting are all built on GenStage. That means
maximum parallelism with the safety of backpressure as jobs are processed.

### Why Kiq?

Many features and architectural choices in Kiq were drawn from existing Elixir
job processing packages like [Exq][exq], [Verk][verk] and [EctoJob][ej]. Each
of those packages are great and have varying strenghts, but they lacked seamless
interop with Sidekiq Pro or Sidekiq Enterprise jobs.

Sidekiq Pro and Enterprise are amazing pieces of commercial software and worth
every cent—your organization should buy them! Kiq is intended as a _bridge_ for
your team to _interop_ between Ruby and Elixir. As an organization embraces
Elixir it becomes necessary to run some background jobs in Elixir, and it must
be just as reliable as when jobs were ran through Sidekiq.

[sk]: https://sidekiq.org/
[skp]: https://sidekiq.org/products/pro.html
[ske]: https://sidekiq.org/products/enterprise.html
[exq]: https://github.com/akira/exq
[verk]: https://github.com/edgurgel/verk
[ej]: https://github.com/mbuhot/ecto_job

### Sidekiq Pro & Enterprise Comaptible Feature Set

Kiq's feature set includes many marquee offerings from Sidekiq, Sidekiq Pro and
Sidekiq Enterprise—plus some additional niceties made possible by running on the
BEAM.

These are the high
level marquee features that Kiq aims to support:

* [x] Integration with Sidekiq's Web UI and support for metrics, processes and
in-progress jobs
* [x] Custom reporters for custom logging, stats, error reporting, etc.
* [x] Reliable job fetching to prevent ever losing jobs before they are
processed
* [ ] Reliable job pushing in the event of network disconnection
* [ ] Unique job support, prevent enqueuing duplicate jobs within a period of time
* [ ] Batch job support, monitor a collection of jobs as a group and execute
callbacks when all jobs are complete
* [ ] Periodic job support, schedule jobs to be enqueued automatically on a
recurring basis
* [ ] Expiring job support, prevent running jobs that have exceeded an
expiration period

Not all of Sidekiq's features are supported. If a feature isn't supported or
planned it is _probably_ for one of these reasons:

1. We get it for free on the BEAM and it isn't necessary, i.e. (leader election,
safe shutdown, multi-process, rolling restarts)
2. We lean on Sidekiq to do the work (Web UI)
3. We enable developers to use custom reporters to do it themselves (stats,
error reporting)

### Design Decisions

* Avoid global and compile time configuration. All configuration can be defined
programatically, eliminating the need for hacks like `{:system, "REDIS_URL"}`.
* Not an application, it is included in your application's supervision
tree
* Testing focused, provide helpers and modes to aid testing
* Extensible job handling via GenStage consumers
* Simplified worker definitions to ease job definition and pipelining

[ent]: https://sidekiq.org/products/enterprise.html

## Installation

If [available in Hex](https://hex.pm/docs/publish), the package can be installed
by adding `kiq` to your list of dependencies in `mix.exs`:
Add `kiq` to your list of dependencies in `mix.exs`:

```elixir
def deps do
Expand All @@ -15,7 +85,83 @@ def deps do
end
```

Documentation can be generated with [ExDoc](https://github.com/elixir-lang/ex_doc)
and published on [HexDocs](https://hexdocs.pm). Once published, the docs can
be found at [https://hexdocs.pm/kiq](https://hexdocs.pm/kiq).
Then run `mix deps.get` to install the dependency.

Kiq itself is not an application and must be started within your application's
supervision tree.

## Usage

Kiq isn't an application that must be started. Similarly to Ecto, you define
one or more Kiq modules within your application. This allows multiple
supervision trees with entirely different configurations.

Define a Kiq module for your application and define an `init/2` callback for
runtime configuration:

```elixir
defmodule MyApp.Kiq do
use Kiq, queues: [default: 25, events: 50]

@impl Kiq
def init(_reason, opts) do
for_env = Application.get_env(:my_app, :kiq, [])

opts =
opts
|> Keyword.merge(for_env)
|> Keyword.put(:client_opts, [redis_url: System.get_env("REDIS_URL")])
|> Keyword.put(:server?, start_server?())

{:ok, opts}
end

defp start_server? do
testing? = Code.ensure_loaded?(Mix) && Mix.env() == :test
console? = Code.ensure_loaded?(IEx) && IEx.started?

not testing? && not console?
end
end
```

Include the module in your application's supervision tree:

```elixir
defmodule MyApp.Application do
@moduledoc false

use Application

alias MyApp.{Endpoint, Kiq, Repo}

def start(_type, _args) do
children = [
{Repo, []},
{Endpoint, []},
{Kiq, []}
]

Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end
end
```

Check the [hexdocs][hd] for additional details, configuration options, how to
test, defining workers and custom reporters.

[hd]: https://hexdocs.pm/kiq

## Contributing

Clone the repository and run `$ mix test` to make sure everything is working. For
tests to pass, you must have a Redis server running on `localhost`, port `6379`,
database `3`. You can configure a different host, port and database by setting
the `REDIS_URL` environment variable before testing.

Note that tests will wipe the the configured database on the Redis server
multiple times while testing. By default database 3 is used for testing.

## License

Kiq is released under the MIT license. See the [LICENSE](LICENSE.txt).
2 changes: 1 addition & 1 deletion lib/kiq.ex
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ defmodule Kiq do
@impl Kiq
def init(_reason, opts) do
for_env = Application.get_env(:my_app, :kiq)
for_env = Application.get_env(:my_app, :kiq, [])
opts =
opts
Expand Down

0 comments on commit 34d4f87

Please sign in to comment.