README.md

minion

Minion is a naive distributed job scheduler.

• minion
  • Endpoints
  • Usage
  • Authentication
  • Job Types
    • dummy
      • example dummy job
    • instance
      • example instance job
    • database
      • example database job
    • service
      • example service job
    • task
      • example task job
  • Create a Job
    • Request
    • Response
  • Update a Job
    • Request
    • Response
  • List Jobs in an account
    • Response
  • List Jobs in a space
    • Response
  • Get a Job
  • Delete a Job
  • Delete all jobs in a group
  • Run a Job
  • IAM permissions
    • S3 repository Example
      • create minion-dev-bucket and create a user with the policy
    • Cloudwatchlogs Example
      • policy example for cloudwatchlogs-creater-consumer
  • Author
  • License

Endpoints

GET /v1/minion/ping
GET /v1/minion/version
GET /v1/minion/metrics

GET /v1/minion/{account}/jobs
GET /v1/minion/{account}/jobs/{group}
POST /v1/minion/{account}/jobs/{group}
GET /v1/minion/{account}/jobs/{group}/{id}
PUT /v1/minion/{account}/jobs/{group}/{id}
DELETE /v1/minion/{account}/jobs/{group}
DELETE /v1/minion/{account}/jobs/{group}/{id}

PATCH /v1/minion/{account}/jobs/{group}/{id}

Usage

Authentication

Authentication is accomplished via a pre-shared key. This is done via the X-Auth-Token header.

Job Types

dummy

A dummy runner job just attempts to execute a template with the given account name and return that string.

example dummy job

{
    "description": "Do some dumb thing to my server",
    "details": {
        "runner": "dummyRunner"
    },
    "group": "space-xy",
    "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
    "modified_at": "2020-02-27T16:22:09Z",
    "modified_by": "someone",
    "name": "dummy-spin1234567",
    "schedule_expression": "* * * ? *",
    "enabled": true
}

instance

An instance runner job executes an action on an instance. Currently supported actions are stop, start and reboot

example instance job

{
    "description": "Start my server",
    "details": {
        "instance_action": "start",
        "instance_id": "i-aaaabbbb11112222",
        "runner": "instanceRunner"
    },
    "group": "space-xy",
    "id": "7cf7433f-f6c2-496e-8fe9-ed4776d130c1",
    "modified_at": "2020-02-28T18:14:26Z",
    "modified_by": "someone",
    "name": "start-spinaaaabbbb11112222",
    "schedule_expression": "* * * ? *",
    "enabled": true
}

database

A Database runner job executes an action against a database instance. Currently supported actions are stop and start.

example database job

{
    "job": {
        "description": "Stop database",
        "details": {
            "runner": "databaseRunner",
            "database_action": "stop",
            "instance_id": "spintst-db000b36"
        },
        "group": "test_rds",
        "modified_by": "someone",
        "name": "stop-tst",
        "schedule_expression": "* * * * *",
        "enabled": true
    }
}

service

A service runner job executes an action on a container service. Currently supported action is scale.

example service job

{
    "description": "Scale service to zero",
    "details": {
        "service_action": "scale",
        "service_cluster": "spindev-cluster-123",
        "service_name": "spindev-svc-123",
        "desired_count": "0",
        "runner": "serviceRunner"
    },
    "id": "d7e88fea-5a43-4a08-a5e9-c9b19baebefd",
    "modified_at": "2020-02-28T18:14:26Z",
    "modified_by": "someone",
    "name": "service-scale-job-test",
    "schedule_expression": "00 18 * * *",
    "enabled": true
}

task

A task runner job executes an action on a container taskdef. Currently supported action is run.

example task job

{
    "description": "Run 3 tasks from a taskdef",
    "details": {
        "task_action": "run",
        "task_cluster": "spindev-cluster-123",
        "task_name": "spindev-svc-123",
        "count": "0",
        "runner": "taskRunner"
    },
    "id": "a8d4d056-fbfe-4d7d-beb4-a64780c6d98c",
    "modified_at": "2021-10-28T18:14:26Z",
    "modified_by": "someone_else",
    "name": "task-run-job-test",
    "schedule_expression": "00 18 * * *",
    "enabled": true
}

Create a Job

POST /v1/minion/{account}/jobs/space-xy

Request

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ]
}

Response

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-27T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true,
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

Update a Job

PUT /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Request

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": false
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ]
}

Response

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-28T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": false
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

List Jobs in an account

GET /v1/minion/{account}/jobs

Response

[
    "space-xy/11e7b876-7a10-4c3e-93a7-e77eb3c68b58",
    "space-xy/6bcfa79f-615e-470d-97c1-687f3357497d",
    "space-ab/747f437a-a4af-48b2-a021-888bb8943a9b",
    "space-ab/7cf7433f-f6c2-496e-8fe9-ed4776d130c1"
]

List Jobs in a space

GET /v1/minion/{account}/jobs/space-xy

Response

[
    "11e7b876-7a10-4c3e-93a7-e77eb3c68b58",
    "6bcfa79f-615e-470d-97c1-687f3357497d"
]

Get a Job

GET /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-28T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

Delete a Job

DELETE /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Delete all jobs in a group

DELETE /v1/minion/{account}/jobs/space-xy

Run a Job

PATCH /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Note: At the moment, this checks if the job exists in the jobs repository and then adds it to the jobs queue. There is a potential race condition, since the jobs queue reads from the local cache when executing jobs, so it may be missing if it was just created and hasn't been cached by the loader yet.

IAM permissions

S3 repository Example

create minion-dev-bucket and create a user with the policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::minion-dev-bucket/*",
                "arn:aws:s3:::minion-dev-bucket"
            ]
        }
    ]
}

Cloudwatchlogs Example

policy example for cloudwatchlogs-creater-consumer

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:ListTagsLogGroup",
                "logs:CreateLogStream",
                "logs:TagLogGroup",
                "logs:DescribeLogGroups",
                "logs:DeleteLogGroup",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "logs:PutRetentionPolicy",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:localdev-*:log-stream:*",
                "arn:aws:logs:us-east-1:012345678910:log-group:localdev-*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "*"
        }
    ]
}

Author

E Camden Fisher camden.fisher@yale.edu

License

GNU Affero General Public License v3.0 (GNU AGPLv3)
Copyright (c) 2020 Yale University