minion

Minion is a naive distributed job scheduler.

minion

Endpoints

GET /v1/minion/ping
GET /v1/minion/version
GET /v1/minion/metrics

GET /v1/minion/{account}/jobs
GET /v1/minion/{account}/jobs/{group}
POST /v1/minion/{account}/jobs/{group}
GET /v1/minion/{account}/jobs/{group}/{id}
PUT /v1/minion/{account}/jobs/{group}/{id}
DELETE /v1/minion/{account}/jobs/{group}
DELETE /v1/minion/{account}/jobs/{group}/{id}

PATCH /v1/minion/{account}/jobs/{group}/{id}

Usage

Authentication

Authentication is accomplished via a pre-shared key. This is done via the X-Auth-Token header.

Job Types

dummy

A dummy runner job just attempts to execute a template with the given account name and return that string.

example dummy job

{
    "description": "Do some dumb thing to my server",
    "details": {
        "runner": "dummyRunner"
    },
    "group": "space-xy",
    "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
    "modified_at": "2020-02-27T16:22:09Z",
    "modified_by": "someone",
    "name": "dummy-spin1234567",
    "schedule_expression": "* * * ? *",
    "enabled": true
}

instance

An instance runner job executes an action on an instance. Currently supported actions are stop, start and reboot

example instance job

{
    "description": "Start my server",
    "details": {
        "instance_action": "start",
        "instance_id": "i-aaaabbbb11112222",
        "runner": "instanceRunner"
    },
    "group": "space-xy",
    "id": "7cf7433f-f6c2-496e-8fe9-ed4776d130c1",
    "modified_at": "2020-02-28T18:14:26Z",
    "modified_by": "someone",
    "name": "start-spinaaaabbbb11112222",
    "schedule_expression": "* * * ? *",
    "enabled": true
}

database

A Database runner job executes an action against a database instance. Currently supported actions are stop and start.

example database job

{
    "job": {
        "description": "Stop database",
        "details": {
            "runner": "databaseRunner",
            "database_action": "stop",
            "instance_id": "spintst-db000b36"
        },
        "group": "test_rds",
        "modified_by": "someone",
        "name": "stop-tst",
        "schedule_expression": "* * * * *",
        "enabled": true
    }
}

service

A service runner job executes an action on a container service. Currently supported action is scale.

example service job

{
    "description": "Scale service to zero",
    "details": {
        "service_action": "scale",
        "service_cluster": "spindev-cluster-123",
        "service_name": "spindev-svc-123",
        "desired_count": "0",
        "runner": "serviceRunner"
    },
    "id": "d7e88fea-5a43-4a08-a5e9-c9b19baebefd",
    "modified_at": "2020-02-28T18:14:26Z",
    "modified_by": "someone",
    "name": "service-scale-job-test",
    "schedule_expression": "00 18 * * *",
    "enabled": true
}

task

A task runner job executes an action on a container taskdef. Currently supported action is run.

example task job

{
    "description": "Run 3 tasks from a taskdef",
    "details": {
        "task_action": "run",
        "task_cluster": "spindev-cluster-123",
        "task_name": "spindev-svc-123",
        "count": "0",
        "runner": "taskRunner"
    },
    "id": "a8d4d056-fbfe-4d7d-beb4-a64780c6d98c",
    "modified_at": "2021-10-28T18:14:26Z",
    "modified_by": "someone_else",
    "name": "task-run-job-test",
    "schedule_expression": "00 18 * * *",
    "enabled": true
}

Create a Job

POST /v1/minion/{account}/jobs/space-xy

Request

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ]
}

Response

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-27T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true,
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

Update a Job

PUT /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Request

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": false
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ]
}

Response

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-28T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": false
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

List Jobs in an account

GET /v1/minion/{account}/jobs

Response

[
    "space-xy/11e7b876-7a10-4c3e-93a7-e77eb3c68b58",
    "space-xy/6bcfa79f-615e-470d-97c1-687f3357497d",
    "space-ab/747f437a-a4af-48b2-a021-888bb8943a9b",
    "space-ab/7cf7433f-f6c2-496e-8fe9-ed4776d130c1"
]

List Jobs in a space

GET /v1/minion/{account}/jobs/space-xy

Response

[
    "11e7b876-7a10-4c3e-93a7-e77eb3c68b58",
    "6bcfa79f-615e-470d-97c1-687f3357497d"
]

Get a Job

GET /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

{
    "job": {
        "description": "Do some dumb thing to my server",
        "details": {
            "runner": "dummyRunner"
        },
        "group": "space-xy",
        "id": "6bcfa79f-615e-470d-97c1-687f3357497d",
        "modified_at": "2020-02-28T16:22:09Z",
        "modified_by": "someone",
        "name": "dummy-spin1234567",
        "schedule_expression": "* * * ? *",
        "enabled": true
    },
    "tags": [
        {
            "key": "foo",
            "value": "bar"
        },
        {
            "key": "baz",
            "value": "biz"
        }
    ],
    "next": "2020-02-27T16:23:09Z"
}

Delete a Job

DELETE /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Delete all jobs in a group

DELETE /v1/minion/{account}/jobs/space-xy

Run a Job

PATCH /v1/minion/{account}/jobs/space-xy/6bcfa79f-615e-470d-97c1-687f3357497d

Note: At the moment, this checks if the job exists in the jobs repository and then adds it to the jobs queue. There is a potential race condition, since the jobs queue reads from the local cache when executing jobs, so it may be missing if it was just created and hasn't been cached by the loader yet.

IAM permissions

S3 repository Example

create `minion-dev-bucket` and create a user with the policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::minion-dev-bucket/*",
                "arn:aws:s3:::minion-dev-bucket"
            ]
        }
    ]
}

Cloudwatchlogs Example

policy example for cloudwatchlogs-creater-consumer

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:ListTagsLogGroup",
                "logs:CreateLogStream",
                "logs:TagLogGroup",
                "logs:DescribeLogGroups",
                "logs:DeleteLogGroup",
                "logs:DescribeLogStreams",
                "logs:GetLogEvents",
                "logs:PutRetentionPolicy",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:*:*:log-group:localdev-*:log-stream:*",
                "arn:aws:logs:us-east-1:012345678910:log-group:localdev-*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "*"
        }
    ]
}

Author

E Camden Fisher camden.fisher@yale.edu

License

GNU Affero General Public License v3.0 (GNU AGPLv3)
Copyright (c) 2020 Yale University

Name		Name	Last commit message	Last commit date
Latest commit History 41 Commits
.github		.github
api		api
cloudwatchlogs		cloudwatchlogs
common		common
config		config
docker		docker
jobs		jobs
k8s		k8s
namesgenerator		namesgenerator
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
go.mod		go.mod
go.sum		go.sum
main.go		main.go
skaffold.yaml		skaffold.yaml

License

YaleSpinup/minion

Folders and files

Latest commit

History

Repository files navigation

minion

Endpoints

Usage

Authentication

Job Types

dummy

example dummy job

instance

example instance job

database

example database job

service

example service job

task

example task job

Create a Job

Request

Response

Update a Job

Request

Response

List Jobs in an account

Response

List Jobs in a space

Response

Get a Job

Delete a Job

Delete all jobs in a group

Run a Job

IAM permissions

S3 repository Example

create minion-dev-bucket and create a user with the policy

Cloudwatchlogs Example

policy example for cloudwatchlogs-creater-consumer

Author

License

About

Resources

License

Stars

Watchers

Forks

Releases 39

Packages 0

Contributors 5

Languages

create `minion-dev-bucket` and create a user with the policy

Packages