REST API and back-end implementation for auto pick

[levinas]

To be used to implement

• ar-get (default or --auto) > contigs.fa

[cbun]

These routes are now available on the test node:

10.0.28.15

# assemblies
/user/USER/job/JOBID/assemblies
/user/USER/job/JOBID/assemblies/<assembly.id>
/user/USER/job/JOBID/assemblies/auto

The return format is different from the route /assembly used in current arast.py. This one gives more information (shock_url, name). Let me know if additional metadata would be useful. You will need to run jobs on this server to completion before you will get any contigs from this.

Example:

`curl 10.0.28.15:8000/user/cbun/job/673/assemblies`

[{"shock_url": "140.221.84.205:8000", "create_time": null, "local_file": null, "filesize": null, "metadata": null, "shock_id": "61f94023-7fd9-40c6-bad1-400957c43443", "filename": "P2_S1_velvet__contigs"}, {"shock_url": "140.221.84.205:8000", "create_time": null, "local_file": null, "filesize": null, "metadata": null, "shock_id": "4ea834d5-90e8-497b-9a3f-9d475f2419c8", "filename": "P1_S1_kiki__kiki"}]

[cbun]

8ddece4 for API

[levinas]

Wonderful.

Sent from my iPhone

On May 22, 2014, at 5:29 PM, Christopher Bun notifications@github.com wrote:

These routes are now available on the test node:

10.0.28.15

assemblies

/user/USER/job/JOBID/assemblies
/user/USER/job/JOBID/assemblies/<assembly.id>
/user/USER/job/JOBID/assemblies/auto
The return format is different from the route /assembly used in current arast.py. This one gives more information (shock_url, name). Let me know if additional metadata would be useful. You will need to run jobs on this server to completion before you will get any contigs from this.

Example:
curl 10.0.28.15:8000/user/cbun/job/673/assemblies
—
Reply to this email directly or view it on GitHub.

[levinas]

Is auto pick N50-based?

[cbun]

No, there is no backend implementation yet. Just an api to work against
for now (and a man behind the curtain). N50 is the first target after
quast parser is put in.
On May 22, 2014 10:25 PM, "Fangfang Xia" notifications@github.com wrote:

Is auto pick N50-based?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/45#issuecomment-43967868
.

[levinas]

Ok, this doesn't affect the one-assembler case. Reopening the ticket just to keep track of it. Maybe we should separate this into two..

[cbun]

I think we can close them if they are sufficient for the demo milestone, and open new ones that target a future milestone.

[levinas]

Ok, I'll close it. We do want at least N50-based auto pick for the real demo. That will be a separate ticket.

For now, is there a hackish way to look at the N50? Or even a mechanism based on a fixed preferences of assemblers (e.g., spades > idba > velvet)...

On May 23, 2014, at 11:47 AM, Christopher Bun notifications@github.com wrote:

I think we can close them if they are sufficient for the demo milestone, and open new ones that target a future milestone.

—
Reply to this email directly or view it on GitHub.

[cbun]

The truly hackish way would be to serve the quast report and print it and parse it: #46 (comment)

Until the compute nodes have stats-generating capability, it will all be a hack. So it is a question of how much energy do we want to spend on a hack for a demo.

I think if the REST API is in place, we can develop against it and have the REST server wave hands until the true implementations are created. This way we are putting energy towards code that will persist rather than be scrapped.

So developing the client against /assemblies/auto should trust that a solution will be in place by the demo. The demo is in August right?

[levinas]

Yes sounds good.

[cbun]

I've decided that the API shouldn't be overly specific and that the clients should do the sorting work rather than the router... So this is how it works now:
#29 (comment)

/user/{user.id}/job/{job.id}/results{?type,tags}

Where type can be things like contigs,report, alignment etc, and tags can easily be appended to metadata by the compute like best or largest_n50. This will make development much easier.

Let me know what you think and close this ticket if you agree

[levinas]

Sounds good.

[sebhtml]

I still believe that /job/xyz should have its own top-level rest endpoint (/job/xyz instead of /user/111/job/xyz). I guess we can agree to disagree on that !

[cbun]

Can you discuss what benefit this has? I have no problem with global job ids if they are more beneficial. Just grabbing the nearest example... this is issue https://github.com/kbase/assembly/issues/45, not https://github.com/issues/453452345245234

[levinas]

In addition to this, I named one benefit for user-level job IDs which is being able to support some range operator:

ar-compare -j 99..105

On May 30, 2014, at 3:03 PM, Christopher Bun notifications@github.com wrote:

Can you discuss what benefit this has? I have no problem with global job ids if they are more beneficial. Just grabbing the nearest example... this is issue #45, not https://github.com/issues/453452345245234

—
Reply to this email directly or view it on GitHub.

[sebhtml]

@cbun

Can you discuss what benefit this has?
I have no problem with global job ids if they are
more beneficial. Just grabbing the nearest example... this is issue
#45, not https://github.com/issues/453452345245234

If I compare things:

a github issue:
octocat/Hello-World#1347 (HTML)
https://api.github.com/repos/octocat/Hello-World/issues/1347 (REST)

an KBase assembly job (I made up the URL):
https://assembly.kbase.us/user/sebhtml/job/1 (REST)

So on second thought, your design is better. But I think it would be cool to share data and jobs
between users like on github.

Also, is there a reference REST API for the assembly service (or a working copy / draft) ?

[sebhtml]

@levinas

In addition to this, I named one benefit for user-level job IDs which is being able to support
some range operator: ar-compare -j 99..105

Jobs could be tagged with "super-cool-set-of-jobs" and then the jobs could be compared with:

curl -X POST -H "Authorization: John Smith" -H "Content-Type: application/json" -d '{"pipeline":"MegaComparator", "tag": "super-cool-set-of-jobs", "message":"xyz", "queue":"xyz", "version":"xyz"}' https://assembly.kbase.us/job/new

One possible problem with ranges is that if one job failed, then the range is a failure, right ?

https://github.com/kbase/assembly/blob/841e27aec5911242916a14d39f97d572513e0936/lib/assembly/client.py