Programmatic_Access.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Programmatic Access to EMBL EBI #\n",
    "******\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the resources at EMBL-EBI are freely available and can be explored from https://www.ebi.ac.uk/services. You can access the resources either using a browser interface or programmatically i.e., using web services. A list of EMBL-EBI Web Services APIs for data retrieval resources is given in https://bit.ly/EMBL-EBI-APIs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some examples to access some of our services programmatically: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Retrieve data from DbFetch ##\n",
    "\n",
    "[DbFetch](https://www.ebi.ac.uk/Tools/dbfetch/) provides an easy way to retrieve entries from various databases at the EMBL-EBI in a consistent manner.\n",
    "\n",
    "You can retrieve data from DbFetch using our web interface http://www.ebi.ac.uk/Tools/dbfetch/\n",
    "\n",
    "Required parameters: \n",
    "1. Database name\n",
    "2. Sequence id\n",
    "\n",
    "Optional Parameters\n",
    "1. Style (default: html)\n",
    "2. Format (default format is often the default format for the specified database)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Retrieve data from DbFetch using REST URL ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To retrieve a protein sequence entry (example WAP_MOUSE) from the Uniprot Knowledgebase, we could open a browser window and try the following URL\n",
    "https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=WAP_MOUSE\n",
    " \n",
    "\n",
    "database is _uniprotkb_ and the sequence id is _WAP_RAT_\n",
    "\n",
    "https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=WAP_MOUSE;style=raw \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run from command line ####\n",
    "\n",
    "You can retrieve the data using _curl_ or _wget_ commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X GET \"https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=WAP_MOUSE;style=raw\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!curl -X GET 'https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=WAP_MOUSE;style=raw' -o wap_mouse.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 30 wap_mouse.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X GET 'https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=WAP_MOUSE;style=raw;format=fasta' -o wap_mouse.fasta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head wap_mouse.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Incorporate the calls into a script ####\n",
    "Suppose you want to bulk download a set of sequence ids in fasta format\n",
    "Create a file (eg: data.txt) with the list of sequence ids (WAP_RAT,WAP_PIG)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using shell command\n",
    "\n",
    "!for line in `cat data.txt`; do curl -X GET \"https://www.ebi.ac.uk/Tools/dbfetch/dbfetch?db=uniprotkb;id=$line;style=raw;format=fasta\" -o $line.fasta; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using file upload\n",
    "save the file as upload.txt in specified format (eg:uniprotkb:WAP_RAT)and use _CURL_ to _POST_ the request"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X POST http://www.ebi.ac.uk/Tools/dbfetch/dbfetch -F \"listfile=@/mnt/c/Training/upload.txt\" -F \"format=fasta\" -F \"style=raw\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Web Service Client ####\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python, Perl and Java clients are provided for EBI Tools Web Services from https://github.com/ebi-wp/webservice-clients."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To simplify the process, we can download one of the clients (e.g. DBfetch) and run it in alternative to using the curl command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the raw client from GitHub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/ebi-wp/webservice-clients/master/python/dbfetch.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One can learn more about available parameters and how to use them by typing python <client_name>.py --help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python dbfetch.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Retrieving the same sequence in fasta format from Uniprot could be done using the Python client as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python dbfetch.py fetchData uniprotkb:WAP_MOUSE fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the above command failed, some dependencies might be missing. See instructions for installing them in https://github.com/ebi-wp/webservice-clients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Run NCBI Blast+ ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to data retrieval, EMBL-EBI provides Web Services for popular Bioinformatics Applications such as NCBI BLAST+, Clustal Omega, InterProScan 5, and HMMER. Programmatic access to these services can be explored from https://www.ebi.ac.uk/Tools/webservices. The common API can be browsed from https://www.ebi.ac.uk/Tools/common/tools/help/\n",
    "\n",
    "Since data needs to be passed to the server for the application to run (i.e. some input sequence data, in this case), the request uses the POST HTTP verb.\n",
    "In this example, we can run NCBI BLAST+ using Swissprot database (uniprotkb_swissprot) and using a UniProt sequence accession number as the input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.Run NCBI Blast+ using REST URL\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X POST --header 'Content-Type: application/x-www-form-urlencoded' --header 'Accept: text/plain' -d 'email=test@ebi.ac.uk&program=blastp&stype=protein&sequence=sp:wap_rat&database=uniprotkb_swissprot' 'https://www.ebi.ac.uk/Tools/services/rest/ncbiblast/run'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, in this case we passed `-X POST` instead of `-X GET`. Additionally, we needed to pass `-d or --data`. The various parameter options were separated with `&` symbol. \n",
    "The parameters were:\n",
    "* `email=test\\@ebi.ac.uk`\n",
    "* `program=blastp`\n",
    "* `stype=protein` \n",
    "* `sequence=sp:wap_rat`\n",
    "* `database=uniprotkb_swissprot`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For retrieving a BLAST output for the previous job with (example `jobid=ncbiblast-R20190225-110224-0172-5825946-p1m`) we need to use a different endpoint `/result/{jobId}/{resultType}`. In this case we can retrieve the default BLAST output format which has been named `out`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!curl -X GET --header 'Accept: text/plain' 'https://www.ebi.ac.uk/Tools/services/rest/ncbiblast/result/ncbiblast-R20191008-111833-0999-94082917-p1m/out' -o blast.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -n 50 blast.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Run NCBI Blast+ using Web Service Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly to what we have done for Dbfetch, we can download the clients and perform various sequence analysis using the available Bioinformatics Applications. In this example we run the same BLAST sequence search as we performed with curl."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: we are getting the raw client from GitHub\n",
    "!wget https://raw.githubusercontent.com/ebi-wp/webservice-clients/master/python/ncbiblast.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the clients can run the job and return outputs on the same call (synchronously as below, or asynchronously)\n",
    "!python ncbiblast.py --email test@ebi.ac.uk --program blastp --stype protein --sequence sp:wap_rat --database uniprotkb_swissprot --outformat out --outfile wap_rat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Useful links\n",
    "\n",
    "EMBL-EBI services and data resources: https://www.ebi.ac.uk/services  \n",
    "EMBL-EBI APIs:https://bit.ly/EMBL-EBI-APIs  \n",
    "EMBL-EBI Web Services General Documentation: https://www.ebi.ac.uk/Tools/webservices  \n",
    "Web Service Clients for EBI Tools and EBI Search: https://github.com/ebi-wp/webservice-clients  \n",
    "RESTful API (SWAGGER) User Interface for EBI Tools: https://www.ebi.ac.uk/Tools/common/tools/help/  \n",
    "RESTful API (SWAGGER) User Interface for EBI Search: https://www.ebi.ac.uk/ebisearch/swagger.ebi  \n",
    "\n",
    "**Contact us via Help and Support at https://www.ebi.ac.uk/support/webservices**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15rc1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}