Data

This pages describes how to load snippets into HiPiler and extract the location of snippets. It also explains how to load HiPiler in a Jupyter notebook.

Config file

HiPiler's configuration is specified in JSON and consists of two main parts: one for the snippet location and display and another one for HiGlass. The snippet configuration is stored under fgm and HiGlass is configured under hgl.

This is an example for the snippet configuration:

{
  "fgm": {
    "fragmentsServer": "//34.199.119.21",
    "fragmentsPrecision": 2,
    "fragmentsDims": 22,
    "fragmentsPadding": 0,
    "fragmentsNoCache": false,
    "fragmentsNoBalance": false,
    "fragmentsPercentile": 99,
    "fragmentsIgnoreDiags": 1,
    "fragments": [
      ["chrom1", "start1", "end1", "strand1", "chrom2", "start2", "end2", "strand2", "dataset", "zoomOutLevel", "size", "distance-to-diagonal", "noise", "_groupA", , "_groupB"],
      ["22", 17395000, 17400000, "coding", "22", 17535000, 17540000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 135000, 0.21559968240959723, 1, 1],
      ["22", 17400000, 17410000, "coding", "22", 17980000, 17990000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 100000000, 570000, 0.19905589933149648, 1, 4],
      ["22", 17650000, 17655000, "coding", "22", 17980000, 17985000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 325000, 0.14815256212601854, 3, 2],
      ...
    ]
  },
  ...
}

The parameters are defined as follows:

• fragmentsServer: URL to a HiGlass server
• fragmentsPrecision [optional; default: 2]: Float precision of the raw matrix snippets to be returned by the server
• fragmentsDims [optional; default: 22]: Pixel dimension of the matrix snippets to be returned by the server. Since HiPiler currently only supports squared snippets only one dimension is needed.
• fragmentsPadding [optional; default: 0]: Number of bins used for padding. This only matters when loading domains.
• fragmentsNoCache [optional; default: false]: If true disables loading snippets from cache. Good for testing.
• fragmentsNoBalance [optional; default: false]: If true disables Cooler's balancing.
• fragmentsPercentile [optional; default: 100]: Percentile at which the values are capped.
• fragmentsIgnoreDiags [optional; default: 0]: Number of diagonals that are ignored, i.e., set to zero.
• fragments: BEDPE-like table of snippet locations. For details see below.

BEDPE-like table

Like a simple spreadsheet, the first row of the BEDPE-like location table contains the header. The following columns are required by HiPiler. Examples are in brackets

• chrom1 [str, e.g.: 'chr1' or '1']
• start1 [int, e.g.: 0]
• end1 [int, e.g.: 0]
• strand1 [str, optional, e.g.: '-', 'minus', 'noncoding', or 'non-coding' for the minus strand; otherwise plus strand is assumed]
• chrom2 (like chrom1)
• start2 (like start1)
• end2 (like end1)
• strand2 (like stand1, optional)
• dataset [str, e.g.: 'CQMd6V_cRw6iCI_-Unl3PQ']
• zoomoutlevel [int, e.g.: 0]

The dataset is the UUID of the Hi-C map where the locations should be extracted. zoomoutlevel at which zoom level the snippets should be extracted, i.e., 2^zoomoutlevel * base resolution, e.g., for zoomoutlevel 3 with base resolution 1KB: 2^3 = 8 and 8 * 1KB = 8KB.

Additionally, an unlimited number of extra columns can be appended to load categorical and ordinal attributes for exploration. Ordinal attributes just need to specified by a unique name, e.g., size, distance-to-diagonal, and noise. They appear as measure for arrangement of snippets in HiPiler. Categorical attributes need must be prefixed with an underscore (_), e.g., _groupA and _groupB. They appear as grouping options in HiPiler.

The local neighborhood (i.e., the Hi-C map) of snippets are shown in HiGlass. The configuration of HiGlass in HiPiler is unrestricted except that one should only configure one row of primary matrices and avoid vertical arrangement of multiple matrices as the vertical space is needed for the detail matrices. For instructions on how to configure HiGlass please visit its wiki.

Examples

A set of example / demo configurations are available here. Look for HiPiler Example.

Import

There are three options to load a configuration:

1. Drag and drop JSON onto the browser window of HiPiler.
2. Go to the home page and click on Select a config.
3. You can direct load hosted config files by going to http://hipiler.higlass.io/?config=URL_TO_YOUR_CONFIG, where URL_TO_YOUR_CONFIG should point to your config. E.g.: http://hipiler.higlass.io/?config=https://rawgit.com/flekschas/8b0163f25fd4ffb067aaba2a595da447/raw/rao-2014-gm12878-mbol-1kb-mres-chr22-loops.json

Export

There are two options to export the snippets / pile configuration, including annotations, as JSON:

Note: You have to be at /explore to export anything.

1. Open the snippet's menu by clicking on More at the bottom and then on Export Piles.
2. Hit CMD + S.

Jupyter notebook integration

Coming soon