Data

This pages describes how to load snippets into HiPiler and extract the location of snippets. It also explains how to load HiPiler in a Jupyter notebook.

CSV

Using a CSV file is the fastest way to explore annotations. The table should have the following form:

chrom1	start1	end1	strand1	chrom2	start2	end2	strand2	dataset	zoomOutLevel	server	coords	pVal	_group
str	int	int	str	str	int	int	str	str	int	str	str	float	str
4	69285000	70075000	+	4	69285000	70075000	+	CQMd6V_cRw6iCI_-Unl3PQ	0	//higlass.io	hg19	0.001	A

The first row defines the data type and the second row gives an example. The last two columns are optional. After coords any additional column that does not start with an underscore (e.g., pVal) is assumed to be some kind of numerical measurement and can be used to explore the annotations in HiPiler. Any additional column starting with an underscore (e.g., _group) are assumed to be a categorical attribute of the annotation and can be used for automatic grouping of annotations in HiPiler.

Examples

A set of example are available here. Look for HiPiler CSV Example. A concrete example is available at https://gist.github.com/flekschas/f3ebccf8156af5d22aa0d90e7cbc925f.

View Config

HiPiler's view config is specified in JSON and consists of two main parts: one for the snippet location and display and another one for HiGlass. The snippet configuration is stored under fgm and HiGlass is configured under hgl.

This is an example for the snippet configuration:

{
  "fgm": {
    "fragmentsServer": "//34.199.119.21",
    "fragmentsPrecision": 2,
    "fragmentsDims": 22,
    "fragmentsPadding": 0,
    "fragmentsNoCache": false,
    "fragmentsNoBalance": false,
    "fragmentsPercentile": 99,
    "fragmentsIgnoreDiags": 1,
    "fragments": [
      ["chrom1", "start1", "end1", "strand1", "chrom2", "start2", "end2", "strand2", "dataset", "zoomOutLevel", "size", "distance-to-diagonal", "noise", "_groupA", , "_groupB"],
      ["22", 17395000, 17400000, "coding", "22", 17535000, 17540000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 135000, 0.21559968240959723, 1, 1],
      ["22", 17400000, 17410000, "coding", "22", 17980000, 17990000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 100000000, 570000, 0.19905589933149648, 1, 4],
      ["22", 17650000, 17655000, "coding", "22", 17980000, 17985000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 325000, 0.14815256212601854, 3, 2],
      ...
    ]
  },
  ...
}

The parameters are defined as follows:

fragmentsServer: URL to a HiGlass server
fragmentsPrecision [optional; default: 2]: Float precision of the raw matrix snippets to be returned by the server
fragmentsDims [optional; default: 22]: Pixel dimension of the matrix snippets to be returned by the server. Since HiPiler currently only supports squared snippets only one dimension is needed.
fragmentsPadding [optional; default: 0]: Number of bins used for padding. This only matters when loading domains.
fragmentsNoCache [optional; default: false]: If true disables loading snippets from cache. Good for testing.
fragmentsNoBalance [optional; default: false]: If true disables Cooler's balancing.
fragmentsPercentile [optional; default: 100]: Percentile at which the values are capped.
fragmentsIgnoreDiags [optional; default: 0]: Number of diagonals that are ignored, i.e., set to zero.
fragments: BEDPE-like table of snippet locations. For details see below.

BEDPE-like table

Like a simple spreadsheet, the first row of the BEDPE-like location table contains the header. The following columns are required by HiPiler. Examples are in brackets

chrom1 [str, e.g.: 'chr1' or '1']
start1 [int, e.g.: 0]
end1 [int, e.g.: 0]
strand1 [str, optional, e.g.: '-', 'minus', 'noncoding', or 'non-coding' for the minus strand; otherwise plus strand is assumed]
chrom2 (like chrom1)
start2 (like start1)
end2 (like end1)
strand2 (like stand1, optional)
dataset [str, e.g.: 'CQMd6V_cRw6iCI_-Unl3PQ']
zoomoutlevel [int, e.g.: 0]

The dataset is the UUID of the Hi-C map where the locations should be extracted. zoomoutlevel at which zoom level the snippets should be extracted, i.e., 2^zoomoutlevel * base resolution, e.g., for zoomoutlevel 3 with base resolution 1KB: 2^3 = 8 and 8 * 1KB = 8KB.

Additionally, an unlimited number of extra columns can be appended to load categorical and ordinal attributes for exploration. Ordinal attributes just need to specified by a unique name, e.g., size, distance-to-diagonal, and noise. They appear as measure for arrangement of snippets in HiPiler. Categorical attributes need must be prefixed with an underscore (_), e.g., _groupA and _groupB. They appear as grouping options in HiPiler.

The local neighborhood (i.e., the Hi-C map) of snippets are shown in HiGlass. The configuration of HiGlass in HiPiler is unrestricted except that one should only configure one row of primary matrices and avoid vertical arrangement of multiple matrices as the vertical space is needed for the detail matrices. For instructions on how to configure HiGlass please visit its wiki.

Examples

A set of example / demo configurations are available here. Look for HiPiler Example.

Import

There are three options to load a configuration:

Drag and drop JSON view config or CSV file onto the browser window of HiPiler.
Go to the home page and click on Select a config.
You can direct load hosted config files by going to http://hipiler.higlass.io/?config=URL_TO_YOUR_CONFIG, where URL_TO_YOUR_CONFIG should point to your config. E.g.: http://hipiler.higlass.io/?config=https://rawgit.com/flekschas/8b0163f25fd4ffb067aaba2a595da447/raw/rao-2014-gm12878-mbol-1kb-mres-chr22-loops.json

Export

There are two things you can export:

The annotations you have explored as CSV that have been augmented with grouping and trashing information Open the snippet's menu by clicking on More at the bottom and then on Export Piles or hit CMD + S.
The view config, which you can send somebody else for exploring the data. Open the snippet's menu by clicking on More at the bottom and then on Export View.

Note: You have to be at /explore to export something.