-
Notifications
You must be signed in to change notification settings - Fork 0
Data
This pages describes how to load snippets into HiPiler and extract the location of snippets. It also explains how to load HiPiler in a Jupyter notebook.
Using a CSV file is the fastest way to explore annotations. The table should have the following form:
chrom1 | start1 | end1 | strand1 | chrom2 | start2 | end2 | strand2 | dataset | zoomOutLevel | server | coords | pVal | _group |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
str | int | int | str | str | int | int | str | str | int | str | str | float | str |
chr4 | 69285000 | 70075000 | + | chr4 | 69285000 | 70075000 | + | CQMd6V_cRw6iCI_-Unl3PQ | -1 | //higlass.io | hg19 | 0.001 | A |
The first row defines the data type and the second row gives an example. The last two columns are optional. After coords
any additional column that does not start with an underscore (e.g., pVal
) is assumed to be a numerical measurement and can be used to explore the annotations in HiPiler. Any additional columns starting with an underscore (e.g., _group
) are regarded as a categorical attribute of the annotation and can be used for automatic grouping of annotations in HiPiler.
zoomOutLevel: When working with TADs or other larger and differently-sized structural features it can be best to let HiPiler choose the optimal resolution for cutting out the data. To do so, set zoomOutLevel
to -1
.
Custom Gene Annotations: Starting with HiPiler v1.4
, If you're working with a custom coordinate system it can be handy to also display custom gene annotations on top of the heatmap. To do so specify the name of the gene annotations or the tileset UUID of the gene annotation track in a special column called _gene_annotations
.
A set of example are available here. Look for HiPiler CSV Example. A concrete example is available at https://gist.github.com/flekschas/f3ebccf8156af5d22aa0d90e7cbc925f.
HiPiler's view config is specified in JSON and consists of two main parts: one for the snippet location and display and another one for HiGlass. The snippet configuration is stored under fgm
and HiGlass is configured under hgl
.
This is an example for the snippet configuration:
{
"fgm": {
"fragmentsServer": "//34.199.119.21",
"fragmentsPrecision": 2,
"fragmentsDims": 22,
"fragmentsPadding": 0,
"fragmentsNoCache": false,
"fragmentsNoBalance": false,
"fragmentsPercentile": 99,
"fragmentsIgnoreDiags": 1,
"fragments": [
["chrom1", "start1", "end1", "strand1", "chrom2", "start2", "end2", "strand2", "dataset", "zoomOutLevel", "size", "distance-to-diagonal", "noise", "_groupA", , "_groupB"],
["chr22", 17395000, 17400000, "coding", "chr22", 17535000, 17540000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 135000, 0.21559968240959723, 1, 1],
["chr22", 17400000, 17410000, "coding", "chr22", 17980000, 17990000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 100000000, 570000, 0.19905589933149648, 1, 4],
["chr22", 17650000, 17655000, "coding", "chr22", 17980000, 17985000, "coding", "CQMd6V_cRw6iCI_-Unl3PQ", 3, 25000000, 325000, 0.14815256212601854, 3, 2],
...
]
},
...
}
The parameters are defined as follows:
- fragmentsServer: URL to a HiGlass server
-
fragmentsPrecision [optional; default:
2
]: Float precision of the raw matrix snippets to be returned by the server -
fragmentsDims [optional; default:
22
]: Pixel dimension of the matrix snippets to be returned by the server. Since HiPiler currently only supports squared snippets only one dimension is needed. -
fragmentsPadding [optional; default:
0
]: Number of bins used for padding. This only matters when loading domains. -
fragmentsNoCache [optional; default:
false
]: Iftrue
disables loading snippets from cache. Good for testing. -
fragmentsNoBalance [optional; default:
false
]: Iftrue
disables Cooler's balancing. -
fragmentsPercentile [optional; default:
100
]: Percentile at which the values are capped. -
fragmentsIgnoreDiags [optional; default:
0
]: Number of diagonals that are ignored, i.e., set to zero. - fragments: BEDPE-like table of snippet locations. For details see below.
Like a simple spreadsheet, the first row of the BEDPE-like location table contains the header. The following columns are required by HiPiler. Examples are in brackets
- chrom1 [str, e.g.: 'chr1' or '1']
- start1 [int, e.g.: 0]
- end1 [int, e.g.: 0]
- strand1 [str, optional, e.g.: '-', 'minus', 'noncoding', or 'non-coding' for the minus strand; otherwise plus strand is assumed]
-
chrom2 (like
chrom1
) -
start2 (like
start1
) -
end2 (like
end1
) -
strand2 (like
stand1
, optional) - dataset [str, e.g.: 'CQMd6V_cRw6iCI_-Unl3PQ']
- zoomoutlevel [int, e.g.: 0]
The dataset
is the UUID of the Hi-C map where the locations should be extracted. zoomoutlevel
at which zoom level the snippets should be extracted, i.e., 2^zoomoutlevel * base resolution, e.g., for zoomoutlevel 3 with base resolution 1KB: 2^3 = 8 and 8 * 1KB = 8KB.
Additionally, an unlimited number of extra columns can be appended to load categorical and ordinal attributes for exploration. Ordinal attributes just need to specified by a unique name, e.g., size, distance-to-diagonal, and noise. They appear as measure for arrangement of snippets in HiPiler. Categorical attributes need must be prefixed with an underscore (_), e.g., _groupA and _groupB. They appear as grouping options in HiPiler.
The local neighborhood (i.e., the Hi-C map) of snippets are shown in HiGlass. The configuration of HiGlass in HiPiler is unrestricted except that one should only configure one row of primary matrices and avoid vertical arrangement of multiple matrices as the vertical space is needed for the detail matrices. For instructions on how to configure HiGlass please visit its wiki.
A set of example / demo configurations are available here. Look for HiPiler Example.
There are three options to load a configuration:
- Drag and drop JSON view config or CSV file onto the browser window of HiPiler.
- Go to the home page and click on Select a config.
- You can direct load hosted config files by going to http://hipiler.higlass.io/?config=URL_TO_YOUR_CONFIG, where
URL_TO_YOUR_CONFIG
should point to your config. E.g.: http://hipiler.higlass.io/?config=https://rawgit.com/flekschas/8b0163f25fd4ffb067aaba2a595da447/raw/rao-2014-gm12878-mbol-1kb-mres-chr22-loops.json
There are two things you can export:
- The annotations you have explored as CSV that have been augmented with grouping and trashing information Open the snippet's menu by clicking on More at the bottom and then on Export Piles or hit CMD + S.
- The view config, which you can send somebody else for exploring the data. Open the snippet's menu by clicking on More at the bottom and then on Export View.
Note: You have to be at /explore
to export something.
Coming soon