Skip to content

Commit

Permalink
Merge pull request #172 from pepkit/dev
Browse files Browse the repository at this point in the history
Release 0.11.1
  • Loading branch information
vreuter authored Apr 24, 2019
2 parents 48cb76e + 844041d commit ddf3097
Show file tree
Hide file tree
Showing 12 changed files with 375 additions and 440 deletions.
2 changes: 1 addition & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

## What is looper?

`Looper` is a pipeline submitting engine. `Looper` deploys any command-line pipeline across samples organized in [standard PEP format](https://pepkit.github.io/docs/home/). You can think of `looper` as providing a single user interface to running, summarizing, monitoring, and otherwise managing *all* of your sample-intensive research projects.
`Looper` is a pipeline submitting engine. `Looper` deploys any command-line pipeline for each sample in a project organized in [standard PEP format](https://pepkit.github.io/docs/home/). You can think of `looper` as providing a single user interface to running, summarizing, monitoring, and otherwise managing all of your sample-intensive research projects the same way, regardless of data type or pipeline used.

## What makes looper better?

Expand Down
328 changes: 122 additions & 206 deletions docs/changelog.md
Original file line number Diff line number Diff line change
@@ -1,233 +1,149 @@
# Changelog

This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html) and [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.

- **Unreleased**
## [Unreleased]

## [0.11.1] - 2019-04-17

- **v0.11 (*2019-04-17*)**
### Changed
- Improved documentation
- Improved interaction with `peppy` and `divvy` dependencies

- Added
## [0.11] - 2019-04-17

- Implemented `looper rerun` command.
### Added
- Implemented `looper rerun` command.
- Support use of custom `resources` in pipeline's `compute` section
- Listen for itemized compute resource specification on command-line with `--resources`
- Support pointing to `Project` config file with folder path rather than full filepath
- Add `selector-attribute` parameter for more generic sample selection.

- Support use of custom `resources` in pipeline's `compute` section

- Listen for itemized compute resource specification on command-line with `--resources`

- Support pointing to `Project` config file with folder path rather than full filepath

- Changed

- Switched to a Jinja-style templating system for summary output

- Made various UI changes to adapt to `caravel` use.

- Using `attmap` for "attribute-style key-vale store" implementation

- Removed Python 3.4 support.

- UI: change parameter names `in/exclude-samples` to `selector-in/exclude`.

- New:

- Add `selector-attribute` parameter for more generic sample selection.


- **v0.10.0 (*2018-12-20*)**

- Changed

- ``PipelineInterface`` now derives from ``peppy.AttributeDict``.

- On ``PipelineInterface``, iteration over pipelines now is with ``iterpipes``.

- Rename ``parse_arguments`` to ``build_parser``, which returns ``argparse.ArgumentParser`` object

- Integers in HTML reports are made more human-readable by including commas.

- Column headers in HTML reports are now stricly for sorting; there's a separate list for plottable columns.

- More informative error messages

- Fixed

- HTML samples list is fully populated.

- Existence of an object lacking an anchor image is no longer problematic for ``summarize``.

- Basic package test in Python 3 now succeeds: ``python3 setup.py test``.

- **v0.9.2** (*2018-11-12*):

- Fixed

- Fixed bugs with ``looper summarize`` when no summarizers were present

- Added CLI flag to force ``looper destroy`` for programmatic access

- Fixed a bug for samples with duplicate names

- Added new display features (graphs, table display) for HTML summary output.


- **v0.9.1** (*2018-06-30*):

- Fixed

- Fixed several bugs with ``looper summarize`` that caused failure on edge cases.


- **v0.9.0** (*2018-06-25*):

- New

- Support for custom summarizers

- Add ``allow-duplicate-names`` command-line options

- Allow any variables in environment config files or other ``compute`` sections to be used in submission templates. This allows looper to be used with containers.

- Add nice universal project-level HTML reporting


- **v0.8.1** (*2018-04-02*):

- Changed

- Minor documentation and packaging updates for first Pypi release.

- Fix a bug that incorrectly mapped protocols due to case sensitive issues

- Fix a bug with ``report_figure`` that made it output pandas code


- **v0.8.0** (*2018-01-19*):

- Changed

- Use independent `peppy` package, replacing ``models`` module for core data types.

- Integrate ``ProtocolInterface`` functionality into ``PipelineInterface``.

- **v0.7.2** (*2017-11-16*):

- Fixed
### Changed
- Switched to a Jinja-style templating system for summary output
- Made various UI changes to adapt to `caravel` use.
- Using `attmap` for "attribute-style key-vale store" implementation
- Removed Python 3.4 support.
- UI: change parameter names `in/exclude-samples` to `selector-in/exclude`.

- Correctly count successful command submissions when not using `--dry-run`.
## [0.10.0] - 2018-12-20

- **v0.7.1** (*2017-11-15*):
### Changed
- `PipelineInterface` now derives from `peppy.AttributeDict`.
- On `PipelineInterface`, iteration over pipelines now is with `iterpipes`.
- Rename `parse_arguments` to `build_parser`, which returns `argparse.ArgumentParser` object
- Integers in HTML reports are made more human-readable by including commas.
- Column headers in HTML reports are now stricly for sorting; there's a separate list for plottable columns.
- More informative error messages
- HTML samples list is fully populated.
- Existence of an object lacking an anchor image is no longer problematic for `summarize`.
- Basic package test in Python 3 now succeeds: `python3 setup.py test`.

- Fixed

- No longer falsely display that there's a submission failure.

- Allow non-string values to be unquoted in the ``pipeline_args`` section.
## [v0.9.2] - 2018-11-12

- **v0.7** (*2017-11-15*):
### Changed
- Fixed bugs with `looper summarize` when no summarizers were present
- Added CLI flag to force `looper destroy` for programmatic access
- Fixed a bug for samples with duplicate names
- Added new display features (graphs, table display) for HTML summary output.

- New

- Add ``--lump`` and ``--lumpn`` options

- Catch submission errors from cluster resource managers

- Implied columns can now be derived

- Now protocols can be specified on the command-line `--include-protocols`

- Add rudimentary figure summaries

- Simplifies command-line help display

- Allow wildcard protocol_mapping for catch-all pipeline assignment

- Improve user messages

- New sample_subtypes section in pipeline_interface

- Changed

- Sample child classes are now defined explicitly in the pipeline interface. Previously, they were guessed based on presence of a class extending Sample in a pipeline script.

- Changed 'library' key sample attribute to 'protocol'

- **v0.6** (*2017-07-21*):
## [0.9.1] - 2018-06-30

- New
### Changed
- Fixed several bugs with `looper summarize` that caused failure on edge cases.

- Add support for implied_column section of the project config file

- Add support for Python 3

- Merges pipeline interface and protocol mappings. This means we now allow direct pointers to ``pipeline_interface.yaml`` files, increasing flexibility, so this relaxes the specified folder structure that was previously used for ``pipelines_dir`` (with ``config`` subfolder).

- Allow URLs as paths to sample sheets.

- Allow tsv format for sample sheets.

- Checks that the path to a pipeline actually exists before writing the submission script.
## [0.9.0] - 2018-06-25

- Changed
### Added
- Support for custom summarizers
- Add `allow-duplicate-names` command-line options
- Allow any variables in environment config files or other `compute` sections to be used in submission templates. This allows looper to be used with containers.
- Add nice universal project-level HTML reporting

- Changed LOOPERENV environment variable to PEPENV, generalizing it to generic models
## [0.8.1] - 2018-04-02

- Changed name of ``pipelines_dir`` to ``pipeline_interfaces`` (but maintained backwards compatibility for now).
### Changed
- Minor documentation and packaging updates for first Pypi release.
- Fix a bug that incorrectly mapped protocols due to case sensitive issues
- Fix a bug with `report_figure` that made it output pandas code

- Changed name of ``run`` column to ``toggle``, since ``run`` can also refer to a sequencing run.

- Relaxes many constraints (like resources sections, pipelines_dir columns), making project configuration files useful outside looper. This moves us closer to dividing models from looper, and improves flexibility.
## [0.8.0] - 2018-01-19

- Various small bug fixes and dev improvements.
### Changed
- Use independent `peppy` package, replacing `models` module for core data types.
- Integrate `ProtocolInterface` functionality into `PipelineInterface`.

- Require `setuptools` for installation, and `pandas 0.20.2`. If `numexpr` is installed, version `2.6.2` is required.
## [0.7.2] - 2017-11-16
### Changed
- Correctly count successful command submissions when not using `--dry-run`.

- Allows tilde in ``pipeline_interfaces``
## [0.7.1] - 2017-11-15

- **v0.5** (*2017-03-01*):
### Changed
- No longer falsely display that there's a submission failure.
- Allow non-string values to be unquoted in the `pipeline_args` section.

- New

- Add new looper version tracking, with `--version` and `-V` options and printing version at runtime

- Add support for asterisks in file paths

- Add support for multiple pipeline directories in priority order

- Revamp of messages make more intuitive output

- Colorize output

- Complete rehaul of logging and test infrastructure, using logging and pytest packages

- Changed

- Removes pipelines_dir requirement for models, making it useful outside looper

- Small bug fixes related to `all_input_files` and `required_input_files` attributes
## [0.7] - 2017-11-15
### Added
- Add `--lump` and `--lumpn` options
- Catch submission errors from cluster resource managers
- Implied columns can now be derived
- Now protocols can be specified on the command-line `--include-protocols`
- Add rudimentary figure summaries
- Simplifies command-line help display
- Allow wildcard protocol_mapping for catch-all pipeline assignment
- Improve user messages
- New sample_subtypes section in pipeline_interface

- More robust installation and more explicit requirement of Python 2.7


- **v0.4** (*2017-01-12*):

- New

- New command-line interface (CLI) based on sub-commands

- New subcommand (``looper summarize``) replacing the ``summarizePipelineStats.R`` script

- New subcommand (``looper check``) replacing the ``flagCheck.sh`` script

- New command (``looper destroy``) to remove all output of a project

- New command (``looper clean``) to remove intermediate files of a project flagged for deletion

- Support for portable and pipeline-independent allocation of computing resources with Looperenv.

- Changed

- Removed requirement to have ``pipelines`` repository installed in order to extend base Sample objects

- Maintenance of sample attributes as provided by user by means of reading them in as strings (to be improved further)

- Improved serialization of Sample objects
### Changed
- Sample child classes are now defined explicitly in the pipeline interface. Previously, they were guessed based on presence of a class extending Sample in a pipeline script.
- Changed 'library' key sample attribute to 'protocol'

## [0.6] - 2017-07-21
### Added
- Add support for implied_column section of the project config file
- Add support for Python 3
- Merges pipeline interface and protocol mappings. This means we now allow direct pointers to `pipeline_interface.yaml` files, increasing flexibility, so this relaxes the specified folder structure that was previously used for `pipelines_dir` (with `config` subfolder).
- Allow URLs as paths to sample sheets.
- Allow tsv format for sample sheets.
- Checks that the path to a pipeline actually exists before writing the submission script.

### Changed
- Changed LOOPERENV environment variable to PEPENV, generalizing it to generic models
- Changed name of `pipelines_dir` to `pipeline_interfaces` (but maintained backwards compatibility for now).
- Changed name of `run` column to `toggle`, since `run` can also refer to a sequencing run.
- Relaxes many constraints (like resources sections, pipelines_dir columns), making project configuration files useful outside looper. This moves us closer to dividing models from looper, and improves flexibility.
- Various small bug fixes and dev improvements.
- Require `setuptools` for installation, and `pandas 0.20.2`. If `numexpr` is installed, version `2.6.2` is required.
- Allows tilde in `pipeline_interfaces`

## [0.5] - 2017-03-01
### Added
- Add new looper version tracking, with `--version` and `-V` options and printing version at runtime
- Add support for asterisks in file paths
- Add support for multiple pipeline directories in priority order
- Revamp of messages make more intuitive output
- Colorize output
- Complete rehaul of logging and test infrastructure, using logging and pytest packages

### Changed
- Removes pipelines_dir requirement for models, making it useful outside looper
- Small bug fixes related to `all_input_files` and `required_input_files` attributes
- More robust installation and more explicit requirement of Python 2.7


## [0.4] - 2017-01-12
### Added
- New command-line interface (CLI) based on sub-commands
- New subcommand (`looper summarize`) replacing the `summarizePipelineStats.R` script
- New subcommand (`looper check`) replacing the `flagCheck.sh` script
- New command (`looper destroy`) to remove all output of a project
- New command (`looper clean`) to remove intermediate files of a project flagged for deletion
- Support for portable and pipeline-independent allocation of computing resources with Looperenv.

### Changed
- Removed requirement to have `pipelines` repository installed in order to extend base Sample objects
- Maintenance of sample attributes as provided by user by means of reading them in as strings (to be improved further)
- Improved serialization of Sample objects
4 changes: 2 additions & 2 deletions docs/config-files.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ or a **pipeline *developer*** (building your own pipeline).

Users (non-developers) of pipelines only need to be aware of one or two config files:

- The [project config](project-config.md): This file is specific to each project and
- The [project config](define-your-project): This file is specific to each project and
contains information about the project's metadata, where the processed files should be saved,
and other variables that allow to configure the pipelines specifically for this project.
It follows the standard `looper` format (now referred to as `PEP`, or "*portable encapsulated project*" format).
Expand Down Expand Up @@ -48,4 +48,4 @@ it uses a pipeline-specific configuration file, which is detailed in the [`pypip
Essentially, each pipeline may provide a configuration file describing where software is,
and parameters to use for tasks within the pipeline. This configuration file is by default named like pipeline name,
with a `.yaml` extension instead of `.py`. For example, by default `rna_seq.py` looks for an accompanying `rna_seq.yaml` file.
These files can be changed on a per-project level using the `pipeline_config` section of a [project configuration file](project-config.md).
These files can be changed on a per-project level using the `pipeline_config` section of a [project configuration file](define-your-project).
Loading

0 comments on commit ddf3097

Please sign in to comment.