Version 2.1.0

Finally, a new PyPI release in a rather long while. Thanks @Tarrasch and @erikbern, and all the contributors!

Added

luigi:

• Notifications: more emails and proper coloring (#1471), improved SMTP handling
• EnumParameter (#1479), DictParameter (#1574)
• Support for Python 3.5 (#1494)
• Process locking on Alpine Linux (#1530) and Windows (#1557)
• Visualizer: resources tab (#1566), GUI functionality to disable a worker (#1564)

luigi.contrib:

• ExternalBigqueryTask (#1434), BigqueryCreateViewTask (#1465)
• Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (#1601)
• ExternalProgramTask, ExternalPythonProgramTask - commonalities for running any external application or script (#1520)
• Support for SFTP (#1585)
• Sped-up Hive client using Metastore (#1533)
• OpenerTarget, a single Luigi target to open multiple file system types (#1555)
• Query base task (giving rise also to luigi.contrib.redshift.RedshiftQuery and luigi.postgres.PostgresQuery) (#1493)
• RedshiftUnloadTask (#1527)
• UploadToSalesforceTask (#1404)
• Support for S3 assumed role (#1596)

Changed

luigi:

• Semi-opaque, hashed task_id (as opposed to TaskName(param1=value1, param2=foo bar)) (#1444)
• More explicit way to handle timelike parameters (date vs datetime) (#1473)
• Optimizations in scheduler algorithm

Removed

luigi:

• Old deprecated (2014) stuff around scheduler and its state (#1592)

luigi.contrib:

• Deprecated classes SparkJob, Spark1xBackwardCompat, Spark1xJob, PySpark1xJob (#1442)

Fixed

luigi.contrib:

• Ensure that FTP RemoteTarget successfully creates temporary files (meaning, in a directory relative to output) (#1515)
• Remove superfluous init_mapper()/init_reducer() calls in LocalJobRunner (#1475)
• Humanly format HadoopJobError (#1528)
• Broken Redshift table creation (#1453)
• Improved Salesforce reliability (#1597, #1600)
• Missing call to post_copy() (#1502)

...and a slew of other additions, fixes, improvements and documentation.

Version 2.0.1

This version contains mostly bugfixes, in particular making Luigi 2.0 working on Windows

Thanks @daveFNbuck @econchick @Tarrasch @dlstadther @DeaconDesperado @jpiper @nresare @dhurlburtusa @stephenpascoe @mbruggmann and some more people I probably forgot!

Pull requests

• #1441 don't print nested stack traces
• #1439 spark: use list2command
• #1447 fix unix socket
• #1446 fix daemon
• #1448 support symlinks in hdfs
• #1431 #1433 worker as a context manager
• #1429 keyboardinterrupt fix
• #1428 make it work without tornado
• #1425 positional parameter qualification
• #1422 fixed broken task trigger
• #1411 mention misspelled task name
• #1418 bigquery fix
• #1417 visualizer fix
• #1413 kwargs for s3
• #1345 retry get_work
• #1315 limit graph size
• #1391 amazon sns support
• #1392 pyspark parameter fix
• #1395 tornado fix
• #1397 spark package fix
• #1400 serialize TaskParameter
• #1402 kwargs for boto s3
• #1388 disable email from cfg
• #1341 get_work fix
• #1340 stop subprocesses on exception
• #1352 fix bug in scheduler
• #1357 fix issues in requires during scheduling
• #1367 fix visualizer w weird task names
• #1372 s3target.path
• #1314 redshift updates
• #1370 gmail authentication for notifications
• #1366 pig fix
• #1331 path warning fix
• #1338 ssh forwarding agent
• #1337 ssh multiplexing
• #1363 windows signal fix
• #1298 tracking fix
• #1346 fix dep graph on firefox

2.0.0

Highlights:

• Improved documentation.
• New visualiser page! #1086, thanks @stephenpascoe
• Execution summaries at end of runs! #1091, thanks @nicolehedblom
• Smarter layout algorithm for the SVG dependency graphs, thanks @daveFNbuck
• Command line interface now has a useful --help, see #1276
• Luigi finally has semantic exit codes! #1264

Contrib:

• Stablization of GCP tools
• AWS EC2 support. #1317, thanks @jfeala
• Sungrid support. #1124, thanks @jfeala (again :))

Scheduling:

• UNIX sockets are now supported, hence also encryption. #1129, thanks @graingert
• Smarter scheduling! #1272
• Speedup in scheduler. #1046
• A --take-lock option to smoother roll out code changes. #1137

Deprecations:

• There have been a lot of deprecations of features regarding Parameters. Before upgrading, please consider taking a look at the new module documentation for parameters.

1.3.0

Notable changes:

• is_global finally removed! (It still exists to not break old code, but it only sets positional=False)
• Initial support for Google Cloud Storage and Google BigQuery. Thanks @mikekap! #999 #1002
• Massive luigid speedup for get_work() response times. #986
• Various bugfixes.
• Standardized FileSystem/Target interface. Again thanks @mikekap! #997
• SQLAlchemy works for multiple engines. Thanks @dylanjbarth! #991

v1.2.1

Same as 1.2.0, only that I didn't forget to update the setup.py file this time.

1.2.1

Same as v1.2.1. Only that we can't have a initial "v" as otherwise readthedocs won't build the documentation for us. See #970 (comment)

v1.2.0

Notable changes:

• Dropped Python 2.6 support. #934
• New experimental "assistant" feature.
• External tasks are now also registered. This might cause breakage in settings like described here
• Many configuration settings have been hoisted to the new configuration system. Particularly for hdfs-related parameters. The --help command will show what new options are available.
• While is_global is not removed yet. It'll get removed anytime soon. Please use the new positional=False setting as a replacement in your parameters. In fact, we suggest to always use positional=False.
• The central scheduler now supports searching within groups. #903
• New dependency graph rendering using D3. #920
• The built in map reduce framework can now use ujson instead of doing eval. It's still not clear to us how much faster it is. #939
• Various bug fixes and other features.

1.1.1

Features better Spark support: #837 #812 #806 and various bug fixes

Version 1.1.0

This mainly features Python 3 support (thanks mostly to @gpoulin!)

1.1.0 corresponds to 0abe27e and has been run internally at Spotify for a while so should be fairly robust.

I would have released latest master as 1.1.0 had it not been for the bugs around Tornado (#824). Hoping to launch an 1.1.1 soon.

setup.py has recorded 1.0.24 for a while now, which is not correct for all commits after 0abe27e. I bumped the version of setup.py to 1.1.1

1.0.24

.24, .23, and .22 were quick fixes of big issues that came up when trying to release .21