Skip to content

v0.81.0
Compare
Choose a tag to compare
@holukas holukas released this 11 Sep 12:49
· 35 commits to main since this release
v0.81.0
0a9f1d7
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

v0.81.0 | 11 Sep 2024

Expanding Flux Processing Capabilities

This update brings advancements for post-processing eddy covariance data in the context of the FluxProcessingChain.
The goal is to offer a complete chain for post-processing ecosystem flux data, specifically designed to work seamlessly
with the standardized _fluxnet output file from the
widely-used EddyPro software.

Now, diive offers the option for USTAR filtering based on known constant thresholds across the entire dataset (similar
to the CUT scenarios in FLUXNET data). While seasonal (DJF, MAM, JJA, SON) thresholds are calculated internally,
applying them on a seasonal basis or using variable thresholds per year (like FLUXNET's VUT scenarios) isn't yet
implemented.

With this update, the FluxProcessingChain class can handle various data processing steps:

  • Level-2: Quality flag expansion
  • Level-3.1: Storage correction
  • Level-3.2: Outlier removal
  • Level-3.3: (new) USTAR filtering (with constant thresholds for now)
  • (upcoming) Level-4.1: long-term gap-filling using random forest and XGBoost
  • For info about the different flux levels
    see Swiss FluxNet flux processing chain

New features

  • Added class to apply multiple known constant USTAR (friction velocity) thresholds, creating flags that indicate time
    periods characterized by low turbulence for multiple USTAR scenarios. The constant thresholds must be known
    beforehand, e.g., from an earlier USTAR detection run, or from results from FLUXNET (
    diive.pkgs.flux.ustarthreshold.FlagMultipleConstantUstarThresholds)
  • Added class to apply one single known constant USTAR thresholds (
    diive.pkgs.flux.ustarthreshold.FlagSingleConstantUstarThreshold)
  • Added FlagMultipleConstantUstarThresholds to the flux processing chain (
    diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain.level33_constant_ustar)
  • Added USTAR detection algorithm based on Papale et al., 2006 (diive.pkgs.flux.ustarthreshold.UstarDetectionMPT)
  • Added function to analyze high-quality ecosystem fluxes that helps in understanding the range of highest-quality data(
    diive.pkgs.flux.hqflux.analyze_highest_quality_flux)

Additions

  • LocalSD outlier detection can now use a constant SD:
    • Added parameter to use standard deviation across all data (constant) instead of the rolling SD to calculate the
      upper and lower limits that define outliers in the median rolling window (
      diive.pkgs.outlierdetection.localsd.LocalSD)
    • Added to step-wise outlier detection (
      diive.pkgs.outlierdetection.stepwiseoutlierdetection.StepwiseOutlierDetection.flag_outliers_localsd_test)
    • Added to meteoscreening from database (
      diive.pkgs.qaqc.meteoscreening.StepwiseMeteoScreeningDb.flag_outliers_localsd_test)
    • Added to flux processing chain (
      diive.pkgs.fluxprocessingchain.fluxprocessingchain.FluxProcessingChain.level32_flag_outliers_localsd_test)

Changes

  • Replaced .plot_date() from the Matplotlib library with .plot() due to deprecation

Notebooks

  • Added notebook for plotting cumulative sums per year (notebooks/Plotting/CumulativesPerYear.ipynb)
  • Added notebook for removing outliers based on the z-score in rolling time window (
    notebooks/OutlierDetection/zScoreRolling.ipynb)

Bugfixes

  • Fixed bug when saving a pandas Series to parquet (diive.core.io.files.save_parquet)
  • Fixed bug when plotting doy_mean_cumulative: no longer crashes when years defined in parameter
    excl_years_from_reference are not in dataset (diive.core.times.times.doy_mean_cumulative)
  • Fixed deprecation warning when plotting in bokeh (interactive plots)

Tests

  • Added unittest for LocalSD using constant SD (
    tests.test_outlierdetection.TestOutlierDetection.test_localsd_with_constantsd)
  • Added unittest for rolling z-score outlier removal (
    tests.test_outlierdetection.TestOutlierDetection.test_zscore_rolling)
  • Improved check if figure and axis were created in (tests.test_plots.TestPlots.test_histogram)
  • 39/39 unittests ran successfully

Environment

  • Added new package scikit-optimize
  • Added new package category_encoders

What's Changed

Full Changelog: v0.80.0...v0.81.0

Contributors

holukas
Assets 2
Loading