v0.71.1

v0.71.1 | 15 Mar 2024

Bugfixes & changes

• Fixed: Replaced all references to old filetypes using the underscore to their respective new filetype names,
  e.g. all occurrences of EDDYPRO_FLUXNET_30MIN were replaced with the new name EDDYPRO-FLUXNET-30MIN.
• Environment: Python 3.11 is now allowed in pyproject.toml: python = ">=3.9,<3.12"
• Environment: Removed fitter library from dependencies, was not used.
• Docs: Testing documentation generation using Sphinx, although it looks very
  rough at the moment.

What's Changed

• Update pyproject.toml for compatibility with python 3.11 by @inkenbrandt in #58
• V0.71.1 by @holukas in #69

New Contributors

• @inkenbrandt made their first contribution in #58

Full Changelog: v0.71.0...v0.71.1

v0.71.0 | High-resolution update

v0.71.0 by @holukas in #66

v0.71.0 | 14 Mar 2024

High-resolution update

This update focuses on the implementation of several classes that work with high-resolution (20 Hz) data.

The main motivation behind these implementations is the upcoming new version of another script, dyco, which will make direct use of these new classes. dyco allows to detect and remove time lags from time series data and can also handle drifting lags, i.e., lags that are not constant over time. This is especially useful for eddy covariance data, where the detection of accurate time lags is of high importance for the calculation of ecosystem fluxes.

*Plot showing the covariance between the turbulent departures of vertical wind and CO2 measurements. Maximum (absolute) covariance was found at record -26, which means that the CO2 signal has to be shifted by 26 records in relation to the wind data to obtain the maximum covariance between the two variables. Since the covariance was calculated on 20 Hz data, this corresponds to a time lag of 1.3 seconds between CO2 and wind (20 Hz = measurement every 0.05 seconds, 26 * 0.05 = 1.3), or, to put it another way, the CO2 signal arrived 1.3 seconds later at the sensor than the wind signal. Maximum covariance was calculated using the `MaxCovariance` class.*

New features

• Added new class MaxCovariance to find the maximum covariance between two variables (diive.pkgs.echires.lag.MaxCovariance)
• Added new class FileDetector to detect expected and unexpected files from a list of files (diive.core.io.filesdetector.FileDetector)
• Added new class FileSplitter to split file into multiple smaller parts and export them as multiple CSV files. (diive.core.io.filesplitter.FileSplitter)
• Added new class FileSplitterMulti to split multiple files into multiple smaller parts and save them as CSV or compressed CSV files. (diive.core.io.filesplitter.FileSplitterMulti)
• Added new function create_timestamp that calculates the timestamp for each record in a dataframe, based on number of records in the file and the file duration. (diive.core.times.times.create_timestamp)

Additions

• Added new filetype ETH-SONICREAD-BICO-CSVGZ-20HZ, these files contain data that were originally logged
  by the sonicread script which is in use in the ETH Grassland Sciences group since the early
  2000s to record eddy covariance data within the Swiss FluxNet. Data were
  then converted to a regular format using the Python script bico, which
  also compressed the resulting CSV files to gz files (gzipped).
• Added new filetype GENERIC-CSV-HEADER-1ROW-TS-MIDDLE-FULL-NS-30MIN, which corresponds to a CSV file with
  one header row with variable names, a timestamp that describes the middle of the averaging period, whereby
  the timestamp also includes nanoseconds. Time resolution of the file is 30MIN.

Changes

• Renamed class TurbFlux to WindRotation2D and updated code a bit, e.g., now it is possible to get
  rotated values for all three wind components (u', v', w') in addition to the rotated
  scalar c'. (diive.pkgs.echires.windrotation.WindRotation2D)
• Renamed filetypes: all filetypes now use the dash instead of an underscore
• Renamed filetype to ETH-RECORD-DAT-20HZ: this filetype originates from the new eddy covariance real-time
  logging script rECord (currently not open source)
• Missing values are now defined for all files
  as: NA_VALUES: [ -9999, -6999, -999, "nan", "NaN", "NAN", "NA", "inf", "-inf", "-" ]

v0.70.1

• Updated (and cleaned) notebook StepwiseMeteoScreeningFromDatabase.ipynb

v0.70.0

v0.70.0 | 28 Feb 2024

New features

• In StepwiseOutlierDetection, it is now possible to re-run an outlier detection method. The re-run(s)
  would produce flag(s) with the same name(s) as for the first (original) run. Therefore, an integer is added
  to the flag name. For example, if the test z-score daytime/nighttime is run the first time, it produces the
  flag with the name FLAG_TA_T1_2_1_OUTLIER_ZSCOREDTNT_TEST. When the test is run again (e.g. with different
  settings) then the name of the flag of this second run is FLAG_TA_T1_2_1_OUTLIER_ZSCOREDTNT_2_TEST,
  etc ... The script now checks whether a flag of the same name was already created, in which case an
  integer is added to the flag name. These re-runs are now available in addition to the repeat=True keyword.
  (diive.pkgs.outlierdetection.stepwiseoutlierdetection.StepwiseOutlierDetection.addflag)
  Example:
  • METHOD with SETTINGS is applied with repeat=True and therefore repeated until no more outliers
    were found with these settings. The name of the flag produced is TEST_METHOD_FLAG.
  • Next, METHOD is applied again with repeat=True, but this time with different SETTINGS. Like before,
    the test is repeated until no more outliers were found with the new settings. The name of the flag produced
    is TEST_METHOD_2_FLAG.
  • METHOD can be re-run any number of times, each time producing a new
    flag: TEST_METHOD_3_FLAG, TEST_METHOD_4_FLAG, ...
• Added new function to format timestamps to FLUXNET ISO
  format (YYYYMMDDhhmm) (diive.core.times.times.format_timestamp_to_fluxnet_format)

Bugfixes

• Refactored and fixed class to reformat data for FLUXNET
  upload (diive.pkgs.formats.fluxnet.FormatEddyProFluxnetFileForUpload)
• Fixed None error when reading data files (diive.core.io.filereader.DataFileReader._parse_file)

Notebooks

• Updated notebook FormatEddyProFluxnetFileForUpload.ipynb

v0.69.0

What's Changed

• Extract binary info by @holukas in #48

Full Changelog: v0.68.1...v0.69.0

v0.68.1

What's Changed

• fixed bugs in flux processing chain by @holukas in #47

Full Changelog: v0.68.0...v0.68.1

v0.68.0

v0.68.0

v0.67.1

Updated docstring

v0.67.0 - Flux processing chain updates

v0.67.0

v0.66.0: ScatterXY plot

What's Changed

• Indev by @holukas in #36
• Remove sphinx autodocs for now by @holukas in #37
• Add scatter plot by @holukas in #41

Full Changelog: v0.64.0...v0.66.0