Duplicated timestamps in csv change height distribution in vpts plot

[bart1]

While exploring some data I noticed the height distribution of some vpts plots looking strange. Here somehow the data get plotted 2 or 3 times as high. It might related to the height datetime combination being replicated in the source file. The lower height distribution seems correct see also (https://crow.aloftdata.eu#/?radar=fikor&date=2024-10-12&interval=72&timedisplay=radarLocal&vpiMode=mtr&vpColorScheme=viridis&lang=en):

require(bioRad)
#> Loading required package: bioRad
#> Welcome to bioRad version 0.8.1
#> using vol2birdR version 1.0.5 (MistNet not installed)
require(lubridate)
#> Loading required package: lubridate
#> 
#> Attaching package: 'lubridate'
#> The following objects are masked from 'package:base':
#> 
#>     date, intersect, setdiff, union
t<-as.Date("2024-10-12")+lubridate::hours(15)+lubridate::minutes(0:(12*8)*5)
int<-as.interval(min(t), max(t))
# r<-bioRad::download_vpfiles(as.character(min(as.Date(t))),
#                             as.character(max(as.Date(t))), "fikor", directory = "/home/bart/")
vptsAloft<-read_vpts("/home/bart/fikor_vpts_202410.csv.gz")
vptsAloft<-vptsAloft[vptsAloft$datetime %within% int]
plot(vptsAloft)
#> Warning in plot.vpts(vptsAloft): Irregular time-series: missing profiles will
#> not be visible. Use 'regularize_vpts' to make time series regular.
#> Warning in plot.vpts(vptsAloft): Dropped 66 profiles with duplicate datetime
#> values

vptsAloft<-vptsAloft[!duplicated(vptsAloft$datetime)]
plot(vptsAloft, main="Dups removed")

require(dplyr)
#> Loading required package: dplyr
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
vroom::vroom('/home/bart/fikor_vpts_202410.csv.gz')  |>
  filter(datetime %within%int) |>
  select(dens, height, datetime) |> group_by(datetime, height) |>
  summarise(dens=tail(head(dens,1),1))|> ungroup()|>
  tidyr::pivot_wider(names_from = "height", values_from = "dens") |>
  select(-datetime) |> as.matrix() |> image()
#> Rows: 200125 Columns: 26
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr   (2): radar, source_file
#> dbl  (21): height, u, v, w, ff, dd, sd_vvp, eta, dens, dbz, dbz_all, n, n_db...
#> lgl   (2): gap, vcp
#> dttm  (1): datetime
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> `summarise()` has grouped output by 'datetime'. You can override using the `.groups` argument.

vroom::vroom('/home/bart/fikor_vpts_202410.csv.gz')  |>
  filter(datetime %within%int) |> group_by(height)|>
  filter(!duplicated(datetime)) |> as.data.frame()|> bioRad:::as.vpts() |> plot()
#> Rows: 200125 Columns: 26
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr   (2): radar, source_file
#> dbl  (21): height, u, v, w, ff, dd, sd_vvp, eta, dens, dbz, dbz_all, n, n_db...
#> lgl   (2): gap, vcp
#> dttm  (1): datetime
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

^{Created on 2025-02-25 with reprex v2.1.1}

[bart1]

Here is what is most likely the correct plot:

require(bioRad)
#> Loading required package: bioRad
#> Welcome to bioRad version 0.8.1
#> using vol2birdR version 1.0.5 (MistNet not installed)
require(lubridate)
#> Loading required package: lubridate
#> 
#> Attaching package: 'lubridate'
#> The following objects are masked from 'package:base':
#> 
#>     date, intersect, setdiff, union
require(dplyr)
#> Loading required package: dplyr
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
t<-as.Date("2024-10-12")+lubridate::hours(15)+lubridate::minutes(0:(12*8)*5)
int<-as.interval(min(t), max(t))

vroom::vroom('/home/bart/fikor_vpts_202410.csv.gz')  |>
  mutate(datetime=strptime(source_file, "fikor_vp_%Y%m%dT%H%M%SZ_0xb.h5", "UTC"))|>
  filter(datetime %within%int) |>  arrange(datetime, height)|>
  bioRad:::as.vpts() |> plot()
#> Rows: 200125 Columns: 26
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr   (2): radar, source_file
#> dbl  (21): height, u, v, w, ff, dd, sd_vvp, eta, dens, dbz, dbz_all, n, n_db...
#> lgl   (2): gap, vcp
#> dttm  (1): datetime
#> 
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

^{Created on 2025-02-25 with reprex v2.1.1}