[Blacklisted] Concern about SYE and UNW rawacf files on mirror converted from dat format

[mts299]

Hello all,

I just noticed that for various times between 2007 - 2016, some rawacf files from SYE and UNW available on the (BAS) data mirror appear to have been produced from a dattorawacf command rather than directly at the radar site.

This is a problem because in 2018, we identified an issue with dattorawacf where ranges with a lag0 power below some threshold (by default 3 dB) were automatically being dropped during the conversion. Because of this thresholding, it was impossible to reproduce the original dat file with a circular conversion of dattorawacf -> rawacftodat. For more details, see this page:

SuperDARN/rst#142

Does anyone know if the original dat files from these radars are still available so we can run the dattorawacf conversion with the updated software?

Cheers,
Evan

[mts299]

2019-07-06

Hello Evan and DDWG,

If I remember/understand correctly,...
SYE still runs Radops 2000 on QNX4 (on probably about 20 year-old PC;-)
and use raw_write command to produce DAT files with '-t 0' option added
meaning thresholding lag0 power at 0dB (not at default 3dB).
(this "-t 0"-like option was applied (by me) probably since SYE started to
use Radops2000 and probably even before it since its operation in 1997
with older Radops486?.)

After some discussion with Kevin at VT in some(many?) years ago,
we started to convert SYE dat files to rawacf with dattorawacf command
her at NIPR before providing them to VT and/or BAS for distribution.
(One reason we moved so was that multiple rawacf files everytime we converted
were all different in binary level though the physical data in them were the same.)
We stores all the original dat and converted rawacf files here at NIPR anyway
(although the 1999 SH dat files issue exist - I found them on distributed CDs!).

The dattorawacf program we use for the conversion is (a slightly modified
version of ) dattorawacf in RST 2.11_m64, i.e., that is based on
dattorawacf ver.1.06 in RST2.11_m64
oldraw.lib ver.1.11 in RST2.11_m64
raw.lib ver.1.14 in RST2.11_m64
which tried to completely disable the lag0 power thresholding when conversion
because I thought that all the data should be transparently converted to a new format.

I don't remember whether we (I and our software technical staff) confirm the
reproducibility of original dat files with the method above- which I can ask .her to check.)

Probably we here should move to the newest RST 4.x? anyway?
Is the dattorawacf program in newest RST very different from one in older RSTs?

Evan, do you have any concrete examples of distributed SYE rawacf files
which do not include any rawacf record below 3dB (systematically for some period) ?

Sorry that I couldn't follow most of your discussion these years,
so I might not correctly understand the situation in the right way...
Please point such issues to me if any, thanks.

Cheers,
Sessai

[mts299]

July 6, 2019

Hi Sessai,

Thank you for the very detailed response. After sending my message yesterday, I did notice that the SYE files had a "thr" value of 0 in the parameter block, indicating that you had already disabled the low power thresholding as you explained. So there should not be any issue with the SYE files. The only change we've made in the newer version of dattorawacf / rawacftodat was to entirely disable the thresholding functionality so that all records are perfectly converted by default, and an error message is printed if the user tries setting the "-t" option.

For UNW, I've just selected another random rawacf file (20140207.1000.00.unw.rawacf) and the "thr" parameter is listed as 3 in the parameter block and the slist/acfd/xcfd arrays on channel A are all less than nrang (70). From the origin command, it looks like this particular file was generated with a dattorawacf command at VT - is that correct Kevin? I can produce a fitacf file from this rawacf with make_fit, but when I try converting back to a dat file with rawacftodat I get a segmentation fault. The same problem does not happen with the dat file I pulled directly off VT's server (dat -> rawacf -> dat).

P.S. If SYE is still running Radops2000 and you have any backups of the code or copies of the original installation script(s) they would be very useful for our software archive (https://github.com/SuperDARN/ros-archive/branches/all)

Cheers,
Evan

[mts299]

2019-07-06

Hello Evan,
I might need to add some more information to you.
I just try to check dat/rawacf files stored on our local storage for Syowa original data.
I noticed that we have (locally) converted sye rawacf files only after 2016
(but there are 2015 sye rawacf files on the mirror, I imagine.)
So sye rawacf files at mirror sites might be converted
not here locally before providing to VT(or BAS) before 2016
but they might have been converted at VT? after providing original dat files to VT?
If the files (up to 2015) contains only above 3dB or so,
they might need to be converted again without any thresholding?

About old radops codes,
I probably have all the source codes for Radops2000
(and probably have Radops386 source codes as well

• at least I found some parts of them here
• I'll try to check them when I have some time...)

Sessai

[mts299]

July 8, 2019
Hi Sessai (and DD-WG members),

I just checked the BAS mirror, and there do not appear to be any SYE or SYS data from 2016-present. We do have some 2016 SYE and SYS rawacf data at Dartmouth, so those files must have been available on the BAS mirror at some point.

Cheers,
Evan

[mts299]

2019-07-08
Hi, Evan,
it is a different issue which is basically my fault!
I just recently tried to contact with Paul (and Kevin)
to recover the uploading them to BAS (and/or VT).
All the symlinks for the files after 2016 are on our
server here and I hope it will be done soon, though
there are some big data gaps and degrade of the
data quality etc due to interference issue with other
instrument at Syowa that happened in early 2016.
Sessai

[mts299]

2019-07-10

Hi Sessai, Evan,

This is a timely conversation!

That's correct, we don't currently have any Syowa data from 2016-present on the BAS mirror. However, I am just about to start syncing the backlog from the NIPR staging server, as Sessai has kindly provided me with the connection details.

If I've followed this thread correctly, the sye and sys converted rawacf files don't have the low power thresholding issue, and so are OK to place on the BAS mirror. Is this correct?

Cheers,
Paul.

[mts299]

2019-07-10

Hi Paul,
Thanks for retrying to restart syncing syowa data.
About the low power thresholding issue, at least sye rawacf after 2016 inclusive
should be ok (they are convert from sye dat files without any thresholding at NIPR)
(but distributed sye rawacf before it were converted probably at VT and
it is not confirmed whether the issue exists or not. - probably ok???)
At sys, rrawacf are obtained at radar site with ROS, so should be no problem.
Sessai

[mts299]

2019-08-05
Hi all,

I just wanted to follow up on this issue to check on its status. It sounds like Sessai has been trying to get the SYE and SYS files back on the BAS mirror, but I'm not sure if there has been any action on the UNW files. Can anyone from La Trobe speak to the dat -> rawacf conversion process for those files?

Thank you,
Evan

[mts299]

2019-08-05
Hey Evan,

As a bit of an update on my end, I've started downloading some of the
sye rawacf files that Sessai's producing. I'm not sure I can commit to
fully catching up due to running lower on data storage.

For the dat files, I can say that ones from possibly 2012 to present
were likely converted to rawacf files here at VT for both sye and unw.
The good news is I've still got the dat files here, so they can be
reprocessed/reconverted.

For the Unwin files, I've spot checked and can see a good number of dat
files for that radar dating back to the beginning in 2004. So, it's
possible I can download those dat files and reconvert them into rawacfs
after the 20060701 transition. Then either forward those onto Univ. of
Saskatchewan or doing this in parallel with BAS (which then Univ. of
Saskatchewan could/might pick up on). Would this make sense?

The dat files should be smaller, though I'll again have to watch my data
storage here.

Evan, have you checked other radars that were producing dat files right
up to the 20060701 flip and when the rawacf files stopped being
converted from dat files? My wonder is if the same thresholding problem
was in place then, then should we be concerned on updating those rawacf
files as well.

Cheers,

-Kevin S.

[mts299]

2020-02-07
Hi Kevin,

I was just searching back through some old emails and realized I never responded to you on this thread. Can you recall the status of this issue? Were the UNW dat files ever reprocessed with the low-power thresholding disabled?

Thanks,
Evan