Use dia_pipe with a given template and calexps from other nights

[BenjaminRacine]

Hi all,

I am trying to use dia_pipe on HSC data.
My issue is that in the way I ran my pipeline up to now, I have an output directory per night, for instance one for HSC-Z night 00817, and one for HSC-R night 01111, where the outputs of jointcal and coaddDriver are.
If I want to run imageDifferenceDriver , it works if I use the directory of coadded night as input, and chose the visits from that same night:

imageDifferenceDriver.py $Template_night --output $output_imdiff --id visit=$VisitZ_from_template tract=9812^9813 patch=0,0^0,1^0,2^0,3^0,4^0,5^0,6^0,7^0,8^1,0^1,1^1,2^1,3^1,4^1,5^1,6^1,7^1,8^2,0^2,1^2,2^2,3^2,4^2,5^2,6^2,7^2,8^3,0^3,1^3,2^3,3^3,4^3,5^3,6^3,7^3,8^4,0^4,1^4,2^4,3^4,4^4,5^4,6^4,7^4,8^5,0^5,1^5,2^5,3^5,4^5,5^5,6^5,7^5,8^6,0^6,1^6,2^6,3^6,4^6,5^6,6^6,7^6,8^7,0^7,1^7,2^7,3^7,4^7,5^7,6^7,7^7,8^8,0^8,1^8,2^8,3^8,4^8,5^8,6^8,7^8,8 --cores 8 --job imageDifferenceDriver --time 100 --longlog --clobber-config --config imageDifference.doSelectSources=True imageDifference.kernelSourcesFromRef=True

But if I want to do something more interesting, where I use a coadd from a night and do DIA on the visits of another night, then it fails.:

imageDifferenceDriver.py $Template_night --output $output_imdiff --id visit=$VisitZ_from_anothernight tract=9812^9813 patch=0,0^0,1^0,2^0,3^0,4^0,5^0,6^0,7^0,8^1,0^1,1^1,2^1,3^1,4^1,5^1,6^1,7^1,8^2,0^2,1^2,2^2,3^2,4^2,5^2,6^2,7^2,8^3,0^3,1^3,2^3,3^3,4^3,5^3,6^3,7^3,8^4,0^4,1^4,2^4,3^4,4^4,5^4,6^4,7^4,8^5,0^5,1^5,2^5,3^5,4^5,5^5,6^5,7^5,8^6,0^6,1^6,2^6,3^6,4^6,5^6,6^6,7^6,8^7,0^7,1^7,2^7,3^7,4^7,5^7,6^7,7^7,8^8,0^8,1^8,2^8,3^8,4^8,5^8,6^8,7^8,8 --cores 8 --job imageDifferenceDriver --time 100 --longlog --clobber-config --config imageDifference.doSelectSources=True imageDifference.kernelSourcesFromRef=True

It doesn’t crash but prints Failure warnings like:

178061 WARN 2020-08-03T12:37:30.942+0200 imageDifferenceDriver ({'visit': 1202, 'tract': '9812', 'patch': '0,0', 'pointing': 817, 'filter': 'HSC-R', 'ccd': 23, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2014-03-28', 'taiObs': '2014-03-28', 'expTime': 360.0})(imageDifferenceDriver.py:52)- Failed imageDifferenceDriver: No coadd PhotoCalib found!

and no output is created.
It makes sense, but I was wondering if there is a way to provide a path for the template that is different from the individual visits path, as one does for ap_pipe with the --template option.
Alternatively, I could rerun the pipeline and use a single path for all outputs, but even then I am not sure how to let imageDifferenceDriver know that I want to use a template from a night and visits from another night.

Is there a plan to allow these kinds of dia analyses?

Thanks,

Ben

[rearmstr]

Hi Ben,

As others commented on slack, what you want to do is not exactly supported in dia_pipe. You can get around some of this by chaining reruns together and making sure the parent repos are setup correctly or modifying the repositoryCfg.yaml file.

For DESC, our mode of operation is to process all the exposures through processCcd in a single repo. You can then create a new template repo (with the single visit repo as the parent) based on a subset of the exposures. Future processing with the template repo then will then have access to all the available visits when running difference imaging.

Given that the gen3 butler is looming on the horizon, I wasn't planning on adding much more functionality. If this solution doesn't work for you I can look at what is done in ap_pipe to add another directory to search for templates.

-Bob

[BenjaminRacine]

Hi Bob,

Thanks for the answer.
I tried a little to modify the repositoryCfg.yaml, but couldn't really figure out how to fix the issue.
I instead reran the step after

singleFrameDriver.py  DATA --rerun ProcessCcdOutputs  --id visit=$AllVisits --longlog --cores 56 --timeout 9999999

using rerun instead of output:

jointcal.py DATA --id visit=$VisitZ_01111 --rerun ProcessCcdOutputs:JC_Z_01111 --longlog 

And then used chaining reruns again:

coaddDriver.py  DATA --rerun JC_Z_01111:coadd_Z_01111 --cores 30  --id tract=9812^9813 patch=0,0^0,1^0,2^0,3^0,4^0,5^0,6^0,7^0,8^1,0^1,1^1,2^1,3^1,4^1,5^1,6^1,7^1,8^2,0^2,1^2,2^2,3^2,4^2,5^2,6^2,7^2,8^3,0^3,1^3,2^3,3^3,4^3,5^3,6^3,7^3,8^4,0^4,1^4,2^4,3^4,4^4,5^4,6^4,7^4,8^5,0^5,1^5,2^5,3^5,4^5,5^5,6^5,7^5,8^6,0^6,1^6,2^6,3^6,4^6,5^6,6^6,7^6,8^7,0^7,1^7,2^7,3^7,4^7,5^7,6^7,7^7,8^8,0^8,1^8,2^8,3^8,4^8,5^8,6^8,7^8,8 filter=HSC-Z --selectId visit=$VisitZ_01111 --longlog --timeout 9999999 --config makeCoaddTempExp.doApplySkyCorr=False 

Now when calling:

imageDifferenceDriver.py  DATA --rerun coadd_Z_01111:dia_onePatch_Z_00817_visits --id visit=$VisitZ_00817 tract=9812 patch=0,3 --cores 10 --job imageDifferenceDriver_rerun_onepatch --time 100 --longlog --clobber-config --config imageDifference.doSelectSources=True imageDifference.kernelSourcesFromRef=True

It worked

I am now stuck at the association step, which for now doesn't output anything, but I am still working on it.
I am also going to open a community ticket to ask if there is a natural way to use the improved WCS and photocalib from jointcal for the single visits in the difference imaging, instead of the processCcd ones.

Ben

Note that I had to download obs_subaru with git clone -b w.2020.23 https://github.com/lsst/obs_subaru.git
cd obs_subaru
setup -j -r
scons
and then edit policy/HscMapper.yaml, as you suggested to Manal.

[wmwv]

I am also going to open a community ticket to ask if there is a natural way to use the improved WCS and photocalib from jointcal for the single visits in the difference imaging, instead of the processCcd ones.

Do you have a particular reason to think that the existing WCS is problematic?

Particularly if you're using Alard+Lupton subtraction, I wouldn't expect that little tweaks to the WCS or flux calibration are important.

[RobertLuptonTheGood]

And if the Wcs is a problem this should be pushed upstream; it's not the job of the diffim code (although it can be tweaked with a suitable choice of basis function --- not something I'm recommending as the same information could have been used to improve the astrometry)

[BenjaminRacine]

Do you have a particular reason to think that the existing WCS is problematic?

Not really. For now at least, using ap_pipe, a lot of the dia sources etc seemed to be dipoles or rings, which sounded to me like a poor astrometry/photometry (and/or some failure of the psf_matching).
I assumed that using a better astro/photo for the single visits would help but to be fair, I don't have any experience.

Particularly if you're using Alard+Lupton subtraction, I wouldn't expect that little tweaks to the WCS or flux calibration are important.

I was using the default for ap_pipe, which I think as AL:
config.differencer.subtract.name='al'
config.differencer.subtract['al'].kernel.name='AL'
It is the same for dia_pipe.
Are you saying that Alard+Lupton shouldn't benefit from jointcal? Even in the sense of having a better template?
Why is that?

[BenjaminRacine]

And if the Wcs is a problem this should be pushed upstream

What do you mean by "pushed upstream"?

[wmwv]

Not really. For now at least, using ap_pipe, a lot of the dia sources etc seemed to be dipoles or rings, which sounded to me like a poor astrometry/photometry

The A+L kernels have some allowance for imperfectly matched images. Yes, it's better if the WCS is right, but unless you can visually see that the alignment is off it shouldn't matter. I.e., load the template and science image in ds9, align by WCS and blink back and forth and several different zoom levels. If that looks fine, then any slight imperfections of the WCS are completely within the ability of A+L to correct.

(and/or some failure of the psf_matching).

Indeed. Significant ringing is much more likely a sign that the image subtraction is attempting deconvolution. I.e., the image being convolved (by default the template) has in fact worse seeing than the image it is being convolved to.

[wmwv]

And if the Wcs is a problem this should be pushed upstream
What do you mean by "pushed upstream"?

He meant that the WCS is bad enough to be a problem, that should be fixed in processCcd.

[RobertLuptonTheGood]

For general-enough AL basis functions it can sort out the astrometry, but I hope we won't come to this. The information that the algorithm uses can be used in the astrometric solver, and should be.
This doesn't help you for now, but let's not make the mistake of using difference imaging algorithms as black magic.