Missing Night Time Images

[ericofpendom]

I noticed this morning whilst reviewing the generated time lapse video that it was very jerky. When I checked the data I have found that there are several times through the night period where 5 or 6 minutes can go by before the next image is captured. I have processed the data through Excel and highlighted in yellow the times when this happens. I would be interested if anyone could shed some light. It doesn't happen through the daytime period. I have attached an Excel workbook with the data.
Thanks in advance.
Missing Images.xlsx

[stevedee]

I suspect your system is deleting images which are too dark.

I had the same problem, so I opened the Editor and lowered the low limit for dim images to 0.1 like this:-

# Remove corrupt or too dim/bright images.
REMOVE_BAD_IMAGES="true"
REMOVE_BAD_IMAGES_THRESHOLD_LOW=0.1
REMOVE_BAD_IMAGES_THRESHOLD_HIGH=90

For me, only the night-time images are important, but you may want to experiment with this value.

[ericofpendom]

Yes I currently have the debug level at 3 and that is how I got the data in the attached spreadsheet.

This is from the same file but a different period, but it shows a removeBadImages report.

2024-09-15T13:26:54.053730+01:00 26 raspberry allsky[94708]: saveImage.sh DAY /home/pi/allsky/tmp/image-20240915132650.jpg DATE=20240915 TIME=132650 AUTOEXPOSURE=1 sAUTOEXPOSURE=(auto) EXPOSURE_US=3982 sEXPOSURE=3.98 ms AUTOGAIN=0 sAUTOGAIN= GAIN=1 AUTOWB=1 sAUTOAWB=(auto) WBR=2.500000 WBB=2 BRIGHTNESS=0 MEAN=0.995055 FULLMEAN=0.973971 BIN=1 FLIP=none BIT_DEPTH=0 FOCUS=1 DARKFRAME=0 eOVERLAY=1
2024-09-15T13:26:55.799467+01:00 26 raspberry allsky[94720]: removeBadImages.sh: File is bad: removed 'image-20240915132650.jpg' (above threshold: MEAN=97.74, threshold = 90)

[ericofpendom]

I have done as you suggested and set the LOW to 0.1. I will see how that behaves and report back tomorrow either way.

[stevedee]

Just playing with my system, I reduced High Limit to 1 which of course results in an: "Multiple bad consecutive bad images." on the System Messages section of the WebUI
So I guess if you are not seeing that kind of message, your problem must lay somewhere else.

[ericofpendom]

I will see more tomorrow. Cheers.

[stevedee]

OK, but I don't think it will help.

The only other suggestion I have is that you disable auto-focus

From the documentation:-
Some RPi and RPi-compatible cameras support auto focus. That feature sounds really useful but those cameras have some limitations:
Auto focus at night takes a long time since the camera has to take several long-exposure images.

[EricClaeys]

@ericofpendom,
You got this message:

removeBadImages.sh: File is bad: removed 'image-20240915132650.jpg' (above threshold: MEAN=97.74, threshold = 90)

so there's no reason to change the THRESHOLD_LOW value.
A mean value of 97.74 is almost pure white, but if you want to see those images (maybe to help debug the problem), change the HIGH value to something like 99.

For the "missing images", would you please attach /var/log/allsky.log? Don't edit the file first. I need to see it exactly as it was created.
In the spreadsheet you have all the yellow lines at the end, but Sept 17 comes before Sept 16 and it most of the log entries for those images appear to have been deleted.

What Pi model do you have?

[ericofpendom]

Thank you again Eric for your help with this, it is much appreciated. I shall do as you suggest and report back.

[ericofpendom]

Hi Eric, just an update. I made a few changes yesterday, mainly I set the night max exposure to 30 seconds and I also set the max gain to 100 and turned on auto gain for night time. I have just had a look at the log file and done a bit of processing through excel and so far today there have been 16 jumps of 3 minutes compared to the same period yesterday, the 18th, there were 80 jumps, some as big as 12 minutes. I have done as you suggested and set manual exposure to 1 second. I will check the logs again tomorrow and update.

[EricClaeys]

Manual Exposure should be 1, which is 1 ms, not 1 second, which is 1000 ms.
Keep me informed please. Eric

[ericofpendom]

Doh!

[ericofpendom]

Hi Eric, just an update on the missing images issue. I am still getting missing images throughout the night time and yesterday I even swapped out the pi4 for an identical model and got the same results this morning. I am thinking now that it could be the sensor overheating due to the repetitive long exposures. Because the daytime images are short exposure I don't think the sensor is affected, hence no missing images through the day. I have spent a few hours trying to find some way of monitoring the sensors temperature but without success. I have started a discussion earlier asking if 5 users would be willing to let me have a copy of their allsky log files so that I can see if they are getting missing images that they haven't noticed. At present I am stumped. I am considering buying another camera so that I can build a second rig for comparison.
Thanks again for any help that you or anyone can give in this matter.

[EricClaeys]

@ericofpendom, if you can identify exactly when some missing images occur and attach the log file, I can compare. Make sure the Debug Level is 3.

[ericofpendom]

HI Eric, I have a copy of the allsky.log from this evening with level 3 debug. I have identified every instance of more than 2 minutes where there is a gap in the image process by using Excel. Would I be able to post a .xlsm (macro enabled) file on here?

[ericofpendom]

Hi Eric, I have actually managed to copy the relevant data over to a standard excel file.
It relates directly to the attached log file. All I have done is identify the minutes from the time stamp (mins column) and in the difference column is the number of minutes skipped. The rows are in pairs, so the first row is the minute of the last image and the second of the pair is the minute of the next image, and so on in pairs down the rows.
I hope that is clear.

allsky_time_jumps.xlsx
allsky.log

[ericofpendom]

Just an update on this issue. After checking and changing everything else, I finally gave in and changed the 2mtr ribbon cable to a 12" cable and the system performed perfectly. I still don't understand why no frames were dropped in the daytime but I suppose I will never know. Thanks to all that responded.

[EricClaeys]

Glad you got it working. Luckily ribbon cables are cheap.