ESP like OTA + rollback able bootloader

[lanmarc77]

Hi everyone,
might it be possible to implement options to the bootloader that allow to "partition" the space of the program memory into either three or two parts so that the bootloader could implement an ESP like behaviour for multiple versions of one program to reside on the chip and switch between them?
With an external watchdog IC this could even implement auto revert in case of failures.
Thanks
Marcel

[SpenceKonde]

A scheme which limited the sketch to the lower half of memory is possible, copying the current code to the high half to back it up. And you could restore than to the primary half of memory. However what you can;t do is have both f them in a usable form at the same time. The absolute location of the application section start must be known at compiletime, which must be where the chip places the inteerrupt vectors vectors to be. This is at an address of 0x200 * BOOTSIZE, and the bootloader cannot modify it's own fuses.

You could use the CRC scan peripheral to implement the integrity checking if you marked the high half of the flash as "application data" (through th CODESIZE fuse) so it wouldn't get combined with the low half of the flash in the "app code" checksum so you don't even need an external IC to autorevert.

However, swapping the active with backup is more work (since you'd need to do it piecewise with a giant buffer in ram)

The copy operations are actually quite easy though they will take about as it does to upload a 64k sketch via SerialUPDI at 345600 baud, (at that speed you are right at the limit of how fast the flash can be written). Backup current sketch would just be setting the FLMPER32 mode, erase 4 0x10000, 0x14000, 0x18000, 0x1C000, then switch the mode to FLWR and loop from start of application section (probably 0x400 or 0x600) to 0xFFFE inclusive, incrementing by 2 (writing by words works, writing by byte is buggy). And reverting is basicall the inverse of that except thatyou can't use the 32-page erase on the first page because that has the bootloader section in it and the erase will jsut fail

[lanmarc77]

Hi
thank you for your answer, from which I am certain I only got half. I hope it is enough to reply and get my questions through.
When thinking about partitioning into three sections I thought of having the first third always contain the running application (your compile time argument). The other two (third) contain usually an old program and the one that is also running in the first third.
When uploading either via USART, RS485 or even LoRa FUOTA the new program is always placed in the old section. If the transfer was successfull the chip is restarted with a mark (e.g. in the EEPROM) to select the old section as containing a new program. The bootloader while running is then copying this section into the first third of the flash and then starts it.
All the different states including special cases like power outages while copying and so on could be atomically kept track of using eeprom status.
This "wastes" a lot of flash memory basically two third but might be ok for those who do not need it or want to use an external flash memory.
Are there any technical issue that you see with this scheme?
Marcel

[SpenceKonde]

One thing in favor of my 3 sections vs your 4 sections (3 = boot + active application + backup application, 4 = (boot + active, and two inactive applications sections) is that the hardware has explicit support for defining exactly three flash sections - "bootloader, "app code" and "app data" with only the bootloader able to write to app data, and code executing from either of the other two sections able to write to app data (Though both sections can be write protected by code in the bootloader, and that write protection is cleared only on reset.

Are you proposing to have the application code be writing the new sketch image to the "old" sections as opposed to the bootloader or what?
I wouldn't think that would be necessart for serial (or RS485 with minor modifications to the bootloader - actually extremely minor, since there's hardware support for controlling the TX enable pin) since optiboot can already do that part - it would need just need to be adapted to do the whole copy/backup/restore stuff. You wouldn;t need to put a mark in eeprom unless you had uploaded an imagethat passed CRC check but didnt function correctly.

[lanmarc77]

Hi,
yes indeed I had exactly the use case in mind you described. Transfer ok but logic of the program is not working as intended.
I understand the elegance of the 3 section version though.
And yes the basic programming flow I would imagen to be in the running application. In this way I can also choose the transmission protocol and do not need to have it optiboot compatible. Especially in LoRaWAN the packages are very small and need to be assembled before reaching a page size.
But I would imagine the write/flash process this to be bootloader "assisted". So the writing itself is done by the bootloader into the correct section as well as setting any marks might it be needed.
Once the new program works also this new program sets a persist marking so that after a reboot/power off no rollback happens. Again this would happen over the same communication channel the firmware came in and with the help of the bootloader. This would also allow the bootloader to transparently decide how it stores the states and where.
Marcel

[SpenceKonde]

There is no reason that you need to involve the bootloader in the writing by the application of a new version into one of the "slots" - see the Flash.h library assuming APP_SPM is set to something other than -1. You mark the end of the APP CODE section with the CODESIZE fuse, and after that, it's app data, and it can be written from the appcode section (but if code is actually executing from within APPDATA 0 but that;'s not possible under your situation anyway because code compiled to to start from 0x400 or whatever is going to crash and burn when it tries to run at 0x10400 or whatever it ends up being located at - it'll only work once transferred into the app section anyway) that can be done entirely from application space. When it comes time to actually "apply" the new firmware, meaning that it;s copied from one of those inactive slots into the appcode section, THAT would have to be done by the bootloader. either you could reboot into it, or - I'd probably leave an entry point in the bootloader at a known address (typically you use offsets of a few bytes before the end of the bootloader - for optiboot on Dx, there's an entrypoint 2 words before the optiboot version (which occupies the final word of BOOTCODE section) for a two-instruction routine to allow code from the application section to overwrite the application section piecewise (see Flash.h for APP_SPM = -1 - that is the bare minimum, SPM Z+, RET - to use it flash.h sets everything up, and just calls that address instead of executing SPM itself.). That';s not what you want here though - you need the bootloader to be able to do the whole erase+copy procedure - erase might just be a loop until the page number hits the end of the app section with NVMCMD set to FLPER,, and then the copy is even easier, set start address of each, and loop, reading two bytes and then writing them to the flash (there is no concept of pages as far as write is concerned only erase has pages!). and finally end it by triggering a softeware reset start with the new firmware.

I'm imaging an implementation of those that could be very simple (space efficient) assembly....

caled from

__asm__ __volatile__ (
"jmp (entrypoint address here)" "\n\t" 
:: "x" (uint16_t) &applicationsectionstart, "y" (uint16_t)&inactivesectionstart), r24 (uint16_t) length
);    // application and inactive sections must be aligned on page boundaries (ie, those two addresses modulo 512 = 0)
__builtin_unreachable(); // It will always software reset out; it also shits all over the working registers so it couldn't reenter application cleanly without more work. But telling the compiler that execution will never come out the other side of the asm can save a bit of flash. . 

Then down in bootloader section *\n\t"'s omitted for the sake f my hands

ldi r16, 0x9D  // SPM CCP signature
ldi r17, 0xD8  // IOREG CCP signature
movw r22, r24 // copy the length. 
ldi r18, 0x08 // FLPER
ldi r19, 0x02 // FLWR
out 0x34, r16
sts 0x1000,r18
cpse r22,r1 //check if low byte of length is 0
rjmp .+4 //skip next two insns if so
ldi r22, 0
subi r23,255 //same as +1 to high bte.
mov r0, r23
andi r0,1
add r0,r23  // this rigamarole ensures that r23 is now an even number of half-pages that we need to erase..
movw r30,r26 // copy application start to z reg -
spm Z
subi r31, 254 //we erased a page, so add 2 to te high byte of the address, there's no add immediate, just subtract immediate) 
subi r23, 2 // and subtract 2 from the length of the section
brne .-8 // if last math operation wasn't 0, jump back 4 instructions (8 bytes), counting from the word following this one  (that's how relative branch/jumps are)
// when we pass this, the application section is eraseyd. 
out 0x34,r16
sts 0x1000,r1 // set nvmcmd to 0 (no command) using r1, the zeroreg. 
out 0x34,r16
sts 0x1000,r19 // set nvm command to FLWR
movw r30, r28 // copy inactive section address to Z
lpm r0, Z+
lpm r1 Z+
movw r28,r30 // we've read two bytes and incremented it twice.
movw r30,r26 // grab the address of the application section to write it to
spm Z+
movw r26,r30
sbiw r24,2 //decrement length by 2.
brne .-18 //if that wasn't zero, jump back to right after we set the nvmcmd to FLWR
ldi r20, 1
out 0x34, r17 
sts 0x0040, r20 // software reset.

78 bytes in the bootlooader space for the copy routine, give or take (that waa just off the top of my head, so take with a handful of salt..

[lanmarc77]

Hi,
I think I understood. I would rather like to have the final copy process to be handled from being booted into the bootloader (so right after a reboot) and not via a callback. I want to make sure that also a power off does not leave the app section useless/inconsistent. If I understood, this could happen if the copy process is interrupted when being executed via a bootloader callback, as the bootloader does not implement anything else then copying from section to section.
I imagine the for me important case where the bootloader must decide itself if a section needs to be copied. This rollback happens if the current application does not work and some kind of trigger reboots (e.g. a power cycle or external watchdog). The app would not be able to do this assuming something went wrong during the relatively long copy process.
But this would mean I would need to pass at least the desired parameters which need to survive a reboot or even poweroff.
Marcel

[lanmarc77]

I ordered some Microchip Curiosity Nano Evaluation kits and hope I will be able to test a few things out.

[lanmarc77]

I was able to build the bootloader using Microchip Studio. This already comes with all the needed libraries. I needed to add the AVR128DB48 to pin_defs_x.h.
While working my way through all the macros I was getting an error when compiling. I inserted two #endif after line 701 closing the #if from 614 and 621. Is this specific to the compiler Microchip Studio uses or a general problem with the .h file?

[SpenceKonde]

Nope, you didn't need to modify anything. I have had nothing but woe and misery using microchip studio, I never touch it anymore.

DA-series bootloader works every bit as well on the DB - there are no differences relevant to the bootloader.

It's possible that bugs have been introduced since it was last rebuilt - I vaguely recall adding a bunch of stuff so I could build for the DD-series, which would require different bootloader binaries, but not actually building them yet, so that could hae been the origin of that), but I know I didn't rebuild the bootloaders a that time, because the parts were expevted to care still haven't been released.

Sorry if I was unclear above. What I was really trying to get across was that you don't have to - and probably don't want to, be doing the whole process in the bootloader. Get the file written to he app data section which you've declared to be inactive( while the app is running and has access to all it's communication tools without duplicating them in the bootloader. Any copying to the active app sectio has to be done by code in the bootloader section - whether you get there with a callback, or a software reset, or whathaveyou is more of a detal

An I mean if the thing only does OTA updates, or requires an otherwise unused pin pin shorted to gtound for non-OTA, since you wouldn't have to do hardly anything under normal conclusions, it could be set to stop using the WDT to exit the bootloader( (checking a pin takes setting one pinnctrl reg, waiting a few clocks, and testing if the pin is low in spite of that - if it wasn't high you could turn off the pullup, and jump to app and you'd essentially be in power up state again, so ou couold despense witth the WDT reset, and use that to help determine if the sketch if broke and needs to be restored from the backup - because that's something that I predict would be troublesome.

[SpenceKonde]

Maybe I was not as clear as I could have been there - you 100% must have a bootloader of some sort to actually copy the code, as only the bootloader section can write to the app data section.

But the storing of the new code into a non-app-code location can totally be done by the application, and may well be better handled with the communication machinery of the full sketch, depending on how the firmware is getting to the device.

[lanmarc77]

First, thank you for taking the time to discuss this with me.
I think I got everything so far. I was also working my way through the datasheet and the different flash section and who can write to what section. So having a "fat" bootloader containing all the needed communication channels is not the way to go, even though the Dx Series seems to support such big bootloaders now. The older AVR Series had size limitations if I remember correctly.
So coming back to different partitions it is at least a bit of logic inside the bootloader needed to actually know from which app data partition to copy into app code and (in case of watchdog reset or power failure or hardware reset). While handing these information to the bootloader via a callback from a running application this would not work in case of a broken application that was copied just before and then activated. Then it would need a power on or hard reset and in this case the bootloader must have enough knowledge from which appcode partition it should rollback to and if it needs to rollback at all.
I imagine using an eeprom area to keep track of these states which app data partition is actually copied inside the app code section and if it is in testing state (meaning automatic rollback on reset) or if it is persisted meaning do not rollback on reset. From what I read from the datasheet we would not need to store the partition sizes as it could be read out from the fuses which determine bootsize and codesize?

My Microchip studio project was compiled for a fixed CPU. I guess this comes with the idea of the studios project idea. Having this set to the AVR128DB64 this of course did not work with the makefile which only contained the AVR128DA series. From what I understood though is the DA and DB series identical for what the bootloader needs. I probably could choose the AVR128DA for compilation and would work in the AVR128DB and then need no change in the makefile?
My boards arrived and I was able to compile and upload the bootloader and it worked! I was able to upload code using Arduino Studio and this projects hardware package. The automatic reset on upload did not work though even though I have switched PF6 to work as reset. Any ideas on that?
Marcel

[lanmarc77]

@SpenceKonde It's giving back time. 🙏

After now digging deeper I opted out for a hardware solution. Initially I wanted to make sure to always be able to have a working "software partner" to be able to communicate with the µC in the field. Even if something really bad like power outages or brown outs are happening during the flash process. While the above ESP like OTA would work it is indeed a lot of work (but possible). And it wastes a lot of flash memory.
Eventually one just needs to be able to reliable get back into the bootloader somehow even if not physically at the location of the device. One could implement an additional reset wire but this needs installing, noise and ESD protection. I developed a hardware based reset that only works with the RS485 protocol.

I attach it for everyone who might be in need of something like this and for you to point here if someone comes around with the same questions I had:

When sending a zero at 300bps (red signal) R2+C1 will charge (blue signal) until it reaches the TTL threshold and U2B will switch to low. At around half of one byte time at 300bps:

The fast shottky diode makes sure, that when a one is send, it will discharge R2+C1 much faster then charging. A stopbit is always send and always 1. So at higher baudrates R2+C1 will not get the chance to charge to much higher voltages before being reset by the diode.
At my desired speed of 57600bps and when sending the worst case continously 0 the level stays way below the TTL threshold:

Now only by chaning the baudrate of the master (no additional wires) one can reset reliable all bus clients if they implement this circuit behind an usually already noise and ESD protected RS485 interface. The circuit was built up and tested.
Thank you for all your technical insight which helped me to implement the software side of an RS485 able bootloader. I have not used Optiboot as I needed a completly different protocol but your code examples helped with a few assembly inlines I needed to implement.
All the best
Marcel