Creating and Linking a Custom Bootloader or Library

This forum handles questions and discussions concerning Microchip’s 8-bit compilers, assemblers, linkers and related tools.

Creating and Linking a Custom Bootloader or Library

Postby AaronD » Thu Jul 31, 2014 10:40 pm

I have a PIC16F1454 that needs to update its firmware via USB. The complete application + USB stack is a little more than 50% of the code space, so I can't just download the new code and switch to it.

What I think I can do is:
  • 1. Tease out a section to be a bootloader that can download code via USB all on its own, but that's about all it can do. Put it as high as I can get it so that the application code can have the rest, starting at the interrupt vector. (instruction 0x0004)
  • 2. Use the old bootloader to download a new bootloader over the top of the application code, including a sacrificial section that starts at the reset vector. (instruction 0x0000) The USB is polled, so once this starts, I don't need interrupts. Don't actually write the reset vector until the new code is verified, in case I lose power partway through.
  • 3. Reboot and use the sacrificial section to copy the new bootloader into upper memory over the top of the old bootloader. Verify the copy, then change the reset vector to jump to it.
  • 4. Reboot again and use the new bootloader to download the new application code. As far as the bootloader is concerned, this is the same as step #2 except that the reset vector jumps to the bootloader just like it already does.
Since I only have 0x2000 words, I want to reuse the same USB stack for both the application and the bootloader, so at least that section of the bootloader has to be available to the application. Or I could keep control in the bootloader and have it make a function call into the application code for every cycle of the main loop. Either way, at least one of them has to know about the other.

They will always be downloaded as a pair, and there should always be enough evidence for the PC side to resume correctly if it gets interrupted. I would think that that would make the linking a little easier than if the versions were not locked to each other.

So how can I do that? How to make 2 projects that can link to each other, or 1 project that can create 2 downloadable files?
AaronD
 
Posts: 11
Joined: Thu Jul 31, 2014 8:44 pm
PIC experience: Experienced Hobbyist

Re: Creating and Linking a Custom Bootloader or Library

Postby ric » Thu Jul 31, 2014 11:31 pm

I can foresee a lot of problems here.
If the application "owns" the interrupt vector, then a faulty image or download could easily "brick" your device.
It would be far safer to have an external serial FLASH chip, and download the new application image into there first, then use that to replace the old application.
Leave the bootloader in charge of the reset and interrupt vectors at all times.
Latest test project, an LED matrix display made from one reel of addressable LEDs. here
User avatar
ric
Verified identity
 
Posts: 659
Joined: Sat May 24, 2014 2:35 pm
Location: Melbourne, Australia
PIC experience: Professional 5+ years with MCHP products

Re: Creating and Linking a Custom Bootloader or Library

Postby AaronD » Fri Aug 01, 2014 10:59 pm

Hi ric,

Unfortunately, an external FLASH chip is unavailable for this project because it's supposed to be as cheap as possible. We practically give these things away as an accessory to our other products. In fact, the PCB is barely big enough to hold the PIC and a few SMT resistors. It'd sure be nice though.

If I'm REALLY careful with how I verify the code before I allow it to run, I think I can make it pretty reliable. If it doesn't check out, leave the reset vector where it is so that the old code runs again next time and makes another attempt to get it right.

Because USB can be polled, the bootloader does not need interrupts, which makes things a little bit easier. Just disable them when downloading new code so that the interrupt vector doesn't get executed before I'm ready, which is also convenient for the flash writing routines that need them off anyway. Let the application enable them as it needs, but the bootloader is completely main-line.

As for bricking the chip, yes, I have two opportunities to do that: one between downloading and copying the new bootloader (reset changes to the copying code), and one between copying and running the new bootloader (reset changes back to the bootloader). But they're only 4ms each: 2ms to erase + 2ms to write. I think the chance of losing power during any particular 4ms interval is pretty low.
Another possibility is that the flash cells wear out and end up with incorrect data, but if that happens, there are probably errors in other places too, which would cause the checksum to fail and so I'd leave the reset vector alone. And the chip is probably done at that point anyway.



Another approach that I considered was to swap the locations of the bootloader and the application: bootloader on the bottom and application on top. That would put the reset and interrupt vectors in the bootloader natively, but it would cause a little bit more latency in my bit-banged, interrupt-driven communications that are part of the application. And there's still the problem of updating both the application AND the bootloader in the field.

So I think the potential brick problem is there no matter how I do it, but easily minimized through careful coding, and having the application at the bottom makes the interrupts slightly faster.

In terms of actual use, the application always owns the interrupt vector, and the bootloader always owns the reset vector, except:
- Interrupts are globally disabled by the bootloader when starting a new download and not enabled until the new application does so.
- The reset vector is "borrowed" by some temporary code that clobbers the old bootloader with a new one.



I think the only thing left is to figure out how to take what I would consider to be one code base (could be multiple projects, but I'd rather not unless I have to), and create two downloadable files:
1. Temporary copying code starting at 0x0000 + a big gap + the new bootloader, which has function calls into file #2
2. Application code starting at 0x0004, or at 0x0000 if you count the reset vector that actually goes to the bootloader in file #1

There will be 100% overlap between the temporary copying code and the application, so that might not work as a single project. But it might work to have that be its own project that expects a 2-byte offset to be stored at the end of itself. When receiving file #1, I'll detect the gap and store the offset, then store the bootloader next to it for now. The copying code then uses that offset to put it where it's supposed to go.

So I might have two projects: one for the temporary copying code, and one for the application + bootloader. After compiling, I would normally have:
1. Copy
2. App + BtLd
What I want is:
1. Copy + BtLd
2. App
I'd need to put main(), a few other functions, and a bunch of constants into upper memory without moving the reset vector. (--code-offset doesn't work because it moves the vectors too) Then put the ISR at 0x0004 and some other functions immediately after it. Then split the final object file at that point and merge it with the other final object file. Can I do that with XC8?
AaronD
 
Posts: 11
Joined: Thu Jul 31, 2014 8:44 pm
PIC experience: Experienced Hobbyist

Re: Creating and Linking a Custom Bootloader or Library

Postby AaronD » Fri Aug 01, 2014 11:30 pm

Or another way of stating my end goal:

Before downloading new code:
This code is entirely from a single project.
Code: Select all
0x0000: Reset Vector
    GOTO Old Bootloader
0x0004: Interrupt Vector
    Application.ISR()
    Application.Init()
    Application.Main_Polled()
    Application.Handle_USB_Comms()
    Application.foo()
    Application.bar()
0x1200: Old Bootloader
    Init
    Check if Application_OK
    if(Application_OK)
        Application.Init()
    while(1)
        Clear_Watchdog()
        Bootloader.USB_State_Machine()
        if(!Bootloader.Handle_USB_Comms())
            if(Application_OK)
                Application.Handle_USB_Comms()
        if(Application_OK)
            Application.Main_Polled()
    endwhile
    Bootloader.Handle_USB_Comms()
    Bootloader.USB_State_Machine()
    Bootloader.USB_Descriptors[]
    Bootloader.foo()
    Bootloader.bar()
0x2000: End of Code Space

After download #1:
This code could be a mixture of projects.
Code: Select all
0x0000: Reset Vector
    Copy New Bootloader to intended location
    Change Reset Vector to GOTO intended location of New Bootloader
    Reboot
0x01FE: End of copying code
    Pointer to intended location of New Bootloader
0x0200: New Bootloader
    Init
    Check if Application_OK
    if(Application_OK)
        Application.Init()
    while(1)
        Clear_Watchdog()
        Bootloader.USB_State_Machine()
        if(!Bootloader.Handle_USB_Comms())
            if(Application_OK)
                Application.Handle_USB_Comms()
        if(Application_OK)
            Application.Main_Polled()
    endwhile
    Bootloader.Handle_USB_Comms()
    Bootloader.USB_State_Machine()
    Bootloader.USB_Descriptors[]
    Bootloader.foo()
    Bootloader.bar()
0x1200: Old Bootloader
    Init
    Check if Application_OK
    if(Application_OK)
        Application.Init()
    while(1)
        Clear_Watchdog()
        Bootloader.USB_State_Machine()
        if(!Bootloader.Handle_USB_Comms())
            if(Application_OK)
                Application.Handle_USB_Comms()
        if(Application_OK)
            Application.Main_Polled()
    endwhile
    Bootloader.Handle_USB_Comms()
    Bootloader.USB_State_Machine()
    Bootloader.USB_Descriptors[]
    Bootloader.foo()
    Bootloader.bar()
0x2000: End of Code Space

After download #2:
This code is entirely from a single project.
Code: Select all
0x0000: Reset Vector
    GOTO New Bootloader
0x0004: Interrupt Vector
    Application.ISR()
    Application.Init()
    Application.Main_Polled()
    Application.Handle_USB_Comms()
    Application.foo()
    Application.bar()
0x1280: New Bootloader
    Init
    Check if Application_OK
    if(Application_OK)
        Application.Init()
    while(1)
        Clear_Watchdog()
        Bootloader.USB_State_Machine()
        if(!Bootloader.Handle_USB_Comms())
            if(Application_OK)
                Application.Handle_USB_Comms()
        if(Application_OK)
            Application.Main_Polled()
    endwhile
    Bootloader.Handle_USB_Comms()
    Bootloader.USB_State_Machine()
    Bootloader.USB_Descriptors[]
    Bootloader.foo()
    Bootloader.bar()
0x2000: End of Code Space

The firmware is now updated.

I didn't show the required error-checking, but you get the idea.
Looking at the first and third panels, how can I layout a project like that with XC8?
AaronD
 
Posts: 11
Joined: Thu Jul 31, 2014 8:44 pm
PIC experience: Experienced Hobbyist

Re: Creating and Linking a Custom Bootloader or Library

Postby AaronD » Tue Aug 12, 2014 9:10 pm

Okay, I think I got it. The manual isn't very clear, but the basic idea is:

1. Write some code and see what section ("psect") the compiler decides to put it in. Search the .map and .lst files for the function and constant names that you used in your C code.
2. Use "#pragma psect oldsection=NewSection" to redirect everything that would normally go in oldsection into a new one that you can control. I believe it's case sensitive, but you can define NewSection any way you want, without spaces or special characters.
  • oldsection can have wildcards in it. For example, text%%u will match any section that starts with "text" and ends with a number.
  • #pragma psect affects the entire source file that it sits in. It can be anywhere in the file. Duplicates are not allowed, that is, you can't redirect similar code different ways in the same source file. You need a separate source file to do that.
3. In Project Properties -> XC8 linker -> Additional options -> Extra Linker Options, use -L-pNewSection=<address> to start it at that address, or -L-pNewSection=OtherSection to start it immediately after OtherSection.

As an example, my relocated HelloWorld project looks like this:
Code: Select all
//Relocate code into upper memory
//  must be in the file itself, can't be #included
#pragma psect maintext=BootloaderCodeMain
#pragma psect text%%u=BootloaderCode
#pragma psect strings=BootloaderConst       //constants normally go in strings, but this pragma bumps them to stringtext; don't know why
#pragma psect stringtext=BootloaderConst    //catch stringtext also


unsigned char foo(unsigned char index);


const unsigned char bar[5] =
{
    0,
    1,
    2,
    3,
    4
};


void main(void)
{
    while(1)
    {
        CLRWDT();
        PORTA=foo(3);
    }
}

unsigned char foo(unsigned char index)
{
    return bar[index];
}

And it has these Extra Linker Options:
Code: Select all
-L-pBootloaderCodeMain=1200h -L-pBootloaderCode=BootloaderCodeMain -L-pBootloaderConst=BootloaderCode

Except for the compiler-generated startup code that I didn't relocate, the entire program now lives at 0x1200 and above.
AaronD
 
Posts: 11
Joined: Thu Jul 31, 2014 8:44 pm
PIC experience: Experienced Hobbyist

Re: Creating and Linking a Custom Bootloader or Library

Postby Ian.M » Wed Aug 13, 2014 12:40 pm

Next problem: making the application's RAM usage compatible with the USB stack in the bootloader!

We've had a very similar discussion to this over at MCHP a few months back, and IIRC XC8 just doesn't have the level of memory allocation control required to make it practical.

There's also the problem of the FLASH erase block size - the bootloader cannot rewrite the interrupt vector without clobbering the reset vector, so there is a high risk of bricking. If you can get round the RAM allocation issues, your best bet would be to put a BRA instruction at the interrupt vector to branch to 0x0024 and tell the compiler to relocate the application by 0x20. That means you never have to touch the bootblock and only adds two cycles to the interrupt latency. You may even move the application up a few more blocks to allow enough room for a machine code routine that does a checksum of the application, and checks an i/o pin to decide whether to boot into the application or the bootloader.
Last edited by Ian.M on Thu Aug 14, 2014 12:35 am, edited 1 time in total.
Ian.M
Verified identity
 
Posts: 95
Joined: Wed May 28, 2014 12:47 am
PIC experience: Professional 1+ years with MCHP products

Re: Creating and Linking a Custom Bootloader or Library

Postby ric » Thu Aug 14, 2014 12:30 am

Ian.M wrote:... put a BRA instruction at the interrupt vector to branch to 0x0024 and tell the compiler to relocate the application by 0x20. ....

Note that Ian specifically mentions using BRA rather than GOTO.
In any PIC16F chip with more than one bank of FLASH, it is not safe to do a GOTO inside an ISR until you have written a known value to the PCLATH register.
Ignoring this can lead to many sleepless nights chasing intermittent crashes!
Latest test project, an LED matrix display made from one reel of addressable LEDs. here
User avatar
ric
Verified identity
 
Posts: 659
Joined: Sat May 24, 2014 2:35 pm
Location: Melbourne, Australia
PIC experience: Professional 5+ years with MCHP products

Re: Creating and Linking a Custom Bootloader or Library

Postby Ian.M » Thu Aug 14, 2014 12:45 am

Oh yes. A GOTO at the start of the ISR on any core that completes the goto address from PCLATH is pure poison.

The only devices that its OK for are PIC18 which uses a two word GOTO, the 'enhanced' baseline core (the one with four stack levels and interrupts)that swaps critical registers as part of its fast context saving so the ISR PCLATH is preserved from one invocation to the next), and midrange devices with a single bank of FLASH as Ric notes.
Ian.M
Verified identity
 
Posts: 95
Joined: Wed May 28, 2014 12:47 am
PIC experience: Professional 1+ years with MCHP products

Re: Creating and Linking a Custom Bootloader or Library

Postby AaronD » Thu Aug 14, 2014 11:52 pm

Ooo, I see a cruel prank depending on how the development team is organized! :twisted:

Anyway, my application and bootloader are in the same project, so the compiler knows about the RAM for both simultaneously. The linking part is proving to be a headache between less-than-descriptive error messages and a less-than-helpful help file.

After talking with a supplier, I solved the potential brick problem my having some fixed startup code in the boot block and then protecting it with the config bits. This code checks a flag that is just outside of the protected area to determine whether to jump to the main code or to an internal copy/verify routine. The erased value goes one way, else the other. The flag is only changed when both the code and the image to copy are valid. An unexpected reset may write flash more than required, but it never runs invalid code. It does create 3 cycles of extra latency to the interrupts (LJMP, boot block is too big for BRA), but after much consideration, I think I can handle that. ;)



I still have two problems:

1.
If I compile my code with no special linker options, it works with no errors. If I add these options, I get fixup errors:
Code: Select all
Extra Linker Options:
-L-pBootloaderCodeMain=1000h -L-pBootloaderCodeUSB=BootloaderCodeMain -L-pBootloaderConstUSB=BootloaderCodeUSB -L-pBootloaderCode=BootloaderConstUSB -L-pBootloaderConst=BootloaderCode -L-pApplicationISR=0200h

ROM ranges:
default,-0-1FF

As best I can tell from the help file, fixup errors mean that the compiler has done its thing already and left symbolic addresses for everything, the linker has put everything where it's supposed to go, and there is now a placeholder that is too small for the address that needs to go there. It says nothing about how to fix it.

Either way, the C code still has one of these in every .c file except for the one that I'm okay with using any remaining free space:
Code: Select all
//Relocate code into custom sections to be linked explicitly
//  must be in the file itself, can't be #included
//  likewise, relocated constants must be in the file itself, not in a header
#pragma psect maintext=BootloaderCodeMain
#pragma psect text%%u=BootloaderCode
#pragma psect strings=BootloaderConst       //constants normally go in strings, but this pragma bumps them to stringtext; don't know why
#pragma psect stringtext=BootloaderConst    //catch stringtext also


2.
Most of the project is spread across several C source files, but the startup code is written in assembly. I'd like to include that into the project as well so that I end up with one hex file to debug with and send to production. For field updates, I can split the hex file later into startup (throw away), application (download 2nd), and bootloader (download 1st). But when I add the assembly file "Startup.s" to the C project, the makefile seems to break:
Code: Select all
make[2]: *** No rule to make target 'build/default/production/Startup.obj', needed by 'dist/default/production/Main.X.production.hex'.  Stop.
make[1]: *** [.build-conf] Error 2
make[2]: Leaving directory 'C:/Users/aaron/MCP/CastleLink_V1/Main.X'
make: *** [.build-impl] Error 2
nbproject/Makefile-default.mk:75: recipe for target '.build-conf' failed
make[1]: Leaving directory 'C:/Users/aaron/MCP/CastleLink_V1/Main.X'
nbproject/Makefile-impl.mk:39: recipe for target '.build-impl' failed

BUILD FAILED (exit value 2, total time: 1s)

I could have the startup code in a separate project and then link the hex file into the C one, but I'd like to make sure that the jump to main() works without duplicating code and forgetting to update one of them. :oops: It's actually a two-stage jump that goes to the end of flash first and then from there to main(). This gives the startup code a known place to go while keeping the bootloader's starting address flexible.
The dual-project idea might work anyway if I could encode that jump in the C project. And it would keep the end of the bootloader from running over it.
AaronD
 
Posts: 11
Joined: Thu Jul 31, 2014 8:44 pm
PIC experience: Experienced Hobbyist

Re: Creating and Linking a Custom Bootloader or Library

Postby ric » Fri Aug 15, 2014 12:16 am

AaronD wrote:...
It does create 3 cycles of extra latency to the interrupts (LJMP, boot block is too big for BRA), but after much consideration, I think I can handle that. ;)
...

I assume you are aware that you can only get away with that on enhanced mid-range chips, because they have already saved the PCLATH register for you.
Standard mid-range chips don't save PCLATH, (and don't have a BRA instruction). A LJMP macro would kill the entry value of PCLATH.
Latest test project, an LED matrix display made from one reel of addressable LEDs. here
User avatar
ric
Verified identity
 
Posts: 659
Joined: Sat May 24, 2014 2:35 pm
Location: Melbourne, Australia
PIC experience: Professional 5+ years with MCHP products

Next

Return to MPLAB XC8

Who is online

Users browsing this forum: No registered users and 4 guests