WWW.DUMAIS.IO
ARTICLES
OVERLAY NETWORKS WITH MY SDN CONTROLLERSIMPLE LEARNING SWITCH WITH OPENFLOWINSTALLING KUBERNETES MANUALLYWRITING A HYPERVISOR WITH INTEL VT-X CREATING YOUR OWN LINUX CONTAINERSVIRTIO DRIVER IMPLEMENTATIONNETWORKING IN MY OSESP8266 BASED IRRIGATION CONTROLLERLED STRIP CONTROLLER USING ESP8266.OPENVSWITCH ON SLACKWARESHA256 ASSEMBLY IMPLEMENTATIONPROCESS CONTEXT ID AND THE TLBTHREAD MANAGEMENT IN MY HOBBY OSENABLING MULTI-PROCESSORS IN MY HOBBY OSNEW HOME AUTOMATION SYSTEMINSTALLING AND USING DOCKER ON SLACKWARESYSTEM ON A CHIP EMULATORUSING JSSIP AND ASTERISK TO MAKE A WEBPHONEC++ WEBSOCKET SERVERSIP ATTACK BANNINGBLOCK CACHING AND WRITEBACKBEAGLEBONE BLACK BARE METAL DEVELOPEMENTARM BARE METAL DEVELOPMENTUSING EPOLLMEMORY PAGINGIMPLEMENTING HTTP DIGEST AUTHENTICATIONSTACK FRAME AND THE RED ZONE (X86_64)AVX/SSE AND CONTEXT SWITCHINGHOW TO ANSWER A QUESTION THE SMART WAY.REALTEK 8139 NETWORK CARD DRIVERREST INTERFACE ENGINECISCO 1760 AS AN FXS GATEWAYHOME AUTOMATION SYSTEMEZFLORA IRRIGATION SYSTEMSUMP PUMP MONITORINGBUILDING A HOSTED MAILSERVER SERVICEI AM NOW HOSTING MY OWN DNS AND MAIL SERVERS ON AMAZON EC2DEPLOYING A LAYER3 SWITCH ON MY NETWORKACD SERVER WITH RESIPROCATEC++ JSON LIBRARYIMPLEMENTING YOUR OWN MUTEX WITH CMPXCHGWAKEUPCALL SERVER USING RESIPROCATEFFT ON AMD64CLONING A HARD DRIVECONFIGURING AND USING KVM-QEMUUSING COUCHDBINSTALLING COUCHDB ON SLACKWARENGW100 MY OS AND EDXS/LSENGW100 - MY OSASTERISK FILTER APPLICATIONCISCO ROUTER CONFIGURATIONAASTRA 411 XML APPLICATIONSPA941 PHONEBOOKSPEEDTOUCH 780 DOCUMENTATIONAASTRA CONTACT LIST XML APPLICATIONAVR32 OS FOR NGW100ASTERISK SOUND INJECTION APPLICATIONNGW100 - DIFFERENT PROBLEMS AND SOLUTIONSAASTRA PRIME RATE XML APPLICATIONSPEEDTOUCH 780 CONFIGURATIONUSING COUCHDB WITH PHPAVR32 ASSEMBLY TIPAP7000 AND NGW100 ARCHITECTUREAASTRA WEATHER XML APPLICATIONNGW100 - GETTING STARTEDAASTRA ALI XML APPLICATION

ARM BARE METAL DEVELOPMENT

2014-11-01

The project

I've always wondered how programming for an ARM cpu is. So I decided to try to make an OS, written 100% in assembly for an ARM development board. I shouldn't say OS though, every time I write an OS, I really only make: memory management, scheduler, mutex, netcard driver, serial port driver and some small application to run on the "os". It's basically just to learn about the architecture of the device.

The ARMv7 architecture offers a lot of cool features that I am not using. I just want to keep things simple for now. Once I get something working good, I will go deeper in the documentation and try some more advanced stuff.

At first, I wanted to use my beaglebone black to run my OS. But then, I found out that qemu can emulate quite a few boards and it would be easier to do. By using qemu, I get the following advantages over using a real board:

  • no need to upload code to the board, I use the image directly
  • can reboot the machine easily while working remotely (no need to physically access the board)
  • very easy to peek in memory with qemu's monitor command "pmsave"
  • can use gdb to debug with qemu
  • no need for a separate bootloader. Can boot kernel directly.

I chose to use the "realview-pb-a8" emulated board in qemu. I have never seen that board, I have no idea what it is. It uses a Cortex-A8. So I was able to get a programing guide for that SoC. I started from there.

The fact that I am using qemu makes things easier but removes a lot of fun. For example, qemu boots my kernel directly. On a real board, I would need to write a bootloader (or use u-boot). I would need to initialized SDRAM, initialize clocks and "power domains" and other board initialization. QEMU boots your kernel directly into RAM and you can run from there. So I wouldn't quite call this "bare-metal" programming. I guess I could only call this project "kernel programming for a Cortex-A8".

Getting started

The first step is to create a small test and actually run it. So I created the following program:

Note the qemu command in the Makefile. This allows me to run the test using "make run". It will emulate the ARM board which is the realview-pb-a8

Board specifications

When starting development on a new board, the first thing you need to do is to get a memory map of the device. Because the board will contain sdram, sram, memory mapped peripheral IO etc... From board to board, the physical location of those elements will change. Here is the memory map for the realview-pb-a8

Physical Memory Layout

Physical addressDescription
0x00000000-0x0FFFFFFFSDRAM mirror
0x10000000-0x1001FFFFPeripherals
0x10020000-0x1005FFFFBoard specific stuff that I don't need just yet
0x10060000-0x1007FFFFOn board SRAM
0x10080000-0x6FFFFFFFBoard specific stuff that I don't need just yet
0x70000000-0x8FFFFFFFSDRAM
0x90000000-0xFFFFFFFFBoard specific stuff that I don't need just yet

A more detailed memory map can be found in the RealView Platform Baseboard for Cortex A8 User Guide.

Booting

Interrupt vector table

This architecture only uses 7 interrupt vectors

0x00Reset
0x04Undefined Instruction
0x08Software Interrupt
0x0CPrefetch Abort
0x10Data Abort
0x14reserved
0x18IRQ
0x1CFIQ

The interrupt vector table must be placed at the begining of the memory. Each entry is 32bits wide. It must be an instruction not an address. So you would typically put a branch instruction to jump to the proper handler. Using qemu, my kernel gets loaded at 0x70010000, so putting the IVT at the begining of my kernel would not work. I had to rellocate the IVT to 0x70000000 once the kernel was running. By the way, on that board the SDRAM starts at 0x70000000 but is mirrored to 0x00000000. Still, qemu starts execution at 0x70010000. but if the IVT is at 0x70000000, the CPU will still see it at 0x00000000 because of the mirror.

Setting up the stack

There are 6 CPU modes in this architecture. Each mode will shadow the register r13 (stack pointer). So they each need their own stack. To set those stacks, you must switch mode and set r13 appropriately. I don't set the User mode stack because this will be done on a per-process basis and System mode uses the same registers as user mode.

msr CPSR_c,#0b11010001 // stack for FIQ mode ldr r13,=STACK_BASE_FIQ msr CPSR_c,#0b11010010 // stack for IRQ mode ldr r13,=STACK_BASE_IRQ msr CPSR_c,#0b11010111 // stack for Abort mode ldr r13,=STACK_BASE_ABORT msr CPSR_c,#0b11011011 // stack for Undefined mode ldr r13,=STACK_BASE_UNDEFINED msr CPSR_c,#0b11010011 // stack for Supervisor mode. And we will stay in that mode ldr r13,=STACK_BASE_SUPERVISOR

Memory Management Unit

Creating a paged memory system is not difficult. The MMU offers a 2 level page table system The level1 table has 4096 entries, each mapping 1Mb of virtual addresses. You could create "section" entries to map those 1Mb to physical memory directly. You would then get pages of 1Mb and only 1 table that takes 16k in memory. But if you want 4k pages, then those entries need to be "Coarse table" entries, meaning that each entry will reference a subtable (a level2 table). Each level 2 table contain 256 entries, mapping 4k of memory. So for a 4k paging system you would have 1 Level1 table with 4096 entries (a total of 16k in size) and 4096 level2 tables containing 256 entries each for a total of 4Mb in size.

Level1, Section
313029282726252423222120191817161514131211109876543210
base addrNS0nGSAP2TEXAP0domainXNCB1PXN

Level1, Page Table (TODO)
313029282726252423222120191817161514131211109876543210
page table addr0domain0NSPXN01

Level2, Small Page (TODO)
313029282726252423222120191817161514131211109876543210
base addrnGSAP[2]TEXAP[1:0]CB1XN

Domains and permissions

The mmu has a concept of domains and access permissions. 16 access domains exist. In a page descriptor, we set the access bits and the domain associated with that page. CP15.register3 contains 2 bits for each domains 0 to 15. These bits determine how page access should be checked. Example: a page is associated to domain 12. CP15.register3 indicates that domain12 is Client. Therefore access permissions in the page will be checked. If domain12 was set to Manager, permissions would have been ignored.

Initializing the MMU

The first thing you need to do is setup the page tables like mentionned above. Obviously, you might want to do an identity mapping for the region of code that is currently running the MMU initialization code so that the mapping does not change after having initialized the MMU.

Level 1 Page Table

70100000 01 40 10 00 01 44 10 00 01 48 10 00 01 4c 10 00 70100010 01 50 10 00 01 54 10 00 01 58 10 00 01 5c 10 00 ... 70103fe0 01 20 50 00 01 24 50 00 01 28 50 00 01 2c 50 00 70103ff0 01 30 50 00 01 34 50 00 01 38 50 00 01 3c 50 00

Level 2 Page Tables (all contiguous)

70104000 fe 0f 00 00 fe 1f 00 00 fe 2f 00 00 fe 3f 00 00 70104010 fe 4f 00 00 fe 5f 00 00 fe 6f 00 00 fe 7f 00 00 ... 70503fd0 f2 4f ff ff f2 5f ff ff f2 6f ff ff f2 7f ff ff 70503fe0 f2 8f ff ff f2 9f ff ff f2 af ff ff f2 bf ff ff 70503ff0 f2 cf ff ff f2 df ff ff f2 ef ff ff f2 ff ff ff

Then you need to configure the CP15 register.

RegisterDescriptionValue
CP15.reg2Translation Table Base RegisterLoad the base address of the Level1 table
CP15.reg3Domain Access control registerMCR p15, 0, , c3, c0, 0 where Rd contains the 32bits we want to write. We will use domain 0 for the kernel for now. So bit 1:0 will be set to 0b11 to allow access without checking permissions. all other domains will be set to 0b00 to unconditionally deny access.
CP15.reg1ControlSet bit0 high to enable the MMU. Do this as the last step

The following registers are also useful but not needed during initialization

CP15.reg5FSRRead this in you fault hander. It is the fault code
CP15.reg6FARRead this in you fault hander. It is the faulty virtual address
CP15.reg8Invalidate TLBUsed to invalidate the TLB. invalidate entire TLB: mcr p15, 0, Rd, c8, c7,0
CP15.reg10TLB LockdownUsed mark a TLB entry as persistent so it does not get overwritten by other entries. can increase performance for pages such as those containing interrupt handling code so that the translation is always cached. We could probably use a level1 table entry as a Section of 1Mb for the kernel and lock it down in the TLB.

Faults

I will not list the different reasons for getting a fault since this is all covered in the reference manual. Basically, if a fault occurs while prefetching an instruction then the Instruction Fetch fault will occur. If the fault occurs while accessing data, a Data fault will occur. The virtual address that caused the fault will be stored in FAR. A more detailed error code will be found in FSR

Processes

To test the multitasking system, I have created some small programs that I build separately and package in the image. The kernel loads the programs in their own process. Ideally, I would have like to create a flash image that contains the kernel at the very begining and then the programs would be appended at the end, kind of like a real hard disk with a bootloader and programs. But I could never get qemu to eumlate a flash file. Even with the "-pflash". The documentation for the board says that when the board is powered on, the flash is mapped at 0x00000000. This will shadow the sdram. To use the sdram,you must remap the flash to some other place. But I could never make that work. I tried creating a flash image and provide it to qemu with the "-pflash" option but that doesn't seem to work. Qemu always wants a kernel file to be provided. I don't know why I can just put my code in a flat binary that would act as flash and get the code running from 0x00000000. The kernel file gets loaded at 0x70010000 which is the sdram. So I am creating a image file containing the kernel and the programs that get loaded in sdram by qemu.

Programs run as domain 1, and in user mode. Their virtual mapping is:

Level1 table entries (1mb mapping)TypePermissionsDescription
0Section PL1 RWX, PL0 -Kernel code. Identity mapping
1-FFSection PL1 -, PL0 -Unmapped
100Page table PL1 RW, PL0 -peripherals, and SRAM. Identity mapping
101-1DFSection PL1 -, PL0 -Unmapped
1E0Page Table PL1 RW, PL0 -Peripherals. Identity mapping
1E1-1FFSection PL1 -, PL0 -Unmapped
200-2FFPage Table PL1 RWX, PL0 RWXProcess code
300-6FFSection PL1 -, PL0 -Unmapped
700-8FFSection PL1 RWX, PL0 -Kernel code. Identity mapping
900-EFFSection PL1 -, PL0 -Unmapped
F00-FFFPage tables PL1 RW, PL0 RWProcess Stack

The task information page

When creating a process, I add it in a list of process. The list of process is a fixed-size list in kernel memory (accessible by any process in privileged mode) that contains a pointer to the L1 table of the process and several other usefull information for the process. This information is used by the scheduler and is formatted like this:

OffsetDescription
0x0000Physical address of the process's L1 page table
0x0004saved r13_irq registers
0x0040Quantum count

Using Software Interrupts

When using the SWI instruction, you need to pass it a parameter that would normally be the function number you would want to call. For example: SWI 0x02. Once you are in the SWI handler, you want to get that parameter to know where to dispatch the handler. but the SWI instruction completely ignores the parameter. It is not given to you in any way when you get in your handler. In order to get this, you need to take r14, which contains the return address and substract 4. That would give you the address of the SWI instruction itself. So you can read at that memory area and see what parameter was provided. That is pretty weird in my opinion. I would rather just put the function number in r0 before calling SWI and read r0 once in the handler. That would illimnate an unncessary memory access. Plus, the page at that location will obviously be in the instruction prefetch cache but since we are using "LDM" to load the instruction in a register to read it, it means we will be looking in the data cache. And the page will most probably not be in that cache. So in my project, I will only pass function parameters in a register and ignore the one provided to SWI.

Something that confuses me is that when calling SWI, you enter Supervisor mode. Then you are in privileged mode. That makes sense, But then r13 and r14 gets shadowed. I'm not entirely sure why I would want that. It actually complicates things when multi-tasking. I guess that in some more complex OS design, this is very usefull.

Multi-Tasking

Saving registers of mode X from mode Y

Assume we have a function called schedule(). This function saves the current context, and reloads the context of the next task to run. In my implementation, this function willa lways be called from the IRQ mode. So the schedule function will be called from a non-user mode. The schedule function will need to store the user-mode context (registers r0 to r14). But from the non-user mode of IRQ, registers r13 and r14 are shadowed. r0-r12 will be the same as the user-mode so we need to find a way to save the r13 and r14 of the user mode. For this, the instruction stm/ldm with "^" can be used to store/load the user-mode registers. this will save/load r0-r12 as usual but the r13 and r14 will be the ones of the user mode.

CPSR and SPSR: While in an exception (therefore in a mode different than user or system) the previous cpsr is saved in spsr. Before returning back from the exception, you must reload spsr back into cpsr. This will change the mode automatically, re-enable interrupts etc. To load this and to load r14 in r15 at the same time, look at the notes below about the LDM instruction.

LDM instruction format: Compared to AVR32 and x86, this is pretty complicated in my opinion. The "ldm" instruction has 3 forms. The first form does what it says it does. But the second form which is: ldm Rn,registers_without_r15^ (yes, there is a "^" at the end) loads all user mode registers while you are in a non-user mode. so it is a way to load user registers while they are shadowed. The third form, ldm Rn,registers_with_r15^, will automatically load spsr back into cpsr. You could also use a data instruction with the "S" flag and R15 as a destination. For some reason, it will conveniently reload spsr back into cpsr at the same time... Go figure. For example: movs r15,14; will reload r15 and also reload spsr back into cpsr. I am wondering why they re-purposed a flag like this. That is one small thing that makes me like x86 more than ARM.

Since I re-enable interrupts after entering SWI, the SVC context must be saved also since a context switch could occur while in a service call (that is actually the whole point of re-enabling interrupts in SWI). So my context-switching code also pushes the r13_svc and r14_svc on the the task's IRQ stack.

Context switching

The schedule function needs to do the following:

  • save registers r0-r14 user-mode
  • save register r14_irq (since this will be done from the IRQ handler)
  • save register spsr (which is the usermode cpsr)
  • change level1 page table for new process
  • flush tlb (unless using ASID)
  • restore r0-r14 (for user-mode)
  • restore the return address in r14_irq
  • restore spsr
  • to return, load r14_irq in r15 and spsr into cpsr

Each task have their own stack and they have their own IRQ mode stack. When entering the schedule() function in IRQ mode, the use-mode registers are pushed onto the IRQ stack. The current spsr and r14_irq is also pushed on the IRQ stack. The r13_irq is then saved in a list somewhere. When time comes to restore the task, the page tables are switched back to that task's page tables, and r13_irq is restored from the list. At this point, the task's IRQ stack has been restored. We can then pop everything from it and the context switch is done. Here is a sample of my schedule function

push(r0-r14)^ mrs r0,SPSR push(r0) // At this point, whole context is saved on stack // Determine what is the next task to run ... // store context: save r13_irq // r0 points to the entry in the process list (as decribed earlier) str r13,[r0,#4] // offset 0x04 is r13 // load new page table ldr r1,[r5] //r5 points to the entry of the next task to run mcr p15,0,r1,c2,c0,0 // flush TLB (note that there are ways to avoid this mov r1,#0 mcr p15,0,r1,c8,c7,0 //load r13_irq ldr r13,[r5,#4] // now restore context on stack pop(r0) // this is just SPSR, only reg available to touch is r14 msr SPSR,r0 pop(r0-r14)^ b returnFromInterrupt

This is a sample only. My schedule() function does a bit more than that. But it gives you the general idea.

When r0-r14 will be restored for the user mode, it would restore the task's context as it was before entering the IRQ to schedule(). r13 and r14 of the user mode will be restored and not the banked ones of the currently executing mode.

Note that the TLB must be flushed when reloading the "translation table base register" in CP15 because the cached TLB entries will continue to correspong to the previous mapping. This is a very expensive operation but we can use the concept of ASID by using the CONTEXTIDR register. By setting a unique task ID in CONTEXTIDR, all page translations that gets loaded in the TLB will be tagged with that ID. When doing a lookup, the MMU will ignore entries that do not match the current CONTEXTIDR. So on a context switch, you would change the ID in CONTEXTIDR. This would create duplicate entries in the TLB but with different IDs. So instead of flushing the TLB, entries will be removed only when the TLB is full. See this article for more information about the TLB and ASID.

The schedule() function is called by the timer IRQ handler. But you might want to call it from other places. For example, if a task wants to yield, it should be scheduled out immediately. I could do this in a SWI handler but trying to change context from the SVC mode brings up other challenges. So to keep things simple, I want to do context switches only from the IRQ mode. For this, it is possible to use a "software IRQ". This is well documented in the GIC documentation.

Download

Here is my source code