WWW.DUMAIS.IO
ARTICLES
OVERLAY NETWORKS WITH MY SDN CONTROLLERSIMPLE LEARNING SWITCH WITH OPENFLOWINSTALLING KUBERNETES MANUALLYWRITING A HYPERVISOR WITH INTEL VT-X CREATING YOUR OWN LINUX CONTAINERSVIRTIO DRIVER IMPLEMENTATIONNETWORKING IN MY OSESP8266 BASED IRRIGATION CONTROLLERLED STRIP CONTROLLER USING ESP8266.OPENVSWITCH ON SLACKWARESHA256 ASSEMBLY IMPLEMENTATIONPROCESS CONTEXT ID AND THE TLBTHREAD MANAGEMENT IN MY HOBBY OSENABLING MULTI-PROCESSORS IN MY HOBBY OSNEW HOME AUTOMATION SYSTEMINSTALLING AND USING DOCKER ON SLACKWARESYSTEM ON A CHIP EMULATORUSING JSSIP AND ASTERISK TO MAKE A WEBPHONEC++ WEBSOCKET SERVERSIP ATTACK BANNINGBLOCK CACHING AND WRITEBACKBEAGLEBONE BLACK BARE METAL DEVELOPEMENTARM BARE METAL DEVELOPMENTUSING EPOLLMEMORY PAGINGIMPLEMENTING HTTP DIGEST AUTHENTICATIONSTACK FRAME AND THE RED ZONE (X86_64)AVX/SSE AND CONTEXT SWITCHINGHOW TO ANSWER A QUESTION THE SMART WAY.REALTEK 8139 NETWORK CARD DRIVERREST INTERFACE ENGINECISCO 1760 AS AN FXS GATEWAYHOME AUTOMATION SYSTEMEZFLORA IRRIGATION SYSTEMSUMP PUMP MONITORINGBUILDING A HOSTED MAILSERVER SERVICEI AM NOW HOSTING MY OWN DNS AND MAIL SERVERS ON AMAZON EC2DEPLOYING A LAYER3 SWITCH ON MY NETWORKACD SERVER WITH RESIPROCATEC++ JSON LIBRARYIMPLEMENTING YOUR OWN MUTEX WITH CMPXCHGWAKEUPCALL SERVER USING RESIPROCATEFFT ON AMD64CLONING A HARD DRIVECONFIGURING AND USING KVM-QEMUUSING COUCHDBINSTALLING COUCHDB ON SLACKWARENGW100 MY OS AND EDXS/LSENGW100 - MY OSASTERISK FILTER APPLICATIONCISCO ROUTER CONFIGURATIONAASTRA 411 XML APPLICATIONSPA941 PHONEBOOKSPEEDTOUCH 780 DOCUMENTATIONAASTRA CONTACT LIST XML APPLICATIONAVR32 OS FOR NGW100ASTERISK SOUND INJECTION APPLICATIONNGW100 - DIFFERENT PROBLEMS AND SOLUTIONSAASTRA PRIME RATE XML APPLICATIONSPEEDTOUCH 780 CONFIGURATIONUSING COUCHDB WITH PHPAVR32 ASSEMBLY TIPAP7000 AND NGW100 ARCHITECTUREAASTRA WEATHER XML APPLICATIONNGW100 - GETTING STARTEDAASTRA ALI XML APPLICATION

NGW100 - MY OS

2012-02-25

Preface

This section describes the implementation of my OS at a high level. Not all of it is working right now but this is the basic idea.

Source code

You can download the current code but it is far from being complete.
Download

The big picture

The OS is not meant to be an operating system like linux or windows where user application can run. The purpose is to provide a basic platform to implement different "firmwares". A firmware is a set of application in this context. Depending on what you want your NGW100 to be, you would run a different firmware (set of applications) that would drive the board through services provided by the OS.

The following picture describe what components will be handled by the OS.

The orange boxes represent modules that will be implemented in the future. The "EDX/LSE" is an OS specific component, don't try to find info about it in the datasheet. This is described here

Note how the USB controller, AC97 device, timers and several other devices are not represented in the graphics. This is because the OS won't take care of these devices. Drivers for these devices will be implemented as applications. So if I want to make my NGW100 device act as a USB storage device, I would have to write an application that implements the USB Mass Storage device protocol and use the file system module of the OS through the API. So building a router with my NGW100 would be a matter of developping applications that would run cooperatively like a serial shell app, packet-routing app, web-server app, usb mass storage app, etc.

Virtual file system

The OS will locate files using the following path structure: /0/dir/file. "/0" is the storage device number and the "/dir/file" is the path of a file on that storage device. The following storage devices will be recognized:
NumberDevice
0Onboard Dataflash
1SD card

Booting

Booting is currently done with u-boot as I don't have a JTAG tool yet. Eventually, I would like to have it boot on its own. Since u-boot initializes stuff for me, I will be own my own when I take the step.

With u-boot, I am downloading the firmware trough a tftp server and I save the image in the parrallel flash at address 0x00100000. I then execute the code from address 0x80100000 (remember, 80000000 is mapped to 0x00000000). Executing from that address makes the code run in priviledged mode.

After the kernel will start running and all initialization is done, it will look in the dataflash for the boot script. This script contains a list of applications (located on dataflash or SD card) that should run at startup and a some other settings.

Boot script

This file will be located at "/0/boot.script". todo: not designed yet

Initialization

The following is a list of things I am doing before attempting entering the idle loop in my OS

  • Reset some CPU settings to default. I looked at what u-boot was doing on reset and I do the same.
  • Configure the PM to enable PLL and run at 150MHZ.
  • Not sure what the SMC is, but I think I have to set it up in order to use ethernet devices
  • Initialize the EVBA and clear interrupt mask flag
  • Initialize SDRAM
  • Initialize the MMU
  • Initialize serial port

Memory Management

Overview

Physical memory usage

Physical memoryDescription
0x00000000 - 0x000FFFFFReserved (u-boot is in there) in parallel flash
0x00100000 - 0x0010FFFFKernel code in parallel flash
0x00110000 - 0x007EFFFFParallel flash. reserved by kernel. Unused yet
0x007F0000 - 0x007FFFFFReserved by u-boot (parallel flash)
0x00800000 - 0x0FFFFFFFINVALID
0x10000000 - 0x1000FFFFPage table(SDRAM)
0x10010000 - 0x11E73FFFprocess memory (SDRAM)
0x11E74000 - 0x11F09FFFgraphic memory
0x11F0A000 - 0x11FAFFFFreserved
0x11FF0000 - 0x11FFFFFFkernel buffers (SDRAM)
0x12000000 - 0x23FFFFFFINVALID
0x24000000 - 0x24000FFFKernel Stack (SRAM)
0x24001000 - 0x24007FFFKernel heap (SRAM)
0x24008000 - 0xFEFFFFFFINVALID
0xFF000000 - 0xFFFFFFFFMemory-mapped IO

Memory mapping viewed by kernel

Address rangeDescriptionRelation to physical memory
0x00000000 - 0x03FFFFFFINVALIDPage translated but not handled by OS
0x04000000 - 0x04000FFFKernel StackPage translated to SRAM
0x04001000 - 0x0400107FProcess TablePage translated to SRAM
0x04001080 - 0x0400108FUSART0 TX bufferPage translated to SRAM
0x04001090 - 0x0400109FUSART0 RX bufferPage translated to SRAM
0x040010A0 - 0x040010AFUSART1 TX bufferPage translated to SRAM
0x040010B0 - 0x040010BFUSART1 RX bufferPage translated to SRAM
0x040010C0 - 0x040010CFUSART2 TX bufferPage translated to SRAM
0x040010D0 - 0x040010DFUSART2 RX bufferPage translated to SRAM
0x040010E0 - 0x040010EFUSART3 TX bufferPage translated to SRAM
0x040010F0 - 0x040010FFUSART3 RX bufferPage translated to SRAM
0x04001100 - 0x0400117FReserved fo MMC driverPage translated to SRAM
0x04001180 - 0x04007FFFKernel heapPage translated to SRAM
0x04008000 - 0x7FFFFFFFINVALIDPage translated but not handled by OS
0x80000000 - 0x800FFFFFReserved (u-boot is in there) in parallel flashCached, Segment translated to flash
0x80100000 - 0x8010FFFFKernel code in parallel flashCached, Segment translated to flash
0x80110000 - 0x807EFFFFParallel flash. reserved by kernel. Unused yetCached, Segment translated to flash
0x807F0000 - 0x807FFFFFReserved by u-boot (parallel flash)Cached, Segment translated to flash
0x80800000 - 0x8FFFFFFFINVALIDN/A
0x90000000 - 0x9000FFFFPage tableCached, Segment translated to SDRAM
0x90010000 - 0x91E73FFFprocess memoryCached, Segment translated to SDRAM
0x91E74000 - 0x91F09FFFgraphic memoryCached, Segment translated to SDRAM
0x91F0A000 - 0x91FAFFFFreservedCached, Segment translated to SDRAM
0x91FA0000 - 0x91FAFFFFdataflash temp bufferCached, Segment translated to SDRAM
0x91FB0000 - 0x91FBFFFFmmc temp bufferCached, Segment translated to SDRAM
0x91FC0000 - 0x91FC017FMACB0 rx descriptor tableCached, Segment translated to SDRAM
0x91FC0180 - 0x91FC01DFMACB0 tx descriptor tableCached, Segment translated to SDRAM
0x91FC01E0 - 0x91FC19DFMACB0 rx bufferCached, Segment translated to SDRAM
0x91FC19E0 - 0x91FC31DFMACB0 tx bufferCached, Segment translated to SDRAM
0x91FC31E0 - 0x91FC31E3MACB0 current RX entryCached, Segment translated to SDRAM
0x91FC31E4 - 0x91FC31E7MACB0 current TX entryCached, Segment translated to SDRAM
0x91FC31E8 - 0x91FC31FFMACB0 net configCached, Segment translated to SDRAM
0x91FC3200 - 0x91FC337FMACB1 rx descriptor tableCached, Segment translated to SDRAM
0x91FC3380 - 0x91FC33DFMACB1 tx descriptor tableCached, Segment translated to SDRAM
0x91FC33E0 - 0x91FC4BDFMACB1 rx bufferCached, Segment translated to SDRAM
0x91FC4BE0 - 0x91FC63DFMACB1 tx bufferCached, Segment translated to SDRAM
0x91FC63E0 - 0x91FC63E3MACB1 current RX entryCached, Segment translated to SDRAM
0x91FC63E4 - 0x91FC63E7MACB1 current TX entryCached, Segment translated to SDRAM
0x91FC63E8 - 0x91FC63FFMACB1 net configCached, Segment translated to SDRAM
0x91FC6400 - 0x91FC67FFARP cacheCached, Segment translated to SDRAM
0x91FC6800 - 0x91FFFFFFreservedCached, Segment translated to SDRAM
0x92000000 - 0x9FFFFFFFINVALIDN/A
0xA0000000 - 0xBFFFFFFFSame as 0x80000000-0x9FFFFFFFSame as 0x80000000-0x9FFFFFFF but uncached
0xC0000000 - 0xDFFFFFFFINVALIDPage translated but not handled by OS

Memory mapping viewed by application process

Address rangeDescription
0x00000000 - 0x000000FFApplication stack
0x00000100 - 0x000001FFsupervisor stack
0x00000200 - 0x000002FFReserved
0x00000300 - 0x00000343Task context
0x00000344 - 0x000003FFReserved
0x00000400 - 0x01FFFFFFUser space. Size may differ, depending on allocated space
0x02000000 - 0x7FFFFFFFINVALID
0x80000000 - 0xFFFFFFFFAccess denied

Memory mapping viewed by system threads

Address rangeDescription
0x00000000 - 0x000000FFThread stack
0x00000100 - 0x00000143Task context
0x00000144 - 0x01FFFFFFReserved
0x02000000 - 0x7FFFFFFFINVALID
0x80000000 - 0xFFFFFFFFkernel space

Paging

The OS will use paging and segmentation at the same time. This means that the first 2gigs of addressing space will be separated in 4k chunks and remapped on the physical space in another way. More information on this can be found in the AVR32 architecture section of this site. Paging allows us to remap the memory in a non-contiguous way.

Application processes will use the SDRAM memory. Only the SRAM and SDRAM memories will be paged. The page table will be stored in the sdram and will be 64K in size (8192 pages). 8192 pages will allow us to maintain 4k pages for the reset of the SDRAM and the whole 32K sram. The sram pages will be used as the heap for the kernel and will be protected. The SDRAM pages will be set as private and will be associated with the owning process ASID. In the page table, a page entry will take 64bit in size. The first 32bit is going to be structured like TLBEHI. This 32bit field will allow us to find an entry on the table based on the requested page number and ASID. The last 32bit will be a copy of what should be copied in TLBELO when finding a match for TLBHI. The first 8 entries in the page table will be reserved for the SRAM. The rest of the entries will reflect the physical SDRAM space in the same order. The first 32bits of each entry do not have to be linear though.

Example Page table:
PageTable 1st 32bitsPage table last 32bits
VPN=0x24000PFN=SRAMBlock0
VPN=0x24001PFN=SRAMBlock1
VPN=0x24002PFN=SRAMBlock2
VPN=0x24003PFN=SRAMBlock3
VPN=0x24004PFN=SRAMBlock4
VPN=0x24005PFN=SRAMBlock5
VPN=0x24006PFN=SRAMBlock6
VPN=0x24007PFN=SRAMBlock7
VPN=0, ASID=1PFN=SDRAMBlock0
VPN=0, ASID=2PFN=SDRAMBlock1
VPN=89, ASID=1PFN=SDRAMBlock2
VPN=9, ASID=1PFN=SDRAMBlock3
......

The page table will also be used by the kernel to track available memory. A page with the "v" flag cleared means that this page is not reserved by any application so that 4k of memory is free. When allocating memory to a process, the "v" flag of the reserved pages will be set.

Scheduler

The scheduler will use a timer to switch task. A task switch will occur about 20 times per seconds. In order to switch tasks, the current context must be saved in the application space. The scheduler will perform the following tasks:

  • Go through the list of process and find the next one to be switched to in a round-robin fashion.
  • Using the stmts instruction, save all registers from ro to r14 to the application space buffer. stmts will save the registers from the application context. We will be executing from INT3 so registers r8 to r14 will be shadowed, meaning that even if we change them, stmts will pickup the right values that were assigned before entering INT3.
  • Save the RSR_INT3 register to the application space. RSR_INT3 contains the system flags before entering INT3. If the interrupt occured between a "cp.w" and a "breq", teh flags need to be preserved.
  • Save the RAR_INT3. This register contains the return address that rete is going to use to resume execution of the interrupted code.
  • Change the TLBEHI's ASID. Since we are switching context, the pages won't be mapped the same way. From now on, accessing the data in the application space will give us the data from the new application space.
  • using ldmts, restore R0-R14.
  • restore RSR_INT3
  • restore RAR_INT3, so we can use rete to resume the task.

Dataflash

The dataflash is accessed with the SPI bus. Since the dataflash is not a protable device and only the MCU can access it, it is a good opportunity to design my own file system. That memory will contain the boot script.

Dataflash file system

todo: not designed yet

ASTERISK FILTER APPLICATION

2012-02-25

Filter application

I use this application to manage a blacklist of telemarketers. Althought this could be easily done with a perl script, I'd rather have a real application for it. This application will allow you to define several groups. In each group, you add a list of callerID, an optional greeting message and a context/extension pair to jump to. If the callerID of an incomming call matches a callerID defined in a group, the group's greeting message will be played and the dialplan will jump to the context/extension defined for that group. If no greeting is defined, none will be played. If no context/extension pair is defined, dialplan execution will continue normaly.

Compiling and Installing

After downloading app_filter.c, copy the file in your asterisk source three and compile it. In my example, I am assuming that your asterisk source code is in /usr/src/asterisk-1.4.21.1/apps/ and that your modules are located in /usr/lib/asterisk/modules/.

cp app_filter.c /usr/src/asterisk-1.4.21.1/apps/ cd /usr/src/asterisk-1.4.21.1 make apps cp apps/app_filter.so /usr/lib/asterisk/modules/

next, go into the asterisk CLI and load the module using "module load app_filter.so". Of course it is also possible to have the module loaded automatically using /etc/asterisk/modules.conf. Now you are ready to use the module

Configuration

You must first create /etc/asterisk/filter.conf. Add the following:

[general] dbtype => 0

"dbtype" must be set to 0 if the same configuration file is going to be used for defining filter groups. Set dbtype to 1 if you prefer mysql. If you are going to be using mysql, you can skip to the next section. Otherwise, here's how to do it. Note that the configuration file is read each time that Filter() is called

You can several sections, in the file. In each section, add as many callerIDs as you want using the "cid:" directive. Define an optional greeting with "greeting", context with "context" and extension to "exten" Complete example:

[general] dbtype => 0 [test1] cid => 1001234567 cid => 1001234566 greeting => greeting1 ; do not add the extension. This file must be present in /var/lib/asterisk/sounds [test2] cid => 1001234565 cid => 1001234564 greeting => not-welcome context => telemarketer-torture exten => s

In this example, caller with callerID 100-123-4565 will hear the "not-welcome.wav" greeting message and will be redirected to extension "s" in the "telemarketer-torture" context

Configuration with mysql

mysql support is not implemented yet

Dialplan

Here is how you should use it in the dialplan:

exten => s,1,Answer() exten => s,2,Filter()

Download

The source code can be found here: app_filter.c

CISCO ROUTER CONFIGURATION

2012-02-25

My private network is built around a cisco 2620xm router and a cisco catalyst 3448-XL My router's image is c2600-adventerprisek9-mz.124-25c.bin, and for the switch: c3500xl-c3h2s-mz.120-5.WC17.bin.

DSL connection

My router has a WIC-1ADSL card. Using this, I can connect with my ISP. Note that these settings might only work for my ISP (Teksavvy, in ottawa)

vpdn enable no ip cef interface ATM0/0 no ip address atm restart timer 300 no atm ilmi-keepalive bundle enable ! dsl operating-mode auto hold-queue 224 in pvc 0/35 pppoe-client dial-pool-number 1 ! ! interface Dialer1 ip address negotiated ip access-group 120 in ip mtu 1492 ip nat outside ip nat enable ip virtual-reassembly encapsulation ppp ip tcp adjust-mss 1452 dialer pool 1 dialer-group 1 no cdp enable ppp authentication pap callin ppp pap sent-username your_username password 0 your_password ppp ipcp dns request accept ppp ipcp address accept ! ip forward-protocol nd ! set default route to go through Dialer1 interface ip route 0.0.0.0 0.0.0.0 Dialer1 ! deny telenet access from outside. access-list 120 deny tcp any any eq telnet access-list 120 permit tcp any any access-list 120 permit ip any any dialer-list 1 protocol ip permit

With these settings, your DSL connection should come up. Any host from the outside of your network will be able to access TCP ports (except 23) on your network. More details on NAT will follow.

VLAN trunking

Consider the following configuration:
VLAN 1 hosts the 10.0.0.0/24 network
VLAN 3 hosts the 10.0.3.0/24 network
VLAN 10 hosts the 10.0.10.0/24 network
VLAN 1 and VLAN3 can talk to each other and to the WAN
VLAN 10 can only talk to the WAN

interface FastEthernet0/0.1 encapsulation dot1Q 1 native ip address 10.0.0.1 255.255.255.0 ip access-group 101 in no ip unreachables ip nat inside ip virtual-reassembly ! interface FastEthernet0/0.3 encapsulation dot1Q 3 ip address 10.0.3.0 255.255.255.0 ip access-group 103 in no ip unreachables ip nat inside ip virtual-reassembly ! interface FastEthernet0/0.10 encapsulation dot1Q 3 ip address 10.0.10.0 255.255.255.0 ip access-group 110 in no ip unreachables ip nat inside ip virtual-reassembly

A subinterface (ie: 0.10) defines a vlan. In this setup, interface 0 is configured to handle VLANs 1,3 and 10 by separating it in 3 different sub-interface. Note that with this configuration, nothing more needs to be done for inter-vlan routing. By assigning an IP paddress to subinterfaces, you tell the router how to route between vlans.

With this configuration, all 3 networks will be able to talk to each other. To prevent VLAN10 to talk to VLAN 1 and 3, you could do the following:

access-list 100 deny ip 10.0.0.0 0.0.0.255 10.0.10.0 0.0.0.255 access-list 100 permit ip any any access-list 100 permit udp any any access-list 103 deny ip 10.0.3.0 0.0.0.255 10.0.10.0 0.0.0.255 access-list 103 permit ip any any access-list 103 permit udp any any access-list 110 deny ip 10.0.10.0 0.0.0.255 10.0.0.0 0.0.0.255 access-list 110 deny ip 10.0.10.0 0.0.0.255 10.0.3.0 0.0.0.255 ! deny telneting in gateway from guest network access-list 110 deny tcp 10.0.10.0 0.0.0.255 host 10.0.10.1 eq telnet access-list 110 permit ip any any access-list 110 permit udp any any

DHCP

The following configuration will setup a DHCP server on the router with a different pool for each networks.

! only hand out ip addresses from 10.0.0.100-10.0.0.149 ip dhcp excluded-address 10.0.0.1 10.0.0.99 ip dhcp excluded-address 10.0.0.150 10.0.0.255 ! only hand out ip addresses from 10.0.3.100-10.0.3.149 ip dhcp excluded-address 10.0.3.1 10.0.3.99 ip dhcp excluded-address 10.0.3.150 10.0.3.255 ! only hand out ip addresses from 10.0.10.100-10.0.10.149 ip dhcp excluded-address 10.0.10.1 10.0.10.99 ip dhcp excluded-address 10.0.10.150 10.0.10.255 ip dhcp pool pool_vlan1 import all network 10.0.0.0 255.255.255.0 default-router 10.0.0.1 dns-server 10.0.0.1 ip dhcp pool pool_vlan3 import all network 10.0.3.0 255.255.255.0 default-router 10.0.3.1 dns-server 10.0.3.1 ip dhcp pool pool_vlan10 import all network 10.0.10.0 255.255.255.0 default-router 10.0.10.1 dns-server 10.0.10.1

DNS

To use the cisco router as a DNS forwarder, the following simple configuration can be usd

ip dns server

NAT / Port forwarding

I never got port range forwarding to work on my router. I ended up writing 100 lines for a range of 100 ports. But this is not shown here for for sake of simplicity

ip nat translation timeout 3600 ip nat translation tcp-timeout 1200 ip nat translation finrst-timeout 15 ip nat translation syn-timeout 45 ! forward port 80 to 10.0.0.4:80 ip nat inside source static tcp 10.0.0.4 80 interface Dialer1 80 ! enable NAT on Dialer1 interface ip nat inside source list 1 interface Dialer1 overload access-list 1 permit 192.168.0.0 0.0.255.255

AASTRA 411 XML APPLICATION

2012-02-25

411 directory

canada411 has changed their webpage. This script does no longer work. I will try to update it when I get more time.
This application allows you to make 411 queries directly from an Aastra phone's XML browser. Basically, all it does is prompt the user for the name of a person and the city he lives in. Then a query is made on canada411.ca and the result is parsed and displayed on the phone. Of course, if canada411 change their output format, this application may not work correctly. I assigned soft button to access this application directly from the idle screen.

Download

411.tar