WWW.DUMAIS.IO

Home Automation SystemLast edited on Mar 6, 2013

The version of this project is obsolete. I am now running this software on a x86-64 machine. Code can be found on github

The idea

I wanted to control and monitor parts of my house. So I thought about building a device that would:

  • Control my Insteon devices
  • Monitor some relay triggers (alarm system PGM, my sump pump, doors)
  • generate wakeup calls
  • control IR cameras
  • generate sounds (alarms, welcome message, ...)

I decided to get a Raspberry PI and make an expansion board to do all this. I made a server application for the rPI and now I can add many features to my house. I wanted my device to do two things only: generate events, and execute commands. So with a script, I can instruct it to execute a command based on any combination of events. Pretty simple, yet very powerful.

Raspberry PI preparation

This is what needs to be done on the raspberry pi:

  • Prepare a Linux image
  • Allow ssh root login and configure network
  • Enable sound card and install alsa-lib
  • Enable serial port
  • Configure a syslog server
  • Configure timezone (make symbolic link /etc/localtime -> /usr/share/zoneinfo/Canada/Eastern)
  • Install berkeley DB (5.3.21)
  • Auto-start server in /etc/rc.local
  • Install bind (DNS server)

For this project I decided to use ArchLinux on the rPI. I used the image ARCHlinuxARM-13-06-2012. I proceeded with copying the image on the flash disk, change the root password, allowed root login through SSH and assigned a static IP to the device. After that, I had a rPI device that I could access throuh SSH.

To enable sounds, I had to create the file /etc/modules-load.d/sound.conf and add a line that states: snd-bcm2835. To use the sound card, I used alsa-lib-1.0.26. I compiled it (as part of my cross-platform toolchain) with:

CC=arm-unknown-linux-gnueabi-gcc ./configure --target=arm-unknown-linux-gnueabi --host=x86_64-unknown-linux-gnu --prefix=/opt/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi --disable-python
make

Then I copied src/.libs/libasound.a and include/*.h in my source folder because I wanted to link statically with alsa so that I don't need to install alsa-lib on the rPI. My application could not find the device "hw:0,0", which should be the analog output on the rPI. I had to deploy a alsa.conf file that defines a device called "homeautomationsounddevice" and use that device when initializing the sound card. I used snd_pcm_open_lconf to specify the local configuration file that I created.

To be able to use the serial port on the rPi, I had to disable the serial console. There is tons of information about this everywhere so I won't go into the details of that.

Since the rPI uses flash memory, I didn't want to wear it out by writing tons of logs on the flash card. So all my debugging output is done through syslog. On the syslog server, I had to edit the bootscripts to start syslog with the "-r" option to listen for incoming syslog messages. Then I had to edit /etc/syslog.conf to log all local7.info logs in a separate file: "local7.info /var/log/syslog-local7"

I had to install a DNS server for my alarm system's IP module to work (more details below). So I installed bind and made it start automatically on boot. I then created a zone for (dhas.ca) with an MX record. The IP module is the only one to use the DNS server so I don't care for domain name collision.

Development

When I need a toolchain for cross-compiling, I usually install everything manually. But this time, for the rPI, I saw that everyone was using "crosstool-ng" which is a script that will build a toolchain for the target platform of your choice. I decided to give it a try. At first it was failing to download the gcc package so I had to look at the logs, find the package in question and then download it manually. I stored it in the tarball folder that it was supposed to be copied in and then everything else went fine. Note that is is very important to specify "softfp" as the FPU. Otherwise, some libraries like Lua will fail. After the toolchain was built, I proceeded with writing a test application and copied it over on the rPI. The application would not execute, I always got an error "No such file or directoy" even though the file really existed. After much searching I tried doing "readelf -a test2 | grep "Requesting"" and I got "[Requesting program interpreter: /lib/ld-linux-armhf.so.3]". that file did not exist on the archLinux that I installed on the rPI. I had to make a symbolic link of that name to /lib/ld-linux.so.3 and then everything was fine. I guess the real fix would be to fix my toolchain, but I'll leave it like that for the moment.

Chassis

I wanted to rackmount my device so I started to look for anything that is rackmountable on kijiji. I was looking for a broken device but then I found this guy who was selling a working hub for 5$. The hub had a power supply that provides 5V and 12V. Perfect for my project.

So I removed everything inside except the power supply. For the front panel, I used a piece of 1/4" MDF and I spray-painted it. In order to get the PCBs to hold in place in the case, I used a sheet of lego that I glued on the bottom. Then I made the PCB hold in place with legos.



I had a hard time cutting the holes needed for the terminal board, the LCD and the RJ45 connector. I used a drill and made several holes until I get this big ugly rectangle-like hole. For the LCD, it wasn't easy aligning everything. Next time I will use a CAD and prototype on cardboard first (thanks to ESawDust for the idea).

The rPI takes care of the PGM using its GPIO. It communicates to the AVR chip using the SPI bus. The AVR chip is only used to drive the LCD and the LEDs on the front panel. The terminal board is used to provide 12V and video connections for my cameras. There is also 4 terminal used to connect my alarm PGMs (or any other relays, like a motion sensor or a device to monitor my sump-pump)

The expansion board I did is not very complicated. It is only used for:

  • Distribute +5 and +12v over some headers
  • Breakout the terminal board on a 2row header
  • Add a RS232-to-TTL converter to and RS232 functionality to the rPi
  • Add a ATMega1284P to drive some LED and a LCD display on the front panel.

The software

Third party libraries

I had to use the bcm2835 library that was written for interfacing the GPIO in C. For this reason, I had to licence my code under GPL. When I get some time, I will write my own and use it instead.I hate GPL. I'd rather just give my code out for free with no strings attached. That's what I usually do. But at least that library is well written and very easy to build and use.

The SIP portion of my software uses resiprocate. I had to cross-compile resiprocate for the ARM platform. There was a couple of hiccups but it compiled fine. I had to do a couple of tweaks in the source code and in the makefiles to make it build but nothing too big. Basically what was missing is some references to the include file "cstddef". I included the file in all failing sources. I also had to modify the Makefile so it would link against librt too. I've seen bigger problems on other software builds. I am linking statically to resiprocate so I didn't need to copy anything on the rPI.

I was able to compile LUA for the rPI with "make linux CC=arm-unknown-linux-gnueabi-gcc" Actualy that command failed. But it doesn't matter because it failed after it was able to create the static library file. So I copied all the header files and the static library in my source folder and I used it that way.

For ortp (the RTP library), I compiled statically but I deployed it in my toolchain's library folders.

CC=arm-unknown-linux-gnueabi-gcc ./configure --target=arm-unknown-linux-gnueabi --host=x86_64-unknown-linux-gnu --prefix=/opt/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi --enable-static
make
make install

I am using Berkeley DB as a storage mechanism for persistant script variables and events. I downloaded db-5.3.21.NC.tar.gz.

cd buil_unix
../dist/configure -host=arm-unknown-linux-gnueabi CC=arm-unknown-linux-gnueabi-gcc RANLIB=arm-unknown-linux-gnueabi-ranlib STRIP=arm-unknown-linux-gnueabi-strip AR=arm-unknown-linux-gnueabi-ar -enable-smallbuild -prefix=/opt/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi CFLAGS="-Os"
make
make install

Software architecture

The rPI board runs a home-made server application. The server contains 4 modules that can talk to each other:

  • VOIP Service. Used to place calls when a module has an event to report or to receive calls to control one of the other modules.
  • PGM Service. Used to monitor triggers such as PGM on an alarm system, sumpump events or a door sensor event.
  • Insteon Service. Used to report events received by an insteon modem or to send a command to the modem.
  • REST Interface. Used for invoking commands on different modules.
  • SMTP server. Used to get emails for events generated by Paradox IP100 module.
  • Web Service. Used to get REST requests over HTTP
  • Event Processor. Used to receive JSON formatted events from different modules.

I am planning on adding another module eventually to interface my to IR cameras.

REST API

/alarm/getlogs
Retrieve all alarm logs
/audio/play
plays files defined by PLAY_STRING on onboard sound device
soundComa separated list of files to play. digits can be used. They will be decoded and the proper sound files will be constructed. I.e: sound1,29,4,sound2 would play sound files: sound1.wav,20.wav,9.wav,4.wav,sound2.wav Note that number reconstruction only work for french grammar and only supports numbers -59..59 inclusively. For playing silence, you can use the number of seconds prefixed by a dollar sign. I.e: sound1.wav,$4,sound2.wav. This would make a 4 second pause between sound1 and sound2
/events/add
add a scheduled event. This event will trigger the LUA script at the defined time
hourhour of the day at which to trigger the event
minminute of the hour at which to trigger the event
nameEvent name
puser defined data
/events/gettime
get current time
/events/remove
remove a scheduled event
idevent ID
/events/show
show all scheduled events
/help
display API documentation
/insteon/addezflora
add a Insteon ezflora module definition
idthe Insteon device ID formated as 0xNNNNNN
nameThe name of the module
/insteon/addiolinc
add a Insteon iolinc module definition
idthe Insteon device ID formated as 0xNNNNNN
nameThe name of the module
/insteon/addmodule
add a Insteon module definition
idthe Insteon device ID formated as 0xNNNNNN
nameThe name of the module
/insteon/clearmodules
reset list of Insteon module definition
/insteon/ezflora/setprogram
Sets a program on the EZFlora
idthe Insteon device ID formated as 0xNNNNNN
pProgram number. 1 to 4
z1Zone 1 timer. 0 to 255 minutes
z2Zone 2 timer. 0 to 255 minutes
z3Zone 3 timer. 0 to 255 minutes
z4Zone 4 timer. 0 to 255 minutes
z5Zone 5 timer. 0 to 255 minutes
z6Zone 6 timer. 0 to 255 minutes
z7Zone 7 timer. 0 to 255 minutes
z8Zone 8 timer. 0 to 255 minutes
/insteon/ezflora/status
Forces an update of EZFlora status
idthe Insteon device ID formated as 0xNNNNNN
/insteon/listmodules
lists all Insteon modules
/insteon/raw
Send raw insteon command
cmd1command byte 1
cmd2command byte 2
d1data byte 1 for extended data
d10data byte 10 for extended data
d11data byte 11 for extended data
d12data byte 12 for extended data
d13data byte 13 for extended data
d14data byte 14 for extended data
d2data byte 2 for extended data
d3data byte 3 for extended data
d4data byte 4 for extended data
d5data byte 5 for extended data
d6data byte 6 for extended data
d7data byte 7 for extended data
d8data byte 8 for extended data
d9data byte 9 for extended data
idthe Insteon device ID formated as 0xNNNNNN
/insteon/refreshalllinksdatabase
retrieve all-link database
/insteon/setcontroller
set Insteon controller ID (PLM)
idthe Insteon device ID formated as 0xNNNNNN
/insteon/switch
Turn on or off a device
actionon/off/toggle
idthe Insteon device ID formated as 0xNNNNNN
level0 to 255. Irrelevant if action is off or toggle
rate0 to 255. This is the ramp rate
subdevfor EZFlora, subdev is 1-7 for valves and 8-11 for programs 1-4. For switches, this is irrelevant
/panel/lcd
Set text on DHAS panel LCD
rowrow index (0 or 1)
strstring to display on LCD
/panel/lcd/reset
reset DHAS panel LCD
/panel/led
set LED status on DHAS panel
actionon/off/blink
idled index
timeduration of LED on or blink. If not provided, 255 will be assumed
/pgm/getlogs
retrieve full list of logged PGM events. if pgm number is given, only logs for that PGM will be returned.
pgmpgm number [0-3]. Optional
/pgm/querypgm
retrieve current status of pgm
pgmpgm number [0-3].
/phone/blf
Will subscribe for presence events for the given extension. The extension must be a known extension in the subscribe context of our UA (if using Asterisk).
extextension
/phone/call
Will call the given extension and optionally play sound when the remote peer answers the call. Placing a call only works if the user agent was previously registered. Called extension must be know by the proxy because direct URI are not supported. To make an intercom call (where the UAS will autoanswer) this needs to be configured on the proxy.
extextention to call
playComa separated list of files to play. digits can be used. They will be decoded and the proper sound files will be constructed. I.e: sound1,29,4,sound2 would play sound files: sound1.wav,20.wav,9.wav,4.wav,sound2.wav Note that number reconstruction only work for french grammar and only supports numbers -59..59 inclusively. For playing silence, you can use the number of seconds prefixed by a dollar sign. I.e: sound1.wav,$4,sound2.wav. This would make a 4 second pause between sound1 and sound2
/phone/play
Play sounds on an active call using given callID.
idcall ID
releaseaftersounds[true/false] if you want the call to be released after sound finished playing
soundComa separated list of files to play. digits can be used. They will be decoded and the proper sound files will be constructed. I.e: sound1,29,4,sound2 would play sound files: sound1.wav,20.wav,9.wav,4.wav,sound2.wav Note that number reconstruction only work for french grammar and only supports numbers -59..59 inclusively. For playing silence, you can use the number of seconds prefixed by a dollar sign. I.e: sound1.wav,$4,sound2.wav. This would make a 4 second pause between sound1 and sound2
/phone/register
Will register the phone service user agent to the given PBX. This is usually done during initialization
pinpin associated to user
proxyPBX IP address
userSIP user to register
/phone/release
release a call using call ID (usually provided in call events)
idCall ID
/phone/showblf
Get the list of active subscriptions to presence events in the system
/phone/showcalls
Get the list of active calls in the system
/thermostat/getstats
retrieve stats
/thermostat/setip
set IP of thermostat
ipIP address of thermostat
/thermostat/setmode
set operating mode
modecool/heat
/thermostat/settemperature
Set thermostat setpoint (farenheit) for given mode
modecool/heat
ttemperature in fareinheit (integer)
/weather/temperature
get current outdoor temperature

Events

When a module needs to trigger and event, it sends a JSON formatted message to the EventProcessor. The event processor then executes a function in a Lua script. The Lua script has a handler that will receive the JSON message. It is up to the user to write his own Lua script to trigger any desired actions upon receiving an event. The script can trigger actions by initiating a RESTful request to the application. The script needs to have one function defined: onEvent(str) which will be called by the EventProcessor.

Timer event that gets triggered every minute of the hour.
{
    "event":"timer",
    "timestamp":"TIMESTAMP"
}
Sun rises in Ottawa, Canada. This will occur at a minute boundary
{
    "event":"sunrise",
    "timestamp":"TIMESTAMP"
}

Sun sets in Ottawa Canada. This will occur at a minute boundary
{
    "event":"sunset",
    "timestamp":"TIMESTAMP"
}

Insteon event
{
    "event":"insteon",
    "id":"device ID",
    "name":"device name",
    "trigger":"unsolicited/ack", ; unsolicited event or ack of a direct message
    "type":"switch", ; only switch for now
    "value":"0..255"
}

PGM state changed
{
    "event":"pgm",
    "pgm":"PGM number", ; 1..4
    "status":"0/1"

}

Phone module event when a call comes it, digits are sent, call is released
and call initiated.
{
    "event":"call",
    "call": {
        "from":"caller"
        "to": "called extension"
        "id": "SIP call ID"
    }
    "callevent":{
        "type":"answered/released/digit"
        "dir": "incoming/outgoing"
        "digit":"digit pressed"
    }
}

SIP Presence events
{
    "event":"presence",
    "device":"SIP device",
    "status":"idle/busy/terminated"
}

IP100 module Alarm events
{
    "event":"alarmstate",
    "time":"time stamp",
    "status":"Armed or Disarmed or Alarm",
    "from":"user that generated the event"
}
User defined scheduled event
{
    "event":"scheduledevent",
    "id":"id"
    "name":"name of event"
    "param":"user defined parameter"
    "min":"minute that event was triggered"
    "hour":"hour that it was triggered"
    "recurent":"true if event will stay in system. false if event will be deleted after being triggered"
}

EZFlora turned off water
{
    "event":"wateroff",
    "id":"ezflora id",
    "name":"device name",
    "type":"1",
    "meter":LITERS_USED, // this is the number of liters used since last start of program or valve
}

EZFlora turned on a valve
{
    "event":"wateron",
    "id":"ezflora id",
    "name":"device name",
    "type":"1",
    "meter":LITERS_USED, // this is the number of liters used since last start of program or valve
    "previousvalve":VALVE_NUMBER: // if another valve was on before, this is the one
    "valve":CURRENT_OPENED_VALVE
}


VOIP module

The phone service will not allow any calls to be placed unless the user agent is sucessfully registered. This is done with the REST command /phone/register?user=&pin=&proxy=. The phone service will accept calls for any users (extension) at its address. It will answer all calls and will trigger an event containing the called extension and all DTMF tones being sent its way.

Insteon

The protocol is a bit difficult to use. I'm not sure why they made so complicated. For example, when sending a "status request" command, we get a response back but we don't get the command number back in the response. So it is hard to know what that response is for. So we have to remember that we were waiting for a response like that. It's diffucult to make a nice asynchronous parser. I can think of a thousand ways to make that protocol easier to use.

When sending an insteon command, you need to wait to get the echo back with a NAK or ACK. If you get an ACK and you are expecting to get a response from a device after issuing that command, you need to wait for the response. Things cannot be done in a trully async manner. If you send a whole bunch of commands and expect to get a whole bunch of responses after, it won't work. So before issuing a command, you need to make sure you got the expected response of a previous command (or use a timeout). At least it works that way for "Status request". If you send 10 "status request" in a row without waiting for the reponse of each, then you won't get all responses. At least that's my experience, I don't remember seeing that in the documentation. The only problem with the way I'm doing things is if I get an unsolicited event (someone opened a light manually) between the echo of my command and the response I was expecting. I am assuming that the PLM will handle such a case for me and send me the events in the right order. Otherwise, the protocol has a major flaw in my opinion.

Script

This is where all the power is at. The server exposes an interface to receive events and trigger actions. The server does not care about which action should occur when an event occurs on a module. This is handled by a Lua script. So with a script like this, I can tell my device to do many things upon receiving an event. I can make a script that says "when you receive a call from device X and he sends digits 1234, turnon the bedroom light" or "when PGM 2 gets activated (by motion sensor in staircase), play sound alarm.wav on rPI analog output" or "at 7PM, turn on outside light" or "when PGM 1 is deactivated (alarm system disarmed), send a call in living room and play greeting.wav". Here is an example of a Lua script for a system:

initiateAction("/insteon/setcontroller?id=0x123456");
initiateAction("/insteon/clearmodules");
initiateAction("/insteon/addmodule?name=bedroom&id=0x7F1FBA");
initiateAction("/phone/register?user=username&pin=password&proxy=172.2.88.8")

function onEvent(st)
    JSON = (loadfile "JSON.lua")()
    local v = JSON:decode(st)
    if v["event"]=="pgm" and v["pgm"]=="1" and v["status"]=="0" then
        initiateAction("/insteon/switch?id=0x1FAA22&action=on")
    end
end

The first lines in the global scope will be executed when the server starts or whenever the script is reloaded. This is a good place to do initialization. In my example, it is where I initialize my insteon devices. These lines are necessary since the server needs to know about the insteon devices. Then there is the onEvent handler. This function will be called whenever an event occurs on a system module. This is where the power of Lua comes into play because I can apply any logic I want using the Lua script. The script will always stay loaded so if a event occurs and the script sets a global variable, this variable will still hold the same value upon receiving the next event. This is true only until the script is reloaded though. Since all global variable values are lost after a script is reloaded, 2 functions are provided to the Lua scripts: getVar(key) and setVar(key,value). These two functions provide peristant storage of values even when the server is restarted.

My server listens for SIGHUP. So whenever the script is modified, issuing "killall -HUP homeautomation" will reload the script

System configuration

This is how I use my device

Alarm system

My alarm system allows me, with the use of a jumper on the board, to specify if I want the PGM to source 12V or sink to ground. I configured it to sink to ground. That way, the PGM will act like a switch. When PGM is activated, it will be connected to ground, when deactivated it will be disconnected. So with a pull-up resistor to the 3.3V pin of the rPI, the GPIO on the rPI will get 0 or 3.3 V depending on the status of the PGM. I configured one of my PGM to activate/deactivate following the arm status of a partition. The second PGM is configured to activate when any of the zones are in alarm and deactive when any of the zone alarms are restored.

Insteon

I currently have 4 light switches. 1 in my bedroom, that turns on in the morning when I get my wakeup call. I have another one for the outside light at the front door. This one gets turned on based on a schedule (sunset until midnight). Another one in the living room gets turned on when the alarm system gets deactivated between sunset and sunrise. The other one is in my basement and has no special purpose.

I use my EZFlora to water my two vegetable gardens at 5 in the morning. I can cancel a scheduled watering or schedule another one.

Features

Alarm status web page

I configured my alarm system to activate/deactivate a PGM (a relay on the alarm system) when we arm/disarm the system. The rPI monitors the PGM using a GPIO pin. The rPI logs all activity about the PGM in a local file. This is done by a homemade server application. The server application also listens for RESTfull queries on a TCP port. Now any computer in my house (including my tablet and smartphone) can make a request to the rPI using a webbrowser and it wil get a JSON formatted response indicating the status of the alarm system or a list of arm/disarm events with a timestamp. This is usefull to know if someone is at home while I am away or to know if the system is currently in alarm.

When I bought my alarm system, I got an IP module that lets me receive events and can let me configure the system. The module isn't that great though. The only events I can receive are email notification of who armed/disarmed the system or which zone is in alarm. I'd like to get more events but I guess I'll have to live with that. The fact that I can only get email notifications also annoys me. I have to specifiy the SMTP server but it absolutely needs to be a domain name with a MX record (no IP address). So I had to hack the following into place:

  • Install bind on the rPi and create a zone with an MX record.
  • Make a fake SMTP server that receives the emails and parses them.

The SMTP server is a module in my homeautomation system instead of a stand alone SMTP server. It is just a module that listens on port 25. The DNS server (bind) has a zone that resolves an MX record to the address of my home automation system. The alarm system IP module has been configured to use that DNS server. When the SMTP server module receives an email, it parses it and trigger to appropriate event It is a complicated setup and it is a shame that I need to use it that way. But now, at least, I have a way to know who armed/disarmed the alarm system and when it occured.

I am currently using the alarm system PGM to know if the system is armed or disarmed. This is very reliable because if an event occurs while my software is down, I can always check the PGM status on load. But I also use the SMTP server to get notifications of who armed and disarmed the system to trigger a welcome call on the phone close to the front door. The welcome call greets me and tells me how many voicemails I got while I left. The voicemail announcement depends on who disarms the alarm, because we each have our own voicemail box.

Phone System

Since I run an Asterisk server at home, all my phones are IP phones. With this in mind, I decided to leverage that in order to call my device and send it DTMF tones to control different actions. So with a phone in the house, I can turn on/off lights and trigger any other actions that my device is capable of. I also have an Aastra phone that has some buttons I can assign XML web pages on. When pressing a button, it fetches an XML document at a URL of my choice and displays the content on the phone. I don't really care about the XML. With these buttons, I execute some REST commands on my device to turn on/off my bedroom lights and setup a wakeup call.

Download

You will not be able to compile the source code because I didn't include any of the GPL stuff. But you couldn't use it any way if you don't have my device. So here is the source code and I hope that at least bits of it will help for other projects. dhas.tar



Sump pump monitoringLast edited on Feb 20, 2013

Goal

When I built my Home Automation System, I wanted a way to monitor my sump pump. There are many ways of doing this. I chose to use a current sensor to detect when the pump is on or off. Since the pump will not draw any current when it does nothing and draw about 3.5A when it is pumping, it is very easy to know when the pump is on or off whith a current sensor. I used the CR9321-npn sensor.

By running the hot wire of my sump pump inside the sensor's hole, the sensor will detect if current is running through. I won't go into the details of this but it works with induction current. The nice thing about this sensor is that it has a transistor inside. So instead of giving you an analog output, it gives you a dry contact that is on or off depending if more than 350mA runs in the wire. So it is easy for me to attach this to my home automation system. The home automation system looks at the device just as if it were a door contact or any other dry contact.

Device




Since I didn't wanna pull apart my sump pump, I made an outlet in which I can plug the sump pump in. The outlet has a power cord and plugs into any other outlet. My special outlet has a terminal block on it that breaks out the sensor's red and black wire. I didn't take any pictures of the inside of the box but it basically looks something like this:


And here is what I do with the data. I know it is a bit overkill to make those statistics but it's always fun to observe a trend with any data sets. This is a snapshot of my status page on my intranet site.




Building a hosted mailserver serviceLast edited on Sep 22, 2012

I guess that with the new buzzwords today we would call this "cloud service" or mailbox in the cloud. What it is really: a service similar to any big providers like GoDaddy or DomainPeople that hosts a mail server for you and let you create email addresses and mailboxes hosted on their servers. Except I wanted to do it myself and offer this type of service to other people. Well not really... I just wanted to create the service but I don't want any customers. It's just a proof of concept. So my server will host mailboxes and email forwarding services for people who already own a domain name and want to have their MX record point to my server. My server also offers a web page to let users manage their email addresses. Oh, and this is all done on an Amazon EC2 virtual machine. So guaranteed uptime.

So in another post (I am now hosting my own DNS and Mail servers on Amazon EC2) , I talked about installing postfix on an EC2 VM. Now I wanna offer my users a way to create mailboxes and forward addresses by themselves instead of having to call me every time to manage their domain. As I said in the other post, a mysql server integration with postfix is probably a good idea for this type of setup. First thing to do is to install the mysql server. I won't go into details here as this is widely covered on the internet. I did a search for "ec2 mysql" to see if there was any special considerations to take since it was my first installation on a non-slackware distribution and I found this: http://www.samstarling.co.uk/2010/10/installing-mysql-on-an-ec2-micro-instance/

After you have a functional mysql server that will automatically run after a reboot, you should create a database for the mail config and create a mysql user that has access to it.

in main.cf:
    virtual_mailbox_domains = mysql:/etc/postfix/vdomains-mysql
    virtual_mailbox_maps = mysql:/etc/postfix/vmbox-mysql
    virtual_alias_maps = mysql:/etc/postfix/valias-mysql
in mysql:
    create database dbmail;
    create user 'mysqlmailuser'@'localhost' IDENTIFIED BY 'pass123';
    grant all privileges on dbmail.* to 'mysqlmailuser'@'localhost' with grant option;
    create table owners
    (
        id INT NOT NULL AUTO_INCREMENT,
        owner VARCHAR(255),
        pass VARCHAR(255),
        PRIMARY KEY(id)
    );

    create table domains
    (
        domainkey VARCHAR(255),
        domain VARCHAR(255),
        owner INT not null,
        FOREIGN KEY(owner) REFERENCES owners(id) ON DELETE CASCADE
    );

    create table aliases
    (
        alias VARCHAR(255),
        fwd VARCHAR(255)
    );

    create table mailbox
    (
        username VARCHAR(255),
        mbox VARCHAR(255)
    );
/etc/postfix/vdomains-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT domain FROM domains WHERE domainkey='%s'

/etc/postfix/valias-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT fwd FROM aliases WHERE alias='%s'

/etc/postfix/vmbox-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT mbox FROM mailbox WHERE username='%s'

I'm not sure why, but when adding/removing an entry in the "domain" table, changes don't get picked up instantly by postfix. You don't need to "postfix reload" but it could take a minute or two. Changes to the alias and mailbox tables are reflected right away though.

Letting users do it themselves

I wanted to provide a way of having different user accounts on the server. Each user owns a different subset of domains that are registered somewhere on some registrar. The users would have set the MX record of their domain to point towards my server. Or they could use my name server and I could setup their MX record but that is a different story. I just need an automated way for the users to add their domain in the list of domains for which postfix accepts email from. I also want users to create their aliases/forwards and mailboxes. This will be done through a web page on which each users will have to login. After they are logged on, a list of their domains will appear and they have the choice of adding/remove domains. When they will click on a domain, they will be able to manage their mailboxes and forwards.

So all this web application does is to add/remove domains, aliases and mailboxes from the mysql tables that postfix uses. Since the changes are in a mysql server, the changes will be immediately available to postfix, giving instant control to the users on their virtual mail server.

Note that this code does not produce a very nice web page layout, it just does the work. A bit more a validation could also be done to improve robustness. It is more of a proof of concept than a real design. Here is the code: ww.zip



I am now hosting my own DNS and Mail servers on Amazon EC2Last edited on Sep 22, 2012


Using an Amazon EC2 virtual machine, I wanted to host my own domain by doing as much as I could myself. By that, I mean that I wanted to host my own DNS server, mail server and web server. So after registering a new domain name, the first thing I did was to change the nameserver of that domain to use my own server instead of the registrar's. After that, all the work is done on my EC2 machine. Note that for this to work, I needed to unblock TCP port 25 and 53 and also UDP port 53 on my virtual server. This can be done in the EC2 management console under the "security groups" section.

Configuring the DNS server

First thing to do is to add the new zone definition in /etc/named.conf. Let's assume that I registered "exampledomain.ca".

zone "exampledomain.ca" IN {
        type master;
        file "exampledomain.ca";
};

Then I need to define the zone in /var/named/exampledomain.ca. I wanted to create a subdomain called "www" so that my web server could be accessible from www.exampledomain.ca. I needed an MX record that points to the mail server that is also hosted on the same server as the DNS server. For this example, let's assume that the public IP address of my server is 1.2.3.4.

$TTL    300
@           IN SOA  exampledomain.ca. www.exampledomain.ca. (
                    2012220104      ;
                    1H      ; refresh
                    15M     ; retry
                    1W      ; expiry
                    1D )        ; minimum

            IN NS   ns.
            IN A    1.2.3.4
            IN MX 10 mail

ns              IN A 1.2.3.4
www             IN A 1.2.3.4
mail            IN A 1.2.3.4

Mail server

after looking for a solution to host multiple domains with virtual mailboxes that are not associated with a unix account, I discovered that that postix might be better for the job than sendmail. after installing postfix and setting it to run automatically at boot time, we need to configure the /etc/postfix/main.cf file. I am only telling it to listen to all network interface and where to look for the list of domains we are hosting and the mailbox definition.

The other nice thing about postfix VS sendmail is that you can easily add more mailboxes/forwards and domains without having to restart postfix. This is very usefull if you wanna make a webpage on which users can create their own mailboxes and add their own domains. But the nicest thing of all is that you can use a mysql server (or several other kinds of DB) so that your web site can configure the server more easily. But for now, I will stick with scripts. Here is an example of my main.cf script:

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
mail_owner = postfix

# listen to all interfaces so we can accept incomming connection from LAN and WAN.
inet_interfaces = all
inet_protocols = all
mydestination = $myhostname, localhost.$mydomain, localhost
unknown_local_recipient_reject_code = 550
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
debug_peer_level = 2
debugger_command =
        PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
        ddd $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.6.6/samples
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES

# mailboxes will be saved in /var/vhosts
virtual_mailbox_base = /var/vhosts

#/etc/postfix/vmbox will contain the list of mailboxes
virtual_mailbox_maps = hash:/etc/postfix/vmbox

#/etc/postfix/vdomains will contain the list of domains we are accepting mail for.
virtual_mailbox_domains = hash:/etc/postfix/vdomains
virtual_alias_maps = hash:/etc/postfix/valias
# The /var/vhosts folder must be writeable by user 4000 and group 4000
virtual_uid_maps = static:4000
virtual_gid_maps = static:4000

A note on postifx lookup tables. Postifx is great because it lets use use different type of lookup tables. In the example posted above, I use hash:/etc/postfix/valias. This is a reference to a Berkely DB file. You need to create a text file with key/pair values separated by a space and generate a DB with the "postmap" command in order to use this. But if you want to use another table type such as a mysql table, then you could specify mysql:/etc/postfix/valias. But in that case, the "valias" file's content would be different. It would contain information on how to access the database table.

More information on different types of tables can be found here: http://www.postfix.org/DATABASE_README.html

The virtual_uid_maps and virtual_gid_maps lines indicate that the mailboxes inside /var/vhosts will be owned by 4000:4000 on the system. This means that you have to create a user with ID 4000 and change ownership of the base folder to that user:group

useradd virtualmailbox -u 4000
chown virtualmailbox:virtualmailbox /var/vhosts

The virtual_mailbox_domains line indicates that postfix needs to look into that file to find the list of domains it is accepting mail for. Everytime that you modify the vdomains of vmbox file, you need to rebuild the hash database by issuing the "postmap /etc/postfix/vdomains" command. Of course, you would replace "vdomains" with "vmbox" if you need to rebuild the vmbox database. virtual mailboxes are files that will be stored on the server. But if you need to forward to another address, then you will need aliases. I'm not sure why, but I discovered that you should not use "virtual_alias_domains" if you want to use mailboxes and aliases for the same domains. Only list the domains in "virtual_mailbox_maps".

# This file will be compiled by the "postmap" tool. It needs
# to contain 2 columns on each line. Since this is just a list
# of domains, we only need one column but the tool will expect 2.
# So you can put anything you want in the second column. I decided
# to put the domain name a second time.
#
exampledomain.ca exampledomain.ca

vmbox file

#because there is a slash at the end of the path, mail will be saved in maildir format
admin@exampledomain.ca  exampledomain.ca/admin/

#because there is no slash at the end of the path, mail will be saved in linux mbox format
test@exampledomain.ca   exampledomain.ca/test

valias file

user1@exampledomain.ca  user1@gmail.com
user2@exampledomain.ca  user2@yahoo.com

Automation

It could become very handy to automate the process of adding domains and user mailbox though a perl of php script. The best way to do it would be by using a database, but let's see how to do it with files first.

Files

  • Add the domain name in /etc/postfix/vdomains
  • Add mailbox in /etc/postfix/vmbox
  • Add forwarding addresses in /etc/postfix/valias
  • Run "postmap /etc/postfix/vdomains"
  • Run "postmap /etc/postfix/vmbox"
  • Run "postmap /etc/postfix/valias"

Of course, this requires that the script locks the files to prevent two process from accessing the same files at the same time. And also, it would be better to schedule the "postmap" instead or running it everytime because if you have a large system where users create several mailboxes in 1 minute, you would want to wait a little bit before running postmap oherwise you would add a great load on your CPU.

Database

With this method, your scripts don't need to lock any files and you don't need to schedule a "postmap". All you need to do is to insert a mailbox/domain in the database and the change is instantaneous. In my opinion, this is the ideal way of doing thing if you have a server that hosts several domains and you allow users to create their own mailboxes. For an example on how to do this, you can refer to this post I did: Building a hosted mailserver service



Deploying a layer3 switch on my networkLast edited on Aug 20, 2012

Lately, I bought a Cisco 3550 switch with 48 ports. My goal was to have a switch that could do the inter-vlan routing instead of relying on the router. This way, if my router goes down, I only lose my internet connection but the voice network and data network on my LAN can still function properly. I was also using the cisco router as my DHCP server, but now it is running on my switch.

The first thing that needs to be configured is the SVI interface. SVI stands for Switch Virtual Interface. It represents a interface to a vlan. You need one SVI for each VLAN that you want to provide routing on. By creating an SVI, you assign it an IP address. This IP address is the address that you will use as the default gateway on all nodes on your VLAN.

! create SVI for VLAN 1
interface Vlan1
 ip address 192.168.1.1 255.255.255.0

! create SVI for VLAN 2
interface Vlan2
 ip address 192.168.2.1 255.255.255.0

Since I wanna host my DHCP server on the switch now instead of the router, I can take the same configuration I had on the router at apply it on the switch.

ip dhcp excluded-address 192.168.1.1 192.168.1.99
ip dhcp excluded-address 192.168.1.150 192.168.1.255
!
ip dhcp pool pool_vlan1
   import all
   network 192.168.1.0 255.255.255.0
   default-router 192.168.1.1
   dns-server 192.168.254.1
!
ip dhcp pool pool_vlan2
   import all
   network 192.168.2.0 255.255.255.0
   default-router 192.168.2.1
   dns-server 192.168.254.1

Notice that the DNS server (forwarder) is on the router and not on the switch. Usually, the DNS server in my setups was on the same device as the subnet's gateway. So on all the devices that were assigned a static IP on my network, I had to change the IP address of the DNS server.

Connecting the router and the switch

There are two ways of doing this: SVI or routed port.

SVI

With this method, I would create a VLAN 10 on my switch for subnet 192.168.10.0/24. I would assign an IP address from that subnet to my router's LAN interface and connect it to an access port that is part of VLAN 10 on the switch.

Routed port

A routed port is a port that behaves like a port on a router. It handles layer 3 protocols. Using this method, you need to create a small network between your router and the switch. You can set the port to become a router port by using the "no switchport" command when configuring your interface. Then, you assign an IP address to that interface. This small network only has two members in it, so it is a /30 subnet. I used 192.168.254/30 and used two IPs in there. So basically, with a routed port, we are creating a peer-to-peer network that will bridge the networks known by the router and the ones known by the router. On the switch, I would configure the routed port like this:

interface fast 0/1
 no switchport
 ip address 192.168.254.2 255.255.255.252

Either way, I need to create routes on the router because the router will not be aware of the other subnets on my switch. It will only be aware of the peer-to-peer network (if using a routed port) or the subnet for the vlan it is part of (if using SVI). But it will not know anything about the other subnets in other vlans. At first, I thought about assigning the router's LAN port an IP address in the VLAN1 subnet and connect it to an access port for VLAN1 on the switch. But then, since I have an ACL that prevents VLAN2 from talking to VLAN1, the VLAN2 wouldn't have access to the router, so no internet for VLAN2. So I thought about creating a VLAN 10 in which my router would be part of, in a new subnet 192.168.254.0/30. Only the SVI on the switch and the physical port on the router would belong to that subnet. But why do this when a routed port does exactly that anyway? So I chose to go with the routed port even though it does the same thing as the SVI method

Routes

The router only know about network 192.168.254.0/30. But because we are doing TCP/UDP port forwarding, it needs to know about 192.168.1.0, 192.168.2.0, 192.168.3.0 etc... So I tell the router: Whatever you have for 192.168.0.0/16, throw it on the FastEthernet0/0 interface. Whatever else you got, ship it on the WAN (my DSL modem). This is done like this:

ip route 192.168.0.0 255.255.0.0 Fast0/0
ip route 0.0.0.0 0.0.0.0 Dialer1

For the switch, things are really simple. The switch knows how to route between VLANs because we have created an SVI for each VLAN. We just need to tell it to send whatever does not belong to a known subnet to the router, through the routed port:

ip route 0.0.0.0 0.0.0.0 Fast0/1


Pages:123456789101112