WWW.DUMAIS.IO

Building a hosted mailserver serviceLast edited on Sep 22, 2012

I guess that with the new buzzwords today we would call this "cloud service" or mailbox in the cloud. What it is really: a service similar to any big providers like GoDaddy or DomainPeople that hosts a mail server for you and let you create email addresses and mailboxes hosted on their servers. Except I wanted to do it myself and offer this type of service to other people. Well not really... I just wanted to create the service but I don't want any customers. It's just a proof of concept. So my server will host mailboxes and email forwarding services for people who already own a domain name and want to have their MX record point to my server. My server also offers a web page to let users manage their email addresses. Oh, and this is all done on an Amazon EC2 virtual machine. So guaranteed uptime.

So in another post (I am now hosting my own DNS and Mail servers on Amazon EC2) , I talked about installing postfix on an EC2 VM. Now I wanna offer my users a way to create mailboxes and forward addresses by themselves instead of having to call me every time to manage their domain. As I said in the other post, a mysql server integration with postfix is probably a good idea for this type of setup. First thing to do is to install the mysql server. I won't go into details here as this is widely covered on the internet. I did a search for "ec2 mysql" to see if there was any special considerations to take since it was my first installation on a non-slackware distribution and I found this: http://www.samstarling.co.uk/2010/10/installing-mysql-on-an-ec2-micro-instance/

After you have a functional mysql server that will automatically run after a reboot, you should create a database for the mail config and create a mysql user that has access to it.

in main.cf:
    virtual_mailbox_domains = mysql:/etc/postfix/vdomains-mysql
    virtual_mailbox_maps = mysql:/etc/postfix/vmbox-mysql
    virtual_alias_maps = mysql:/etc/postfix/valias-mysql
in mysql:
    create database dbmail;
    create user 'mysqlmailuser'@'localhost' IDENTIFIED BY 'pass123';
    grant all privileges on dbmail.* to 'mysqlmailuser'@'localhost' with grant option;
    create table owners
    (
        id INT NOT NULL AUTO_INCREMENT,
        owner VARCHAR(255),
        pass VARCHAR(255),
        PRIMARY KEY(id)
    );

    create table domains
    (
        domainkey VARCHAR(255),
        domain VARCHAR(255),
        owner INT not null,
        FOREIGN KEY(owner) REFERENCES owners(id) ON DELETE CASCADE
    );

    create table aliases
    (
        alias VARCHAR(255),
        fwd VARCHAR(255)
    );

    create table mailbox
    (
        username VARCHAR(255),
        mbox VARCHAR(255)
    );
/etc/postfix/vdomains-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT domain FROM domains WHERE domainkey='%s'

/etc/postfix/valias-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT fwd FROM aliases WHERE alias='%s'

/etc/postfix/vmbox-mysql
    user = mysqlmailuser
    password = pass123
    dbname = dbmail
    query = SELECT mbox FROM mailbox WHERE username='%s'

I'm not sure why, but when adding/removing an entry in the "domain" table, changes don't get picked up instantly by postfix. You don't need to "postfix reload" but it could take a minute or two. Changes to the alias and mailbox tables are reflected right away though.

Letting users do it themselves

I wanted to provide a way of having different user accounts on the server. Each user owns a different subset of domains that are registered somewhere on some registrar. The users would have set the MX record of their domain to point towards my server. Or they could use my name server and I could setup their MX record but that is a different story. I just need an automated way for the users to add their domain in the list of domains for which postfix accepts email from. I also want users to create their aliases/forwards and mailboxes. This will be done through a web page on which each users will have to login. After they are logged on, a list of their domains will appear and they have the choice of adding/remove domains. When they will click on a domain, they will be able to manage their mailboxes and forwards.

So all this web application does is to add/remove domains, aliases and mailboxes from the mysql tables that postfix uses. Since the changes are in a mysql server, the changes will be immediately available to postfix, giving instant control to the users on their virtual mail server.

Note that this code does not produce a very nice web page layout, it just does the work. A bit more a validation could also be done to improve robustness. It is more of a proof of concept than a real design. Here is the code: ww.zip



I am now hosting my own DNS and Mail servers on Amazon EC2Last edited on Sep 22, 2012


Using an Amazon EC2 virtual machine, I wanted to host my own domain by doing as much as I could myself. By that, I mean that I wanted to host my own DNS server, mail server and web server. So after registering a new domain name, the first thing I did was to change the nameserver of that domain to use my own server instead of the registrar's. After that, all the work is done on my EC2 machine. Note that for this to work, I needed to unblock TCP port 25 and 53 and also UDP port 53 on my virtual server. This can be done in the EC2 management console under the "security groups" section.

Configuring the DNS server

First thing to do is to add the new zone definition in /etc/named.conf. Let's assume that I registered "exampledomain.ca".

zone "exampledomain.ca" IN {
        type master;
        file "exampledomain.ca";
};

Then I need to define the zone in /var/named/exampledomain.ca. I wanted to create a subdomain called "www" so that my web server could be accessible from www.exampledomain.ca. I needed an MX record that points to the mail server that is also hosted on the same server as the DNS server. For this example, let's assume that the public IP address of my server is 1.2.3.4.

$TTL    300
@           IN SOA  exampledomain.ca. www.exampledomain.ca. (
                    2012220104      ;
                    1H      ; refresh
                    15M     ; retry
                    1W      ; expiry
                    1D )        ; minimum

            IN NS   ns.
            IN A    1.2.3.4
            IN MX 10 mail

ns              IN A 1.2.3.4
www             IN A 1.2.3.4
mail            IN A 1.2.3.4

Mail server

after looking for a solution to host multiple domains with virtual mailboxes that are not associated with a unix account, I discovered that that postix might be better for the job than sendmail. after installing postfix and setting it to run automatically at boot time, we need to configure the /etc/postfix/main.cf file. I am only telling it to listen to all network interface and where to look for the list of domains we are hosting and the mailbox definition.

The other nice thing about postfix VS sendmail is that you can easily add more mailboxes/forwards and domains without having to restart postfix. This is very usefull if you wanna make a webpage on which users can create their own mailboxes and add their own domains. But the nicest thing of all is that you can use a mysql server (or several other kinds of DB) so that your web site can configure the server more easily. But for now, I will stick with scripts. Here is an example of my main.cf script:

queue_directory = /var/spool/postfix
command_directory = /usr/sbin
daemon_directory = /usr/libexec/postfix
data_directory = /var/lib/postfix
mail_owner = postfix

# listen to all interfaces so we can accept incomming connection from LAN and WAN.
inet_interfaces = all
inet_protocols = all
mydestination = $myhostname, localhost.$mydomain, localhost
unknown_local_recipient_reject_code = 550
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
debug_peer_level = 2
debugger_command =
        PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin
        ddd $daemon_directory/$process_name $process_id & sleep 5
sendmail_path = /usr/sbin/sendmail.postfix
newaliases_path = /usr/bin/newaliases.postfix
mailq_path = /usr/bin/mailq.postfix
setgid_group = postdrop
html_directory = no
manpage_directory = /usr/share/man
sample_directory = /usr/share/doc/postfix-2.6.6/samples
readme_directory = /usr/share/doc/postfix-2.6.6/README_FILES

# mailboxes will be saved in /var/vhosts
virtual_mailbox_base = /var/vhosts

#/etc/postfix/vmbox will contain the list of mailboxes
virtual_mailbox_maps = hash:/etc/postfix/vmbox

#/etc/postfix/vdomains will contain the list of domains we are accepting mail for.
virtual_mailbox_domains = hash:/etc/postfix/vdomains
virtual_alias_maps = hash:/etc/postfix/valias
# The /var/vhosts folder must be writeable by user 4000 and group 4000
virtual_uid_maps = static:4000
virtual_gid_maps = static:4000

A note on postifx lookup tables. Postifx is great because it lets use use different type of lookup tables. In the example posted above, I use hash:/etc/postfix/valias. This is a reference to a Berkely DB file. You need to create a text file with key/pair values separated by a space and generate a DB with the "postmap" command in order to use this. But if you want to use another table type such as a mysql table, then you could specify mysql:/etc/postfix/valias. But in that case, the "valias" file's content would be different. It would contain information on how to access the database table.

More information on different types of tables can be found here: http://www.postfix.org/DATABASE_README.html

The virtual_uid_maps and virtual_gid_maps lines indicate that the mailboxes inside /var/vhosts will be owned by 4000:4000 on the system. This means that you have to create a user with ID 4000 and change ownership of the base folder to that user:group

useradd virtualmailbox -u 4000
chown virtualmailbox:virtualmailbox /var/vhosts

The virtual_mailbox_domains line indicates that postfix needs to look into that file to find the list of domains it is accepting mail for. Everytime that you modify the vdomains of vmbox file, you need to rebuild the hash database by issuing the "postmap /etc/postfix/vdomains" command. Of course, you would replace "vdomains" with "vmbox" if you need to rebuild the vmbox database. virtual mailboxes are files that will be stored on the server. But if you need to forward to another address, then you will need aliases. I'm not sure why, but I discovered that you should not use "virtual_alias_domains" if you want to use mailboxes and aliases for the same domains. Only list the domains in "virtual_mailbox_maps".

# This file will be compiled by the "postmap" tool. It needs
# to contain 2 columns on each line. Since this is just a list
# of domains, we only need one column but the tool will expect 2.
# So you can put anything you want in the second column. I decided
# to put the domain name a second time.
#
exampledomain.ca exampledomain.ca

vmbox file

#because there is a slash at the end of the path, mail will be saved in maildir format
admin@exampledomain.ca  exampledomain.ca/admin/

#because there is no slash at the end of the path, mail will be saved in linux mbox format
test@exampledomain.ca   exampledomain.ca/test

valias file

user1@exampledomain.ca  user1@gmail.com
user2@exampledomain.ca  user2@yahoo.com

Automation

It could become very handy to automate the process of adding domains and user mailbox though a perl of php script. The best way to do it would be by using a database, but let's see how to do it with files first.

Files

  • Add the domain name in /etc/postfix/vdomains
  • Add mailbox in /etc/postfix/vmbox
  • Add forwarding addresses in /etc/postfix/valias
  • Run "postmap /etc/postfix/vdomains"
  • Run "postmap /etc/postfix/vmbox"
  • Run "postmap /etc/postfix/valias"

Of course, this requires that the script locks the files to prevent two process from accessing the same files at the same time. And also, it would be better to schedule the "postmap" instead or running it everytime because if you have a large system where users create several mailboxes in 1 minute, you would want to wait a little bit before running postmap oherwise you would add a great load on your CPU.

Database

With this method, your scripts don't need to lock any files and you don't need to schedule a "postmap". All you need to do is to insert a mailbox/domain in the database and the change is instantaneous. In my opinion, this is the ideal way of doing thing if you have a server that hosts several domains and you allow users to create their own mailboxes. For an example on how to do this, you can refer to this post I did: Building a hosted mailserver service



Deploying a layer3 switch on my networkLast edited on Aug 20, 2012

Lately, I bought a Cisco 3550 switch with 48 ports. My goal was to have a switch that could do the inter-vlan routing instead of relying on the router. This way, if my router goes down, I only lose my internet connection but the voice network and data network on my LAN can still function properly. I was also using the cisco router as my DHCP server, but now it is running on my switch.

The first thing that needs to be configured is the SVI interface. SVI stands for Switch Virtual Interface. It represents a interface to a vlan. You need one SVI for each VLAN that you want to provide routing on. By creating an SVI, you assign it an IP address. This IP address is the address that you will use as the default gateway on all nodes on your VLAN.

! create SVI for VLAN 1
interface Vlan1
 ip address 192.168.1.1 255.255.255.0

! create SVI for VLAN 2
interface Vlan2
 ip address 192.168.2.1 255.255.255.0

Since I wanna host my DHCP server on the switch now instead of the router, I can take the same configuration I had on the router at apply it on the switch.

ip dhcp excluded-address 192.168.1.1 192.168.1.99
ip dhcp excluded-address 192.168.1.150 192.168.1.255
!
ip dhcp pool pool_vlan1
   import all
   network 192.168.1.0 255.255.255.0
   default-router 192.168.1.1
   dns-server 192.168.254.1
!
ip dhcp pool pool_vlan2
   import all
   network 192.168.2.0 255.255.255.0
   default-router 192.168.2.1
   dns-server 192.168.254.1

Notice that the DNS server (forwarder) is on the router and not on the switch. Usually, the DNS server in my setups was on the same device as the subnet's gateway. So on all the devices that were assigned a static IP on my network, I had to change the IP address of the DNS server.

Connecting the router and the switch

There are two ways of doing this: SVI or routed port.

SVI

With this method, I would create a VLAN 10 on my switch for subnet 192.168.10.0/24. I would assign an IP address from that subnet to my router's LAN interface and connect it to an access port that is part of VLAN 10 on the switch.

Routed port

A routed port is a port that behaves like a port on a router. It handles layer 3 protocols. Using this method, you need to create a small network between your router and the switch. You can set the port to become a router port by using the "no switchport" command when configuring your interface. Then, you assign an IP address to that interface. This small network only has two members in it, so it is a /30 subnet. I used 192.168.254/30 and used two IPs in there. So basically, with a routed port, we are creating a peer-to-peer network that will bridge the networks known by the router and the ones known by the router. On the switch, I would configure the routed port like this:

interface fast 0/1
 no switchport
 ip address 192.168.254.2 255.255.255.252

Either way, I need to create routes on the router because the router will not be aware of the other subnets on my switch. It will only be aware of the peer-to-peer network (if using a routed port) or the subnet for the vlan it is part of (if using SVI). But it will not know anything about the other subnets in other vlans. At first, I thought about assigning the router's LAN port an IP address in the VLAN1 subnet and connect it to an access port for VLAN1 on the switch. But then, since I have an ACL that prevents VLAN2 from talking to VLAN1, the VLAN2 wouldn't have access to the router, so no internet for VLAN2. So I thought about creating a VLAN 10 in which my router would be part of, in a new subnet 192.168.254.0/30. Only the SVI on the switch and the physical port on the router would belong to that subnet. But why do this when a routed port does exactly that anyway? So I chose to go with the routed port even though it does the same thing as the SVI method

Routes

The router only know about network 192.168.254.0/30. But because we are doing TCP/UDP port forwarding, it needs to know about 192.168.1.0, 192.168.2.0, 192.168.3.0 etc... So I tell the router: Whatever you have for 192.168.0.0/16, throw it on the FastEthernet0/0 interface. Whatever else you got, ship it on the WAN (my DSL modem). This is done like this:

ip route 192.168.0.0 255.255.0.0 Fast0/0
ip route 0.0.0.0 0.0.0.0 Dialer1

For the switch, things are really simple. The switch knows how to route between VLANs because we have created an SVI for each VLAN. We just need to tell it to send whatever does not belong to a known subnet to the router, through the routed port:

ip route 0.0.0.0 0.0.0.0 Fast0/1


C++ JSON libraryLast edited on Jul 9, 2012

After spending some time trying to find a good JSON library for C++, I realized that all the libraries out there are too heavy to use. Some of them look very good but their usage looks heavy. So I decided to write my own. My library is compliant with RFC 4627 except that it doesn't support unicode and numbers in exponential format .

Seriously, this library is really easy to use and has no dependencies (other than STL). I cannot find another C++ json library that is that simple to use.

Usage examples

The library exposes one object that is used to do everything you need. the "JSON" object. So there is no need to include a whole bunch of header files and use a whole bunch of class. You only need the JSON object to do everything you need. The object exposes these methods:

MethodDescription
JSON& operator[](int i);Access a list item
JSON& operator[](std::string str);Access an object member
std::string str();Get value of item
void parse(std::string json);Parse a JSON document
std::string stringify(bool formatted=false);serialize JSON object
JSON& addObject(const std::string& name="");Add object
JSON& addList(const std::string& name="");Add List
JSON& addValue(const std::string& val,const std::string& name="");Add string value
JSON& addValue(int val, const std::string& name="");Add integer value
JSON& addValue(double val, const std::string& name="");Add double precision FP value

Reading a JSON document

The library is very simple to use. Just compile it and it will output a "test" executable and a .a that you can link against. Then let's say you have the following JSON document:

{
    "obj1":{
        "member1":[
            "val5",
            "val4"
        ],
        "member2":"val3"
    },
    "list1":[
        "listItem1",
        "listeItem2",
        {
            "listObject1":"val2"
        }
    ],
    "value1":"val1"
}

The following code is an example on how to use the library:

JSON json;
std::string val;
std::string str = someFunctionThatReadsAJSONDocumentFromFileOrNetworkOrWhatever();
json.parse(str);
val= json["obj1"]["member1"][0].str(); // would give "val5"
val= json["list1"][1]["listObject1"].str(); // would give "val2"

Each access to a member will return a JSON object. So you only have 1 class to use at all time. So you can create a new variable each time or you can access all members by chaining the function calls.

val= json["obj1"]["member1"][0].str(); // would give "val5"
JSON& j1 = json["obj1"];
JSON& j2 = j1["member1"];
val = j2[0].str(); // would give "val5"
val = j2.str();    // would give "{list}"
val = j1.str();    // would give "{object}"

Invalid paths

The nice thing about this is that you don't need to worry about null objects. If you try to access an invalid member, you will get an invalid JSON object. But if you try to access another member from an invalid JSON object, you will also get an invalid JSON object. You will never get a NULL object that could crash your application.

val= json["obj1"][1].str(); // would give "{invalid}" because obj1 is an object, not a list
val= json["obj2"].str(); // would give "{invalid}"
val= json["obj2"]["member2"].str(); // would also give "{invalid}"
val= json["list"][100].str(); // would give "{invalid}"
val= json["list"][100]["something"].str(); // would also give "{invalid}"

Writing a JSON document

There are 3 methods provided to add items in the JSON document:

  • addObject(const std::string& keyName="")
  • addList(const std::string& keyName="")
  • addValue(const std::string& val, const std::string& keyName="")

All 3 functions have an optional keyName parameter. That is because if you add an item to an Object, you need to specify the key name that will be used in the parent. Again, I wanted to have a simple interface without having to force the programmer to use different classes if using a list or an object. So this here is the behavior of those function calls if you provide the key name or not.

ActionResult
addXXX on Object and provide key name item added and parent uses keyName
addXXX on Object and don't provide key name item added and a key name is auto generated
addXXX on List and provide key name item added and key name ignored
addXXX on List and don't provide key name item added
addXXX on Value and provide key name operation is ignored
addXXX on Value and don'tprovide key name operation is ignored

After adding items in the json object, you can then serialize it with stringing().

JSON json;
json.addObject("obj1");
json.addList("list1");
json.addValue("val1"); // key autogenerated because key name not provided
json["list1"].addValue("item1");
json["list1"].addValue("item2");
std::string val = json.stringify(true);

Would output:

{
    "obj1":{
    },
    "list1":[
        "item1",
        "item2"
    ],
    "key2":"val1"
}

Download

Project can be found on github



ACD Server with resiprocateLast edited on Jul 9, 2012

This is another project I've been working on using resiprocate. I'm using this ACD server with Asterisk. All devices, including the ACD server are talking with asterisk. The ACD server registers with asterisk just like any other phone would. When my gateway (FXO) sends a call to asterisk, asterisk routes it to the ACD server. When an agent logs on the ACD, the ACD server checks the contact header and establishes a presence subscription to that contact. In my case, this is always going to be an extension on Asterisk.

Features

Feature list:

  • multiple queues
  • Agents can be members of many queues
  • Queues are called with AOR (i.e: telemarkerqueue@acdserver.local)
  • most-idle agent routing (MIA)
  • redirect on no answer (to next MIA, with configurable timeout)
  • music while waiting
  • welcome message + periodic announcements while in queue
  • supports g.711 uLaw only
  • supports SIP info only for DTMF
  • REST api with JSON formatted responses
    • list of queues and calls with state, source/destination etc.
    • list of agents with state, idle time etc.

Unfortunately, there is too much I want to do so here is the list of other features I would like to add when I find the time

  • Adding agents & queues dynamically without restart
  • Ringall queues
  • Calling an agent directly (agent@acdserver.local)
  • Force a call out of the queue using REST api (unattended transfer to any device on same call server).
  • Use one thread only for all RTP sessions
  • Prevent agent to log on more than one phone and more than one agent on same phone whatever codec I use)

Using it

Here is a typical scenario:

  • Agent dials *44 (asterisk routes this to sip:acdlogin@acdserver.local as per dialplan)
  • ACD Server answers and prompts for agentID
  • ACD finds agent in internal list
  • ACD subscribe with calling phone (using contact header).
  • agent becomes available when a notify indicates IDLE.
  • agent dials *45 (asterisk routes this to sip:acdlogout@acdserver.local as per dialplan)
  • ACD unsubscribes and set Agent as unavailable

Asterisk

In sip.conf, under the profile for my acdserver, I set:

subscribecontext=acdpresence
context=acdserver

When phone1 calls the acdserver, the contact header will appear to be sip:phone1@asterisk.local to the ACD server (because asterisk is a B2BUA). So when the ACD server will try to subscribe to that device, it will need to have a corresponding entry in the dialplan. This is how I setup my dialplan:

[acdserver]
exten => _.,1,Dial(SIP/${EXTEN})

[acdpresence]
exten => _.,hint,SIP/${EXTEN}

It is discouraged to use "_." but currently, this is my only option. I'll try to find something. But this is because I don't have a naming convention for my devices. If all your devices are called phone1, phone2, phone3 etc. then you could use exten => _phone.,1,Dial(SIP/${EXTEN})

API

The server listens for incomming requests in the form of a RESTful API. The responses are sent as JSON data. I prefer JSON over xml since it is easier to parse with javascript and it also looks nicer in my opinion. I used my own json library which you can also find on this website. The server currently supports 2 requests.
curl -X GET "http://127.0.0.1:242/queues" would return

{
    "queues": [
        {
            "name": "queue1",
            "calls": [
                {
                    "id": "335099e931f09ad46ea75b8a451ad65d@192.168.0.3",
                    "state": "assigned",
                    "from": "gateway@192.168.0.3",
                    "to": "queue1@192.168.0.3:5071"
                }
            ]
        },
        {
            "name": "queue2",
            "calls": [ ]
        },
        {
            "name": "queue3",
            "calls": [ ]
        }
    ]
}

curl -X GET "http://127.0.0.1:242/agents" would return

{
    "agents": [
        {
            "id": "2771",
            "state": "idle",
            "idletime": "14",
            "device": "",
            "memberof": ["queue1","queue2","queue3"]
        },
        {
            "id": "2772",
            "state": "loggedout",
            "idletime": "0",
            "device": "",
            "memberof": ["queue2"]
        },
        {
            "id": "2773",
            "state": "loggedout",
            "idletime": "0",
            "device": "",
            "memberof": ["queue3"]
        }
    ]
}

Download

This code is experimental and is a mess right now. A lot of it can change at any time. The only libraries you will need is resiprocate and ortp.
download



Pages:123456789101112