Asterisk attack

There was a lot of talk about this being the next menace after email spam. I’m not actually sure what it’s called for VoIP systems, but my Asterisk setup has started to be attacked over the last few days. Lots of entries like:

[Aug 27 19:20:30] NOTICE[18826] chan_sip.c: Registration from '"742"<sip:742@a.b.c.d>' failed for '' - No matching peer found
[Aug 31 10:13:10] NOTICE[18826] chan_sip.c: Registration from '"1002" <sip:1002@a.b.c.d>' failed for '' - Wrong password

Lots of messages get logged a second and I noticed this as suddenly CPU load on my PC jumped up quite a bit.

For the moment I’ve routed these addresses via the interface lo0 so they won’t bother me any more, but I need to come up with a better solution.

First I’m curious if applications like Asterisk or FreeSwitch have any built-in anti-abuse controls to recognise bad behaviour and to disable those abusers. I’m pretty sure that I’ve not read about anything for Asterisk, and I’m currently reading the FreeSWITCH book I bought but haven’t come across this mentioned yet.  Seems that applications like this may need to have these controls added at some time, just as sendmail, postfix and most mail servers have had to adjust to a hostile world.

The other option of course is to use a firewall or packet filter to limit the incoming traffic rate from a single IP to port 5060 or whereever the SIP connection is being accepted so that when going over the limit the ip will be blocked for some time. iptables can do this I think so I’m going to have to read about how to configure and set that up.

There are other applications designed to watch logs and use them to automatically add temporary blocks. fail2ban is one of these. I’ll also have to see if I can configure it for this task.

So if this has happened to you how do you protect your VoIP systems from that hostile world of the Internet?

Don’t forget to monitor your nameservers

As I mentioned in a past article I got my IPv6 connectivity working so started working on setting up various IPv6 services. One of these was to setup my name server so it also worked on IPv6.

This worked fine, but recently I lost my IPv6 connectivity but thought no more about it. I’m trying to get it working again but that’s not happened yet.

Over the last few days I’ve had some intermittent issues logging in to my home pc from my laptop and I couldn’t figure out why. Eventually I had a look at the DNS requests and of course some requests were being made to my non-reachable name server. This generated some network delays while the resolver timed out and then tried talking to one of the IPv4 name servers and correctly retrieving the information.

So conclusion, if you setup yourself with a name server for your domain and include access on IPv6 then ensure that you monitor it. I do run nagios at home and intend to add some extra checks so that I can see if there are problems. Even when playing with IPv6 if you don’t have things setup properly it can affect non-IPv6 activity…

Once the unreachable IPv6 AAAA entry was removed from my name server things went back to normal.

Using LVM snapshot filesystems for development database instances

The Problem

Developers often need to have a development database copy of the live production system you are using in able to allow them to test their code and to test new functionality and make schema changes to the database for this new functionality to work.

That’s normal and happens everywhere. A typical DBA task is to make a copy of the live system, sometimes to remove any confidential or sensitive information which perhaps the development database users should not be able to see, and then give them access to this development instance. The developers then “hack away”, changing their code and perhaps things in the database until they are ready to put these new changes into production when they then come along and discuss how to apply these changes into the live systems.

Once the development database has been created it soon becomes stale so often the developers want a new up to date copy to be made to “simplify” their testing.

This is all fine until the database size begins to grow and this process of dumping and copying the data for the developers takes hours rather than minutes and therefore can only be done on a daily or weekly basis.

I have recently been experimenting with the use of mounting the development database instance on an  LVM snapshot of the original filesystem where the live system is running.  This procedure does not have to be Linux specific but should work with any OS or storage which provides a facility to make a filesystem snapshot based on the contents of another filesystem.

So what does this mean in practice?

Previous Behaviour:

Situation: server1 has a live production copy of the database. We want to make a copy to server2. server2 is already prepared with a configuration which will work based on the copy of the production data.

Procedure: stop server1, copy the filesystem holding the database to server2, start server1. start server2. [the copy procedure takes hours.]

New Behaviour:

Situation: server1 has a live production copy of the database (probably via a slave), and space/memory for a second development instance to run concurrently on the same server.

Procedure: stop server1 (live instance), make a LVM snapshot of the live filesystem (the snapshot size can be much smaller than the live filesystem size), start server1 (live instance), start server1 (development instance). [the copy procedure takes just a few seconds.]

Since I do this with the live system being a slave, I tend to also include a routine to disable replication information on the snapshot filesystem by removing the appropriate files. It may also be necessary adjust the grants on the dev-instance so that it is appropriate for the new set of db users.

To all intents and purposes when you login to the development instance it looks like an up to date copy of live system. You can make as many changes as you like as long as the number of disk blocks on the snapshot which get changed don’t exceed the snapshot size. At this point the snapshot filesystem becomes invalid and mysqld is unable to access it. Mysqld gets a bit upset about this, but you just kill it and then go and rebuild the instance again if this happens – it only takes a few seconds.

This works pretty well and speeds things up for the developers. I can create a new development environment from the live system in seconds rather than hours. The disk storage requirements also tend to drop significantly. It also helps the devs. If you do a daily refresh of this development instance then it allows the developers to test any schema changes which will be needed to be applied to the live system much more easily as “going back to the current live state” is so easy.

So if you haven’t done something like this it might be worth giving it a go.

This is an example of the output from a script I’m currently using:

[root@myhost ~]# clone_instance -s40G -d live-instance dev-instance
Aug 22 10:57:00 myhost clone_instance[16405] Cloning live-instance to dev-instance with a snapshot volume of size 40G
Aug 22 10:57:00 myhost clone_instance[16405] /mysql/live-instance is mounted as expected
Aug 22 10:57:00 myhost clone_instance[16405] Device /dev/volgroup1/live-instance is mounted on /mysql/live-instance, having volume group: volgroup1, logical volume: live-instance
Aug 22 10:57:00 myhost clone_instance[16405] live-instance is defined in /etc/my.cnf [mysqld1]
Aug 22 10:57:00 myhost clone_instance[16405] dev-instance is defined in /etc/my.cnf [mysqld2]
Aug 22 10:57:00 myhost clone_instance[16405] Found defaults file /root/.my-live.cnf needed to shutdown live-instance
Aug 22 10:57:00 myhost clone_instance[16405] Going to viciously kill any processes using files under mount point: /mysql/dev-instance
Aug 22 10:57:02 myhost clone_instance[16405] Unmounting /mysql/dev-instance
Aug 22 10:57:02 myhost clone_instance[16405] Removing existing SNAPSHOT LV /dev/volgroup1/dev-instance
Logical volume "dev-instance" successfully removed
Aug 22 10:57:03 myhost clone_instance[16405] SNAPSHOT LV /dev/volgroup1/dev-instance removed
Aug 22 10:57:03 myhost clone_instance[16405] Shutting down live-instance [mysqld1] using mysqldmin and defaults file /root/.my-live.cnf (as 'mysqld_multi stop 1' does not seem work properly)
Aug 22 11:02:34 myhost clone_instance[16405] Creating new snapshot LV dev-instance (40G) based on /dev/volgroup1/live-instance
Logical volume "dev-instance" created
Aug 22 11:02:35 myhost clone_instance[16405] Restarting live-instance [mysqld1] using mysqld_multi start 1
Aug 22 11:02:35 myhost clone_instance[16405] Mounting SNAPSHOT LV /dev/volgroup1/dev-instance on /mysql/dev-instance
Aug 22 11:02:35 myhost clone_instance[16405] Cleaning up log files on SNAPSHOT LV
Aug 22 11:02:35 myhost clone_instance[16405] Removing replication information from SNAPSHOT LV
Aug 22 11:02:35 myhost clone_instance[16405] Found defaults file /root/.my-dev.cnf needed to access dev-instance
Aug 22 11:02:35 myhost clone_instance[16405] Starting SNAPSHOT DB instance dev-instance [mysqld2] using: 'mysqld_multi start 2'
Aug 22 11:02:35 myhost clone_instance[16405] Clone procedure complete.
[root@myhost ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
160G  135G   26G  84% /mysql/live-instance
160G  134G   27G  84% /mysql/dev-instance
[root@myhost ~]# lvs
LV             VG        Attr   LSize   Origin         Snap%  Move Log Copy%  Convert
live-instance  volgroup1 owi-ao 160.00G
dev-instance   volgroup1 swi-ao  40.00G live-instance    2.06
[root@myhost ~]#

The longest part of the procedure is shutting down the active live slave. The rest of the time is insignificant.
The “dev experience” when using this dev-instance is just as before. The instance works, can be modified and behaves just as you would expect.

Review of IPv6 Network Administration

Originally submitted at O’Reilly

This essential guide explains what works, what doesn’t, and most of all, what’s practical about IPv6–the next-generation Internet standard. A must-have for network administrators everywhere looking to fix their network’s scalability and management problems. Also covers other IPv6 ben…

A little outdated – needs updating

By Simon Mudd from Madrid, Spain on 8/18/2010
3out of 5

Pros: Well-written, Easy to understand

Cons: Too basic

I forgot to check the date this book was published before buying it and that was a mistake. The content was pretty good but as it was written in 2005 a lot of what it says regarding versions of operating systems and the current status of software is well out of date. That’s unfortunate and could probably be easily corrected. I think that while the target audience is not clear, I am looking from a hobbyist’s perspective as to how to setup my current home network to IPv6 and I did not really get enough information. The fact that wifi is pretty common now, but that if the wifi router you use does not support IPv6 (which is typical) then you have to make tunnels in your own IPv4 lan to achieve connectivity between hosts.

I have many other O’Reilly titles and some of them are very good. I was hoping for more from this one especially as I had previously bought IPv6 Essentials some time back.


IPv6 Reading list

I have just returned from holiday and while it’s not normal reading during some of my spare time I managed to read a couple of books on IPv6 I’d recently bought to get me up to scratch and refresh my memory on the topic.

These books are:

The first book is quite good, and while a little old provided a good solid background for me on the subject. A lot of the information was not relevant to my interest in setting up a networked SOHO network but that’s not really a problem. The background was interesting.

The second book was a bit of a mistake. I’d ordered it as it was a second edition and therefore up to date but I hadn’t realised that it was by Microsoft. Not a problem really but of course the focus was very much on Windows and Windows commands to achieve many things. I was surprised that Microsoft don’t support ripng (IPv6) or OSPF v3 even if they don’t really expect people many windows administrators to actually use them.  There’s also quite a lot of talk about ISATAP and TEREDO tunnelling which is not even mentioned in the Cisco book. Whether that’s because the Cisco book is not fully up to date or simply because these are rather Windows centric tunnelling mechanisms I’m not so sure.

So in the end the 2 books were useful. Certainly Cisco’s documentation and I liked the routing info which was useful.

I’m curious if you can recommend other IPv6 specific books, especially those related more to Linux and perhaps setting up systems in a SOHO environment. Comments welcome.

I’m also considering setting up another router with openwrt as this seems to be a good way to get the connectivity independently of the IPv4 infrastructure I currently have. That however is still work in progress and while my IPv6 tunnel is down I can’t make much progress.