Archive for February, 2011

log_processlist.sh – script for monitoring MySQL instances

Thursday, February 24th, 2011

I thought I would upload a small script I use which I find quite useful. log_processlist.sh is normally run minutely from cron and collects statistics from any MySQL instance it finds. Data is kept for up to a week and then rotated. This allows you to check what happened over a period of time, for example if an instance crashes or seems to have a lot of load.

I have found it quite invaluable on the instances I’m monitoring and once setup it’s usually nice and silent. If you find this useful or have suggestions for improvements please let me know.

A few more things to add to my desired SOHO FreeSWITCH configuration

Friday, February 11th, 2011

Since my last post I’ve spoken some more on #freeswitch and also freeswitch-users and have some ideas of how to implement some of the things I wanted to do on my SOHO PBX setup.

Additionally I’ve also added several other tasks which I think are important. So here is where you can find the updated post: http://blog.wl0.org/voip-configuration-requirement-for-freeswitch/.

I’ve not fully understood the configuration in FreeSWITCH using 2 different ports for internal and external traffic, especially as most SIP systems just use the default SIP port 5060. As such I want to configure FreeSWITCH so it works on a single port
rather than using the 2 ports the default configuration uses.

Since the requirements list has become quite long I want to try to document how I get to the final setup in the hope it will allow someone else to do this more quickly than it’s taken me.

If you see a configuration / description of FreeSWITCH behaviour which is wrong, or a better way to implement these features please let me know. This is a learning process for me.

Freeswitch – progress and frustration

Wednesday, February 2nd, 2011

I’ve not posted for a while and thought it about time I gave an update on my progress trying to get FreeSWITCH to work.

Bluebox

I previously mentioned that I was trying out Bluebox and it looked promising. I did play around with it for a while but had various issues which I had trouble solving.  One of these was due to the way the configuration is stored. Bluebox configuration is stored in a database and then auto-generated. This means that the configuration files have entries for things like sip_interface_1, trunk_1, trunk_2, etc… When looking in the log files this can be a bit confusing as it’s not apparent which “real entry these names refer to”.

2011-01-09 22:40:33.837612 [NOTICE] sofia_reg.c:342 Registering trunk_1
2011-01-09 22:40:33.837612 [ERR] sofia_reg.c:1612 trunk_1 Registration Failed with status DNS Error [503]. failure #37
...
2011-01-11 12:49:56.927418 [DEBUG] switch_core_state_machine.c:77 sofia/sipinterface_1/203@bluebox.example.com Standard ROUTING
2011-01-11 12:49:56.935221 [INFO] mod_dialplan_xml.c:331 Processing 1001 <1001>->003491xxxxxxx in context context_2

I’d suggest that the configuration optionally allows the user to provide their own names so that you can entries such as the_name_of_my_sip_provider instead.

There were also some configuration issues with registrations and dialing not working as I had expected.  To be honest I did get offers of help from some of the Bluebox developers, but did not think it right to expect them to hand-hold me debugging certain software issues. In the end I could not get Bluebox to work the way I expected and so gave up. That is not to say that Bluebox is bad, but I think it needs to include some more tools to help diagnose the status of various things.  After all if you provide a GUI interface it helps if you also provide some sort of GUI test tools.  Perhaps I will try again later and see if  some of the issues I had experienced have been resolved.

Back to Manual Configuration File editing

In the end I decided to try again and edit the configuration files manually. I used the Bluebox install iso as a base (running inside vmware) and this time decided before I started to try to write up a requirements document and work through it. That can be found here. I decided to run the configuration inside a vmware guest while I was testing as this keeps the configuration behind my router’s NAT setup and in theory FreeSWITCH is not visible to the outside world so is safe while I learn and adjust the configuration.

The results of doing things this way have been promising but I notice several things which perhaps for a FreeSWITCH newbie may not be ideal.

  • I believe that the default FreeSWITCH logging may be too verbose. This is great for developers who understand all the logging messages and may be looking for a specific log entry but can be rather intimidating for someone who is trying to see if something worked or not and if not why not. For those type of people a less verbose log level may be appropriate, and of course you can later increase logging if you need to get the details of why something is not working as expected. So I’m not sure that verbose logging is ideal for the default freeswitch configuration.
  • xml file breakage. I’ve occasionally edited the xml configuration files incorrectly and when running reloadxml get the frustrating message about a failure to parse the file and some weird line number. Figuring out what entry is wrong and why from this message is quite tricky.  I only recently came across some comment saying that the preparsed xml files get written to log/freeswitch.xml.fsxml.  This helps a lot as you see the final configuration file generated from the various individual files but I believe mention of this should be more prominently commented in the various tutorials. It’s easy to miss. That said FreeSWITCH could at least report the full path of the xml filename where the error is found.
  • Related to the same issue is the fact that, as far as I can see, comments need to be in the format <!– some stuff –>. It is not clear to me whether white space around these comments is allowed always or only sometimes as the “xml breakage” I seem to have triggered was often caused by adding several lines of comments as above, often with white space before or after the comments. I did not expect that to make a difference but it seems that in some places it does.  I’d love some clarification on this and if it’s possible to make FreeSWITCH’s xml parser slightly more tolerant that would be nice.

Default configuration FreeSWITCH security concerns

Note: the comments here relate to the 1.0.6 version of FreeSWITCH. I’ve tried to build newer versions on my Linux box but the build fails. Checking out 1.0.6 builds without problems but I realise that a lot of changes have happened since. It would be nice to test on a newer version.
The default/sample configuration seems to be very open which is great for learning as a lot of functionality is enabled by default. However, this seems to be a perfect opportunity for SPIT (which seems to be the term used for the equivalent of VoIP SPAM).  While the default password for the extensions is configurable in vars.xml it is not apparent that it may be really important that you change this to something really secure.  I was working with FreeSWITCH behind a NAT’d router so was pretty relaxed about this, thinking they couldn’t reach the software, as it was not accessible on my public IP address.  However, that assumption seems not to be as true as I thought. It seems that registering with an external SIP gateway opens a hole out, but also opens a hole through the firewall to FreeSWITCH. That’s scary and I have not seen it mentioned anywhere.

A few days ago I noticed a lot of FreeSWITCH logging of something apparently trying to dial various international numbers. Looking at the logs it seems that someone had managed to guess one of my extension’s (not a default extension) passwords and was trying to dial out again to various numbers including places like Somalia.  As my configuration was not complete, and the default gateway not setup, this was failing but I was rather horrified at how easily someone could get in.  I was lucky and this could have cost me a lot of money.

I still don’t understand how the attacker found/figured out the extension that was being used to dial out from. Not being a default extension it seems it must have been guessing a large number and finally found one that worked.

However, this really reminded me of a similar situation several years back with sendmail. Before qmail, postfix and several other newer mail servers were popular most UNIX boxes came pre-installed with sendmail. Sendmail is great: you can configure it to do anything but the configuration file is rather complex and hard to really understand. At the time the configuration also assumed that it was fine to resend mail and often sendmail was configured as an open relay.  Newer MTAs (and sendmail too now) are configured with a slightly different attitude: the Internet is hostile and there are lots of people who, if they can, may try to abuse your system.  So now the software is configured to be very restrictive of what mail it will accept and forward.

How does this relate to FreeSWITCH? I’ve seen attacks to my current Asterisk setup (though not successfully) and with the attack of my FreeSWITCH configuration it seems that VoIP software should have something similar. The configuration should be designed to allow the least amount of access necessary and also to have controls in place to mitigate attack attempts.  Things that come to mind are:

  • the external profile should log authentication failures by default
  • FreeSWITCH should have some sort of rate limiting configured, so that someone trying to access FreeSWITCH frequently will trigger this and be ignored for a while, with the issue logged clearly.
  • It should be more obvious how to configure network ACLs for extensions, and these should be configured by default.  Postfix by default allows relaying (registering an extension) only from hosts in my networks which by default is the network of the network interface. This can of course be adjusted but is a good safe starting point.  FreeSWITCH’s default configuration should perhaps be similar, only allow registrations from clients on my network.
  • It seems that to register the client can use FreeSWITCH’s ip address or domain name.  Once someone outside has figured out where the SIP listener is located it’s quite easy to try to register with different users at the ip address. If you only accept the registration to user@my.domain.com then the attacker has to know what the correct configuration realm is before they can make much progress. That’s much harder for them. So it again seems like a good idea to NOT allow registrations directly to the IP address (at least by default).
  • For trunk connections if you have a DID number you expect the VoIP provider to call you. In this case it seems good that the example configuration clearly allow you to limit where the incoming calls to that incoming trunk number may come from, and whether they should be password protected or not.
  • Allow rate limiting of calls to a gateway, or from an extension. That’s different to rate limiting registration attempts to a specific extension, or from a specific ip address.  Also allow the configuration and limiting of the number of concurrent connections that may be allowed through a trunk/gateway.

I believe that some and perhaps  all of these features are possible in FreeSWITCH but they are not documented explicitly in the default configuration in a way which you really take note. It would be helpful if that were addressed.  You expect to have a mail server on all unix servers, and they should be secure. Many people using voice software probably expect the same.

Additionally a hacked mail server does not cost you money. A hacked PBX does.

FreeSWITCH cookbook

One thing that I would like to see is a FreeSWITCH cookbook, something that gives examples of lots of different types of configuration and is complete enough to use to solve different tasks that many people using FreeSWITCH may need to get
resolved. Like many other cookbooks I’ve seen (for perl, mysql) comments on the how and why help understand things better.  This could be part of the existing FreeSWITCH book, though perhaps a separate book would be more appropriate.

The FreeSWITCH configuration files are complex and there are many of them. I think this is part of what makes the software hard to follow for someone who is not using it on a day to day basis. Compare that with Asterisk which basically
has 2 configuration files you are going to change: sip.conf and extensions.conf. FreeSWITCH has many more.  This is why I think that you see a lot of people playing with Asterisk: it’s configuration is reasonably straight forward to understand and doing the basics is quite simple. From there you can hack away adding things that you want to improve. So a FreeSWITCH cookbook would be great. It could mention many of the things that I’ve mentioned in this post but others such some of ones I mention here below are issues I’m still working on resolving:

  • how to setup dialing to a trunk with a fallback trunk if the call fails.
  • common voicemail setups
    • set different languages for different extensions
    • set different languages for an outside caller depending on how he dials FreeSWITCH (which incoming DID number is used)
    • setup of a common voicemail mailbox (typical in a SOHO environment) shared by various extensions
  • how to setup FreeSWITCH in a new language (what’s required to make this work). FreeSWITCH seems to come with English and Russian and there are some external non-free language sound files, but from what I can see no more free ones. Can we “hack” or use a set of Asterisk sound files (using as many of them that “match”) in the meantime as Asterisk has sound support in many more languages? A short guide of what is wanted, or perhaps some sort of fundraising to bulid these would be nice.  I’d be willing to donate something if this were coordinated by the FreeSWITCH developers. If I could figure out what is needed then I may do this myself.  My specific interest is in Spanish, but it would be nice to have some “British English” sound files. Not strictly necessary of course but would be nice.
  • How to change the dial tones to sound like different countries. I live in Spain but have never really figured out how to change the dial tones to match what my family members expect to hear. Though I think that the default setup seems to match pretty closely, it would be nice to match this properly.  Again specific examples for different countries is useful for everyone.
  • Setting up the equivalent of Asterisk’s macros. I’ve seen in several places that I’d like to do the same thing and can’t figure out how to do what I’ve done in Asterisk which is setup a macro to parameterise the functionality. Something as simple as:
    • dial_gateway(gateway_name,gateway_name_file,number_to_call) which makes my configuration
      • say “using gateway <gateway_name>” (taken from some wmv files)
      • set up recording of the call
      • dial the number via the specified gateway
    • for calls to an extension(extension_number,language)
      • ring for a while and if picked up, setup recording and answer
      • if not picked up redirect to voicemail (in the appropriate language)

These are just some examples of things which it seems need to be grouped together as common functionality and so far I have not figured out how to do this in FreeSWITCH short of duplicating the configuration for each extension or trunk.

Summing up

In the meantime I am going to try and figure out how to tighten down my configuration so that it feels safe. Having seen various
attack attempts and one that partly succeeded I really want to be able to trust the configuration to not be the cause of a large bill
at some time in the future.  I like FreeSWITCH and would not have persevered with it if I did not think that the software is good.
It just seems to me that the documentation, while available, is not there in a form which many of us can digest easily.  The FreeSWITCH 1.0.6 book has been very helpful but I believe needs to cover more detail than now perhaps in a second edition. I think that more and better documentation will encourage more people to use this software. Sometimes I wonder if I am the only one who finds FreeSWITCH tricky to configure.