Archive for May, 2011

Thoughts about extlookup() in puppet

Wednesday, May 11th, 2011

To manage some of the database servers I look after I use puppet.

I’ve recently been trying to clean up quite an extensive set of puppet recipes I’d written. A suggestion that has been made was to remove the data from the recipes and a pointer was made to the new extlookup functionality. That is keep data and code separate and make it easier to modify the behaviour without having to modify the code itself.

In my case I manage a few sets of database servers. Similar groups of boxes are configured the same way. One specific function I was looking at to improve was this snippet.

case $domain {
'dom1.example.com': { s_db::master_vif::check { 'master vif': interface => 'eth0:123', ip_address => '10.1.xxx.yyy' } }
'dom2.example.com': { s_db::master_vif::check { 'master vif': interface => 'eth0:456', ip_address => '10.2.xxx.yyy' } }
}

The intention here being that on the primary master the interface would be enabled and on the standby master the interface would NOT be enabled.

Note: this snippet applies to a pair of boxes but there are several other pairs to configure in the same way, each with their own specific settings.

The extlookup() functionality only allows a single “lookup value” to be searched and implicitly rather than explicitly where to look for this value. The snippet above has 2 “parameters” and if I have several pairs of boxes: MDB_A, MDB_B, MDB_C (the name is not part of the domain or the hostname) then using the extlookup() functionality I’m going to have to setup a lot of data files and this looks unwieldy, especially as I have other resources defined which take several parameters.

So I thought about how I would do this to make the lookup facility more generic and visible.

What seems better to me would be something which is a bit more generic and based on looking up a < “config_item”, “lookup_value” > pair in a single location to get a single value as now. That would make the extlookup() prototype probably change to something like:

extlookup( “config_item”, [ list of variables to use ] , “default_value”, [ lookup "tables/csv files" ] )

parameters being:
[1] config_item ( grouping of values to lookup )
[2] array of values to apply the lookup on, done in order as now but more explicitly shown
[3] default value
[4] explict array of locations of the data files.

The .csv files would have 3 columns: config_item,lookup_value,result_value

Using something like this I can then simplify the snippet above to:

$interface_name = extlookup( 'MDB_A-vif-interface', [ $hostname, $domain ], '', [ 'dba_config' ] )
$ip_address = extlookup( 'MDB_A-vif-ip-address', [ $hostname, $domain ], '', [ 'dba_config' ] )

s_db::master_vif::check { 'master vif': interface => $interface_name, ip_address => $ip_address } }

This is clearer as the configuration is more explicit then the current extlookup() definition.

If performance of having a large number of entries in a single file were a concern then talking to a DB backend and doing something like the following would work:

SELECT lookup_value FROM some_table WHERE config_item = ? AND lookup_key = ?

Also while in this example above I can use $hostname or $domain as a retrieval key in several other boxes I probably can’t but might need to lookup on another value such as server_type or db_name.
Having the explicit [ .... ] list where I can add any string I want to be looked up give more flexibility and is clearer.

Using the current (puppet 2.6) mechanism would require me, if I understand it correctly, to configure different
.csv files to be searched for each “configuration parameter”. I may also need in several places to override the $extlookup_precedence. In any case this can’t be overridden several times within the same module which is what I would need if I want to lookup different variables.

That doesn’t strike me as being very useful. It’s also unclear from the current extlookup() call itself where the data is being retrieved from. Using these external variables seems to me to make the calls behave like magic.

So my question was to ask how others who are using the extlookup() functionality how they cope with more complex settings than the system settings which depend on say $hostname or $domain etc. and whether my proposed extension makes any sense.

Thanks for your thoughts.

2011-05-12, update

Another snippet of something I’d like to do which seems easier to fulful using my proposed extlookup() setup.

$notifications_1 = 'address11@example.com'
$notifications_2 = 'address2@example.com'
$notifications_3 = 'address3@example.com'

case $hostname {
'host01':  { $email = $notifications_1 }
'host02': { $email = $notifications_2 }
default:   { $email = $notifications_3 }
}

This would seem more naturally expressed as:

$email = extlookup( 'notification_email', [ $hostname ], $notifications_3 )

or perhaps the following if you want to only check for boxes in your domain.

$email = extlookup( 'notification_email', [ $hostname, $domain ] )

but most importantly with a single data file containing:
notification_email,host01,address1@example.com
notification_email,host02,address2@example.com
notification_email,example.com,default_email@example.com