Thoughts about extlookup() in puppet

To manage some of the database servers I look after I use puppet.

I’ve recently been trying to clean up quite an extensive set of puppet recipes I’d written. A suggestion that has been made was to remove the data from the recipes and a pointer was made to the new extlookup functionality. That is keep data and code separate and make it easier to modify the behaviour without having to modify the code itself.

In my case I manage a few sets of database servers. Similar groups of boxes are configured the same way. One specific function I was looking at to improve was this snippet.

case $domain {
'': { s_db::master_vif::check { 'master vif': interface => 'eth0:123', ip_address => '' } }
'': { s_db::master_vif::check { 'master vif': interface => 'eth0:456', ip_address => '' } }

The intention here being that on the primary master the interface would be enabled and on the standby master the interface would NOT be enabled.

Note: this snippet applies to a pair of boxes but there are several other pairs to configure in the same way, each with their own specific settings.

The extlookup() functionality only allows a single “lookup value” to be searched and implicitly rather than explicitly where to look for this value. The snippet above has 2 “parameters” and if I have several pairs of boxes: MDB_A, MDB_B, MDB_C (the name is not part of the domain or the hostname) then using the extlookup() functionality I’m going to have to setup a lot of data files and this looks unwieldy, especially as I have other resources defined which take several parameters.

So I thought about how I would do this to make the lookup facility more generic and visible.

What seems better to me would be something which is a bit more generic and based on looking up a < “config_item”, “lookup_value” > pair in a single location to get a single value as now. That would make the extlookup() prototype probably change to something like:

extlookup( “config_item”, [ list of variables to use ] , “default_value”, [ lookup “tables/csv files” ] )

parameters being:
[1] config_item ( grouping of values to lookup )
[2] array of values to apply the lookup on, done in order as now but more explicitly shown
[3] default value
[4] explict array of locations of the data files.

The .csv files would have 3 columns: config_item,lookup_value,result_value

Using something like this I can then simplify the snippet above to:

$interface_name = extlookup( 'MDB_A-vif-interface', [ $hostname, $domain ], '', [ 'dba_config' ] )
$ip_address = extlookup( 'MDB_A-vif-ip-address', [ $hostname, $domain ], '', [ 'dba_config' ] )

s_db::master_vif::check { 'master vif': interface => $interface_name, ip_address => $ip_address } }

This is clearer as the configuration is more explicit then the current extlookup() definition.

If performance of having a large number of entries in a single file were a concern then talking to a DB backend and doing something like the following would work:

SELECT lookup_value FROM some_table WHERE config_item = ? AND lookup_key = ?

Also while in this example above I can use $hostname or $domain as a retrieval key in several other boxes I probably can’t but might need to lookup on another value such as server_type or db_name.
Having the explicit [ …. ] list where I can add any string I want to be looked up give more flexibility and is clearer.

Using the current (puppet 2.6) mechanism would require me, if I understand it correctly, to configure different
.csv files to be searched for each “configuration parameter”. I may also need in several places to override the $extlookup_precedence. In any case this can’t be overridden several times within the same module which is what I would need if I want to lookup different variables.

That doesn’t strike me as being very useful. It’s also unclear from the current extlookup() call itself where the data is being retrieved from. Using these external variables seems to me to make the calls behave like magic.

So my question was to ask how others who are using the extlookup() functionality how they cope with more complex settings than the system settings which depend on say $hostname or $domain etc. and whether my proposed extension makes any sense.

Thanks for your thoughts.

2011-05-12, update

Another snippet of something I’d like to do which seems easier to fulful using my proposed extlookup() setup.

$notifications_1 = ''
$notifications_2 = ''
$notifications_3 = ''

case $hostname {
'host01':  { $email = $notifications_1 }
'host02': { $email = $notifications_2 }
default:   { $email = $notifications_3 }

This would seem more naturally expressed as:

$email = extlookup( 'notification_email', [ $hostname ], $notifications_3 )

or perhaps the following if you want to only check for boxes in your domain.

$email = extlookup( 'notification_email', [ $hostname, $domain ] )

but most importantly with a single data file containing:

Published by

Simon J Mudd

Born in England, I now live in Spain, but spent a few years living in the Netherlands. I previously worked in banking (financial markets) both in IT and as a broker, but IT has always had a stronger influence. Now working at as a Senior Database Administrator. Other interests include photography, and travel. Simon is married, with two children and lives in Madrid.

2 thoughts on “Thoughts about extlookup() in puppet”

  1. Ggod post, extlookup was never designed to fix all problems. It exist in a space between getting to know puppet and needing an ENC. Writing good ENCs can be hard so it struck me there could be middle ground where a simple configurable datasource might save time and for many people it might be enough all together. It let’s you get going down the proper data handling path quickly but in many cases eventually given enough complexity an ENC is needed.

    Anyway, fwiw you can give extlookup a 4th parameter being a specific CSV file to search before the usual order. This might get you closer to making it a but nicer without writing code.

  2. You’ve said before that extlookup() isn’t expected to be the solution to everything, so I should perhaps not be so critical.

    However, the basic idea of extlookup() is completely right: You want a mechanism to allow you to retrieve a configuration parameter and look that up based on a series of criteria starting from “very specific” to more generic and finally optionally provide a default value if no match has been found. That makes a lot of sense and is clean.

    What I dislike about the extlookup() mechanism is that it’s harder to override the lookup parameters (if you need to do this frequently several times in the same class). Several configuration parameters I’m using are “site” dependent and the site is not directly part of the normal variables provided by factor such as $hostname, $domain, etc. So sometimes I want to lookup a parameter per site (master_database_address) but other times using the default search parameters is fine. Mixing this in a single class seems tricky and additionally is not explicit.

    You’ve suggested I look at the code and see if I can adapt it to fit my requirements. That seems a sensible idea. I had hoped to stick clear of Ruby but in the end it looks like that’s not going to be possible. Let’s see if I can come up with something that works for me.

Leave a Reply