Getting nagged by Nagios

Nagios is an entirely usable service monitoring system - I'm aware of at least three implementations within the University Computing Service alone. There are some aspects of its design (or, I suspect, its evolution) that I don't particularly like, but all in all it's much, much better than nothing.

An important feature is its powerful and capable configuration system. As usual this is a two edged sword because you have to understand the resulting complexity to take advantage of the power. I have two things to offer that might help: some general configuration guidelines, and a diagram.

Guidelines

• Keep the number and size of edits to the distribution-supplied config files to a minimum. 
• Arrange that related names and aliases share a common prefix (e.g. foo in what follows) so that they sort together (host names, which are most usefully based on DNS names, can be an exception to this)
• Keep to a minimum the number of distinct names created (e.g. use foo as both a contact and a contact group name and so don't create foo-contactgroup) 
• Note that, unlike most other 'names', service_description isn't a global name and only needs to be unique within the set of hosts that provide it. It doesn't need and shouldn't have a foo prefix, and should be a short, human-readable, description of the service
• Keep names (especially those commonly displayed in the web apps) as short as reasonably possible
• Express group membership by listing group name in the individual objects, NOT by listing the individual objects in the group definition (or if you want, the other way around but be consistent!)
• Use inheritance wherever possible to avoid replicating common settings
• Use a naming convention so that separate groups of people can create global names without clashing
• Store information belonging to each group of people in a predictable location (e.g. always put host information in files or directories starting "host") to make navigation easier
• Optimise the web-based display 

The UCS's Unix Support group has further developed these guidelines into a configuration standard that helps them monitor large numbers of services with minimal configuration work and with several different groups of people maintaining the configuration. Part of the key to this is putting the configuration for each separate group of services into separate sub-directories and incorporating this information into the main nagios.cfg with 'cfg_dir'.

Object relationships

Nagios's Template-Based Object Configuration is one of its most powerful features but it's difficult to get you head around it when starting out. Here's a diagram that might help - it shows most of the relationships between the various objects:

Authentication: Think before you proxy

There are three obvious ways to add authentication such as Ucam Webauth or Shibboleth or OpenID to a web application:

• Add the authentication code to the application itself (what the Java world calls 'Application Managed Security')
• Implement the authentication in the web server or container hosting the application (Java's 'Container Managed Security')
• Implement the authentication in an HTTP proxy in front of the application

The proxy option has some obvious quick gains: a single proxy can protect multiple applications; it's normally easy to leverage existing 'container managed' implementations to build such proxies; using a proxy probably requires much less modifications of existing applications (sometimes none). But I'm going to argue that this approach, while appropriate and even necessary in some circumstances, should be avoided where possible. Here are some reasons:

• Security: to be useful, the proxy has to pass on information (such as user name at the very least) to its downstream client and obviously has to do this is a way that can't be influenced by an attacker. About the only option (if you are sticking to HTTP for communication between the proxy and its downstream client) is by adding extra HTTP request headers. It's not easy to be absolutely sure  that these headers can't somehow be injected by an attacker (by talking directly to the downstream client, by supplying them in a request and convincing the proxy to pass them on, etc., etc.).
• Availability of attributes: Shib and other similar systems potentially make lots of information about the authenticating user available as attributes. Unless you arrange for your proxy to pass all of these on (and there could be lots and you might not know in advance what they all are) then your clients will loose out. There's also the problem that (in theory, if not in practise) that users are able to control what information about them is disclosed to which services. With multiple services behind a single proxy it's going to be a case of one size having to fit all.
• Confused logs: clients behind proxies see all connections as coming from the proxy. This makes their logs less useful than they might be and frustrates any attempt to limit access by IP address (which is actually a bad thing anyway IMHO, but that's another story). There are ways around this, though most then suffer from the security problems mentioned above.
• One more system: the proxy is one more system to build, maintain and in particular to understand. I have a theory that authentication services in some big web applications are a bit 'hit and miss' (and 'electronic journals' are big offenders here) precisely because they are big, layered applications that almost no one (or perhaps actually no one in a few cases) really understands.

I think my underlying feeling would be that a proxy is an entirely appropriate, if not quite as simple as it first appears, way to implement authentication for some particulare application, but that care should be taken not to slip into using one in every case. Like they say, once you've got a hammer everything starts looking like a nail...

Core data stores in a University context

I've just come across what I think is a really interesting post about core data stores in a University context:

http://alexbilbie.blogs.lincoln.ac.uk/2010/02/04/core-dot-lincoln/

I haven't thought this through fully, but on first glance it seems to me that much of what's identified as needed at Lincoln would be directly relavent in Cambridge. The idea that University systems should be making interfaces to raw data available, for use by other University systems and potentially by others, has been bubbling around in the back of my mind for a while (and is something that I know is already happening in systems such as dSpace and the Streaming Media Service) and this post seems to be helping to crystalise some of those thoughts. Perhaps I'll come back to this topic in a later post.

Promiscuous JavaScript considered dangerous

A trick commonly used to incorporate content from one web site into pages provided by another is for the provider site to make available a piece of JavaScript which inserts the required content into any page when loaded and executed. There are lots of examples - see for example Talks.cam 'Custom views', or the Met Office's Weather widgit.

This is realy neat, and has the huge advantage that it doesn't need any server-side support to impliment (typically RSS/Atom would need work on the server to collect and format the feeds) and can be deployed by just about anyone with editing access to the pages in question and the ability to cut and paste.

But it comes at a huge price. JavaScript that you include in your pages (wherever it comes from) has largely unfettered access to those pages. So it can rewrite you content, read cookies (and so steal authentication sessions), post content back to your server, etc., etc. So consider, before doing so, that adding JavaScript to your pages that is controlled by someone else amounts to giving that 'someone else' editing rights to your page. Are you happy to do that?

Of course reputable providers of this sort of JavaScript won't do anything like this deliberatly. But you do need to consider that they might do it accidently - it's particularly easy for them to mess up other JavaScript on you pages (for example the Met Office widgit above sets the JavaScript global variable moDays without any check if it's being used elsewhere). This trick is oftern used to provide feed information that may have been originally supplied to the provider site by even more third parties. There's the danger that the provider's cross site scripting protection may be incomplete, opening the posibility of unknown third parties being able to inject JavaScript into your pages, which is bad. And if you are relying on someone else's cross site scripting protection you need to know what character encoding they are doing it for and to be sure that you are setting a compatible one (this is discussed in a very helpful and to their credit very honest page supplied by Talks.cam that describes the risks of using their 'Promiscuous JavaScript' offering).

Note that I'm NOT saying that you shouldn't use this trick (as a supplier or as a provider). I am suggesting that you should be aware of the risks and make an apropriate assement of how they affect any particular deployment before proceding.

Apache configuration file layouts

A traditional Apache configuration consists of one file (httpd.conf) that contains all the required configuration directives. However a single file is a problem for  packaging systems where different packages are responsible for different aspects of Apache's operation. For them it's much easier if they can contribute one or more files containing configuration fragments and if these are then incorporated into the Apache configuration using the 'Include' directive. While convenient for the packaging system, this is less convenient for the system administrator who now finds his Apache configuration spread across multiple files in several directories. Here are a two diagrams showing the configuration file layout in two common Linux distributions - Debian (and so Ubuntu), and SLES:

Debian

Note that the mods-enabled and hosts-enabled directories contain symlinks to files actually stored in the parallel mods-available and hosts-available directories,  and that commands, a2enmod, a2dismod, a2ensite and a2dissite, are provided to manipulate these symlinks. Within the mods-{available,enabled} directories, the *.load files contain the Apache configuration directive to load the module in question; the coresponding *.conf files contain configuration directives necessary for the module. httpd.conf is included for backwards compatability and to support installing 3rd party modules directly via apxs2. See the file /etc/apache2/README for more details.

SLES

The files shown in yellow boxes, all of which appear in the /etc/apache2/sysconfig/ directory, are regenerated automatically from information in /etc/sysconfig/apache2/ on Apache startup and so shouldn't be hand edited. See comments at the top of /etc/apache2/httpd.conf for more information.

This diagram is for SLES10 and Apache 2; similar arrangements were used with SLES 9 and Apache 1.3, with 'apache2' replaced by 'httpd' in filenames. In SLES 9 it was necessary to run SuSEconfig to regenerate the files based on sysconfig information.

2010-02-23: Debian diagram amended - the 'master' file was incorrectly labelled httpd.conf and should have been apache2.conf. Apart from anything else, you can't have httpd.conf including itself!

Paul Walk's 'Infrastructure service anti-pattern

Following on from Service-to-service communication, I've just seen a excelent blog posting in paul walk's weblog entitled 'An infrastructure service anti-pattern' which makes an excellent case for how machine APIs should be used. Well worth reading.

Service-to-service communication

The University of Cambridge's Raven service works well enough for interactive logins using a web browser, but doesn't (and was never intended to) support non-interactive authentication, or authentication between one service and another, rather than between people and services. Here's a set of suggestions for filling this gap and for supporting general service-to-service communication - I happen to like these today but I'm making no promises that I won't have changed my mind by tomorrow.

For 'proxied' or non-interactive authentication on behalf on individuals I'd recommend OAuth. This is essentially a standardised protocol for establishing a token that grants one service limited, delegated access in a user's name to another service. There's a good example of how it could work in the Beginner’s Guide to OAuth – Part II : Protocol Workflow. OAuth is gaining significant traction in social networking applications.

For service to service communication I'd recommend SSL/TLS using mutual authentication by certificate. Since we are assuming that authentication is required we should also assume that confidentiality is necessary so the protection offered by SSL/TLS seems appropriate.

Certificate trust could just be established bilaterally between pairs of services, but the complexity of this grows with the square of the number of services involved. Better would be to establish an in-house Public Key Infrastructure with a central in-house Certification Authority (CA) that could issue certificates for this purpose. Some difficult policy decisions will be needed about who is allowed to apply for certificates in the name of which services, but once made it should be possible to largely automate the CA by providing a Raven-authenticated web interface for certificate management. Note that these certificates would need to identify 'services', rather than just computers, so the parties to a conversation could for example be the 'CS IP Register Database' and the 'Department of Important Studies Network Management system'. We'd need to sort out a naming convention. An important service provided by the CA would need to be the maintenance of a Certificate Revocation List.

Authorisation I'd leave to the services involved. Both OAuth and certificate authentication establish the 'identity' of a party in a conversation and it should be easy enough to use this identity within whatever authorisation system a service already uses. For example, Lookup could be adapted to allow certificate DNs to appear alongside user identifiers as members of groups used for access control. 

Finally we need to identify protocols by which services can communicate. I suggest something lightweight and vaguely 'REST'ish. Authorities differ on what exactly REST requires, but here I just mean a basic CRUD interface carried over HTTP and mapped onto HTTP primitives PUT, GET, POST, DELETE, etc. Data should probably be serialised using simple XML, though other formats such as JSON are a possibility. Existing XML schema can be used where appropriate, for example the  Atom Syndication Format can be used to represent lists (particularly search results), and the Atom Publishing Protocol is probably worth considering to support the creation and modification of resources (see Wikipedia for an introduction to Atom).

The advantage of this approach is that it provides a lightweight and technology neutral interface using tools (HTTP servers and clients) that are widely available and reasonably well understood. It even allows an amount of  experimentation using nothing but a web browser. It also opens the possibility of in-browser manipulation of data, especially if results are available in JSON. Against this there's the need for an API design for each new service and the requirement for programming work at both the client and server ends. One way  of supporting this is to distribute at least one example client library with each new API. An important selling point for this approach is the fact that it underpins almost all of the current 'cloud' offerings - see the Google Data Protocol, Amazon Web Services, Yahoo Social API, etc.

There are other posibilities for filling the various slots mentioned above - obvious ones being SSH to provide confidentiality and strong mutual authentication, and SOAP to provide interservice communication. I happen to think (today, as mentioned above) that the set I've listed here would currently provide the best solution. Why might be the subject of subsequent posts.