All change: New job, new challenges

I've worked for the University of Cambridge Computing Service and subsequently for University Information Services for almost 17 years. At the end of this month I'm going to step back from my current responsibilities and go and do something almost entirely new.

With the support of UIS's management and at my request I'm about to be seconded to work with Dr Ian Lewis, University Director of Infrastructure Investment, on the Intelligent City Platform that he's developing as part of the University's collaboration with Connecting Cambridgeshire's Smart Cambridge' project  and the Greater Cambridge Partnership (formerly The City Deal). This involves a whole group of technologies (remote sensing, mapping, GIS, data handling, etc., etc.) that I've always been interested in but never had the time to work with. I'm looking forward to the new opportunities.

Over the years I've done quite a few things for the Computing Service and UIS, many of which I'm still involved with managing and developing. Plans are afoot for rearranging and reassigning those responsibilities, but at least in the short term this change is going to throw additional work on my already overstretched colleagues. If you are a consumer of any of the things I've previously been responsible for then please cut my colleagues a little slack while these reorganisations complete. One advantage of the agreed secondment arrangement is that for the short term I'll still be available to provide answers to all those questions that I've somehow failed to answer in the documentation.

Onward and upwards...

x509 certificate chaining

Our x509 certificate supplier recently change the root and intermediate certificates needed to use them (without warning, which was unhelpful). Sorting this out caused me to have to re-learn how certificate chaining is supposed to work.

As far as I can see, the primary rule is that the 'Issuer' DN of one certificate must match the 'Subject' DN of a certificate corresponding to the key that signed the first certificate.As an optimisation or hint, certificates can contain a 'Authority Key Identifier' which should match the 'Subject Key Identifier' of a certificate corresponding to the key that signed the first certificate. Quite what a key identifier is is not defined though there are suggestions. In most cases it's some sort of digest of the corresponding public key.

If you really want to know how certificate paths are built and validated then see RFC 4158 and RFC 5280.

Updating SAML Service Provider keys for Shibboleth IdsP

SAML Service Providers (SPs) maintain one or more key pairs for use when interacting with Identity Providers (IdPs). The public part of these key pairs are distributed to IdP via SAML metadata or otherwise.

SP keys are used for two purposes:

1. The SP uses them to sign authentication requests
2. IdP use them to encrypt attribute assertions in responses for decryption by the SP (encryption is optional, but commonly enabled)

Keys need to be replaced occasionally. This can be difficult to achieve without service disruption because the public parts are distributed and it's impractical to update all copies simultaneously. For a Shibboleth IdP, coping with replacement of an SP key used for signing isn't problematic -- a new key can be added to the SP's metadata ahead of deployment and the IdP will be happy with signatures made with either. However once two keys appear in the metadata the IdP may use either to encrypt assertions and this will only work if the SP is happy to perform decryption with either key.

For Shibboleth SPs this isn't a problem, because they can be configured to accept multiple keys for decrypting assertions while using a nominated key for creating signatures. There's a well known procedure for rotating the key on such a SP, see for example documentation on the UK federation site and in the Shiboleth Consortium Wiki. This amounts to:

• Create new key pair on the SP
• Configure the SP to accept either key for decryption but not use the new one for signing (by adding the new one with use="encryption")
• Add the new key to the SPs metadata for both use="signing" and use="encryption" (or just without a 'use' attribute)
• Wait for the metadata to propagate, or directly update all IdPs
• Switch the SP to use the new key for signatures (by removing use="encryption" from the new one and adding it to the old one)
• Completely remove the old key from the SPs metadata
• Wait for the metadata to propagate, or directly update all IdPs
• Remove the old key from the SP configuration

Some SPs don't support this. However it is only encryption that causes a problem. The main reason for encrypting assertions is to prevent users seeing what the IdP is saying about them, but in man cases there's nothing in the assertions that the user can't see by other means. So temporarily disabling encryption while rolling a SP key is a possibility. This means that the key can be rolled without downtime with the following revised procedure:

• Setup custom relying party on the IdP for the SP that suppresses encryption. Something like this:

<rp:RelyingParty id="<SP entityID>"
   provider="<IdP entityID>"
   defaultSigningCredentialRef="IdPCredential" >

   <rp:ProfileConfiguration xsi:type="saml:SAML2SSOProfile" 
       encryptAssertions="never" 
       encryptNameIds="never" />

 </rp:RelyingParty>

• Add the new key to the SP's metadata
• Wait for the metadata to propagate, or just update all IdPs.
• Switch the SP to use the new key
• Remove old key from metadata
• Wait for the metadata to propagate, or just update all IdPs.
• Remove the custom relying party configuration

Note that this requires coordinated work on every IdP with which the SP works so it's not practical on a large scale. It will however work where there's essentially a one-to-one SP -> IdP relationship, as in many SaaS scenarios.

Tools such as the Firefox SAML tracer, the Chrome SAML Dev Tool extension and the SAML Chrome panel are  really helpful for checking that each step of the process has completed before moving in to the next.

Mitigating recent TLS vulnerabilities

Recently discovered vulnerabilities in TLS (also known as SSL -- the protocol implementing secure web browsing and securing other activities such as email) can and should be mitigated by appropriate server configurations. Existing defaults and previously-recommended configurations may require attention to address these issues. While these vulnerabilities have been addressed in recent versions of major clients, not everyone runs up-to-date versions and not all access is from major clients.

Vulnerabilities addressed by this advice include 'POODLE' (CVE-2014-3566), 'Freak' (CVE-2015-0204), and 'LogJam' (CVE-2015-4000).

What represents a 'best' configuration depends on the capabilities of the servers involved, and of their expected clients. The best security can only be obtained on up-to-date software and only with configurations that may exclude some older clients. The following advice will provide a reasonable level of security but should be reviewed in the light of specific requirements.

The following advice is intended to be generic; specific configuration advice for some platforms appears below.

Suggestions (in order of importance):

1) Ensure that the SSLv2 and SSLv3 versions of the protocol are disabled.

Note that IE6 on Windows XP will be unable to communicate with servers that don't support SSLv3, but given its age this should be acceptable -- many major services already disable SSLv2 and SSLv3.

2) Adjust the cryptographic suites supported to exclude the following:

• all 'export' suites
• any using symmetric encryption with keys less that 128 bits
• any using signatures based on the MD5 hash algorithm
• any using symmetric encryption based on the RC4 algorithm

3) Configure Diffie-Hellman (DH) key exchange to use at least 2048-bit groups. Additionally generate a unique 2048-bit group for use in Diffie-Hellman key exchange on each server. As an alternative, it may be appropriate to disable all cryptographic suites that rely on Diffie-Hellman key exchange. Plan to upgrade systems that can't be appropriately configured.

Note that this advice does not apply to Elliptic-Curve Diffie-Hellman key exchange (ECDH) which does not currently have any known vulnerabilities. Note also that Java 1.6 and 1.7 clients may be unable to communicate with servers offering Diffie-Hellman key exchange using groups over 1024-bits long.

4) Support TLSv1.2 - plan to upgrade any systems that can't do so.

One way to test your configuration is to use the SSL Labs server test page. Aim to eliminate any issues flagged 'VULNERABLE' or shown in red, and to reduce or eliminate any marked 'WEEAK' or shown in orange. It should be possible to achieve an overall rating of at least 'B' and preferably 'A' but don't be guided entirely by the overall rating shown. The 'Handshake Simulation' section of the report can be helpful when evaluating the impact of any configuration change on clients.

Implementations:

Apache:

Add the following directives to httpd.conf or equivalent and ensure that they are not being overridden elsewhere:

    SSLProtocol all -SSLv2 -SSLv3
    SSLCipherSuite ALL:!aNULL:!eNULL:!LOW:!EXP:!MD5:!RC4

TLSv1.2 is automatically available on systems running OpenSSL 1.0.1 or above but not otherwise - plan to upgrade systems that include only lower versions

By default Apache 2.2 only supports a fixed 1024-bit Diffie-Hellman group - plan to upgrade it. Apache from 2.4, and patched versions of 2.2 in some Linux distributions, support longer fixed groups. Unique groups can be created with the command

    openssl dhparam -out dhparams.pem 2048

and loaded into patched versions of Apache 2.2, and into Apache 2.4 with the directive

    SSLOpenSSLConfCmd DHParameters dhparams.pem

Restart Apache after making these changes.

IIS:

See elsewhere for instructions on disabling SSLv2 and SSLv3.

To set cryptographic suites:

• Open the Group Policy Object Editor (i.e. run gpedit.msc in the command prompt).
• Expand Computer Configuration --> Administrative Templates --> Network --> SSL Configuration Settings.
• In the right pane, open the SSL Cipher Suite Order setting.
• A reasonable cipher suite list (from Bulletproof SSL and TLS, Ch 15) would be:

      TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256
      TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384
      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA_P256
      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA_P256
      TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256
      TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384
      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA_P256
      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA_P256
      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256
      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 
      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

    If this excludes some old but necessary clients then consider adding:

      TLS_RSA_WITH_AES_128_CBC_SHA
      TLS_RSA_WITH_AES_256_CBC_SHA
      TLS_RSA_WITH_3DES_EDE_CBC_SHA
      TLS_RSA_WITH_RC4_128_SHA

Reboot after making these changes

nginx

The Nginx project have published instructions on how to disable SSLv3 on Nginx.

To configure cipher suites place the following in the website configuration server block in /etc/nginx/sites-enabled/default (see the LogJam pages):

      ssl_ciphers 'ALL:!aNULL:!eNULL:!LOW:!EXP:!MD5:!RC4';

Custom Diffie-Helman groups can be created with the command:

      openssl dhparam -out dhparams.pem 2048

and loaded into nginx with the following configuration:

      ssl_dhparam dhparams.pem;

Tomcat

See elsewhere for instructions on disabling SSLv2 and SSLv3.

Configuring cipher suites (see Bulletproof SSL and TLS):

* With the APR/Native connector

•  Set the 'SSLCipherSuite' attribute of the 'Connector' XML element in your $TOMCAT_HOME/conf/server.xml file:
  
  SSLCipherSuite = "ALL:!aNULL:!eNULL:!LOW:!EXP:!MD5:!RC4

* With the JSSE connector

• Set the 'ciphers' attribute of the 'Connector' XML element in your $TOMCAT_HOME/conf/server.xml file:
  
  ciphers = "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
  TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
  TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
  TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
  TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
  TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384,
  TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
  TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
  TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
  TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,
  TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
  TLS_RSA_WITH_AES_128_CBC_SHA,
  SSL_RSA_WITH_3DES_EDE_CBC_SHA"

Information Systems - what we do

Building on yesterday's list of technologies use by the Information Systems team here at the University, here are some of the services that we run. Much of this will probably only make sense to people inside the University.

• Forum
• The authentication wrapper for Google Apps @ Cambridge
• Managed Web Server
• Managed Wiki Service
• Raven web authentication
• TLS Certificate Scheme
• University Map
• Web Search (spidering back-end, the search.cam interface is run by others)
• Webtools Web Server (hosting miscellaneous UIS infrastructure)

In addition, we contribute to a number of services run by other parts of UIS:

• CamCERT
• Falcon CMS
• Git Repository Hosting Service
• Network Traffic Analysis (for Traffic Charging and intrusion detection by CERT)
• UIS News Service

So now you know.

An Information Systems colophon

I've followed the work of Government Digital Service (GDS - the people behind GOV.UK) for a while. It seems to me that an organisation dedicated to “leading the digital transformation of government” probably knows a thing or two that's relevant to the digital transformation of a university.

GDS have a blog, and their list of technologies that they use is interesting. Work I've been doing recently means that I've created a similar list of the technologies used within my Information Services team here at the University. For what its worth this what it looks like:

Core servers

• Most services run on virtual machines provided by an internal VMware cluster
• Most run under Linux - SuSE Enterprise, Ubuntu, or Debian
• High-availability clusters use Corosync & Pacemaker
• Servers are built using our internal BES build system
• We have one legacy service using Solaris

Base software products

• Some services are based on existing software, including phpBB and Mediawiki
• The Web Search service is provided by Funnelback

Technologies

We rely on, and in some cases also support, a range of technologies:

• Ucam WebAuth, Shibboleth, SAML for web authentication
• LDAP and the University's Lookup service (and its API) for authorisation
• TLS, PKI and X509 certificates for web security
• OpenStreetMap and its API for geographic information
• AbFab, EAP, PAM, and RADUIS as part of investigations into Moonshot
• Google APIs for Google Apps integration
• Netflow for network traffic analysis

Applications (incl. Frameworks, etc)

• Our current favorite development environment is Python and Django
• Other systems use C, Perl (with HTML::Mason), Java, JavaScript, PHP, Ruby (with Rails)
• System administration relies heavily on shell (normally Bash)
• Our ‘Managed CMS’ service is based on Plone
• Web Sites run under Apache and Tomcat

Database and other storage

• MySQL
• PostgreSQL with Slony for replication

Monitoring, managing and alerting

• Services are monitored with Nagios
• Collectd collects statistics and Graphite and Grafana visualise them
• Servers are increasingly built and manage with Ansible
• We are moving towards Git (with an internal Gitolite service) for version control, but also use (at least) SVN, CVS, RCS and SCCS!

Supporting Tools

• Central RT system for support tickets
• Trac and Mantis for bug tracking
• We've been experimenting with LeanKit and Asana for program and project management

Why I think php is a bad idea

Update: a friend reminds me of http://me.veekun.com/blog/2012/04/09/php-a-fractal-of-bad-design/ which covers the same topic from a different angle.