Making the UK Federation usable

About a year ago, in Bye, bye Athens... hello UK Federation, I questioned the rather grand claims being made about the then new UK Access Management Federation for Education and Research, notably that it would deliver "improved services to users", and wondered what the reality would be like.

I think we're still waiting to find out to be honest but there doesn't yet seem to be much evidence that anything has really improved over what we had before - certainly not in terms of usability for the end-user!

Last week I attended a meeting set up by the JISC-funded Service Provider Interface Study project, looking specifically at usability issues within the federation as things currently stand.

Firstly... hats off to both the project team and JISC for being so open about the issues. The meeting was a real eye-opener (for me at least), not only in that it demonstrated just how poor usability is across all the players that make up the federation, but also in the realisation that, actually, most service provider access control is done via IP address checking rather than by SAML-based authentication, in part because the usability issues are so great. For most users, SAML only comes into play when they are off-site (at least according to the service providers present at the meeting). Note: I appreciate that this was also the case under the old Athens system... I mention it here only because it seems to me that the continued use of IP address checking hasn't been widely acknowledged in the way the federation is generally presented, so it came as something of a surprise (to me at least).

Usability problems hit almost every aspect of the Federation as it is currently deployed, from the point that the end-user is initially asked to sign-on right thru to the ways in which service provider services are personalised (or not). Overall usability is made worse because the end-to-end experience is distributed across several players - the service provider, the identity provider, and (optionally) a 'where are you from?' service - each of which can, and do, make different decisions about naming and design. The result is a confusing experience for the end-user, combining poor usability of the individual components in the system coupled with a lack of consistency between the different parts, leading to a situation where it must be near impossible, for example, to write user-support documentation (i.e. help pages) covering everything in a comprehensive form. Even trivial issues, such as what 'sign-on' is called and where it is positioned on the page, are handled differently by different players.

It seems to me that privacy and security issues seem to have driven much of the thinking behind the Federation in its current form. At one point I asked the meeting whether anyone had actually asked real end-users whether they would like to have the option of sharing more information between the identity provider and the service provider in order to enjoy a more seamless and usable experience overall (even if it theoretically compromised their privacy in some way)? I'm not sure I got a clear answer... but it is hard not to draw the conclusion that the Federation has been designed by people with a primary interest in the technology rather than the user experience. A bit like the 'good old days' when we let the techies have full control over firewall policies, disregarding the fact that people actually had jobs that they needed to get done.

I'm sorry if all this seems very blunt but the current deployments are so un-friendly that something has got to be done - otherwise we might as well just bite the bullet and go back to having separate login accounts for every service we access (something that most people are perfectly accustomed to these days anyway!).

So... I want to focus on two, related, aspects of usability for the remainder of this post: naming the authentication process and discovering the identity provider.

Firstly... what do you call the process by which you gain access to a service provider in a SAML-based world? How are things 'branded' if you like? This is a non-trivial question to answer but a great example of how largely technical considerations (like the need for federations) have been allowed to get in the way of user-oriented usability issues. It's also something that the OpenID crowd have got cracked. That's not to say that there aren't other problems with OpenID - there are - but at least they have a single global brand (and associated logo) which makes it easy for any user, anywhere in the world to realise when they are being asked to sign-in using their OpenID.

What's the equivalent brand in the SAML world? There isn't one. Nobody pushes the use of a "SAML sign-on" (quite rightly in my opinion) since it would be meaningless to people. Shibboleth, as I've argued before, names a particular bit of software rather than an approach, and so is inappropriate. Some service providers in the UK still use 'Athens' (because it's what end-users are used to!) - again, clearly wrong in a SAML world. That leaves branding at the level of the federation... but who on earth wants to refer to their "UK Access Management Federation login" - what a horrible mouthful that is. And remember... most service providers offer their services globally, so if we brand things at the federation level then service providers have to start asking their users which federation they are part of - something that I suspect most of us neither know nor care!

That brings us on to my second usability issue. In a SAML-based world, service providers have to direct the end-user back to their institution in order that they can sign-in using their institutional username and password before being redirected back to the target service. This is typically done using a 'where are you from' service, either stand-alone on the network or embedded into the service provider website. Typically, this involves the end-user selecting from a pull-down list of identity providers (there are over 500 in the UK Federation currently), optionally preceded by a pull-down list of possible federations.  This is horrible.

I'd like to propose a new rule of thumb for the design of user-interfaces in a SAML world... if we ever have to explicitly ask the end-user to choose from a list which federation they are part of, then we have a totally borked approach and we need to do something different. This seems obvious to me - yet it is exactly the direction we are heading in right now :-( .

I'd actually go much further and say that if we ever have to explicitly ask the end-user to tell us which institution they are a member of just so they can sign-in to something, then we have similarly broken the system - but I appreciate that is a more difficult part of the process to remove given the current technology. We can, however, make the selection of the institution rather easier than scrolling thru a list of 500 (or 1000, or 5000) names. How about looking at the way TheTrainLine let you select a station name? How about using the JQuery Auto-Complete function to narrow down the list of available names as the end-user begins to type? Here's a demo of just that. Much more intuitive than a pull-down list.  (Thanks to my colleague, Mike Edwards, for the sample code to build this, based on the JISC "what do we do?" page.)

It'll be interesting to see what recommendations the Service Provider Interface Study project comes up with.  Here are mine.  Let's stop thinking in terms of asking the end-user what federation they belong to and think instead of questions they are likely to know the answers to.  What is the name of the institution you belong to? What's the URL of your institutional website? What country are you in?  Let's make the machines do the hard work of sorting out which federation is relevent.  In short, let's start building user-interfaces, no... scrub that... let's start designing the system as a whole such that usability and the needs of the end-user are put first rather than being tacked on as an after-thought!

Finally... I'm surprised that publishers, and other service providers, aren't driving this much more forcibly than they appear to have done to date.  There was a strong feeling in the meeting that things have got much worse (in usability terms) since the demise of Athens - yet the publishers present seemed rather defeatest about what they could do about it.  Every time the usability of the system breaks, a service provider somewhere stands to lose a customer and while they are not typically paying for content individually it ultimately all adds up.  Publishers should be pressing the UK Federation (and all other federations) for a system that works end-to-end, not just because of their own self-interests, but because of the interests of their primary user-base.  I also think that they should be working much more closely together to bring greater consistency to the way that SAML-based sign-on is presented to the end-user.

Comments

[Richard Padley]

Andy,

You are quite correct to point to the appalling design of the current WAYF system as an pristine example of poor user experience. In addition to the auto-complete refinements you've suggested I suggest its possible to cache IP and GEOIP lookups every time a user visits the WAYF, and use these to present a sensible default option based on past user choices. In this way the system "learns" as the first few users go through the pain of selection.

In reality this is why IP address authentication is still dominant. But this is not without its own problems - see my blog post http://blogs.semantico.com/discovery-blog/2009/06/ip-address-authentication-considered-harmful/

[Jonathan Rochkind]

This is a very important point, and you have some good ideas. I've been noticing this as a HUGE problem for any real-world Shibboleth adoption too. I hope someone listens to you and does something about it!

If the federation attends to it and decides how the login should be branded/identified, I suspect they could have quite a bit of luck in getting content providers to adhere to their branding specs.

And then I'd need to wait for the US shibboleth federation to follow suit. And then providers that want to follow both the US and the UK federations branding specs... are going to have a pain. Hmm.

[Jonathan Rochkind]

Oh, and yeah, IP address authentication is HUGELY problematic.

1) On campus, at least at my organization (and I suspect I'm not alone) we can't actually reliably correspond IP addresses to the units that licenses might exist for. It's not possible. Pretty soon ALL of our outgoing traffic is going to be NAT/PAT'd to a small group of IP addresses (as per current network best practices especially when you're running out of IP addresses) without regard to internal organizational unit -- yet we license content for internal organizational units. Nightmare!

2) For off-campus users, this means relying on proxy, which puts another piece of software, and another couple network journeys, between the user and the service, potentially really slowing things down for them, and also giving us another thing that can (and sometimes does) break or malfunction and keep them out. And from experience, that's not just 'potential', in both cases.

We've really got to get away from IP authentication, it's a big mess. But the usability problems in Shibboleth style login are pretty immense at the moment.

[Nicole Harris]

Hi Andy

I think these are mostly fair comments! A few observations:

The project is testing with end-users and previous testing with end-users has come up with some interesting results. When given the option of a promoted branded click-on and a list of institutions, more than two thirds of the users were instantly drawn to the list. Users can cope with lists - just think of all the times you have to choose your country from a list when filling in your address. It's important not to assume users are dim :-)

One of the things we are hoping the study will produce is a recommendation around the need (or lack of need) for a brand that all the federations around the world are open to discuss - we've established this already. Anything that is specifically country branded is doomed to fail.

I think the central UK federation WAYF needs improving and I am frustrated that the federation has been slow to improve this function. It is however supposed to be a lowest common denominator service for publishers who don't want to do it themselves, so perhaps it has been left a bit too long for these reasons.

However, very few publishers are using the central WAYF, and are developing directly on their own interfaces. Therefore we need the publishers to work with us on this and follow the recommendations that are provided - this so far has been the hardest part of the whole work to achieve. I think the workshop was an excellent step in the right direction.

I don't think these usability issues are immense or unsolvable, and the recommendations from the study will be fairly simple and elegant I hope. The challenge is the implementation, and this is a challenge that JISC is bravely taking on.

We've been trying to persuade institutions to move away from on-campus IP reliance for years - all throughout the Athens days as well as the time since the federation came along - but have yet to convince them despite the problem with IP that have been very well described by other commentaries. I think this is somewhere we have failed, but will continue to try to articulate a convincing argument :-)

[AndyP]

@Nicole,
Yes, people are used to pull-down lists... but long ones are nearly always inherently unusable and the approach definitely doesn't scale to the kind of numbers of entries we are likely to have if we aggregate organisations from across multiple federations (or even inside single bigger federations like the UK one).

I'm certainly not arguing that "users are dim"... I'm arguing that users deserve a usable system, especially given how much money has been invested in it.

FWIW... I *do* think the usability issues are immense. I think we have some very difficult usability challenges ahead if we want to end up with something that is usable across the system as a whole.

Finally, note that the JQuery code I suggested was intended to be used within publisher sites rather than at the central WAYF - the quicker we can completely get rid of the central WAYF the better as far as I'm concerned - it's a horrible part of the process.