Influence, connections and outputs

Martin Weller wrote an interesting blog post on Friday, Connections versus Outputs - interesting in the sense that I strongly disagreed with it - that discussed a system for assessing an individual's "prominence in the online community of their particular topic" by measuring their influence, betweenness and hubness (essentially their 'connectedness' to others in that community). Martin had used the system to assess the prominence of people and organisations working in the area of 'distance learning', suggesting that it might form a useful basis for further work looking at metrics for the new forms of scholarly communication that are enabled by the social Web. The algorithm adopted by the system was not available for discussion so one was left reacting to the results it generated.

I reacted somewhat negatively, largely on the basis that the system ranked Brian Kelly's UK Web Focus blog 6th most influential in that particular subject area. This is not a criticism of Brian (who is clearly influential in other areas), but the fact remains that Brian's blog contains only three posts where the phrase 'distance learning' appears, two of which are in comments left by other people and one of which is in a guest post - hardly indicative of someone who is highly influential in that particular subject area?

In passing, I note that Brian has now also commented on this and Martin has written a follow-up post.

Why does Brian's blog appear in the list? Probably because he is very well connected to people who do write about distance learning. Unfortunately, that connectedness is not sufficient, on its own, to draw conclusions about his level of influence on that particular topic, so the whole process breaks down very quickly.

My concern is that if we present these kinds of rather poor metrics in any way seriously in counterpoint to more traditional (though still flawed) metrics like the REF we will ultimately do harm in trying to move forward any discussion around the assessment of scholarly communication in the age of the social Web.

To cut a long story short (you can see my fuller comments on the original post) I ended by suggesting that if we really want to develop "some sensible measure of scholarly impact on the social Web" then we have to step back and consider three questions:

• what do we want to measure?
• what can we measure?
• how can bring these two things close enough together to create something useful?

To try and answer my own questions I'll start with the first. I suggest that we want to try and measure two aspects of 'impact':

• the credibility of what an individual has to say on a topic,
• and the engagement of an individual within their 'subject' community and their ability to expose their work to particular audiences.

These two are clearly related, at least in the sense that someone's level of engagement in a community (their connectedness if you like) clearly increases the exposure of their work but is also indicative of the credibility they have within that community.

Having said that, my gut feeling is that credibility, at least for the purposes of scholarly communication, can only really be measured by some kind of a peer-review (i.e. human) process. Of course, on the Web, we are now very used to infering credibility based on the weighted number of inbound links that a resource receives, not least in the form of Google's PageRank algorithm. This works well enough for mainstream Web searching but I wouldn't want it used, at least not at any trivial level, to assess scholarly credibility or impact. Why not? Well a couple of things immediately spring to mind...

Firstly, a link is typically just a link at the moment, whether it's a hyperlink between two resources or the link between people in a social network. The link carries no additional semantics. If paper A critiques paper B then we don't want to link between them to result in paper B being measured as having more credibility/impact than it otherwise would have done had the critique not been written.  (This is also true of traditional citations between journal articles of course, except that peer review mechanisms stop (most of) the real dross from ever seeing the light of day.  On the Web, everything is there to be cited.)

Secondly, if we just consider blogging for a moment, the way a blog is written will have a big impact on how people react to it and that, in turn, might affect how we measure it. Blogs written in a more 'tabloid' style for example might well result in more commenting or inbound links than those written in a more academic style. We presumably don't want to end up measuring scholarly impact as though we were measuring newspaper circulation?

Thirdly, any metrics that we choose to use in the future will ultimately influence the way that scholarly communication happens. Take blog comments for example. A comment is typically not a first class Web object - comments don't have URIs for example. One can therefore make the argument that writing a comment on someone else's blog post is less easily measurable than writing a new blog post that cites the original. One might therefore expect to see less commenting and more blog-post writing (under a given set of metrics). While this isn't necessarily a bad thing, it seems to me that our behaviour should be driven by what works best for 'scholarly communication' not by what can be most easily 'measured'.

As I said in my first comment on Martin's post, "connectedness is cheap". On that basis, we have to be very careful before using any metrics that are wholly or largely based on measures of connectedness. The point is that the things we can measure easily (to return to the second part of my question above) are likely to be highly spammable (i.e. they can be gamed, either intentionally or by accident). Yes, OK... all measures are spammable, but some are more spammable than others! If we want to start assessing academics in terms of their engagement and output as part of the social Web then I think we need to start by answering my questions above rather than by showcasing rather poor examples of what can be automated now, except as a way of saying, "look, this is hard"!

Comments

[PeteJ]

I don't mean to hijack the discussion with technical nitpicking (honest!), but it isn't true (or at least it's not true in all cases) that weblog comments don't have URIs.

I guess it may depend on stylesheet/theme how visible they are, but certainly, TypePad, Blogger and WordPress assign, and can expose, URIs for comments.

On this (TypePad) weblog, we publish comments with a URI (the target of the hyperlink in the "timestamp"), and those URIs are used in links the list of "most recent comments" in the right-hand column.

(Hmm, actually there seem to be two different URIs used! See e.g.

http://efoundations.typepad.com/efoundations/2009/06/jiscmail-and-social-bookmarking.html?cid=6a00d8345203ba69e201156fda3741970c#comment-6a00d8345203ba69e201156fda3741970c (from your initial comment)

http://efoundations.typepad.com/efoundations/2009/06/jiscmail-and-social-bookmarking.html#comment-6a00d8345203ba69e201156fda3741970c (from the hyperlink in the right-hand column))

Similarly on Martin's (TypePad) weblog, your initial comment on his post has the URI

http://nogoodreason.typepad.co.uk/no_good_reason/2009/06/connections-versus-outputs.html?cid=6a00d8341c0c0e53ef0115703a0070970c#comment-6a00d8341c0c0e53ef0115703a0070970c

A comment by Martin on a post of Brian's (WordPress):

http://ukwebfocus.wordpress.com/2009/05/11/how-is-he-embracing-web-2-0-how-is-web-2-0-changing-he/#comment-71958

A comment by Peter Miller on a post of Art's (Blogger):

http://artfossett.blogspot.com/2009/06/faithless.html?showComment=1243968061575#c4759073298554540052

In some cases(!), they might not be the "prettiest"/most human-readable URIs in the world, but they do their job, so I think weblog comments can be cited/counted/measured as well as posts, if that is what we want to do.

[Martin]

You knew I'd come back on this Andy ;)

As I said in the follow up post, I think any metric would be _part_ of a package. But I think to abandon any metric and say only human peer review is valuable will be to miss a trick. What I want is for an academic who is seeking promotion, say, to be able to produce a case which has comments and peer review on their online work, and also to be able to say 'in subject X I have scored the following on the HE Academic Influence Measure' - ie it is a reasonably reliable, objective measure that adds weight to their case.
It would be a shame with _all_ this data flying around for us not to find some use for it surely?
On your third point: "any metrics that we choose to use in the future will ultimately influence the way that scholarly communication happens."
Yes, absolutely agree - this is always the problem, the experimenter effect/heisenberg uncertainty principle - as soon as we measure something we change behaviour. But this applies to peer review also - in order for a peer review system to be open and transparent they need clearly defined guidelines as to what they are judging as good quality, reputation, etc. These guidelines then become what people try and meet, thus changing their behaviour. In some ways my proposal was precisely because at the moment such guidelines aren't well suited to dealing with the type of activity that takes place online, we could do with some other measure also.
And I still think because someone doesn't write 'distance learning' specifically in their posts doesn't mean that the subject matter itself isn't of relevance to a range of distance learning topics.
But we aren't going to agree I guess - if you're at ALT-C perhaps we can settle it with a game of rock-paper-scissors?

[AndyP]

@PeteJ, thanks for the correction. Whatever the technical situation, it feels (to me at least) that commenting on someone else's post adds less to my online 'output' than does writing a new blog post of my own. If I was worried about my output being 'measured' (in any sense) then that might influence my behaviour in ways that were primarily designed to meet the needs of the 'measuring mechanism' rather than the needs of the 'conversation'. That wouldn't be a good thing [tm].

@Martin, the funny(?) thing is that I wrote my blog post partly for the reasons I outline above in response to @PeteJ - I wanted the intellectual effort that I'd put into commenting on your posts to turn into something more (in terms of my own 'output') than just comments on someone else's blog. To take this to its most trivial level, it's easier for me to report to my boss that I've written X blog posts this month, than it is to report that I've spent time commenting on your blog - even if the latter is a more effective use of my time. This is a shame but it also reflects the reality of life :-(

I'm less worried than you about us agreeing to disagree. As I said before, I think we want to get to roughly the same place - we just want to start from different places. That probably isn't a bad thing anyway!