« How uncool? Repository URIs... | Main | A National Research Data Service for the UK? »

March 05, 2009

JISC bid evaluation process - some suggestions

Brian Kelly has a blog post about the use that was made of Twitter during the writing and evaluation of a recent JISC call.

I can't comment on the writing side of things, because I didn't do any, and I haven't done any analysis on the way Twitter was used as part of the evaluation process.  However, I do admit to getting a little hot under the collar about over-length bids and I did publicly vent my frustration on Twitter, which resulted in a little discussion.

Actually, it was all a storm in a tea cup as far as I was concerned because, somewhat embarrassingly, I'd mis-read the email telling me which bids I'd been assigned to mark and the one that was over length (~18 sides rather than 12 I think) was one that I wasn't even supposed to be looking at!

Doh :-(

Anyway, for what it's worth, here are my suggestions for the evaluation process in the future:

  1. Set a hard rule, maximum X pages at minimum Y point size and stick to it.
  2. Make the page length include everything: cover sheet, FOI statement, appendices, etc. That way, there is no room for argument.
  3. Require that all bids are submitted as 2 PDF files (yes, I know, I normally say I hate PDF but in this case, having everything look exactly the same is important), one containing the bid itself (as per the above rules) and one containing copies of all the letters of support.  That way, evaluators don't have to print out all the letters of support unless they really want to.
  4. Automatically reject any bid that doesn't stick to these rules (i.e. this is an admin job).

I know this sounds harsh but the point is that if you relax the rules you end up doing so in slightly arbitrary ways (i.e. you do it for the people you know but not for others or you have some vague unwritten cut-off in your head about what is acceptable).  Further, if you allow some long bids, you are effectively being unfair on those bidders who comply with the rules.  Finally, if people can't write a bid that sticks to the rules, what are their deliverables going to be like?

If you send the long bids thru to evaluators but tell them not to read past a certain page (which is current JISC practice I think) then you leave individual evaluators with a difficult problem.  In the case of the bid that I was marking but not supposed to be marking(!) for example, the information about partners was in the additional pages but I knew them well anyway.  What was I supposed to do?  Give them a zero score for that section or give them the benefit of the doubt?

Note: I'm not suggesting here that there is anything significantly wrong with the current process and in the main it works well.  But I did find this round of marking more difficult than usual (the nature of the 12/08 call was quite complex) and I suppose that highlighted some issues for me that I've probably ignored in the past.  It didn't help that I'd left all my marking to the last minute (as usual)!

TrackBack

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8345203ba69e20112793733ae28a4

Listed below are links to weblogs that reference JISC bid evaluation process - some suggestions:

Comments

Wow this is so different from my first pass at bid evaluation! Good to know the other side, I think Paul mentioned also printing stuff (which I think is what you are saying is the most annoying aspect for you?). For me it was an entirely online affair using FoxitReader which allows for annotation on PDFs. Though I use three screen when doing this stuff plenty of screen space.

For me it has got to be actually breaking PDF all together and getting into an HTML submission process. We could do so much more if the data was marked up in specific fields. For example being able to text-mine the "standards used" area or the "methodlogies" topic. Let alone the metadata we could mark up for name/authority fields and especially financial/budget numbers.

Also, the page thing seems to be moot when every word processor has a word count, why wouldn't we use a word count?

I really am curious as to other methodology on how they mark stuff up, hopefully this post can act as a gumtree?

I think there are two separate problems here - bid length and how much you discount what you already know about the bid and people bidding. For example, if some people you knew wrote a bid of the correct length but didn't include bios etc. what would you do?

David,
I did mean to suggest word-count rather than page-count but forgot. Agree that this is a perfectly sensible alternative.

Re: HTML rather than PDF. Again I agree (from a theoretical perspective) but think that you must live in an alternate reality somewhere? Can you imagine the hell-hole of broken HTML rubbish JISC would attract (mostly exported from MS Word I suggest) if they moved to HTML submission?

Deary me...

BTW, please note the timestamp on my 'bollox' tweet - 7.16am on a Sunday morning. That wasn't some kind of timezone glitch - that was genuine. No wonder I was so grumpy!

Juliette,
you're right... I'm mixing up issues to a certain extent. In the scenario you present I'm not sure what I would do tbh. But given that bid marking isn't anonymous I'd probably (I know I shouldn't admit this but the reality is that it is the nature of the beast)... I'd probably treat the bid slightly different depending on whether I knew the bidders or not. It's inevitable isn't it? Either that or I'd return it unevaluated on the basis that I couldn't mark it impartially. Yeah, that's what I'd do (unless I'd left everything until the Sunday morning before the bid marking deadline of course!). Doh :-(

Re "Can you imagine the hell-hole of broken HTML rubbish JISC would attract (mostly exported from MS Word..." <- LOL! Hell hole indeed! Though in this case just making the submission field strip all mark up would seem to be easy enough? Then again, I am not against PDF so long as the UTF-8 is encrypted in the layers. <-- Did some fun stuff with Yahoo Keyword term extractor and Wordle that helped me process the bids in the overall context.

Ah, interesting... I was going to suggest that someone should do a Wordle of all the bids submitted under the 12/08 call - but then wondered if there was some confidentiality issue that might mitigate against doing so? I think the result would be of potential interest to the community.

TO a man with a hammer, everything looks like a nail. To a man with a repository, every process looks like a workflow on a deposit.

JISC should make more use of their repository: all bid submissions should be DEPOSITED. The automatic ingest procedure could count the number of pages and refuse to accept it. The letters of support could be uploaded separately.

Andy, of your 4 suggestions I think 1 (make a rule and stick to it) is there already. It's just that there's disagreement about where the 'sticking' is done. 2 isn't workable, as one of the sheets (the FOI form) can potentially be multi-page for reasons that have nothing to do with the quality of the bid. I'm neutral on 3. But I disagree about 4. The current position allows for bids to be rejected at any point for not following the rules. If the calls were hugely over-subscribed, then rejection at the submission point, or shortly thereafter, would be sensible. But we often aren't in that position.

Personally, I ignore everything beyond the page limit and I would not have a problem if the extraneous pages were stripped before I saw the bid. I make it clear in my comments that I've done this. Sometimes, there's still a cogent case for a project in the first 10 pages and I think it deserves a good mark for that. In other cases there's so little left that I just mark it as a C because of insufficient information.

But evaluation panels need the flexibility to either throw out all bids which are over limit (and I've been on panels which have done that) or accept and even fund such bids on the grounds that, even truncated, they are better proposals than the others. The problems - such as missing budgets - can easily be addressed. The rules are clear, and the rules say that the Executive can choose when to apply them :-)

The comments to this entry are closed.

About

Search

Loading
eFoundations is powered by TypePad