Sunday, October 26, 2008

Performance Evaluation Strategies

Cross-posted from my other blog. Apparently, aspects of the performance tracking system I've developed might be useful to a wider audience than I'd anticipated!

Here's the text of an email I sent to a brand new (and very bright) graduate student recently. The student had asked me to "have a quick look at" a one page proposal for a small internal stipend competition.

---------------------
"Hi [name]

The bulk of the application is fine – I’ve attached an edited version with track changes on so you can see what I’ve done.

In addition to these small changes, I think the first paragraph needs some more detailed work. The major problem is that the hypothesis is extremely broad and doesn’t reflect what your experiments are actually going to address. [Using Method A] isn’t going to determine whether [very broad hypothesis, worthy of at least one Nobel if eventually proven] is actually true, so you will need to tighten up the hypothesis. Instead I would frame it in terms of your sentence on [description of what you are actually doing] – so something like this:

Hypothesis: that [Outcome B] can be derived by [using Method A], and that this [outcome]reflects the heterogeneity of [Behaviour C] in response to [Condition D].

Feel free to rewrite that sentence! But this is a better reflection of what your experiments will actually be testing.

The hypothesis is currently hidden in the middle of the first introductory paragraph. You need to make it stand out more: put it in bold type, at the end of that first paragraph. I also think it would be better to change the emphasis and order of the sentences leading up to the hypothesis – this very short introduction needs to be intensely focused on the content of the hypothesis and research proposal, with each concept leading logically to the next. At the moment it is a bit choppy, with no obvious connection between adjacent sentences. You need to lead your reader through this section since they might not have the background to make these leaps without guidance.

This order might work, but again please do play around with it until you’re completely satisfied!

1) [Very basic, established fact] (introduce the idea of [Outcome B])
2) This leads to heterogeneity in [Behaviour C]
3) Possible connection between the [heterogeneity of Behaviour C] and response to [Condition D]
4) The problems caused by using [current] approaches that [suck are outdated]
5) Therefore the superiority of [Method A].

Then the hypothesis.


I’d be happy to take another look at this proposal, and especially the first paragraph, once you’ve had the chance to make these edits.

Good luck!

Cath

---------------------

In contrast, my work on the actual proposal sections of the last three reworked / resubmitted grants we put in, with a combined budget well into 7 figures, mostly involved the following:

- Correcting typos
- Standardising the use of alternative spellings (not yet automated, alas)
- Correcting verb/subject disagreements (including "the data is", a pet peeve)
- Correcting preposition use by ESL writers (things like "we will respond from situation X by...")

As I mentioned before, this last round of submissions was not typical in that there were no new grants. But even when I do get involved with new grant applications, the same Catch 22 situation arises:

  • I am at my most useful when helping the people who have the least experience in preparing grant applications, and less useful when helping the experienced senior PIs.
  • The Big Decision (whether to try and find the money to keep me when my contract runs out) will be made by several of the most senior PIs.

What to do?

My strategy is in part adapted from my time in industry, where no job is ever truly safe, and has since evolved to better suit my current job. It is all very different from being a postdoc, when productivity is much more obvious, there are no scary formal reviews, and you know you are SOL after three years anyway!

1) Record keeping.

As I've mentioned before, I have retained the "lab book" habit from my time in the lab. I write all my activities, grouped by project, into a notebook at the end of each day. It's quick and relatively painless; unfortunately the quality of my record keeping tends to suffer in the frenzy of deadline week.

Every month or so I transfer all of my scribbles into an Excel spreadsheet, which contains all of my grants, manuscripts and other projects (for there are many). I have columns for dates, agency/journal, PI, title, funding/acceptance status, and - most importantly - my contribution. This is currently just a string of activities in an unformatted list; I once tried to develop an easy check list system, but with every grant and every PI being different I found it impossible to define consistent categories. Regardless, I should be able to pull out these data (please note correct verb/subject agreement) and insert them into a written progress report or PowerPoint presentation within an hour or two, although I have not actually had to do this yet.

I also archive all received and sent emails, sorted by project, and file hard copies of, for example, the PI's original draft of the grant proposal with my red pen corrections on it.

2) Blowing my own trumpet.

This is the tricky part for me! I mean, I'm English. I would rather keep my trumpet blowing to formal reviews (and blog posts, heh), but I do make myself drop my most significant achievements into conversation (e.g. "Oh by the way, Dr X got her grant" to my immediate supervisor at the end of an unrelated conversation). I hate to contribute to the flood of emails we all get, but I will CC people if appropriate; I copied the student's supervisor (a senior PI who was out of town in the week leading up to the deadline) when I sent the email above, for example.

I've also commandeered the large whiteboard by my desk, which is seen by everyone who visits me and/or my immediate supervisor. As well as a list of everyone's vacation dates and upcoming grant deadlines (its original purpose), it now has a list of "Grants Under Review", with agency and PI listed. Rather than erasing them once the decision is made, I keep them up for as long as I can, complete with an indication of whether they were funded. I just had to erase the older grants at the top of the list to make way for the latest batch of submissions, but there were a good number of successful applications listed up there for the last few months.

Sometimes someone will help me out, for example by copying one of the senior PIs on an email in which they thank me for helping them with their project. This is a rarity though and I can't rely on it.

3) Covering my ass.

This is where my industry experience comes in!

I email a copy of my Excel spreadsheet (see #1) to my Gmail address after every major update. You know, just in case I am terminated without notice and can't access my work files...

I also keep an email folder in my work account labeled "Feedback". Any time I get any significant positive OR negative feedback, it goes in here, and is forwarded to my Gmail account every month or so. I can use the positive feedback to make a case for keeping / reinstating me, and the negative feedback to help me avoid making the same mistake twice. I know I wasn't the only one at my former company who did this!

-------------

I recently described this system to someone with a similar job to mine, and she thought that most of it was a good idea - although I didn't tell her about some of my more paranoid ass covering! Of course she then told me that her reviews to date have been incredibly informal... but then she's on a permanent contract.

Hopefully my system will help me to convince the senior PIs that I'm worth keeping... I'll keep you posted!

No comments: