Connect. Learn. Get what you need. The CTSI helps research teams work faster and better.

Menu

Category Archives: Evaluation

Dr. Ann Dozier, Lead Evaluator for the University of Rochester Clinical and Translational Science Institute (CTSI).

Dr. Ann Dozier, lead evaluator for the University of Rochester Clinical and Translational Science Institute, contributed to a recent publication in the Clinical and Translational Science Journal entitled “Developing Common Metrics for the Clinical and Translational Science Awards (CTSAs): Lessons Learned.” The paper describes an effort by the national CTSA Evaluation Key Function Committee to develop and test a methodology for identifying common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics.

Dr. Dozier offered this comment: “This project further reinforced the importance of engaging stakeholders (potential end users) in the process of common metric development and how different the CTSAs are in what and how their internal data are gathered and maintained.”

The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

Share this:

Like this:

The CTSI’s latest retreat, held on November 11, concentrated on the theme of “renewing our strategic focus; preparing for action.” Participants were generally people with formal roles in the CTSI and effort support.

First of all, thanks to everyone who took part. We invited 85 people, 70 responded that they would come, and all but two showed up. This, and all the written feedback we received, demonstrates real commitment, and we thank you for it.

I have a thick stack of paper on my desk which contains input from many of you on the questions we asked you to think about. Here’s a review and summary of your input:

We asked retreat participants to provide their thoughts on concepts, ideas or just words that should be included in statements of CTSI mission, vision and values. I read each and every written response provided, whether we got it before, during or after the retreat. I actually read each one several times (and special thanks to all of you who listened to your 3rd grade teachers regarding good handwriting).

I’ve created three summaries of your input. For the first, I reviewed input regarding the CTSI mission statement. I recorded each suggested concept, and counted up the number of times that concept appeared across all of the feedback provided. Then, I grouped the concepts into categories. I admit that this analysis is subjective, but it seems illuminating nonetheless. Here’s the result:

In words: in your feedback, you mentioned resources that the CTSI provides or could provide 26 times. You mentioned various high level goals that the CTSI could pursue about 24 times. And so on. For all of you data nerds, I’ve provided all the details in a separate post (click here to take a look).

After our new Senior Project Research Associate Dr. Kathleen Holt took a look at this first graph, she noted that I appeared to focus on nouns, and suggested that I go back and take a look at the verbs you used in your input. So I did that, and the graph below was the result:

Also very interesting perspectives: somewhat different than the first look, but similar in some ways as well. Again, you’ll find the details in a separate post (here).

Finally, I repeated the process with the concepts, ideas and words you suggested should be part of a CTSI statement of values. Here’s the result:

If you’d like to review the full compilation of responses, click here.

So – what do you think? You can use the “leave a comment” feature to share your thoughts, or email me directly.

What’s next? Over the next month or so the Operations Committee will review this information and their own thoughts, and work toward new or revised statements of mission, vision and values for the CTSI. But this is only the beginning of the strategic planning process. After mission, vision and values comes the development of a customer value proposition, strategic themes and results, objectives, related performance measures and targets, strategic initiatives and execution. You’ll be hearing more!

Share this:

Like this:

In a separate post (here), I provide a summary of your feedback from the November 11, 2014 CTSI retreat. This post contains details, for those of you that want to ponder the issues a bit more deeply.

The table below summarizes feedback regarding a CTSI mission statement. The first column is the category I assigned; the second is the concept or idea as extracted from your feedback, and the third columns shows how many times that concept was mentioned across all feedback received.

Mission Concepts – Details

Category

Concept

Mentions

Collaboration

connection

8

Collaboration

team science

6

Collaboration

collaboration

4

Collaboration

multidisciplinary

3

Collaboration

synergy

2

Community

community engagement

6

Community

integration

3

Community

outreach

2

Community

community centered

1

Population Health

community health

3

Population Health

clinic

2

Population Health

health

2

Population Health

patients

2

Population Health

population health

1

Population Health

prevention

1

Population Health

population

1

Education

training/education

12

Education

mentoring

2

Education

knowledge

2

Help

facilitate

6

Help

help/assistance

4

Help

guidance

3

Help

enable

2

Help

accelerate

2

Help

access

2

Help

catalyst

1

Resources

expertise

7

Resources

funding

6

Resources

resources

6

Resources

service

2

Resources

infrastructure

2

Resources

solutions

1

Resources

tools

1

Resources

support

1

Efficiency

efficiency

5

Efficiency

relevance

2

Efficiency

quality

2

Communication

reputation

2

Communication

awareness

2

Communication

communication

2

Communication

dissemination

1

Character

trust

3

Character

compassion

2

Character

commitment

1

Goals

translation

8

Goals

innovation

7

Goals

discovery

2

Goals

success

2

Goals

opportunity

2

Goals

science

1

Goals

evolution

1

Goals

enhance

1

Another way to look at this information is through a Wordle. In the image below, each concept shows up. The size of the concept is proportional to the number of times it was mentioned. Click on the image to get a larger view.

The following table is similar to the previous one, but focuses on the verbs used in your feedback to describe the CTSI’s mission.

Share this:

Like this:

You see my name in your inbox, and you think to yourself, “Not her again!!!! What does she want now?!?!” I understand your pain, but I admit it: I am a harasser. That is my job. As the Senior Information Analyst for the Clinical and Translational Science Institute, I am charged with the daunting task of collecting evaluation data for a multi-million dollar Federal grant. It is not easy, and it requires information to be collected from many different sources, most likely including you.

Don’t despair! If I contact you for information or to complete a survey there is probably a GREAT reason behind it. Here is a list of the most common reasons that I will be in your inbox:

You’re a faculty member: Congratulations!! All that hard work and schooling paid off and now you are officially a member of the faculty at highly respected University. The CTSI highly regards the opinions of the University of Rochester faculty and often surveys them for awareness of and satisfaction with our services and programs.

You are a funded investigator: Congratulations!! You have successfully obtained a grant, which is not easy feat! The CTSI is interested in the research climate at the University and periodically collects data from you about your projects and research collaborations.

You received funding from the CTSI: Congratulations!! You were picked out of a group of very strong candidates to receive support through our pilot funding or educational programs! The CTSI surveys awardees and trainees twice per year about outcomes that resulted from the support and any success stories you would like to share.

You utilized CTSI programs or services: The CTSI offers a vast array of services and programs that assist investigators at the University in conducting clinical and translational science. Our offerings include expert consultation services, the Clinical Research Center, the Center for Community Health, the Research Navigator Program, and others. The CTSI collects information about your use of these services and the benefits that result, such as publications and grants.

In closing, I would like to say that I am a human being, not an evil robot. I am actually quite nice in real life. If you would like me to stop harassing you, the solution is quite simple. Just send me the data. And thank you for helping us improve the CTSI’s programs and services and show that we’re using federal funds wisely.