I’m writing this after having read Christina’s post on her thoughts on the PhD process, and following a few of her links to other PhD students who are blogging their own progress. As I’m going through a little slump at the moment, I thought it might be useful to write a short post on where I’m at right now, to review what I’ve done so far.

A few weeks ago I spent 3 days on a writing workshop with colleagues in my department who are also registered for their PhD’s (there are 4 of us), where I worked on my systematic review (see the proposal). I managed to trim the original 103 articles that I gathered during my first, second and third search rounds, to about 60. Then I went through those 60 with a more critical eye, removing what wasn’t appropriate. Finally I narrowed the list down to 20 articles that we eventually conducted independent critical reviews on, and came to consensus with my supervisor, where we finally agreed on 7 articles that matched my inclusion criteria. The article is now ready to be written up, although I’m uncertain of the format. The outcome of the systematic review will be a peer-reviewed publication that identifies some of the ways in which blended learning has been applied in clinical education, and which will inform the development of my own module (one of the later objectives).

My fourth year research group has just finished capturing the data they gathered from a survey we drew up together, where they looked at the role of social networks to facilitate reflective learning. This survey forms part of my first objective, as well as the first component of my SAFRI project (which will later include focus group interviews with staff members, and an additional survey of the students). Immediately after conducting the survey, I have also held workshops with 2 classes so far, to facilitate the process of working within the network, and will be completing workshops with the last 2 classes in the next few weeks. Tomorrow the group will submit an outline of the first few sections of a draft article, and I’ll be presenting some tentative results at the SAAHE conference next week (see the abstract).

I’ve also recently finished a first draft of an article based on a small, wiki-based project I ran in our department last year (you can still comment on it). Strictly speaking it’s unrelated to my PhD as it doesn’t fit into the proposal, but is still work in a related field. Finally, I gave a presentation on PLE’s to the Centre for Teaching and Learning at Stellenbosch University. Again, PLE’s are not explicitly addressed in my PhD proposal, but as I’m leaning more and more towards that concept as having great potential in reflective learning, I think it might ultimately end up playing an important part in the project.

Now that I look back at my progress over the past 6 months, maybe a short break is in order…?

This reflection on the role of feedback in education is based on a mailing list discussion as part of the SAFRI programme, as well as on a few of the assigned readings. I thank the organisers of the session, as well as all the participants in the conversation.

Guidelines for effective feedback

The assigned readings seem to focus on providing readers with a list of guidelines for providing effective feedback and while this list is not exhaustive, it serves as a starting point. Feedback is better when:

It is promptly given

It is specific to the task being assessed, and to the objectives that were set (of course, this assumes that the student has set objectives for the task)

It is performance-based i.e. the feedback is along the lines of what you did, as opposed to who you are. One participant made the point that while it is performance-based, it is done to improve performance, and not to evaluate it

It does not focus on too many things

It helps the student plan for future learning

Feedback should seek to encourage self-assessment, self-reflection and self-awareness

Should / Could be better received if the supervisor / clinician allows the student to comment first e.g. What did they think went well? What did they think could be improved? (one participant said that “…a lot of feedback that I have to give to students is actually on written assignments [which]…means that we cannot start with the student’s agenda. There is no interactive discussion, no possibility to listen, respond to non-verbal cues, observation, etc.”

Positive feedback should be given first (builds self-esteem and encourages better performance)

This positive feedback should not be followed with a qualifier e.g. “You did well, but…”

“Listen and ask, don’t tell and provide solutions”

I’m glad someone pointed out that this (long) list is wonderful in ideal situations but is impractical in many real-world situation, which is where we find ourselves most often e.g. large student:teacher ratios, overloaded faculty and limited time to spend with students. On further reflection, it seems clear that this issue is not specific to feedback, and affects almost everything we want to do as teachers. The problem with teaching is that the way we want to do it (i.e. small groups with focused attention) doesn’t scale very well and we need to come up with a fundamentally different approach to teaching. My own view is that the internet and various associated technologies can enhance communication in ways that do scale, and that therein lies part of the solution.

Complexity in feedback

My own initial thought on a list of guidelines was that it was a wonderful “how to” for providing effective feedback to students. However, as I progressed through the conversation, I began to have my doubts, starting with this one. I worry that having a list of guidelines may deceive us into thinking that if we follow the list then we’ve given good feedback. This is like a student thinking that following a list of questions is a good way to conduct an assessment. Giving feedback seems to be a dynamic process, affected by context (e.g. social / cultural background, type of placement / task) and an understanding of the person/ality receiving it. Someone suggested that the complex process of giving / receiving feedback was also about the identity and character of the receiver and that “…in order to protect the integrity of their beliefs and knowledge, [they] will reject corrective feedback and find ways to devalue it”. Some studies have identified the importance of eliciting thoughts and feelings before giving feedback, which might go some way to alleviating this.

One of the participants insightfully related this back to the MBTI session in March, where it was clear that “knowledge of personalities is useful when giving and receiving feedback. What we know about ourselves and others is important feedback management”. In relation to this, another participant raised the point that “…the issues of culture, gender, religion, belief, age are very important in feedback. Also, feedback for average students is different than feedback for failing students [as well as for] students who think they’re doing well [but] are actually failing”. I think it’s important to note that feedback is dynamic, contextual and complex.

Inappropriate feedback

I found a common theme in the conversation that went along the lines of “feedback drives learning”. This may be a matter of semantics but I’d like to challenge the idea that it does drive learning and suggest that it can drive learning. This may seem pedantic but I think it’s important, because when we say (or imply) that it does, we’re operating under the assumption that all feedback is equal, which it clearly isn’t. This was pointed out by several participants, who suggested that feedback can be inappropriate “…because of how, when or where it is presented”. Some teachers seem to be guilty of using feedback to highlight their own skills and knowledge, while at the same time making it clear that the student lacks these things (or in one horrifying example, actively humiliating them).

When using my own experiences to make a point, I often include examples of my mistakes. I think that as role models, we need to model our failures (and to elaborate on how we moved past them) as well as our successes. Students seem to have an idea that we’re infallible, which unfortunately makes them believe they must be too. If we can highlight that we’re also subject to errors of judgement and prone to forgetfulness (a big one for me), we show them that we’re human and go a long way to establishing trust.

Feedback as a skill

There was a suggestion that giving “feedback is a skill that has to be learnt” and that we should emphasise its importance. This was taken further with the idea that receiving feedback is also a skill that needs to be learnt (e.g. listening, reflecting, analysing), and that we need to spend a lot more time preparing students to receive feedback effectively and with the right attitude / mindset. One participant spoke of students who receive feedback defensively, negating it whether it was appropriate or not. I liked the idea raised that feedback should not only emphasise knowledge, but also more generic skills like effective communication, conflict resolution, etc.

Feedback in teaching and learning

It was pointed out that feedback is not only about a student-teacher interaction, it can also be between peers or colleagues. In fact, feedback is “…for helping all who are interested in self development and actualization of goals, not only students”. This suggests to me that feedback between peers could be an important component of peer teaching, an area of that I’m increasingly interested in.

Feedback should also be given to students who are performing well. This will help to dispel the notion that feedback = criticism. It was pointed out that as teachers, we often have a tendency to focus on the student’s weakness (I know that I’m guilty of this), possibly as a remnant of our own experiences of being students, when this is how we received it. One of the dangers of focusing on the negative only, is inducing a lack of confidence on the students part, where they become incapable of identifying their strengths. Thus, feedback should be used to highlight strengths as well as weaknesses, in order to promote learning.

Feedback and evaluation

Before beginning, see the third point in the guidelines. All too often (as was pointed out by some participants), feedback = marks, and there is no action required after receiving the mark (or if there is, the response is along the lines of “What must I do to get a higher mark”). I wonder if we’re not the problem. We make the assumption (and model behavior showing) that feedback and marks are related, whereas they don’t need to be. Marks are quantitative, while feedback is qualitative. Marks are summative, feedback is formative. Apples are apples and oranges are oranges. When one person is evaluating another, there’s no real objectivity, and so quantitative measures of competence don’t seem to me to be a good fit.

We should distinguish between formative and summative assessment, and their relative relationships to feedback. In formative assessment, feedback is essential as its nature is to facilitate learning. In summative assessment, feedback is irrelevant because the nature of the examination is to evaluate, which we’ve seen is not the role of feedback.

I think we need to get away from this idea that feedback and marks are necessarily related. Of course they can be, but it doesn’t mean they must be. We have to disconnect feedback, which is about learning, and marking, which is about evaluation (and not a very good form of evaluation at that). If students are “marks driven” it’s because we’ve put marks at the centre of our curricula. How do we de-emphasise marks…well, by not marking (it’s radical, I know).

Feedback and reflection

In order for feedback to be effective and of any value, we must first identify the relationship between the task, the feedback, reflection on the task and feedback, and finally, acting on the feedback. Without first making sure that the receiver understands this, the feedback is of no value. We know that reflection drives deep learning, understanding and professional development, yet we leave little space in the curriculum for structured reflection (and forget about teaching students how to reflect). It was great to read the comment, “we do not learn from experience, but rather from reflecting on experience, and feedback must facilitate this reflection for the student”. I would argue that unless the feedback results in a behavioral change, it is ineffective. Of course, if the student cannot focus on anything but the mark, we clearly haven’t established for them the relationship between the task, the feedback, the reflection and the action.

“No feedback” as feedback?

There was a question of whether “no feedback could also be feedback”, and I agree with one response which stated that it was “…the most negative and most useless form of ‘feedback’ in that it borders on pure and simple indifference”, as well as being “regressive and inhibiting”. I’m reminded of a line from “A man for all seasons]]”, in which Thomas More states that …”the maxim of the law is ‘Silence gives consent'”. When I was a student and was not explicitly told to improve (and to do this), then it meant there was nothing to improve, which I found disturbing, knowing what kind of student I was. I worry that if we give no feedback, we risk the student believing that their performance is exemplary (which it may very well be, but then tell them that), or worse, that they don’t know what to believe, leading to anxiety and confusion.

Feedback as a form of academic literacy

I’ve also been thinking about feedback as a form of literacy, which was touched on by one participant, who suggested that “…conversation is a form of feedback, a sort of negotiation of understanding in which mutual feedback becomes recursive leading to shared understanding. In this way, feedback is not one person who ‘knows’ helping another person who doesn’t ‘know’, rather it is mutual. By asking myself questions before and during the feedback, I am learning at the same time the other is learning”. I would take this in a more general direction and suggest that feedback is useless unless the student is at least familiar with the culture of the tribe, which includes the language and conventions we use.

We can give as much feedback as we like, but if we’re speaking a language the student doesn’t understand (I don’t mean English, etc.), we’re wasting everyone’s time. And in case you think that by final year, our students understand our “tribe”, think about this simple example: The first year student is starting out in the culture of higher education, the second year student is starting out in the culture of physiotherapy, the third year student is starting out in the culture clinical practice, and the final year student is starting out in the culture of being a physiotherapist. We need different languages and approaches for each of these different cultures / literacies / “tribes”.

Final thoughts

For most of the discussion, feedback was treated as a “thing”, rather than a process or interaction. My own view is that feedback is neither “given” nor “received”, but is a process that people participate in

How many of us ask our students for feedback on our feedback?

How many of use ask our students for feedback on our teaching? One participant had this to say “Should each of us offer the students who receive our teaching the opportunity to give feedback on our teaching as part of their learning? This may be a little threatening, but on the occasions that I have done so in the past it has mostly been encouraging and affirming. Occasionally uncomfortable.”

If we do ask for it, how many of us reflect on it and make an associated change in our teaching practice?

In my department we have bi-monthly research meetings, where we report back on our research activities (we get dedicated research time and so need to account for how we’ve used that time). We’ve recently changed the format of the meeting to include a topical discussion or presentation, rather than simply reporting progress back to the group.

Today was our first session in this new format and we discussed some of the issues around supervision. Everyone in the department is responsible for supervising either a group of 4th year physiotherapy students as they go through the process of conducting a research project, or a group of postgraduate students who are working on their degrees. Here are some notes I took during the session.

Contracts are useful, not only between the supervisor and student, but between the students themselves (in cases of group research). We often find that some students don’t participate or contribute to the process as much as the rest of the group, which has the potential to cause conflict, especially when marks are involved. While conflict does give the opportunity to work through conflict management processes with the group (not desirable, but still a learning opportunity), a contract that outlines responsibilities, roles, expectations, accountability and consequences, may go some way to reducing it’s likelihood.

The issue of supervision style was raised, and it was pointed out that often supervisors will have a different way of approaching supervision. In addition, students have different ways that they want to be supervised. It seems that managing expectations in terms of communicating personal preferences is important to begin with, and probably to keep reviewing during the process.

As a research supervisor, it’s important to have prepared a plan for moving the project forward. At PhD level, it’s largely the responsibility of the student to manage this, but undergraduate and Masters students (generally) aren’t ready to completely manage their own projects. One useful project management tool that was suggested was to use Gantt charts to plan the project over time.

Finally, we need to go over what “academic writing” actually means. Students are often under the impression that it’s about using big words and complicated sentence structure, when all we’re really looking for is argument construction and defense. Academic writing is about conveying ideas in simple language, something our students don’t always understand.

I enjoyed the session, and although it lacked structure, it gave us the opportunity to discuss the issues faced during the supervision process. It’s always nice to be able to draw on the experience of professors and seasoned researchers, and I’m looking forward to our next sessions. I’ll be presenting a session on systematic reviews with a colleague in 2 weeks time, and will be putting that up here too.

Edit: I just had a colleague point out that we also discussed the role of feedback in supervision. I haven’t included those details in this post, because I have a more comprehensive post dealing with feedback that I’m working on at the moment, which can be found here.

Earlier today I had a productive meeting with one of my supervisors where, in addition to updating her on my progress, we discussed my reluctance to get “too involved” in my research projects. This is something I’ve touched on before and will probably continue returning to it during the course of my research.

I come from a quantitative background where many believe that the RCT is the gold standard of research, and where the focus on objectivity and distance is meant to reduce or avoid researcher bias. My supervisor, on the other hand, has a good understanding of participatory action research (PAR), in which researcher interaction is not only acceptable, it’s essential. While PAR is the methodological approach I’m tending towards, I still don’t really have my head wrapped around it (note: I’m not trying to compare RCTs and PAR, or to say anything substantial about either one).

I keep wondering “how much” engagement is appropriate before I start to influence people, and I keep having to be reminded that it’s not the “correct amount” of interaction that I need to worry about, but the objectives I’m trying to achieve that should be my guiding light. I can (should?) influence the interactions with my colleagues and students, as long as I’m documenting not only the interaction, but my own progression and reflection. To be honest, this is new ground for me and it’s not a process I’m entirely comfortable with. It seems very…messy.

Yesterday I had a meeting with my supervisor to discuss the assignments I’m going to run as part of the first objective of my PhD. Together with a systematic review and a survey, I was interested in using student and staff participation in a social network to derive additional data that would help me form a baseline understanding of their attitudes and skills around teaching and learning practice, as well as establish the level of digital and information literacy within the department.

After joining the SAFRI programme, I incorporated the social network idea into my SAFRI project, but unconsciously ended up with a different agenda. Instead of using the network to highlight potential problem areas and the challenges of teaching with technology, it morphed into me trying to demonstrate the effectiveness of using a social network to facilitate reflective practice. In hindsight, it’s clear that the 2 projects were at odds with one another, and the objectives were definitely not aligned.

When my supervisor pointed out that there was inconsistency in the 2 projects I really struggled to accept it. I was adamant that my methods were fine and she suggested that I hand over facilitation of the assignments within the network to other staff who didn’t have such a high personal stake in the success of the project, and I strongly disagreed. I found several reasons to explain why I had to be the person to run it, the strongest of which was that “…no-one else will try as hard as I will to make sure it works”. Which kind of made her point.

When I went away and thought about our conversation I reviewed my objectives for the 2 projects, and then it was clear that they really were 2 different projects. One was suggesting that this would be a useful tool to describe the current state of affairs, which I know will be less than ideal. The other was intent on proving that the network would be a positive tool, rather than describing what would happen if we just incorporated one into the department.

After the painful realisation that I’d let my personal desire for this project to succeed override my objectivity as a researcher, I agreed to let others lead the social network assignments, with guidance from me. This will greatly reduce the impact of researcher bias, as well as synchronise the objectives of the 2 projects. As it stands now, it will more accurately describe the state of the department in terms of attitudes and skills around teaching and learning, and the levels of digital and information literacy, which will give me valuable data that will inform the next objectives of my study.

This was a great learning experience for me, and a warning of the dangers of getting too close to one’s project. There are some situations where the researcher can be an integral part of the project, but this experience has shown me when it would be detrimental to the process.

Day 2 of the first residential session felt really productive, although I’m not sure if that was because I worked hard or because I spent most of the day staring at my tiny netbook screen (I love my NC10 but it’s definitely not for 8+ hour shifts). Before beginning this session, each of use had to have some ideas around a project that we’d be running over the next few years, as well as an idea for a study to run within that project.

The main theme of today was to make sure that we could conceptualise the research and project as 2 components of an integrated work, and to create strong links between the various parts of the study. The aim of the SAFRI programme is to develop leadership skills and research capacity within the domain of medical education in Africa, so the research project is a critical part of the programme. I was surprised to find that even though the emphasis on doing good research was only a recent development within the programme, it already seems to have been tightly integrated.

Most of today was spent writing (laptops allowed) and working closely with our supervisors (each Fellow is assigned a supervisor who helps to guide their project and studies). For anyone who joined this programme thinking it’d be a nice afterthought to add to their CV’s, today would’ve put that idea to rest. It’s intense, and there’s a lot of pressure to produce a good piece of work.

Like this:

These are some of the notes I made during the presentation on preparing a proposal, but include some points that relate to the general process of conducting research

There are 3 things a proposal should try to address:

What? – What contribution will this research make? What is it about? What do you want to study?

Why? – Why should be bother? Why is it significant? Why do you want to study this?

How? – How are you going to study this? What tools and techniques will you use? Who else will be involved? How long is it going to take?

There is no “one way” to write a proposal, rather conform to the norms of your department. How do you find your “voice”, and what is the voice of your department? Bear in mind that the proposal voice is more tentative and uncertain than the thesis voice, which has defiinite ideas to convey

Research = Inquiry

PhD research = Making a contribution to a field, linking it to what already exists

Your research question should be real i.e. the answer is not readily available

Hypothesis = “a bold guess” → I think the answer could be…

Research journey

Ask a question

Begin reading (the “river of words” – who else has asked this question, where do they live, what did they find i.e. what is already out there, what already exists

Could lead to the question changing → you find your question has been answered but there are other, related questions that need answering

Interact with others who are looking at similar questions

Research design is central, may include a redesign following a pilot

Be aware of gatekeepers and how to get around them

Know when to stop gathering data, and when to start analysing it (if this is addressed in part in the proposal, it can give guidance to this process)

Following the analysis, you may have to adapt or even to discard some ideas if the data doesn’t support it

Be careful of despair when your ideas aren’t supported

Following analysis comes synthesis (this is the hardest part where many candidates drop out) → putting the thesis together / linking all the ideas

During the writing / rewriting process, you may have to return to the “river of words” to review significant contributions

What is the contribution of the editor and supervisor of the final document?

What do you have to say when all is said and done i.e. what is your contribution to the current understanding and knowledge base of your field?

Be wary of those who presume to “know it all”. How will you defend the unique nature of your research?

Differing interpretations of the same data can be a result of using different theoretical frameworks that underly the analysis

The proposal should lay out what the researcher wants to do, but should also include limitations i.e. significance (what it will include and why that’s important) vs. limitations (what will be excluded and why)

Negotiate with the supervisors as to what wasn’t done and write it up i.e. explain to examiners and readers what was left out of the study

The literature review is being written and rewritten throughout the process, because you’re reading throughout the journey

Make sure the research question is clear and concise. What is the background to the question? Why is it relevant now and didn’t arise 10 years ago? What makes “now” a good time to try and answer the question?