Main menu

Sub menu

What goes wrong when students reflect on their research methods

In 2017 Elena and I ran the IT3010 course at NTNU on the topic of qualitative research methods in computer science. This was the first time we had the course run as a realistic run-up to a research conference with papers presented by students. The last deliverable in the course was a reflection report where each student was asked to reflect on the who process. Here I want to summarize some tips to students after having graded this last deliverable.

Beware! Reflection reports can easily become “bla bla” text that does not say anything of substance. Try to avoid this by reflecting! Before starting to write each section, try to think for five minutes (without checking your social network) about “what do I want to say here?” If you don’t, then you end up with bull*t and it is very easy for your professor to see that.

As previous point, a general comment is that the reports are not substantiated and don’t related to the course text book. The reflections become shallow when they relate only to personal opinions of the writer. Having personal opinions and reflections are central but they need to be substantiated. Ideally you need to write your report with frequent references to the text book, and reflect on what worked as the book said, what did not work that way, what you did as consequence, etc. For example, did you follow the text book guidelines when doing the initial literature study? If yes, what do you think about that process? If no, what process did you follow and why, and what was the result?

Unfortunately most of your reflection reports read like a chronological description of “what happened.” They lack real reflection. One way to fix this can be to think “what happened” first, then “why did it happen that way”, then continue with “what was my role in that happening that way” and “what other options could have been available,” “what did I learn in the process” etc.

Some reports listed a lot of things that did not work as intended, without doing a reflection on why that was the case and what else could have worked. For instance, if you did a survey and with hindsight you say it did not work, try to write a little about why, and why for instance a case study would have been better.

You reflect a lot on your contribution. But this reflection is often limited to describing the deliverables and how they can be used by others, especially practitioners. In research you need to take care of the novelty of your contribution as well as its usefulness. If you made a good deliverable it is not good if someone else had already made that deliverable. A discussion of contribution of research contribution therefore needs to reflection on how novel the contribution was in addition to how useful it was.

The part on research strategy often fails to answer exactly why you chose the specific strategy, and what were the alternatives. For doing this you need to go back to the formulation of the research question and show why e.g. survey was a good strategy and case study was a bad one. For instance, if your RQ is “what is the prevalence of using sharing platforms among Norwegian students?” then you can easily say survey is the way to go because you want to investigate prevalence (how big is the percentage of students using the platform). But if your RQ is “what are the benefits of using sharing platforms among Norwegian students?” then it is not obvious that you should use a survey. You might rather answer it using case study, ethnography, etc. You need to confront your choice with other available choices. Your reflection should also take into account access to users, type of data generation you can practically use, the length of your project, the resources you have, etc.

The previous point applies equally to your data generation methods. Most reflections lack a part on how the chosen methods helped address the research question. They become details of how the methods were used, and don’t mention how well they helped answer the research questions, what were the alternative methods, and what could have been done differently, if anything. A discussion of the data generation method in the context of the strategy is also a plus –e.g. if you used questionnaire data as part of a case study, or structured interviews as part of a survey.

A lot of the text in the reports were about details of data collection and analysis. Try rather to look at the big picture. For instance, in addition to writing what data generation tools you used try to see if you can reflect on the overall approach and whether it could have been done differently. Or, instead of writing in length about how much time it took to arrange a meeting, try to see if the data answered your research questions, etc. This citation from one of the reports is an example of a high-level reflection that we are looking for: “I believe that much of the difficulty in developing an explicit research contribution stems from the haphazard way the final research question was created, and how the question was created to fit the report, rather than the report being a result of the question.” Or the following reflection from another report: “Reading the book more and studying the theory would have helped the most. If I knew about all the strategies and data generation methods more thoroughly, I could have suggested more and different solutions to the one we ended up using…When choosing research strategy and data generation methods we didn’t plan enough what to do with the acquired data. Too late did we start to actually analyze the data and draw conclusions..”

Many groups experienced some challenges regarding the distribution of the work in the group. Some suggestions from some of the students were: 1) open communication about and with those who do not contribute, 2) fixed group work time from the start of the course, 3) creating a group contract, i.e. roles and who is responsible for what part of the work, 4) more pressure on participation when research questions and the research plans are made so that you get buy-in from all the members, 5) clarification of action items, maybe using tools like Trello.

Some reports go into details of who did what and who did not do their part of the work etc. We would prefer that this type of detail was not in the report, especially references to other group members using their names. It is a much researched fact that each one of us thinks he/she does a much better job than everyone else. So you don’t need to write about it!

Most reports refer to the fact that experiments, design and creation, ethnography would have required much more time to do than a survey. Note that in a real world situation this might be true. But it is totally possible to set up small design and creation research projects –for instance a co-design workshop with some users. In fact some of the tasks would have become much more interesting if other methods than survey were used.

The reflection on the presentation at the student conference in most of the reports is a description of how the PPT presentation was developed. This is OK. What is missing is a reflection on what it meant to participate in the conference, the process of presenting your results, of being exposed to questions. Almost all reports missed a reflection on what questions were asked and what that meant to their research, which is the main point of participating to a conference.