The scientific literature is biased. Positive results are published widely, while negative and null results gather dust in file drawers (1, 2). This bias functions at many levels, from which papers are submitted to which papers are published (3, 4). This is one reason why p-hacking is (consciously or unconsciously) used to game the system (5). Furthermore, researchers often give a biased interpretation of one’s own results, use causal language when this isn’t warranted, and misleadingly cite others’ results (6, 7). Studies which have to adhere to a specific protocol, such as clinical trials, often deviate from the protocol by not reporting outcomes or silently adding new outcomes (8). Such changes are not random, but typically favor reporting positive effects and hiding negative ones (9). This is certainly not unique to clinical trials; published articles in general frequently include incorrectly reported statistics, with 35% including substantial errors which directly affect the conclusions (10-12). Meta-analyses from authors with industry involvement are massively published yet fail to report caveats (13). Besides, when the original studies are of low quality, a meta-analysis will not magically fix this (aka the ‘garbage in, garbage out’ principle).

Note that these issues are certainly not restricted to qualitative research or (semi-)positivistic paradigms, but are just as relevant for quantitative research from a more naturalistic perspective (14-16).

This list could go on for much longer, but the point has been made; everybody lies. Given the need to be skeptical, how should we read the scientific literature?

Using reflective reasoning to prevent bias

Reading is simple, but reading to become informed is much harder. This is especially the case when we are dealing with scientific theories. To aid you in this endeavor I will borrow the ‘reflective reasoning’ method from medical education. It has been proven that it There is some evidence that it enhances physicians’ clinical reasoning, increases diagnostic accuracy, and reduces bias (17-19).

Step One. Pick a theory. This can be your own theory, or any theory present in the academic literature. We will call this theory the diagnosis.

Step Two. Now list all the symptoms which are typical of this diagnosis. In other words: which data/studies support the theory? The key step is to differentiate between findings in the following manner:

Which findings support the theory?

Which findings contradict the theory?

Which findings are expected given the theory, but are missing?

Why can this be helpful? Because by our nature we fixate on findings which confirm what we already believe (20). These questions can help reduce confirmation bias and give you a much more balanced perspective on the literature.

If you are not aware of any contradictory or missing evidence then take this as a sign that you might have been reading a biased section of the literature.

Step Three. In addition to the initial theory, list all alternative theories which could potentially explain the same array of findings and again list all the three types of findings, like this:

Theories

Confirming findings

Contradictory findings

Findings which are expected, but missing

Theory A

Findings 1-3

Findings 4-5

Findings 6-9

Theory B

Findings 2-5

Finding 1

Findings 10-11

Theory C

Findings 1-4

Findings 2-3, 5

Findings 6-11

Why is this step so important? Because most finding can be explained by multiple theories, just as any given symptom can be explained by multiple diagnoses. Should we only check whether a particular theory is supported by some data, than any theory would suffice because every theory has some support. In the above example, theory B and C both have the same level of supporting findings, but differ dramatically in the amount of contradictory and expected-but-missing findings.

It is a given that findings can differ in the quality of evidence they provide (from uninformative to very convincing) but also in their specificity; does a finding support only one theory, or does it fit in many models? If a theory is based mainly on findings which are also explained by other theories, it’s not a strong theory.

In the end, a theory is more than the sum of its supporting or contradicting findings. Nevertheless, carefully reflecting on the quantity and quality of evidence for any theory is an essential step for being a critical reader.

Why you should not be a skeptical scientist

No matter how critical or reflective you are, you will always remain biased. It’s human nature. That’s why you should not be a skeptical scientist by yourself.

Step Four. Invite others to take a very, very critical look at the theories you use and write about. In other words, ask others to be a ‘critical friend’. For a truly informative experience, invite them to be utterly brutal and criticize any and every aspect of whichever theory you hold dear, and then thank them for showing you how you lie a different perspective.

Luckily, there just happens to already exist an excellent platform where academics relentlessly criticize anything that is even remotely suspect. It’s called Twitter. Get on it. It’s fun and very informative.

More tips for the skeptical scientist

In addition to the reflective reasoning procedure, here are some more tips which can help you become a more critical, or skeptical, scientist. Do you have tips of your own? Please share!

Play advocate of the devil: For every finding which is used to support a theory/claim, try to argue how it can be used to contradict it and/or support a different theory.

Use these wonderful (online) tools to check: whether there is evidence for p-hacking (21), whether reported statistics such as p-values are correct (22 or 23), and whether reported Likert-scale summaries are plausible (24).

Check the repeatability of a finding: For every finding, find at least one other study which reports the same finding using the same procedure and/or a different procedure. Likewise, actively search for contradicting findings.

Doing a review or meta-analyses? Do all of the above, plus make funnel plots (25).

Read the References section.

Even if you’re not a fan, try pre-registration at least once.

Use the free G*Power tool to post-hoc calculate the power of published studies, and use it to a-priori to plan your own studies (26).

When reporting empirical data, strive to visualize it in the most informative way. Bar plots are easily one of the least informative visualizations. Use more informative formats instead, such as the pirate plot in the image below (27).

Last time, I was thrilled to announce that my data had arrived. Now, as I am looking though the piles and piles of questionnaires, new questions have risen in my mind:

Why would someone cut out squares from the pages of their questionnaire?

I said: “tick the boxes in black or blue pen”. Not: “tick them in red, orange, yellow, lightblue or barely visible pencil”!

Why do these kids leave so many questions unanswered? I didn’t put that question mark in for nothing!

Why does a significant portion of the student population think my picture on the announcement letter ought to have a beard?

But the most important thing I discovered was that not all students fill in the questionnaire seriously. Of course, that could have been expected, but naïve as I was, I thought it wouldn’t be so bad. While scanning in the hardcopy questionnaires into the computer that can read the answers for me, I came across a group of 6 questionnaires that were all filled out in a zigzag pattern. Obviously, this had been a group of silly teenagers who were sitting together and had come up with the idea of transforming the questionnaire into a zigzag artwork. Unfortunately, I could not use these questionnaires anymore. From then on, before I scanned the questionnaires, I manually looked through all of them to discover any other potential jokers. While doing this, I also discovered some other nice (and some not so nice, and some even very rude) artworks of students on the questionnaires. I made a selection of the nice ones as a keepsake.

“Good luck with your research, stranger”. Well thank you, stranger!

Nice hat!

“This makes me cranky”

An assembly of some nice artworks.

A Harry Potter-with-a-beard look and a mathematical problem.

Some students thought I should be a pirate instead of a researcher. Time for a career swap?

It is a little like snail mail: students responded to my “letter”. Although not very extensive or polite sometimes, it was a peculiar and surprising form of communication. I have enjoyed looking for the most creative outbursts on the paper.

Which one is your favourite? And have you ever experienced students drawing on your questionnaire or picture – and if so, did they also think you were meant to be a pirate?

As you all might have noticed, we went offline for a little while. As much fun as we have when blogging about our PhD-lives, we need ideas for interesting blogposts. So, we took a little break, allowing ourselves to have experiences we could later write about. And look, here we are, back with new blogposts :-).

This week, I am taking you with me in a flashback to my stay in Berkeley. The previous time I wrote, it was mainly about my preparations for and my first experiences in Berkeley. Now, I have been back for a little over two months, and will write about what useful knowledge and experiences I gained. Because even though I was there for ‘just’ six weeks, what I brought back will last me months! In this blogpost I will focus on two of those insights I’ve gained. Read more

So, last Friday two weeks ago I left the gray-skied Netherlands for my big adventure to the United States. I was going to visit UC Berkeley for six whole weeks! I would be able to meet faculty from UC Berkeley and talk to them about my and their research. Before my departure I was, of course, nervous:

I had a wish-list made for what to get for my research from this trip, would I be able to cross off everything on it?

I would have to talk to many new people, every time explaining my research all over again and have it open for discussion

In this blogpost I want to share some of my experiences with you regarding these two points I just made.

My wish-list

It all started for real last Monday. I had my first meeting with my contact at UC Berkeley. In the 45 minutes I was in her office things went really fast; names of people I should really meet flew by, we discussed what I wanted to get from my stay at UC Berkeley, and I was suggested a book to read. After this meeting I soon realized that I probably won’t have to worry about checking off everything on my wish-list; it will probably get done. For, all the people whose names were mentioned during the meeting, would be told of my visit at UC Berkeley, which gave me an opening of contacting these people myself. Of course, this still gave me the, sometimes difficult, task of introducing myself a little over email, thinking of the reason why I would meet that specific person, and asking whether the person would have time to meet with me. Luckily, up until now everybody has responded positively. In addition, I have also visited some classes in the teacher education programs. And I am visiting some research group meetings. Thus, even after only two weeks, I can honestly say much on my wish-list has already been crossed off. I can start looking for some extra wishes :-).

I think what really helped in getting so far with my wish-list, is that I communicated it and I keep communicating (parts of) it to the people I meet with. For, they are the ones who can help me getting everything done. And although I now use this strategy on a visit abroad, I think this is very helpful in many different situations, also. I mean, if there is anything you want to do during your PhD, communicate it with the people around you. It is good if you want to find stuff out for yourself. I’m not saying you shouldn’t do that. But you will notice that with some help from others, you might get a little further, maybe the person you’ve just communicated your wishes to, knows just that specific person who can help you in achieving your goal.

Meeting new people everyday

The fact that I am crossing wishes off my wish-list, means that I have to meet new people everyday. That also means that over and over again, I have to explain my research. You would say that if I have to do that so often, it becomes easier and I can just hit the replay button that makes my brain tell my mouth to talk about my research. If only. It means that every time again, I look critically at what I am explaining and to whom. First of all, for almost everyone around here, I have to tell more about the context in which my research is taking place, than I would have to in the Netherlands; not everybody is familiar with the Dutch educational system. Second, you have to think about who you have in front of you and what his/her research interests are. I have experienced that the meeting will follow a much more natural course, if you find some common ground to talk about. For my meetings up to now, I didn’t have a whole set of questions prepared. I did some research about the person, came up with two, for me important, questions and went to see the person. Sometimes, I haven’t even used the question I had made up before, but it was good I had them. We would just find concepts to talk about that have kept both our interests and the meeting would just go. Still, I find it every time a challenge to think about how I am going to talk about my research with the next person, and what questions I want to ask him/her. So far, however, it has really paid off; at the moment I am really thinking over some concepts and my theoretical framework. But, the story behind that will follow in a later blogpost.

Well, this blogpost has been a lot about my experiences, but what message do I want to bring to you with this?

Make a wish-list (about what you want to achieve: during your time as a Phd-student; at a conference; when visiting another university, etc.) and communicate it

Look up people you are meeting and think about how you want to tell your story to them

A while ago, my co-worker and fellow blogger Tim posted a tweed likening getting you PhD to a MMORPG. I thought this was a funny and also pretty accurate description and today I want to write about one of my favourite PhD side-quests: Teaching.

For context, I write this as a full time PhD student who is not expected to teach that much. About 10% of your appointment is a normal amount of teaching for PhD students in the Netherlands. This may be very different from the situation of some of you. Some of you may be teachers/PhD candidates whose main quest is actually teaching. If it is, my post may not be that useful for you, but hopefully you have some nice tips to add for those of us who are inexperienced teachers.

Before I started my PhD I worked as a Teaching Assistant in various courses for two and a half years and I’ve been involved in the teaching of two different courses since I started my PhD. I love teaching and think it is very rewarding, but it is also very tiring and time-consuming. Today I share some tips to make your teaching experience as good as possible:

Educate yourself

If you’re a new PhD student teaching a course or tutorial may seem intimidating. Especially if you have never taught before! There are several ways to educate yourself.

You can try and see if your university offers workshops or courses for teachers to improve their general teaching methods. You can also talk to more experienced co-workers and see how they approach their teaching. This can be in general, or related to the specific course you’re expected to teach. Experienced teachers hopefully will have hands on information that you can use while preparing for teaching.

Be well prepared

A thorough preparation is vital, especially if you expect that you will be teaching a course several times. Of course preparation is time consuming, but it is a timesaver if you are able to take your slides from previous semester. A thorough preparation can also help with nerves on your side. If you are really well versed in the subject you will be teaching, you will feel more secure standing in front of the students.

Plan!

Teaching can be a serious time consumer. Especially if you have to grade student work and provide individualised feedback. One tip that a co-worker gave me is to set aside a specific moment during the week for these type of activities. That way you know how much work you actually have to do and you have some control over the amount of time you spend. Try to be realistic in your planning. I for one know that I cannot go straight from teaching to being focused on my research, so I’ll plan for some transition time as well.

What’s in it for you(r research)?

When your teaching duties consist of thesis supervision it may be possible for you to combine teaching and your own research. Often, the students will get involved in your project and (hopefully) collect some of your data. But even when your teaching is not that directly linked to your research there may still be links. Maybe the content of the course you’re teaching is closely related and you can use your theoretical insights during instruction. But even if there is no way to connect your teaching to your research, teaching a course is relevant to your personal and professional development.

Enjoy it

Whatever you do, enjoy your teaching experience!

Happy questing and hopefully this side-quest will help you level up!

Have you been teaching and what are your tips for PhD students who are starting to teach?

Just before our summerbreak, we uploaded the post ‘useful tips when attending a conference’. In this post several PhD-students from different universities in the Netherlands provided, as the title says, tips for when you are about to attend a conference. Just last week, I attended two European conferences together with some of my colleagues, the JURE (Junior Researchers conference) and the EARLI (European Association for Research on Learning and Instruction). In this blogpost I’ll reflect on how the tips from the post mentioned above helped me and expand on them further.

Last week Tessa and I gave you an overview of our experiences from a conference (the ORD2015) we recently attended. We really hope you had fun reading our experiences and maybe related to it from your own experiences. This time I would like to be more helpful with my blogpost by providing tips when attending conferences. Or, actually I should say that I am only helpful by showing these tips to you… Other people have provided them; I asked PhD students from ICLON and other institutes to provide tips for when you are attending a conference. The idea is that everyone (presenter, non-presenter, already having a large network, in a presentation not completely on your subject, etc.) attending a conference should find some useful tips in here.

Before the conference

Jorine Vermeulen (PhD student at CITO and University of Twente) told me to be sure to prepare before attending the conference: Take at least half a day in advance to check out the program and make a selection of must go to session as well as sessions you can possibly skip for an extra break, some working, or a meet up with some people you really want or need to talk to about your project.

Daniël van Amersfoort (PhD student at Welten Institute, Open University) gave a tip you should think about when signing up for the conference: Always visit pre- and post-conferences or, in case it is being organized, a doctoral consortium; most of the times these meetings are more profound than the conference itself. In addition, these are perfect places to meet people you will probably also see at the conference!

In addition he says you should not only try to schedule where you are to meet new people, but: If it is a bigger conference and there are specific people, or even ‘big names’ you want to meet, email them months before the conference. Using this approach, I have sat at a table with almost half my bookcase ;-)!

So, try to stick with how you planned it 😉

Preparing a presentation for the conference? Bas Agricola (part-time PhD student at Utrecht University) has some tips for you:

Use the more standard building blocks for your presentation (theoretical framework and research question, method, results, and conclusion and discussion). This way it will be recognizable for you audience.

Considering your PowerPoint, Prezi, or other way of presenting you use; do not use too many words on your slides. Try to prevent from writing full paragraphs, use key words.

During the conference

The time has finally come. It is time for the conference! Time to do something with your preparations. But also, to be spontaneous; you didn’t plan that extra break for meeting specific other people? No problem, according to Nienke Woldman (PhD Student at Wageningen University and president of VPO): Do not only attend presentation sessions. Make sure to have spare time to talk to other researchers. Making new contacts and maintaining ‘old’ ones is at least as important at a conference as exchanging information during sessions. Most of the times the scheduled breaks are too short for this. So, be spontaneous, decide suddenly to skip a session, grab a cup of coffee and see who you end up talking to. Sometimes at these, more quiet moments, you make really good contacts!

Daniël also made this point when I asked him for tips. Although he added: Do not stick with your colleagues the whole conference. You see these people often enough; go your own way. (also hearing Fleetwood Mac in you head right now?! I am….)

By the way, if you haven’t contacted those ‘big names’ do not think you’ve missed your chance. You have something to discuss with them? Do not hesitate to approach then and ask if you could discuss your research with them over breakfast or lunch.

Of course, during conferences the prepared presentations mentioned above are being presented. Are you presenting yourself? Bas has a tip for you: Share your doubts with the audience. Being transparent on the subjects you still have questions about will provide the best input for discussion and questions. And this in turn, will provide you with new information and thoughts to take home.

The idea during presentation sessions is that the audience is not a passive listener of what the presenter as to tell. Bas says: Actually, the same I just said about the presenter counts for the audience; share your thoughts. You do not have to be fully informed on the research subject of the presentation. You can always provide some new viewpoints on methodology and share your questions, criticism and feedback. Sometimes these new viewpoints are very refreshing for a researcher who is most of the time surrounded by his own research (subject).

How you can do this and what you can get from being an active listener, Tim (PhD student at ICLON, also blogger here on the research blog) shares with you.

Having attended several conferences as a non-presenter I would like to share some tips about ‘that what is not being said’. These can range from implicit assumptions, logical fallacies in reasoning, leaps of faith or simply not stating the obvious. To better understand the presenter and whatever it is (s)he is presenting about it can be very useful to make the implicit, explicit. For example:

Make notes (mentally or physically) during a presentation on which implicit assumptions the speaker makes about any theoretical models being used. To what extent does the presented study rely on the mentioned model or theory? Is it being assumed that this is the only theory? A complete theory? Is it being contrasted with alternative theories?

For studies which use a sample to generalize to a population: what exactly is the sample a sample of? What are the limits of generalizing the findings of the study? How representative (in size and characteristics) is the sample of the specific population?

For studies which use inference statistics: to what extent are the analyses explorative (e.g. no a-priori established hypotheses were made and many analyses were performed) or confirmatory (e.g. a-priori established hypotheses were tested)? What does this tell you?

For the ‘conclusions’ part of a presentation: are the conclusions sufficiently backed up by what was said before or are there hidden reasoning steps? If the given conclusions would not be true, would the presented study be able to found evidence for this? What are the alternative explanations or conclusions which may also have some – or maybe more – merit?

The goal is not to be overly skeptical, but to be sufficiently critical to look beyond what is being presented and (hopefully) learn more. This does not only help you to better understand presentations at a conference but might also prove helpful for your own studies, articles and presentations. But that’s just my assumption.

Now Jorine, Daniël, Bas, Nienke and Tim have flooded you with tips, I wish you a very fruitful conference the next time you attend one!

Sometimes, when two people in a relationship each have their own house, their own businesses and value their independence, they could have a LAT-relationship: Living Apart Together. Sometimes, when a bunch of colleagues each have their own offices, their own projects and value their own intellectual paths, they could have a WAT-relationship: Working Apart Together. A PhD can be considered quite an individual job. Read more

These past few weeks since my last blogpost I have been busy writing, as you probably might have guessed, considering this blogpost is ALSO about writing (is there anything else I can talk about at the moment?!). It wasn’t always as easy as I wanted to (I experienced enough terrifying moments), but last Friday I actually finished a new version of my paper 🙂 !!! Yay!

In order to get to that new version, I used a very productive approach and an useful app in addition to the three approaches I told you last time. Also I looked into another approach and some apps, in case the ones I chose weren’t sufficient. These approaches and apps are:

My name is Francesc Esteve, I am from Spain (SP), and I have just presented my doctoral dissertation at the Rovira i Virgili University. During the last semester I did a research stay at the ICLON Graduate School of Leiden, and I like to explain my international experience in this post.

I started my PhD four years ago. After finishing the theoretical framework and data collection, I thought it might be a good idea to stay a while at another European university, to share and compare my results, draw future lines of work and finish writing my work.

I’ve tried to summarize this experience in 5 points detailed below, five things to highlight, that I’ve learned or that’ve surprised me.

Translate:

Interaction & Sharing

Please, feel free to add comments and questions to the posts by clicking on the title of the post. At the bottom you can put your comments.
Please share the post in your network by clicking on the icon (-s) below the posts.