Cognitive Sciences Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It's 100% free, no registration required.

Background: I'm having hard time motivating myself to study. In my first year I was a very good student, I had the motivation to study because I wanted to compete, it was first and maybe hardest year. I wanted to show that I am good and better. I passed my exams in time with good grades. Then, I started studying less and less, finding it hard to motivate and focus on learning. I don't study and don't have motivation to study because I am afraid of failing an exam. I am afraid that if I study hard for an exam and then fail I will feel that I am stupid.

Questions

Is there a name for the phenomena of not putting effort into study because of fear of failure which in turn results in actual failure?

Its called self-handicapping. You built up your self esteem based on initial results which later regressed to the mean. This demotivated you (as you say). I am not trying to say that you are not a good student, just that the self-esteem you probably have is detrimental to your growth. I experienced something similar after my first year.
–
AsheeshRAug 28 '13 at 3:39

Also, I believe that not studying is usually more complex and not the result of a single cause. E.g. I usually procrastinate until the point where the relation between time spent studying and exam results is the most effective, see the 80/20 rule: 20% of the time I might spend studying causes 80% of the results, so I only study for those 20% of the time -- subconsciously I know how long I can procrastinate. And why don't I use all the time that I have to achieve 100%? Because studying is boring and I hate it. So procrastination is balancing boredom and results.
–
whatAug 28 '13 at 16:52

Self-handicapping is the process by which people avoid effort in the hopes of keeping potential failure from hurting self-esteem.

What causes this?

The main and most relevant reason here is the preservation of your own self-image and/or managing your public image. If you were to try your very best and still fail, then the only causal attribution would have been dispositional, that is, the only way to explain it to yourself would have been that, you are just not good enough.

However, when you place impediments in your path and essentially prevent yourself from performing to your fullest, you are creating scapegoats for your failures. This will allow you to attribute the failure to situational factors and not dispositional ones, allowing you to maintain your image of self-competence and/or self-efficacy.

Are there any strategies for overcoming this cycle?

Self-handicapping occurs in many different fashions but most of them tend to be variations of procrastination and lack of effort/practise. So, by tackling either of these, you will be able to reduce the chances of handicapping yourself.

Other than that, research has shown that by focussing on long-term mastery in a field that you like, self-handicapping behavior can be reduced.

In three studies with German high-school and college students, we found empirical evidence for the assumed moderator effect of mastery goals. In studies 1 and 3, performance-avoidance goals were remarkably lower associated with self-handicapping in the group of students highly endorsing mastery goals compared to students who proved to be less mastery oriented. In studies 2 and 3, moreover, individuals' self-esteem was less related to self-handicapping when students strongly emphasized mastery goals.