I was able to get the value of the integral as 1.
Now, for the second part of the problem. I have to find the value of 'r' at which the probablity P of finding the electron is maximum.
[tex]P = 4\pi r^2 \psi^2[/tex]

Try plotting your function is as a function of [tex] r/a [/tex] and you will clearly see that [tex] P(r) [/tex] is not maximized at [tex] r/a = 4 [/tex]. I think somebody has made a mistake here, but the wavefunction you gave is correct for the 2s state. The supposed answer must be wrong. It possible to obtain a quadratic equation for the maximum if you use some information about the wavefunction to find one of the factors.

Try plotting your function is as a function of [tex] r/a [/tex] and you will clearly see that [tex] P(r) [/tex] is not maximized at [tex] r/a = 4 [/tex]. I think somebody has made a mistake here, but the wavefunction you gave is correct for the 2s state.

Exactly.In fact, if you forget to multiply by [tex] r^2 [/tex](a very common and easy to commit error), you get the incorrect answer [tex] r/a = 4 [/tex].
Also, in
[tex]P = \frac{r^2}{8a^3} \left(2 - \frac{r}{a}\right)^2 e^{\frac{-r}{a}} [/tex]
take the r^2 inside and then differentiate. You will get a quadratic after cancelling the common terms.

Thanks Physics Monkey and Siddharth.
I did take the r2 term inside and differentiated it, but I dont get a quadratic term. I have posted the equation I got in post #3. Anyway, I'll post again:
[tex]\frac{8r^3}{a^2} - \frac{16r^2}{a} + 8r - \frac{r^4}{a^3} = 0[/tex]
Is it possible to solve this equation or am I going wrong somewhere?

As dextercioby already noted, there is a common factor of [tex] r [/tex] that you can obviously factor out. A simple way to make progress on the cubic is to note the following: the probability density passes through zero at [tex] r = 2a [/tex], but it is also always greater than or equal to zero. What does this tell you about one of the roots of your equation?

As dextercioby already noted, there is a common factor of [tex] r [/tex] that you can obviously factor out. A simple way to make progress on the cubic is to note the following: the probability density passes through zero at [tex] r = 2a [/tex], but it is also always greater than or equal to zero. What does this tell you about one of the roots of your equation?

Yes, I can factor out the common 'r'.
[tex]\frac{8r^2}{a^2} - \frac{16r}{a} - \frac{r^3}{a^3} +8 = 0[/tex]

Probability density is greater than or equal to zero. Does this mean the roots are real?

The probability density is always greater than or equal to zero. Therefore, when the density equals zero that point will be a ... ?

The point will be a minimum, right?

I need to clarify certain doubts or else this thread will become FUBAR.
Just as for particle in the box, there are some positions where the probability is zero. The radial probability distribution functions P(r) indicate the relative probability of finding the electron within a thin spherical shell of radius 'r'.
So if the P(r) = 0, the point is a minimum so at what value of 'r' is P(r) maximum?
How does this compare to 4a, the distance between the electron and the nucleus in the n=2 state of Bohr model?