Recall the chain rule. If two expressions which are functions of x are multiplied, we do the following:
h(x) = f(x)*g(x)
h'(x) = f'(x)*g(x) + f(x)*g'(x)
As we can see, the derivative of h(x) is the derivative of f(x) multiplied by the original g(x), plus the derivative of g(x) multiplied by the original f(x).
For example,
h(x) = 2x * ln(x)
We know that
derivative of 2x = 2, and
derivative of ln(x) = 1/x. Thus
h'(x) = 2*ln(x) + (2x)*(1/x)

In the problem, f(x) = x^8√(5-3x). We can rewrite this as,
f(x) = (x^8) * (5-3x)^(1/2)
We know that
the derivative of x^8 = 8x^7
the derivative of (5-3x)^(1/2) = (1/2)(-3)(5-3x)^(-1/2)
Using chain rule, you'll get
f'(x) = (8x^7)*(5-3x)^(1/2) + (x^8)*(-3/2)*(5-3x)^(-1/2)
If you simplify this, you'll get the answer that you typed in there:
f'(x) = x^7(80-51x) / 2√(5-3x)