where the time evolution operator [itex] \exp(-\frac{i}{\hbar}\hat{H}t) [/itex] allows you to find the state vector [itex] |\psi (t) \rangle [/itex] at any time if you know the state vector [itex] \psi (0) \rangle [/itex] at time 0 and the Hamiltonian. This is the true significance of the Hamiltonian.

It is true that "operator" [itex]\frac{\partial}{\partial t} [/itex] also formally describes the infinitesimal time evolution. However, in order to apply it to [itex] | \psi (0) \rangle [/itex] you must already know the state vector at a later time [itex] | \psi (\Delta t) \rangle [/itex]. So, you must know the solution of the (time evolution) problem, which you are trying to solve in the first place.

Isn't it also true that if you express the Hamiltonian in terms of the time derivative you are choosing a particular basis in which to express it? You could also choose to write it, for e.g., in the momentum basis. For that reason, I'd say that the Dirac notation is more general, being independent of a choice of basis.

Isn't it also true that if you express the Hamiltonian in terms of the time derivative you are choosing a particular basis in which to express it?

I don't think so. As far as I understand is the time parameter [itex]t[/itex] in non-relativistic QM (what we are discussing here) something different as position and momentum. Well, physically this is trivial. What I mean is the following:

A quantum state is given as a vector in a Hilbert space. Which Hilbert space this is depends on your chosen perspective. In position representation the quantum state is represented by a function [itex]\psi:\mathbb{R}^3\to\mathbb{C}[/itex].

The time evolution of [itex]\psi[/itex] is mathematically realised by a curve [itex]\psi_t[/itex] in this Hilbert space, so time is "just" the parameter of this curve. Since the Hamiltonian is an operator on the Hilbert space, it can include derivatives in [itex]x[/itex] (for position representation) but cannot include something like time-derivatives.