A Quantum Field Theory - Why quantise fields?

As I understand it, the need for quantum field theory (QFT) arises due to the incompatibility between special relativity (SR) and "ordinary" quantum mechanics (QM). By this, I mean that "ordinary" QM has no mechanism to handle systems of varying number of particles, however, special relativity predicts the possibility of particle creation and thus systems with a variable number of particles. "Ordinary" QM also runs into problems such as negative probabilities and a breakdown of causality.

My question though, is why fields? Why not some other theoretic construction? It makes sense that quantities that were described by classical fields previously, such as the electromagnetism, will be quantum mechanically be described by some sort of "quantum field", but I'm struggling to fully understand the motivation for constructing quantum fields for which particles are emergent quantities (i.e. excitations of their underlying quantum field)?!

I've read Weinberg's paper "What is Quantum Field Theory, and What Did We Think It Is?" , and in it he states that QFT is an inevitable consequence of trying to construct a quantum theory that obeys the principles of SR (namely Lorentz invariance) and also the Cluster Decomposition Principle. Is this sufficient motivation from the start to consider fields?

Here are a couple of ideas I have (although I'm not sure if they're correct at all?!):

1) Is it simply the case that, in order to satisfy Lorentz invariance, interactions must be expressed in terms of density fields, i.e. $$V(t)=\int\,dx\mathcal{H}(t,\mathbf{x})$$ where ##\mathcal{H}(t,\mathbf{x})## is a Lorentz scalar and commutes for spacelike separations, $$\left[\mathcal{H}(t,\mathbf{x}),\mathcal{H}(t,\mathbf{y})\right] =0$$ for ##\left(\mathbf{x}-\mathbf{y}\right)^{2}\geq 0##.
Furthermore, the cluster decomposition principle requires that one constructs ##\mathcal{H}(t,\mathbf{x})## out of creation and annihilation operators, however in order to construct a Lorentz scalar out of such operators they must be "coupled" in some way. The most natural solution to this is to construct ##\mathcal{H}(t,\mathbf{x})## out of fields. With this in mind, we are motivated to consider quantum fields as opposed to a particle approach.

2) Since single-particle mechanics is "out of the window" if one includes SR, we need to consider constructing multi-particle states. The most elegant way to do this is to introduce the formalism of second quantisation - instead of asking the question "which particle is in which state", which is meaningless since the particles are indistinguishable, we instead ask "how many particles are in each single-particle state". By doing this we can represent a quantum state of a many-body system in terms of the number of particles in each of the available single-particle states. Furthermore, one can introduce so-called creation and annihilation operators to add and remove particles from each single-particle state, respectively. In this sense, a many-body quantum state can be constructed by acting on the vacuum state of the theory with the creation operator.
Now, in principle, since particles can be created (or annihilated) from the vacuum at any point in spacetime and also should be able to propagate through spacetime continuously, one needs to construct a field of creation and annihilation operators - a set for each spacetime point - a so-called "quantum field". In this sense, a quantum field acts on the vacuum to create a particle, and since this is the minimum possible excitation of the quantum field, the particles created are a quanta of the underlying quantum field.

Was this the primary motivation for describing particles in terms of fields?

The primary motivation was and is to describe reality.

It turned out that one needs fields, as particles are incompatible with relativitiy. It also turned out that fields explain particles easily (in the limit where it makes sense to speak of particles) through the concept of elementary excitations, while the other direction was blocked by inconsistency.

But historically, it was a matter of trial and error, and the particle and field picture coexist beside each other.

It turned out that one needs fields, as particles are incompatible with relativitiy. It also turned out that fields explain particles easily (in the limit where it makes sense to speak of particles) through the concept of elementary excitations, while the other direction was blocked by inconsistency.

But historically, it was a matter of trial and error, and the particle and field picture coexist beside each other.

Ok. Is there any credence to what Weinberg wrote in his book "The quantum Theory of Fields: Volume I", about the usage of fields being an inevitable consequence of requiring that a quantum theory satisfies Lorentz invariance and the cluster decomposition principle?

Also, why are particles incompatible with relativity? Is it because one cannot construct a particle wavefunction that accounts for particle creation?

Ok. Is there any credence to what Weinberg wrote in his book "The quantum Theory of Fields: Volume I", about the usage of fields being an inevitable consequence of requiring that a quantum theory satisfies Lorentz invariance and the cluster decomposition principle?

Yes and no. See the topic ''Is there a multiparticle relativistic quantum mechanics?'' from Chapter B1 of my theoretical physics FAQ.

Ok. Is there any credence to what Weinberg wrote in his book "The quantum Theory of Fields: Volume I", about the usage of fields being an inevitable consequence of requiring that a quantum theory satisfies Lorentz invariance and the cluster decomposition principle?

I would dare to disagree with Weinberg here. In your quote above I would prefer replacing "an inevitable consequence" with "one convenient method".

Weinberg showed beautifully how one can pack particle creation and annihilation operators into linear combinations called quantum fields. Then interaction operators in the Fock space can be built as certain polynomials in fields and the Poincare invariance of the theory will be ensured. Within this logic fields can be regarded simply as a mathematical tool for easier formulation of a Poincare invariant theory, where particles can be created and destroyed and particle interactions are cluster separable.

However, nobody has proved that this field-based approach is the only available way for designing such good theories. Actually, there are alternative approaches that do not involve quantum fields and yet provide satisfactory relativistic theories of particle interactions. You can find relevant articles by searching phrases like "unitary dressing transformation" or "Greenberg-Schweber". One benefit of doing QFT without fields is that particle self-interaction can be avoided, so the renormalization problem does not even appear.

There are Quantum systems in Minkowski space whose natural description is not fields, but rather loops or flux tubes. However it is always possible to write these theories as fields possessing an unphysical local symmetry (gauge symmetry), so you don't lose anything with the field formalism.

Weinberg has a nice discussion of this with regard to the photon in Chapter 5 Vol. 1