This warning seems related to the fact that the trait did not evolve
according to the specified model (i.e., Brownian motion). I simulated
some data with:

Advertising

tr <- rphylo(50, .1, 0)
x <- rTraitCont(tr)

ace(x, tr)$sigma did not return NA out of 1000 replications. I changed
the second line to:

x <- rnorm(50)
and NAs were returned in 18 cases.

In your case, since you simulated the data, it could be that their
distribution does not conform well to the Brownian motion model. Maybe
you could transform them or discretize them and use a discrete trait
model (i.e., Markovian model, type = "d" in ace).

I am playing around with synthetic cophylogenetic data. In one example,
which represents a very extreme case of high agreement between the host
and parasite phylogenies (the data was generated with 51 cospeciation, 2
sorting and 1 duplication events), I get a warning when using ace() to
label the cophylogenetic nodes of the parasite tree:

fit.P<-ace(Pfreq[,2],obj$trees[[2]])
Warning message:
In sqrt(1/out$hessian) : NaNs produced
Where Pfreq[,2] is a vector of frequencies:
> Pfreq[,2]