My understanding of your particular example is limited but just an observation - in case you do not want to do inference over certain variables but simply sample from them, you can do so directly by calling dist.sample(). All sites designated with pyro.sample will be used to generate hamiltonian trajectories in the integrator. So in your example, if X_hyp is not something that needs to be treated as a latent site, you should use:

Some minor things: use torch.tensor instead of torch.Tensor, no need to unsqueeze to make X a 2D tensor (GP module allows X 1D for most kernels), use torch.clamp(min=0) instead of torch.abs() for L2, use GPRegression forward method instead of creating simple_gp function. use gp.kernels.Matern52 and set_prior methods instead of creating Matern52 kernel.