Suppose we are given a set of $M$ pairs $\{(\vec{x}^{(i)},\vec{y}^{(i)})\}$, with
$\vec{x}^{(i)}\in\mathbb{R}^N$,
$\vec{y}^{(i)}\in\mathbb{R}^N$,
$M\gg N$ such that

$\vec{y}^{(i)} = Q^{(i)} \vec{x}^{(i)}$,

where $Q^{(i)}$'s are unknown orthogonal matrices. It is known however that they can all share the same set of eigenvectors:

$Q^{(i)} = U D^{(i)} U^{-1}$

How can $U$, and therefore $Q^{(i)}$'s, be recovered just from $\{(\vec{x}^{(i)},\vec{y}^{(i)})\}$?

My thoughts so far:

I'm hoping this can be reduced to an eigenproblem, or something easy like that, but I don't see how it can be done. The best approach I have in mind involves numerical solution of a nonlinear system of equations: e.g. require, for each $i$ that the elements of
$U^{-1} \vec{y}^{(i)}$
and the corresponding elements of
$U^{-1} \vec{x}^{(i)}$
have the same moduli (using the fact that the eigenvalues of an orthogonal matrix have moduli of $1$)