> If a process has two or more threads running in it, what are the rules> regarding one thread referencing memory in a different thread's stack?

There are no rules in the hardware. It is possible for a thread to
reference another thread's stack in a flat memory. And then there are
thread implementations which are even loose, they are not strictly OS
recognized thread. These threads all run in application domain.

The simplest solution is to attack the problem at the language level.

At the language level, all your parallelism optimizations are possible
wrt stacks as per C & C++. Once "decided" that the stack is unaliasable across
threads at the language level, the next step would be to
decide about security be it at the language lever, hardware level or
both. Normally,
without security, the hw need not care about aliasing as the compiler wont.

With security, all the following layers(there are more layers if one
looks at them)
can cooperate to enforce some discipline:

Language

Compiler

Hardware

[Operating system enforced security remains the greatest leveller of them
all but it doesnt belong in this forum. However, if you care , then you
can ship a DLL/Shared Library as one of your layers in security]

Sincerely,
Seima Rao.
[Some hardware has a fully shared address space, some doesn't. The question
is basically what doesn't. I'm fairly sure I've seen something where threads
can't see each other's stacks, but I don't remember what it is. On most
current architectures, it'd require messing with virtual memory tables which
would make switching threads a lot slower, basically the same as process
switches. -John]