R.
Gidal said in the name of R. Hiyya b. Joseph in Rab's name: In the case
of a house, the inside of which is not ten [hand breadths in height]
but its covering makes it up to ten, it is permitted to carry on the
roof over the whole [area]; but within, one may carry only four cubits.

A diagram might help:

The point is that the space inside the house is less than 10 tefahim high, say 9 tefahim as I drew it. A reshut ha-yahid needs to be a space enclosed by walls 10 tefahim high, so the inside of this house doesn't qualify. The roof of the house, on the other hand, is like a platform 10 tefahim high, so it is a reshut ha-yahid.

I note that the ceiling cuts off the top of the interior reshut. I might have thought that the interior is indeed enclosed by a 10-tefah wall, if you were to imaginarily blow a skylight out of the roof that takes away the ceiling, like this:

Clearly the reshut isn't measured that way. The ceiling cuts off what you can measure from the inside.

I also note that this isn’t a very comfortable house, unless maybe you
live in the Shire. 10 tefahim is about 3 feet. (Hobbits are 2 to 4 feet tall, according to
Wikipedia.) Obviously the point with this case is the theory of reshuyot, not because it’s directly practical, but because it’s interesting. And it’s about to get a lot more interesting.

Said
Abaye: But if one digs out four square [handbreadths] and makes it up
to ten, carrying over the whole is permitted. What is the reason? [The
rest] is [as] cavities of a private domain, and such are [themselves] a
private domain.

The diagram:

The 10-tall volume of space above the 4-by-4 hole in the ground qualifies as a reshut ha-yahid; the rest of the interior of the house becomes permitted along with it as horei reshut ha-yahid, “cavities” in the walls of the room.

So, the peshat seems to be that to have a reshut ha-yahid you in fact need a volume of space that’s 10 by 4 by 4 tefahim. And that space needs to be partitioned from the public by walls.

But what exactly are the walls in this case? Cue the rishonim.

The rishonim contrast this hole in the ground to a similar hole that's been dug inside a sukkah, discussed on Sukkah 4a:

If a sukkah is less than 10 tefahim tall, which is the minimum allowed height, and you dig a hole in the ground inside the sukkah to complete the 10 tefahim, the sukkah is only kosher if the edges of the hole are 3 tefahim from the wall or less.

Why does this hole need to be near the walls of the sukkah, but the hole in a reshut ha-yahid can be as far as you want from the walls of the house?

(The rishonim take for granted that the reshut ha-yahid hole can be far from the walls, and I note that there’s a simple proof that it’s so: the hole need only be 4 by 4 tefahim, yet it allows you to carry beyond 4 amot inside the room; ergo, the room is over 4 amot wide, and the hole must be multiple amot away from at least some of the walls.)

Tosafot say it’s a special rule by sukkah that the walls need to adjoin the sekhakh. For the sides of the hole to be part of those walls, they can’t be disjointed by more than 3 tefahim. For a reshut ha-yahid, though, we don’t care how wide or disjointed the walls are. We can have a reshut that’s 4 tefahim wide, with a wall 4 amot thick, and the wall happens to have a huge cavity 9 tefahim tall and 3.9 amot deep.

The Rosh has different interpretation for the walls of the reshut ha-yahid:

An interior wall that zigzags horizontally over 4 amot is no good according to the Rosh. The reason a house with a hole has a reshut ha-yahid inside is thanks specifically to the exterior wall of the house, which is your typical straight 10-tefahim-tall wall.

To understand this opinion, it seems to me that we need to be careful how we distinguish the dinim of reshut ha-yahid.
There are two requirements, as I mentioned: partition from the public,
and a minimum volume of space. To the Rosh, both these requirements have a height of
10 tefahim, but they are totally distinct! You can
have one yardstick on the inside from the hole to the ceiling, and
another yardstick outside from the ground to the roof. One yardstick
gives you your volume, the other gives you your partition. In contrast,
as far as I can tell from Tosafot, the only yardstick that matters is
the one inside the house. Once you have ten tefahim inside, both the partition and the volume of space are good.

It appears to me, then, that the following case is a nafka minah between Tosafot and the Rosh:

This house is only 8 tefahim tall, so you have no reshut ha-yahid
on the roof. What if you dug a hole inside to complete ten tefahim? You
certainly have a proper 10-by-4-by-4 volume inside, and you’re
partitioned from the public. For Tosafot that’s enough. But for the Rosh
that’s not enough, because nowhere do you have a proper 10-tefahim high wall!

Finally, the Daf Digest cites the hilluk of R’ Chaim Brisker that the sukkah has a din of 10-tefah “walls,” whereas the reshut ha-yahid is a 10-tefah space that simply needs to be enclosed. This sounds very similar to what I was medayyek from Tosafot. The difference might be that Tosafot require 10-tefah “walls” for the reshut ha-yahid as well, but allow them to have a disjointed shape; whereas R' Chaim doesn't have any height requirement for the wall. I don’t see any a nafka minah. But at least R’ Chaim would certainly accept Tosafot’s side of the nafka minah I drew above. I wouldn’t be surprised if he actually makes it himself, but I don’t know where Daf Digest found him to look him up.