Has been mentioned for OpenMW before. Type "navmesh" in the search filter.
Will almost certainly be used post 1.0.

My apologies for not posting a save game. I didn't know if you'd want/need one. I tried to attach a save file just now, but the server won't accept it. I have a save where I'm looking at Huleeya with the behavior occurring as soon as I can figure out how to upload it.

Well, you could just e-mail it to me. My e-mail is [email protected].
That said, I was able to get to this mission without too much time spent. Using the console to cheat really helps.

I hope we're talking about the same behavior. He'll stop occasionally to perform an animation (such as scratching his head), but most of the time he just twirls in circles, both clockwise and counter clockwise. He seemed "trapped" in the square immediately inside the door. If this isn't related to the problem you've been investigating, and you want me to open a ticket, please ask. I've assumed it was the same issue you were troubleshooting already.

Here’s what I did

Use console to complete quests prior to this one.

COC into the Vivic, Black Shalk Cornerclub.

Toggled God Mode on player. (Note, I also gave player a speed of 300).

Huleeya kills all three thugs. I don’t see any “Running in Circles” (RiC)

I walk to door of corner club. Huleeya follows me. I see no RiC

I exit door of corner club, and walk down corridor. I note that on exit, Huleeya and I are at same place. I don’t see any spinning when Huleeya follows me. I do note that he goes in middle of corridor, which makes it difficult to get past him when I get to end of wrong corridor.

Now, there are 2 ways I know RiC can occur.
When using AiFollow or AiTravel, the actors path is checked every 0.25 seconds to see if it needs to be recalculated.
The recalc check is, has target moved by more than 10 units. (10 units is a tiny distance, there’s usually 100 or more units between waypoints in the path grid.) What this means is that when running AiFollow, the path is recreated every 0.25 seconds.
When new path is created, it sets the nearest path grid as the first waypoint of the path.
And, as actor is usually closer to the waypoint he’s just passed than the waypoint he’s heading towards, this sends the Actor back the way he came.

When using AiCombat, the pathfinder checks every 0.25 seconds to see if Actor has reached the current waypoint. However, in 0.25, the Actor may have gone past the waypoint, so the actor then turns around and goes back to the waypoint. Obviously, 0.25 seconds later the actor has gone past the waypoint again. Rinse and repeat.

The AiFollow and AiTravel should have been fixed. (When new path is created, it’s compared with old path, and the first node of the new path is discarded if it’s not wanted.)
AiCombat is fixed (mostly) by my PR, which hasn’t been accepted into trunk yet.

Looking at the script for this mission.
Huleeya starts with AiWander (and a distance of 0), so should not wander at all.
After you say follow me, Huleeya should be set to AiFollow. So, logically, you should be seeing the AiFollow bug.
(Note, if you make player stand still then Actor should not recalculate path.)

I thought the AiCombat might be possible. Need to know exactly what you did.

My other thoughts.
Could you be blocking Huleeya? (I was playing with player's speed set to 300)
Did RiC occur in the Black Shalk Cornerclub, or when you entered the Vivic Waist?
What happens if you set other NPCs to AiFollow and get them to follow you around the cornerclub, or into Vivic waist?
What if you start a fight with them? Do they run in circles when chasing you?

SetSpeed on NPC may help to confirm problem
If set speed low, say around 5, then problem with AiCombat should not occur, as NPC won’t overshoot waypoints.
If set speed very high, then problem with AiFollow should go away, as NPC moves sufficiently far to reach next waypoint.

I'm running a development build from commit ac1f64b5593f742fe34017dd07077a6c0da71956 on July 28, so it sounds like I should already have the AIFollow fix you're referring to, but maybe not.

Yes, that should have the fix there. Question: Are you actually building the source, or using one of Ace's nightly builds?
If building from source, are you willing/able to add logging to isolate the problem?

There is some “doubling back to nearest waypoint” in AiCombat, when Line of Sight on target is lost.

Was a report of “running in circles” for one of the quests. (Huleeya in “Vivic, Black Shalk Cornerclub” when doing quest “Vivec Informants”, http://www.uesp.net/wiki/Morrowind:Vivec_Informants but I've been unable to reproduce, and the person who reported it hasn't responded to me. If anyone could see if this is reproducible in the latest build, that would be very helpful.

I do have some thoughts about how to finish fixing the running in circles, but they do introduce a small increase in the actors getting “stuck”. So, Scrawl has vetoed them.

So, I'm thinking I should now focus on the “getting unstuck” logic.
Notes.
There are at least 3 pieces of “unstuck” logic.

AiPackage::pathTo(). Checks every 0.5 seconds. Checks for door (and opens), otherwise, assumes has bumped into a person. And responds by changing movement vector by 45%.
There's also a call to zTurn() in line 111. But that's probably useless. Because we call zTurn() again in line 124.

AiWander::evateObstacles(). Is similar to AiPackage::pathTo() in that it checks for doors. (Although the door logic is different.) And if not stuck by a door, again, assumes is bumping into another actor. Evasion logic is similar to AiPackage::pathTo(). Of note, uses ObstacleCheck class. (AiPackage::pathTo() notes that ObstacleCheck class doesn't work.) Also of note, the mStuckCount will go up to 200 before resting. (i.e. actor spends 200 seconds walking into wall before stopping.)

AiCombat has it's own stuck logic. Somewhat more difficult to figure out as logic is dispersed. But I think it's mostly, “If stuck, don't use shortcutting.”

My thoughts:
The “detecting is stuck” logic doesn't keep much history. In fact, it works by recording position of last test, and checking for difference. I've seen Obstacle check bouncing between two positions, because as soon as it detects movement it assumes the actor is “unstuck.”

I'm thinking the “unstick” logic might need to keep more history (and have more states.)
Basic idea something like.

When “stuck” is first detected, and it's not a door, assume it's another actor and apply existing logic. (As this does work. You can see actors passing other actors in a few seconds.

If the “stuck” is not fixed within say, 3 seconds, need to do something more drastic. Looking at vanilla, when it gets stuck, actor starts moving randomly a small distance from existing position. A drunkard's walk might work. (I think this has already been suggested.)

Other notes, where I've seen actors get stuck.

When the nearest pathgrid point is on the other side of an obstacle. (Note, I think actor could have jumped over the obstacle.)

When actor grazes a wall. You can see this at the Ebonheart well that's directly south east of the point when you “coe 1, -13” Some of the pathgrid's edges come very close to the well, and the player's collision cylinder hits the well.

With assistance of Asmo, I believe I've solved the mystery of Huleeya running in circles in the Black Shalk Cornerclub.

When Player enters the Black Shalk Cornerclub, Huleeya has AiWander with a distance of 0.
Player then talks to Huleeya selecting "travel together" and "No, you should wait here for now".
At this point Huleeya is given an AiWander with a distance of 128.
This is just big enough to cover two path nodes.
So, the "turning in circle" is Huleeya walking between the two nodes.
It's reasonably clear if you open the console and "TogglePathGrid".

If you run Vanilla, you can see the same behaviour, although in Vanialla, Huleeya travels a little further so it's easier to see he's moving between 2 path nodes.
Also, occasionally he moves to one of the other nearby nodes. (i.e. He moves between 4 nodes, but mostly just the two nodes that he moves between in OpemMW.

I've experimented with commenting out various parts of pathfinding code, and tried to understand how pathfinding works.

If I understand this correctly, A* pathfinding is used to find path from pathnodes, but after that, it's simply casting rays to see whether the actor sees the target or not. Some Z-level thing was included, I'm not sure I understand that. Regarding the actors running towards obstacles, I can think of three solutions:

1) Implement automatic nodes and A* for detecting obstacles for close vicinity pathfinding

2) Implement a "Problem solving mode" for actors. E.g. I'm not at attack distance, I have been pathfinding for 3 seconds without moving enough (treshold distance) -> Go to pathFindingProblemSolvemode, which is something like run away, don't shortcut, generate a different path not ending at the same node.

Some people have worked with recastnavigation. Perhaps this is easier to implement than options 1 & 2.

I noticed in the demo video of this feature that the FPS dipped below 20 at some points when a lot or even a few NPCs were near the player.

It does not matter if NPC's are "near player" or are not "near player". We update collisions for all NPCs in AI radius (nearly 100 meters).
So this slowdown is more likely a physics subsystem performance issue, rather than pathfinding one.