Use SocialGraph API because it lets you work with ID's rather than normal objects, which is faster. Make sure you max out the page size to minimize the number of queries. There are examples in the downloadable code.

Get the list of IDs of all your followers.

Get the list of IDs of all the other person's followers.

Use the LINQ Except operator to find the the other person's follower IDs that aren't in your follower IDs.