This setup allows me to redirect to the default 401.htm page (found in inetpub/custerr/), but I'm unable to redirect it. Even with the xml above (401-c.htm is found in the same directory) it only directs me to the default 401 page.

I don't have a clue what's going on here. Can anyone help?

Edit:
An alternative web.config setup I've tried that also did not work:

The system kept asking me for credentials even though I entered them correctly. When I closed the login box it did not give me my custom error page. Only a blank page with absolutely no content.

Edit2:
I've concluded that none of the above solutions actually work as they intercept the 401 error BEFORE it reaches SharePoint. I need a solution like the one @GavinB and @Amit Kumawat suggests though for some reason this does not work either. It seems I need a way to rewrite the actual data that SharePoint sends back with the 401 error. I've tried this: http://www.sharepointblues.com/2010/02/12/custom-error-page-adapter/ but cannot seem to find the correct assembly for the "controlType" attribute.

I'm sorry, but I cannot set any answer as correct since none of the proposed solutions have worked for me. I get either a blank page or the default 401.
–
ThomasDec 13 '11 at 6:55

Myself, with method 2, I have found that it does work for a 404, but not for a 401 - something to do with the login process. You can do this via the IIS GUI - I've a question here on the difference: serverfault.com/questions/466387/…
–
BGMJan 12 '13 at 21:35

Ok, the problem isn't the redirection then but your page. Start with baby steps, copy the OOTB page and use that in your solution with some small edits to the HTML to verify that the re-direction is working. Then slowly change the page to add branding or remove functionality. I have the above approach & code working perfectly
–
GavinBDec 7 '11 at 20:10

Oh also, the URL you see is the OOTB 401 page URL but it does load the custom page.
–
GavinBDec 7 '11 at 20:11

Error 401 is for Unauthorized Access (or Access Denied). In a SharePoint site, "Access Denied" is managed at two levels - IIS and SharePoint.

The web.config changes you did are meant for IIS. As "access denied" inside SharePoint site is managed by SharePoint, for IIS there is no 401 and hence no redirection.

To set custom 401 page for a SharePoint site, you have to use SharePoint API only.
Also, Your custom 401 page should be accessible to a user even if he is unauthorized on the site. It means your custom 401 page should not throw a 401 itself to unauthorized user!

By the way, you may loose "Sign in as different user" functionality if you change default access denied page in SharePoint .

So this means I have to set up anonymous access on my SharePoint installation right? I was hoping there was a way to avoid that. Thanks for the answer though. I'm pretty sure this is the best solution now.
–
ThomasDec 7 '11 at 14:32

I think no need for SharePoint site (or library) to be anonymous, if the custom 401 page is in file system. You can assign anonymous access just to the virtual directory(custerr) where your 401 page is placed
–
Amit KumawatDec 7 '11 at 15:16

Ok, so I tried this: /_layouts/ErrorPages/ is a folder that I've explicitly given anonymous access through IIS manager. It can now (verified) be reached by users not logged in. Then I tried to redirect using the solution proposed by GavinB below. Problem is I still get a blank page when the user aborts the authentication. Why, oh why is this so hard?
–
ThomasDec 13 '11 at 7:05

Use httpwatch in IE and see how web requests and response are working. This will give you better idea what is happening. Let me know your observation. If you don't have httpwatch, download it from here httpwatch.com/download.
–
Amit KumawatDec 13 '11 at 8:09

Sounds like #1 should work fine. I am thinking that the Browser File Handling of the web application might be preventing you from seeing the HTM file. You can update that setting to Permissive, and I believe HTM files should start to work fine.