Introduction

If you've ever used the File | Save As... menu in Internet Explorer, you might have noticed a few interesting options IE provides under the Save As Type drop-down box:

The options provided are:

Web Page, complete

Web Archive, single file

Web Page, HTML only

Text File

Most of these are self-explanatory, with the exception of the Web Archive (MHTML) format. What's neat about this format is that it bundles the web page and all of its references, into a single compact .MHT file. It's a lot easier to distribute a single self-contained file than it is to distribute a HTML file with a subfolder full of image/CSS/Flash/XML files referenced by that HTML file. In our case, we were generating HTML reports and we needed to check these reports into a document management system which expects a single file. The MHTML (*.mht) format solves this problem beautifully!

This project contains the MhtBuilder class, a 100% .NET managed code solution which can auto-generate a MHT file from a target URL, in one line of code. As a bonus, it will also generate all the other formats listed above, too. And it's completely free, unlike some commercial solutions you might find out there.

Background

I know people assume the worst of Microsoft, but the MHTML format is actually based on RFC standard 2557, compliant Multipart MIME Message (MHTML web archive). So it's an actual Internet standard! Web Archive, a.k.a. MHTML, is a remarkably simple plain text format which looks a lot like (and is in fact almost exactly identical to) an email. Here's the header of the MHT file you are viewing at the top of the page:

To generate a MHTML file, we simply merge together all of the files referenced in the HTML. The red line marks the first content block; there will be one content block for each file. We need to follow a few rules, though:

Use Quoted-Printable encoding for the text formats.

Use Base64 encoding for the binary formats.

Make sure the Content-Location has the correct absolute URL for each reference.

Not all websites will tolerate being packaged into a MHTML file. This version of Mht.Builder supports frames and IFrame, but watch out for pages that include lots of complicated JavaScript. You'll want to use the .StripScripts option on sites like that.

Using Mht.Builder

MhtBuilder comes with a complete demo app:

Try it out on your favorite website. The files will be generated by default in the \bin folder of the solution. Just click the View button to launch them. Bear in mind that for the Web Archive and complete tabs, all the content from the target web page must be downloaded to the /bin folder, so it might take a little while! Although I don't provide any feedback events yet, I do emit a lot of progress feedback via the Debug.Write, so switch to the debug output tab to see what's happening in real time.

There are four tabs here, just like the four options IE provides in its Save As Type options. In MhtBuilder, these are the four methods being called, in the order they appear on the tabs:

As of Windows XP Service Pack 2, HTML files opened from disk result in security blocks. In order to avoid this, we need to add the "Mark of the Web" to the file so IE knows what URL it came from, and can thus assign an appropriate security zone to the HTML. That's what the blnAddMark parameter is for; it causes the HTML file to be tagged with this single line at the top:

<!-- saved from url=(0027)http://www.codeproject.com/ -->

The other thing we need to do when saving these files is fix up the URLs. Any relative URLs such as:

<imgsrc="/images/standard/logo225x72.gif">

must be converted to absolute URLs like so:

<imgsrc="http://www.codeproject.com/images/standard/logo225x72.gif">

We do this using regular expressions, which gets us a NameValueCollection of all the references we need to fix. We loop through each reference and perform the fixup on the HTML string.

We use a similar technique to get a list of all the files we need to download, which are then downloaded via my WebClientEx class. Why use that instead of the built in Net.WebClient? Good question! Because it doesn't support HTTP compression. My class, on the other hand, does:

HTTP compression is a no-brainer: it increases your effective bandwidth by 75 percent by using standard GZIP compression-- courtesy of the SharpZipLib library.

Conclusion

Creating MHTML files isn't hard, but there are lots of little gotchas when dealing with HTML, regular expressions, and HTTP downloads. I tried to document all the difficult bits in the source code. I've also tested MhtBuilder on dozens of different websites so far with excellent results.

There are many more details and comments in the source code provided at the top of the article, so check it out. Please don't hesitate to provide feedback, good or bad! I hope you enjoyed this article. If you did, you may also like my other articles as well.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Share

About the Author

My name is Jeff Atwood. I live in Berkeley, CA with my wife, two cats, and far more computers than I care to mention. My first computer was the Texas Instruments TI-99/4a. I've been a Microsoft Windows developer since 1992; primarily in VB. I am particularly interested in best practices and human factors in software development, as represented in my recommended developer reading list. I also have a coding and human factors related blog at www.codinghorror.com.

Comments and Discussions

The question I have is, is there a way to catch an http error when the url is openned and the mht is created. I have had a problem reciently where an error will occur and the file will still be created and display the error because it is a valid URL. If it's possible for this check to be put in place please let me know.

Hi.. firstly, thanks for the code. Its exactly what we wanted. However we have one problem. Our dev and QA environments work under http and the code works fine for that but when the site runs under HTTPS(in UAT/LIVE environment), it throws following error:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel

What I am doing is providing an HTML url for a file which gets created dynamically in the application folder (e.g. 'https://servername/Forms/PrintFiles/HTMLPage.html where 'servername' is the server where the site is hosted)

In IE, when I save as MHTML, relative files are not embedded. If I have an HTML page with relative files, I save it as an mhtml, then remove the local files, I see no images in the mhtml. Am I doing something wrong. Is this how this code behaves? (Since it is supposed to mimic IE).

we are using this piece of code for converting a lot of our reports which are in ASP to MHTML files which represents as an image of that week.
The problem we are facing is the MHTML file being generated is just adding a reference of all the images and CSS to our production site after archiving also. suppose the main site is down or we are offline then these webpage archived reports are not showing up properly, So any of you can you tell us some tweaking of this code to make it working.

This is sort of an aside, but I figure that people who look at this page probably have a great familiarity with MHTML, and I need it to solve my problem. I have a webpage which contains a base64 string encoding a .png file. I also know the dimensions of the file etc. But the page will not know the image's URL.

I want to use this image as the background for one of the elements in my page. In Firefox/Safari/Opera, I can just use the "data: URI", i.e.

Unfortunately, Internet Explorer does not support the data: URI. But I figure that IE must have this functionality, because it would be ridiculous if it didn't. And it looks to me like MHTML is the most likely way that one can get this done with IE.

Does anyone know if this is possible, and if so, could you please provide a short code snippet explaining how?

...Looking at you blog Jeff I found this post from Kyle who was talking about a fix to open the mht in word.
I applied the fix, and hey presto.
cheers
Moose
---------------------
Here's what Kyle said to do:

I've made two code changes to allow for the file to be opened in Word 2003. This made it work for me anyway.

Kyle

In builder.vb starting on line 474 change the procedure to the following:

Hi,
Thank you very much for providing this well crafted code. I need to save html as "Web page complete", "Web page archive", "Web page as PDF" for an application to backup blogs. Currently I am doing it using your code . What I am interested in is to show the download progress as the web page is being saved. So I was thinking of combining the functions provided in MHT builder into the extended web browser control found at http://www.codeproject.com/csharp/ExtendedWebBrowser.asp.

Hello,
great job!! I've tried your library and I found it very usefull: my best compliments.
But I've found an error saving a particular web page: the url is "http://www.flcgil.it/notizie/news/2006/dicembre/firmato_il_contratto_di_lavoro_dell_enea_si_inizia_a_parlare_seriamente_dei_precari".

When I save the page, it gives me an exception: "System.IO.PathTooLongException: The path is too long after being fully qualified. Make sure path is less than 260 characters." But the url is 131 characters!
The exception is thrown saving the page in mht (with the method SavePageArchive, setting the file storage on disk as temporary or permanent) and in Web page complete (with SavePageComplete).
I suppose that during the saving, the library saves temporary html files where the name of the file, added to the url, exceeds 260 characters. If I'm right, the only solution is to give new shorter name to this temporary files.

I think I've fixed this bug.
I've noticed two reason for the failure in saving my page (http://www.flcgil.it/notizie/news/2006/dicembre/firmato_il_contratto_di_lavoro_dell_enea_si_inizia_a_parlare_seriamente_dei_precari):

1) this page probably references itself, so the recursively download of the externally referenced files never ends. Yes, there is a property AllowRecursiveFileRetrieval that I can set to false to avoid this, but I want to be sure to download all necessary files.
My idea is to permit the recursion only to a certain level of depth; reaching that limit, I suppose that the file is autoreferencing and I stop the recursive download. I've made the following changes in ExternalFile.vb:

'-- if this is an HTML file, it has dependencies of its own;
'-- download them into a subfolder
If (wf.IsHtml Or wf.IsCss) And recursive Then
wf.DownloadExternalFiles(st, level, recursive)
End If
End If
End Sub

In the file Builder.vb, in the functions SavePageComplete, GetPageArchive and SavePageArchive, when I call the method DownloadExternalFiles I initialize the depth of the recursion to zero:
_HtmlFile.DownloadExternalFiles(st, 0, _AllowRecursion)

2) When creating a new file name, it shouldn't be too long. This can happen expecially if the title of html page is used as file name. So I've modified the function MakeValidFilename in ExternalFile.vb:
Private Function MakeValidFilename(ByVal s As String, Optional ByVal enforceLength As Boolean = False) As String
If enforceLength Then
End If
'-- replace any invalid filesystem chars, plus leading/trailing/doublespacesDim name As String
name = Regex.Replace(Regex.Replace(s, "[\/\\\:\*\?\""""\\|]|^\s+|\s+$", ""), "\s{2,}", " ")'enforce the maximum length to 25 characters
If name.Length > 25 Then
Dim extension As String
extension = Path.GetExtension(name)
name = name.Substring(0, 25 - extension.Length) & extension
End If
Return name
End Function

(Maybe the optional parameter enforceLength was added to do something similar).
There is also a function MakeValidFilename in the file Builder.vb, but I can't see when it is called, so I haven't modified it.

With this changes, I can save my web page. Has anybody done something similar? Is there something I've missed?

Thanks dear for this cute artical and good class library , but my question now is if i publish my created presentation ( 3 slides for example ) to htm format from Microsoft PowerPoint and i tried to save it using your library , each time i clicked in any link in the htm presentation, the mht.dll saves only first slide

i think the problem results that the URL of the browser not changed even if i click in any link in the htm presentation, and the first file ( called fram.htm ) and this page contains first slide only URL , so the mht.dll detects only this slide page.

and i expect that the solution will be if i can save mht files from brwser cache ( like file save as in the browser)

First of all thank you for the wonderful article and project. My question is not directly related to this project but I am posting it here in hope for getting some help. I am trying to write a proxy server to share internet connection. I know there are several small utilities available for this purpose but I wanted to do it my self so I can enhance it as I need. There is nice project (SSLProxy) at GotDotNet with source code but it is all in C# and I feel much comfortable using VB.NET. Also that project is pretty big to be converted to VB.NET. I wrote a small class using SOCKETS but it is not stable and sometimes it misses chunk of stream; especially when more then 2 connections are active.
Any help or suggestion is much appreciated.

Before my contribution, I should state I enjoyed very much reading through the code !!!

I think I have found and fix two small bugs in the WebFile class.

Bug #1
The code wrongly assumes that the URL and <Base HREF=..> are identical. To fix it, I made three changes:
1) I added a private member to the class:Private _BaseUrlFolder As String

2) _BaseUrlFolder is set in the ProcessHtml() method:
If BaseUrlFolder <> "" Then
If BaseUrlFolder.EndsWith("/") Then
_BaseUrlFolder = BaseUrlFolder.Substring(0, BaseUrlFolder.Length - 1)
Else
_BaseUrlFolder = BaseUrlFolder
End If
End If

I really want to know how to take multiple html files at once. is Me.url a array??
I Iuput "http://www.codeproject.com (ENTER) http://www.google.com" into the Target URL,but it show "unable to download 'http://www.codeproject.com/%0D%0Ahttp:/www.google.com': The remote server returned an error: (400) Bad Request."

How can I do??

Sorry,I know little about VB.NET and my English is terriable.
please help me~~

I was hoping to point this fantastic program at a URL and it will make each and every html file on that site into mht. It seems to take the index page only and then stops. It also saves the name as the title, rather than the actual file name...

I really like how this works but if this is able to do an entire site, and names the files using the actual file name rather than the title, could you let me know.

UrlExt = Path.GetExtension(Url)
'-- do we need to use a proxy to get to the web?
If _ProxyUrl <> "" Then
Dim wp As New WebProxy(_ProxyUrl)
If _ProxyAuthenticationRequired Then
If _ProxyUser <> "" And _ProxyPassword <> "" Then
wp.Credentials = New NetworkCredential(_ProxyUser, _ProxyPassword)
Else
wp.Credentials = CredentialCache.DefaultCredentials
End If
wreq.Proxy = wp
End If
End If

'-- does the target website require credentials?
If _AuthenticationRequired Then
If _AuthenticationUser <> "" And _AuthenticationPassword <> "" Then
wreq.Credentials = New NetworkCredential(_AuthenticationUser, _AuthenticationPassword)
Else
wreq.Credentials = CredentialCache.DefaultCredentials
End If
End If

'-- sometimes we need to transfer cookies to another URL;
'-- this keeps them around in the object
If KeepCookies Then
If _PersistedCookies Is Nothing Then
_PersistedCookies = New CookieContainer
End If
End If

'-- sometimes URL is indeterminate, eg, "http://website.com/myfolder"
'-- in that case the folder and file resolution MUST be done on
'-- the server, and returned to the client as ContentLocation
_ContentLocation = wresp.Headers("Content-Location")
If _ContentLocation Is Nothing Then
_ContentLocation = ""
End If

'-- if we have string content, determine encoding type
'-- (must cast to prevent Nothing)
_DetectedContentType = wresp.Headers("Content-Type")
If _DetectedContentType Is Nothing Then
_DetectedContentType = ""
Else
_DetectedContentType = ContentTypeFromExtension(UrlExt)
End If
If IsBinaryFromExtension(UrlExt) Then
_DetectedEncoding = Nothing
Else
If _ForcedEncoding Is Nothing Then
_DetectedEncoding = DetectEncoding(_DetectedContentType, _ResponseBytes)
End If
End If

End Sub

3. Maybe not necessary, because I added it before creating the functions in 1)
In External.vb:

3a. After Private _ContentType As String I added:
Private _ContentTypeBefore As String

3e. In Private Sub AddMatchesToCollection I added:
Dim urlRegex2 As New Regex("^files*:///\w+", RegexOptions.IgnoreCase)
and changed:
If Not urlRegex.IsMatch(value) Then
into:
If Not urlRegex.IsMatch(value) And Not urlRegex2.IsMatch(value) Then

We have an application for real-estate brokers, who receive every day new data concerning new real estate properties on the market.
Every day, they send this information to their prospects.
For this, they generate a HTML file and mail this as an attachment.
(I have created a HTML template file and the data is automatically merged, based on this template which results in a HTML file).

The problem was and still is, that they also want to show photos of the real estate properties.
I used to explain them that HTML is just plain text and if they want their recipients to see the photos, I would have to place hard references to the location of these photos (on their local server or after upload on their webserver).

For the real estate brokers, that use Outlook/Exchange I created a solution using the same generated HTML file mentioned above based on these articles (http://www.outlookcode.com/d/code/htmlimg.htm or http://www.dimastr.com/redemption/objects.htm).

For the other real estate brokers I was looking for a MHT(ML) solution using the same generated HTML file.

The name of this HTML file to be converted could have the following format:
file:///C:/Documents%20and%20Settings/ron.OMAWEB/Local%20Settings/Temp/PeriodiekAanbod/Obj_1506.html

The name of the photos could have the following format:
<img class="imgfoto" src="file:///C:\Documents%20and%20Settings\ron.OMAWEB\Local%20Settings\Temp\PeriodiekAanbod\Thumbnails\121313001211000w000649400000001000.jpg" alt="Cornelia van Arkeldijk 58" />

You can see an example of the local HTML save in IE as a webarchive on:
http://home.wanadoo.nl/rsegijn/example.mht

You mentioned that at first you also supported local HTML files.
Could you please help me on my way?
I think I have to set some property UriScheme.File or UriScheme.Http
And based on this property use FileWebRequest and FileWebResponse or HttpWebRequest and HttpWebResponse.
And in case of a FileWebResponse set the Content-Type to "text/html", because FileWebResponse always returns "application/octet-stream".
And in Sub SetUrl match to "file:///[^/'""]+" and _Url.LastIndexOf("/") > 8 in case of UriScheme.File
And a lot of regex handling have to be adjusted (very interesting stuff, but I have NO idea what it does, even after reading a regex tutorial on the web .
etc.
etc.

Integrated your code into our application so that we could send MHT- formatted data to Interfax.Net for faxing -- it works great; thanks!! Next question is that we retain the MHT data in a database and would like to view it from inside the app -- with HTML, we simply write the text into the response object using Response.Write. For MHT data, this displays the MHT headers, etc. -- any idea what we're doing wrong? We think it might be as simple as setting the Response.ContentType or adding some headers, but we've not been able to find the magic combination that works... TIA -- john

Glad to see that someone else will claim to have the TI994A in their computer background. Taught myself assembly language for it, and even went as far as self publishing a technical manual for expanding it via the PEB, as well as demonstrating the only TI994A with a Motorola math coprocessor. But that is all in the basement now.

hi
thx for that great code !
i need to save local html's generated by code into .mht files and i don't know how to proceed ..., even with CDO i didn't knwo how to do it, it seems like it takes only url's like ur code, i had a look at ur code didn't understand all but u r using regular expressions for that aim and i'm not good in regex

How is code this granular supposed to help anyone trying to do this? With how little help there is on converting to mHTML you'd think you'd give a more straightforward example, rather than a multi-class application that is tough to follow. Thanks for the help.......This isn't supposed to be a coding contest.