Introduction

If you've ever used the File | Save As... menu in Internet Explorer, you might have noticed a few interesting options IE provides under the Save As Type drop-down box:

The options provided are:

Web Page, complete

Web Archive, single file

Web Page, HTML only

Text File

Most of these are self-explanatory, with the exception of the Web Archive (MHTML) format. What's neat about this format is that it bundles the web page and all of its references, into a single compact .MHT file. It's a lot easier to distribute a single self-contained file than it is to distribute a HTML file with a subfolder full of image/CSS/Flash/XML files referenced by that HTML file. In our case, we were generating HTML reports and we needed to check these reports into a document management system which expects a single file. The MHTML (*.mht) format solves this problem beautifully!

This project contains the MhtBuilder class, a 100% .NET managed code solution which can auto-generate a MHT file from a target URL, in one line of code. As a bonus, it will also generate all the other formats listed above, too. And it's completely free, unlike some commercial solutions you might find out there.

Background

I know people assume the worst of Microsoft, but the MHTML format is actually based on RFC standard 2557, compliant Multipart MIME Message (MHTML web archive). So it's an actual Internet standard! Web Archive, a.k.a. MHTML, is a remarkably simple plain text format which looks a lot like (and is in fact almost exactly identical to) an email. Here's the header of the MHT file you are viewing at the top of the page:

To generate a MHTML file, we simply merge together all of the files referenced in the HTML. The red line marks the first content block; there will be one content block for each file. We need to follow a few rules, though:

Use Quoted-Printable encoding for the text formats.

Use Base64 encoding for the binary formats.

Make sure the Content-Location has the correct absolute URL for each reference.

Not all websites will tolerate being packaged into a MHTML file. This version of Mht.Builder supports frames and IFrame, but watch out for pages that include lots of complicated JavaScript. You'll want to use the .StripScripts option on sites like that.

Using Mht.Builder

MhtBuilder comes with a complete demo app:

Try it out on your favorite website. The files will be generated by default in the \bin folder of the solution. Just click the View button to launch them. Bear in mind that for the Web Archive and complete tabs, all the content from the target web page must be downloaded to the /bin folder, so it might take a little while! Although I don't provide any feedback events yet, I do emit a lot of progress feedback via the Debug.Write, so switch to the debug output tab to see what's happening in real time.

There are four tabs here, just like the four options IE provides in its Save As Type options. In MhtBuilder, these are the four methods being called, in the order they appear on the tabs:

As of Windows XP Service Pack 2, HTML files opened from disk result in security blocks. In order to avoid this, we need to add the "Mark of the Web" to the file so IE knows what URL it came from, and can thus assign an appropriate security zone to the HTML. That's what the blnAddMark parameter is for; it causes the HTML file to be tagged with this single line at the top:

<!-- saved from url=(0027)http://www.codeproject.com/ -->

The other thing we need to do when saving these files is fix up the URLs. Any relative URLs such as:

<imgsrc="/images/standard/logo225x72.gif">

must be converted to absolute URLs like so:

<imgsrc="http://www.codeproject.com/images/standard/logo225x72.gif">

We do this using regular expressions, which gets us a NameValueCollection of all the references we need to fix. We loop through each reference and perform the fixup on the HTML string.

We use a similar technique to get a list of all the files we need to download, which are then downloaded via my WebClientEx class. Why use that instead of the built in Net.WebClient? Good question! Because it doesn't support HTTP compression. My class, on the other hand, does:

HTTP compression is a no-brainer: it increases your effective bandwidth by 75 percent by using standard GZIP compression-- courtesy of the SharpZipLib library.

Conclusion

Creating MHTML files isn't hard, but there are lots of little gotchas when dealing with HTML, regular expressions, and HTTP downloads. I tried to document all the difficult bits in the source code. I've also tested MhtBuilder on dozens of different websites so far with excellent results.

There are many more details and comments in the source code provided at the top of the article, so check it out. Please don't hesitate to provide feedback, good or bad! I hope you enjoyed this article. If you did, you may also like my other articles as well.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

Share

About the Author

My name is Jeff Atwood. I live in Berkeley, CA with my wife, two cats, and far more computers than I care to mention. My first computer was the Texas Instruments TI-99/4a. I've been a Microsoft Windows developer since 1992; primarily in VB. I am particularly interested in best practices and human factors in software development, as represented in my recommended developer reading list. I also have a coding and human factors related blog at www.codinghorror.com.

I have been here before in the past but I never downloaded this program, cause I didn't want to install dotNet again.. as of yet. so I went elsewhere. The visit here led me to finding another MHT editor that didn't need dotNET and in doing so, I found a bug in the program, and it lead to the me playing around with Windows some more as I was already doing. This led me to play around with Outlook and some MHT and EML files and...

People email me and I will tell you how to use OUTLOOK Express as a MHT editor and how to convert EML to MHT's without any added software. Yes, without any added software. WINDOWS already comes with an MHT editor.

If you read the Article above, the KEY words are the MIME technology. Not only can you use OUTLOOK Express as an MHT editor, you can convert your HTML pages to MHT and Convert your EML to MHT.

NO SOFTWARE TO DOWNLOAD. In addition, I will tell you how to Open an EML in Internet Explorer;no software to download either.

It's cool to download stuff like this that is free, however, all you need is someone with an bit of engineering know-how and is nice enough like me to point out something that is not discussed on the web as I have never come across it being discussed about what I do with OUTLOOK express.

I was going to post it here, but I going to post it on my blogspot instead. So like I said, just ask..

This solution is awesome, just what I was looking for! I noticed in ealier messages that file:///... based requests were originally supported, but that support was dropped. For anyone still interested, I was able to add that support (at least for my purposes) as follows:

'-- do we need to use a proxy to get to the web? If _ProxyUrl <> "" Then Dim wp As New WebProxy(_ProxyUrl) If _ProxyAuthenticationRequired Then If _ProxyUser <> "" And _ProxyPassword <> "" Then wp.Credentials = New NetworkCredential(_ProxyUser, _ProxyPassword) Else wp.Credentials = CredentialCache.DefaultCredentials End If wreq.Proxy = wp End If End If

'-- does the target website require credentials? If _AuthenticationRequired Then If _AuthenticationUser <> "" And _AuthenticationPassword <> "" Then wreq.Credentials = New NetworkCredential(_AuthenticationUser, _AuthenticationPassword) Else wreq.Credentials = CredentialCache.DefaultCredentials End If End If

'-- c.) Added If/Then check and type cast handling for _IsFileWebRequest = False: If _IsFileWebRequest = False Then '-- note that, if present, this will trigger a 304 exception '-- if the URL being retrieved is not newer than the specified '-- date/time If ifModifiedSince <> DateTime.MinValue Then CType(wreq, HttpWebRequest).IfModifiedSince = ifModifiedSince End If End If

'-- d.) Added If/Then check and type cast handling for _IsFileWebRequest = False: If _IsFileWebRequest = False Then '-- sometimes we need to transfer cookies to another URL; '-- this keeps them around in the object If KeepCookies Then If _PersistedCookies Is Nothing Then _PersistedCookies = New CookieContainer End If CType(wreq, HttpWebRequest).CookieContainer = _PersistedCookies End If End If

'-- sometimes URL is indeterminate, eg, "http://website.com/myfolder" '-- in that case the folder and file resolution MUST be done on '-- the server, and returned to the client as ContentLocation _ContentLocation = wresp.Headers("Content-Location") If _ContentLocation Is Nothing Then _ContentLocation = "" End If

'-- if we have string content, determine encoding type '-- (must cast to prevent Nothing) _DetectedContentType = wresp.Headers("Content-Type") If _DetectedContentType Is Nothing Then _DetectedContentType = "" End If

'-- f.) Added If/Then check and handling for _IsFileWebRequest = True (wanted to force to text/html for my purposes, but only for the initial HTML text content): If _IsFileWebRequest = True AndAlso _LoadedInitHTML = False Then _DetectedContentType = "text/html;charset=UTF-8" End If

If Me.ResponseIsBinary Then _DetectedEncoding = Nothing Else If _ForcedEncoding Is Nothing Then _DetectedEncoding = DetectEncoding(_DetectedContentType, _ResponseBytes) End If End If

'-- g.) Added If/Then check and handling for _IsFileWebRequest = True (wanted to force to utf-8 for my purposes, but only for the initial HTML text content): If _IsFileWebRequest = True AndAlso _LoadedInitHTML = False Then _DetectedEncoding = System.Text.Encoding.GetEncoding("utf-8") End If

First of all this is a really good project and very useful. The project works well for lot of webpages.I would like to save for example the following webpage http://www.mapunderwriting.co.uk/ but the quality is bad. When i inspect the source code a stylesheet is missing. If I add the stylesheet missing manually(taken from the file saved by IE), it does not solve my issue. What could be the issue here ?

The question I have is, is there a way to catch an http error when the url is openned and the mht is created. I have had a problem reciently where an error will occur and the file will still be created and display the error because it is a valid URL. If it's possible for this check to be put in place please let me know.

Hi.. firstly, thanks for the code. Its exactly what we wanted. However we have one problem. Our dev and QA environments work under http and the code works fine for that but when the site runs under HTTPS(in UAT/LIVE environment), it throws following error:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel

What I am doing is providing an HTML url for a file which gets created dynamically in the application folder (e.g. 'https://servername/Forms/PrintFiles/HTMLPage.html where 'servername' is the server where the site is hosted)

In IE, when I save as MHTML, relative files are not embedded. If I have an HTML page with relative files, I save it as an mhtml, then remove the local files, I see no images in the mhtml. Am I doing something wrong. Is this how this code behaves? (Since it is supposed to mimic IE).

we are using this piece of code for converting a lot of our reports which are in ASP to MHTML files which represents as an image of that week.The problem we are facing is the MHTML file being generated is just adding a reference of all the images and CSS to our production site after archiving also. suppose the main site is down or we are offline then these webpage archived reports are not showing up properly, So any of you can you tell us some tweaking of this code to make it working.

This is sort of an aside, but I figure that people who look at this page probably have a great familiarity with MHTML, and I need it to solve my problem. I have a webpage which contains a base64 string encoding a .png file. I also know the dimensions of the file etc. But the page will not know the image's URL.

I want to use this image as the background for one of the elements in my page. In Firefox/Safari/Opera, I can just use the "data: URI", i.e.

Unfortunately, Internet Explorer does not support the data: URI. But I figure that IE must have this functionality, because it would be ridiculous if it didn't. And it looks to me like MHTML is the most likely way that one can get this done with IE.

Does anyone know if this is possible, and if so, could you please provide a short code snippet explaining how?

...Looking at you blog Jeff I found this post from Kyle who was talking about a fix to open the mht in word. I applied the fix, and hey presto. cheersMoose---------------------Here's what Kyle said to do:

I've made two code changes to allow for the file to be opened in Word 2003. This made it work for me anyway.

Kyle

In builder.vb starting on line 474 change the procedure to the following:

Hi, Thank you very much for providing this well crafted code. I need to save html as "Web page complete", "Web page archive", "Web page as PDF" for an application to backup blogs. Currently I am doing it using your code . What I am interested in is to show the download progress as the web page is being saved. So I was thinking of combining the functions provided in MHT builder into the extended web browser control found at http://www.codeproject.com/csharp/ExtendedWebBrowser.asp.

Hello, great job!! I've tried your library and I found it very usefull: my best compliments.But I've found an error saving a particular web page: the url is "http://www.flcgil.it/notizie/news/2006/dicembre/firmato_il_contratto_di_lavoro_dell_enea_si_inizia_a_parlare_seriamente_dei_precari".

When I save the page, it gives me an exception: "System.IO.PathTooLongException: The path is too long after being fully qualified. Make sure path is less than 260 characters." But the url is 131 characters!The exception is thrown saving the page in mht (with the method SavePageArchive, setting the file storage on disk as temporary or permanent) and in Web page complete (with SavePageComplete).I suppose that during the saving, the library saves temporary html files where the name of the file, added to the url, exceeds 260 characters. If I'm right, the only solution is to give new shorter name to this temporary files.

I think I've fixed this bug.I've noticed two reason for the failure in saving my page (http://www.flcgil.it/notizie/news/2006/dicembre/firmato_il_contratto_di_lavoro_dell_enea_si_inizia_a_parlare_seriamente_dei_precari):

1) this page probably references itself, so the recursively download of the externally referenced files never ends. Yes, there is a property AllowRecursiveFileRetrieval that I can set to false to avoid this, but I want to be sure to download all necessary files.My idea is to permit the recursion only to a certain level of depth; reaching that limit, I suppose that the file is autoreferencing and I stop the recursive download. I've made the following changes in ExternalFile.vb:

'-- if this is an HTML file, it has dependencies of its own; '-- download them into a subfolder If (wf.IsHtml Or wf.IsCss) And recursive Then wf.DownloadExternalFiles(st, level, recursive) End If End IfEnd Sub

In the file Builder.vb, in the functions SavePageComplete, GetPageArchive and SavePageArchive, when I call the method DownloadExternalFiles I initialize the depth of the recursion to zero:_HtmlFile.DownloadExternalFiles(st, 0, _AllowRecursion)

2) When creating a new file name, it shouldn't be too long. This can happen expecially if the title of html page is used as file name. So I've modified the function MakeValidFilename in ExternalFile.vb:Private Function MakeValidFilename(ByVal s As String, Optional ByVal enforceLength As Boolean = False) As String If enforceLength Then End If '-- replace any invalid filesystem chars, plus leading/trailing/doublespacesDim name As String name = Regex.Replace(Regex.Replace(s, "[\/\\\:\*\?\""""\<\>\|]|^\s+|\s+$", ""), "\s{2,}", " ")'enforce the maximum length to 25 characters If name.Length > 25 Then Dim extension As String extension = Path.GetExtension(name) name = name.Substring(0, 25 - extension.Length) & extension End If Return nameEnd Function

(Maybe the optional parameter enforceLength was added to do something similar).There is also a function MakeValidFilename in the file Builder.vb, but I can't see when it is called, so I haven't modified it.

With this changes, I can save my web page. Has anybody done something similar? Is there something I've missed?

Thanks dear for this cute artical and good class library , but my question now is if i publish my created presentation ( 3 slides for example ) to htm format from Microsoft PowerPoint and i tried to save it using your library , each time i clicked in any link in the htm presentation, the mht.dll saves only first slide

i think the problem results that the URL of the browser not changed even if i click in any link in the htm presentation, and the first file ( called fram.htm ) and this page contains first slide only URL , so the mht.dll detects only this slide page.

and i expect that the solution will be if i can save mht files from brwser cache ( like file save as in the browser)

First of all thank you for the wonderful article and project. My question is not directly related to this project but I am posting it here in hope for getting some help. I am trying to write a proxy server to share internet connection. I know there are several small utilities available for this purpose but I wanted to do it my self so I can enhance it as I need. There is nice project (SSLProxy) at GotDotNet with source code but it is all in C# and I feel much comfortable using VB.NET. Also that project is pretty big to be converted to VB.NET. I wrote a small class using SOCKETS but it is not stable and sometimes it misses chunk of stream; especially when more then 2 connections are active. Any help or suggestion is much appreciated.

I really want to know how to take multiple html files at once. is Me.url a array?? I Iuput "http://www.codeproject.com (ENTER) http://www.google.com" into the Target URL,but it show "unable to download 'http://www.codeproject.com/%0D%0Ahttp:/www.google.com': The remote server returned an error: (400) Bad Request."

How can I do??

Sorry,I know little about VB.NET and my English is terriable.please help me~~

I was hoping to point this fantastic program at a URL and it will make each and every html file on that site into mht. It seems to take the index page only and then stops. It also saves the name as the title, rather than the actual file name...

I really like how this works but if this is able to do an entire site, and names the files using the actual file name rather than the title, could you let me know.