The HTML Help compiler has some real limitations when it comes to character sets and the toc/index; it's amazing that the compiler hasn't been updated in such a long time.

I haven't yet found a good way to configure the character set for the index but you can change the font -- is it possible the default index font doesn't support Turkish? You can change the font with something like the following near the top of the hhk file (in the body section):

The compiler is apparently not Unicode enabled -- I found this link with issues other people had with Japanese (obviously different that Turkish but maybe there are some clues there...):
http://www.helpware.net/FAR/far_faq.htm

into the HTML files, and compile them using HTML Help Workshop. The text
displays correctly in the main window and in the TOC.

The one problem remaining is that the Index window contains some incorrectcharacters. The index includes indexterm elements and the documenttitles. It is clear when comparing the 1254 and 1252 encodings that the
incorrect characters are coming from the codepoints in 1252 instead of1254. For example, "small dotless i" (0xFD in Windows 1254) is replacedwith "small y acute" (0xFD in Windows 1252).

The help index comes from param elements like this contained in <object>elements in the HTML output:

<param name="Keyword" value="My turkish title">

It seems when the help compiler collects this data, it loses the connection
with the windows-1254 meta information that was at the top of the HTML fileit came from. I have not found a way to specify that the keyword indexshould be handled in the 1254 encoding.