By default, Spotlight excludes many hidden, System, and developer directories from the metadata index for simplicity and ease of use by the majority of users. These exclusions include things like hidden UNIX directories and ADC Reference Documentation. It would be very helpful for some users to have some of these excluded areas available for Spotlight searching. While mdimport -f will allow you to manually force Spotlight to add a directory to its index, many of Spotlight's command-line tools make references to filter rules. I wondered where these rules might be kept, and how they might be altered.

I was hoping there would be a text-based configuration file somewhere, but this appeared at first glance to not be the case. It turns out the excluded directories are hardcoded into Spotlight's mds binary, located in /System: Library: Frameworks: CoreServices.framework: Versions: A: Frameworks: Metadata.framework: Versions: A: Support: mds. You can do a strings on the above binary to see them -- or here's the full list of excluded directories.

However, while looking in the .Spotlight-V100 directory at the root of the drive, I found a file called _rules.plist, which can be used to tweak the exclusions.

The contents of this file, by default, are (note that some rows have been split for a narrower display -- you shouldn't be copying and pasting this bit anyway!):

To add excluded directories to the metadata index, _rules.plist can be modified by adding an array of directories that you'd like searched:

<array>
<string>/private</string>
<string>/usr</string>
</array>

Add this section directly below the <key>INCLUDE<key> line. In this example, the /private and /usr directories have been added to the metadata index. After this change, execute sudo mdutil -E / to remove the index and force the volume to reindex. After this, Spotlight has indexed and is able to search /private (including things like /etc) and /usr.

A side effect of this modification is that these areas now become searchable by the Finder's conventional "Find" functionality (Find...; Command-F; or search field in Finder window). Before, a search for httpd.conf yields no results (unless you happen to have other files named "httpd.conf" files around in non-excluded areas). After adding /private, which contains /etc (which is really a symbolic link to /private/etc), "httpd.conf" now appears in search results at its expected location of /private/etc/httpd/httpd.conf.

Further, be advised that explicitly excluded items may need to be explicitly included. For example, Spotlight specifically excludes /Developer/ADC Reference Library by default. Adding /Developer alone to the INCLUDE stanza may not index it; you may need to explicitly include /Developer/ADC Reference Library to cause it to be indexed. Note that I did not yet test this theory; this point is mentioned for information only.

The only "bummer" about Spotlight so far is that there dont appear to be content importers available for system files like .conf, .plist, and other "generic" files like hostconfig.

I may be missing something but doing mdimport on a directory like /private only imports metadata like create/modify dates, and a couple other somewhat "low value" bits of info. You cant search inside the files.

I saw a suggestion to use SetFileInfo (a cli tool installed with the Developer Tools) that will change the file Kind to a generic text file...which of course will get indexed.

Can we move away from using ':' to separate paths? It's really old-skool and the Unix kernel uses posix paths (with / ) for everything these days. The : is really only seen in AppleScripts and some old classic apps; but for a 10.4 hint to use : is really ancient ...

I use a slash on shorter paths that can be copied and pasted. However, on long paths, which I pad with spaces so they'll wrap, I switched to colons.

Why? Because I was tired of answering tech support emails about why a copy and pasted path didn't work -- people were just copying the whole line and pasting it into the Terminal. I'm open to other suggestions, but since switching, I haven't had one person email asking why a pasted path didn't work...

I used pico too and it doesn't work - unless you remove the extra <array/> after the </array> - - if that makes sense.
If you don't - the _rules.plist just gets over-written back to default.
The following worked for me:

After I remove the index and it is re created mine defaults back to the original. Is this what is supposed to happen?
What I do know is that Spotlight is not showing any files or folders that I added to _rules.plist

This of course assumes you have the dev tools installed. It will open the _rules.plist in a su'd Property List editor, then you just add siblings for the INCLUDE element and simply fill in as many paths as you want.

This might sound idiotic, but it's great for me because later on I can delete all the APIs I don't care about!

Just to test it I copied the whole thing. My ContentIndex.db grew to 96 megs, and the "mds" process has RSIZE of 7.4 megs, VSIZE of 46 megs, when I invoke Spotlight and do a search.

On my system it took 2 minutes to copy and 10 minutes for Spotlight to index (as evidenced by CPU monitor or load averages in UNIX "w" command). My system is a dual G5 so I imagine other users might take as much as an hour (but you only have to do it once :-)

Here is a one-liner that will search out all Text files on your drive and set the filetype to TEXT. Modifying what the grep(s) search for and the "TEXT" part of SetfFile this one liner can be used for other types of files that the "file" command can distinguish. Nake sure you use the "file" command to see what it outputs for the type of file you want to change, and use GetFileInfo on example file of that type to see the four-char code needed.

Thanks for this sub-hint. I'm a pikey who's still on 10.4 and I use a personal wiki to keep random notes. I set up MediaWiki on the system Apache install yonks ago, but it got annoying lately that Spotlight didn't index it. I installed MoinMoin Desktop Edition thinking its plain text files would be the answer but Spotlight didn't seem to index the files except the metadata. Running SetType -t TEXT on the "pages/*/revisions/" trees fixed the problem (which would seem to contradict kirkmc's comment). It's still not perfect - every time you add a new page or edit an existing one you have to set the type on the new file. I'll address this with a cron job to pick those up every hour or so, and just hope that anything I've been editing/creating more recently than that is fresh in my mind anyway :)

I expect that in 10.5 there's a way to write a plugin to feed Spotlight the right data without this hack (perhaps even linkifying the results so they're URLs), or a way to simply configure Spotlight differently, but by now I think I'm better off waiting for 10.6.

I ran into a problem with Spotlight. I couldn't find php.ini because
it's in /private/etc/apache2 and /private is not indexed by
Spotlight.

Meanwhile, the current OS X version is 10.5 and the path of the
file to be altered is:
/.Spotlight-V100/Store-V1/Exclusions.plist

Also, for most users, it's still hard to edit a hidden file that also
needs super user privileges. Maybe you could mention it can be
done by dropping you text editor on Pseudo first. Pseudo let's you
use applications with root rights.

Was this hint to allow Tiger 10.4 to index the names of files within bsd directories such as "/usr" or to index their content as well?

In Snow Leopard 10.6, the names of system files are indexed by default, but not the content. If I copy a file from a directory such as "/usr" to the desktop, for instance, Spotlight will index the content as well.

If this hint also allows content to be indexed in Tiger 10.4, I don't know what to do for Snow Leopard 10.6. The _rules.plist file no longer exists and the Exclusions.plist file that is in /.Spotlight-V100/Store-V1/ does not have a tree for including directories, just a tree for excluding directories.