@rivervanrain
I haven't used GWT, but I've run my robots.txt through various robot file validators that can be found online and it seems to be correct. I'm not sure how the robots.txt file work for Elgg, but my crawler just isn't... view reply

This site does have users who create content with access rules to their liking. We will filter the results depending on the access of the logged in user (or public user).
Our application is not hosted on the internet (rather within the our... view reply

I've currently installed Elgg v1.12.5, with robots.txt in the root directory of Elgg with the following content:
User-agent: *
Disallow:
Previously, I used the built-in robots.php with the same rules.
The crawler that I...

@rivervanrain
I haven't used GWT, but I've run my robots.txt through various robot file validators that can be found online and it seems to be correct. I'm not sure how the robots.txt file work for Elgg, but my crawler just isn't able to get into the private/closed content.

@steve_clay
We're using the Google Search Appliance, and I've tried to do the authentication through form (or cookie store method) but it doesn't pick up the login form...