ABAKUS

Pretender hat geschrieben:Trotzalledem wird im Google Index z.b. Seiten aus diesem Ordner angezeigt

Pretender hat geschrieben:... aber immerhin hat Google diese Links verfolgt...

marc75 hat geschrieben:hast du php?

Code: Alles auswählen

if(stristr($_SERVER['HTTP_USER_AGENT'], 'googlebot')) {
 header("HTTP/1.0 404 Not Found");
 echo 'Kein Zugriff';
 exit();
}

.htaccess

Code: Alles auswählen

RewriteEngine on 
RewriteCond %{HTTP_USER_AGENT} ^googlebot 
RewriteRule !error403\.html

eventuell muss noch ein Status Code mitgesendet werden.

Netscape 6.0

Pretender hat geschrieben:Ich habe im Moment keine Logiles aber ich gehe doch davon aus, wenn die URL`s im Google-Index auftauchen, daß Google Sie auch gespidert hat.

Matt Cutts hat geschrieben:You might wonder why Google will sometimes return an uncrawled url reference, even if Googlebot was forbidden from crawling that url by a robots.txt file. There’s a pretty good reason for that: back when I started at Google in 2000, several useful websites (eBay, the New York Times, the California DMV) had robots.txt files that forbade any page fetches whatsoever. Now I ask you, what are we supposed to return as a search result when someone does the query [california dmv]? We’d look pretty sad if we didn’t return www.dmv.ca.gov as the first result. But remember: we weren’t allowed to fetch pages from www.dmv.ca.gov at that point. The solution was to show the uncrawled link when we had a high level of confidence that it was the correct link. Sometimes we could even pull a description from the Open Directory Project, so that we could give a lot of info to users even without fetching the page.