Google-Crawl: Von kurz nach lang...

Beitrag von **Luise** » 11.09.2004, 09:21

Southmedia hat geschrieben:Luise, könntest du Ausschnitte aus dem Logfile posten bitte? Danke.

Logfile:

1094692793*64.68.82.141*crawler13.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*Startseite**
1094693077*64.68.83.155*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Startseite**
1094694861*64.68.82.168*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*Referenzen**
1094696266*64.68.82.185*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*E---**
1094696266*64.68.82.185*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*design**
1094696311*64.68.82.146*crawler13.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*M---**
1094696554*64.68.82.184*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*edv**
1094696716*64.68.82.170*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*Impressum**
1094696744*64.68.82.184*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*M---**
1094697182*64.68.82.143*crawler13.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*O---**
1094697717*64.68.82.167*crawler14.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*M---**
1094698259*64.68.82.14*crawler10.googlebot.com*Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;*Batmans**
1094700664*64.68.83.155*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Startseite**
1094701952*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Startseite**
1094701992*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Referenzen**
1094702389*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*M---**
1094702787*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*M---**
1094703115*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*design**
1094703280*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*M---**
1094703426*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*edv**
1094703688*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*O---**
1094703808*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Batmans**
1094704137*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*E---**
1094704238*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Impressum**
1094704339*64.68.83.141*crawl34.googlebot.com*Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;*Startseite**

Statistik

Code: Alles auswählen

3544 09.09.04 03&#58;19&#58;53 crawler13.googlebot.com unknown unknown Startseite 
3545 09.09.04 03&#58;24&#58;37 crawl34.googlebot.com Netscape 6 unknown Startseite 
3546 09.09.04 03&#58;54&#58;21 crawler14.googlebot.com unknown unknown Referenzen 
3547 09.09.04 04&#58;17&#58;46 crawler14.googlebot.com unknown unknown E--- 
3548 09.09.04 04&#58;17&#58;46 crawler14.googlebot.com unknown unknown design 
3549 09.09.04 04&#58;18&#58;31 crawler13.googlebot.com unknown unknown M---
3550 09.09.04 04&#58;22&#58;34 crawler14.googlebot.com unknown unknown edv 
3551 09.09.04 04&#58;25&#58;16 crawler14.googlebot.com unknown unknown Impressum 
3552 09.09.04 04&#58;25&#58;44 crawler14.googlebot.com unknown unknown M--- 
3553 09.09.04 04&#58;33&#58;02 crawler13.googlebot.com unknown unknown O--- 
3554 09.09.04 04&#58;41&#58;57 crawler14.googlebot.com unknown unknown M---
3555 09.09.04 04&#58;50&#58;59 crawler10.googlebot.com unknown unknown Batmans 
3556 09.09.04 05&#58;31&#58;04 crawl34.googlebot.com Netscape 6 unknown Startseite 
3557 09.09.04 05&#58;52&#58;32 crawl34.googlebot.com Netscape 6 unknown Startseite 
3558 09.09.04 05&#58;53&#58;12 crawl34.googlebot.com Netscape 6 unknown Referenzen 
3559 09.09.04 05&#58;59&#58;49 crawl34.googlebot.com Netscape 6 unknown M---
3560 09.09.04 06&#58;06&#58;27 crawl34.googlebot.com Netscape 6 unknown M--- 
3561 09.09.04 06&#58;11&#58;55 crawl34.googlebot.com Netscape 6 unknown design 
3562 09.09.04 06&#58;14&#58;40 crawl34.googlebot.com Netscape 6 unknown M--- 
3563 09.09.04 06&#58;17&#58;06 crawl34.googlebot.com Netscape 6 unknown edv 
3564 09.09.04 06&#58;21&#58;28 crawl34.googlebot.com Netscape 6 unknown O--- 
3565 09.09.04 06&#58;23&#58;28 crawl34.googlebot.com Netscape 6 unknown Batmans 
3566 09.09.04 06&#58;28&#58;57 crawl34.googlebot.com Netscape 6 unknown E--- 
3567 09.09.04 06&#58;30&#58;38 crawl34.googlebot.com Netscape 6 unknown Impressum 
3568 09.09.04 06&#58;32&#58;19 crawl34.googlebot.com Netscape 6 unknown Startseite

Im Übrigen hatte ich auch einen "crawl35" am 03.09.

*Beitrag editiert, da ich doch noch Logfiles aus meinen persönlichen Logs hatte, allerdings von einer anderen Domain*

von **Anzeige von ABAKUS** »

SEO Consulting bei ABAKUS Internet Marketing
Erfahrung seit 2002

persönliche Betreuung
individuelle Beratung
kompetente Umsetzung

Jetzt anfragen: 0511 / 300325-0.

Beitrag von **Southmedia** » 11.09.2004, 10:28

Dein Logfileprogramm macht wohl aus "Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)" dann "Netscape 6"... haste keine normale Apache-Logs?

Beitrag von **Luise** » 11.09.2004, 10:47

Southmedia hat geschrieben:Dein Logfileprogramm macht wohl aus "Mozilla/5.0 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)" dann "Netscape 6"...

was mein Programm daraus macht, finde ich gar nicht so interessant, sondern viel mehr, daß da überhaupt etwas ist, wo es was draus machen kann. Der "normale" Bot hat so was ja nicht.

Southmedia hat geschrieben:haste keine normale Apache-Logs?

Aber natürlich

Code: Alles auswählen

64.68.82.141 - - &#91;09/Sep/2004&#58;03&#58;19&#58;53 +0200&#93; "GET / HTTP/1.0" 200 6235 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.83.155 - - &#91;09/Sep/2004&#58;03&#58;24&#58;36 +0200&#93; "GET /robots.txt HTTP/1.1" 200 177 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.155 - - &#91;09/Sep/2004&#58;03&#58;24&#58;37 +0200&#93; "GET /robots.txt HTTP/1.1" 200 177 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.155 - - &#91;09/Sep/2004&#58;03&#58;24&#58;37 +0200&#93; "GET / HTTP/1.1" 200 6527 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.82.168 - - &#91;09/Sep/2004&#58;03&#58;54&#58;20 +0200&#93; "GET /robots.txt HTTP/1.0" 200 177 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.168 - - &#91;09/Sep/2004&#58;03&#58;54&#58;21 +0200&#93; "GET /referenzen/ HTTP/1.0" 200 6538 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.185 - - &#91;09/Sep/2004&#58;04&#58;17&#58;46 +0200&#93; "GET /referenzen/f---.html HTTP/1.0" 200 3326 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.185 - - &#91;09/Sep/2004&#58;04&#58;17&#58;46 +0200&#93; "GET /design/ HTTP/1.0" 200 3043 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.146 - - &#91;09/Sep/2004&#58;04&#58;18&#58;31 +0200&#93; "GET /referenzen/z---.html HTTP/1.0" 200 3345 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.184 - - &#91;09/Sep/2004&#58;04&#58;22&#58;34 +0200&#93; "GET /edv/ HTTP/1.0" 200 3080 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.170 - - &#91;09/Sep/2004&#58;04&#58;25&#58;16 +0200&#93; "GET /impressum/impressum.html HTTP/1.0" 200 4006 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.184 - - &#91;09/Sep/2004&#58;04&#58;25&#58;44 +0200&#93; "GET /referenzen/z---.html HTTP/1.0" 200 3311 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.143 - - &#91;09/Sep/2004&#58;04&#58;33&#58;02 +0200&#93; "GET /referenzen/f---.html HTTP/1.0" 200 3332 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.167 - - &#91;09/Sep/2004&#58;04&#58;41&#58;57 +0200&#93; "GET /referenzen/z---.html HTTP/1.0" 200 3338 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.14 - - &#91;09/Sep/2004&#58;04&#58;51&#58;00 +0200&#93; "GET /referenzen/batman.html HTTP/1.0" 200 4073 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.170 - - &#91;09/Sep/2004&#58;05&#58;10&#58;35 +0200&#93; "GET /impressum/kontakt.html HTTP/1.0" 200 3957 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.82.168 - - &#91;09/Sep/2004&#58;05&#58;18&#58;42 +0200&#93; "GET /impressum/disclaimer.html HTTP/1.0" 200 7634 "-" "Googlebot/2.1 &#40;+http&#58;//www.google.com/bot.html&#41;"
64.68.83.155 - - &#91;09/Sep/2004&#58;05&#58;31&#58;04 +0200&#93; "GET /robots.txt HTTP/1.1" 200 177 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.155 - - &#91;09/Sep/2004&#58;05&#58;31&#58;04 +0200&#93; "GET / HTTP/1.1" 200 6527 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;05&#58;52&#58;32 +0200&#93; "GET / HTTP/1.1" 200 6527 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;05&#58;53&#58;12 +0200&#93; "GET /referenzen/ HTTP/1.1" 200 6634 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;05&#58;59&#58;49 +0200&#93; "GET /referenzen/z---.html HTTP/1.1" 200 3525 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;06&#58;27 +0200&#93; "GET /referenzen/z---.html HTTP/1.1" 200 3407 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;11&#58;55 +0200&#93; "GET /design/ HTTP/1.1" 200 3202 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;14&#58;40 +0200&#93; "GET /referenzen/z---.html HTTP/1.1" 200 3518 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;17&#58;06 +0200&#93; "GET /edv/ HTTP/1.1" 200 3239 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;21&#58;28 +0200&#93; "GET /referenzen/f---.html HTTP/1.1" 200 3512 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;23&#58;28 +0200&#93; "GET /referenzen/batman.html HTTP/1.1" 200 4267 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;25&#58;13 +0200&#93; "GET /robots.txt HTTP/1.1" 200 177 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;25&#58;13 +0200&#93; "GET /impressum/index.php?id=disclaimer HTTP/1.1" 200 7730 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;28&#58;57 +0200&#93; "GET /referenzen/f---.html HTTP/1.1" 200 3422 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;30&#58;38 +0200&#93; "GET /impressum/impressum.html HTTP/1.1" 200 4102 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"
64.68.83.141 - - &#91;09/Sep/2004&#58;06&#58;32&#58;19 +0200&#93; "GET / HTTP/1.1" 200 6527 "-" "Mozilla/5.0 &#40;compatible; Googlebot/2.1; +http&#58;//www.google.com/bot.html&#41;"

Ich habe im Übrigen noch andere Logs, wo unter der selben IP zur selben Zeit der "Media Partners" Bot auf einer anderen meiner Seiten unterwegs ist.

Beitrag von **Nexus** » 12.09.2004, 16:21

Hi,

hab auch mal Logs gewälzt. Dabei ist mir aufgefallen das der neue GoogleBot jetzt endlich mit accept-encoding: gzip vorbeikommt.
Das sollte einiges an Traffic sparen

Gruß
Nexus