The FreeBSD Diary

Google appears to be redirecting all search queries submitted from non-recognized broswers to a 403-Forbidden page. I noticed the problem tonight when using the links browser. Similar queries from the same IP with Opera, IE, or lynx succeed. Search attempts by other links users from other IPs resulted in the same message. Additional tests showed that fetch and wget also failed.

The 403-Forbidden message refers you to their terms and conditions. I'm sure Google are trying to block automated queries, which do not comply with their T+C. I've written to them regarding this. I am also confident this problem will be fixed soon and that they were not targeting any particular tool directly.

The original page (with my IP address removed) is available at google-links-forbidden-original.php.

You can get around the problem by changing the referrer field via Setup->Network Options->HTTP options->Fake User-Agent. I used Opera/7.10 (UNIX; U) [en] and was able to get past their blockage.

For wget, you can do something like:

wget --user-agent="Opera/7.10 (UNIX; U) [en]" http://www.google.com/search?hl=en\&ie=ISO-8859-1\&q=things\&btnG= Google+Search

And for fetch, you can set the HTTP_USER_AGENT environment variable:

export HTTP_USER_AGENT="Opera/7.10 (UNIX; U) [en]"