January 29th, 2007

How to set the cURL user agent string with PHP

I just found out that my free link checking tool is being blocked by some websites. My guess is that it's because it's sending a black User-Agent string. I'm going to have to spoof it, say it's FireFox or something. Here's how to do that with cURL and PHP:

// spoofing FireFox 2.0
$useragent="Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1";

$ch = curl_init();

// set user agent
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
// set the rest of your cURL options here
 

3 Responses to ' How to set the cURL user agent string with PHP '

Subscribe to comments with RSS or TrackBack to ' How to set the cURL user agent string with PHP '.

  1. James said,

    on November 1st, 2007 at 5:02 pm

    I can confirm this is the case when trying to scrape Google. With a user agent, the organic results are between and . Without one, they are not included.

  2. James said,

    on November 1st, 2007 at 5:02 pm

    Between and
  3. James said,

    on November 1st, 2007 at 5:04 pm

    Between HTML comments!

Leave a reply

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

*
To prove you're a person (not a spam script), type the security word shown in the picture.
Anti-Spam Image