Ad

Friday, August 01, 2008

Web Scraping: Gathering "Related Searches" Keyword Data From Google Search Results

Web Scraping: Gathering "Related Searches" Keyword Data From Google Search Results

You might first want to read Basic Web Scraping: Pulling Out Data From Google Search Results.

In the other Basic Web Scraping post we created a Simple Web Scraper that retrieved Google’s Search Result Pages for some keyword and scraped the “Number of Pages” indexed for the keyword. That was a good starting point in learning web scraping, that’s why in this post we’re going to extend that scraper to return one more information, “Related Keywords”.

Actually, Related Searches is a bunch of keywords that Google displays at the bottom of search result pages. These are not displayed all the keywords that you search for, but only for keywords that are somewhat broad. These “related Searches” are the keywords that Google has evaluated to be related with the one searched for.

To scrape this, we’d first need to analyze the HTML code of Google Search Result. We need to find out where the Related Searches block is in the code, typically it’d like:

<h2 class=r>Searches related to:<b> computer programming</b></h2><table border=0 cellpadding=0 cellspacing=0 style="margin-top:6px"><tr style="font-size:84%"><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+schools&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=1">computer programming <b>schools</b></a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+careers&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=2">computer programming <b>careers</b></a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+languages&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=3">computer programming <b>languages</b></a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+c%2B%2B&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=4">computer programming <b>c++</b></a></td><td>&nbsp;</td></tr><tr style="font-size:84%"><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+information&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=5">computer programming <b>information</b></a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=basic+computer+programming&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=6"><b>basic</b> computer programming</a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=computer+programming+for+dummies&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=7">computer programming <b>for dummies</b></a></td><td>&nbsp;</td><td style="padding:0 30px 6px 0" valign=top><a href="/search?hl=en&client=firefox-a&rls=org.mozilla:en-US:official&hs=lS4&q=game+programming&revid=92395746&sa=X&oi=revisions_inline&resnum=0&ct=broad-revision&cd=8"><b>game</b> programming</a></td><td>&nbsp;</td></tr></table>

Remember our goal here is to find out certain footprint or consistent block in the code (which always will be on every page we request), out of which the needed data (Related Searches) can be pulled out.

Here, we can safely assume it to be:

<h2 class=r>Searches related to:

and

</td></tr></table>

So if we do a search for the above block in the code, we can easily pull the required information.

Now, we can follow the following steps to scrape the required information:

  1. Retrieve the required page.
  2. Look for the consistent part of code that uniquely encloses or identifies the required information. Scrape that block.
  3. Pull the required data out.

Following is the code listing:

<html>
<head>
<title>Google Result Scraper</title>
</head>

<body>
<p align="center" style="font-size:500%"><font color="#0000FF">G</font><font color="#FF0000">o</font><font color="#FFFF00">o</font><font color="#0000FF">g</font><font color="#00FF00">l</font><font color="#FF0000">e</font> 
  <font size="2"><br />Result Scraper</font></p>

<?php
$s
=$_GET['s'];

if(isset(
$s))
{
    
//*******F I R S T   P A R T*********
    //Find the number of pages indexed for the searched term
    //*From previous part
    
echo "<p><i>Search for $s</i></p>";

    
$s=urlencode($s);

    
$main_data=file_get_contents("http://www.google.com/search?hl=en&q=".$s."&btnG=Google+Search");
    
    
//strip off HTML
    
$data=strip_tags($main_data);
    
//now $data only has text NO HTML

    //these have to ound out in the fetched data
    
$find='Results 1 - 10 of about ';
    
$find2=' for';

    
//have text beginning from $find
    
$data=strstr($data,$find);

    
//find position of $find2
    //there might be many occurence
    //but it'd give position of the first one, 
    //which is what we want, anyway
    
$pos=strpos($data,$find2);

    
//take substring out, which'd be the number we want
    
$search_number=substr($data,strlen($find), $pos-strlen($find));

    echo 
"Pages Indexed: $search_number";
    
    
    
//********S E C O N D   P A R T*******
    //Find related searches
    
    
echo "<h3>Related Keywords</h3>";

    
//these have to found out in the fetched data
    
$find='<h2 class=r>Searches related to:';
    
$find2='</table>';
    
$find3='<table border=0';

    
//have text beginning from $find
    
$data=strstr($main_data,$find);

    
//find position of $find2
    //there might be many occurence
    //but it'd give position of the first one, 
    //which is what we want, anyway
    
$pos=strpos($data,$find2);

    
//take substring out, this is the block of data
    //that we'd fetch required information off of.
    
$related_search_block=substr($data,strlen($find), $pos-strlen($find));
    
    
//pull out the required data, stripping off "Searches related to.."
    
$related_searches=strstr($related_search_block,$find3);
    
//now we only have the keywords, of course formatted as is
    //with tables
    
    //strip off HTML, therefore taking off table and various
    //other formattings
    
$related_searches=strip_tags($related_searches);
    
//now we have the keywords (plain text) sepearted with 
    //each other using "&nbsp;'
    
    //explode it
    
$keywords=explode('&nbsp;',$related_searches);
    
//now we a nice array having all the keywords
    
    //print the keywords
    //nicely formatted using <ol> (ordered list)
    
echo "<ol>";
    foreach(
$keywords as $keyword)
    {
        if(
$keyword!='' && $keyword!=' ')
        echo 
"<li>
                $keyword
             </li>"
;
    }
    echo 
"</ol>";    
}
else
{
?>

<form name="form1" id="form1" method="get" action="">
  <div align="center">
    <p> 
      <input name="s" type="text" id="s" size="50" />
      <input type="submit" name="Submit" value="Find Related Keywords" />
    </p>
  </div>
</form>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>
  <?php
}
?>
</p>
<p align="right"><font size="2">by <a href="http://learning-computer-programming.blogspot.com/">Learning 
  Computer Programming</a></font></p>
</body>
</html>

Although this script is not very useful as it is (after all it’s meant for learning purpose) you can make it more useful by extending it. one thing that’d be nice is to hav it search for related keywords and again take each keyword as a base for further searches. This way it’d be able to supply you with many keywords related to the one entered. Best thing is, the keywords will be Googly Related!

Previous Posts:

4 comments:

  1. Hello,

    I'm Susan, of the TechnoSnack's team and I wish to inform you that we are opening a new blog aggregator about Computers & Internet news.
    We put it on-line some hours ago and the link is: http://www.technosnack.com.

    The main objective of this project is creation of a "virtual dashboard" of posts coming from many specialized blog and information about Computers & Internet world, with news about Linux, Windows, Mac, Open sources, Security, Graphics, Symbian and more on...

    The key feature is that news come directly from blogosphere. We wish to show a preview of posts, with a link "Read more..." to signed blogs. If users are interested in news, they are redirected to your blog and can read entire post directly from your blog!


    So, the different signed blogs can increase their visibility and reach more visitors, all over the world!

    We think that in a little of time it can send more visitors to re gistered blogs, contributing to diffusion of know-how about Computer and Technology world.

    I visited your blog and I think it has very interesting and useful posts!


    So, are you interested in this idea, with your blog?
    If yes, then you can register your blog, using the specific "Registration Form"!

    REGISTRATION IS ABSOLUTELY FREE!

    The only thing we ask to you is to insert TechnoSNACK banner in your blog to promote this project. Or, if you prefer, you can insert a link in your blogroll.

    If you like (we whould be happy, but it is not mandatory :-), you can write a post regarding TechnoSNACK project in your blog, to promote this idea.


    Bye!
    Susan - TechnoSnack's Team

    ReplyDelete
  2. Hi me again new to php apps I've tried script here
    http://link-directory.org/testing/script2.php
    I get Google's font 'Related Keywords' is this what this particular script is meant to do? If so this is great. I'm wanting to attempt an extended version of this script to retrieve full results page. Any tips on this? cheers

    ReplyDelete
  3. Hi Jay,

    I have responded to your comment there, no that is what it does.

    When you search for some keyword sometimes Google tells you about the possibly related keywords towards the bottom of the page. Actually, this script fetched those information from there and displays it.

    Apparently, this script is not working due to what I said in the post below:

    http://learning-computer-programming.blogspot.com/2008/06/basic-web-scraping-scraping-google.html

    ReplyDelete
  4. Your article saved us Many hours in while making keyword Research Thank you.

    Ahmed Moheildeen

    ReplyDelete

You are free to comment anything, although you can comment as 'Anonymous' it is strongly recommended that you supply your name. Thank You.

Please don't use abusive language.