Web Scraping a Search Results Page with OutWit Hub Data Extractor

A tutorial on how to (1) scrape a search engine results page using (2) a custom scraper created with OutWit Hub data extractor, and (3) export to an Excel spreadsheet. Companion blog post at www.marketing2oh.com shows how to create basic scrapers for Google, Yahoo and Bing.

---

Closed Caption:

hi I'm tailstock content marketing
consultant and founder of marketing 20
our video today demonstrates the tools
and techniques covered in the marketing
to a blog post had to scrape a search
engine results page for your SEO project
and will introduce you to a helpful
entry firefox browser extension called
outwit hub we can use that with hub for
web data extraction and to import that
data into an Excel spreadsheet or
database
this is really helpful when we want to
analyze search engine rankings for here
is keywords or when we want to review
which our website pages have been
indexed by search engine today I'm going
to show you how to create a web scraper
for google without with hub and use
outwit hub to scrape a google search
results page and last how to export that
data into an Excel spreadsheet so let's
get started to follow along i recommend
that you visit out with technologies
homepage at out wet
oh you twi t dot com to download and
install a copy of that with hub after
installing and restarting your firefox
browser you'll see a new button on your
navigation to a bar next to the web
address field our project today we're
going to navigate to google to google
some page where we're going to do a site
search for a specific domain do this we
type in this site command SI te followed
by a colon and then the domain name in
which case in this case is going to be
Jim stone designs for you . com now
since i already know that there are over
a hundred pages for gemstone designs
website
we're going to use the advanced search
so we can see as many pages as possible
and select the maximum number of results
per page 102 can see now
and as we scroll down we've got a full
page of results
100 results and an additional 15 on the
second page
okay we've gotta search results now the
hundred and fifteen pages that have been
indexed by google for gemstones designs
and let's go to that out with hub button
again and our browser menu and click on
it to get the outwit hub browser to
appear here showing us of course in the
central browser window the page that
were examining and i'm going to mention
this point that I would help comes with
built-in functionality that does a
really nice job on similar pages of
capturing that information that's
presented in table format or list on
that will also attempting guess at the
information that you're looking for but
to say time what we're interested in is
creating our own custom scraper and so i
want you to click on the source button
and it's going to show us to windows
here now in the center one showing us
the source code and then on the ride and
editor window with that we're going to
use to create your custom scraper so
let's click on new and to create our
scraper and it says to us we want to
create a scraper that is going to be
applied to URL starting with google.com
that's absolutely correct and we're
going to give a name to this we're going
to call it obviously Google scraper
excuse me and I'm interested in
capturing just three fields here i'm
interested in capturing the page title
page description and lastly the URL to a
particular page and now we need to
examine the source code and we're going
to be looking for unique markers in
front of and behind the information
typically HTML
in front of and behind the particular
information we're looking to collect so
scrolling down to the first search
result and examining the code HTML in
front of that first item on the title of
the page gemstone designs we can see
that the code includes an HTML for a
great starting list h3 type of heading
and endlessly lastly a link which is
specific to this particular page so I'm
going to collect that middle line of the
HTML that says this is going to be an h3
class equals are tagged and then we're
going to look behind the title and
capture the eighth copy and paste the
clothes h3 and proceed to the patron
scription you can see just one tag here
after the h3 for the div classical ass
and now when we look at the end of the
page description
we've got a code here none of it is a
close div i'm going to try a couple of
items here that includes the site tag
that begins the URL link to the URL from
to be up to the particular page so I'm
going to capture and paste the brake and
the side tag and then of course the
citag open citag and close surround the
particular URL that were interested in
and so we capture both of those and
there you have it let's try it by saving
it and click on scraper 20 results we
can see the page title description URL
in the three
columns to the right and if we scroll
down we can see that we've captured all
100 items now at this point what i
recommend doing is selecting all and
exporting by going to file export
selection as Excel and we're going to
save that to our desktop now
alternatively we could use the catch
button immediately below but it's not
working as well as actually exporting
what we have in this window at least 44
now so i recommend going ahead and
exporting the contents here and we'll
take a look at the excel spreadsheet
that we've created see the title
description URL hide the source you were
here you'd better look and scroll down
and there's a hundred items so we've
captured on looks like the results that
we were after and with me to do the same
again for the second page of search
results in the additional 15 items but
again for now i recommend this is the
best way to collect all of the results
in that search window i hope you enjoyed
the video here please visit our website
marketing 2.com to read the entire blog
post which will give you some additional
examples for custom scrapers and also to
you can learn more about our services
www mr clean 2.com thanks so much