Real-Time SERPs Scraping with Zenserp – Review

0
144

Google has a long history of giving a really hard time for SEO developers attempting to scrape publicly available data off their search results pages. Indeed, SERP scraping is officially against Big G’s terms of service. However, the practice does not violate the Computer Fraud and Abuse Act (CFAA), and thus direct legal actions by Google to curtail the practice have fallen flat. In this article, we are going to talk about Highly Scalable Real-Time SERPs Scraping with Zenserp. Let’s begin!

So, while scraping is not actually illegal or hurting anyone’s feelings in practice, developers must still contend with the roadblocks. That is designed by Google to impede their data collection. Common headaches include Captchas and IP blocking. However, cease-and-desist letters have been sent in the past, this tactic is increasingly rare.

But what if you did not have to deal with these nuisances at all to get your precious data? Enter Zenserp, a core offering from Austria-based SaaS Industries.

Zenserp in a nutshell | SERPs Scraping with Zenserp

The Zenserp API is a dream tool for SEO devs looking for a turnkey solution to clear common scraping hurdles. It is dead-simple to use and integrate, thanks to SaaS Industries’ user-centric development cycle and broad language compatibility (Browser, Curl, Python, Node.js, PHP). What’s more, their voracious appetite for consumer feedback and feature requests has led to April 2019’s huge API update to include a comprehensive array of data types, including Google images, video, maps, news, shopping, and also trends.

Combined with competitive pricing and world-class customer service. You truly have an all-in-one solution to your SERP-scraping woes in Zenserp.

Getting started with Zenserp

So, what’s it like to actually use Zenserp? Below, we cover the process of getting started with Zenserp. Then, we take a dive into their API documentation to get a closer look at Zenserp’s robust functionality.

Dashboard overview | SERPs Scraping with Zenserp

Assume you’re just taking a peek at what Zenserp’s all about, you’ll click the bright green “Sign Up for Free Plan” button on their home page. When you’ve created and verified your account, click “Login” to access your Dashboard.

The page you land on displays an overview of your most important usage information, including your API key, monthly request quota, API usage over time. Additionally, documentation of any failed requests that may have occurred. There’s also a handy Request Builder section that directs new Zenserp users towards a wealth of useful resources. Most of this is stuff we’ll cover in the Documentation section of this review, so hang tight on that.

Let’s take a quick look at all the sidebar options:

  • Dashboard – You are here already! This is simply a quick link to return to your overview page at any time.
  • Playground – Appropriately named, this page offers a fully visual (i.e., no coding required) and interactive interface for you to mess around with Zenserp’s various endpoints. Here, you can generate some sample responses, and use them to configure your API request. Don’t rely on this 100%, as there are corner cases beyond the capability of this tool. Nevertheless, we love how you have the option to export your experimentation to code (Browser, Curl, Python, Node.js, PHP) when you find something that really works.
  • Documentation – Details the guts of the Zenserp API. We cover this in-depth below.
  • Profile/Security – Basic account configuration, including your avatar, contact information, and password management. Nothing fancy.
  • Billing – Consists of three options, where you can upgrade/downgrade/cancel your subscription level, manage your payment method and VAT ID, plus a free field where you can request text be included on your Zenserp invoices for making accounting easier.

All of this is pretty standard fare for any API, but we appreciate how simply laid-out everything is. What you see is what you get, and Zenserp doesn’t attempt to bury subscription settings in obscure corners of your profile.

The Zenserp API documentation | SERPs Scraping with Zenserp

With the basics out of the way, now we can get into the good stuff: Zenserp’s documentation. Here, you’ll see exactly what the API is capable of, and how to use it. While full API implementation into your application is beyond the scope of this review, we’ll briefly touch on each endpoint, some key parameters, and provide some example responses.

Basic functionality

Starting off the documentation page is an aptly named “Getting Started” section, which details your most basic API functions capable of interacting with all other endpoints. Let’s take a look:

  • Authentication – This shows you how to use your API key for authentication. There is an example of use in the header, followed by GET and POST request examples. Easy peasy.
  • status – Allows you to request the number of remaining requests per your subscription limits. After all, why bother logging into your Zenserp Dashboard when you can just have your app tell you directly?
  • batch – Appending this endpoint allows you to make up to 1,000 searches in a single batch. You’ve got the option to request all batches at once, specific by batch ID, or to submit a new batch or merge existing ones together as needed.
  • Error codes – Zenserp also has the ability to return errors as they occur. Including 403 Forbidden when you’ve hit your monthly limit or used an incorrect API key, 404 when no results have been found, or 500. Whenever there’s a server error. Don’t forget that failed requests also appear on your Dashboard.

The /search endpoint | SERPs Scraping with Zenserp

The core functionality of Zenserp is here. By appending /search to your API request. You’ll receive a comprehensive array of data on basically anything you can think to collect SERPS for. Your query will return a wide array of results, including:

  • Organic search results – Querying organic SERPs returns all kinds of data, including item ranking, its title, both displayed and destination URL, several types of reviews, whether the link is a Google AMP link, strings representing the page description, and plus the Local 3-Pack which contains business info.
  • Paid results – You’ll also get a full overview of paid results. The info returned includes paid position (top, right, or bottom), title, the AdWords tracking URL, breadcrumbs, the visible URL as it appears on Google, page description, plus any paid product placements (including position, title, associated images, URL, and of course, price).
  • Answer box/featured snippet/top story – Pure gold to any SEO. Easily see what performs best to gain these coveted spots on SERPs, then build a strategy to replicate those results.
  • Related searches/did you mean – Zenserp enhances your keyword research by returning valuable data on all related searches to your query. You can also anticipate how Google responds to common errors users make when searching out your query.

Here’s a cut-down version of an example that appears in the Zenserp documentation, with “Pied Piper” as the query. You can see the different types of results that appear in order: the Wikipedia entry (1), questions related to the query (2), a website with an exact-match domain name (3), a popular article on the Pied Piper’s origins (4), plus some related videos on the topic (5). You’ll also see some related search phrases, plus the total number of results returned.

Example

{
    "query":{
        "apikey":"YOUR-API-KEY",
        "q":"Pied Piper",
        "url":"https://google.com/search?q=Pied+Piper&hl=en&gl=us&sourceid=chrome&ie=UTF-8"
    },
    "organic":[
        {
            "position":1,
            "title":"Pied Piper of Hamelin - Wikipedia",
            "url":"https://en.wikipedia.org/wiki/Pied_Piper_of_Hamelin",
            "destination":"https://en.wikipedia.org › wiki › Pied_Piper_of_Hamelin",
            "description":"The Pied Piper of Hamelin is the titular character of a legend from the town of Hamelin (Hameln), Lower Saxony, Germany. The legend dates back to the Middle ...",
            "isAmp":false
        },
        {
            "position":2,
            "questions":[
                {
                "question":"What is the meaning of the Pied Piper?"
                },
                {
                "question":"What is the story of the Pied Piper?"
                },
                {
                "question":"Is the pied piper a fairy tale?"
                },
                {
                "question":"Is Pied Piper a real company?"
                }
            ]
        },
        {
            "position":3,
            "title":"Pied Piper",
            "url":"http://www.piedpiper.com/",
            "destination":"www.piedpiper.com",
            "description":"The old Pied Piper “flute player” logo was referencing a fairytale character and in no way was meant to suggest a sexual act involving either the mouth or ear of ...",
            "isAmp":false
        },
        {
            "position":4,
            "title":"The Disturbing True Story of the Pied Piper of Hamelin ...",
            "url":"https://www.ancient-origins.net/myths-legends/disturbing-true-story-pied-piper-hamelin-001969",
            "destination":"https://www.ancient-origins.net › myths-legends › disturbing-true-story-pie...",
            "description":"Aug 14, 2014 - Many are familiar with the story of the Pied Piper of Hamelin. Few realise however, that the story is based on real events, which evolved over ...",
            "isAmp":false
        },
        {
            "position":5,
            "videos":[
                {
                "title":"Pied Piper",
                "url":"https://www.youtube.com/watch?v=VEZ_Ui6d9AM",
                "parsed":{
                    "source":"YouTube",
                    "date":"May 2, 2018",
                    "channel":"BTS - Topic"
                }
                },
                {
                "title":"The True History of The Pied Piper of Hamelin | Fairy Tales ...",
                "url":"https://www.youtube.com/watch?v=obIAwAAcAAg",
                "parsed":{
                    "source":"YouTube",
                    "date":"Aug 4, 2017",
                    "channel":"Jen Campbell"
                }
                },
                {
                "title":"The Pied Piper: a modern day fairy tale",
                "url":"https://www.wvi.org/child-health-now/video/pied-piper-modern-day-fairy-tale",
                "parsed":{
                    "source":"World Vision International",
                    "date":"Aug 22, 2013",
                    "channel":"WVIAdvocacy"
                }
                }
            ]
        }
    ],
    "related_searches":[
        {
            "title":"pied piper instrument",
            "url":"https://google.com/search?hl=en&gl=us&q=pied+piper+instrument&sa=X&ved=2ahUKEwj-vPeI3oXnAhVJJBoKHQ6iDYkQ1QIoAHoECBMQAQ"
        },
        {
            "title":"pied piper poem",
            "url":"https://google.com/search?hl=en&gl=us&q=pied+piper+poem&sa=X&ved=2ahUKEwj-vPeI3oXnAhVJJBoKHQ6iDYkQ1QIoAXoECBMQAg"
        },
        {
            "title":"moral lesson of pied piper story",
            "url":"https://google.com/search?hl=en&gl=us&q=moral+lesson+of+pied+piper+story&sa=X&ved=2ahUKEwj-vPeI3oXnAhVJJBoKHQ6iDYkQ1QIoAnoECBMQAw"
        },
        {
            "title":"the pied piper of hamelin summary",
            "url":"https://google.com/search?hl=en&gl=us&q=the+pied+piper+of+hamelin+summary&sa=X&ved=2ahUKEwj-vPeI3oXnAhVJJBoKHQ6iDYkQ1QIoA3oECBMQBA"
        },
        {
            "title":"pied piper of hamelin story pdf",
            "url":"https://google.com/search?hl=en&gl=us&q=pied+piper+of+hamelin+story+pdf&sa=X&ved=2ahUKEwj-vPeI3oXnAhVJJBoKHQ6iDYkQ1QIoBHoECBMQBQ"
        },
    ],
    "number_of_results":34800000
}

Refining your request by search type | SERPs Scraping with Zenserp

Beyond the default SERPs request, you can also refine your Google Search query by search type (Image, Video, Maps, Shopping, News, Shopping, etc.). Simply set the tbm (“to be matched”) parameter to any of the following options:

  • isch – Returns Google Image Search results. This request returns a litany of data including position, thumbnails, source URLs, titles, and more. (For Reverse Image Search, you’ll run the image_url parameter instead of tbm=isch)
  • vid – Returns Google Video Search results. You’ll get info like source domain, video URL, title, description, plus any extensions or thumbnails.
  • lcl ­– Another goldmine of SEO data, the lcl parameter returns Google Maps Search results. You’ll see business/location name, website URLs, whether or not the result is paid-for, any stars or reviews, plus vital information like address, phone numbers, hours of operation, notable prices, and even the type of business. You also have the option to request directions to these businesses.
  • nws – Need to keep up to date with current events and developments in your industry? Add the nws parameter to your query to return real-time Google News Search results. See names of articles, their links, dates published (and parsed by your API), the publisher, a brief article description, plus any available thumbnails.
  • shop – Access Google Shopping Search results with the shop parameter. Let’s take a look at some sample results (heavily truncated for the sake of readability) for water heaters in Atlanta, GA:

Shopping EndPoint

Besides the core /search endpoint, the Zenserp API also features /shopping. That enables you to parse Google Shopping Product Pages. You’ll append the product_id parameter to your /shopping query, with the option to specify your desired search geo-location (location), country code (gl), and/or Web Interface Language (hl). You can also change which region variant of Google Search you want to use with search­_engine.

Trend Endpoint  | SERPs Scraping with Zenserp

Google Trends requests are made by appending /trends to your request (though Zenserp’s documentation incorrectly states that you should use /shopping). Your Trends results are powered by the keyword[] parameter, where you pass your keyword phrase as the argument within the []. Zenserp’s documentation has a handy reminder about which operators Trends uses (+ for “or”, – for “exclude”, space for “and”, and quotes “ “ for an exact match), which will be instrumental in your query.

You can further specify timezone (tz), category ID (cat), the timeframe for your results (hourly, daily, monthly, yearly in many subdivisions). The type of Google property to search (like, images, Flights, YouTube, Finance, etc.). The geolocation or Google location code (gl), and even the resolution within that region (essentially, setting whether you search country-wide, city-wide, regionally, etc.)

So, let’s say you wanted to see how Kanye West has been trending over the past month. You’d make the request like so:

https://app.zenserp.com/api/v1/trends?apikey=YOUR-API-KEY&keyword%5B%5D=kanye%20west&timeframe=today%201-m

Performance | SERPs Scraping with Zenserp

Now that we’re done with the features overview, how does Zenserp actually perform? After all, all this nice data is useless if you can’t actually access it because of Google’s stonewalling programmatic data collection efforts.

In our experience, Zenserp performed its tasks without a hiccup, no matter what we searched, and where in the world we targeted our queries. The secret weapon here is Zenserp’s incredible worldwide proxy network, which redirects your requests where they need to go while changing your IP address to fit. While Zenserp doesn’t publicize the actual numbers of their server network, it proved sufficiently robust to never throw up a single error when we made our test requests.

Also included under the hood of the Zenserp API is a sophisticated Captcha solver. Once again, we do not have details on just how they made this work. But work it did­, and consistently too.

Pricing

Zenserp’s infrastructure is enough to satisfy any scale of operation. They offer a wide range of subscription price points so that you only pay for exactly what you need. Whichever tier you choose, you can pay a low monthly fee, or snag a 20% discount on a yearly plan.

In addition to the standard tiers, Zenserp offers High Volume Plans. That is locked into a monthly payment scheme. Each tier includes all premium features of the MEDIUM-VERY LARGE plans. And scales your monthly quota according to how much you pay:

  • PREMIUM – $599.99 and 250,000 requests per month
  • PROFESSIONAL – $1,399.99 and 750,000 requests per month
  • ENTERPRISE – $1,599.99 and 1,000,000 requests per month

SERPs Scraping with Zenserp

SaaS Industries has produced an incredible offering with its Zenserp API. It’s customizable and highly scalable thanks to its robust infrastructure. Yet dead-simple to integrate into your app thanks to easy parametric logic and wide coding language compatibility.

Our testing yielded consistent results without ever throwing up an error. Although we suspect the service is not invulnerable to errors on the scale required by large enterprises. No matter what type of request we made, the results were always timely and meaningful. It is enormously helpful to be able to refine your SERPs request by search type, and as such it’s clear that Zenserp offers an invaluable yet affordable tool to any SEO dev.

If we had any criticism of the service, it’s that the documentation could use a touch more natural language to explain its contents. Of course, an experienced developer should have no problems understanding what’s on offer, so this is likely a non-issue. Even if it is, all but the smallest payment tiers include concierge onboarding and personalized customer support, so you’ll always be able to address your needs perfectly.

Conclusion

Alright, That was all Folks! I hope you guys like this SERPs Scraping with Zenserp article and find it helpful to you.  Give us your feedback on it. Also if you guys have further queries related to this article. Then let us know in the comments section below. We will get back to you shortly.

Have a Great Day!

Also See: User Guide On View Chrome Cache On Windows 10

Leave a Reply