Have you ever been looking at a webpage and wanted to quickly crawl all the links found on that webpage?
Then one by one, you painstakingly right-click and “Copy link address” each link, so you can crawl at another time.
You can spend several moments just trying to “quickly” crawl various web pages to extract valuable webpage information like meta data, redirects, canonicals, etc.
I want to share a pretty nifty way of quickly crawling various web page links more efficiently all the while remaining highly targeted in your research.
We’ll be using a Chrome extension called Linkclump and the SEO crawling tool, Screaming Frog.
In our case, we’ll be using a Search Engine Results Page (SERP) to find the links we want to crawl with Screaming Frog.
Let’s begin
1) Add Linkclump Chrome Extension
The first thing you’ll need to do is add the Linkclump extension to your Chrome browser. The Linkclump extension allows you to better interact with the various links found on a webpage.
It’s important to remember that links play a huge role in SEO. They help guide the user experience across the web and, in SEO, they are able to pass search authority to webpages.
With Linkclump you’re able to copy Anchor Text, URLs, HTML and, in our case, can help you perform quick, accurate research on a webpage’s links.
First, you’ll want to configure the extension to only copy URLs associated with links to your clipboard (the clipboard is a location in your computer’s memory that temporarily stores cut or copied data that can be pasted to a new location).
2) Copy Links from Webpage
Now that we have the Linkclump extension configured, let’s perform a quick search within Google.
In our case, let’s Google “linkclump”.
Once the search results have been loaded, we can then extract the URLs from each result using Linkclump’s right click + drag feature.
What I’m doing in the image above is copying the URLs associated with each search result link. Once you’ve dragged over the links and let go of the right-click, the URLs are copied to your clipboard.
3) Paste into Screaming Frog
If you don’t have Screaming Frog, I highly suggest you download it here. It’s an amazing crawling tool and helps you to better understand the makeup of a URL and website. It allows you to pull in meta data, status codes, and a plethora of over on-page elements.
Once downloaded and opened, you’d have to configured Screaming Frog to “List” mode. Reason being, you’ll have a list of URLs copied to your clipboard that you can paste to Screaming Frog, which will be crawled.
Below is a short video of the process of taking your copied URLs, pasting and crawling them in Screaming Frog.
Summary
And that’s it! You’ve successfully and quickly performed ad hoc crawling research on the various elements for a specific set of URLs. It may not seem like much, but don’t underestimate the potential here. You can,
- Crawl competitors within a SERP and quickly understand how their on-page elements may be driving their performance
- You can apply to almost any search engine, include YouTube and Bing
- You can crawl a website’s internal links with precision, rather than having to rely on an extensive crawl
I hope this makes your linking research much more accurate and productive. Happy crawling!
