There are several motives you could want to find every one of the URLs on an internet site, but your specific objective will establish Anything you’re looking for. As an example, you might want to:
Detect each individual indexed URL to investigate difficulties like cannibalization or index bloat
Accumulate recent and historic URLs Google has observed, specifically for web site migrations
Obtain all 404 URLs to recover from submit-migration problems
In Each and every state of affairs, only one tool received’t Provide you with every little thing you will need. Unfortunately, Google Lookup Console isn’t exhaustive, along with a “web page:instance.com” research is limited and challenging to extract data from.
On this put up, I’ll stroll you thru some applications to make your URL record and in advance of deduplicating the information employing a spreadsheet or Jupyter Notebook, depending on your web site’s sizing.
Previous sitemaps and crawl exports
If you’re on the lookout for URLs that disappeared from your Stay website a short while ago, there’s an opportunity somebody in your crew could possibly have saved a sitemap file or simply a crawl export ahead of the variations have been manufactured. Should you haven’t currently, look for these data files; they could frequently present what you'll need. But, when you’re reading this, you probably did not get so Blessed.
Archive.org
Archive.org
Archive.org is a useful Software for Website positioning tasks, funded by donations. In the event you search for a site and select the “URLs” solution, you may access as many as ten,000 shown URLs.
Nevertheless, Here are a few limitations:
URL limit: You may only retrieve as much as web designer kuala lumpur 10,000 URLs, which happens to be inadequate for bigger web sites.
Excellent: Quite a few URLs may very well be malformed or reference source files (e.g., images or scripts).
No export option: There isn’t a built-in method to export the list.
To bypass The dearth of an export button, use a browser scraping plugin like Dataminer.io. On the other hand, these limits imply Archive.org might not present an entire Alternative for larger sites. Also, Archive.org doesn’t indicate regardless of whether Google indexed a URL—however, if Archive.org identified it, there’s a good probability Google did, also.
Moz Professional
Whilst you might commonly use a link index to discover external websites linking to you, these tools also find URLs on your internet site in the process.
How to use it:
Export your inbound hyperlinks in Moz Pro to secure a rapid and straightforward listing of target URLs out of your web-site. In the event you’re addressing a massive Web site, think about using the Moz API to export information past what’s workable in Excel or Google Sheets.
It’s important to Take note that Moz Pro doesn’t confirm if URLs are indexed or discovered by Google. Having said that, due to the fact most web sites implement exactly the same robots.txt regulations to Moz’s bots since they do to Google’s, this technique generally functions effectively being a proxy for Googlebot’s discoverability.
Google Lookup Console
Google Search Console offers quite a few important sources for building your list of URLs.
Links stories:
Similar to Moz Pro, the One-way links area provides exportable lists of goal URLs. Regretably, these exports are capped at one,000 URLs Every. You may implement filters for specific webpages, but due to the fact filters don’t utilize to your export, you would possibly have to depend upon browser scraping instruments—restricted to 500 filtered URLs at a time. Not best.
Functionality → Search Results:
This export offers you a list of internet pages acquiring look for impressions. When the export is restricted, You can utilize Google Lookup Console API for bigger datasets. Additionally, there are no cost Google Sheets plugins that simplify pulling additional substantial information.
Indexing → Web pages report:
This section supplies exports filtered by problem form, even though they are also constrained in scope.
Google Analytics
Google Analytics
The Engagement → Web pages and Screens default report in GA4 is a wonderful resource for collecting URLs, having a generous limit of 100,000 URLs.
Better still, it is possible to implement filters to build distinctive URL lists, correctly surpassing the 100k Restrict. One example is, if you would like export only blog site URLs, comply with these methods:
Stage one: Incorporate a segment into the report
Action 2: Simply click “Produce a new phase.”
Phase 3: Outline the segment that has a narrower URL pattern, such as URLs that contains /blog/
Observe: URLs located in Google Analytics might not be discoverable by Googlebot or indexed by Google, but they provide valuable insights.
Server log information
Server or CDN log documents are Potentially the ultimate Instrument at your disposal. These logs capture an exhaustive listing of each URL path queried by people, Googlebot, or other bots in the recorded interval.
Criteria:
Facts measurement: Log information can be large, numerous sites only retain the last two weeks of information.
Complexity: Analyzing log documents is usually difficult, but several tools can be found to simplify the procedure.
Incorporate, and great luck
After you’ve gathered URLs from each one of these resources, it’s time to combine them. If your web site is sufficiently small, use Excel or, for much larger datasets, applications like Google Sheets or Jupyter Notebook. Guarantee all URLs are constantly formatted, then deduplicate the listing.
And voilà—you now have an extensive listing of current, old, and archived URLs. Excellent luck!