Crawling December: CDNs and crawling

Content delivery networks (CDNs) are particularly well suited for decreasing latency of your website and in general keeping web traffic-related headaches away. This is their primary purpose after all: speedy delivery of your content even if your site is getting loads of traffic. The “D” in CDN is for delivering or distributing the content across the world, so transfer times to your users is also lower than just hosting in one data center somewhere. In this post we’re going to explore how to make use of CDNs in a way that improves crawling and users’ experience on your site, and we also look at some nuances of crawling CDN-backed sites.