As a hobbyist developer on a tight budget, my servers generally run on inexpensive VPSes (Referral Link). They're not configured to autoscale. Floods of unexpected traffic cause them to fall over. This fragility is by design, so that my hosting costs won't cause a budget overflow.
On the modern web, however, this presents a problem.
We Begin Our Story ...
On the World Wide Web of ages long past, otherwise known as the 2000s and 2010s, a bargain was struck with the search engines. They would be allowed to scrape websites, so long as they behaved responsibly and obeyed the rules outlined in each site's robots.txt file.
The robots.txt file declared what was to be considered "private property, no trespassing" – and, in some technical cases, "abandon all hope ye who enter here".
This arrangement was beneficial to the websites because it boosted their visibility in search results, bringing in more views. And it benefitted the search engines by giving them better and more accurate results.
The search engines entrenched themselves as the foundational way to find anything on the internet.
And with so many eyes on them, it was inevitable that they would embrace advertising as their revenue centre. But this blog post isn't about ads. This is about something more recent, and much worse for the Web.