How big the web is? Yes, we all know there are some really big fat numbers here, but how big? Google unveils some stats from time to time, and that gives us the picture.
You see the pattern: in the last 14 years the web (or what Google extracts from the web) became really, really big thing. Note the order of magnitude: for 10 years (from 1998, when Google started with 25 million pages to 2008, when they celebrated their 1st trillion) the web grown 40,000 times. And even if that sounds big, it’s nothing compared to the next part of the chart. For the next 4 years, we got something I would call exponential growth: from 1 to 30 trillion webpages. This is over 7 trillion pages growth per year. The web of today is 1.2 million times bigger than 1998.
30 billion web pages indexed by Google in 2012
Let’s translate the 30 trillion number into something meaningful: you will need over 500 terabytes of storage to save the URLs only. Give it one backup copy, and it’s a petabyte. 500 2TB hard drives. Just for the list of the pages. Don’t ask how much more you need to keep the actual content
There are, currently, about 4300 pages for each human being on this planet. Yes — if every single human alive reads 4300 pages right now, we’ll read the whole web. Go figure…
Another statistic from the big G: they served something like 1,2 trillion searches in 2012. About 100 billion searches per month. Multiply by 10 (the default number of results). Imagine that there are no repetitions and Google returns different URL each and every time. Even then – 2/3 of the URLs indexed will never be shown to any searcher. Now you see how powerful the holy algorithm is. Even if we restrict it to show each page only once, it will trash 2/3 of all webpages it knows. Does SEO matter?
1,2 trillion searches in 2012, 100 billion per month
But there is another number: in 2012 there are 246 million domains registered (according to Verisign). almost half of them (about 120 million) are .com and .net domains. About 13% of all .com and .net the domains doesn’t contain any web page. 21% contain only one and 66% are multi-page domains. To make things simpler — let’s assume it’s the same for all TLDs. This will give us some funny numbers. There are 25 million domain names without any web page published. There are about 50 million single-page domains. So, almost all of these 30 trillion pages are published on just over 160 million domains. Again: 160 million domains are serving close to 30 trillion web pages. 66% of the domain names in use are responsible for 99,99% of the web. That gives us something like 180,000 pages average for each of these multi-page domains. So, is your site close to the average?
29,9 trillion webpages published on 160 million domains
But don’t panic! Think about wikipedia, or blogger, or wordpress, or youtube. Wikipedia alone is responsible for almost 30 million of these pages (to put it in context: wikipedia alone is bigger than the whole web back in 1998). And these multi-million page sites are human built. Imagine what the bots can do. So, don’t feel bad if your site doesn’t have hundreds of thousands of pages But you may start to think about automation…
the whole web in 1998 is smaller than wikipedia today
I get so many questions in my head reading these numbers, but I will stop here. This post was supposed to be here only for fun. Still — I can’t stop myself from asking at least two of these questions. How will Google handle the growth in the coming years? I expect something between linear and exponential growth, thus meaning we will see the number doubled in year or two… And the more important one: is there a limit of some kind? I have no idea how big the web could possibly be in a decade from now, but comparing to the past, it sounds really interesting… Can the web grow forever? Is even the sky a limit?
Well, we’ll see…
But I will stop here and go back to work now, because with just few hundreds pages this site is far, far away from the 180,000 average