
So there can be certain parts of sites that are available via a search engine, and other (none interesting stuff) that isn’t. It’s not so much being evil, it’s more about being a responsible net-citizen. When a webmaster excludes a directory it’s usually full of techie stuff (ie programming) that would actually damage a sites ranking on a search engine.


Telling a search engine not to look at something doesn’t indicate it’s exciting, it usually indicates the exact opposite. These files tell the search engines what they can and can’t look at on the site. There is a file on web servers called robots.txt, and one called sitemap.xml. Gyp the Cat dot Com hides information from search engines. Most sites hide some information from search engines. 2 – Hiding Information from Search Engines Is Rare & Indicates Some Deep Web The only way you could get on is if the company in question gave you a log on.ĭeep web yes, interesting probably no. However, since it was relating to the companies operational factors (and not full of alien autopsies) it would make pretty boring reading to even the most ardent of conspiracy theory buff. Now this system mentioned above was accessible via the internet, and had some pretty heavy security features (for pretty obvious reason). This intranet also had a 9.6Gb database full of other information. I’ve worked with some pretty large organisations in my time, one smaller one had 7.1Tb of office documents, images, and the odd movie on their intranet.

But not for the reasons you probably think. Yes, it’s true, the “Deep Web” exists, and people store on it information they don’t want you (as Joe Public) to have.

“A website that is not available through searching for it on Google and the like”Īfter having some quite heated conversations over the last couple of weeks (with some quasi-technically savvy people I should add), I’ve put this list together to hopefully clarify some points, I appreciate it’ll take away quite a lot of the mystery of the whole thing, but here we go…ġ – The deep web is a portal to information that others don’t want you to have The Deep Web (otherwise known as DarkNet, Undernet or the Invisible Web), has a simple definition:
