In an earlier post we talked a little bit about vampires and how many vampire “experts” may not have been aware of the connection a doctor named Dolphin had made in 1985 between the modern legends surrounding those creatures of the night, and a rare disease called Porphyria. Well today I want to talk about something quite similar, but in another area in which so many of us are involved with, at so many different times each day, especially any of us that write or read these very Blogs that both you and I are enjoying at this very moment. I’m speaking, of course, of the Internet in general, and Internet Search Engines in particular. Even more specifically, I’m talking about the search engines we’re all familiar with, the ones only very few of us are familiar with, and those that we would all be better off knowing nothing about at all. But let me digress for just a moment.
Two of my greatest joys in life have always been reading, and writing. And in relation to both of those, and the fact that I was born in Canada, I tend to find the English language pretty fascinating also. One of the things I like best about this language so many of us share, is the way new phrases sneak into general usage almost each and every day. What I dislike, (call me crazy) is when I see or hear a word or a phrase that is quite obviously being used incorrectly. Like decimate, that was always one of my pet peeves. You could always get me going simply by telling me about the movie you just watched, or the book you just read, where the villain of the piece got all upset about something, and so he “decimated the village to the last man”. Well good luck with that! You see, the word decimate is derived from the Latin root “deci” which meant 1/10th, and decimation was what the Romans called it when they punished an entire village or town by lining up all the men, and then killing every 10th one of them. If you think about it, no matter how mad the commander might have been, even if he continued the practice until he couldn’t continue it any longer, there would still be nine men left. Its impossible to decimate a town to the last man. And this, long-winded fellow that I am, brings me to the reason for this post; the modern phrase “Google it.”
Now I’m pretty certain that its generally accepted just what this phrase is supposed to mean when used in our modern English vernacular. Person A asks person B what they know about a particular subject, to which person B replies, “I don’t know, why don’t you Google it?” The implication is that if you plug the subject identifier into the Google search engine, you will be able to discover all the information regarding that subject that is currently residing on the Internet. It doesn’t really have to be Google exactly, it may be the Yahoo search engine, the Bing search engine, or any of several other traditional main-stream search engines that are available by which you can investigate the World Wide Web. But therein lies the rub. All of these search engines that I’ve just mentioned, and even those that I haven’t mentioned by name, search only the World Wide Web, and nothing else. Now most of you probably know where I’m going with this, but there are probably a few out there that are asking, “So what else is there, and even if there is something else, isn’t the WWW enough?” Let me try and answer that for those who are curious.
First of all, the World Wide Web is the part of the Internet that has been cataloged, and indexed, by Google, Yahoo, Bing and the others, using basically what are known as web spiders, or web crawlers (no relation to your friendly, neighborhood, Peter Parker), that perform their magic quite simply by starting with one popular web page, cataloging every word on that page, then following every link on that page, and repeating the same process from the arrival points of each of those links, again and again until it (the web spider) falls into a spider trap, or hits a firewall, at which point it is stopped. Perhaps a password is required, which a “bot” is unable to provide, or perhaps a query, or an answer to a query is required. And once again, that’s the problem. If we just count the part of the entire Internet that can be reached, and indexed by these spider-bots, or crawlers, or whatever you choose to call them, before they encounter something that stops their progress, then various estimates arrived at anywhere ranging from 2001-2009 or later, have arrived at a number in the area of between 40-50 billion web pages, with approximately 56% of those being in English. These pages (English and otherwise) constitute what is known as the Surface Web, or the Open Web, and these are pretty much the only pages that can be reached by using the search engines that we have been talking about. If it was just about numbers, it really might seem like it should be enough to satisfy anyone. I mean, surely it’s a reasonable percentage of the entire Internet, most likely, the majority of it, right? Wrong.
First of all, even if it was just about numbers, the fact is that in comparison to the portion of the Internet that can be accessed by traditional search engines like those mentioned above, the second level of the Internet, variously known as the Invisible Web, or Deep Web, or Deepnet, or Hidden Web, which cannot except on the rarest of occasions ever be accessed by Google, or those other search engines that we talked about earlier, is the part of the Internet that is in the majority, and not just by a little bit either. Though its almost impossible to put even good guesses on the size of the Deep Web because the majority of the information is hidden, or locked in data-bases, some early estimations suggest the Invisible Web is 4000 to 5000 times larger than the Surface Web. This means that most of the Internet’s information is so far down it cannot be seen or retrieved by conventional search engines because those contents do not technically exist until they are dynamically created as the result of a specific search. An exception to un-indexed specialized information lies in Google Scholar, which allows you to search scholarly journals only.
Now in addition to the fact that the Deep Web contains 4000-5000 times as much information as the Surface Web, or World Wide Web, there is also the matter of quality of information. What kind of information does one find when they learn how to delve into the secrets of the Invisible Web? Let me offer you an excerpt from one of the reference articles I used in researching this post. “The information found on the Invisible Web is highly specialized and usually contained within databases. This highly specialized data allows you to find relevant, specific and scholarly information whether your research is narrowly focused or broadly focused. In addition, most of the information contained within these databases is maintained by academic institutions and written or prepared by scholars and experts in specific fields. As far as reliability and validity, the resources found on the Invisible Web are often the same as those you would find by physically browsing a large academic library’s print scholarly resources. They give you reliable, scholarly sources of information that allows you to write a stronger, better researched paper or assignment.³
So in essence, the Invisible Web is ideal for students, professionals, researchers, writers, and countless others. To use it requires the same kind of knowledge one requires for hunting down information in any library setting, though a solid background in university-level library research is always a major plus. With that kind of skill at one’s disposal, however, just imagine an Internet 4-5 thousand times more expansive than what you are already used to. It is, however, not the tool of professionals only. I have only 2 years of university to my name, and I have used the Deep Web at least casually for a number of years quite successfully, and I feel certain, any person of average intelligence can manage to do the same. In order for a person to travel in the Deep Web, a non-standard set of search engines will need to be used and I have included a link to a web-page that highlights ten of these for those of you who would like to give one or more of them a try.
And for those those of you who still feel that the Invisible Web is just too mysterious a place to explore on your own, there is absolutely no real need to bother with it. First of all, the Surface Web contains far more information than any one person will ever go through in several lifetimes as it is, and the way things usually go in these areas, it probably won’t be much longer before those wonderful guys at Google or Yahoo, or one of the other search engine suppliers comes up with a tool that will make it a simple matter for everyone to surf the Invisible Web as easily as they now traverse the Surface Web. Of course, by it’s very definition, when that time arrives, the whole thing will be called the Surface Web, and the Invisible Web will just be a concept that has faded into obscurity. And that leaves us with just one other Web to mention, and that’s all I’m going to do, is mention it…very briefly.
A while back, I believe it was a few of those wonderful fellows from one of those alphabet companies like the CIA, or the FBI, or ACME, or somebody like that, decided that what they really needed was a way to use the Internet whereby their activities could never be traced back to them. After a bunch of brainstorming, enter a system that was structured much like an onion, which thus became the emblem of a Network system known as TOR. Now this system, by directing every piece of information it handled through literally thousands of encrypted networks all over the world, pretty much achieved the desired result, that anyone using this system had a very good chance to remain anonymous, to everyone, forever. And then, what should have been the obviously expected result of creating this Frankenstein’s Internet Monster, resulted. Every criminal, terrorist, sicko, pervert, and other assorted form of human scum, flocked to this system as a means of communicating terrorist plots, assassination attempts, sending ransom notes and death threats, exchanging child pornography, and just about every other kind of illegal activity one can imagine. Not surprisingly, this Network, as well as others that now work on the same principle, are known as The Dark Web, and other than to say that there is absolutely no reason for any decent person (other than the law-enforcement types who have to battle those who use it), to ever wish to be on it, that’s all I choose to say about it.
So do yourselves a big favor, and stick with The Surface Web, or if you’re feeling adventurous, The Invisible Web. You really wouldn’t like the people you’re likely to meet on the Dark Web. Charles Manson might not like the people he would meet on the Dark Web. Stay healthy and safe. We’ll see you next time. Until then…