As soon as the World Wide Web began expanding in the early 1990s, a problem quickly emerged of how to locate content on the web. The web had been designed to enable computers connect and share info. However, people soon realized that finding the right info was becoming challenging. Before the advent of search engines like Google, Yahoo and Bing, one of the ways created for finding information was through web directories.
A web directory is basically a collection of website links. The links are typically organized according to categories such as sports, history, travel, entertainment, and so on. Each category typically has a number of subcategories. A person searching for information can browse by category or type a query in a search box. A web directory is sometimes referred to as a subject directory or link directory, and a good example is DMOZ.
A Brief History of Web Directories
The first recorded web directory was created in 1992 by the father of the internet, Tim Berners-Lee. It was a simple list of webservers, edited by Sir Tim, and hosted on the server at CERN. One of first pages from 1992 is still preserved for historical purposes on the W3C archives. Although Sir Tim’s directory is nowhere as sophisticated as today’s web directories, it nevertheless laid a foundation on which future innovators built their directory methodologies.
The first batch of popular web directories began to emerge in the mid 90s. Three of the most popular all started in 1995. They are Lycos’ TOP 5%, LookSmart and The Starting Point Directory. Even internet giant Yahoo!, when it initially launched in 1995, started out as a collection of website directories i.e. Yahoo! Directory. Of these early pioneers, only Yahoo and The Starting Point Directory are still operational. Lycos’ Top 5% closed shop in 2000 and LookSmart followed suit in 2006.
The decade from 1995 to 2005 can be best described as “the golden era of web directories.” This is when the use of web directories was at its highest. Between 1995 and 2000, it was a question of necessity. Web crawlers hadn’t yet taken off. As such, web directories were the only available way of finding information on the web. All the major tech giants were investing in web directories. One of the directories unleashed in this time period was The Mozilla Directory (DMOZ), which was released in 1998. DMOZ is currently one of the largest web directories in the world.
However, the period from 95 to 2000 saw an exponential rise in the number of websites on the web. This put a serious strain on web directories. By then, web directories were edited by people. As such, the personnel demands required for keeping up with new upcoming websites were becoming too high. People began looking for automated options for indexing websites – a search that led to the unveiling of web crawlers with the emergence of Google in 1999. In a way, the emergence of Google marked the beginning of the decline for web crawlers.
Between 2000 and 2005, search engines began providing an alternative to web directories. Google was becoming increasingly popular. Even Yahoo, an early provider of search options, switched from the directory approach to web crawlers in 2002. However, most internet users were still in love with web directories. As such, even Google unveiled a directory option – Google Directory – which was actually a mirror of DMOZ. By the year 2005, internet users had warmed up to search engines, and the relative importance web directories began slowly going into decline.
From 2005 to date, web directories have been in decline. In 2006, one of the pioneers in the industry, LookSmart closed shop. In 2011, Google Directory was also shut down. As for Yahoo!, it started off by shutting down its country-based directories. Finally, the company announced in September this year that Yahoo Directory will be closed down on 31st December, 2014.
A Bleak Future?
The announcement of Yahoo’s plan to close its directory service has caused some analysts to predict that the future of the web directory industry is bleak. However, such predictions may be quite alarmist. It is true that web crawlers make it easier and cheaper to index websites than the manual approach taken by web directories.
However, web directories still carry a feeling of authenticity. The fact that a website has been looked at by a human being rather than an algorithm gives it a feeling of authority. Studies show that people trust the listings in web directories far more than they do the results in SERPs. For people who are seeking specialist information (such as academic and technical information), this trust is critical. This is why specialist directories like Business.com and Starting Point Directory are becoming more popular by the day.
The only challenge that a web directory provider has to contend with is keeping the directory list up-to-date. This can be a huge task, given the fact that the websites have to be reviewed manually. Getting people to review the websites can be a complex and expensive task. There are two major approaches which are being used by the successful web directories to solve this problem.
The first is to introduce a paid option. This is the approach that was taken by The Starting Point Directory. Faced by increasing maintenance costs, the company re-launched as a paid service in 2006. This enabled it to weather the storms that caused its fellow pioneers to sink. With a paid option, website owners basically pay to be reviewed by the website directory.
Upon review, a website meeting the requirements of the directory is listed. Those who don’t meet the requirements can be given constructive feedback which can enable them to qualify for listing. There are different variations to the paid up option — with some directories charging higher prices to appear near the top of the list (or on the directory’s first pages).
The second option is to use voluntary reviewers and editors. This is the approach taken by DMOZ. As of February 2014, there were over 97,500 editors working on the DMOZ directory. The vast majority of these are volunteers. Using the voluntary option basically saves on the costs of reviewing websites.
Successful web directories are also beginning to employ indexing technologies in order to complement the work of human editors and reviewers. Such technologies are specifically used to keep track of the continuing availability of the listed websites. For instance, DMOZ employs a web crawler called Robozilla which is tasked with tracking the status of all websites listed on DMOZ. It flags off websites which seem to have disappeared or moved, and the editors can then check them out.
In a nutshell, web directories first emerged in order to enable people to quickly find info on the web. But then, the appearance of automated search indexing provided a cheaper and more efficient alternative to their manual-based directory listing. This caused many major directories to close down. However, web directories still have an important role to play in providing a reputable way of finding information. Their uniqueness comes from the fact that the websites they list are verified by human beings — a fact which will always make them a first choice for those who are looking for reputable information on the web.