Fifth International World Wide Web Conference
May 6-10, 1996, Paris, France

Abstract:

We report on our examination of pages from the World Wide Web. We have analyzed data collected by the Inktomi Web crawler (this data currently comprises over 2.6 million HTML documents). We have examined many characteristics of these documents, including: document size; number and types of tags, attributes, file extensions, protocols, and ports; the number of in-links; and the ratio of document size to the number of tags and attributes. For a more limited set of documents, we have examined the following: the number and types of syntax errors and readability scores. These data have been aggregated to create a number of ranked lists, e.g., the ten most-used tags, the ten most common HTML errors.

Keywords:

HTML, statistics, tools, World Wide Web.

Introduction

Such an analysis of the content of HTML documents is of interest for several reasons:

• Evolution of HTML. Unused features and extensions that do not achieve a reasonable level of acceptance should be deprecated and, eventually, eliminated. This prevents the accretion of useless language features.
• Improving Web content. Widespread awareness of poor natural and markup language usage will promote the spread of helpful tools and practices.
• Control of HTML. The marketplace perceives the relative ability of vendors to force acceptance of new, non-standard language extensions as market ``strength.'' Understanding the true acceptance level of such extensions can help fight vendor disinformation.
• Sociological insights. Many interesting sociological observations may be derived from the content of Web pages.

• User studies. User surveys [COMM95, PITK94b, PITK95a, PITK95b, RISS95, YAHO95] and browser usage studies [CATL95, PITK94a] have become very common. Such studies focus on high-level user issues (e.g., choice of software, available connectivity) and low-level user-browser interaction (e.g., use of the back button). The information extracted, though valuable, is wholly user-centric.
• Content analyses of small data sets. There have been some attempts to perform simple analyses of the content of the Web. For example, the original Lycos project at Carnegie Mellon University's Center for Machine Translation [MAUL94] tracked a number of interesting statistics while their data set was relatively small. These included:
  • content of title and headings
  • 100 top keywords and first 20 lines
  • word frequency count
  • file size (bytes, words)
  • URL types
  • most-linked-to URLs
• Structural analysis. The CMU Lycos project generated at least one complete graph of their data set. The project's commercial successor, Lycos, Inc., now tracks the 250 most-linked-to sites as a side-effect of their indexing [LYCO95]. Other projects have focused on (graph-oriented) structural analysis as well. These include several Web visualization systems (e.g., Webspace [CHI95] and the Navigational View Builder [MUKH95]). For the most part, such visualization has been very small-scale and limited in scope. More sophisticated analyses are possible, combining both structural analysis and semantic modelling. A project at Xerox PARC [PIRO95] is conducting such analyses over small data sets.

Tools

• Web Data Collection
• Data Extraction and Manipulation
• Natural (English) Language Analysis
• Markup (HTML) Language Analysis

The data presented in this study comes entirely from Inktomi. The high speed of the crawler enables us, for the first time, to consider taking ``snapshots'' of the Web and analyzing them. As of this writing, the Inktomi team has crawled twice. The first set of runs, from July to October 1995, collected 1.3 million unique HTML documents. The second set of runs, in November 1995, collected 2.6 million unique HTML documents.

HTML Data Extraction and Manipulation: libink

libink consists of four major subcomponents:

• HTML parser. libink contains a simple flex-based HTML scanner. We found existing parsers too slow (especially true in the case of parsers written in scripting languages) or difficult to modify. The libink scanner is small, enabling us to make it both fast and relatively robust, as well as highly configurable. Like the W3C SGML/HTML lexical analyzer [CONN95], our scanner uses a callback interface to handle various events (e.g., recognition of a tag and its attributes). The W3C lexical analyzer, however, is not configurable.
• URL parser. The URL parser, unlike many freely-available implementations, conforms to RFC 1808 [FIEL95].
• Domain name service (DNS) translation and caching. We use Internet addresses to reduce hostname aliasing in our data. To speed up the lookup process, we provide a wrapper around the standard name service routines that caches all URL hostnames.
• General hash table services. The various lookup tables on which libink relies sometimes exceed the capacity of a single machine's physical memory. Therefore, in addition to in-memory hash tables, libink provides interfaces to striped on-disk hash tables (using GNU DBM) as well as hash-partitioned distributed hash tables (using ONC RPC). The distributed hash tables support 1ms turnaround on hash table lookups, which is far better than the 20-30ms required to fetch a hash table page from secondary storage.

Natural Language Analysis: style

Since most HTML documents do not conform to an internationalization standard, we applied heuristics to screen out non-English documents. We filtered out documents that contained any character with the high bit set (indicating a non-ASCII character set) or containing character sequences indicating known encodings (such as the Shift-JIS encoding of the Japanese character set).

Markup Language Analysis: weblint

Results

Here, ``other'' includes all domains other than the given top-level domains. For example, ``other'' contains all non-US top-level domains (such as Germany's .de).

We analyzed a variety of properties of these documents. In this paper, we present results on the following:

• Document Size
• Tag/Size Ratio
• Tag Usage
• Attribute Usage
• Browser-specific Extension Usage
• Port Usage
• Protocols Used in Child URLs
• File Types Used in Child URLs
• Number of In-links
• Readability
• Syntax Errors

Figure 1 presents different views of the size distribution. On first inspection, this distribution appears to be exponential (the magenta line represents the location of the mean). However, further zooming indicates a curve before the distribution begins to taper off. The final graph in Figure 1 contains a semilog plot of the same data (in which the sizes are plotted logarithmically and the number of documents is plotted arithmetically).

These simple size distribution plots proved to be very useful in detecting several problems with the data set. Many of the outliers were caused by one of two major classes of errors:

• Problematic URLs: when faced with incorrect URLs that contain valid prefixes, some HTTP servers return the file matching the valid prefix. For example, the data set contains hundreds of documents with URLs of the form http://bazaar.com/underground2.html/..., all of which are identical to http://bazaar.com/underground2.html. There does not appear to be a general way for a client program (such as a crawler) to differentiate this situation from a site containing a large number of identical files.
• CGI Error Responses: some of the most popular CGI programs, such as NCSA imagemap and CERN HTImage, report errors with messages containing HTTP status ``200'' (success). Because the image map programs all happen to return fixed error messages, we were able to detect and eliminate those particular messages, but there (again) does not appear to be any general way for a client to distinguish ``200'' error messages from valid documents.

Figure 2: Tag/Size Ratio

Tag Usage

We examined the most popular tags. The top graph of Figure 3 shows the top ten tags (ranked according to the number of documents in which the tag appeared at least once). The bottom graph indicates the average number of occurrences of the tag per document.

We also examined the least popular tags. Several tags, BDO, COLGROUP, and NOEMBED were used zero times in our data set of over 2.6 million HTML documents. A number of other tags appeared a very limited number of times.

Attribute Usage

We examined the most popular attributes. Figure 4 shows the top ten attributes (ranked according to the number of documents in which the attribute appeared at least once). HREF appeared an average of 14 times per document.

We also examined the least popular attributes. Several attributes, ACCEPT-CHARSET, AXIS, CHAROFF, and CONTROLS, were used zero times in our data set of 2.6 million HTML documents. A number of other attributes appeared a very limited number of times.

Browser-specific Extension Usage

Figure 5 shows the percentage of documents in which the four most popular extensions are used. The usage of most of these features has risen dramatically, indicating wide user acceptance. Other features, such as BLINK, have not experienced such growth.

Figure 6 indicates the popularity of various proposals for dynamic addition of functionality to browsers. APP and APPLET support SunSoft's Java ``applet'' language, DYNSRC supports VRML markup, and EMBED supports Netscape's third-party ``plug-in'' modules. All have enjoyed significant growth, though the oldest and most popular method (Java, first released in May 1995 [KARP95]) still has very low usage.

Port 80, the standard HTTP port, was used for approximately 94% of the documents. Port 70 (the standard Gopher port) was used for approximately 0.3% of the documents (this number is slightly lower than the 1% usage of port 70 observed in our earlier data set). We checked many of the documents being served from port 70; all the ones we examined were in fact HTML documents. Ports 8000, 8001, and 8080, and 8888 accounted for the majority of the remaining documents. The strong preference for ``8'' and ``80'' in the non-standard ports is presumably related to the standard port number ``80''

Protocols Used in Child URLs

The in-link entries marked with (*) indicate sites that are highly self-referential. That is, these sites (by inspection) appear to contain a great number of links to their own top-level pages. It would probably be instructive to count only links from outside a given site.

The numbers presented in Table 5 represent the scores of the different domains on the Kincaid readability test. Higher numbers represent more grammatical and lexical complexity. Lower numbers represent more simple structure and word choice. Documents with lower numbers are considered to be more ``readable''. The ``other'' domain is excluded because it represents extraordinarily diverse sources.

Conclusions

Truisms

Second, we observed empirically that the Web changes exceptionally quickly. Many properties of the documents in our first data set have altered in the months since the data was collected. The largest document in our data set was 1.6Mbytes; we checked the current size of that same document. It has grown to 9Mbytes. As another example, many of the most popular URLs in the first data set no longer exist.

Future Directions

Structural graph analysis has many applications in this area. In particular, analysis of the kind practiced by sociologists in structural network analysis [WASS94] promises insight. However, existing social network algorithms are several orders of magnitude more complex than is viable for a data set of this size. Significant work would have to be done to make such analysis feasible.

It would also be interesting to allow user-defined queries against the data set. The simplest functionality would be to allow a user to ascertain how a form-specified URL compared with the data set. A more interesting and complex interface would allow the user to define arbitrary queries on the data set.

REFERENCES

[BOWE96]

N. Bowers, ``Weblint Home Page (version 1.013),'' Khoral Research, Inc., Albuquerque, NM, Jan. 1996. Available as http://www.khoral.com/staff/neilb/weblint.html.

[CATL95]

L. D. Catledge and J. E. Pitkow, ``Characterizing Browsing Strategies in the World-Wide Web,'' Proc. 3rd Int. World Wide Web Conf., Darmstadt, Germany, Apr. 1995. Available as http://www.igd.fhg.de/www/www95/proceedings/papers/80/userpatterns/UserPatterns.Paper4.formatted.html.

[CHER81]

L. L. Cherry, ``Writing Tools - The STYLE and DICTION Programs,'' Computer Science Technical Report No. 91 (TM 79-1271-13), Bell Laboratories, Murray Hill, NJ, Feb. 1981. Revised version reprinted as L. L. Cherry and W. Vesterman, ``Writing Tools - The STYLE and DICTION Programs,'' 4.4 BSD User's Supplementary Documents, Computer Science Research Group, Berkeley, CA, 1994.

[CHI95]

E. H. Chi, ``Webspace Visualization,'' The Geometry Center, Univ. of Minnesota, Minneapolis, MN. Available as http://www.geom.umn.edu/docs/weboogl/webspace/webspace.html.

[COMM95]

CommerceNet Consortium, ``The CommerceNet/Nielsen Internet Demographics Survey,'' Menlo Park, CA, 1995. Available as http://www.commerce.net/information/surveys/execsum/exec_sum.html.

[CONN95]

D. Connolly, ``A Lexical Analyzer for HTML and Basic SGML,'' W3C Working Draft, World Wide Web Consortium, Cambridge, MA, Dec. 1995. Available as http://www.w3.org/pub/WWW/TR/.

[FIEL95]

R. Fielding, ``Relative Uniform Resource Locators,'' RFC 1808, June 1995. Available as http://www.cis.ohio-state.edu/htbin/rfc/rfc1808.

[FRYS94]

H. Frystyk and H. W. Lie, ``Towards a Uniform Library of Common Code: A Presentation of the World Wide Web Library,'' Proc. 2nd Int. World Wide Web Conference, Chicago, IL, Oct. 1994. Available as http://www.ncsa.uiuc.edu/SDG/IT94/Proceedings/DDay/frystyk/LibraryPaper.html.

[HANN95]

M. J. Hannah, ``HTML Reference Manual,'' Sandia National Laboratories, Albuquerque, NM, Dec. 1995. Available as http://www.sandia.gov/sci_compute/html_ref.html.

[KARP95]

R. Karpinski, ``Hot Java Arrives: Sun Aims to Revolutionize the Web,'' InteractiveAge, May 22, 1995. Available as http://techweb.cmp.com/ia/15issue/15hotjava.html

[LYCO95]

Lycos, Inc., ``The Lycos 250 and Hot Lists,'' Pittsburgh, PA, Sep. 1995. Available as http://www.lycos.com/lists/index.html.

[MAUL94]

M. L. Mauldin and J. R. R. Leavitt, ``Web Agent Related Research at the Center for Machine Translation,'' 1994 Meeting of the ACM Special Interest Group on Networked Information Discovery and Retrieval, McLean, VA, Aug. 1994. Available as http://fuzine.mt.cs.cmu.edu/mlm/signidr94.html, Carnegie Mellon Univ., Jul. 1994.

[MUKH95]

S. Mukherjea and J. D. Foley, ``Visualizing the World-Wide Web with the Navigational View Builder,'' Proc. 3rd Int. World Wide Web Conf., Darmstadt, Germany, Apr. 1995. Available as http://www.igd.fhg.de/www/www95/proceedings/papers/44/mukh/mukh.html.

[PIRO95]

P. Pirolli, J. Pitkow and R. Rao, ``Silk from a Sow's Ear: Extracting Usable Structures from the Web,'' Xerox PARC, Palo Alto, CA, Nov. 1995. Submitted for publication.

[PITK94a]

J. E. Pitkow and K. Bharat, ``WEBVIZ: A Tool for World Wide Web Access Log Visualization,'' Proc. 1st Int. World Wide Web Conf., Geneva, Switzerland, May 1994. Available as http://www1.cern.ch/WWW94/PrelimProcs.html.

[PITK94b]

J. E. Pitkow and M. M. Recker, ``Results From The First World-Wide Web User Survey'', Georgia Institute of Technology, Atlanta, GA, Jan. 1994. Available as http://www.gatech.edu/pitkow/survey/survey-1-1994/survey-paper.html.

[PITK95a]

J. E. Pitkow and M. M. Recker, ``Using the Web as a Survey Tool: Results from the Second WWW User Survey,'' Proc. 3rd Int. World Wide Web Conf., Darmstadt, Germany, Apr. 1995. Available as http://www.igd.fhg.de/www/www95/proceedings/papers/79/survey/survey_2_paper.html.

[PITK95b]

J. E. Pitkow and C. Kehoe, ``The GVU Center's 3rd WWW User Survey,'' Georgia Institute of Technology, Atlanta, GA, Apr. 1995. Available as http://www.cc.gatech.edu/gvu/user_surveys/survey-04-1995/.

[RISS95]

M. Rissa and C. Oy, ``WWW User Survey Results,'' Helsinki, Finland, Feb. 1995. Available as http://www.mroy.fi/dec94.htm.

[WASS94]

S. Wasserman and K. Faust, ``Social Network Analysis: Methods and Applications,'' Cambridge University Press, Cambridge, UK, 1994.

[YAHO95]

Yahoo, Inc., ``Survey Says...'' Mountain View, CA, Aug. 1995. Available as http://www.yahoo.com/docs/survey/first.html and http://www.yahoo.com/docs/survey/index.html.