A model of USENET newsgroups dynamics: Implementation and results

Abstract

We present an implementation of the multilevel text processing model of discussions in USENET groups proposed earlier (NLDB '02 Proceedings). In the statistical processing phase, a discussion thread is SGML tagged to include the relevant information about parent-child relationships among the postings as well as other meta data of postings and threads. This tagged output is then processed by a generic information retrieval system. Various relevant metrics that measure properties of discussions (such as thread focus, relevance of posting, discussion density, etc.) are defined and computed. The subsequent semantic component (utilizing tools like electronic lexicons, POS taggers and parsers) has been implemented to work in a modular fashion to allow inclusion or exclusion of some of its subcomponents. The user may tune this module to its minimal level to process semantics of individual words only, or up to its maximal level to include words with their full contexts. We also present evaluation data assessing the performance of the system with or without some of its modules.