Mining the Internet for Competitive Technical Intelligence

Competitive Intelligence Magazine, Society of Competitive Intelligence Professionals, Volume 10, Number 5, September-October 2007

Alan L. Porter, David J. Schoeneck, and Paul R. Frey (Search Technology, Inc.) and
Diana M. Hicks and Dirk P. Libaers (Georgia Tech)

Published: 2007


In a 2005 issue of CIM we described a “tech mining” approach to generate competitive technical intelligence (CTI) (Porter, 2005). Tech mining addresses managerial issues by deriving empirical knowledge, primarily from patent and research publication abstract databases. This article extends the resource base to be mined to the internet. Technology intelligence detects opportunities based on early identification of emerging technologies pertinent to a company’s business interests. Moreover, it identifies areas in the competitive landscape with limited or no competition, where corporate strategy can exploit these areas. Competitive intelligence tracks competitor activities to spot threats early. CTI blends elements of both.


Software can aid in extracting intelligence from database searches — for example, by retrieving research publication abstract records on “fuel cells” or patents assigned to “International Fuel Cells” (now a unit of United Technologies). Extracting knowledge to meet strategic intelligence needs is well and good, but companies want more!

Table 1 presents a larger picture of competitive technical intelligence resources. These resources exploit technological content from publicly accessible and clients’ confidential databases, and also extract information from business and general databases such as LexisNexis and Factiva. This kind of empirically derived knowledge from databases and the internet should be complemented by suitable tacit knowledge from individuals. For instance, first map the hot spots of fuel cell research and development (R&D) activity, then have technical experts refine and interpret the prospects (Table 1 E). Additionally, tap business experts to explore the ramifications of enhanced technical capabilities (Table 1 F). Users of CTI information want answers to their questions rather than nicely defined puzzle pieces. That’s a tall order, but there are practical ways to extend the information compilation to include the internet (cells C and D in Table 1). We first draw upon search engines such as Google to augment our databasederived results from the internet, then look at specific sites. For instance, our fuel cells search identifies an active research center at Georgia Tech. We would then look up their web site to check whether key researchers are still located there, see their most recent research efforts, and obtain contact information. But we need more.

A typical searcher is looking for ONE result. Sometimes this is recovering a previously known source; other times it is discovering a new one (Battelle, 2005). For CTI purposes, we often want to capture an entire body of information. Taking the fuel cell illustration, we identified a set of active R&D center web sites. We then probed further by profiling what fuel cell types those active centers emphasize to spot trends as key centers shift toward emerging technologies, or to discern the range of applications. Here’s how we developed this type of internet-derived intelligence. ... (View the full paper in PDF format)

Keywords: Technology intelligence; data mining; Internet; Google; innovation; small business; research

Porter, A. L., Schoeneck, D. J., Frey, P.R. (Search Technology, Inc.), and Diana Hicks, Dirk P. Libaers (Georgia Institute of Technology), 2007. This article first appeared in the September/October 2007 issue of "Competitive Intelligence Magazine" Volume 10, Number 5.
Society of Competitive Intelligence Professionals


Home  Contact Us          © 2001 - 2018 Search Technology, Inc.  |  Privacy Statement