Data mining is the process of sorting through large amounts of data and picking out relevant information. Sorting is any process of arranging items in some sequence and/or in different sets and accordingly it has two common yet distinct meanings ordering: arranging It is usually used by business intelligence organizations, and financial analysts, but is increasingly being used in the sciences to extract information from the enormous data sets generated by modern experimental and observational methods. Business intelligence ( BI) refers to technologies applications and practices for the collection integration analysis and presentation of business Information and A financial analyst (or securities analyst, research analyst, equity analyst, investment analyst) is a person who works with Financial analysis A data set (or dataset) is a collection of Data, usually presented in tabular form It has been described as "the nontrivial extraction of implicit, previously unknown, and potentially useful information from data"^[1] and "the science of extracting useful information from large data sets or databases. Information as a concept has a diversity of meanings from everyday usage to technical settings Debt AIDS Trade in Africa (or DATA) is a Multinational non-government organization founded in January 2002 in London by U2 's A data set (or dataset) is a collection of Data, usually presented in tabular form A Computer Database is a structured collection of records or data that is stored in a computer system "^[2] Data mining in relation to enterprise resource planning is the statistical and logical analysis of large sets of transaction data, looking for patterns that can aid decision making. Enterprise resource planning ( ERP) is the planning of how business resources (materials employees customers etc ^[3]

Contents 1 Background 2 Privacy concerns 3 Notable uses of data mining 3.1 Terrorism 3.2 Games 3.3 Business 3.4 Science and engineering 4 A typical example 5 See also 6 References 7 Sources 8 Books 9 External links

Background

Traditionally, analysts have performed the task of extracting useful information from recorded data, but the increasing volume of data in modern business and science calls for computer-based approaches. Information as a concept has a diversity of meanings from everyday usage to technical settings Debt AIDS Trade in Africa (or DATA) is a Multinational non-government organization founded in January 2002 in London by U2 's As data sets have grown in size and complexity, there has been a shift away from direct hands-on data analysis toward indirect, automatic data analysis using more complex and sophisticated tools. A data set (or dataset) is a collection of Data, usually presented in tabular form The modern technologies of computers, networks, and sensors have made data collection and organization much easier. A computer is a Machine that manipulates data according to a list of instructions. A sensor is a device that measures a physical quantity and converts it into a signal which can be read by an observer or by an instrument Data collection is a term used to describe a process of preparing and collecting Data - for example as part of a Process improvement or similar project However, the captured data needs to be converted into information and knowledge to become useful. Information as a concept has a diversity of meanings from everyday usage to technical settings Knowledge is defined ( Oxford English Dictionary) variously as (i expertise and skills acquired by a person through experience or education the theoretical or practical understanding Data mining is the entire process of applying computer-based methodology, including new techniques for knowledge discovery, from data. Methodology (also called manner) is defined as "the analysis of the principles of methods rules and postulates employed by a discipline" Knowledge discovery is a concept of the field of Computer science that describes the process of automatically searching large volumes of Data for patterns that ^[4]

Data mining identifies trends within data that go beyond simple analysis. Through the use of sophisticated algorithms, users have the ability to identify key attributes of business processes and target opportunities.

Although data mining is a relatively new term, the technology is not. Companies for a long time have used powerful computers to sift through volumes of data such as supermarket scanner data to produce market research reports. Continuous innovations in computer processing power, disk storage, and statistical software are dramatically increasing the accuracy and usefulness of analysis.

The term data mining is often used to apply to the two separate processes of knowledge discovery and prediction. A prediction is a statement or claim that a particular Event will occur in the Future in more certain terms than a forecast. Knowledge discovery provides explicit information that has a readable form and can be understood by a user. Forecasting, or predictive modeling provides predictions of future events and may be transparent and readable in some approaches (e. Forecasting is the process of Estimation in unknown situations Predictive modelling is the process by which a model is created or chosen to try to best predict the Probability of an outcome g. rule based systems) and opaque in others such as neural networks. Traditionally the term neural network had been used to refer to a network or circuit of biological neurons. Moreover, some data-mining systems such as neural networks are inherently geared towards prediction and pattern recognition, rather than knowledge discovery.

Metadata, or data about a given data set, are often expressed in a condensed data-minable format, or one that facilitates the practice of data mining. Metadata ( meta data, or sometimes metainformation) is "data about data" of any sort in any media Common examples include executive summaries and scientific abstracts.

Data mining relies on the use of real world data. This data is extremely vulnerable to collinearity precisely because data from the real world may have unknown interrelations. An unavoidable weakness of data mining is that the critical data that may explain the relationships is never observed. Alternative approaches using an experiment based approach such as Choice Modelling for human-generated data may be used. Choice modelling attempts to model the decision process of an individual or segment in a particular context Inherent correlations are either controlled for or removed altogether through the construction of an experimental design. Design of experiments, or experimental design, is the design of all information-gathering exercises where variation is present whether under the full control of the experimenter

Recently, there were some efforts to define a standard for data mining, for example the CRISP-DM standard for analysis processes or the Java Data-Mining Standard. CRISP-DM stands for CRoss Industry Standard Process for Data Mining Independent of these standardization efforts, freely available open-source software systems like RapidMiner and Weka have become an informal standard for defining data-mining processes. RapidMiner (formerly YALE (Yet Another Learning Environment is an environment for Machine learning and Data mining experiments Weka (Waikato Environment for Knowledge Analysis is a popular suite of Machine learning software written in Java, developed at the University of Waikato

Privacy concerns

There are also privacy and human rights concerns associated with data mining, specifically regarding the source of the data analyzed. Privacy is the ability of an individual or group to seclude themselves or information about themselves and thereby reveal themselves selectively Human rights refers to the "basic Rights and freedoms to which all humans are entitled Data mining provides information that may be difficult to obtain otherwise. When the data collected involves individual people, there are many questions concerning privacy, legality, and ethics. ^[5] In particular, data mining government or commercial data sets for national security or law enforcement purposes has raised privacy concerns. ^[6][7]

Notable uses of data mining

Terrorism

Data mining has been cited as the method by which the U. S. Army unit Able Danger had identified the September 11, 2001 attacks leader, Mohamed Atta, and three other 9/11 hijackers as possible members of an Al Qaeda cell operating in the U. Able Danger was a classified military planning effort under the command of the U Mohamed Atta ( ar محمد عطا السيد) (September 1 1968 &ndash September 11 2001 was an Egyptian Islamist, S. more than a year before the attack.

It has been suggested that both the Central Intelligence Agency and the Canadian Security Intelligence Service have employed this method. near as long as it used to be several months ago It has been actively summarized and split into sub-articles and there is a dynamic talk page discussion of all The Canadian Security Intelligence Service ( CSIS) (Service canadien du renseignement de sécurité ( SCRS) is the primary Intelligence agency of the ^[8]

Previous data mining to stop terrorist programs under the US government include the Terrorism Information Awareness (TIA) program, Computer-Assisted Passenger Prescreening System (CAPPS II), Analysis, Dissemination, Visualization, Insight, and Semantic Enhancement (ADVISE), Multistate Anti-Terrorism Information Exchange (MATRIX), and the Secure Flight program Security-MSNBC. These programs have been discontinued due to controversy over whether they violate the US Constitution's 4th amendment.

Games

Since the early 1960s, with the availability of oracles for certain combinatorial games, also called tablebases (e. In complexity theory and computability theory, an oracle machine is an Abstract machine used to study Decision problems It can be visualized This article is on the theory of combinatorial games For the theory that includes games of chance and games of imperfect knowledge see Game theory An endgame tablebase is a computerized Database of all Chess positions within certain endgames The tablebase reveals the game-theoretical value g. for 3x3-chess) with any beginning configuration, small-board dots-and-boxes, small-board-hex, and certain endgames in chess, dots-and-boxes, and hex; a new area for data mining has been opened up. Dots and Boxes (also known as Boxes, Squares, Paddocks, Square-it, Dots and Dashes, Dots, Smart Dots, or simply This is the extraction of human-usable strategies from these oracles. Current pattern recognition approaches do not seem to fully have the required high level of abstraction in order to be applied successfully. Instead, extensive experimentation with the tablebases combined with an intensive study of tablebase-answers to well designed problems and with knowledge of prior art i. e. pre-tablebase knowledge is used to yield insightful patterns. Berlekamp in dots-and-boxes etc. Elwyn Ralph Berlekamp (born September 6, 1940 in Dover, Ohio, United States of America) is a professor emeritus of Mathematics and John Nunn in chess endgames are notable examples of researchers doing this work, though they were not and are not involved in tablebase generation. John Denis Martin Nunn (born April 25, 1955 in London) is one of England 's strongest Chess players and once belonged to the world's Chess is a recreational and competitive Game played between two players. In Chess, the endgame (or end game or ending) refers to the stage of the game when there are few pieces left on the board

Business

Data mining in customer relationship management applications can contribute significantly to the bottom line. Customer relationship management ( CRM) is a term applied to processes implemented by a company to handle its contact with its customers Rather than contacting a prospect or customer through a call center or sending mail, only prospects that are predicted to have a high likelihood of responding to an offer are contacted. More sophisticated methods may be used to optimize across campaigns so that we can predict which channel and which offer an individual is most likely to respond to - across all potential offers. Finally, in cases where many people will take an action without an offer, uplift modeling can be used to determine which people will have the greatest increase in responding if given an offer. Data clustering can also be used to automatically discover the segments or groups within a customer data set. Clustering is the classification of objects into different groups or more precisely the partitioning of a Data set into Subsets (clusters

Businesses employing data mining quickly see a return on investment, but also they recognize that the number of predictive models can quickly become very large. Rather than one model to predict which customers will churn, a business could build a separate model for each region and customer type. Then instead of sending an offer to all people that are likely to churn, it may only want to send offers to customers that will likely take to offer. And finally, it may also want to determine which customers are going to be profitable over a window of time and only send the offers to those that are likely to be profitable. In order to maintain this quantity of models, they need to manage model versions and move to automated data mining.

Data mining can also be helpful to human-resources departments in identifying the characteristics of their most successful employees. Information obtained, such as universities attended by highly successful employees, can help HR focus recruiting efforts accordingly. Additionally, Strategic Enterprise Management applications help a company translate corporate-level goals, such as profit and margin share targets, into operational decisions, such as production plans and workforce levels. ^[9]

Another example of data mining, often called the market basket analysis, relates to its use in retail sales. Affinity analysis is a data analysis and Data mining technique that discovers co-occurrence relationships among activities performed by (or recorded about specific individuals If a clothing store records the purchases of customers, a data-mining system could identify those customers who favour silk shirts over cotton ones. Although some explanations of relationships may be difficult, taking advantage of it is easier. The example deals with association rules within transaction-based data. In Data mining, association rule learning is a popular andwell researched method for discovering interesting relations between variablesin large databases Not all data are transaction based and logical or inexact rules may also be present within a database. A Computer Database is a structured collection of records or data that is stored in a computer system In a manufacturing application, an inexact rule may state that 73% of products which have a specific defect or problem, will develop a secondary problem within the next six months.

Related to an integrated-circuit production line, an example of data mining is described in the paper "Mining IC Test Data to Optimize VLSI Testing. "^[10] In this paper the application of data mining and decision analysis to the problem of die-level functional test is described. Experiments mentioned in this paper demonstrate the ability of applying a system of mining historical die-test data to create a probabilistic model of patterns of die failure which are then utilized to decide in real time which die to test next and when to stop testing. This system has been shown, based on experiments with historical test data, to have the potential to improve profits on mature IC products.

Science and engineering

In recent years, data mining has been widely used in area of science and engineering, such as bioinformatics, genetics, medicine, education, and electrical power engineering. Bioinformatics is the application of information technology to the field of molecular biology Medicine is the art and science of healing It encompasses a range of Health care practices evolved to maintain and restore Human Health by the Education encompasses both the Teaching and Learning of Knowledge, proper conduct, and technical competency Electric power is defined as the rate at which Electrical energy is transferred by an Electric circuit.

In the area of study on human genetics, the important goal is to understand the mapping relationship between the inter-individual variation in human DNA sequences and variability in disease susceptibility. Deoxyribonucleic acid ( DNA) is a Nucleic acid that contains the genetic instructions used in the development and functioning of all known In lay terms, it is to find out how the changes in an individual's DNA sequence affect the risk of developing common diseases such as cancer. Cancer (medical term Malignant Neoplasm) is a class of Diseases in which a group of cells display uncontrolled This is very important to help improve the diagnosis, prevention and treatment of the diseases. The data mining technique that is used to perform this task is known as multifactor dimensionality reduction. Multifactor dimensionality reduction (MDR is a Data mining approach for detecting and characterizing combinations of Attributes or Independent variables ^[11]

In the area of electrical power engineering, data mining techniques have been widely used for condition monitoring of high voltage electrical equipment. Condition monitoring is the process of monitoring a parameter of condition in machinery such that a significant change is indicative of a developing failure The purpose of condition monitoring is to obtain valuable information on the insulation's health status of the equipment. Data clustering such as self-organizing map (SOM) has been applied on the vibration monitoring and analysis of transformer on-load tap-changers(OLTCS). Clustering is the classification of objects into different groups or more precisely the partitioning of a Data set into Subsets (clusters A self-organizing map (SOM is a type of Artificial neural network that is trained using Unsupervised learning to produce a low-dimensional (typically two dimensional Using vibration monitoring, it can be observed that each tap change operation generates a signal that contains information about the condition of the tap changer contacts and the drive mechanisms. Obviously, different tap positions will generate different signals. However, there was considerable variability amongst normal condition signals for the exact same tap position. SOM has been applied to detect abnormal conditions and to estimate the nature of the abnormalities. ^[12]

Data mining techniques have also been applied for dissolved gas analysis (DGA) on power transformers. Dissolved gas analysis, or DGA, is a diagnostic and maintenance tool used in Machinery. A transformer is a device that transfers Electrical energy from one circuit to another through inductively coupled Electrical conductors DGA, as a diagnostics for power transformer, has been available for centuries. Data mining techniques such as SOM has been applied to analyse data and to determine trends which are not obvious to the standard DGA ratio techniques such as Duval Triangle. ^[13]

A fourth area of application for data mining in science/engineering is within educational research, where data mining has been used to study the factors leading students to choose to engage in behaviors which reduce their learning^[14] and to understand the factors influencing university student retention. ^[15]

Other examples of applying data mining technique applications are biomedical data facilitated by domain ontologies,^[16] mining clinical trial data,^[17] traffic analysis using SOM,^[18] et cetera. Biomedical research (or experimental medicine) in general simply known as medical research, is the Basic research or Applied research conducted Traffic analysis is the process of intercepting and examining messages in order to deduce information from patterns in Communication.

A typical example

A useful example is the large database of thousands of beautiful photos of all kind, collected - and enhanced almost daily - since more than a decade by the Norwegian "anonymous" Svein Ulvund. This huge database is available under www. vossnow. net, [1] (Voss is a town in the Norwegian district of Hordaland near Bergen).

To exploit that photo-collection is not easy. For example, as a typical task, one may want to find out photos from a specific place, e. g. the famous Nærøyfjord nearby. The Nærøyfjord ( Nærøyfjorden) is a Fjord in the municipality of Aurland in Sogn og Fjordane, Norway. Note that the name of that fjord involves letters which are not easily available on a non-Norwegian keyboard. However, to get around that problem one can find out towns on the fjord without the above-mentioned unfortunate property, e. g. Gudvangen. Gudvangen is a village in Aurland municipality in Sogn og Fjordane county Norway. This can be done with the help of a conventional map, or on the internet. Thus, finally one enters the searchpage of the database, using the two entries Gudv (which suffices for Gudvangen) and the year and month, e. g. Mai 2006, say (not May 2006, since the search machine "speaks Norsk"). Then one gets still a rather large collection of photos from the Nærøyfjord, but now the number is reduced in such a way that it makes sense to have a final selection, either personally or otherwise.

This simple example also makes two points obvious: (i) the necessity to automatize the process and (ii) some of the problems involved.

See also

• Data warehouse
• Data analysis
• Structured data analysis (statistics)

References

Sources

• Kurt Thearling, An Introduction to Data Mining (also available is a corresponding online tutorial)
• Dean Abbott, I. Philip Matkovsky, and John Elder IV, Ph. D. An Evaluation of High-end Data Mining Tools for Fraud Detection published a comparative analysis of major high-end data mining software tools that was presented at the 1998 IEEE International Conference on Systems, Man, and Cybernetics, San Diego, CA, October 12-14, 1998.
• Mierswa, Ingo and Wurst, Michael and Klinkenberg, Ralf and Scholz, Martin and Euler, Timm: YALE: Rapid Prototyping for Complex Data Mining Tasks, in Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-06), 2006.
• Peng, Y. , Kou, G. , Shi, Y. and Chen, Z. "A Systemic Framework for the Field of Data Mining and Knowledge Discovery", in Proceeding of workshops on The Sixth IEEE International Conference on Data Mining Technique (ICDM), 2006

Books

• Peter Cabena, Pablo Hadjnian, Rolf Stadler, Jaap Verhees, Alessandro Zanasi, Discovering Data Mining: From Concept to Implementation (1997), Prentice Hall, ISBN 0137439806
• Ronen Feldman and James Sanger, The Text Mining Handbook, Cambridge University Press, ISBN 9780521836579
• Pang-Ning Tan, Michael Steinbach and Vipin Kumar, Introduction to Data Mining (2005), ISBN 0-321-32136-7 (companion book site)
• Bing Liu, "Web Data Mining: Exploring Hyperlinks, Contents and Usage Data", Springer, 2007, ISBN-10: 3-540-37881-2
• Galit Shmueli, Nitin R. Patel and Peter C. Bruce , Data Mining for Business Intelligence (2006), ISBN 0-470-08485-5 (companion book site)
• Richard O. Duda, Peter E. Hart, David G. Stork, Pattern Classification, Wiley Interscience, ISBN 0-471-05669-3, (see also Powerpoint slides)
• Phiroz Bhagat, Pattern Recognition in Industry, Elsevier, ISBN 0-08-044538-1
• Ian Witten and Eibe Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations (2000), ISBN 1-55860-552-5, (see also Free Weka software)
• Mark F. Weka (Waikato Environment for Knowledge Analysis is a popular suite of Machine learning software written in Java, developed at the University of Waikato Hornick, Erik Marcade, Sunil Venkayala: "Java Data Mining: Strategy, Standard, And Practice: A Practical Guide for Architecture, Design, And Implementation" (Broché)
• Weiss and Indurkhya, Predictive Data Mining, Morgan Kaufman
• Yike Guo and Robert Grossman, editors: High Performance Data Mining: Scaling Algorithms, Applications and Systems, Kluwer Academic Publishers, 1999
• Trevor Hastie, Robert Tibshirani and Jerome Friedman (2001). The Elements of Statistical Learning, Springer. ISBN 0387952845 (companion book site)
• Pascal Poncelet, Florent Masseglia and Maguelonne Teisseire (Editors). Data Mining Patterns: New Methods and Applications , Information Science Reference, ISBN 978-1599041629, (October 2007).

External links

• Data Mining at the Open Directory Project
• Data Rich and Information Poor

The Open Directory Project ( ODP) also known as dmoz (from directory

© 2009 citizendia.org; parts available under the terms of GNU Free Documentation License, from http://en.wikipedia.org
Dapyx Software network: MP3 Explorer | Ebook Manager | Zenithic