IvoaKDD < IVOA

---+ IVOA Knowledge Discovery Interest Group

---

%TOC%
---

---++ Charter

Knowledge discovery is the task of processing and analyzing data-sets with the aim of extracting new knowledge. This area spans widely across multiple disciplines, including visualization, remote data exploration, machine learning techniques, statistical methods, workflow orchestration, and polymorphic data access. To support the process of discovery, the KD-IG interacts closely with the other working/interest groups and feeds back requirements of the scientific community.

The activities of the KD-IG include the following items with a strong emphasis on the first two points:

   1 _Participating in the definition of new data preservation and exchange formats with respect to support machine learning algorithms._
   1 <em>Introducing uncertainties and probabilistic description to VO standards and services.<br /> <br /></em>
   1 Presenting and collecting best practice examples of scientific data analytics in astronomy.
   1 Defining requirements for implementing and adding machine learning capabilities to services.
   1 Coordinating and unifying the access to data visualization functionalities.
   1 Discussing the aspect of data provenance with respect to data used to derive/train models.
   1 Introducing proper statistical scoring and evaluation methods as services.
   1 Contributing to the discussion on scripting and orchestrating the scientific discovery workflow.
   1 Supporting the development of dedicated knowledge discovery applications.
<!-- <p>We will develop and test scalable data mining algorithms and the accompanying new standards for VO interfaces and protocols, so that these algorithms can be discovered and used transparently within VO science workflows or in standalone data exploration applications. Therefore the activities of the KDD-IG will be:</p> <ol> <li> Support the definition of an ontology of the KDD tasks required by the astronomical community. This ontology will be used to define programming and documentation standards. </li> <li> Make an inventory of existing methods relevant for astrophysical applications (more than 100 new KDD models and methods appear every month on specialized journals). </li> <li> Identify reference data sets to be used for comparing, debugging and testing methods and tools. </li> <li> Foster the implementation, using available VO standards and methods, of general purpose data exploration and data mining methods which will allow the general user to seamlessly exploit the complex data repositories offered by the VO. </li> <li> Provide/receive feedbacks to/from the WGs in order to improve the usability of VO tools and standards. </li> <li> Provide/receive from the community information to improve both the usability and the potentialities of Data Mining tools under the VO. </li> <li> Define and pursue specific science cases which will be used to showcase the VO capabilities to the community. </li> </ol><p>More important than anything else, we wish to use this IG as an arena where different groups can share experiences and plan future developments.</p> <p> </p> -->
---++ Motivations

During the [[IVOA.InterOpMay2009][Strasbourg InterOp Meeting]] it emerged the need for an Interest Group on Data Mining (KD-IG) as an indispensable step to bridge the Virtual Observatory Infrastructure with the expected VO science. In fact, _"...Data mining, or KDD, is the semi-automatic discovery of patterns, associations, changes, anomalies, and statistically significant structures and events in data. In other words, traditional data analysis is assumption driven as a hypothesis is formed and validated against the data. Data mining, in contrast, is discovery driven as the patterns are automatically extracted from data....&rdquo;_ <!-- <p>As such, Data Mining (DM) can be considered as the “frontier” of VO enabled science since it represents the only way to capture and reveal the scientific knowledge (patterns, trends, correlations, etc.) hidden behind the complexity of Massive Data Sets.</p> -->

Data Mining is a rapidly evolving set of methodologies which needs to be imported under the VO umbrella and not just another application. As such, DM cannot be just a tool or a suite of tools offered by a group of developers to a &ldquo;passive community&rdquo;. Data Mining involves a large number of researchers across many domains. The astronomical community, which has only recently entered the Massive Data Sets era, makes use of just a handful of methods and tools which very often are far from optimal. The synergy of different expertise present in the IVOA makes it the ideal arena for exploring new and more modern approaches.

KD-IG requires a strong and continuous interaction with the scientific community which, besides testing the proposed solutions, methods, and tools, will also provide feedback and inputs aiming at extending the scientific capabilities of the VO.

The KD-IG will interfaces to many other IVOA working and interest groups: Applications, Semantics, Time Domain, Data Models, Grid & Web Services, and Resource Registry. This cross- discipline nature is also a primary reason to create a specific IG. Data Mining, in fact, addresses sophisticated and extreme modes of usage which require a careful orchestration and fine tuning of standards, methods, and tools provided by the other IVOA WGs and IGs. Typical examples are the automatic extraction of bases of knowledge from VO archives using VO ontologies; the transparent access to large computational facilities regardless the computational paradigm; the automated switching from asynchronous to synchronous mode of data access; and the extreme usage of workflows and advanced visualization methods. Furthermore, effective KD requires the possibility for an inexperienced user to contribute, or at least seamlessly use under the VO infrastructure, his/her own KD routines and methods. This situation puts strong requirements on security issues and opens new problems for ticketing and scheduling. In other words, the KD-IG will provide feedback to the solutions implemented by the WG&rsquo;s and, by posing new operational problems, will stimulate the development and adoption of new solutions and standards.

We also wish to stress that, in ultimate analysis, the goal of the KD-IG is to allow the VO to produce new scientific knowledge publishable in astronomical journals. On the one end its activities will contribute to demonstrate to the community the power and necessity of federated access to the vast VO universe of data and, on the other, KD-IG will illustrate the power and performance of data mining algorithms to facilitate and accelerate astronomical discovery within this data universe.

---++ KD-IG Meetings

| *IG Session*  | *At Meeting*  | *When*  | *Where*  |
| [[IVOA.InterOpMay2010KDD][IVOA.InterOpMay2010KDD]] | IVOA.InterOpMay2010 | May 2010 | Victoria |
| [[IVOA.InterOpMay2011KDD][IVOA.InterOpMay2011KDD]] | IVOA.InterOpMay2011 | May 2011 | Naples |
| [[IVOA.InterOpMay2016-KDD][IVOA.InterOpMay2016-KDD]] | IVOA.InterOpMay2016 | May 2016 | Cape Town |
| [[IVOA.InterOpMay2017-KDD][IVOA.InterOpMay2017-KDD]] | IVOA.InterOpMay2017 | May 2017 | Shanghai |
| [[IVOA.InterOpOct2017KDD][IVOA.InterOpOCt2017KDD]] | IVOA.InterOpOct2017 | October 2017 | Santiago |
| [[IVOA.InterOpMay2018KDIG][IVOA.InterOpMay2018KDD]] | IVOA.InterOpMay2018 | May 2018 | Victoria |
| [[IVOA.InterOpMay2019KDIG][IVOA.InterOpMay2019KDD]] | IVOA.InterOpMay2019 | May 2019 | Paris |
| [[IVOA.InterOpMay2020KDIG][IVOA.InterOpMay2020KDD]] | IVOA.InterOpMay2020 | May 2020 | Sydney virtual |
| [[IVOA.InterOpNov2021KD][IVOA.InterOpNov2021KDD]] | IVOA.InterOpNov2021 | Nov 2021 | virtual |
---++ Other Interesting Meetings for KD-IG

| *Meeting*  | *When*  | *Where*  | *Docs* |
| [[http://hea-www.harvard.edu/AstroStat/CAS2010/][Challenges and Methods for Massive Astronomical Data]]  | August 2010 | [[http://www.cfa.harvard.edu/][CfA]] | [[IvoaAstroStatslides][Slides]] |

<!-- <h2 class="TML">Next tasks</h2> <p> </p> <ol> <li> Definition of a taxonomy of Data Mining models. This taxonomy will contribute to the Standard Vocabulary of the Semantics WG2. </li> <li> Definition of the requirements which a Data Mining model needs to match in order to be imported under the VObs standards. </li> <li> Inventory of existing Data Mining models of relevant astrophysical interest. </li> <li> Definition of standard template data sets for Data Mining models test and debugging. </li> <li> Definition of standard data sets to be used as bases of knowledge for debugging and test of supervised methods. </li> <li> Definition of procedures to extract and validate robust bases of knowledge from the VObs data archives using the VObs ontology. </li> <li> Study of the scalability of Data Mining models under different computing infrastructures (definition of best benchmarks). </li> </ol>-->

---++ Related Topics

In the past hot topics had been identified. These are following the priorities emerged during the first KD-IG meeting held at the [[IVOA.InterOpMay2010KDD][IVOA.InterOpMay2010KDD]] in Victoria, and singled out by the Chair in his welcome message to the members.

Please follow the links and edit the specific pages:

   1 [[IvoaKDDDictionary][Dictionary]] of Data Mining terms;
   1 [[IvoaKDDCensus][Census]] of Data Mining and Machine Learning tools and methods of astronomical interest;
   1 [[IvoaKDDTemplate][Template]] datasets for algorithm benchmarking;
   1 A user [[IvoaKDDguide][guide]] for Knowledge Discovery in Databases in Astronomy;
   1 Knowledge Discovery in Databases and VO [[IvoaKDDstandards][standards]];
   1 Specific [[IvoaKDDfields][fields]] of applications of KDD in astronomical research;

---++ Members

*Chair:* [[RaffaeleDAbrusco][RaffaeleDAbrusco]]

<strong>Vice Chair:</strong> Yihan Tao

<br /> <!--
      * Set ALLOWTOPICRENAME = IVOA.TWikiAdminGroup
-->
This topic: IVOA > WebHome > IvoaKDD
Topic revision: r44 - 2022-09-07 - YihanTao