Enabling Large-Scale Mining Software Repositories (MSR) Studies Using Web-Scale Platforms

The Mining Software Repositories (MSR) field analyzes software data to uncover knowledge and assist software developments. Software projects and products continue to grow in size and complexity. In-depth analysis of these large systems and their evolution is needed to better understand the characteristics of such large-scale systems and projects. However, classical software analysis platforms (e.g., Prolog-like, SQL-like, or specialized programming scripts) face many challenges when performing large-scale MSR studies. Such software platforms rarely scale easily out of the box. Instead, they often require analysis-specific one-time ad hoc scaling tricks and designs that are not reusable for other types of analysis and that are costly to maintain. We believe that the web community has faced many of the scaling challenges facing the software engineering community, as they cope with the enormous growth of the web data. In this thesis, we report on our experience in using MapReduce and Pig, two web-scale platforms, to perform large MSR studies. Through our case studies, we carefully demonstrate the benefits and challenges of using web platforms to prepare (i.e., Extract, Transform, and Load, ETL) software data for further analysis. The results of our studies show that: 1) web-scale platforms provide an effective and efficient platform for large-scale MSR studies; 2) many of the web community’s guidelines for using web-scale platforms must be modified to achieve the optimal performance for large-scale MSR studies. This thesis will help other software engineering researchers who want to scale their studies.

Department dc:contributor.department

Computing

Year dc:date.issued

2010

Author dc:creator

• Shang, Weiyi

Advisor dc:contributor.supervisor

• Hassan, Ahmed E.

• Software Engineering
• Mining Software Repositories
• MapReduce
• Hadoop
• Pig

Language dc:language.iso

eng

Handle dc:identifier.uri

http://hdl.handle.net/1974/5693

OAI identifier oai:identifier

oai:queensu.scholaris.ca:1974/5693