One-sentence Summary

The authors propose OSS PESTO, an open-source tool that addresses the industry gap in framework-supported evaluation by integrating real-time GitHub data collection with customized evaluation using selected OSS models to provide practitioners with intuitive guidance for comparing and selecting suitable open source software.

Key Contributions

• The paper introduces OSS PESTO, an open source software project evaluation and selection tool that enables practitioners to compare GitHub repository candidates using established assessment models.
• The system automates data collection from GitHub repositories and integrates customizable evaluation frameworks with configurable weight functions to improve control over internal and external product quality.
• This GitHub-oriented implementation provides a customization-friendly architecture that supports direct candidate comparison and model adjustment to streamline evaluation workflows for industry and academic users.

Introduction

Open source software has become foundational to modern IT infrastructure, yet the massive scale of repositories on platforms like GitHub makes selecting the right libraries and frameworks a complex challenge for developers and organizations. While academic literature offers numerous evaluation frameworks, most existing tools suffer from rigid architectures, limited customization options, discontinued maintenance, and a lack of seamless integration with live repository metrics. To bridge this gap, the authors introduce OSS PESTO, an open-source evaluation platform that automates real-time GitHub data collection and enables practitioners to apply customizable, model-driven scoring criteria. This tool streamlines the OSS adoption workflow by combining flexible assessment parameters with reliable, locally cached data, ultimately empowering technical teams to make faster and more informed software decisions.

Method

The authors leverage a modular architecture for OSS PESTO, designed to support a systematic evaluation of open-source software (OSS) projects using data from GitHub. The framework is structured around three primary modules: a data crawler, a server, and a client, which together enable the identification, evaluation, and display of OSS candidates. As shown in the figure below, the process begins with candidate identification, where the data crawler module extracts repository data from GitHub via its APIs. This module allows users to specify target projects or select repositories within a defined range of stars, facilitating the acquisition of relevant datasets. The collected data is stored locally in a CSV format, with each row representing an OSS candidate and its associated metrics.

Following data collection, the server module processes the dataset using a configurable evaluation model. This model is defined in a JSON configuration file, which users can modify to select specific evaluation factors and metrics based on their preferences. The server integrates with a MongoDB database to manage data storage and employs ReactJS for backend implementation. The configuration file maps evaluation categories to data features, enabling dynamic customization of the evaluation process. For instance, a user can define a category such as "Popularity" and associate it with metrics like the number of watchers or stars by specifying corresponding header and accessor fields.

The client module, also built using ReactJS, is responsible for presenting the evaluation results to the user. It displays the candidate OSS projects along with the selected attributes and comparison outcomes. The client renders the data in a structured format, allowing users to compare projects across the chosen metrics. The configuration file directly influences the client's display, ensuring that only the relevant metrics and categories are presented. The interaction flow proceeds as follows: the data crawler identifies candidates, the server processes the data according to the selected model and user preferences, and the client visualizes the evaluation outcomes for comparison.

Experiment

The experiments validate the applicability of the OSS PESTO framework by analyzing the community dynamics, support structures, and technical attributes of three JavaScript frameworks using GitHub data. Qualitative results highlight distinct operational profiles, with Vue demonstrating higher popularity, Angular exhibiting broader community engagement and organizational support, and Redux maintaining a leaner technical footprint with fewer dependencies. Additionally, applying alternative evaluation models to the identical dataset yields consistent comparative insights, confirming the tool's reliability and adaptability for standardized open-source software assessment.

The authors conduct an experiment to evaluate three JavaScript frameworks using the OSSPAL model, focusing on community, support, and software technology attributes. The results show differences in popularity, activity, and technical characteristics among the frameworks, with Vue having the highest watch and star counts, Angular showing higher engagement in issue discussions, and Redux demonstrating fewer dependencies and open issues. Vue is the most popular framework in terms of watches and stars compared to the others. Angular has higher engagement in issue-related activities such as comments and pull requests. Redux has fewer dependencies and open issues compared to the other frameworks.

The study employs the OSSPAL model to comparatively evaluate three JavaScript frameworks across community dynamics, support structures, and technical architectures. Qualitative assessment reveals distinct operational profiles for each ecosystem, with Vue demonstrating the strongest overall popularity and user interest. Angular exhibits superior collaborative engagement through active issue tracking and contribution workflows. Meanwhile, Redux stands out for its streamlined technical footprint, maintaining significantly fewer external dependencies and unresolved issues than its counterparts.