Simulation testing is considered the supplement to ensure the safety of Autonomous driving (AD) and advanced driving assistance (ADAS's) systems in terms of time and costs. However, it is very difficult and challenging when the simulation results are unexpected. This work presents a simulation-based metamorphic testing (MT) approach to test the ADAS system, implementing the European new car assessment program (Euro NCAP) standards on OpenStreetMap (OSM). We first defined input patterns and relations related to autonomous driving, following the principles of MT. To assess the approach, we executed three tests in two steps at both the design and system levels. Our results show that none of the three (source) tests detected any collisions. However, for follow-up test cases, the ego vehicle failed to apply brakes to avoid a collision when the speed changed. A real-life issue in the system was immediately revealed and confirmed by the development team. We then designed a mechanism and
The testing of large databases faces the test oracle problem, namely, that it is difficult to verify execution results against expected outcomes. Rigger and Su applied metamorphic testing through query partitioning and ternary logic partitioning techniques to alleviate the challenge. In Part (A) of our project, we conduct an in-depth investigation and have identified a gap between the two techniques. We propose a disjoint partitioning approach to address it. In Part (B), we conduct a comprehensive investigation into the metamorphic testing of DBMS by comparing disjoint partitioning with metamorphic relation output patterns (MROPs) by Segura et al. We propose an exhaustive collection of MROPs for DBMS. To the best of our knowledge, this is the first project to integrate in-depth and comprehensive approaches to tackle the diverse challenges in DBMS testing. In Part (C), we conduct an empirical case study of their applications to OceanBase, the DBMS associated with the world's fastes
Concerns have been growing over fake news and its impact. Software that can automatically detect fake news is becoming more popular. However, the accuracy and reliability of such fake-news detection software remains questionable, partly due to a lack of testing and verification. Testing this kind of software may face the oracle problem, which refers to difficulty (or inability) of identifying the correctness of the software's output in a reasonable amount of time. Metamorphic testing (MT) has a record of effectively alleviating the oracle problem, and has been successfully applied to testing fake-news detection software. This paper reports on a study, extending previous work, exploring the use of MT for fake-news detection software. The study includes new metamorphic relations and additional experimental results and analysis. Some alternative MR-generation approaches are also explored. The study targets software where the output is a real/fake news decision, enhancing the applicab
Owing to big data, DBMS testing faces the oracle problem, that is, it is difficult to verify execution results against expected outcomes. Rigger and Su applied metamorphic testing to alleviate the challenge. We propose a disjoint-partitioning approach to extend their work. We have conducted an empirical case study on OceanBase, the DBMS associated with the world's fastest online transaction processing system. Even though Ocean- Base has been extensively tested and widely used in the industry, we have unveiled various hidden failures and crashes.
Context: Simulation-based testing is essential when developing Advanced Driver-Assistance Systems (ADASs) and autonomous driving (AD) systems, producing fast, high-quality test results, at relatively low cost. Simulation testing relies on the quality of the ADAS simulation platform: If the simulation platform is faulty, then the simulation results may be incorrect, and hence useless. However, because of the lack of suitable test oracles mechanisms to determine the correctness of the software output or behavior it can be too difficult (or expensive) to verify or validate ADAS/AD simulation platforms, a situation known as the oracle problem. Objective: To alleviate the oracle problem and better understand ADAS simulation software. Methods: We develop geometric-transformation-based metamorphic testing approaches, and report on empirical studies conducted on the verification and validation (V&V) of three popular simulation platforms for ADAS development: Simulink, CarMaker and 51Si