TY - GEN
T1 - DeepCT
T2 - 26th IEEE International Conference on Software Analysis, Evolution, and Reengineering, SANER 2019
AU - Ma, Lei
AU - Juefei-Xu, Felix
AU - Xue, Minhui
AU - Li, Bo
AU - Li, Li
AU - Liu, Yang
AU - Zhao, Jianjun
N1 - Funding Information:
ACKNOWLEDGEMENTS This work was partially supported by 973 Program (No. 2015CB352203), Fundamental Research Funds for the Central Universities (No. AUGA5710000816) of China, and JSPS KAKENHI Grant 18H04097. We gratefully acknowledge the support of NVIDIA AI Tech Center (NVAITC) to our research.
Publisher Copyright:
© 2019 IEEE.
PY - 2019/3/15
Y1 - 2019/3/15
N2 - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.
AB - Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.
UR - http://www.scopus.com/inward/record.url?scp=85064176909&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85064176909&partnerID=8YFLogxK
U2 - 10.1109/SANER.2019.8668044
DO - 10.1109/SANER.2019.8668044
M3 - Conference contribution
AN - SCOPUS:85064176909
T3 - SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering
SP - 614
EP - 618
BT - SANER 2019 - Proceedings of the 2019 IEEE 26th International Conference on Software Analysis, Evolution, and Reengineering
A2 - Shihab, Emad
A2 - Lo, David
A2 - Wang, Xinyu
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 24 February 2019 through 27 February 2019
ER -