A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

Maryam Vahdat Pour, Zhuo Li, Lei Ma, Hadi Hemmati

Research output: Chapter in Book/Report/Conference proceedingConference contribution

24 Citations (Scopus)

Abstract

Over the past few years, deep neural networks (DNNs) have been continuously expanding their real-world applications for source code processing tasks across the software engineering domain, e.g., clone detection, code search, comment generation. Although quite a few recent works have been performed on testing of DNNs in the context of image and speech processing, limited progress has been achieved so far on DNN testing in the context of source code processing, that exhibits rather unique characteristics and challenges.In this paper, we propose a search-based testing framework for DNNs of source code embedding and its downstream processing tasks like Code Search. To generate new test inputs, we adopt popular source code refactoring tools to generate the semantically equivalent variants. For more effective testing, we leverage the DNN mutation testing to guide the testing direction. To demonstrate the usefulness of our technique, we perform a large-scale evaluation on popular DNNs of source code processing based on multiple state-of-the-art code embedding methods (i.e., Code2vec, Code2seq and CodeBERT). The testing results show that our generated adversarial samples can on average reduce the performance of these DNNs from 5.41% to 9.58%. Through retraining the DNNs with our generated adversarial samples, the robustness of DNN can improve by 23.05% on average. The evaluation results also show that our adversarial test generation strategy has the least negative impact (median of 3.56%), on the performance of the DNNs for regular test data, compared to the other methods.

Original languageEnglish
Title of host publicationProceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages36-46
Number of pages11
ISBN (Electronic)9781728168364
DOIs
Publication statusPublished - Apr 2021
Externally publishedYes
Event14th IEEE International Conference on Software Testing, Verification and Validation, ICST 2021 - Virtual, Porto de Galinhas, Brazil
Duration: Apr 12 2021Apr 16 2021

Publication series

NameProceedings - 2021 IEEE 14th International Conference on Software Testing, Verification and Validation, ICST 2021

Conference

Conference14th IEEE International Conference on Software Testing, Verification and Validation, ICST 2021
Country/TerritoryBrazil
CityVirtual, Porto de Galinhas
Period4/12/214/16/21

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence
  • Software
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding'. Together they form a unique fingerprint.

Cite this