[FAQAS] Progress on test generation for the ASN.1 case study

Dear Thanassis,

We have applied our mutation analysis framework to the auto-generated code produced by the ASN1CC compiler, particularly the auto-generated code has been produced with the ASN.1 grammar you've shared with us during the last year.

The analysis was done in two phases, that is, phase (1) mutation analysis, and phase (2) test generation. During phase (1), we applied mutation analysis to the auto-generated code (i.e., the test.c source code), and we generated 5.323 code-driven mutants, that is, 5.323 syntactic altered versions of the original code. Through its test suite, we identified 3.104 mutants, but 2.219 mutants remain undetected, giving a mutation score of 58.13%.
During phase (2), we applied test generation to the 2.219 live mutants, and we produced test inputs that killed mutants for 1.717 of them. Considering this augmented test suite, the mutation score is now 90.57%.

Now, we started analyzing on the details the content of these 1.717 tests, but we noticed something peculiar about one of them, that I'll describe in the following:

The original mutated code is the following one and concerns the function T_INT_IsConstraintValid:

flag T_INT_IsConstraintValid(const T_INT* pVal, int* pErrCode) {
flag ret = TRUE;

(void)pVal;

ret = ((*(pVal)) <= 50UL);

*pErrCode = ret ? 0 : ERR_T_INT; // original statement
*pErrCode = ret ? -1 : ERR_T_INT; // mutated statement

return ret;

}

Our test generation tool tells us that for this case, an input value that kills the mutant is pVal=0. We found this a bit strange, since the original test suite already considers pVal=0 as an input (it also considers pVal=50).

To be sure, we also checked the test source code that invokes T_INT_IsConstraintValid:

flag T_INT_enc_dec(const T_INT* pVal, int* pErrCode, const char* filename)
{
static T_INT decodedPDU;
flag ret = TRUE;
...
// validate decoded data
ret = T_INT_IsConstraintValid(&decodedPDU, pErrCode); // pErrCode seems to be never checked
if (ret) {
ret = T_INT_Equal(pVal, &decodedPDU);
pErrCode = ret ? 0 : 4;
if (ret) {
char buf[1024];
strcpy(buf, filename);
FILE fp = fopen(strcat(buf,".dat"), "wb");
fwrite(bitStrm.buf, 1, bitStrm.count, fp);
fclose(fp);
}
}
...
}

And it seems that after the invocation of T_INT_IsConstraintValid, the value of pErrCode is never checked, and probably this is the reason why the mutant that modifies the value of pErrCode is not detected. To summarize, the test suite lacks a verification of the value contained by pErrCode (i.e., it shall check that it does not contain an error code).

Is our conclusion correct?

We will progress with analyzing the rest of the generated tests but first we would like to have some feedback from your side.

Thanks in advance,

––

Oscar Cornejo, PhD
Research Associate – SVV group
SnT – Interdisciplinary Centre for Security, Reliability and Trust

UNIVERSITÉ DU LUXEMBOURG

CAMPUS KIRCHBERG
29, avenue John F. Kennedy
L-1855 Luxembourg Kirchberg
T +352 46 66 44 5376 | oscar.cornejo@uni.lu

mail from Maxime

Maxime.Perrotin@esa.int
12:50 μ.μ. (πριν από 6 ώρες)
προς Thanassis.Tsiodras, εγώ

Hi both,

I believe that the "issue" is here:

errCode has been mutated inside IsConstraintValid, and then set again to 0 without an intermediate check, so indeed the mutation is not detected.
The point is that the "if(ret)" is the same condition that set errCode to the mutated value inside IsConstraintValid

So instead of if (ret), we should/could have if (0 == errCode && ret)

Kind regards
Maxime

ESA - European Space Agency

Maxime Perrotin
Proba-3 Plaftorm & Payload Software Engineer
TASTE Lead Engineer
System, Software and Technology Department

ESTEC
Keplerlaan 1, PO Box 299
NL-2200 AG Noordwijk, The Netherlands
maxime.perrotin@esa.int | www.esa.int
T +31 71 565 4923 | F +31 71 565 5420