Technical topic: Advanced testing with MSC and Python scripts

From TASTE
Jump to: navigation, search

Introduction

This section describes the possibilities offered by TASTE to automate the validation of a system using the full power of ASN.1, Python, and the MSC (Message Sequence Charts).

The general process offered for the validation of a system is the following:

1) Create a TASTE system that contains at least one GUI component. GUIs are automatically generated during the build process and provide and interface allowing to communicate with the main running binary (your application), via message queues or sockets.

2) Run your application and the GUI. The GUI allows to plot data in real-time, and to record traces of execution using sequence diagrams (MSCs).

3) Record a simple scenario to get the template of a test script for your system.

4) Possibly edit and refine your scenario using the TASTE MSC Editor

5) Run the system again and execute the recorded scenario - it will check that the system behaves exactly the same way (regression testing)

6) Go further than MSC - use the power of Python to automate the validation

The Python API provided by TASTE allows powerful checks to be automated on the running system. You can execute several scenario in parallel, record data, wait for explicit messages and check or ignore parameters, etc.

Features

Check the case study below to get the full picture, with a concrete example.

The Python API for testing allows to:

  • Run different scenarii (test cases) in parallel. For example, one scenario can be used to check the reception of periodic data (housekeeping, monitorings), while ignoring other protocol information, while another scenario can control the initialization and sending of control and acknoledgment data... Possibilites are infinite, and depend on the complexity of your system in terms of external interface and protocol.

Scenarios are implemented using a Python decorator - making the code light and easy to read:

   @Scenario
   def MyScenario(queue):
       """ Send periodic messages - Can run in parallel with other scenarii """
       for i in xrange(10):
           queue.sendMsg('runstep', '{ FALSE, TRUE }')
           time.sleep(1)


  • Send a message to the running system, using the ASN.1 value notation:
   queue.sendMsg('message_name', '{ name "Maxime", age 35 }')   # Use ASN.1 value notation for the message content
  • Wait for a specific message and parameters. Parameters are optional and can be checked or not. It is possible to wait for a message while ignoring other messages, or raise an error if another message is received in place of the one that is expected.
   queue.expectMsg('Hello', '{ name *, age 35 }')   -> Must receive the "Hello" message with the parameter "age" having value 35. Name is not checked.
   queue.expectMsg('Hello', '*', ignoreOther=True)  -> Wait until it received "Hello", whatever the parameters
  • Wait for any message, leaving the user the responsibility for checking the content
   (msgId, val) = queue.getNextMsg(timeout=10)   
   if msgId == 'Hello':
       print 'My name is', val.name.Get()  # Note the Python/ASN.1 API that allows to access to fields as if there were parameters

Case Study

This short tutorial explains how to use the TASTE validation features on a concrete example.

  • Creation of a data view
$ taste-create-data-view

Fill the data view with this simple ASN.1 model:

TASTE-Dataview DEFINITIONS ::=
BEGIN
  MyInteger ::= INTEGER (0..255)
  MyTest ::= SEQUENCE (SIZE(2)) OF BOOLEAN
END

Save and quit.

  • Creation of an interface view
$ taste-create-interface-view
TASTE Interface View

Select the language GUI for Function2. For this example, we set language SDL for Function0 and Ada for Function1. What counts is the interface with the GUI (Function2). We have defined three sporadic interfaces:

runStep, with a parameter of type MyTest

result, with a parameter of type MyTest

counter, with a parameter of type MyInteger


  • Implementation of the user code for Function0

Right-click on "Function0" and select "Open SDL editor"

We made a simple SDL state machine using TASTE internal editor OpenGEODE:

Opengeode

Save and quit, the Ada code will be generated automatically.

  • Implementation of the user code for Function1

Right-click on "Function1" and choose "Edit Ada code"

Fill in the blanks:

package body function1 is
    procedure ada(outparam: access asn1sccMyTest) is
        begin
            outparam.all := asn1SccMyTest'(Data => asn1SccMyTest_array'(1 => false, 2 => true));
        end ada;
end function1;


Save and quit (back to the Interface View editor)

  • Create a deployment view and build the system

From the menu Tools->Edit Deployment View

Create a single processor board, choose Linux32 target, and bind the 3 functions to it; name the partition "demo", it will be the name of the binary.

Then exit, to get back to the Interface View.

  • Build the system

Menu Tools->Build the system (C or Ada runtime, it does not make any difference)

At the end of the build, exit the interface view editor and go to the binary directory:

$ cd binary.c/binaries     # or binary.ada/binaries
  • Run the system: first the main binary, then the GUI:
$ ./demo &
$ ./GUI-function2

The generated GUI will open:

Generated GUI

Click on the MSC button on the top of the window to record a trace of the scenario. You can interact with the system, and plot numerical data. Send at least one "runStep" message, to get an MSC that you can later edit and generate a Python script from.

  • Quit the GUI and run it again
$ ./GUI-function2

You will see the previously recorded scenario. Click on "Run" to execute it. It should display "Test case successful". Click on "Edit" to open the MSC editor. You can modify the sceneario - and run it again.

The execution is checked against the MSC data. If received messages are different from specified in the MSC, the test case will be considered failed.

When you run the MSC it is first converted to a Python script, that you can edit at will.

You can run your modified Python test scripts either from the GUI (using the Open button) or from the command line.

The following is an example of a complete test script, showing the execution of two scenarii in parallel, one sending periodic data to the main binary, while the other one is consuming it.

#!/usr/bin/env python
#
# Automatically generated Python sequence chart (MSC) implementation

import os
import sys
import signal
import time
taste_inst = os.popen('taste-config --prefix').readlines()[0].strip()
sys.path.append(taste_inst+'/share/asn1-editor')

from Scenario import Scenario, PollerThread
from PySide.QtCore import QCoreApplication, Qt
from udpcontroller import tasteUDP

# To plot the result after all scenarios are completed, use Matplotlib
import matplotlib.pyplot as plt

status = 0
function2 = None
parallel = None
poller = None

@Scenario
def Exercise_function2(self):
    """ First scenario """
    global status
    plotData = []
    # First wait for an explicit message
    if -1 == self.expectMsg('result', ' { FALSE , TRUE }', lineNo=12, ignoreOther=False):
        print 'FAILED'
        status = -1
        return status
    while True:
        # Wait for any message, and print it 
        (msgId, val) = self.getNextMsg(timeout = 10)
        if msgId:
            print 'Received', msgId
            val.PrintAll()
            if msgId == 'counter':
                # Save Data for further post-processing
                plotData.append(val.Get())
        else:
            print 'timeout... not received anything...'
            # End of the scenario: Plot the recorded data
            plt.plot(plotData)
            plt.ylabel('counter value')
            plt.show()
            print plotData
            return 0

@Scenario
def ParallelScenario(self):
    """ Run in parallel with the other scenario - send periodic messages """
    for i in xrange(10):
        self.sendMsg('runstep', '{ FALSE, TRUE }')
        time.sleep(1)


def runScenario(udpController=None, callback=None):
    """ Function handling the execution of the scenarios """
    global function2
    global parallel
    global poller
    poller = None
    poller = PollerThread()
    
    function2 = Exercise_function2()
    parallel = ParallelScenario()

    if callback:
        function2.done.connect(callback)
        parallel.done.connect(callback)

    poller.slots.append(function2.Q)
    poller.start()
    function2.start()
    parallel.start()


def killThreads():
    """ Function called just before the application ends.. Reporting and cleaning threads """
    function2.wait()
    parallel.wait()
    poller._bDie = True
    poller.wait()
    if status != 0:
        print '[ERROR] At least one scenario failed...'

if __name__ == "__main__":
    """ For CLI execution (when not called from the GUI). Much better performance (no GUI to update) """
    signal.signal(signal.SIGINT, signal.SIG_DFL)
    udpController = None
    app = QCoreApplication(sys.argv)
    app.aboutToQuit.connect(killThreads)
    runScenario(udpController, app.quit)
    app.exec_()
    sys.exit(status)