Technical topic: Advanced testing with MSC and Python scripts
Introduction
This section describes the possibilities offered by TASTE to automate the validation of a system using the full power of ASN.1, Python, and the MSC (Message Sequence Charts).
The general process offered for the validation of a system is the following:
1) Create a TASTE system that contains at least one GUI component. GUIs are automatically generated during the build process and provide and interface allowing to communicate with the main running binary (your application), via message queues or sockets.
2) Run your application and the GUI. The GUI allows to plot data in real-time, and to record traces of execution using sequence diagrams (MSCs).
3) Record a simple scenario to get the template of a test script for your system.
4) Possibly edit and refine your scenario using the TASTE MSC Editor
5) Run the system again and execute the recorded scenario - it will check that the system behaves exactly the same way (regression testing)
6) Go further than MSC - use the power of Python to automate the validation
The Python API provided by TASTE allows powerful checks to be automated on the running system. You can execute several scenario in parallel, record data, wait for explicit messages and check or ignore parameters, etc.
Features
Check the case study below to get the full picture, with a concrete example.
The Python API for testing allows to:
- Run different scenarii (test cases) in parallel. For example, one scenario can be used to check the reception of periodic data (housekeeping, monitorings), while ignoring other protocol information, while another scenario can control the initialization and sending of control and acknoledgment data... Possibilites are infinite, and depend on the complexity of your system in terms of external interface and protocol.
Scenarios are implemented using a Python decorator - making the code light and easy to read:
@Scenario def MyScenario(queue): """ Send periodic messages - Can run in parallel with other scenarii """ for i in xrange(10): queue.sendMsg('runstep', '{ FALSE, TRUE }') time.sleep(1)
- Send a message to the running system, using the ASN.1 value notation:
queue.sendMsg('message_name', '{ name "Maxime", age 35 }') # Use ASN.1 value notation for the message content
- Wait for a specific message and parameters. Parameters are optional and can be checked or not. It is possible to wait for a message while ignoring other messages, or raise an error if another message is received in place of the one that is expected.
queue.expectMsg('Hello', '{ name *, age 35 }') -> Must receive the "Hello" message with the parameter "age" having value 35. Name is not checked. queue.expectMsg('Hello', '*', ignoreOther=True) -> Wait until it received "Hello", whatever the parameters
- Wait for any message, leaving the user the responsibility for checking the content
(msgId, val) = queue.getNextMsg(timeout=10) if msgId == 'Hello': print 'My name is', val.name.Get() # Note the Python/ASN.1 API that allows to access to fields as if there were parameters
Case Study
This short tutorial explains how to use the TASTE validation features on a concrete example.
- Creation of a new TASTE system
$ taste # or space-creator
Fill the data view with this simple ASN.1 model:
TASTE-Dataview DEFINITIONS ::= BEGIN MyInteger ::= INTEGER (0..255) MyTest ::= SEQUENCE (SIZE(2)) OF BOOLEAN END
Select the language GUI for Function2. For this example, we set language SDL for Function0 and Ada for Function1. What counts is the interface with the GUI (Function2). We have defined three sporadic interfaces:
runStep, with a parameter of type MyTest
result, with a parameter of type MyTest
counter, with a parameter of type MyInteger
- Implementation of the user code for Function0
Right-click on "Function0" and select "Open SDL editor"
We made a simple SDL state machine using TASTE internal editor OpenGEODE:
Save and quit, the Ada code will be generated automatically.
- Implementation of the user code for Function1
Right-click on "Function1" and choose "Edit Ada code"
Fill in the blanks:
package body function1 is procedure ada(outparam: access asn1sccMyTest) is begin outparam.all := asn1SccMyTest'(Data => asn1SccMyTest_array'(1 => false, 2 => true)); end ada; end function1;
Save and quit (back to the Interface View editor)
- Create a deployment view and build the system
From the menu Tools->Edit Deployment View
Create a single processor board, choose Linux32 target, and bind the 3 functions to it; name the partition "demo", it will be the name of the binary.
Then exit, to get back to the Interface View.
- Build the system
Menu Tools->Build the system (C or Ada runtime, it does not make any difference)
At the end of the build, exit the interface view editor and go to the binary directory:
$ cd binary.c/binaries # or binary.ada/binaries
- Run the system: first the main binary, then the GUI:
$ ./demo & $ ./GUI-function2
The generated GUI will open:
Click on the MSC button on the top of the window to record a trace of the scenario. You can interact with the system, and plot numerical data. Send at least one "runStep" message, to get an MSC that you can later edit and generate a Python script from.
- Quit the GUI and run it again
$ ./GUI-function2
You will see the previously recorded scenario. Click on "Run" to execute it. It should display "Test case successful". Click on "Edit" to open the MSC editor. You can modify the sceneario - and run it again.
The execution is checked against the MSC data. If received messages are different from specified in the MSC, the test case will be considered failed.
When you run the MSC it is first converted to a Python script, that you can edit at will.
You can run your modified Python test scripts either from the GUI (using the Open button) or from the command line.
The following is an example of a complete test script, showing the execution of two scenarii in parallel, one sending periodic data to the main binary, while the other one is consuming it.
#!/usr/bin/env python # # Automatically generated Python sequence chart (MSC) implementation import os import sys import signal import time taste_inst = os.popen('taste-config --prefix').readlines()[0].strip() sys.path.append(taste_inst+'/share/asn1-editor') from Scenario import Scenario, PollerThread from PySide.QtCore import QCoreApplication, Qt from udpcontroller import tasteUDP # To plot the result after all scenarios are completed, use Matplotlib import matplotlib.pyplot as plt status = 0 function2 = None parallel = None poller = None @Scenario def Exercise_function2(self): """ First scenario """ global status plotData = [] # First wait for an explicit message if -1 == self.expectMsg('result', ' { FALSE , TRUE }', lineNo=12, ignoreOther=False): print 'FAILED' status = -1 return status while True: # Wait for any message, and print it (msgId, val) = self.getNextMsg(timeout = 10) if msgId: print 'Received', msgId val.PrintAll() if msgId == 'counter': # Save Data for further post-processing plotData.append(val.Get()) else: print 'timeout... not received anything...' # End of the scenario: Plot the recorded data plt.plot(plotData) plt.ylabel('counter value') plt.show() print plotData return 0 @Scenario def ParallelScenario(self): """ Run in parallel with the other scenario - send periodic messages """ for i in xrange(10): self.sendMsg('runstep', '{ FALSE, TRUE }') time.sleep(1) def runScenario(udpController=None, callback=None): """ Function handling the execution of the scenarios """ global function2 global parallel global poller poller = None poller = PollerThread() function2 = Exercise_function2() parallel = ParallelScenario() if callback: function2.done.connect(callback) parallel.done.connect(callback) poller.slots.append(function2.Q) poller.start() function2.start() parallel.start() def killThreads(): """ Function called just before the application ends.. Reporting and cleaning threads """ function2.wait() parallel.wait() poller._bDie = True poller.wait() if status != 0: print '[ERROR] At least one scenario failed...' if __name__ == "__main__": """ For CLI execution (when not called from the GUI). Much better performance (no GUI to update) """ signal.signal(signal.SIGINT, signal.SIG_DFL) udpController = None app = QCoreApplication(sys.argv) app.aboutToQuit.connect(killThreads) runScenario(udpController, app.quit) app.exec_() sys.exit(status)