I start to write documentation/specification about this new functionality.
The doc can be found in the doc/asn1-protocol directory.

The main purpose is to describe the impacts in terms of :
- modelling
- validation
- implementation

Indeed, specification of user-defined protocols would also have an impact on the underlying deployment and also the modelling of the hardware (buses, drivers and so on). This report try to detail the impact of this feature on each part.

Since Jerome would also be part of this (it will impact Ocarina and PolyORB-HI-Ada at some point), I would like to have his approval about these specifications before starting the implementation. So, please provide me a feedback about this document.

At this time, only modelling and validation are described. The actual implementation of the protocol is describe but all changes on the implementation are not yet completely specified.

I reviewed the document, I put my comments here to ease tracing and discussion

First of all, I like the approach, it is a nice addition to make Ocarina and the runtime more agnostic to the underlying constructs, so this is a nice improvement!

I just have minor comments on naming schemes used:

• Naming of data for the marshallers:

1.abstract implementation myprotocol.impl
2.subcomponents
3. marshaller: subprogram marshaller_func.impl;
4. unmarshaller: subprogram unmarshaller_func.impl;
5. associated_type: data prot_type.impl;
6.end myprotocol.impl;

the name "associated_type" is not descriptive enough, I'd rather call it request_type, packet_type or message_type (with a preference to the first one)

• Features of marshallers/unmarshallers

1.subprogram marshaller_func
2.end marshaller_func;

-> these subprograms have no parameters. This bug me, I'd rather see a parameter, referencing explicitely the C type and the ASN.1 type, like in your last schema

my fear is that if we don't, we would have to introduce special hacks in the backend to handle these subprograms differently, and add the parameters "magically". Of course, this is already what I did for the devices, so this can be considered as a minor point for now, until we clarify this pattern

• About TASTE::Protocol_Type => do we need an enum or a string?
  you opted for an enumeraiton, but I wonder whether a string is not more efficient for that, after all, it is simply an identifier to be used at TASTE-DV level (presumably), so the user may wish to opt for various names depending on encoding, packet format etc, and ASN1USER seems too restrictive for that.

Also, AFAICT, Ocarina does not process correctly the _Class property, so no checks are done. This has to be checked

Hi Jérôme,

I'm glad you like this proposal ! So, there are my comments :

1. For the name that references the request_type : this was just a suggestion, of course, we can use a name that is more explicit.

2. Subprograms of marshallers subprograms is not yet specified because ... I don't know at this point what it would be. It would depend on the changes on the implementation and so, it is still not clear for me what would be the impact on the implementation.

I'm a little bit stuck at this time because I want to keep the protocol and the driver implementations separated. In other words, keep a driver independent from a protocol so that we could use a driver with all potential protocols. On the other hand, today, we have to adapt each driver for each protocol, which lead to several problems you know as good as me !

For Protocol_Type, yes, it is better to have an enumeration than a string. Having a string will make us UML-style guys ! Having a limited range of value is always better and avoid any misconception in the model itself.

The point with protocol_type is to know how it will be used.
- if it is an enumeration, any external user who want to support a different enumerator would have to update the reference property sets
- if it is a string, there is no need to update it ..

My point is that a string would be better for future extensions by external users

But actually, we don't even need this property: you can bind the connection to the bus (implying the use of a driver) and then a virtual bus (the protocol)

So we do not even need this property (it is not used in the checks you proposed)

the only use I can see is for buildsupport or TASTE-DV to do the correct binding to the virtual bus; Ocarina would not need to process this property.

This is not used in the validation but would likely be used.

Indeed, this property is used to differentiate a user that wants to use the regular protocol and the ASN.1 generated protocol. In addition, if the protoocol is user-defined, then, we have to check that the Implemented_As is correctly assigned. Yes, I know, you'll tell me that we can consider that if the user specifies the Implemented_As, one can assume the value of the property. On my side, I think that an explicit definition is always better and specifies precisely what the user wants. In addition, it offers a way to differentiate the use of standard marshalling functions (using the traditional stack of pohic) and standard functions using ASN.1.

And I don't think we will need to extend the value. If we really need that, we can change it in Ocarina properties ... but anyway, this would also require a change in the graphical tool, which would require more efforts.

Best,

Follow-up.

I was thinking about your proposal and in fact, I think it would be useful and also avoid useless properties. Indeed, at some point, if the user want to specify the standard ASN.1 generated by TASTE, he can do that by himself and specify the Implemented_As property with appropriate values (the one that are generated). In addition, if the user want just to use the regular transport mechanism, he just have to omit to specify the Implemented_As property on the virtual bus.

So, yes, it make sense and I think we can remove the Protocol_Type property.

Thanks for your comment, I'm waiting for more comments on this document ! :)

Best,

Could you attach the document to the ticket please?

There it is. waiting for comments (and time to do the implementation !)

Follow-up,

Requested by Jerome : the models that correspond to the case study. I don't attach them to the bug report since it would be updated and it is better to keep track of changes using subversion (and models change more likely than a PDF file ...).

So, you can see the model here in the following directory of the repository : doc/asn1-protocol/model

Best,

When re-reading the models, and thinking about a potential implementation, I found multiple issues to discuss:

We have two layers, one for the driver, one for the protocol. The idea of the protocol layer is to define multiple encoding schemes for the packets to be sent/received.

Sending is easy: marshall header + payload, then send. The payload is an already marshalled ASN.1 buffer, the header is (in case of Ada) length + destination (combination of receiver thread + port). We can certainly (Julien and I) agree on a common ASN.1 scheme for this part.
We just need to send the sizeof the buffer bytes, and this is done.

Receiving is more complex: the driver needs to know some information to process the message. It needs (at least) to look for the length of the packet it will receive, for instance for serial or even TCP/IP. (Note: there is an implementation issue for which I'll open a separate ticket to discuss this). So here, the issue is to receive a partial chunk (header magic string for finding the beginning of a new packet + length) prior to receive other elements.

This means a more complex model. Not a big deal to add intermediate marshallers in the AADL model itself.

But this raises also a question on ASN.1 marshallers themselves, and would call for potential additions to asn1scc (or clarification from Thanassis).

Let me explain (Sorry for using C instead of ASN.1, I'm not that comfortable yet with it). Suppose the packet has the following structure (simplified for the discussion)

typedef struct {
  uint_32 length;;           // length of the packet
  node_t destination;     // who will receive the packet
  asn_1_buffer payload; // payload
} request_type;

Thanassis, can asn1scc generates a global marshaller/unmarshaller for request_type, and in addition marshallers/unmarshallers for each subblock?

Each marshaller for subblocks would do the following

void marshall (uint_32* length, request_type _asn_1 *request);

that is, "poke" directly in the ASN.1 marshalled buffer the corresponding value

Same for unmarshall, that would "peek" directly in the buffer the value

void unmarshall (uint_32 *length, void *buffer);

Note I use here a void* (could be unsigned char *) to imply that the unmarshaller will operate on a potentially incomplete buffer

With these functions, the driver will then use these functions to process incoming streams properly.

May you please comment on the ASN.1 part? Do not hesitate to ask for details if I'm not clear enough

See my reply here : 0000759

There shall be one ASN.1 module generated by Ocarina and listing all entity identifiers:

Entities DEFINITIONS ::=
BEGIN
    Entity-ty ::= ENUMERATED { thread1, thread2, ... }
END

Then one ASN.1 type per inter-partition communication, defining the Packet type:

Packets DEFINITIONS ::=
BEGIN
IMPORT Entity-ty FROM Entities

Packet-ty ::= SEQUENCE {
    destination  Entities.Entity,
    payload CHOICE {}
}

ASN1Scc will generate a structure similar to the one you describe, and an encoder/decoder function for Packet-ty.

The decoder will put the result in the C or Ada structure that it has generated.
The CHOICE will be a union in C (and equivalent in Ada), so you will immediately know which message it is, to send it to the receiving thread (the payload is still encoded at this level).

Does it answer you question?

Additional information:

These ASN.1 structures (Packet-ty and Entity-ty) will be generated by you (Ocarina). The only thing your code will do is fill/read this structure and send it.

The user can intervene in one case: he shall be able to provide an ACN file associated to this structure, in which case the encoding function you will call is Encode_ACN_Packet_ty instead of Encode_UPER_Packet_ty.

The ACN file will allow the user to specify a different binary encoding (e.g. PUS-compliant) for the packet: inside it it will be possible for example to change the order of the fields, to change the integer values for Entity-ty values, etc.)

This does not address my concern, let me rephrase it, in the hope it clarifies thing. If not, let us schedule a teleconf

1. When receiving, the driver needs to know how much bytes to wait for, so that it knows when a packet "ends", and avoid waiting forever (think of the serial driver)

2. packet-ty has to be optimised so that when sent, it sends only the number of required bytes (see other ticket, but I do not see why we should waste precious satellites bandwidth and energy)

3. so when receiving, the driver will do the following:

a. wait for incoming packet (polling or interrupt)
b. receive a chunk
c. check the chunk is the beginning of a packet (magic value like 0xCAFE)
d. receive the size
e. use one of the preallocated buffer
f. wait for up-to size bytes (modulo whether size includes header or not)
g. delegate the unmarshalling to the protocol, and processing to the runtime

so, what my driver needs, and that is not answered by your answers is

1. how to know when a packet starts, e.g. magic value. Not mandatory, but usually more robust
2. the size of the packet

We want to have the full packet definition in ASN.1, and I have no objection, we can have it. But I ned confirmation for more features that I actually know

My question is whether we can have

Packet-ty ::= SEQUENCE {
   size INTEGER,
   destination Entities.Entity,
   payload CHOICE {}
}

and have partial unmarshallers to

1. validate we have the beginning of a packet, not of the full packet, but only for the first few bytes (again, not mandatory, but nice to have)

2. have a peek function to know the size, to be ready to process the remaining of the buffer (this is what I called partial unmarshallers, out of an incomplete ASN.1 buffer, I want to access the size)

without size, I'll have to wait for the maximum number of bytes;
without accessors for size, or a function that tells me "OK, you've received what forms a valid ASN.1 buffer, no need to wait for more", then we would be sub-optimal

so my question is really not on what to do with a full packet, I know this part
my question is: how to know I have a full packet, ready for unmarshalling

Ok, sorry for the misunderstanding. I'll answer on your two points, with the PUS approach in mind

1) Magic value: this is not needed. The cutting into small chuncks has to be done on the payload and not on the Packet-ty.
In the PUS header there are fields that indicate how to cut/reconstruct messages.
The header part should therefore contain additional fields (packet sequence control).
The driver shall always be waiting for a header

2) on the size: you are correct. The header shall have a fixed size, including a field with the payload length.

your driver shall:

a. wait for the size of the header (small, fixed size)
b. decode the header (or only the payload size, which will be at a fixed place, the last 16 bits in case of PUS header)
c. wait for the remaining bytes
d. decode the whole packet, possibly reconstruct chunks, and send to application code

Reading back the root of this ticket:

http://code.google.com/p/buildsupport/issues/detail?id=7 [^]

I found out that Julien already implemented part of the feature in POHIC ?!

Can you clarify?

Follow-up.

Answer to 0000767

Yes, part of the feature was implemented in Ocarina. It consists in mapping the actual PO-HI-C protocol in ASN.1. But in our case, we want to provide the ability to define your own protocol, not just map a fixed protocol from C to ASN.1.

Answer to 0000764

The developer that implements the protocol knows how much data has to be received by the driver. So, we don't need a specific header. Some times, the protocol will use a fixed data size, sometimes not. It's up to the guy that implement the protocol and we let him decide how to handle that.

Then, one can wonder about how we can know the data size that has to be used. In fact, if you read the specification I proposed, each bus has a configuration table that defines : the marshallers function AND the amount of data size to be used. So, you don't need the size.

Finally, about the magic key, you can add it and it would be protocol dependent and provided by the marshallers function.

However, I have to admit that I wonder if the specifications are really useful since it seems that it is not read :/

Follow-up (again).

So, please indicate what are the problems with the proposed document and what are the problems to proceed to the implementation.

Thanks,

Regards,

I strongly disagree on the fact that you know before hand the size of the request to be received, imagine a system that may process small tcs and one giant tc, if the size is aligned on the maximum size of the messages, then we are inefficient. This is OK for small demos, this is not for realistic projects.

So, in this context of variable-length messages, when receiving a message, I'll first receive a partial chunks that I need to process to know the size of the full message. This chunk will be ASN.1 or whatever encoded. For the whatever, it is up to the representation scheme to define, and protocols like GIOP did that

For ASN.1, I need functions to peek values in a ASN.1 buffer. So Thanassis, is this possible?
Peek should be enough, to
1/ read the full size of the message (with payload) and
2/ extract the payload
3/ extract other elements like receiver thread, etc

I am trying to bring it all back to mind: re-reading the discussion above, I don't understand the comment "we cant send the maximum data size all the time": indeed we don't - as you can see in the generated "invoke_ri.c" stubs...

size_IN_buf_tc=Encode_UPER_TC_T(IN_buf_tc, sizeof(asn1SccTC_T), IN_tc);
if (-1 == size_IN_buf_tc) {
#ifdef __linux__
        printf ("** Encoding error in router_RI_displayer_put_tc!!\n");
#endif
        /* Major error, we must stop the application and let the FDIR/Watchdogs recover */
        exit (-1);
}
/* Call to VM callback function */
vm_async_router_displayer_put_tc(IN_buf_tc, size_IN_buf_tc);

...so only the ACTUAL encoded data are passed to the vm_... function.
So what is this "we send the MAX SIZE" thing? I don't understand...

In case I was not clear: The comment above (795) is just meant to say that the Encode/Decode ASN.1 functions do offer optimal space management - I dont understand why they can't be used to that effect in terms of the driver code...

Thanassis, please do not mix what happens at the application level from what happens at the transport level. We are discussing the latter, and for the implementation, we always send a message of maximum size, by adding padding if required.

from the LEON SERIAL DRIVER (po_hi_driver_leon_serial.c)

__PO_HI_MESSAGES_MAX_SIZE is a macro computed by Ocarina, it is the size of the biggest message to be sent/received, then we do that:

while (tr < __PO_HI_MESSAGES_MAX_SIZE)
{
if (read (po_hi_c_driver_leon_serial_fd_read, &(msg.content[tr]), 1) == 1)
{
tr++;
}
}

and

n = write (po_hi_c_driver_leon_serial_fd_write, &(msg.content[0]), __PO_HI_MESSAGES_MAX_SIZE);

so your answer, although valid w.r.t. to the application part is no longer valid for what happens at driver level

so again, what I'm asking for is all the more simple, and is detailed in message 764 above, and summed in message 794

The only reason I asked is because I want to understand the issue.
Since we want to pass "extra" data, of the form...

Data ::= SEQUENCE {
  payloadLength INTEGER,
  target SomeEnumeratedType,  -- chooses amongst the N communicating entities
  payload OCTET STRING (SIZEOF(...))  -- carries the actual payload data
}

If the communicating channel is not multiplexed, then we could apply
a simple separation, like this:

Header ::= SEQUENCE {
  payloadLength INTEGER,
  target SomeEnumeratedType
}
Payload ::= OCTET STRING (SIZE(...))

...and send header first, payload second, with whatever ASN.1 encoding we want.
You decode the header, and then...

while(length--) {
   read one byte into payload space for the chosen enum target  ...

}

If, instead, the communication channel IS multiplexed (so header and payload
packets may be mixed) then all we need to do is use an ACN spec, to control
the encoded bitstream, so that length and target are in fixed locations:

ASN.1 file:

MY-MODULE DEFINITIONS AUTOMATIC TAGS ::= BEGIN
  maxSize INTEGER ::= 16384

  OrbPacket ::= SEQUENCE {
    destination     DestinationID,
    payloadData     OCTET STRING (SIZE(0..maxSize))
  }

  DestinationID ::= ENUMERATED {
    dest1,
    dest2,
    dest3
  }
END

ACN file:

MY-MODULE DEFINITIONS ::= BEGIN
  OrbPacket[] {
      payloadDataLen INTEGER [encoding pos-int, size 32, endianness big],
      destination    [encoding pos-int, size 32, endianness big, encode-what indexes],
      payloadData    [size payloadDataLen]
  }
  DestinationID[]
END

If you pass the ACN spec as well to ASN1SCC (use the -ACN flag), the receiving code can just do:

Receive first 4 bytes - payload length, guaranteed to be in big-endian
Receive next  4 bytes - target enum, guaranteed to be in big-endian
while(length--) {
    read one byte into payload space for the chosen enum target  ...
}

Jerome : you don't know the size in advante BUT the guy that implements the protocol knows how much data it should received. In other words, ASN.1 will provide data structures and programming facilities to marshall the packets but after, this is the responsability of the programmer/designer to :
1. Specify how much data should be received/sent at each communication operating
2. Enable the mapping between the protocol-level data and PolyORB data. That is exactly why we also map AADL entities into ASN.1 : by using it, the developer can make by himself the translation between protocol-level values and aadl/pohic-values -for example, the mapping between one protocol-level id and one task/port).

Hope that I'm clear.

Best,

Thanassis answer from message 798 is exactly the information I was asking for. Thanks
so now, we need to define the ASN.1 and ACN precisely to continue on the modeling part, and see the code we can use for the processing of the first bytes as depcited by Thanassis

I think there is a totally misunderstanding : there is no predefined ACN or ASN.1 model that has to be defined : the purpose and the scope of this bug is to let the user defines by himself its own ASN.1 and ACN model to implement its own protocol.

the understanding is clear: simply replace "the" by "a" (the joy of gallicism ..)

without an initial model we cannot start; and we already have some elements from the various exchanges we got so far. So let us start prototyping work by defining a simple ASN.1 file, and see if we have all elements to converge

ergghh .... the configuration of polyorb is already proposed in the specs ...

So, you can just add :

Payload-T ::= BIT STRING (0 .. 5000)

Packet-T ::= SEQUENCE {
outport Port-T,
payload Payload-T
}

And then you are !

A first shot has been made to integrate not only ASN.1 protocols but all user-defined protocols. It can of course includes protocol defined by the user and that use ASN.1.

Please look at misc/experiments/protocol-asn1 to see an example with a very basic and simple protocol that just send application data directly on the network.

All comments are welcome !

I don't really understand how this will work with TASTE, what to define where, etc.

Do we need a new property on the bus component in the deployment view to specify which protocol we want to use on a given inter-node link?

Then what information do we need to capture at model level?

(The example is in pure AADL form - difficult to understand/map to TASTE)

So, this was designed and added in PolyORB-HI-C, I assign this ticket to Jerome, waiting that it is implemented in the Ada runtime.

Thanks for him, but this does not answer the question :-)

What should we do/add in TASTE to support this feature?

Do you need any help to progress with this issue ?

No, unless you have a time machine ;)
The implementation is underway, but it requires severe refactoring all along the place, hence this delay

In the meantime, Jerome, please tell me if you see any defect in the implementation.

Then, for Maxime, you have to consider that integrating user-defined bus would not be so difficult ... but will require modification in the graphical tools for sure ... (at least to specify the protocol implementation ...).

No update since 2011, closing ticket