Anonymous | Login | Signup for a new account | 2022-06-30 04:50 UTC | ![]() |
Main | My View | View Issues | Change Log | Roadmap |
View Issue Details [ Jump to Notes ] | [ Issue History ] [ Print ] | ||||||||
ID | Project | Category | View Status | Date Submitted | Last Update | ||||
0000004 | Taste | [All Projects] General | public | 2010-08-05 08:24 | 2021-12-02 10:07 | ||||
Reporter | maxime | ||||||||
Assigned To | hugues | ||||||||
Priority | high | Severity | feature | Reproducibility | N/A | ||||
Status | closed | Resolution | fixed | ||||||
Platform | OS | OS Version | |||||||
Summary | 0000004: Use ASN.1 for packet encoding in Polyorb | ||||||||
Description | I would like to propose a way to improve the marshalling process in In essence the problem is the following: We need a packet format that is network and processor-neutral (endianess- At the moment this is something that you do manually in PolyORB:
Your functions take care of alignments and endianess. I think this whole packet building should be delegated to ASN.1, since - What I propose is that for each concurrency view, buildsupport also
When you give this file to the ASN.1 Compiler (asn1scc), it produces 1) a C or Ada data structure that corresponds to the Packet type, and a C 2) one function called "Encode_Packet (Packet *p, char *OutBuffer) " and This is exactly what is needed by PolyORB prior to sending the message to PolyORB code could be adapted to fill the fields of the data structure Now see the benefits: a) No risk of endianess issues (as we have now) (b) is critical, if we want to support the TM/TC format. We can use ACN to | ||||||||
Tags | No tags attached. | ||||||||
Attached Files | ![]() | ||||||||
![]() |
|
(0000695) user2 2011-05-12 15:41 |
I start to write documentation/specification about this new functionality. The main purpose is to describe the impacts in terms of : Indeed, specification of user-defined protocols would also have an impact on the underlying deployment and also the modelling of the hardware (buses, drivers and so on). This report try to detail the impact of this feature on each part. Since Jerome would also be part of this (it will impact Ocarina and PolyORB-HI-Ada at some point), I would like to have his approval about these specifications before starting the implementation. So, please provide me a feedback about this document. At this time, only modelling and validation are described. The actual implementation of the protocol is describe but all changes on the implementation are not yet completely specified. |
(0000696) hugues (administrator) 2011-05-13 10:44 |
I reviewed the document, I put my comments here to ease tracing and discussion First of all, I like the approach, it is a nice addition to make Ocarina and the runtime more agnostic to the underlying constructs, so this is a nice improvement! I just have minor comments on naming schemes used:
1.abstract implementation myprotocol.impl the name "associated_type" is not descriptive enough, I'd rather call it request_type, packet_type or message_type (with a preference to the first one)
1.subprogram marshaller_func -> these subprograms have no parameters. This bug me, I'd rather see a parameter, referencing explicitely the C type and the ASN.1 type, like in your last schema my fear is that if we don't, we would have to introduce special hacks in the backend to handle these subprograms differently, and add the parameters "magically". Of course, this is already what I did for the devices, so this can be considered as a minor point for now, until we clarify this pattern
Also, AFAICT, Ocarina does not process correctly the _Class property, so no checks are done. This has to be checked |
(0000697) user2 2011-05-13 11:47 edited on: 2016-11-21 12:59 |
Hi Jérôme, I'm glad you like this proposal ! So, there are my comments :
I'm a little bit stuck at this time because I want to keep the protocol and the driver implementations separated. In other words, keep a driver independent from a protocol so that we could use a driver with all potential protocols. On the other hand, today, we have to adapt each driver for each protocol, which lead to several problems you know as good as me ! For Protocol_Type, yes, it is better to have an enumeration than a string. Having a string will make us UML-style guys ! Having a limited range of value is always better and avoid any misconception in the model itself. |
(0000698) hugues (administrator) 2011-05-13 14:00 |
The point with protocol_type is to know how it will be used. My point is that a string would be better for future extensions by external users But actually, we don't even need this property: you can bind the connection to the bus (implying the use of a driver) and then a virtual bus (the protocol) So we do not even need this property (it is not used in the checks you proposed) the only use I can see is for buildsupport or TASTE-DV to do the correct binding to the virtual bus; Ocarina would not need to process this property. |
(0000700) user2 2011-05-13 15:22 |
This is not used in the validation but would likely be used. Indeed, this property is used to differentiate a user that wants to use the regular protocol and the ASN.1 generated protocol. In addition, if the protoocol is user-defined, then, we have to check that the Implemented_As is correctly assigned. Yes, I know, you'll tell me that we can consider that if the user specifies the Implemented_As, one can assume the value of the property. On my side, I think that an explicit definition is always better and specifies precisely what the user wants. In addition, it offers a way to differentiate the use of standard marshalling functions (using the traditional stack of pohic) and standard functions using ASN.1. And I don't think we will need to extend the value. If we really need that, we can change it in Ocarina properties ... but anyway, this would also require a change in the graphical tool, which would require more efforts. Best, |
(0000702) user2 2011-05-13 20:28 |
Follow-up. I was thinking about your proposal and in fact, I think it would be useful and also avoid useless properties. Indeed, at some point, if the user want to specify the standard ASN.1 generated by TASTE, he can do that by himself and specify the Implemented_As property with appropriate values (the one that are generated). In addition, if the user want just to use the regular transport mechanism, he just have to omit to specify the Implemented_As property on the virtual bus. So, yes, it make sense and I think we can remove the Protocol_Type property. Thanks for your comment, I'm waiting for more comments on this document ! :) Best, |
(0000728) maxime (administrator) 2011-05-26 13:02 |
Could you attach the document to the ticket please? |
(0000735) user2 2011-05-26 13:29 |
There it is. waiting for comments (and time to do the implementation !) |
(0000739) user2 2011-05-27 16:09 |
Follow-up, Requested by Jerome : the models that correspond to the case study. I don't attach them to the bug report since it would be updated and it is better to keep track of changes using subversion (and models change more likely than a PDF file ...). So, you can see the model here in the following directory of the repository : doc/asn1-protocol/model Best, |
(0000756) hugues (administrator) 2011-06-04 20:07 edited on: 2016-11-21 12:59 |
When re-reading the models, and thinking about a potential implementation, I found multiple issues to discuss: We have two layers, one for the driver, one for the protocol. The idea of the protocol layer is to define multiple encoding schemes for the packets to be sent/received. Sending is easy: marshall header + payload, then send. The payload is an already marshalled ASN.1 buffer, the header is (in case of Ada) length + destination (combination of receiver thread + port). We can certainly (Julien and I) agree on a common ASN.1 scheme for this part. Receiving is more complex: the driver needs to know some information to process the message. It needs (at least) to look for the length of the packet it will receive, for instance for serial or even TCP/IP. (Note: there is an implementation issue for which I'll open a separate ticket to discuss this). So here, the issue is to receive a partial chunk (header magic string for finding the beginning of a new packet + length) prior to receive other elements. This means a more complex model. Not a big deal to add intermediate marshallers in the AADL model itself. But this raises also a question on ASN.1 marshallers themselves, and would call for potential additions to asn1scc (or clarification from Thanassis). Let me explain (Sorry for using C instead of ASN.1, I'm not that comfortable yet with it). Suppose the packet has the following structure (simplified for the discussion)
Thanassis, can asn1scc generates a global marshaller/unmarshaller for request_type, and in addition marshallers/unmarshallers for each subblock? Each marshaller for subblocks would do the following
that is, "poke" directly in the ASN.1 marshalled buffer the corresponding value Same for unmarshall, that would "peek" directly in the buffer the value void unmarshall (uint_32 *length, void *buffer); Note I use here a void* (could be unsigned char *) to imply that the unmarshaller will operate on a potentially incomplete buffer With these functions, the driver will then use these functions to process incoming streams properly. |
(0000757) hugues (administrator) 2011-06-04 20:15 |
Reminder sent to: ttsiodras May you please comment on the ASN.1 part? Do not hesitate to ask for details if I'm not clear enough |
(0000761) maxime (administrator) 2011-06-06 20:12 edited on: 2016-11-21 12:58 |
See my reply here : 0000759 There shall be one ASN.1 module generated by Ocarina and listing all entity identifiers:
Then one ASN.1 type per inter-partition communication, defining the Packet type:
ASN1Scc will generate a structure similar to the one you describe, and an encoder/decoder function for Packet-ty. The decoder will put the result in the C or Ada structure that it has generated. Does it answer you question? |
(0000762) maxime (administrator) 2011-06-07 04:46 |
Additional information: These ASN.1 structures (Packet-ty and Entity-ty) will be generated by you (Ocarina). The only thing your code will do is fill/read this structure and send it. The user can intervene in one case: he shall be able to provide an ACN file associated to this structure, in which case the encoding function you will call is Encode_ACN_Packet_ty instead of Encode_UPER_Packet_ty. The ACN file will allow the user to specify a different binary encoding (e.g. PUS-compliant) for the packet: inside it it will be possible for example to change the order of the fields, to change the integer values for Entity-ty values, etc.) |
(0000764) hugues (administrator) 2011-06-07 07:07 edited on: 2016-11-21 13:02 |
This does not address my concern, let me rephrase it, in the hope it clarifies thing. If not, let us schedule a teleconf
a. wait for incoming packet (polling or interrupt) so, what my driver needs, and that is not answered by your answers is
We want to have the full packet definition in ASN.1, and I have no objection, we can have it. But I ned confirmation for more features that I actually know My question is whether we can have
and have partial unmarshallers to
without size, I'll have to wait for the maximum number of bytes; so my question is really not on what to do with a full packet, I know this part |
(0000765) maxime (administrator) 2011-06-07 08:27 |
Ok, sorry for the misunderstanding. I'll answer on your two points, with the PUS approach in mind 1) Magic value: this is not needed. The cutting into small chuncks has to be done on the payload and not on the Packet-ty. 2) on the size: you are correct. The header shall have a fixed size, including a field with the payload length. your driver shall: a. wait for the size of the header (small, fixed size) |
(0000767) maxime (administrator) 2011-06-08 09:00 |
Reading back the root of this ticket: http://code.google.com/p/buildsupport/issues/detail?id=7 [^] I found out that Julien already implemented part of the feature in POHIC ?! Can you clarify? |
(0000789) user2 2011-06-23 08:04 |
Follow-up. Answer to 0000767Yes, part of the feature was implemented in Ocarina. It consists in mapping the actual PO-HI-C protocol in ASN.1. But in our case, we want to provide the ability to define your own protocol, not just map a fixed protocol from C to ASN.1. Answer to 0000764The developer that implements the protocol knows how much data has to be received by the driver. So, we don't need a specific header. Some times, the protocol will use a fixed data size, sometimes not. It's up to the guy that implement the protocol and we let him decide how to handle that. Then, one can wonder about how we can know the data size that has to be used. In fact, if you read the specification I proposed, each bus has a configuration table that defines : the marshallers function AND the amount of data size to be used. So, you don't need the size. Finally, about the magic key, you can add it and it would be protocol dependent and provided by the marshallers function. However, I have to admit that I wonder if the specifications are really useful since it seems that it is not read :/ |
(0000790) user2 2011-06-23 08:09 |
Follow-up (again). So, please indicate what are the problems with the proposed document and what are the problems to proceed to the implementation. Thanks, Regards, |
(0000794) hugues (administrator) 2011-06-23 12:56 |
I strongly disagree on the fact that you know before hand the size of the request to be received, imagine a system that may process small tcs and one giant tc, if the size is aligned on the maximum size of the messages, then we are inefficient. This is OK for small demos, this is not for realistic projects. So, in this context of variable-length messages, when receiving a message, I'll first receive a partial chunks that I need to process to know the size of the full message. This chunk will be ASN.1 or whatever encoded. For the whatever, it is up to the representation scheme to define, and protocols like GIOP did that For ASN.1, I need functions to peek values in a ASN.1 buffer. So Thanassis, is this possible? |
(0000795) ttsiodras (administrator) 2011-06-23 13:16 edited on: 2016-11-21 13:06 |
I am trying to bring it all back to mind: re-reading the discussion above, I don't understand the comment "we cant send the maximum data size all the time": indeed we don't - as you can see in the generated "invoke_ri.c" stubs...
...so only the ACTUAL encoded data are passed to the vm_... function. |
(0000796) ttsiodras (administrator) 2011-06-23 13:22 |
In case I was not clear: The comment above (795) is just meant to say that the Encode/Decode ASN.1 functions do offer optimal space management - I dont understand why they can't be used to that effect in terms of the driver code... |
(0000797) hugues (administrator) 2011-06-23 13:30 |
Thanassis, please do not mix what happens at the application level from what happens at the transport level. We are discussing the latter, and for the implementation, we always send a message of maximum size, by adding padding if required. from the LEON SERIAL DRIVER (po_hi_driver_leon_serial.c) __PO_HI_MESSAGES_MAX_SIZE is a macro computed by Ocarina, it is the size of the biggest message to be sent/received, then we do that: while (tr < __PO_HI_MESSAGES_MAX_SIZE) and n = write (po_hi_c_driver_leon_serial_fd_write, &(msg.content[0]), __PO_HI_MESSAGES_MAX_SIZE); so your answer, although valid w.r.t. to the application part is no longer valid for what happens at driver level so again, what I'm asking for is all the more simple, and is detailed in message 764 above, and summed in message 794 |
(0000798) ttsiodras (administrator) 2011-06-23 14:20 edited on: 2016-11-21 12:58 |
The only reason I asked is because I want to understand the issue.
If the communicating channel is not multiplexed, then we could apply
...and send header first, payload second, with whatever ASN.1 encoding we want.
} If, instead, the communication channel IS multiplexed (so header and payload ASN.1 file:
ACN file:
If you pass the ACN spec as well to ASN1SCC (use the -ACN flag), the receiving code can just do:
|
(0000799) user2 2011-06-23 14:21 |
Jerome : you don't know the size in advante BUT the guy that implements the protocol knows how much data it should received. In other words, ASN.1 will provide data structures and programming facilities to marshall the packets but after, this is the responsability of the programmer/designer to : Hope that I'm clear. Best, |
(0000800) hugues (administrator) 2011-06-23 14:31 |
Thanassis answer from message 798 is exactly the information I was asking for. Thanks |
(0000801) user2 2011-06-23 14:48 |
I think there is a totally misunderstanding : there is no predefined ACN or ASN.1 model that has to be defined : the purpose and the scope of this bug is to let the user defines by himself its own ASN.1 and ACN model to implement its own protocol. |
(0000802) hugues (administrator) 2011-06-23 15:13 |
the understanding is clear: simply replace "the" by "a" (the joy of gallicism ..) without an initial model we cannot start; and we already have some elements from the various exchanges we got so far. So let us start prototyping work by defining a simple ASN.1 file, and see if we have all elements to converge |
(0000803) user2 2011-06-23 15:25 |
ergghh .... the configuration of polyorb is already proposed in the specs ... So, you can just add : Payload-T ::= BIT STRING (0 .. 5000) Packet-T ::= SEQUENCE { And then you are ! |
(0001088) user2 2011-08-11 16:46 |
A first shot has been made to integrate not only ASN.1 protocols but all user-defined protocols. It can of course includes protocol defined by the user and that use ASN.1. Please look at misc/experiments/protocol-asn1 to see an example with a very basic and simple protocol that just send application data directly on the network. All comments are welcome ! |
(0001089) maxime (administrator) 2011-08-16 15:01 |
I don't really understand how this will work with TASTE, what to define where, etc. Do we need a new property on the bus component in the deployment view to specify which protocol we want to use on a given inter-node link? Then what information do we need to capture at model level? (The example is in pure AADL form - difficult to understand/map to TASTE) |
(0001179) user2 2011-09-21 08:47 |
So, this was designed and added in PolyORB-HI-C, I assign this ticket to Jerome, waiting that it is implemented in the Ada runtime. |
(0001352) maxime (administrator) 2011-11-03 16:12 |
Thanks for him, but this does not answer the question :-) What should we do/add in TASTE to support this feature? Do you need any help to progress with this issue ? |
(0001353) hugues (administrator) 2011-11-03 16:21 |
No, unless you have a time machine ;) |
(0001355) user2 2011-11-03 18:33 |
In the meantime, Jerome, please tell me if you see any defect in the implementation. Then, for Maxime, you have to consider that integrating user-defined bus would not be so difficult ... but will require modification in the graphical tools for sure ... (at least to specify the protocol implementation ...). |
(0003808) maxime (administrator) 2021-12-02 10:07 |
No update since 2011, closing ticket |
![]() |
|||
Date Modified | Username | Field | Change |
2010-08-05 08:24 | user2 | New Issue | |
2011-03-30 11:38 | maxime | Reporter |
user2 => maxime |
2011-03-30 11:38 | maxime | Assigned To |
=> user2 |
2011-03-30 11:38 | maxime | Priority |
normal => high |
2011-03-30 11:38 | maxime | Severity |
minor => major |
2011-03-30 11:38 | maxime | Reproducibility |
have not tried => N/A |
2011-03-30 11:38 | maxime | Status |
new => assigned |
2011-04-14 19:28 | maxime | Severity |
major => feature |
2011-05-12 15:41 | user2 | Note Added: 0000695 | |
2011-05-13 10:44 | hugues | Note Added: 0000696 | |
2011-05-13 11:47 | user2 | Note Added: 0000697 | |
2011-05-13 14:00 | hugues | Note Added: 0000698 | |
2011-05-13 15:22 | user2 | Note Added: 0000700 | |
2011-05-13 20:28 | user2 | Note Added: 0000702 | |
2011-05-26 13:02 | maxime | Note Added: 0000728 | |
2011-05-26 13:29 | user2 | File Added: specs.pdf | |
2011-05-26 13:29 | user2 | Note Added: 0000735 | |
2011-05-27 16:09 | user2 | Note Added: 0000739 | |
2011-06-04 20:07 | hugues | Note Added: 0000756 | |
2011-06-04 20:15 | hugues | Note Added: 0000757 | |
2011-06-06 20:12 | maxime | Note Added: 0000761 | |
2011-06-07 04:46 | maxime | Note Added: 0000762 | |
2011-06-07 07:07 | hugues | Note Added: 0000764 | |
2011-06-07 08:27 | maxime | Note Added: 0000765 | |
2011-06-08 09:00 | maxime | Note Added: 0000767 | |
2011-06-23 08:04 | user2 | Note Added: 0000789 | |
2011-06-23 08:09 | user2 | Note Added: 0000790 | |
2011-06-23 12:56 | hugues | Note Added: 0000794 | |
2011-06-23 13:16 | ttsiodras | Note Added: 0000795 | |
2011-06-23 13:16 | ttsiodras | Note Edited: 0000795 | View Revisions |
2011-06-23 13:22 | ttsiodras | Note Added: 0000796 | |
2011-06-23 13:30 | hugues | Note Added: 0000797 | |
2011-06-23 14:20 | ttsiodras | Note Added: 0000798 | |
2011-06-23 14:21 | user2 | Note Added: 0000799 | |
2011-06-23 14:31 | hugues | Note Added: 0000800 | |
2011-06-23 14:48 | user2 | Note Added: 0000801 | |
2011-06-23 15:13 | hugues | Note Added: 0000802 | |
2011-06-23 15:25 | user2 | Note Added: 0000803 | |
2011-08-11 16:46 | user2 | Note Added: 0001088 | |
2011-08-16 15:01 | maxime | Note Added: 0001089 | |
2011-09-21 08:47 | user2 | Note Added: 0001179 | |
2011-09-21 08:47 | user2 | Assigned To |
user2 => hugues |
2011-11-03 16:12 | maxime | Note Added: 0001352 | |
2011-11-03 16:21 | hugues | Note Added: 0001353 | |
2011-11-03 18:33 | user2 | Note Added: 0001355 | |
2016-11-21 11:02 | maxime | Description Updated | View Revisions |
2016-11-21 11:03 | maxime | Description Updated | View Revisions |
2016-11-21 12:58 | maxime | Note Edited: 0000798 | View Revisions |
2016-11-21 12:58 | maxime | Note Edited: 0000761 | View Revisions |
2016-11-21 12:59 | maxime | Note Edited: 0000756 | View Revisions |
2016-11-21 12:59 | maxime | Note Edited: 0000697 | View Revisions |
2016-11-21 13:02 | maxime | Note Edited: 0000764 | View Revisions |
2016-11-21 13:06 | maxime | Note Edited: 0000795 | View Revisions |
2021-12-02 10:07 | maxime | Note Added: 0003808 | |
2021-12-02 10:07 | maxime | Status |
assigned => closed |
2021-12-02 10:07 | maxime | Resolution |
open => fixed |
Copyright © 2000 - 2011 MantisBT Group |