Parsing fragmented TCP stream

Jun 11, 2012 at 8:14 PM

Hi all,

I've written a parser of relative complex proprietary protocol and it works fine on single frame, but I'm not able to activate packet reassembly. I tried  follow recommendation in the discussion http://nmparsers.codeplex.com/discussions/203503 but with no result and referred HTTP parser seems to me too complex for me as newby...

A message of my protocol is fully transparent:

 UINT16 Length;

BLOB(Length) Payload;

so I've doubts how to indicate the first frame as in the example above.

Please, could  somebody help?

Thx, pf

 

Jun 12, 2012 at 6:54 PM

Is your protocol repsonsible for doing the fragmentation?  Does the Length above indicate the total length and then only the first packet has the length with some of the payload, and remaining packets have the rest of the payload?  Is the resulting packet (the length and payload) the same as your transport protocol?

If the answers above are all yes, then you'll need to have some indication of which is the first packet vs. fragments.   It might be possible to use a hueristic, but it seems that you could mistake any random UINT16 as a length.  If you don't have any indication of the first, then you'll have to do something more complex and keep state manually.  There are some limitations, like missing messages could mess up the state.  So if you don't have a clear indication in the protocol that a message is the first fragment vs not, then let us know and we can see if there's another option for your case.

Paul

Jun 12, 2012 at 7:54 PM
Edited Jun 13, 2012 at 7:22 AM

Hi Paul,

thanks for you reply.  Sorry I cannot copy&paste part of my protocol determining message kind because its rather complex and hundreds lines long, so I'll try to describe my attempts to reassemble frames verbaly:

 

My protocol writes UINT16 (big endian) as length of a message body and then sends a stream having that length  i.e. only the first packet begins with data length definition, so I'd say that total TCP.Payload length should be 2+defined length and I need to parse the  complete reassembled stream.

Each message body is composed of header and inner body, in the header, there are multi-criterial conditions to determine message kind dependend on networkdirection, hence I have nested switch commands, determining a message kind.

At the beginning, I reset  IsFirst to false and set total length. In the each valid branch I set IsFirst to true and possible remaining data in such known branches I've defined as BLOB with name UnknownData (following referred example). Behind those nested switch commands, there is another BLOB definition, named ContinuationData, which should contain data of an non-recognized message.

I've inserted the POST command from the example, created the conversation as well and registered Payload becide TCP registration.

But I still do not see any effect: in the packet list, there are still continuous packets, my parser operates just on the starting packet. If I do manual reassembly, in the new window I can see concatenated PayloadHeader, providing I comment out Payload registration. Once the payload registraction is there, the lines with PyaloadHeader disappears.

--

Another thing is not clear to me as a newbie: how this reassembly should work from user perspective: should it perform reassembly automatically and my protocol should be shown in the 'basic' capture screen or I have to perform a manual reassembly to open new window with reassemled packets and my protocol should be shown there?

Thanks for any help, pf

 

 

 

Jun 13, 2012 at 5:39 PM

To answer your last question first, you don't get reassembly by default.  You need to hit the reassembly button in the UI and then a new window opens up.  This new window contains all the original data, plus the newly inserted frames.  I have a video on the blog (http://blogs.technet.com/b/netmon/p/usagevideos.aspx), which might help understand how this works from the UI perspective.

To make sure we are on the same page with regards to how to implement reassembly for your case using IsFirst and a Length, let me show a small example of what the parameters for PayloadStart should look like over multiple frames.  For example, say we have this data.

Frame 1: Length=100, Blob of 40 bytes

Frame 2: Blob of 40 bytes

Frame 3: Blob of 20 bytes

The params to PayloadStart for each frame would be:

Frame 1: PayloadStart(NetworkDirection, 0, 0, 0, Length, true, 0, RssmblyIndStartBit + RssmblyIndLengthBit + RssmblySelfBit)

Frame 2: PayloadStart(NetworkDirection, 0, 0, 0, 0,         false, 0, RssmblyIndStartBit + RssmblyIndLengthBit + RssmblySelfBit)

Frame 3: PayloadStart(NetworkDirection, 0, 0, 0, 0,         false, 0, RssmblyIndStartBit + RssmblyIndLengthBit + RssmblySelfBit)

So your NPL code might look like this:

[RegisterBefore(...)]
[PayloadStart(NetworkDirection, 0, 0, 0, Property.MyProtLength, Property.IsFirst, 0, RssmblyIndStartBit + RssmblyIndLengthBit + RssmblySelfBit)]
Protocol MyProt
{
	[Property.MyProtLength]
	UINT16 Length;
	BLOB(Length) myblob;
}

So the trick is not to make Property.IsFirst follow the right pattern.  The length isn't as important because I beleive it's ignored when IsFirst, is false.  What makes this tricky is that we don't maintain state with properties as their scope is at the frame level.  So now you have to create state using conversation properties and MultiValueState arrays. 

Do you have any indication in your frame structure that can differentiate the first message from fragments?

 

Jun 13, 2012 at 7:10 PM
Edited Jun 14, 2012 at 6:56 AM
PaulLong wrote:

So your NPL code might look like this:

[RegisterBefore(...)]
[PayloadStart(NetworkDirection, 0, 0, 0, Property.MyProtLength, Property.IsFirst, 0, RssmblyIndStartBit + RssmblyIndLengthBit + RssmblySelfBit)]
Protocol MyProt
{
	[Property.MyProtLength]
	UINT16 Length;
	BLOB(Length) myblob;
}

So the trick is not to make Property.IsFirst follow the right pattern.  The length isn't as important because I beleive it's ignored when IsFirst, is false.  What makes this tricky is that we don't maintain state with properties as their scope is at the frame level.  So now you have to create state using conversation properties and MultiValueState arrays.

 Do you have any indication in your frame structure that can differentiate the first message from fragments?

Hi Paul.

great thanks for your help and illustrative explanation of Payload params. My implementation was the same as you recommend  except: for continuous frames I din't reset Length param (this parameter has been set to random number from the start of continuous frame, because first packet has been determinated later).

First frame indication I had implemeted already,  so once I cleared Length param before  continuous BLOB, it seems that my parser is able to parse reassembled frames (I don't have implemented parsing of long messages yet).

Once again, thank you very much for Your help,

pf

 

P.S. It seems that length IS NOT IGNORED for continuous frames.