Parsing protocol in TCP stream

Mar 2, 2010 at 4:57 AM
Edited Mar 2, 2010 at 4:59 AM

Hi, I'm trying to write a parser for a proprietary protocol and as a newby need a couple of pointers.

This is a brief protocol description:

The protocol uses variable length messages. All messages have a similar structure, but different messages contain different payloads (which I would like to parse too).
The basic structure of a message is:

STX (0x02), Type (1 byte), Length (1 byte), PAYLOAD (as per length), ETX (0x03)

This is sent over a TCP socket on a predefined port.

I'm trying to write a parser and need some help:

a) I've managed to register the Parser by adding a RegisterBefore tag, with the appropriate name and port. Something like [ RegisterBefore (TCP.TCPPayload.SIP,MyProtocol, 1234) ]

b) How do I verify that the first byte is an STX (0x02)? Can I discard a frame that does not start with 0x02?

c) How do I handle different message types? It almost looks like I should have nested protocols. I.e. MyProtocol hooks into the TCPPayload and then MessageA hooks into MyProtocol.Type for certain values of Type if that makes sense?

d) Would the Network Monitor automatically put different TCP packets together? I.e. what if one of my messages got split over multiple TCP packets? Would that not cause the parser to fail or are the frames assembled automatically?

Thank you in advance. 

 

Mar 2, 2010 at 6:16 AM

Thanks for using Netmon! First for your last question, the answer is yes, you need to click "Reassemble" button in the tool bar.

Here is a sample parser to answer your other questions:

[RegisterBefore(...)]
protocol YourProtocol = "You may put some description/fields here"
{
	switch 
	{
		case UINT8(FrameData, FrameOffset) == 0x02:
			struct
			{
				UINT8 STX;
				UINT8 Type;
				UINT8 Length;
				switch (Type)
				{
					case 0x01:
						Message1 Message1;
					case 0x02:
						...
					default:
						BLOB(FrameLength - FrameOffset) UnknownPayload;
				}
			}
		default:
			BLOB(FrameLength - FrameOffset) UnknownData;
	}
}

struct Message1 = "Something"
{
	Blah YourField1;
	...
}
Please let us know if you need any further assistance.
Thanks, 
Luther
Mar 2, 2010 at 6:46 PM
Edited Mar 2, 2010 at 6:46 PM

Thank you very much Luther for your help. I got it working great with a couple of minor changes - which I've added below in case it helps someone else.

 

 

[RegisterBefore(...)]
protocol YourProtocol = "You may put some description/fields here"
{
	switch (UINT8(FrameData, FrameOffset))
	{
		case 0x02:
			struct
			{
				UINT8 STX;
				UINT8 Type;
				UINT8 Length;
				switch (Type)
				{
					case 0x01:
						Message1 Message1;
					case 0x02:
						...
					default:
						BLOB(FrameLength - FrameOffset) UnknownPayload;
				}
                               UINT8 ETX;
			}
		default:
			BLOB(FrameLength - FrameOffset) UnknownData;
	}
}

struct Message1 = "Something"
{
	Blah YourField1;
	...
}

 

 

 

Mar 6, 2010 at 3:46 PM

Hi Luther,

My protocol parser works fine but I'm hitting a snag that I need a bit more help with. I've tried following the documentation but just can't seem to get it right.

The issue is related to reassembling data for my messages from different packets.
In my packet capture I'm getting parts of the message split across different TCP packets. The packet reassembly that Network Monitor provides does not seem to solve the problem. I suspect it's because it's not actually TCP continuation frames. I.e. the sender is doing multiple sends for one logical message and hence the first couple of bytes come in frame 1 and then some more in frame 2.

From the documentation it seems like I should be able to define a conversation and allow for reassembly of my messages.

In the protocol above, I have the type and length - so in theory - could I specify where "my" payload starts and how long it is. This will allow network monitor to reassemble my message across frames, right?

Could you perhaps provide me a simple example of how I could achieve this?

Your help appreciated as always.

 

Mar 8, 2010 at 3:21 AM

Hi,

Yes, you are right, some reassembly code need to be written. A good real example for you to look at is HTTP parser. I'm also trying to provide some code snippet for you.

Please note, I only examined the first byte and assumed that all frames starting with 0x02 is the begining of a message. This may not be true since the data may also contains 0x02. You may need to add more restrictions, such as examining the message type etc. Let me know if you need the accurate approach (will have to use multi-value storage with conversation).

[RegisterBefore(...)]
[Post.PayloadStart(	/*Post is a must since the properties won't be available before the frame is parsed */
			NetworkDirection, /* Only traffic on same direction should be reassembled */
			0, //identifier
			0, //sequence token
			0, //next sequence
			Property.YourProtocolLength, //total payload length
			Property.YourProtocolIsFirst, //is first
			0, //is last
			RssmblySelfBit /* Indicates that it's not reassembling a payload protocol but itself */
				+ RssmblyIndStartBit 
				+ RssmblyIndLengthBit
		)
]
protocol YourProtocol = "You may put some description/fields here"
{
	switch
	{
		// Conversation must be built for reassembly
		// Since it's self reassembly, we use this trick to avoid building same conversation twice
		case ! (Conversation.YourProtocol.id > 0):
			[BuildConversationWithParent]
			struct{}
	}
	
	switch (UINT8(FrameData, FrameOffset))
	{
		case 0x02:
			[Property.YourProtocolIsFirst = true]
			struct
			{
				UINT8 STX;
				UINT8 Type;
				[Property.YourProtocolLength]
				UINT8 Length;
				switch (Type)
				{
					case 0x01:
						Message1 Message1;
					case 0x02:
						...
					default:
						BLOB(FrameLength - FrameOffset) UnknownPayload;
				}
				UINT8 ETX;
			}
		default:
			BLOB(FrameLength - FrameOffset) ContinuationData;
	}
}
Mar 10, 2010 at 8:14 PM

Luther, like always, your help is appreciated.
I've got the reassembly working beautifully.

One thing I had to change (simply due to our protocol definition):

 

[Property.YourProtocolLength = Length + 4]

This is due to the fact that the Length in my protocol is the "payload" length and excludes the STX, TYPE, LEN and ETX bytes. 

One more question:
Is there any benefit for adding my protocol to payload.npl? From what I could tell, it simply changes the Protocol that is shown in the capture to my actual protocol for reassembled frames (which is nice).

 

 

 

Mar 11, 2010 at 3:36 AM

Glad to know that you've got reassembly working :)

As you've already found out, the benefit of adding the protocol to payload.npl is to let netmon know which parser should be invoked to parse the reassembled payload (in this case, not payload but your protocol itself since "RssmblySelfBit" specified).