URL Parser

Apr 30, 2009 at 8:49 AM
I'm new to the whole idea of using parsers in general but have been reading a lot the past few days and find it extremely useful and interesting.  Most of the documentation I'm seeing refers to simply monitoring (I do understand that the name is Network Monitor..)  But I'm wondering if in any way you can modify packets that meet certain criteria.  For example writing a URL sniffer to allow certain URLs (msdn, yahoo, etc) and block others (.contains(graphic-material)).  It seems very plausible to write a parser that can parse out packets containing data from a white-list or black-list - I'm just wondering given the APIs if it would be possible to modify packets to filter out HTTP requests.

The basic approach from one developer and one attempt using winsock libraries is described below.

[ from http://www.codeproject.com/KB/IP/HTTPFilter.aspx ]

The solution is fairly simple. The following sequence is used to filter a matched web request:
  1. Capture packets.
  2. Distinguish TCP packets which are destined to port 80 or 8080.
  3. Look for a 'GET /' in the first four bytes of the data. If not found, continue with the next captured packet.
  4. Perform a Boyer-Moore pattern match over the data against the blacklist keywords. If not found, continue with the next captured packet.
  5. Send a TCP reset to the server and a block page to the client.
[end]


If anyone is having any pointers it'd be appreciated.  Thanks in advance.

Nick
May 1, 2009 at 2:14 PM
The Network Monitor driver is a filter driver which means it only looks at the packet but doesn't affect their content.  It seems you want block the traffic, which would not be possible with the API.  However you could certinaly view the traffic and only capture traffic based on your algorithm above.  That would be fairly striaght forward.  But the outcome would be a capture file with only information you allowed to pass.

Now this is not to say you'd couldn't some how use the API to accomplish what you want.  But the blocking part would have to occur with a NDIS driver you write.  But the API could still assist in the parsing part.  Performance might also be an issue here as our parsers are more built for accuracy than performance.

Tahnks,

Paul
May 7, 2009 at 3:38 AM
Paul - I appreciate the response.  This was what I had figured, just wondering if you had any other hints on modifying packets via an NDIS wrapper via data picked up by the Network Monitor APIs.  I've figured out how to write the filters I've needed (which was quite trivial thanks to good documentation) but am still having trouble starting and stopping capturing from a c# app.  I can open an already existing capture file, apply different display filters and get those fields I'm looking which is pretty close to what I need.  What I'd like to do is automate starting a capture and parsing from the open and running capture file.  Does that sound reasonable to do?

Creating a capture file with the C# class I've created below hasn't been working out for me yet.  I've tried looping through all of my adapters and starting capturing on all of them, tried hardcoding the wlan adapter index and just starting on it..don't know what else to do to get it to create a capture file and start capturing..  I havent tested my WWAN adapter and will do so later but at a minimum want to monitor my WLAN adapter's traffic.  Anything look out of place to you?

    public class NetworkMonitor
    {
        private const ulong ERROR_SUCCESS = 0;
        private const ulong ERROR_NOT_FOUND = 1168;
        private const ulong ERROR_RESOURCE_NOT_AVAILABLE = 5006;


        private const string captureFileName = "localcapture.cap";
        private string captureFileNamePath = Application.StartupPath + captureFileName;
        string path = "C:\\Capture\\capture.cap";

        uint ret;
        uint capSize;
        private IntPtr myCapFile;
        private IntPtr myCaptureEngine;

        uint wwanAdapterIndex;
        uint wlanAdapterIndex;
        int numberOfAdapters = 0;        
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public NetworkMonitor()
        {
            ret = 0;
            InitCapturing();
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//
        private void InitCapturing()  //could make public and take out of the constuctor...
        {
            try
            {
                OpenCaptureEngine();
                EnumerateAndConfigureAdaptors();
                ConfigAdaptor();
                CreateCaptureFile();          
            }
            catch (Exception e)
            {
                //LOG ERROR                
            }            
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//
        private void OpenCaptureEngine()
        {
            try
            {
                //Open the capture engine

                uint ret = NetmonAPI.NmOpenCaptureEngine(out myCaptureEngine);

                if (ret != 0)
                {
                    MessageBox.Show("Error opening capture engine.");
                    NetmonAPI.NmCloseHandle(myCapFile);
                    return;
                }
                //MessageBox.Show("Capture Engine Opened OK");
            }
            catch (Exception e)
            {
                //LOG ERROR                
            }            
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//
        private void EnumerateAndConfigureAdaptors()
        {
            try
            {
                /*
                uint adapterCount;
                NetmonAPI.NmGetAdapterCount(myCaptureEngine, out adapterCount);
                MessageBox.Show(adapterCount.ToString() + " adapters found");

                
                for(uint n=0; n< adapterCount;n++)
                {
                    //ret = NetmonAPI.NmConfigAdapter(myCaptureEngine, n, FrameIndicationCallback, myCapFile, NmCaptureCallbackExitMode.ReturnRemainFrames);
                    
                    ret = NetmonAPI.NmConfigAdapter(myCaptureEngine, n, new CaptureCallbackDelegate(FrameIndicationCallback), IntPtr.Zero, NmCaptureCallbackExitMode.ReturnRemainFrames);
                    //ret = NetmonAPI.NmConfigAdapter(myCaptureEngine, n, FrameIndicationCallback, IntPtr.Zero, NmCaptureCallbackExitMode.ReturnRemainFrames);

                    if(ret != ERROR_SUCCESS)  //there are no errors configuring the device(s)...
                    {
                        //wprintf(L"Error configuration adapter %d, 0x%X\n", n, ret);
                    }
                    else
                    {
                        numberOfAdapters++;
                    }

                    
                }
                */
                //MessageBox.Show(numberOfAdapters.ToString() + " adapters...");
               
                //here you'll decide which adapter to get.. either WWAN OR WLAN
                //NetmonAPI.nm


                wlanAdapterIndex = 4;  //whenever you give it index 5 it will fail to configure the adaptor...because only 5 adapters, not 6



                wwanAdapterIndex = 0;

                //MessageBox.Show("Enumerated Adaptors OK");
            }
            catch (Exception e)
            {
                //LOG ERROR                
            }            
        }

        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        private void CreateCaptureFile()
        {
            try
            {
                ret = NetmonAPI.NmCreateCaptureFile(path, 20000000, NmCaptureFileFlag.WrapAround, out myCapFile, out capSize);
                //ret = NetmonAPI.NmCreateCaptureFile(captureFileNamePath, 20000000, NmCaptureFileFlag.WrapAround, out myCapFile, out capSize);
                if (ret != 0)
                {
                    MessageBox.Show("Error Opening Capture File");
                    return;
                }
                //MessageBox.Show("Capture File Created OK");
            }
            catch (Exception e)
            {
                //LOG ERROR                
            }            
        }
       
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        private void ConfigAdaptor()  //need to write two different callback methods for each adapter, even maintain separate files / separate monitors?
        {
            //return;

            try
            {
                ret = NetmonAPI.NmConfigAdapter(myCaptureEngine, wlanAdapterIndex, new CaptureCallbackDelegate(FrameIndicationCallback), myCapFile, NmCaptureCallbackExitMode.ReturnRemainFrames);

                if (ret != 0)
                {
                    MessageBox.Show("Error configuration adapter.");
                    NetmonAPI.NmCloseHandle(myCaptureEngine);
                    NetmonAPI.NmCloseHandle(myCapFile);
                    return;
                }
                //MessageBox.Show("Configured Adaptor OK");
            }
            catch (Exception e)
            {
                //LOG ERROR                
            }            
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public void FrameIndicationCallback(IntPtr hCapEng, UInt32 ulAdatIdx, IntPtr pContext, IntPtr hRawFrame)
        {
            try
            {
                IntPtr capFile = pContext;
                NetmonAPI.NmAddFrame(capFile, hRawFrame);
            }
            catch (Exception e)
            {
                
            }  
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public uint StartCapture()
        {
            try
            {
                //don't need to use Promiscuous
                return NetmonAPI.NmStartCapture(myCaptureEngine, wlanAdapterIndex, NmCaptureMode.LocalOnly);
                //NetmonAPI.
                /*
                for (uint i = 0; i < numberOfAdapters; i++)
                {
                    NetmonAPI.NmStartCapture(myCaptureEngine, i, NmCaptureMode.LocalOnly);
                }
                 */
                //return 0;
            }
            catch (Exception e)
            {
                //LOG ERROR
                return 1;
            }  
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public uint PauseCapture()
        {
            try
            {
                return NetmonAPI.NmPauseCapture(myCaptureEngine, wlanAdapterIndex);
            }
            catch (Exception e)
            {
                //LOG ERROR
                return 1;
            }  
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public uint ResumeCapture()
        {
            try
            {
                return NetmonAPI.NmResumeCapture(myCaptureEngine, wlanAdapterIndex);
            }
            catch (Exception e)
            {
                //LOG ERROR
                return 1;
            }  
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//        
        public uint StopCapture()
        {
            try
            {
                uint returnMe = NetmonAPI.NmStopCapture(myCaptureEngine, wlanAdapterIndex);
                NetmonAPI.NmCloseHandle(myCaptureEngine);
                NetmonAPI.NmCloseHandle(myCapFile);
                return returnMe;
            }
            catch (Exception e)
            {
                //LOG ERROR
                return 1;
            }            
        }
        //**********************************************************************************************************//
        //**********************************************************************************************************//       
    }
}

May 7, 2009 at 3:31 PM
I might be helpful to print out the return codes and send those to us, but I'm thinking this is related to working with multi-threaded apps and the capture engine.  Look in the help under "Working with C# Managed Code" under "Introduction to the Network Monitor API".  That will actually point you to "Working with Multi-Threaded Code" which will hopefully help you out.

Basically, you might have to adjust the threading model or your application, driver or the engine in order to capture.  Placing [STAThread] on you Main, is one way to verfy this is your issue.

But again if this doesn't help, please tell me which function is failing and what the return code is.

Thanks,

Paul
May 8, 2009 at 5:46 AM

Paul, thanks again for your quick reply.

 

I don't think its a threading issue as when packets are sent I do get callbacks on the correct adapter and everything:

 

[one of my log files with return codes]

[5/7/2009 4:32:08 PM] : Enter OpenCaptureEngine
[5/7/2009 4:32:09 PM] : OpenCaptureEngine returncode = 0
[5/7/2009 4:32:09 PM] : Enter EnumerateAndConfigureAdapters
[5/7/2009 4:32:09 PM] : 0) EnumerateAndConfigureAdapters returncode = 0
[5/7/2009 4:32:09 PM] : 1) EnumerateAndConfigureAdapters returncode = 0
[5/7/2009 4:32:09 PM] : 2) EnumerateAndConfigureAdapters returncode = 0
[5/7/2009 4:32:09 PM] : 3) EnumerateAndConfigureAdapters returncode = 0
[5/7/2009 4:32:09 PM] : 4) EnumerateAndConfigureAdapters returncode = 0
[5/7/2009 4:32:09 PM] : Enter CreateCaptureFile
[5/7/2009 4:32:09 PM] : CreateCaptureFile returncode = 0
[5/7/2009 4:32:13 PM] : Enter StartCapture
[5/7/2009 4:32:13 PM] : StartCapture return code = 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Capture Engine: 65536 | Adapter Index: 4 | P Context: 0 | Raw Frame: 524288
[5/7/2009 4:32:14 PM] : Enter FrameIndicationCallback | Frame Count: 0

 

It just dawned on me that I may not actually need to save the capture file, but to try and process (parse) the raw packets as they come in and log the URL hosts (and to later try to modify the packets via NDIS if they're blacklisted hosts).  Is this an approach you've used before or is saving the capture file necessary?  Does the above log look like I'm capturing data correctly?  It will always shows the same Capture Engine and Raw Frame etc values on every capture / pause and resume and even after a restart of the application these are always the same.  Using the Network Monitor UI I know how to apply capture filters, and using C# I know how to parse through capture files and apply display filters.  I'm just looking to do those all on the fly in my code.

May 8, 2009 at 2:28 PM
Edited May 8, 2009 at 2:49 PM

You could do all your calculation in the callback.  The potential problem here is that could slow down processing of frames.  With a heavy load or slow CPU, frames could buffer and fill the disk up.  This is the main reason we wrote the example as we did.

Looking at your callback, the Context seems to be zero instead of the handle to the capture file.  I see 4 different commented NmConfigureAdapters so perhaps you were trying different things.  With the version in EnumerateAndConfigureAdaptors, it seems zero is what you are sending.  Given the value is zero in your example above I think this will never work.  You could also print out the return code form NmAddFrame, though I'm sure wiht a 0 as a handle it will fail.

Can you verify you are actually sending the capture file handle to the callback?

Paul

Feb 15, 2010 at 5:59 PM

Hi etnaelk,

We actually had someone else encounter your problem recently starting off your code sample.  We were finally able to determine the issue was with your call to NmConfigAdapter.

You need to keep a reference to the callback delegate as shown in our help file examples.  Otherwise, if you create the object inline as you have done, .NET will try and automatically garbage collect the reference as nothing it knows about references it (our API is native and not managed by the .NET runtime).  This will cause your program to crash after some amount of time whenever .NET decides to garbage collect.

Please see the reference code below as a simple starting guide to correctly implement a program that can capture and record to a file for long periods of time.

Thanks,
Michael Hawker | Program Manager | Network Monitor

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.NetworkMonitor;

 

namespace NetmonCap
{

    class Capturer
    {

        static ulong Count;
        string filename;
        IntPtr capFile;
        IntPtr capEngine;

 

        private CaptureCallbackDelegate pCaptureCallBack = new CaptureCallbackDelegate(CaptureCallback);

        public Capturer(string filename)
        {

            this.filename = filename;

        }

 

        public void Start()
 
       {

            uint size;
            uint errno;

            // Creates a capture file which will store the last 10MB of traffic captured
            errno = NetmonAPI.NmCreateCaptureFile(this.filename, 10000000, NmCaptureFileFlag.WrapAround, out this.capFile, out size);

            if (errno != 0)
            {

                Console.Write("Error Creating File");

            }

 

            errno = NetmonAPI.NmOpenCaptureEngine(out this.capEngine);
            if (errno != 0)
            {

                Console.Write("Error Creating Engine");

            }

            // The "1" below represents the index of the adapter to capture on, this is just a simple example which captures on one fixed adapter.
            errno = NetmonAPI.NmConfigAdapter(capEngine, 1, pCaptureCallBack, this.capFile, NmCaptureCallbackExitMode.DiscardRemainFrames);
            if (errno != 0)
            {

                Console.Write("Error Configuring Capture");

            }

            errno = NetmonAPI.NmStartCapture(capEngine, 1, NmCaptureMode.Promiscuous);
            if (errno != 0)
            {

                Console.Write("Error Starting Capture");

            }

        }

 

        public void Stop()
        {

            uint errno;

            errno = NetmonAPI.NmStopCapture(capEngine, 1);
            if (errno != 0)
            {

                Console.Write("Error Stopping Capture");

            }

 

            NetmonAPI.NmCloseHandle(capEngine);
            capEngine = IntPtr.Zero;

            NetmonAPI.NmCloseHandle(capFile);
            capFile = IntPtr.Zero;
        }

 

        private static void CaptureCallback(IntPtr hCaptureEngine, UInt32 ulAdapterIndex, IntPtr pCallerContext, IntPtr hFrame)
        {           

            if (pCallerContext != IntPtr.Zero)
            {

                Count++;
                NetmonAPI.NmAddFrame(pCallerContext, hFrame);
                Console.WriteLine("Saved " + Count + " Frames");

            }

        }

    }

}