7

I'm scraping a site with HttpWebRequest, but the site is returning an error. The page works fine when I hit it from my browser. I'd like to compare them to see what may be causing the error. I know how to intercept the request from my browser to inspect the headers, but how do I view the data sent by HttpWebRequest?

Improve this question

5 Answers 5

Reset to default
9

In order to compare what you do in code, and what the browser does, I am sure a HTTP debugging tool such as Fiddler, would be the easiest solution.

Fiddler acts as a proxy between client and server, and displays all information sent over the HTTP protocol.

It is possible that you will need to configure your .NET app to use the proxy that Fiddler provides. This blog post provides details on the subject.

Improve this answer
7

http://www.fiddler2.com/fiddler2/ is a great tool for such things.

Improve this answer
0
2

The Net panel of Firebug will show all requests, including headers.

EDIT: Saw you already knew how to do it in a browser as soon as I posted. Try the Headers property:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.stackoverflow.com");
request.Method = "GET";
HttpWebResponse response = (HttpWebResponse) request.GetResponse();
Console.WriteLine(response.Headers);
Improve this answer
5
  • 1
    I know how to view the request from my browser. I want to know how to see what .NET is doing with HttpWebRequest.
    – Jeremy Stein
    Commented Jun 1, 2009 at 15:04
  • 1
    I want to see the request headers, not the response headers. When I check request.Headers, they're empty. I'm guessing the Headers collection is used only for specifying headers, not for inspecting what it's going to use?
    – Jeremy Stein
    Commented Jun 1, 2009 at 15:12
  • Updated with the answer in C#. HttpWebRequest and HttpWebResponse both have the Headers property that'll give you what you're looking for.
    – Chris Doggett
    Commented Jun 1, 2009 at 15:12
  • When I try it with my sample code, replacing response.Headers with request.Headers, I get: Host: stackoverflow.com Connection: Keep-Alive
    – Chris Doggett
    Commented Jun 1, 2009 at 15:13
  • Looks like request.Headers isn't populated until after the GetResponse() call, as it's empty on mine if I move the Console.WriteLine up a line.
    – Chris Doggett
    Commented Jun 1, 2009 at 15:14
0

You can get the headers from a HTTPWebRequest via the Headers property. From MSDN: http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.headers.aspx

Improve this answer
1
  • 1
    Yes, but see the comments on Chris Doggett's answer above. The headers are not populated until after the request is submitted.
    – Jeremy Stein
    Commented Sep 30, 2009 at 12:56
-1

I don't know if there's a general solution. But if you're using Firefox, either of two add-ons will help: Firebug, or LiveHTTPHeaders.

Improve this answer
1
  • 1
    I know how to view the request from my browser. I want to know how to see what .NET is doing with HttpWebRequest.
    – Jeremy Stein
    Commented Jun 1, 2009 at 15:06

Not the answer you're looking for? Browse other questions tagged or ask your own question.