The other day I was using Fiddler to examine some web traffic to automate some tasks. I then stumbled upon a cool Fiddler extension called Request to Code (by Chad Sowald), which takes any HTTP(s) request and converts it to C# code using the System.Net.WebRequest and System.Net.WebResponse classes. Unfortunately, the extension does not support PowerShell. As I am using the System.Net.WebClient class anyway (as I am quite lazy and this class encapsulates the aforementioned classes) and as I never took the time to find out how to write a Fidderl extension I decided to create a quick conversion script from scratch. And really within a couple of lines I got it somehow to work – except … for some gory details as I found out when trying to set a view different headers.

With the WebClient object you can add any headers you like via either adding them directly one by one or by adding them in one shot via a NamedValueCollection:

$wc = New-Object System.Net.WebClient;<br />
# one by one<br />
$wc.Headers.Add($HeaderName1, $HeaderValue1);<br />
$wc.Headers.Add($HeaderName2, $HeaderValue2);<br />
# one shot<br />
$Headers = New-Object System.Collections.Specialized.NameValueCollection;<br />
$Headers.Add($HeaderName, $HeaderValue);<br />
$wc.Headers.Add($Headers);

However when submitting the actual request you receive a relatively meaningless exception:

Exception calling "UploadString" with "3" argument(s): "An exception occurred during a WebClient request."<br />
At line:1 char:1<br />
+ $response = $wc.UploadString($Uri, $Method, $TextBody);<br />
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br />
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException<br />
    + FullyQualifiedErrorId : WebException

Digging deeper into the exception revealed the true cause (at least partially):

PS C:\data\Scripts\FiddlerPoSh&gt; $Error[0].Exception.InnerException.InnerException |fl * -force<br />
Message        : This header must be modified using the appropriate property or method.<br />
                 Parameter name: name<br />
ParamName      : name<br />
Data           : {}<br />
InnerException :<br />
TargetSite     : Void ThrowOnRestrictedHeader(System.String)<br />
StackTrace     :    at System.Net.WebHeaderCollection.ThrowOnRestrictedHeader(String headerName)<br />
                    at System.Net.WebHeaderCollection.Add(String name, String value)<br />
                    at System.Net.HttpWebRequest.set_Headers(WebHeaderCollection value)<br />
                    at System.Net.WebClient.CopyHeadersTo(WebRequest request)<br />
                    at System.Net.WebClient.GetWebRequest(Uri address)<br />
                    at System.Net.WebClient.UploadDataInternal(Uri address, String method, Byte[] data, WebRequest&amp;<br />
                 request)<br />
HelpLink       :<br />
Source         : System

A closer look at “Remark section of the WebClient.Headers Property actually confirmed this observation (personal note to myself: next time read documentation first…). There it also reads to use WebRequest directly when using restricted headers. In my case nothing I really wanted.

Having a look at the headers that are directly supported by the WebRequest object I decided to discard them and only insert the “other” headers:

# Get all defined headers<br />
[System.Net.HttpRequestHeader]|gm -Static -Type Property<br />
   TypeName: System.Net.HttpRequestHeader<br />
Name               MemberType Definition<br />
----               ---------- ----------<br />
Accept             Property   static System.Net.HttpRequestHeader Accept {get;}<br />
AcceptCharset      Property   static System.Net.HttpRequestHeader AcceptCharset {get;}<br />
AcceptEncoding     Property   static System.Net.HttpRequestHeader AcceptEncoding {get;}<br />
AcceptLanguage     Property   static System.Net.HttpRequestHeader AcceptLanguage {get;}<br />
Allow              Property   static System.Net.HttpRequestHeader Allow {get;}<br />
Authorization      Property   static System.Net.HttpRequestHeader Authorization {get;}<br />
CacheControl       Property   static System.Net.HttpRequestHeader CacheControl {get;}<br />
Connection         Property   static System.Net.HttpRequestHeader Connection {get;}<br />
...<br />
# Add header via<br />
$wc.Headers.Add([System.Net.HttpRequestHeader]::UserAgent, "PowerShell System.Net.WebClient");

The next issue was the GZIPped response from the webserver. However, it came down relatively easy to just a few lines of code to convert string data to byte arrays and decompress them by using the help from the system.Text.Encoding and System.IO.Compression namespaces.

so in the end the whole thing was a little bit more complex than expected, but still practible to implement:

<br />$Payload = Get-Content Get-Payload.txt;
# TODO check if datatype System.Array or string and number of lines

# Create WebClient object
$wc = $null;
$wc = New-Object System.Net.WebClient;

# get request line
$PayloadRequest = $Payload[0];
# (GET|POST) (URI) (HTTPVersion)
# no way to set the HTTP version with WebClient
$fReturn = $PayloadRequest -match '^([^\ ]+)\ ([^\ ]+)\ (.+)$';
if(!$fReturn) {
	Log-Error $fn "No valid PayloadRequest: '$PayloadRequest'";
	throw($gotoFailure);
} # if
$Method = $Matches[1];
$Uri = $Matches[2];
$HttpVersion = $Matches[3];

$wc.Headers.Clear();
$Headers = New-Object System.Collections.Specialized.NameValueCollection;
$aLineBody = New-Object System.Collections.ArrayList;
$ProcessingHeader = $true;
foreach($line in $Payload) {
	Log-Debug $fn "Processing request line[$($line.Length))] '$line' ...";
	if($ProcessingHeader) {
		if(0 -ge $Line.Length) {
			Log-Debug $fn "Empty line detected. Reached end of header section.";
			$ProcessingHeader = $false;
			continue;
		} # if
		$fReturn = $Line -match '^([^:]+):\ (.+)$';
		if(!$fReturn) {
			Log-Debug $fn "Current request line does not contain a valid header: '$Line'. Skipping ...";
			continue;
		} # if
		$HeaderName = $Matches[1];
		$HeaderValue = $Matches[2];
		Log-Debug $fn "Found header '$HeaderName': '$HeaderValue'";
		switch($HeaderName) {
		'Connection' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Date' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Host' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Content-Length' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Proxy-Connection' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Transfer-Encoding' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Range' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'Expect' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		'If-Modified-Since' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		# Only skip this header to prevent gzip,deflate responses
		#'Accept-Encoding' { Log-Warn $fn "Detected restricted header. Skipping ..."; continue; }
		default {
			$Headers.Add($HeaderName, $HeaderValue);
			#$wc.Headers.Add($HeaderName, $HeaderValue);
		}
		} # switch
	} else {
		$aLineBody.Add($line);
	} # if
	
} # foreach
# Add parsed headers
$TextBody = $aLineBody -join "`r`n";
$wc.Headers.Add($Headers);
$wc.Headers

if($Method.ToUpper() -eq 'GET') {
	$response = $wc.DownloadString($Uri);
} else {
	#$response = $wc.UploadString($Uri, $Method, $TextBody);
	$enc = [system.Text.Encoding]::UTF8;
	$aByte = $enc.GetBytes($TextBody);
	$aResponse = $wc.UploadData($Uri, $Method, $aByte);
} # if
$Error[0].Exception.InnerException.InnerException

$enc = [system.Text.Encoding]::UTF8;
$aByte = $enc.GetBytes($TextBody);
$aResponse = $wc.UploadData($Uri, $Method, $aByte);
[string] $Response = '';
if($wc.ResponseHeaders.Get("content-encoding").ToLower() -eq "gzip") {
	$ms = New-Object System.IO.MemoryStream (,$aResponse);
	$s = New-Object System.IO.Compression.GZipStream -ArgumentList $ms, $([System.IO.Compression.CompressionMode]::Decompress);
	$reader = New-Object System.IO.StreamReader($s);
	$Response = $reader.ReadToEnd();
elseif($wc.ResponseHeaders.Get("content-encoding").ToLower() -eq "deflate") {
	$ms = New-Object System.IO.MemoryStream (,$aResponse);
	$s = New-Object System.IO.Compression.DeflateStream -ArgumentList $ms, $([System.IO.Compression.CompressionMode]::Decompress);
	$reader = New-Object System.IO.StreamReader($s);
	$Response = $reader.ReadToEnd();
} else {
	$Response = $enc.GetString($aResponse);
} # if
# $Response now holds the string (decompressed) repreentation
# $aResponse hold the compressed or uncompressed byte array (depending on the Content-Encoding header