A few days ago our team spent some time stress testing our web services.
We divided the team into 'Attackers' and 'Defenders' - the attackers' goal was to generate traffic that would bring down our system, the defenders' goal to understand the attacks and come up with innovative ways to block them - our team leader called this 'War Games'.
If you're under 25 and you don't know what 'War Games' is, you just
pissed me off; see http://en.wikipedia.org/wiki/WarGames
One of the early attacks involved the user of JMeter (http://jmeter.apache.org/ ) - you don't need much CPU or bandwidth to generate tons of GET requests - and we found that a couple of users running JMeter with a broadband connection could hit our servers hard with literally thousands of requests per second - causing invalid and unnecessary traffic
A stress test client that can be used nefariously -
JMeter: http://jmeter.apache.org/
After doing some research, our team (the 'Defenders') came up with a couple of ideas on how to block such traffic. One of these ideas made use of an IIS extension provided by Microsoft called "Dynamic IP Restrictions", which I'll call from now on "Dynamic IP".
IIS Extension: http://www.iis.net/download/dynamiciprestrictions
The 'Dynamic IP' IIS extension is provided as a MSI installer - since our services are deployed to Windows Azure, we found that we needed to use Azure "Startup Tasks" to properly install and configure the extension on our virtual machines in the cloud.
Below I show in detail -
- Local Development in IIS - how to download, install and configure the 'Dynamic IP' extension when using a local instance of IIS.
- Command Line Configuration - how to configure the IIS extension using the command line.
- Local Development in Emulator and Azure Startup Tasks - how to make sure the extension works when running in the local Windows Azure emulator and how to ensure the installation of the extension doesn't run every time you run in the emulator locally.
- Deployment to Cloud - Window Azure Fabric - how to deploy, install and configure the extension to in the cloud and how to verify it works when running in Windows Azure VM instances.
A few notes to hopefully avoid some trolls -
- This won't stop all denial of service attacks, but that doesn't mean that we should do nothing - and end up at the mercy of all attacks.
- I write these tutorials to contribute to the .NET community - as a buddy once told me: "your stuff is what we should be seeing on MSDN"; that made me happy - but I'm always looking for constructive criticism on the content or style.
- As always, my posts contain tons of screenshots - that saves me tons of typing but also ensures all steps are covered.
Hat tip to my co-worker Matthew T. who did a lot of this work and came up with some of the solutions below.
How to stop a 'Denial Of Service' (DoS) attack on your ASP.NET web site,
aka “Using the 'Dynamic IP Restrictions' IIS Extension on Windows Azure"
You can follow along with these steps on your own, but below I also provide links that allow you to download all projects and solutions referenced in these screenshots. For any of the screenshots below, click on the screenshot to zoom in.
Pre-requisites
- A yearning to learn cool new Microsoft .NET technologies
- Visual Studio 2010 + SP1
- IIS (not IIS Express)
- Azure SDK (version 1.6 was used when this article was written).
- Windows Azure subscription (to deploy and run in the cloud).
- Tip: use the 'Web Platform Installer' (http://www.microsoft.com/web/downloads/platform.aspx ) to easily install and configure IIS and the Azure SDK.
You'll also need to install the 'Dynamic IP Restrictions' IIS extension - you can download it from http://www.iis.net/download/dynamiciprestrictions
Once you've downloaded and installed the 'Dynamic IP Restrictions' IIS extension, we are ready to create our test project.
You can download the full solution, including the azure and web role projects -
Download DynamicIPIISExtensionOnAzure.zip (612KB)
Extract the two folders, open the ‘C:\sandbox\DynamicIPIISExtensionOnAzure\DynamicIPIISExtensionOnAzure.sln’ solution with Visual Studio (projects made with Visual Studio 2010 SP1 + Azure SDK 1.6) and follow along.
1. Local Development - IIS
In this section, we'll set up our test project and configure the "Dynamic IP" IIS extension - and we'll then test it out locally using IIS.
Let's start with a new project, open Visual Studio and create an "ASP.NET Web Application"; let's call it "DynamicIPIISExtensionOnAzure" and put it under "C:\sandbox" -
Tip: we are creating an "ASP.NET Web Application", but everything we are doing works for MVC sites, WCF services, etc…
We should now have a project and solution under "C:\sandbox\DynamicIPIISExtensionOnAzure", with default files Visual Studio has created for us; compile the project and make sure you get no errors, then save it all, including the solution.
It should look like this -
Next we'll create a new site in IIS and configure it to host our new website. There are multiple ways to help you resolve a host name, in this case let's just edit the C:\Windows\System32\drivers\etc\hosts file and add an entry for -
127.0.0.1 dynamicipiisextensiononazure.localhost.com
Make sure you can ping 'dynamicipiisextensiononazure.localhost.com'
Tip: if you have a local DNS server, configure it so all requests to *.localhost.com resolves to 127.0.0.1; this allows all developers to easily setup different sites with different host names, and easily resolve to their localhost. If you prefer, change localhost.com to *.dev.yourdomain.com or whatever domain you work with locally.
Now open up IIS Manager -
Right click on 'Sites' and choose 'add web site', then fill it with the details below -
Tip: Click on 'Application Pools' then double-click on "DynamicIPIISExtensionOnAzure" and make sure you are using the .NET Framework 4.0; use the latest.
Now open your favorite browser and point it to http://dynamicipiisextensiononazure.localhost.com - you should see a page that looks like this:
Ok, we are done! Thank you for reading, good-bye.
We just got started, so fasten your seatbelts, and let's now create a page that will help us test the "Dynamic IP Restrictions" IIS extension.
To learn about all of the options available to configure the extension, check this great article by 'N. Lala' -
Using Dynamic IP Restrictions
http://learn.iis.net/page.aspx/548/using-dynamic-ip-restrictions/
Go read it now, so we are on the same page on how the extension works.
…
No really, go read it, I'll wait.
Next we are going to create a page that allows us to test the IIS extension by simulating a higher number of requests per second from the same IP that we allow. This is the same test page that N. Lala described in the article above.
Right-click on the "DynamicIPIISExtensionOnAzure" project to add a new page, let's call it "test.aspx" -
Copy and paste this code into the new page -
<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="test.aspx.cs" Inherits="DynamicIPIISExtensionOnAzure.test" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <script runat="server">
protected void Page_Load(object sender, EventArgs e)
{ System.Threading.Thread.Sleep(3000);
}
</script><html xmlns="http://www.w3.org/1999/xhtml"> <head id="Head1" runat="server"> <title>Dynamic IP IIS Extension on Windows Azure Test</title> </head> <body> <form id="form1" runat="server"> <div> <h1>Dynamic IP IIS Extension on Windows Azure Test.</h1> </div> </form> </body>
</html>
The page should look like this now -
Note: I like working with a dark background, so some of these screenshots may look a bit different than the standard Visual Studio color scheme.
Compile the project and open the 'test.aspx' on your browser - it should look like this -
Click refresh a few times, it should look the same. Don't hurt yourself, but refresh as fast as you can... and it should still look the same.
Now, we get to one of the juicy parts - open up IIS Manager and you should see the "Dynamic IP Restrictions" extension installed -
Notice that you can configure the "Dynamic IP" extension for one site or for all sites, at the computer level -
Double-click on the "Dynamic IP" extension and you'll be able to configure it -
Read the article mentioned above if you are still not sure what these parameters mean.
2. Command Line Configuration
You can use the UI to configure the extension, but there's also another way. The 'Dynamic IP' extension stores its configuration in the IIS file applicationHost.config -
Open up a cmd line window as an administrator and run this (one line) -
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/dynamicIpSecurity /denyByConcurrentRequests.enabled:"True" /denyByConcurrentRequests.maxConcurrentRequests:"2" /denyByRequestRate.enabled:"True" /denyByRequestRate.maxRequests:"10" /denyByRequestRate.requestIntervalInMilliseconds:"1000" /commit:apphost
Notice we set maxConcurrentRequests to '2'.
Tip: use AutoHotKey to copy and paste (CTRL-V) the above command directly into the cmd line window. You should not get any errors, it should look like this -
Load the applicationHost.config file in your favorite text editor and check the <dynamicIpSecurity> section.
Tip: use NotePad++ (http://notepad-plus-plus.org/) and you'll get notifications as the file changes -
To verify our changes are in effect, let's go back to IIS Manager, (if needed) close, then re-open the 'Dynamic IP' extension configuration page -
Notice the UI is reading from the updated configuration file and the 'Maximum number of concurrent requests' is now '2'. You can use the UI to make settings changes, for example change the 'Deny Action Type' to 'Send 403 (Forbidden)' -
You'll notice the applicationHost.config file changes, the <dynamicIpSecurity> node is updated with a 'denyAction=Forbidden' -
We can now append '/denyAction:"Forbidden"' to our command, like this (one line) -
%WINDIR%\system32\inetsrv\appcmd.exe set config -section:system.webServer/security/dynamicIpSecurity /denyAction:"Forbidden" /denyByConcurrentRequests.enabled:"True" /denyByConcurrentRequests.maxConcurrentRequests:"2" /denyByRequestRate.enabled:"True" /denyByRequestRate.maxRequests:"10" /denyByRequestRate.requestIntervalInMilliseconds:"1000" /commit:apphost
It should look like this -
Close, then re-open the 'Dynamic IP' extension configuration page and you should see the 'Deny Action Type' is set to 403 -
Now let's go back to the 'test.aspx' page on your browser; refresh the page fast - after a few times you should see the 403 error -
You've just stopped "anonymous" Chinese script kitties (like these jerks that attacked WordPress in 2011) from sending hundreds of thousands of requests from a few machines and bringing down your servers. Nicely done!
Tip: if you need to find out what files an application is modifying, use SysInternal's Process Monitor - http://technet.microsoft.com/en-us/sysinternals/bb896645 ; it works when running desktop apps or command line apps; we used it to find out which file(s) were changing when we made changes to the 'Dynamic IP' extension, using IIS Manager or appcmd.exe - in this case IIS' applicationHost.config.
3. Local Development - Windows Azure Emulator and Startup Tasks
Up to this point, we've installed and configured the IIS extension, and we tested it out. Next we'll make sure it runs in the local Windows Azure Fabric, the 'emulator'.
Right-click on the 'DynamicIPIISExtensionOnAzure' solution and choose "Add > New Project"; for the project type, select "Windows Azure Project", then name it 'DynamicIPIISExtensionOnAzure.deploy' -
The current version of the Windows Azure SDK forces you to add a new web or worker role, so add a new 'Visual C# > ASP.NET Web Role' (we'll delete it later) -
Now, let's add the earlier ASP.NET site we created as a web role to our Azure project; right-click on 'Roles > Web Role Project in solution…' -
Select the 'DynamicIPIISExtensionOnAzure' project we worked on earlier -
You can now remove 'WebRole1' from the Azure project's 'Roles' and delete it - your solution should now look like this -
Here's what it should look like in file explorer -
Right-click on the solution and choose 'Build', make sure all projects compile without errors -
When deploying your web role to the Azure cloud, we need to install the IIS extension and configure it. Earlier we found out how to configure the IIS extension using the command line, it's now time to use that in what Azure experts call a 'Startup Task' (http://msdn.microsoft.com/en-us/library/windowsazure/gg456327.aspx).
Create a folder, name it 'c:\sandbox\DynamicIPIISExtensionOnAzure\Startup' -
Now using your favorite text editor (should it be NotePad++ by now?), copy and paste the text below into a new file we’ll name ‘DynamicIPRestrictions.cmd’
@echo off setlocalif "%EMULATED%"=="true" goto :EOF
REM Install Dynamic IP Restrictions IIS Extension
Startup\dynamiciprestrictions_beta2_x64.msi /qnREM Configure Dynamic IP Restrictions IIS Extension
%WINDIR%\system32\inetsrv\appcmd.exe set config
-section:system.webServer/security/dynamicIpSecurity
/denyAction:"Forbidden"
/denyByConcurrentRequests.enabled:"True"
/denyByConcurrentRequests.maxConcurrentRequests:"2"
/denyByRequestRate.enabled:"True"
/denyByRequestRate.maxRequests:"10"
/denyByRequestRate.requestIntervalInMilliseconds:"1000"
/commit:apphost
Notes:
- The first time you may want to remove the line "if "%EMULATED%"=="true" goto :EOF" - this allows you to test the startup task locally.
- Once you get it working, put the line back, so you don't install the MSI and run appcmd.exe every time you run in the Azure emulator.
- It might take a few tries before you get it right, un-install the MSI manually before trying again.
- This sounds time consuming, and it is - but it takes way less time to test out your azure startup task this way, then by actually deploying your roles to the cloud.
Save the file to 'c:\sandbox\DynamicIPIISExtensionOnAzure\Startup\DynamicIPRestrictions.cmd' -
Copy the MSI file you downloaded earlier from http://www.iis.net/download/dynamiciprestrictions into the 'c:\sandbox\DynamicIPIISExtensionOnAzure\Startup' folder; since all Windows Azure virtual machines run 'Windows 2008 Server R2 x64', make sure you use the x64 MSI file -
Go back to Visual Studio, make sure you 'show all files', then select the 'Startup' folder, right-click and choose 'Include In Project' -
In 'Solution Explorer', right-click on 'DynamicIPRestrictions.cmd > Properties', then make set 'Build Action' to 'Content', and 'Copy to Output Directory' to 'Copy always'; do the same for the MSI file -
Your project should now look like this -
Now, under the 'DynamicIPRestrictionsOnAzure.deploy' project, double-click on the ServiceDefinition.csdef file and replace it with this -
It should look like this -
Compile and make sure there are no errors, your solution in file explorer should now look like this -
Now run the 'DynamicIPRestrictionsOnAzure.deploy' project. Visual Studio will create the azure package, and deploy it to the local Azure emulator -
In your system tray, you should see the Windows Azure tray icon and a message like this -
If you open up the emulator, it should look like this -
If you click on the 'Service Details' you can get the port, in this case our site is running in the emulator on port 8080 -
Open up your browser, and point it to http://127.0.0.1:8080/test/aspx - you should see this -
Start refreshing the page fast, like a squirrel on a bullet - and the 'Dynamic IP Restrictions' rules should kick in - you should see something like this -
We've verified the 'Dynamic IP" IIS extension works in the emulator and we've tested our startup task to make sure it installs the MSI and configures the IIS extension correctly.
4. Deployment to Cloud - Window Azure Fabric
Up to this point we have worked with the IIS extension locally and we've verified it's working as expected in both IIS and the local Azure emulator. It's time now to launch it to space, and deploy to the Windows Azure cloud.
Note: you will need a subscription to Windows Azure to continue.
Point your browser to https://windows.azure.com/ and you should see the 'Windows Azure Portal' - a Silverlight piece of art (I am not being sarcastic, I really like it). Create a new service under one of your subscriptions, call it "DynamicIPIISExtensionOnAzure" -
For the DNS name, chose 'DynamicIPIISExtensionOnAzure.cloudapp.net' - if not available chose something else.
Go back to Visual Studio, right-click on the '.deploy' project and choose 'Publish' -
Note: in our organization we use Team Foundation Server (TFS) and we have deployment builds that will do some of this work for us. To keep things simple, in this article I use Visual Studio to manually deploy our service.
Go through the 'Windows Azure Publish' wizard and choose your subscription -
Make sure you configure 'Remote Desktop', define:
- the username as 'dynamicip'
- the password as 'IISextension456' (or whatever you prefer, just remember it)
Target the 'Production' slot and click 'Publish' -
Visual Studio will do its magic and start the deployment process -
It will compile and create the Azure package and will upload it to your storage account -
Go back to the Windows Azure portal, and after a few moments you will see your service being updated -
Since we chose the 'Production' slot, our web role is initialized by the Azure Fabric controller -
Back in Visual Studio, we are getting similar feedback -
Tip: that's what's great about desktop and Silverlight applications - there's no need to 'refresh the screen' like you need to with non Ajax browser apps - it's all seamless (did you know Visual Studio 2010 is written in WPF?)
After a few more moments, the fabric controller is done and we see our web role is starting up -
In Visual Studio, we see the deployment is complete -
And on the Azure portal we see our web role is 'Ready' to rock'n'roll -
Now open up your browser and point it to http://DynamicIPIISExtensionOnAzure.cloudapp.net:8080/test.aspx (or whatever DNS host you chose when you setup your Azure service above) - and you should see our friendly message -
Now start refreshing the page like a Duracell Bunny fully charged - and we should see the 'Dynamic IP' extension… now working in the Azure cloud!
Congratulation, you successfully deployed your web site to the Windows Azure Cloud and stopped evil Russian hackers from bringing down your service (28% of DDOS attacks in H2 2011 originated in Russia and Ukraine).
Bonus Points - Remote Desktop (RDP) to Azure VM
If you need to troubleshoot the configuration of the IIS extension when your web role is running in the cloud, one way to do it is by using a remote desktop connection, just like you'd connect to a physical server.
Back on the Windows Azure Portal, select one instance of the DynamicIPIISExtensionOnAzure as show below -
We only deployed one instance, so it's name is DynamicIPIISExtensionOnAzure_IN_0, the "IN" stands for "Instance". It's easy to deploy 2 or 20 instances, by simply editing your ServiceDefinition.csdef file.
After you've selected DynamicIPIISExtensionOnAzure_IN_0, click on 'Connect'; if using IE, you may see the dialog below ' -
Click 'Open' and you'll see the standard RDP dialog, click 'Connect' -
Enter the username and password we defined above:
- the username as 'dynamicip'
- the password as 'IISextension456' (or whatever you defined)
You'll get the helpful dialog you're connecting to your virtual machine in the cloud -
Followed by the certificate warning, click 'Yes' -
You're now logged in on your Windows Azure virtual machine - running in Hong-Kong, Singapore, Dublin, or Chicago - depending on which data center you chose. Start IIS Manager, and you should see the 'Dynamic IP Restrictions' IIS extension installed -
Double click on it, and you should see the settings you defined in your startup task, using appcmd.exe -
If you want to go the extra mile, open up the applicationHost.config -
Use notepad, and you should see our familiar settings in the familiar XML -
Pretty amazing times we live in - we can create and deploy a new web service in literally minutes, at fractions of the cost from just a few years back. The world is full of possibilities.
Download DynamicIPIISExtensionOnAzure.zip (612KB)
Links
- Download the "Dynamic IP Restrictions" IIS extension
- Official help on using the "Dynamic IP Restrictions" IIS extension
- Great blog post on the "Dynamic IP Restrictions" IIS extension
Our test site -
- Our test page, running locally in IIS
- Our test page, running locally in the Azure Emulator
- Our test page, running in the Windows Azure cloud
I'm @ehuna, for more info see http://about.me/ehuna
Good times!
Tip: this blog post was written thanks to some heavy background music by Metallica, Disturbed, and the soundtracks from Tron, Inception, and ‘The Matrix’.
Comments