Wednesday, January 22, 2014

Internet outage in China on Jan 21

Yesterday China witnessed one of the largest Internet outages ever. I have three theories about why this outage may have occurred - two related to the Falun Gong but my third theory is that the Chinese authorities set out to attack the mirror websites.

From 15:30 to 16:30 (China time) on January 21, DNS lookup to any domain would incorrectly resolve to 65.49.2.178. Websites inside and outside of China were affected. Even Baidu and Sina were inaccessible. Only software using IP directly (e.g. QQ, VPNs) worked during that time. Attempts to visit any website redirected to http://65.49.2.178, which didn’t respond during that time.  The overwhelming traffic to this IP likely crashed the server.
Timeline
Event
15:15
GFW DNS poisoning begins.
15:17
Local DNS servers began to cache incorrect responses. Some large websites in China began to be affected e.g Sina Weibo.
Incorrect DNS continue to spread through Chinese DNS servers. Major websites including Baidu, Sina affected.
15:39
DNS poisoning lifted by GFW. But local DNS resolvers cached incorrect responses. Users continued to experience outage.
16:00
ISPs around China were manually flushing DNS caches and connections were gradually restored.
I have conclusive evidence that this outage was caused by the Great Firewall (GFW). DNS poisoning is used extensively by the GFW. Some articles that have appeared about this outage suspected that the root DNS server in China was hacked and all domains hijacked to 65.49.2.178. This could explain why DNS servers in China were poisoned. However, during that time, I see that a lookup to 8.8.8.8, a public DNS operated by Google, returned bogus results if the lookup was done from China. In fact, the Google public DNS was not poisoned; the bogus response 65.49.2.178 could only have been returned by GFW.  If the Chinese root DNS server was hacked, a DNS lookup in China via 8.8.8.8 should have returned a correct response. See the below image from Zhihu.

But why did GFW poison all domains and effectively block all website traffic in China?
This action must have been unintentional. 65.49.2.178 is owned by Dynamic Internet Technology according to an IP lookup, and they are behind the famous circumvention tool FreeGate. Currently, http://65.49.2.178 is a mirror site for dongtaiwang.com, a news portal operated by Falun Gong groups.

Blocking 65.49.2.178

One hypothesis is that GFW might have intended to block the IP but accidentally used that IP to poison all domains.

Hackers

Many Chinese media stated that yesterday’s outage may have been due to a hacking attempt. The IP is operated by Dynamic Internet Technology, “mortal enemy number one” of the Chinese government. Some are suggesting Dynamic Internet Technology is behind the outage. However, hacking into a root DNS resolver is not enough to cause this outage, as I explained earlier in this post. They have to hack into GFW. If they are indeed capable of doing that, they can accomplish so much more than messing the entire Chinese internet up. In addition, 65.49.2.178 during that time was not serving any content and with such traffic, it looks more like a DDOS attack against 65.49.2.178. They couldn't use that IP to spread sensitive content during that time. However, from today, they have indeed started to use http://65.49.2.178 to distribute mirrors and stopped within a few hours.

Blocking the mirror sites

The mirror site has attracted considerable attention and GFW has tried multiple times to block us. Backend servers are automatically rotated and the GFW automatically scans new URLs and DNS poisons them.  DNS poisoning is not commonly used compared to connection reset. GFW seems to only use DNS poisoning as a last resort when connection reset fails to block a site. The mirror forces GFW to add hundreds of rule-sets to DNS poisoning daily and perhaps it caused the system crash. This is supported by the fact that the new backend domains are no longer automatically blocked.
The backends are improved to prevent automatic discovery from GFW. Perhaps the script operated by GFW acquired a “null” domain from the mirror site and consequently blocked everything.

No comments:

Post a Comment