We present GFWeb, a novel system designed for large-scale, longitudinal measurement of the Great Firewall (GFW) of China's Web censorship capabilities. GFWeb enables comprehensive, large-scale measurements of the GFW's HTTP and HTTPS blocking mechanisms over an extended period. Over 20 months, GFWeb tested 1.02 billion domains, detecting 943K and 55K pay-level domains (PLDs) censored by the GFW's HTTP and HTTPS filters, respectively. This represents the most extensive dataset of censored domains discovered to date, many of which had not been identified by previous studies. The findings from GFWeb highlight the need for comprehensive measurement systems that can capture the full spectrum of the GFW's censorship mechanisms. Systems relying on a single protocol or bidirectional interference may miss significant aspects of the GFW's operations. Future measurement efforts should consider testing from both sides of the GFW and across multiple protocols to obtain a complete picture of Web censorship.
In the ever-evolving landscape of global Internet governance, the Great Firewall (GFW) of China stands as one of the most sophisticated and opaque systems of Web censorship [2]. It employs an array of filtering mechanisms that continuously adapt to new evasion strategies and emerging technologies. This article expands on our research paper presented at USENIX Security '24 [1], which introduces GFWeb—a novel system designed for large-scale, longitudinal measurement of the GFW's Web censorship capabilities. Analyzing measurement data collected by GFWeb over 20 months, from February 2022 to September 2023, our work provides a deeper understanding of the GFW's mechanisms, evolution, and the broader implications for Internet freedom.
The GFW is a multifaceted censorship apparatus deployed by the Chinese government to control the flow of information on the Internet. It employs various techniques, including DNS poisoning [3, 4, 5, 6, 7, 8, 9], keyword filtering [10, 11, 12, 13], and TCP/IP blocking [10, 14, 15, 16, 17], to enforce its policies. Understanding the GFW's operational details is crucial for developing effective censorship circumvention tools and fostering a free and open Internet. China's Internet censorship framework, conceptualized in the late 1990s as part of the Golden Shield project [18, 19], serves as the government's tool for controlling the flow of online information. Often dubbed "the Great Firewall", this system comprises middleboxes distributed across border autonomous systems (ASes) and managed in a centralized manner [11, 13, 20].
The GFW's on-path devices intercept and inspect network traffic, selectively blocking connections based on predefined blocklists. Its Web censorship mechanisms span multiple layers, covering all three fundamental protocols of the Web: DNS, HTTP, and HTTPS. DNS poisoning is done by exploiting the race condition of UDP-based DNS resolution, injecting false responses when it detects DNS queries for censored domains. HTTP filtering inspects unencrypted Web traffic and blocks connections based on specific keywords or host names. HTTPS filtering examines the Server Name Indication (SNI) field during the TLS handshake to block encrypted connections. The filtering middleboxes begin tracking a TCP connection's state when it sees the first SYN packet from the client and, upon detecting a censored domain in the HTTP Host header or the Server Name Indication (SNI) extension in the TLS Client Hello, tears down the connection by injecting RST/ACK packets to both the client and the server. Figure 1 illustrates the GFW's DNS, HTTP, and HTTPS filtering mechanisms.
Measuring the GFW's censorship presents significant challenges due to its sophisticated and evolving mechanisms. Prior measurement efforts have been limited by scope, time, and/or methodology. Most studies focused on specific aspects over short periods, leading to gaps in understanding the GFW's dynamic censorship landscape. The GFW's stateful blocking, residual censorship, and asymmetric interference further complicate large-scale measurement efforts. Furthermore, relying on volunteers or public servers for data collection introduces potential risks, highlighting the need for innovative measurement approaches like GFWeb to continuously monitor the GFW's Web censorship at scale.
GFWeb is designed to uncover the domain blocklists used by the GFW for Web censorship. It leverages the GFW's bidirectional and loss-tolerant blocking behavior to test hundreds of millions of domains monthly.
The design of GFWeb is depicted in Figure 2, illustrating its probing approach using machines located at both sides of the GFW. The system initiates with the collection of domain test lists from various sources, including domain zone files, the Tranco list, the Citizen Lab test lists, and the Common Crawl dataset. These domains are used by probing machines located outside China to send SYN and PSH/ACK packet pairs towards controlled machines inside China. The loss-tolerant design of the GFW allows GFWeb to trigger censorship responses without completing the three-way TCP handshake, enabling efficient and scalable probing.
When the GFW detects a censored domain in the HTTP or HTTPS request, it responds by injecting three RST/ACK packets to both the client (probing machines) and the server (controlled machines inside China). The responses help identify which domains are censored by observing these injected packets.
The probing process is continuous and iterative, with steps to analyze the fully qualified domain names (FQDNs) detected as censored and subsequently creating regular expressions to understand the blocking patterns. The system then performs additional probing to confirm the censorship of pay-level domains (PLDs) by sending more SYN and PSH/ACK packet pairs and observing for RST/ACK responses. The process is run from both sides of the GFW, allowing GFWeb to scale up its measurement and uncover the comprehensive set of domains censored by the GFW.
Note: A pay-level domain (PLD) is the highest level of a domain name that can be directly registered and controlled by an individual or organization. It's typically the part of the domain name immediately below the public suffix. For example, in www.example.com, “example” is the PLD, while in www.bbc.co.uk, “bbc” is the PLD. PLDs are used to identify the primary domain of a website and are more precise than second-level domains (SLDs) as they accommodate various global domain structures.
Evolution of the GFW: Our study reveals that the GFW has been upgraded to mitigate several issues previously identified by researchers, such as overblocking. More specifically, while developing GFWatch, our previous system for DNS censorship measurement presented at USENIX Security '21, we observed that the GFW was using overly broad regular expressions to block domains, such as *torproject.org, which resulted in collateral damage by also inadvertently blocking unrelated domains that included the string “torproject.org” within their names, such as mentorproject.org and ventilatorproject.org, which have no association with the Tor Project. This broad matching caused more than 41K innocuous websites to be inaccessible in China, highlighting a significant flaw in the GFW's censorship mechanism. With our new system, GFWeb, we observed that the GFW has since addressed this issue indicating an ongoing evolution in its filtering policies.
Cross-Protocol Blocking: Our study also reveals significant insights into the cross-protocol blocking behavior of the GFW, showing that the blocklists for DNS, HTTP, and HTTPS are not identical but rather complementary. As shown in Figure 3, the GFW's HTTP filter has the largest blocklist, followed by DNS and HTTPS filters. Each filter operates on different blocklists that complement each other, forming a comprehensive censorship framework. We find an average of 528K and 24K base censored domains per month for HTTP and HTTPS filters, respectively. GFWatch discovers an average of 247K base censored domains per month. The differing sizes of these can be attributed to the distinct technical characteristics, implementation costs, and strategic purposes of each protocol's filtering mechanism.
Specifically, HTTP traffic is unencrypted, allowing for more granular content-based filtering using Deep Packet Inspection (DPI). This fine-grained control means the GFW can inspect the entire content of HTTP requests and responses, leading to a larger and more specific blocklist. HTTP filtering can block individual pages based on specific keywords, resulting in a higher number of blocked domains.
DNS filtering operates at the forefront of the GFW, manipulating DNS queries to block access to certain domains even before any HTTP or HTTPS traffic is initiated. It is the least expensive in terms of resources because it involves simple on-path inspection and forgery of DNS responses. This mechanism blocks fewer domains compared to HTTP but still plays a critical role in the overall censorship strategy.
HTTPS traffic is encrypted, limiting the GFW's ability to inspect the content of Web communications. The GFW relies on the Server Name Indication (SNI) field in the TLS handshake to identify the domain of the visited website. This makes HTTPS filtering more complex and operationally expensive, as DPI techniques are less effective when it comes to parsing TLS packets with many more header fields compared to HTTP packets. Consequently, the HTTPS blocklist is smaller, focusing on high-priority domains that warrant the additional resource overhead.
Our findings of different blocklists across protocols can lead to straightforward circumvention strategies. For example, if a domain is blocked only by the DNS filter, importing the domain's IP address into the hosts file can bypass the DNS block. Similarly, if a domain is blocked by the HTTP filter, accessing the domain via HTTPS can circumvent the block. For instance, the domain geti2p.net is blocked by the DNS and HTTP filters but not by the HTTPS filter. By (1) importing the domain's correct IP address into the hosts file to bypass the DNS filter and (2) accessing the domain via HTTPS, one can still reach the website. This underscores the importance of understanding the GFW's multi-layered filtering architecture for effective circumvention strategies.
Asymmetric Interference and Prolonged Residual Censorship: Our study also uncovered two new behaviors of the GFW's Web censorship: asymmetric interference and prolonged residual censorship. Asymmetric interference refers to the GFW's blocking behavior that is not bidirectional and symmetric. We identified approximately 1,000 domains that trigger HTTPS filtering only when probed from inside China, but not from outside. This finding challenges the conventional belief that the GFW's filtering mechanisms are bidirectional and symmetric, posing a risk of false negatives in external measurement systems that do not account for this asymmetry. For instance, domains related to circumvention tools or sensitive content, such as torproject.org, exhibit this selective blocking, thereby eluding detection when probed externally.
Additionally, the GFW demonstrates prolonged residual censorship, wherein once a domain triggers asymmetric blocking, subsequent connections with the same three-tuple (source IP, destination IP, and destination port) are consistently dropped for an extended period—up to 350 seconds—rather than merely injecting reset packets. This behavior creates extended periods of unreachability for affected domains, complicating continuous probing strategies used by automated evasion tools like Geneva. The prolonged traffic dropping can mislead these tools into concluding that an evasion strategy is effective when the censorship still actually persists. These findings emphasize the need for comprehensive and nuanced approaches to measuring and circumventing the GFW's evolving censorship tactics.
While the ultimate goal of our study is to investigate the GFW's Web censorship, we encountered some interesting cases of localized network interference by major cloud providers and ISPs in China.
Hosting Providers’ Redirection: Our measurement machines in China are located in the data centers of two major cloud providers: Aliyun and QCloud. While analyzing network traffic collected by GFWeb, we found that both providers have deployed DPI middleboxes to interfere with HTTP connections and attempt to redirect users to warning pages when certain domains are requested despite the location of the request client (i.e., inside or outside China). Specifically, Aliyun injects a redirection to batit.aliyun.com/alww.html and QCloud injects a redirection to dnspod.qcloud.com/static/webblock.html as shown in Figure 4. These pages warn that the domain being requested is not registered with the Chinese government. For a domain to be hosted from within China, it needs to be registered with the Chinese Ministry of Industry and Information Technology (MIIT) and obtain an Internet Content Provider (ICP) license.
Both providers’ DPI middleboxes are deployed as on-path devices and intercept packets in a stateless manner. In other words, these redirection injections can be triggered without initiating a complete TCP handshake (i.e., packets ❶-❸ in Figure 1(b)). An HTTP PSH/ACK packet (❹) with the payload of a trigger domain is sufficient to prompt the middleboxes to inject a redirection towards the side of the connection that sends the PSH/ACK packet. Bock et al. recently found that this behavior could be weaponized for TCP-based amplification attacks since the injected warning pages are much larger compared to the original HTTP request.
Over the course of our study, Aliyun and QCloud middleboxes have interfered with 36.5M and 39.1M FQDNs, respectively. Clustering these domains by their PLDs, we find a total of 7.8M and 6.8M PLDs whose HTTP requests were redirected to Aliyun and QCloud ICP warning pages. It's worth noting that this interference is not caused by the GFW and should not be considered as nation-state censorship. Instead, they are caused by the hosting providers enforcing the ICP license requirements since a benign domain like example.com also triggers the redirection of QCloud's middleboxes.
China's “Anti-Fraud” Redirection: In late 2021, there were some anecdotal reports of new network interference events across major state-owned ISPs in China, including the three largest operators: China Telecom, China Unicom, and China Mobile [21]. Many users reported that their browsing sessions were redirected to a warning page showing an “Anti-Fraud” message. As indicated in Figure 5, the warning page advises users that the site they are trying to access is suspected of fraud and asks them to install an app developed by the State Anti-Fraud Center from the Android or the Apple app stores.
GFWeb has also observed this network interference. Applying the limited time-to-live (TTL) probing approach, we confirm that it is indeed caused by the ISPs. While we did not have direct access to vantage points within these ISPs, one of our measurement machines in China is in a data center whose upstream provider is China Telecom, allowing us to observe the redirection injected by this ISP's middleboxes. Similarly to the GFW, they can inspect and inject packets bidirectionally. They are also deployed as on-path devices and have a loss-tolerant design (i.e., can be triggered without a complete TCP handshake).
GFWeb observed a total of 2.3M redirection attempts caused by China Telecom's middleboxes. In addition to two URLs reported on the Tor Project's GitLab (i.e., URLs 1 and 2 in Table 1) [21], we have also observed six other URLs that share the same pattern but were injected with lower frequency. Our data also indicates that URLs ending with parameter1 and parameter2 were not deployed until February 2023.
| Index | # Triggered | Redirection URLs |
|---|---|---|
| 1 | 1.7M (75.2%) | 182.43.124.6 |
| 2 | 182K (7.9%) | 182.43.124.6/fzyujing?parameter |
| 3 | 123K (5.3%) | 0.0.0.0/fzyujing?parameter1 |
| 4 | 86K (3.7%) | 0.0.0.0/fzyujing?parameter |
| 5 | 74K (3.2%) | 182.43.124.6/fzyujing?parameter1 |
| 6 | 67K (2.9%) | 182.43.124.6/fzyujing?parameter2 |
| 7 | 26K (1.1%) | 0.0.0.0 |
| 8 | 6K (0.3%) | 0.0.0.0124.6/fzyujing?parameter |
Table 1 also lists some injected URLs that we deem as “buggy” because they contain either an invalid or non-routable IP address (i.e., 0.0.0.*). We believe that these URLs are a result of misconfiguration because the injection of these URLs will not lead to any redirection, and thus they are not effective for the ISP's intended purpose.
GFWeb observed 478K unique FQDNs that trigger these injections. Still, we could not find any patterns that could explain why these domains were targeted, since they only triggered the redirection for a short period of time. Looking up the IP addresses of these domains, we find that less than half of them are associated with an IP address, while the remaining are either not associated with any IP address or not existing (i.e., NXDOMAIN). Instead of targeting specific domains that are truly associated with fraud, this observation suggests that the redirection could have been randomly triggered in an opportunistic manner to redirect users to the warning page, persuading them to install the government's “anti-fraud” app. This is evident by the fact that one of the trigger domains was baidu.com, which is obviously not fraudulent.
The original anecdote also reported that OONI and Censored Planet observed this network interference happened via DNS injection as well. However, we could not reproduce redirection cases caused by DNS injection of 182.43.124.6 anymore. This strengthens our hypothesis that the redirection is performed in an opportunistic manner and does not target any particular domains for a long period of time. At the time of writing, we are still observing this network interference from China Telecom and will continue monitoring for any change in the future.
Censorship Measurement: The discovery of asymmetric interference underscores the importance of conducting measurements from both sides of the GFW since filtering policies can be different depending on the probing direction and the domain being tested.
Furthermore, measurement systems that function based on continuous probing against remote servers using the same destination ports will need to be aware of the two different types of residual censorship (i.e., “keeps injecting” and “keeps dropping”) and take appropriate actions to avoid incorrect inferences. More specifically, the residual censorship that “keeps injecting” packets may cause false positives as subsequent benign probes would still trigger the GFW to emit forged packets, whereas the residual censorship that “keeps dropping” subsequent traffic may cause false negatives due to the absence of forged packets that are usually anticipated. The co-existence of these two types of residual censorship in HTTPS filtering also suggests that it is non-trivial to determine whether a domain is blocked or not based solely on the presence or absence of forged packets if residual censorship is not taken into account. To that end, it is important to design measurement approaches that can sidestep the residual censorship to avoid both false positives and false negatives.
Censorship Circumvention: The GFW's Web censorship is composed of multiple layers of filtering based on different blocklists and protocols. While various efforts have attempted to circumvent the GFW's Web censorship at different layers, an effective circumvention solution will need to tackle the GFW's multi-layered filtering architecture. Otherwise, circumvention solutions that only target a single filtering layer may not be sufficient to achieve the desired result.
Similarly to the suggestion for censorship measurement, probing-based evasion techniques like Geneva or DeResistor also need to be aware of the two different types of residual censorship to avoid being tricked into thinking that the censorship has been successfully evaded when it is actually not, especially when the residual censorship is of the “keeps dropping” type.
Using Measurement Data: Internet censorship measurement is a challenging task. Each measurement system is designed with different resources and constraints. Consumers of censorship measurement data (e.g., journalists, researchers, and policymakers) should be aware of the strengths and drawbacks of each system and consider multiple measurement results from different protocols and data sources to obtain a more complete picture of the censorship landscape. When it comes to determining the censorship status of a domain, it is important to gather results from multiple systems and protocols to obtain a more conclusive result.
DNS, HTTP, and HTTPS filtering middleboxes together form the primary pillars of the GFW's Web censorship. In this work, we present GFWeb, a longitudinal measurement system designed to discover domain blocklists used by the GFW for censoring Web access. Over the course of 20 months, GFWeb has tested over a billion fully qualified domains, and detected 943K and 55K pay-level domains censored by the GFW's HTTP and HTTPS filtering middleboxes, respectively. Our study not only complements prior efforts by providing a more comprehensive view into the GFW's Web censorship over time, but also reveals several new findings, including the GFW's asymmetric blocking behavior and patches of overblocking.
The implications of our investigation extend beyond academic circles, touching on the fabric of global Internet governance and the ongoing struggle for digital freedom. The adaptive nature of the GFW signals a future where Internet censorship will become more nuanced and technically complex, posing significant challenges for circumvention technologies and international policy efforts.
In light of these insights, our work underscores the need for a reinvigorated approach to understanding and combating Internet censorship. The dynamic between censorship and circumvention is not static; it evolves as part of a larger geopolitical and technological landscape, with implications for global Internet freedom, the free flow of information, and the resistance against digital authoritarianism.
As we keep operating GFWeb, we hope that our data will not only provide fresh insights into technical observations but also promptly update the public regarding changes in the GFW's blocking policies and support other initiatives, especially those focusing on censorship detection and circumvention.
