Smart, data-driven governance or digital dystopia? Inside China’s social credit system
The TV series Black Mirror’s near-future dystopian stories blended familiar elements of modern technology with the fictional, speculative, and often outright scary potential applications of this technology.
The appeal of Black Mirror is that the technology featured in the often chillingly Orwellian nightmares bore more than a passing resemblance to technology we use every day, highlighting the ethical knife-edge society balances on.
For the first time since 2019, in June 2023 a fresh batch of Black Mirror stories are being released. While little information about the contents of the episodes have been revealed ahead of launch, we can be sure for more tales of alarming dystopia that linger in the mind long after.
In the episode, the system runs like a particularly nefarious version of Uber driver ratings. Find yourself with a low rating? No access to that particular restaurant, car, home, travel destination or social event for you. Nearing 5 stars out of 5? Enjoy the privileges of being part of society’s elite circle of influencers!
Rather than being some speculative vision of a distant science fiction future, it turns out that Nosedive was uncomfortably close to reality. A very similar system in China has bought together cutting-edge technology like computer vision, facial recognition, and machine learning algorithms to create a holistic tracking and ranking system unlike any other seen before.
So how does this scary vision of the future work?
Big Brother meets Big Data
China has long been a surveillance state. Depending on who you ask, it is estimated that 54% of the world’s CCTV cameras are situated in China, totalling around 400 million. Other sources put the figure closer to 200 million, which still equates to a chilling 1 camera per 7 citizens. Whichever number is true, it goes without saying that it’s hard to find some private space in the world’s most populated country.
The application for this surveillance system is changing over time, with facial recognition technology, computer vision and machine learning algorithms further extending the potential uses for such established and widespread visual monitoring.
In May 2021, it was reported that a camera system that uses artificial intelligence and facial recognition technology intended to reveal states of emotion in people was being tested in Xinjiang, China. This already throws up numerous ethical points, but this could just be the tip of the iceberg.
The broader system that’s getting a lot of attention in China is the social credit system. First trialled in Rongcheng, this system aims to match together all possible data capture points - your online presence, general behaviour (habits, finances, purchases, hobbies, opinions etc) with facial recognition, body scanning and geo-tracking information to create a full picture of who an individual is and where they are. From this picture is formed a social credit score.
The overall goal of the system is to provide a complete assessment of each individual, or organisation’s trustworthiness.
Cheong-Jia, manager from AliPay, the financial credit system of tech giant Alibaba Group, summarises the broad purpose of such a score:
“Once a person has a score, all their credit behaviour in life is recorded and can be evaluated based on that number. Our goal is to ensure that if people keep their promises they can go anywhere in the world. If people break their promises, they won’t be able to move an inch.”
If that doesn’t sound dystopian enough, the Chinese government issued a planning document for the social credit system in 2015, which featured a line straight out of the Orwellian playbook: “keeping trust is glorious and breaking trust is disgraceful.”
Unlike the Black Mirror episode, this is not a system where people rate each other. This system is purely to be used by the government, and ratings are ascribed and removed based on the government’s view of your actions. Some of the ways you can have your credit scored lowered include, but are not limited to:
Speaking ill of the government on the internet (full-scale, broad-daylight political dissidents are usually locked up)
Going into your overdraft
Tax evasion
Buying too many video games or spending too long playing them
Speeding/parking tickets
Have more than two children
The system can be used for individual people, but other versions of the technology can also be used for companies and government organizations. As of the end of 2020, these “corporate” and government self-discipline areas of the social credit system appear to be more advanced than other, further-reaching areas of use.
As of 2022, it’s estimated that 80% of China is covered by the system. So how does it work?
What do the social credit scores mean?
The aim of the system is to create ‘model’ citizens in the eyes of the Chinese government. Everyone starts with 1,000 points. Anyone who boosts the figure to 1,300 points through good behaviour gets an AAA rating. Anyone whose points total falls below 600, however, lands in the worst category, with a D rating.
Credit scores can change in real time. This can include what people purchase. For example, if a person was to buy a lot of alcohol from a shop it would suggest dependence and they would lose a couple of points. This is particularly troubling when one considers how ratings affect everyday life and access to provisions across society. Higher social credit scores can allow for:
Priority for healthcare and treatments
Priority for schooling
The ability to undertake higher education
Access to property
Access to certain jobs
If your credit score is at the lower end of the scale, you could find yourself losing the above privileges; being under constant scrutiny from government officials, unable to move up the credit ratings for 5 years, financially punished, stripped of qualifications, and at worst on a national blacklist.
An article from Development and Cooperation gives an example of just how widespread the impact of this can be:
“China’s Ministry of Tourism recently revealed that more than 20 million people were denied air and rail tickets last year because their social record was not considered good enough.”
Is this smart data-driven governance or a tool of social control?
Critics of China’s social credit system argue that it marks the complete and final eradication of privacy from peoples lives. Others see the complete data collection that this system produces as beneficial for planning, management, and governance across all aspects of society, including infrastructure, supplies and resourcing.
With data capture points increasing year on year, data-driven governance has the potential to radically reshape how decisions, both proactive and reactive, are made. In The Rise of Data-Driven Governance, Current History, David Chandler highlights the potential benefits of data-driven governance:
“In international governance, data-led approaches are said to have the potential to build self-governing societal capacities for resilience and adaptation. Through real-time reflexive awareness, advocates claim, risks and problems can be managed as soon as they arise. In the international arena, this can be seen particularly in ambitions for the prevention of—and speedy responses to—disaster, conflict, and health and environmental problems.”
Indeed, in China itself the system is proving to be quite popular, predictably most of all among those who are enjoying a high credit score. The Mercator Institute for China Studies reveals:
“Educated and wealthy urban Chinese have an overwhelmingly positive view of commercial and government-run systems that rate the “trustworthiness” of citizens, businesses and social organizations. Rather than perceiving them as instruments of surveillance, they see them as a way to protect consumers from food scandals or financial fraud – and to access benefits connected to a high social credit score.”
While some may also argue that this positive compliance is just a by-product of the system itself (after all, there’s little appetite for protest or dissent in a country where you can be stripped of all social privileges just for negatively talking about the government on an internet forum) there’s no doubt that it can have some uses. In rare cases like missing persons when the window of opportunity to track them can diminish rapidly in the first few hours, facial recognition technology and holistic monitoring can be useful in helping pinpoint their last known whereabouts. But is this worth the price?
Who owns the data?
Needless to say, a system as widespread as this involves the collection of vast swarths of data. Who owns this is a tricky question, as the broad answer seems to be ‘the government’ – whatever definition you apply to that. But the sources of this data and who has access to it are far more tricky to pin down, as D+C explains:
“Everyone will be able to check their own rating. But government agencies, banks, shopping platforms, tour operators and even airlines will also have access to the information. No one can opt out of the system. Everyone will get an account and will be required to register themselves using their social-insurance number.”
Most of this data has come from big tech companies. Alibaba, which sells more than Amazon through online sales, has the data of 800 million people.
The Social Credit System remains the least digitised of China’s tech-driven monitoring and surveillance initiatives. It relies heavily on human investigations, reports, and decisions. This also leaves room for traditional vectors of individual and political influence, and the speed at which this can be done is startling:
“On social-media sites, images are starting to emerge from pilot cities showing what future surveillance operations might look like. In a local police-control room, hundreds of images captured by surveillance cameras on a Beijing intersection appear on a digital wall. Conspicuous behaviour automatically triggers a zoom. The facial recognition software checks the persons’ features against the database and identifies them within seconds.”
A vision of the Western future?
It’s easy to think that this is all just China – but in reality, this could be a vision of a future across the world. Big Tech companies in the US have already landed themselves in hot water over the past decade about data usage and ownership. The comforts and ease of life that tech and Big Data bring us are all well and good, but we’re sure to face more difficult ethical conversations as we stare into the black mirrors of our future.
After years of middling progress, artificial intelligence innovation has rapidly become a hot topic in technology, particularly Computer Vision - a crucial piece of the puzzle in China’s plans for total survelliance. We spoke to Lead Data Scientist Antonios Ntelidakis about his work in this rapidly growing field
To search our latest jobs in technology, click here
