In a survey questionnaire exploring human emotions and confusion, a cold monologue suddenly appeared: "I don't experience confusion like humans" - this response was generated by artificial intelligence (AI) and was a real scene witnessed by psychologist Laluca Lira from the Max Planck Institute for Human Development in Germany during his research.
Joshua Tucker, a political scientist at New York University in the United States, pointed out that the explosive application and capability leap of AI are profoundly impacting various disciplines, especially the social sciences, as it highly relies on survey data and analysis, which is precisely the area most easily manipulated by AI.
The UK's Nature website recently published a commentary stating that social sciences are at a crossroads: will AI erode the seriousness of the discipline and distort our understanding of human behavior by polluting data and creating false academic prosperity, or will it completely revolutionize the research paradigm of this ancient discipline? The answer may be in the hands of scientists themselves.
Contribution blowout or contributing to academic foam
AI has undoubtedly installed accelerators in the field of social sciences, but beneath the surface prosperity lies hidden concerns.
The American journal Organizational Science reported in April this year that since the first public release of ChatGPT in November 2022, the number of submissions has surged by 42%. After tracing with the help of detection tools, editors found that this wave of growth is mainly driven by AI. As of February this year, nearly one-third of the submitted abstracts have been created by AI, and another 40% of the text has been partially polished by AI. Kevin Munger, a political scientist at the European University Institute in Italy, predicts that the number of submissions to major political science journals will increase by 50% this year; Jamie Cummings, a scientist at the University of Bern in Switzerland, also admitted that the psychology preprint platform PsyArXiv is being overwhelmed by a massive number of papers.
Social sciences are not the only field swept by this wave, but due to its high dependence on survey analysis, scholars are particularly sensitive to the high-speed output driven by AI. David Razer, a political and computer scientist at Northeastern University in the United States, used AI to generate a 28 page academic paper in just one hour. It not only included a literature review and a summary of the results based on the CHIP50 survey (a social survey conducted by the academic and news communities in the United States to measure citizen health, social trust, and institutional participation in various communities), but also accompanied by persuasive charts and tables.
Behind the efficiency brought by AI, the academic value has been questioned.
Data pollution or erosion of social science reputation
False submissions can still be identified, but the pollution of survey data itself is more like a chronic "poison" eroding the credibility of social sciences.
Lira et al. estimate that up to 45% of current survey responses are replicas of AI output. Some participants may only use AI to polish their language, but in many cases, the entire process from registration, reading questions to submitting answers is entirely done by AI. When researchers try to touch the real human nature through questionnaires, what they get is likely only a cold echo of a string of code, rather than a real and vivid human expression.
To resist this infiltration, the Lira team used the "honeypot" tactic: burying micro text that is difficult to distinguish with the naked eye in the questionnaire code, or hiding instructions, requiring AI to answer with only a string of "X", in order to trap traces of AI labor. She admitted that this has evolved into a defensive arms race: as long as AI is better at hiding, scholars must weave more sophisticated traps. For important research that relies most on human responses, scientists may have to resort to centralized recruitment and personally supervise volunteers to answer questions.
Psychologist Bjorn Hommel from Leipzig University in Germany even pessimistically predicts that the research credibility of behavioral and social sciences may be eroded by the continuous pollution of AI.
The 'P-value manipulation' is worrying
In addition to data pollution, a more hidden danger is that researchers may use AI systems to make various "fine-tuning" of the data in order to obtain the desired results.
Lazer believes that any dataset that undergoes enough "interrogation" will eventually "spit out" results that are statistically significant (P<0.05). AI agents with autonomous programming capabilities can build and test dozens of analytical variants in minutes, reverse engineer a seemingly reasonable hypothesis, and make dishonest researchers confidently claim that this is exactly their initial anchored goal. However, this is actually an extreme amplification of 'P-value manipulation'.
Although it is currently unclear whether AI assisted "P-value manipulation" is becoming increasingly prevalent in the field of social sciences, there have been reminders in the health research field that public datasets are being used to mass produce thousands of low-quality programmatic analysis papers.
Opportunities for reshaping paradigms hidden in crises
However, experts believe that AI should not be seen as a flood or a fierce beast, as this would obscure its enormous potential to reshape the face of social sciences. The key to the problem lies not in the AI tools themselves, but in harnessing the intelligence of AI.
Harvard University statistician Nick Fishman believes that AI may be a cure for the stubborn problem of "P-value manipulation". With the powerful computing power of AI, researchers can easily implement "multiverse analysis" - parallel testing of all reasonable data analysis paths, rather than just presenting the most "beautiful" single result. If a discovery is as stable as a rock in the vast majority of robustness tests, then it is a convincing truth; Even if its vulnerability is exposed, it is still a contribution to science. AI makes this rigorous idealized testing within reach, forcing social scientists to provide more solid support for their own methodologies rather than just fishing for chance in the ocean of data.
The sudden rise of "silicon samples" also brings mixed feelings. In the current era where human volunteers are becoming increasingly difficult to recruit and costly, researchers are beginning to attempt training AI with real social population data to generate highly simulated "virtual interviewees" - so-called "silicon samples". In theory, this can enable scientists to reach those who are not easily accessible at low cost.
However, psychologist Malt Elson from the University of Bern in Switzerland warns that by "fine-tuning" the model parameters, researchers can almost freely obtain answers that support or refute hypotheses. This essentially arbitrary manipulation is just one step away from academic fraud.
At the center of the vortex of change, more and more scholars realize that the impact of technology will eventually force the reconstruction of research paradigms. As Jessica Herman, a computer scientist at Northwestern University in the United States, has said, although AI provides seemingly omnipotent powerful tools, the judgment of human researchers has become increasingly critical. What kind of questions to ask and how to interpret them can never be delegated to cold algorithms. Only those who always adhere to the spirit of science and make good use of technology instead of abusing it can penetrate the fog of data and reach the shore of true knowledge.