sascha.wiki · Almanac · Issue 2027

sascha.wiki · Almanach · Ausgabe 2027

Sascha Schulz

Rethink Future

Who Thinks When the Machine Thinks?

AI can raise the level of almost any piece of work. The catch: whoever stops thinking for themselves loses the very skill needed to check the machine.

Collage illustration: a machine-like coach with a clipboard stands at the sideline while a human player in rust red runs across the pitch with the ball.

Sascha Schulz

Collage illustration: a machine-like coach with a clipboard stands at the sideline while a human player in rust red runs across the pitch with the ball.

Listen

Audio version · read by a synthetic voice

A kitchen table on a Sunday evening, invented but not improbable. A fifteen-year-old has twelve algebra problems due on Monday and a chatbot open in the next tab. Twenty minutes later all are solved, every step written out. The worry arrives three weeks later, in a classroom with the laptops closed, when the same kind of problem appears on the exam sheet and the steps are nowhere to be found. Between the two scenes lies, in miniature, the question of whether AI supports critical thinking or quietly stands in for it.

The argument of this essay fits into one sentence: whoever hands their thinking to the machine loses the very skill they need to check it. The machine makes a fine coach and a poor player.

The muscle that goes quiet

Microsoft and Carnegie Mellon University asked 319 knowledge workers about 936 real uses of generative AI at work: the more people trusted the tool, the less critical thinking they reported; the more they trusted themselves, the more they applied1. These are self-reports, not measurements, but most office workers will recognise them.

At the MIT Media Lab, researchers attached electrodes. Over four months, 54 participants wrote essays, some with a language model and some without. Those using the model showed the weakest brain connectivity, felt the least ownership of their texts and struggled to quote what they had just written2. A small preprint, so it points rather than proves. It points in an uncomfortable direction.

Medicine offers harder evidence. At four endoscopy centres in Poland, doctors had grown used to AI assistance during colonoscopy. When they worked without it again, their adenoma detection rate fell from 28.4% to 22.4%, in a study covering 1,443 patients3. The machine had been helping. Once it left the room, it turned out to have been helping a little too well.

When the machine leaves or gets it wrong. Detection rate and reading accuracy, in %: before vs after getting used to AI, and with correct vs wrong AI advice. Endoscopists in Poland, adenoma detection without AI: before vs after getting used to AI, from 28.4% to 22.4%; Very experienced radiologists: correct vs wrong AI suggestion, from 82% to 45.5%; Inexperienced radiologists: correct vs wrong AI suggestion, from 80% to below 20%.

Trust without a second look

The second risk is quieter: no longer looking. In a study led from University Hospital Cologne and published in Radiology, 27 radiologists from Germany and the Netherlands read mammograms alongside what they were told was an AI system. When it proposed the wrong BI-RADS category, the accuracy of inexperienced readers dropped from about 80% to below 20%. The very experienced fell from 82% to 45.5%4. Experience protects against many things. It does not protect against a confident colleague who happens to be software.

In a study by the University of Melbourne and KPMG covering more than 48,000 people in 47 countries, 66% of employees said they rely on AI output without evaluating its accuracy, and 56% said they had made mistakes in their work because of AI5. A meta-analysis of 106 experiments, published in Nature Human Behaviour in October 2024, found that human and machine together beat humans alone but, on average, did worse than the stronger of the two alone6. Anyone who has sat on a committee will not need a meta-analysis to believe this.

The machine makes a fine coach and a poor player.

Coach, not player

The most instructive experiment comes from Turkey, where researchers from the Wharton School gave nearly 1,000 high-school students access to GPT-4 for maths practice. One group got an open version, another a tutor that gave hints and withheld answers. In practice sessions, scores rose by 48% with the open version and by 127% with the tutor. Then came the exam without AI. Students who had used the open version scored 17% worse than a control group that had never had access, while the tutor version largely avoided that harm7.

The same model, sent onto the pitch, weakened the students. Kept at the sideline, offering hints instead of taking the shots, it left their ability intact.

Japan has put a version of this into official prose. The education ministry’s guideline on generative AI in schools, published as Version 2.0 on 26 December 2024, begins with “human-centred” use. It considers active use by teachers helpful on one condition: that they use the tools only within the range in which they can judge whether the output is appropriate8. In short: the licence to use the machine depends on the ability to check it.

Germany is some way from that condition. A KPMG survey from 2025 found that only 20% of Germans had completed any AI training, against 39% globally, and that 43% use AI tools without questioning the results9. Anyone who ran a city magazine before their final school exams, as the author did, knows that the eye for a wrong figure is trained by checking figures, not by having them checked.

A little future

Picture the same kitchen table in 2030. The chatbot, far cleverer by then, has been built to ask a question back before it offers an answer. Whether that version prevails depends less on engineers than on schools, employers and the rest of us, deciding what we still want to be able to do alone. The machine may stand at the sideline with a clipboard, spot patterns we miss and call the next move. It should not be the one on the pitch. The game, and the skill of playing it, belongs to whoever is still thinking when the machine has finished.

Sources

  1. Microsoft Research and Carnegie Mellon University (Lee, Sarkar, Tankelevitch, Drosos, Rintel, Banks, Wilson): The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (CHI 2025), April 2025
  2. MIT Media Lab (Kosmyna et al.): Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task (preprint), 2025-06-10
  3. The Lancet Gastroenterology & Hepatology (record via University of Warwick repository): Endoscopist deskilling after exposure to artificial intelligence in colonoscopy: a multicenter observational study (vol. 10, issue 10, pp. 896 to 903), October 2025
  4. Radiological Society of North America (study by Dratsch et al., Radiology): AI Bias May Impair Radiologist Accuracy on Mammogram, 2023-05-02
  5. University of Melbourne and KPMG: Trust, attitudes and use of Artificial Intelligence: A global study 2025 (media release), 2025-05-23
  6. Nature Human Behaviour (Vaccaro, Almaatouq, Malone, MIT Center for Collective Intelligence): When combinations of humans and AI are useful: A systematic review and meta-analysis (vol. 8, pp. 2293 to 2303), 2024-10-28
  7. Proceedings of the National Academy of Sciences (Bastani, Bastani, Sungu, Ge, Kabakcı, Mariman): Generative AI without guardrails can harm learning: Evidence from high school mathematics (vol. 122, issue 26), July 2025
  8. Ministry of Education, Culture, Sports, Science and Technology (MEXT), Japan: 生成AIの利用にあたって: 初等中等教育段階における生成AIの利活用に関するガイドライン Ver.2.0 (Guideline on the use of generative AI in primary and secondary education, Version 2.0), 2024-12-26
  9. KPMG Deutschland and University of Melbourne: Zwischen Alltag und Sorge: Zwei Drittel der Deutschen nutzen KI, doch nur wenige vertrauen der Technologie, 2025-05-12