AI Thinking

When software starts deciding for us

Is AI stealing our Agency?

Katie Lips, CEO and Founder of Rebel Futures/10 February 2026
When software starts deciding for us

Over the past two decades, I’ve worried about many things in the digital world: privacy, security, misinformation, addiction, exploitation, and so on. Yet there’s something else going on at the moment that I don’t feel is receiving attention or being articulated in the right way.

As we move towards AI-driven systems that interpret, infer, and decide things for us, our role in the digital world is changing. Today I’m exploring this shift, not from the technical perspective of how AI works, but from the psychological perspective of how it affects us.

In 2020, my job was to protect consumers in the digital space; a wide remit, indeed. My long career in digital technology meant I’d seen it all, from the early optimism of the web to the acceleration of the explosion of social media via mobile, to the early days of AI. By the early 2020s, chatbots, early LLMs, and voice transcribers were gaining ground. The lines had already blurred between product and consumer; to a greater extent, consumers were already very much ‘the product’ in many e-commerce spaces; our every micro interaction, our views, opinions, ratings, reviews, connections, and influence were analysed, and reanalysed by platforms we’d never see or even know existed.

Consumers (a broad term to describe any of us in buying mode) didn’t love the digital world. My employer, Which?, ran a regular consumer index that expertly surfaced consumer sentiment and trust in digital systems. It was the mixed bag you’d expect; while some people were happy to ‘pay with their data’ for goods and services, others were more cautious. Certain environments, e.g. supermarket loyalty, were reasonably deemed less risky than others, e.g. life insurance and banking. Scams and cybercrime dominated the headlines generally; people balanced their desire to exploit the internet for all it has to offer with not getting too exploited themselves. GDPR was well established, and yet there was still a lot of noise about compliance, specifically via cookie policies. Websites had to show a cookie banner, or else. It seems childishly innocent that that’s what many businesses concerned themselves with at the time. 2020 was not long ago.

What’s come since 2020, since the global pandemic, since remote working, and since economic decline is an internet our 2020 selves might barely know. To many, it may look the same on the outside; there are the facades of websites, mobile apps, SaaS platforms, and government services, but behind the scenes, there’s been a huge shift. Behind the friendly user interface, what happens in these digital environments has, in many cases, changed beyond recognition.

What I’m talking about is the shift in how services operate using AI. Pre AI, software was deterministic; you entered some data, and it gave you back a result. The same result. Every time. Nowadays, many digital services you use simply don’t work like that any more. Instead of being neatly powered by data in a database that will give you the same answer a million times, what you get back when you enter your data, the same data, in the same way, may indeed change each time you enter it. Not only might the results change, but you might not understand why. Not only might you, as a user, not understand why your answer may change each time, but the software itself - and its owners - might also not understand why. No one may be able to explain this to you or to each other. This is where the relationship between humans and software changes. What’s more, the inputs you’ve given, and what you choose to do next, will be used to further train AI-powered systems, affecting the outputs that future users will experience. You are using a fluid system that may or may not ever give you the same answer twice, and will use your data to learn and improve its answers in the future.

If you work in this space, you’ll know this as the much-referenced shift from deterministic to probabilistic outcomes. This shift didn’t begin with generative AI; machine learning systems have been shaping recommendations and rankings for much longer. Yet in recent years, generative and adaptive models have dramatically accelerated the move from predictable systems to ones that do more of the work humans used to do. You might well understand the tech, maybe even how to build AI infrastructure or models, and as an AI leader, you may even understand the impact AI has on organisations and their ability to manage and govern tech platforms. But do we understand the impact AI is having on user relationships with digital services? Or, more simply, the effects this relationship shift is having on people? And do consumers, users, or patients understand what’s going on?

The shift from web to mobile changed human behaviour and led to the feeling, for many, that connection is better digitally than in real life (IRL), that there’s something ‘out there’ for us to attend to at all times. At this point, I can’t stop the mildly moronic image popping into my head of couples sitting in fancy restaurants gazing into their respective smartphones, instead of at each other. So, what will shift from database-powered to AI-powered change for us? AI hasn’t physically moved anything, unlike mobile. We still use the same devices, the same apps and services, even. So is the shift even noticeable yet? But what if Data 1.0’s cosy notion of housing itself in a database suited us humans and our desire to be in control? And what if AI-powered Data 2.0 undermines that and confuses the hell out of us?

I’m no AI hater; far from it. I love the opportunity of new and emerging technology, but as someone interested in human behaviour and psychology, I enjoy understanding its effect.

We should at least attempt to explore the unintended consequences of the platforms we build, right?

In the Data 1.0 world of the humble database, however complex it might be, there was no room for nuance, no alternative answers, no making it up as you go along. There may have been some clever ‘if this then that’ suggestions (“people who read this book also read” was Amazon’s super early trailblazing example of a recommendation engine). As a user, you’d use a service to give you information: product recommendations, ticket prices, weather information, matches for a hot date, bank balances, and then transact: pay someone, connect with someone, book a holiday, buy a dress, etc. You might also share your own content: write a blog post, Tweet, post on Insta, Pinterest, etc.

Over time, of course, social media and e-commerce systems became more sophisticated and could surface content and suggestions in a way that started to feel more fluid, more personal, more timely. Most of us are entirely used to that. Systems became connected, and as consumers, we began to understand that recommendations we might see on one platform were suggested to us because of our interactions on another platform. Simple web retargeting or worrying that the conversation we’d had about starting couch25k means you’re suddenly seeing ads for running shoes on Facebook. Everyone seems to get this, and while worrying that it could feel intrusive, the consensus was that no one knew how to turn it off, so people just put up with it. I fear retargeting alone played a big role in eroding trust in the digital system.

From a consumer perspective, though, and from a subconscious one at that, let’s look at what actually happens when you interact and transact digitally in a Data 1.0 world versus a Data 2.0 world.

First, you trust the system is going to store your data safely and that it won’t come to harm, or get lost or stolen. That’s the most basic assumption. Then you trust the software has been tested ‘enough’ and that any ‘information’ the software gives you is correct. Then, you continue safe in the knowledge that if you came back tomorrow and did all this again, you’d get the same result (dynamic pricing aside). As a relatively intelligent human, you can understand what is happening at a basic level. Even if you are not technical, you can understand that you are asking a question and getting an answer, then performing a well-designed task. Time and again. If there are decisions to be made, you can make your decision based on information you can broadly trust. For example, if you are in the market for car insurance, after you hand over a bunch of detailed data about yourself and your requirements, you are offered information about a list of relevant products.

The important point here is that it is up to you to read between the lines of what is being presented to you. It’s up to you to understand the nuance of why one product may be better than another, and to know that the software can’t tell you everything. It’s up to you to decide. Up to you and your amazing brain, which is capable of processing information, nuance, feelings, and even intuition, all at once, to allow you to make a decision. You use software to perform a simple task. As then, as a human, you do the more complex work of making a complex decision. You have agency.

In a Data 2.0, AI-powered world, however, software does more of the decision-making. The results and suggestions you might see are determined not by the data you entered, but by data from people like you, in situations like yours, presented according to which option is most relevant to you, in that context, at that very minute. Plus, software doesn’t just return information; it interprets, prioritises, nudges, and persuades. You no longer have so much cognitive load. Over time, that can feel helpful, after all, who wants to do more work? But by taking away our need to make decisions, the digital space is taking away our agency. Taking away our evaluation, reflection, nuance, judgment, skepticism, optimism, and doubt.

This is not about AI making “bad” decisions for us; it’s more probably that AI will make statistically better decisions. This is about what happens when our human role in decision-making quietly diminishes; we become less important. Once we understand that as consumers we are playing less of a role in our own decision-making, what happens then? I think we lose trust; here’s why.

When we have more of a role in the decision, we can subconsciously back up a decision to purchase something, to support a cause, to apply for a job, to insure our health, and so on, based on factors we know and understand. When we play less of a role in the decision we feel, and know that that responsibility, that agency is being taken away from us. At this point, we need to blindly trust in the digital system or reject it.

I believe that, as consumers, we need more trust in an AI-powered system than in the 1.0 world.

This isn’t a case of a lack of adoption due to a lack of digital skill or enablement. It’s the danger that if you build a super clever product that does too much for humans, that’s too clever, humans feel stupid. Ignored. Irrelevant.

Digital businesses and their product teams may strive for effective AI governance across safety, accuracy, and compliance with various regulations. Yet they often fail to acknowledge the role agency plays in customer engagement. They fail to offer users the interactions that can drive trust in AI-powered decision-making. I think it’s vital for product leaders, as we build more and more capable software, that we don’t recognise the need to earn trust, and don’t dumb down the role humans want to play in the digital space.

Katie Lips is CEO and Founder of Rebel Futures. She helps organisations grasp the opportunities at technology inflection points, from the early web and pre-iPhone mobile to cloud-based SaaS and now AI. She has held senior roles at EY and Aimia, and was Head of Digital Policy at Which?. Katie holds a Harvard Kennedy School qualification in Leading in Artificial Intelligence and is a Founder Institute graduate.

Related: explore our work on AI Strategy.

base44
Edit with Base44