Why Everything You Know About Software Is About to Change

Why Everything You Know About Software Is About to Change

There's a moment, somewhere around 2023, when building software quietly turned into a different kind of job.

You used to be able to draw a picture of your product on a whiteboard and that picture would actually mean something. Input on the left, business logic in the middle, output on the right. Every arrow had a name. Every box had a specification. If you wrote the code carefully, you could stand in front of a stakeholder and say, with full confidence, "When the user clicks this button, this exact thing will happen, in this exact way, every single time."

That sentence is now obsolete.

Not because buttons don't work anymore. They do. Not because software stopped running. It didn't. But because the things you can build — the things your users now expect — include behaviors that no specification can fully describe. A button that "summarizes this document" doesn't return a single fixed string. A search box that "finds what you meant" doesn't always know what you meant. A spreadsheet that "spots the trend" might spot a different one tomorrow, on the same data, for the same user.

The old picture on the whiteboard still works for some parts of your product. But for the parts that matter most — the parts your competitors are already rebuilding — it's the wrong picture. And drawing the wrong picture, again and again, is how good teams ship mediocre AI products.

This chapter is about that gap. We'll look at the kind of software you were trained on, why its core promise is breaking under the weight of probability, and the three concrete shifts you need to internalize before you touch a single Figma frame for an AI feature. We'll then walk through a real-world redesign — Notion's onboarding transformation around Notion AI — to see how this plays out at a company you probably already use. Finally, we'll give you a checklist you can apply on Monday morning to tell whether the product in front of you is genuinely AI-native, or just AI-flavored.

Let's start with what we're actually trying to escape.

The Software You Grew Up With (and Why It's About to Behave Differently)

Open your laptop and think about the apps that defined your career. For many of you reading this, that list probably includes things like a banking app, a calendar, a CRM, a code editor, or a project management tool. Now think about what every one of those apps has in common, regardless of what they do.

They are deterministic.

That word — deterministic — is doing a lot of quiet work in the software industry, and most people never stop to define it. So let me be clear about what it means, because we're going to spend the rest of the chapter unpacking why it matters.

A piece of software is deterministic when the same input, processed by the same logic, will always produce the same output. Click "Calculate" in a spreadsheet, and 2 + 2 will return 4 every single time, on every device, in every country, until the heat death of the universe. Submit a form with valid data, and you'll get the same success screen. Type "Hello" into a chat field and press Enter — the message goes to the same place, in the same order, with the same notification.

This is not a bug. This is the entire foundation that the software industry was built on.

It is why we can write unit tests. It is why we can do code review. It is why we can promise users that "your data is safe" and mean it in a precise way. It is why a bug report from a customer in Berlin can be reproduced by a developer in Bangalore. It is why your CEO can stand on stage and demo the product without fear of the buttons doing something weird.

Determinism gave us a hundred miracles we forgot were miracles.

It gave us reliability: software that does the same thing 99.999% of the time is software you can put a person's paycheck behind.

It gave us debuggability: when something goes wrong, there is a chain of cause and effect that you can trace, log, and replay.

It gave us predictability for users: after two minutes with most apps, you know what every button does. That knowledge transfers. You don't have to relearn how to check your bank balance every time a bank updates its app.

It gave us scalable design patterns: because everyone — users, designers, developers, lawyers — could assume that software behaves consistently, we invented things like buttons, dropdowns, modals, form fields, and breadcrumbs, and we could teach them once and use them forever.

And it gave us contracts: legal, technical, and mental contracts about what the software will and won't do. "Your data will not be shared." "This calculation uses this formula." "Clicking Submit creates an order." Those are statements of fact in a deterministic system. In a probabilistic one, they become much trickier.

Now here's the uncomfortable part. Every one of those miracles was built on a single assumption: that the output of the system can be fully described in advance. That someone, somewhere, has written down exactly what the software will do for every possible input it might receive.

AI doesn't honor that assumption. AI can't, by its very nature.

When you ask a large language model to summarize a meeting transcript, you are not executing a function with a documented output. You are sampling from a probability distribution over all the possible summaries the model could plausibly generate. Sometimes it will give you a three-bullet summary. Sometimes it will give you a paragraph. Sometimes it will include the names of people, sometimes it won't. Sometimes it will invent a detail that wasn't in the transcript (we'll talk about this — it's called hallucination — and we'll talk about it a lot).

That variability isn't a flaw that future versions of the model will fix. It's the engine. It's how the model handles tasks it has never seen before. It's how it generalizes. If you asked it to behave exactly the same way every time, you'd be asking it to stop being a model and start being a lookup table.

So we are not facing a temporary problem. We are facing a category change.

And here is the part that catches most product teams off guard: the things that made deterministic software great are not just weakened in the AI era. Some of them become actively dangerous if you apply them naively. Treating a probabilistic system as if it were deterministic — which is the default instinct for everyone who learned their craft before 2020 — is the single most common mistake in AI product design today.

Let's look at exactly how that mistake shows up.

The Probability Problem: Why AI Breaks Your Old Playbook

Imagine, for a moment, that you are a quality assurance engineer at a company that makes tax-preparation software. Your job is to make sure that when a user enters a W-2 with $50,000 in box 1, the app correctly computes the federal tax owed. You have written hundreds of tests. Each one specifies an input (a number in box 1), an expected output (a specific dollar amount), and a tolerance (zero cents of error).

You can do this because tax software is deterministic. The Internal Revenue Code is a function. Your code is a function. Functions compose. You can test exhaustively, or close to it.

Now imagine that your company decides to add an AI feature: "Ask TaxBot: Should I itemize?" A user types a question. The model reads their W-2 and their deductions and produces a recommendation. Your QA team sits down to write tests.

What does the test look like?

If the user has $80,000 in W-2 income, $12,000 in mortgage interest, and $8,000 in charitable donations, should TaxBot say "Yes, itemize" or "Itemizing might save you money" or "It depends on your other deductions"? Should it lead with the math or with the conclusion? Should it mention state taxes? Should it use a casual tone or a formal one?

The answer to every one of those questions is: the model gets to decide. Not your developers. Not your designers. Not