Loading
Why do simple business questions take weeks to answer?
Because in most companies every answer is assembled by hand: someone hunts for where the data lives, waits for access, exports it, stitches spreadsheets together and reconciles the numbers that disagree - and then does it all again for the next question. Data engineering builds that factory properly: pipelines bring the data to one governed home, cleaned, joined and current, so a question becomes a query. On a platform like Databricks, the hunting, waiting and repairing are engineered away once - and the three-week question becomes a three-minute one.
On a Monday morning, someone senior asks a simple question. "Which of our products actually lose us money once returns are counted?"
It is a fair question. The data exists. Sales live in one system, returns in another, product costs in a spreadsheet that finance keeps carefully up to date. Everyone in the room nods, someone says "we'll pull that together," and the meeting moves on.
Three weeks later, an answer arrives. It is presented carefully and caveated twice - "depending on how you count bundles" - and half the room quietly doubts it anyway. Nobody did anything wrong. Everyone involved was competent and worked hard. That is the uncomfortable part: three weeks and a shrug is what this way of working is built to produce.
Follow the question on its journey and the time explains itself.
First comes the hunt: working out where the data actually lives - which system, which export, whose spreadsheet, which of the three files named final is final. Then the wait: asking for access, and asking again, because the person who grants it has a job of their own. Then the assembly: exports and downloads, copied into spreadsheets, columns that almost line up, product codes that almost match. Then the repair: the numbers that disagree, the missing week, the returns that have no matching sale. And then the argument: two teams, two totals, one meeting to decide whose number the answer will use.
Only the last stop is actual thinking - and it gets whatever time is left.
Look at where the time went. Almost none of it was spent answering the question. It went into reaching the data, moving the data and repairing the data - by hand, again, the way it went for the last question, the way it will go for the next one.
The slow part of a slow answer is never the thinking. It is reaching the data.
Step back far enough and you can see the machine. Every company runs an answer factory: questions go in, numbers come out. In most companies, that factory is made of people. Analysts stitching exports together. A finance manager reconciling totals at month-end. A developer doing a favor, again, because only they know where the real table is.
The factory works, barely, and its costs never appear on any budget line. Decisions wait on answers that arrive after the moment that needed them. Sharp, expensive people spend their weeks as couriers between systems. And every answer is a little different from the last one, because hand-built things always are.
The biggest cost is the quietest one. When answers take three weeks, people learn to stop asking. The follow-up question - the interesting one, the one the first answer was supposed to provoke - never gets asked, because nobody wants to be the reason for another three-week project. A company cannot see the questions it lost. There is no report for curiosity that gave up.
When answers are expensive, people stop asking questions.
The instinctive fix is more effort. Hire another analyst. Buy another dashboard tool. Ask everyone to be more careful with the spreadsheets. None of it works for long, because the factory is not understaffed. It was never built.
Companies engineered their operations long ago - nobody processes an order or runs payroll by hand. But the data those systems produce was left to fend for itself, and the gap is filled by hand labor. Data engineering is the discipline that builds the missing factory: the part of the company that makes answering fast, boring and repeatable.
↺ returns to Collect
Built once, this loop replaces the factory made of people.
Built once, this loop replaces the human courier work entirely. The reaching, moving and repairing still happens - but it happens inside the machine, on schedule, the same way every time. Answering stops being a project and becomes a query.
None of this is theoretical. It is what a modern data platform is for, and Databricks makes a concrete example.
On an estate like this, the three-week question is a query someone runs before the meeting ends. Not because anyone worked harder - because the hunting, waiting, stitching and repairing were engineered away once, for every question that follows.
One caveat, and it matters. The platform is the building, not the factory. What makes the estate trustworthy is the engineering done inside it: pipelines built well, checks that reflect how the business actually counts things, definitions agreed once instead of re-argued per spreadsheet. A platform bought and left empty changes nothing.
The platform is the building. The engineering is the factory.
The return on an engineered data estate is usually sold as speed: reports that used to take weeks now take minutes. That undersells it.
The real return arrives slowly, in behavior. When an answer costs minutes, people ask again. They ask the follow-up. They check the odd hunch that would never have justified a three-week project. Curiosity stops being expensive, so there is more of it - and a company that asks more questions of its own business simply knows itself better than one that learned to stay quiet.
Fast answers change what a company dares to ask.
The three-week question was never really about the three weeks. It was about every question that went unasked because of them. Build the factory, and you do not just get your answers faster. You get your curiosity back.
With one painful question, not a platform program. Pick a question the business asks every month, engineer its path end to end - data flowing in on schedule, checks applied, one governed table, a query anyone can run - and let the result argue for the next one. The factory gets built one production line at a time.
Before - and AI raises the stakes. An AI assistant answering from hand-stitched, disagreeing data industrializes the confusion. The same engineered foundation that makes human answers fast is what makes AI answers trustworthy. Build it once and both get better.