Let’s set a baseline: if you understand the mindset of your customer, you know full well whether a product you’re looking at is one they’d want to buy. Technical knowledge is strictly unnecessary.
That’s, of course, not extremely helpful to a manager looking to figure out if an in-progress product is going in the right direction. I want to know if development is proceeding apace before a (potentially useless) product is “completed”. Heck, I want to know what stock I can put in whether or not that done” status will actually happen anywhere near the estimated time frame.
Hiring (aside)
First thing’s first, we have to build the right team. It’s fairly established wisdom at this point that technical interviews are ideal - give a sample project small enough to complete in a reasonable time frame. Ask candidates to write a piece of code and then discuss it with them.
Here’s some basic advice (as this isn’t the main trust of this post), though this does largely require a competent and trusted technical interviewer.
Things to look for
Can they do the basics?
Communication style
How do they respond to a change in requirements?
Are they moving in a direction that will eventually yield a working solution to a difficult problem?
Things to avoid looking for
Being a stickler for syntactical correctness (we have compilers, linters, documentation, and search engines for this)
Trivia questions, brain teasers, and in-group jargon (even when they relate to computer science; if someone can write decent SQL they understand the difference between DML and DDL, even if they don’t use those terms regularly)
Years of experience with your chosen toolchain (everyone is entitled to preferences, but developers who aren’t able to learn new technologies easily are going to need to be micromanaged)
How to bootstrap, though? How do you find and build trust with the type of person who can conduct these interviews and take responsibility for building products that work?
Bootstrapping
If you’re forced to just go off of a resume, references, and a non-technical interview …
Successfully shipping code
You’re looking less for commercial success and more for “did the product do what it was designed to do?” The Wright Brothers flight at Kitty Hawk didn’t carry any paying passengers. Conversely, the vast profits generated by big tech provide cover for more than a few failed projects that never got off the ground.
There is no shortage of people with impressive (at first glance) resumes who’ve never shipped a single project. They worked on niche or experimental products at big businesses that were ultimately dropped in cost-cutting seasons. That’s not necessarily a red flag - I can’t infer that Apple canceled its Mars mission because of engineering failures - but it’s not a green one, either. The iRover’s lead software developer never proved themself.
Diversity of references
You want someone who is trusted both by other engineers and by business partners. That balance is important: engineering teams' can have an insular view, and business partners don’t always know who is responsible for success versus who just conveyed it to them.
Do internal demos early and often
A project that’s on the right path will have something that can be shown to managers in the first few weeks. (one week is short, one month is long)
The question you should be asking is does this work end-to-end? If that answer isn’t “yes” by your first or second demo, the project is in trouble. Once it is a yes, the questions then become “is this better than last time,” “is this improving fast enough,” and “are we working on the right things.”
Analogies and pitfalls
Putting myself in your shoes, let’s pretend I’m touring a startup car factory (a topic I know next to nothing about).
I’m looking for the ability to turn the thing on, see that it produces sufficient torque, and see that it can be steered left and right.
I’m not looking for:
“Attention to detail.” Installing comfy leather seats and high-res digital dashboard on a prototype car that doesn’t yet roll is a distraction.
Look at all those wires and blinking lights! This is complex and above my head; it must be good.
People looking busy. This is easy to fake, or more likely, people may be hurriedly going nowhere.
A demonstration of third-party components. A high schooler can connect a battery to a motor and get it to spin. They’re not on their way to being a car manufacturer.
These are all easy traps to fall into:
“I don’t know anything about code but I do know what a good website looks like, I’ll use that as a proxy for the quality and direction of the prototype.” Sure. I don’t know anything about medicine and I do know what good handwriting looks like, but it’s not how I’d choose a surgeon.
The idea that “this is too complicated for you to understand” is a common technique of fraudsters. More prosaically, it might be an indication that someone’s work is too complicated for them to understand (and thus incoherent), if they’re unable to explain it.
Most people will generally agree that a roomful of programmers sitting at their desks for long hours and producing lots of lines of code doesn’t equal productivity, but it’s just so automatic. This is how Hollywood would portray a bustling office; we need to constantly remind ourselves that Hollywood sets aren’t producing anything besides films.
The last one is possibly the trickiest, especially in an industry that prides itself on code reuse and de-duplication of efforts. I’m not suggesting that you build your own data center, develop your own front-end framework, design system, CRM, etc. I am saying that merely standing up a React app on AWS and launching a popup window to Salesforce doesn’t mean you’ve actually done anything of value, even as a prototype.
The last two are best thought of in concert. “The most important decisions you make are not the things you do, but the things you decide not to do.” Necessary advice, but simply deciding not to reinvent the wheel is not sufficient to make one Steve Jobs. At a certain point, there’s no substitution for understanding your supply chain and value proposition.
Once you’ve gotten off the ground
The first step is the hardest - once you’ve gotten a thing working end-to-end, and you’re able to iterate, measuring progress is much more approachable.
Story points are great, but they only work in the context of an existing team and at least prototype product, and they’re only meaningful in relative terms. Think of them like prices - doubling your inventory or producing a fancier widget with a price tag that’s 50% higher doesn’t mean anything until you’ve started selling the widgets.