Nova AI - your QA agent 🎛

Plus: CEO and Co-Founder Zach Smith on how AI-powered QA can revolutionize the developer experience...

February 13, 2025

CV Deep Dive

Today, we’re talking with Zach Smith, Co-Founder and CEO of Nova AI.

Nova AI is a QA agent designed to revolutionize end-to-end testing with AI. Founded by Zach and his co-founder Henry Li, Nova leverages cutting-edge generative AI and user session data to automatically generate and maintain accurate tests based on real-world user experiences. Built on a Kubernetes-first architecture, Nova delivers enterprise-grade flexibility and security while providing fast, reliable testing solutions that integrate seamlessly into existing workflows.

Today, Nova has already seen adoption by enterprise clients, including a multi-billion-dollar e-commerce giant, helping teams improve testing coverage, streamline QA processes, and achieve significant ROI gains. With features like real-time session data analysis and click-to-deploy infrastructure, Nova is empowering businesses to tackle complex testing challenges at scale while reducing overhead.

In this conversation, Zach shares how Nova came to life, the challenges of building a product for enterprise use, and his vision for how AI-powered QA can revolutionize the developer experience.

Let’s dive in ⚡️

Read time: 8 mins

Our Chat with Zach 💬

Zach - welcome to Cerebral Valley! First off, introduce yourself and give us a bit of background on you and Nova AI. What led you to co-found Nova AI?

Hey there, my name is Zach, and I’m the CEO and Co-Founder of Nova AI. I’ve been an engineer my entire career, primarily focused on building for developers. Whether it’s internal customer tools, APIs, or developer tooling, that’s always been my sweet spot. Over the years, I’ve worked on a range of technologies, particularly in CI/CD (Continuous Integration/Continuous Delivery), where the goal was helping developers ship more reliably and faster at scale.

My journey has taken me through several financial services companies like MasterCard and TD Ameritrade (before it was acquired by Schwab). After that, I joined a Bay Area startup called Armory, which worked on CD technology using Spinnaker, a Netflix open-source project. It was during this time that I gained a deeper understanding of the biggest bottlenecks in delivery pipelines. While companies can have the most sophisticated release automation in place, if QA and other validation processes aren’t equipped to keep up, releases remain blocked across all applications. Automated quality checks and manual testing often became persistent obstacles, preventing smooth and continuous delivery despite advancements in deployment technology.

Eventually, I joined Google to work on Distributed Cloud, a private, air-gapped Kubernetes-based cloud for governments and financial institutions. Later, I worked on an innovation project before leaving to start Nova AI.

As I mentioned earlier, my co-founder Henry and I have seen a lot of the challenges around developer productivity and releasing software efficiently. With generative AI, we’re seeing something unprecedented over the last two decades—a big enough ROI gain compared to legacy competitors and systems. Traditionally, you’d be aiming for a 5%, 10%, or maybe 15% marginal improvement, but now, with AI, we’re able to achieve 5x, 10x, or even 20x efficiency gains. That’s what’s finally enabling companies to invest in testing efforts and see meaningful ROI.

After meeting with a few hundred developers and VPs of Engineering, we were able to confirm some of our initial thoughts, which ultimately led us to start Nova. There was a slight gap between when Henry and I joined forces—I left my previous role a little before Henry—but we ended up partnering right around the pre-seed stage. I like to say we started at the same time because, in spirit, we did. That’s a bit about me and how the company came together.

How would you describe Nova to the uninitiated developer or AI team?

Nova is a QA agent that automatically generates and maintains end-to-end tests with AI, based on user behavior. 

One of our big differentiators is bringing together observability and infra CI/CD expertise. We realized that session data or user behavior data was missing from what all these testing companies were doing. Why aren’t they using that? When we started incorporating it, we saw 20, 30, 40, even 50% increases in accuracy across all our benchmarks. It became clear that we needed to use this and let our customers use it too. At a high level, it’s a QA agent for automatically generating and maintaining end-to-end tests based on critical user experiences.

Who are your users today? Who is finding the most value in what you're building at Nova? 

We learned pretty quickly that early-stage startups are usually excited to try shiny new tools, but typically don’t have the time or budget for end-to-end testing. Henry and I are both engineers, not salespeople, so while these smaller startups were willing to test the product, it rarely translated to consistent usage or meaningful revenue. Early on, we had a hunch that our product would be best suited for enterprise customers. Coming from large companies ourselves, we knew how it should work for that space.

We initially tried to get traction with smaller startups but realized it wasn’t the right fit. So, we took a step back and shifted our focus to Series C and Series D companies and beyond. These are the businesses that start to allocate budget, assign headcount, and invest in tools to solve problems like this. That’s when we doubled down on targeting small, medium, and large enterprises. Just yesterday, we inked a deal with a multibillion-dollar e-commerce giant that does $10 billion in annual revenue.

This kind of customer represents our ideal fit—they face the types of problems we know best, and our solution strikes the right balance between flexibility and customization without the overhead and complexity you’d typically see with tools like LambdaTest or BrowserStack. In a nutshell, we’re fully focused on enterprise customers now.

Which existing use-case for Nova has worked best? Any customer success stories you’d like to share? 

One company we’re working with is a massive physical retailer. One of the things that really resonated with them was our session data feature. When we released it a few months ago, we introduced the ability to monitor and track user session behavior data, which allows us to help generate and automate the creation and maintenance of tests more effectively.

After implementation, they began receiving millions of requests and events weekly, uncovering key insights about user behavior. It became clear that their assumptions about feature usage were often incorrect. Engineers and QAs could now see firsthand how users interacted with their platform, leading to more effective testing and improvements.

One major discovery was a critical feature with no test coverage. In the first meeting after enabling session data, the team immediately noticed it appearing frequently in the logs. They were shocked to realize how heavily it was used and how much of a testing gap existed. This was a powerful validation of how session data can drive better decision-making and quality improvements.

Which existing use-case for Nova has surprised or delighted you the most? How are you measuring the impact or results that you’re creating for your customers? 

When developers write tests, they usually talk about coverage—like, what's the percentage of coverage? They look at how many buttons are on the site and how many are covered by tests. But that’s not really how things work in the real world. Users clicking through the site might have completely different sequences of events, like adding something to a cart and then checking out. A lot of buttons aren’t actually used in those pathways. So, we had to rethink what coverage really means.

Coverage, to us, is about whether the buttons or workflows users are actually using are being tested and working properly. If a workflow breaks—like adding to a cart and checking out—your users can’t buy anything, and they’ll leave your site and go to a competitor. You might lose that business forever.

It’s not just about making sure the code works; we’re laser-focussed on maintaining and improving the user experience. Developers need to make sure the code they ship won’t break anything critical, but also that it enhances the experience so users stick around. To measure this kind of impact, we realized we couldn’t just talk to developers and QA teams—we also needed to involve product teams and leadership. Everyone needs to work together toward revenue goals and ensure the business grows.

This means building new things while maintaining what’s already there. Users need to trust that the site will work, and the teams—whether it’s QA, engineers, or product managers—need to understand how their work directly impacts the business. It’s all about keeping things running smoothly for users while meeting the company’s goals.

Could you share a little bit about how Nova actually works under the hood? 

We’re a Kubernetes-first platform. Like I mentioned earlier, we’ve always been biased toward building for enterprise, so we knew we needed full customization across the platform. Enterprises have so many different requirements—security, scaling, you name it. Instead of using a serverless platform, we decided we needed control over all the nuts and bolts, which is why we went with Kubernetes from day one. The whole platform runs on Kubernetes. We can move it between clouds or install it air-gapped in a customer’s environment. That’s the architecture style we chose.

From an AI perspective, we’re not tied to any specific vendor. What matters most to us is performance. Right now, the biggest factor is time—how long it takes to get a response back while still delivering great results. There’s a lot of room for improvement in that trade-off. For example, some testing products use an AI agent to run real-time steps, but that can take forever if you’re waiting 15 seconds for an API call to OpenAI. Instead, we use the AI agent to generate the steps, but once that’s done, subsequent runs don’t use AI. This makes everything much faster because we don’t have to wait for those slow API calls.

​​Right now, we’re focused on picking the best model off the shelf. For customers who need even more accuracy, we can fine-tune based on their session data to get really strong results.

The AI models we’re seeing are constantly being updated, with new versions coming out all the time. The expectation is that every few months, an even better model will drop, which pushes the whole industry forward. We’re still in the early stages of this, and while what we’re doing is really exciting, we know there’s so much more potential.

As new models are released later this year, we’re confident we’ll continue to improve both as a company and in the service we provide. We already have a lot of great ideas that we can’t implement just yet but will likely be able to tackle soon. It’s an exciting time, and we’re looking forward to putting these advancements into developers’ hands.

How do you see Nova evolving over the next 6-12 months? Any specific developments that your users should be excited about? 

Our pipeline has grown like 10x or 20x over the last couple of months. Without getting into specific numbers, it’s shaping up to be a big sales year for us. We’ve found a sweet spot in the market where we can actually compete and win over the competition.

This year, we’re really focused on sales and continuing to drive innovation. We like to partner closely with our customers—meeting with them every week if possible—to figure out what’s working, what’s not, and what areas we can improve. We’re constantly iterating on the product based on that feedback.

We’re also raising our seed round, which is a big priority right now. After that, it’s all about sales and innovating for our customers.

How would you describe the culture at Nova? Are you hiring, and what do you look for in prospective team members joining the company? 

We’re hustlers from day one. You have to be able to put in the hours—working until midnight, 2:00 a.m., 4:00 a.m., whatever it takes. You’ve got to have the passion for it and tough skin. This journey is a roller coaster. Just a few days ago, we thought a deal we’d been working on for seven months was about to get kicked down the road for six months. We were devastated. Then, of course, it landed. It’s the highs and lows. You need to have tough skin to know you’re going to get beat up sometimes, but you’ve got to be able to pick yourself back up and keep moving forward.

That’s the number one thing we look for in a team—hustling, working hard, all that good stuff. You’ve got to be able to put in the time. We’re hiring for some go-to-market roles and on the engineering side, specifically full-stack with kubernetes, golang, and frontend experience.

We’ve been around the block, worked at different places, and seen the pains. I think we have a really unique mindset of asking, “What’s your pain? What’s your problem?” We want to understand those problems so we can build the right solutions. We’re not here to say, “This is the solution you need.” A lot of our customers want to work with us because we take the time to really understand their issues. We meet with them every week to ask, “What’s your problem? Did the solution we gave you actually solve it?”

If not, we ask, “What improvements do we need to make? Is it a smooth experience? How can we improve?” That’s our approach—customer first, always. It’s been key for us. At the same time, we’ve got the engineering chops to step in and actually solve the problem, as I mentioned earlier.

Anything else you'd like our readers to know about the work you’re doing at Nova?

The one thing we’d like to call out is that while we’ve talked a lot about the existing offering and what we’re doing for enterprise, we’re really excited about our QA desktop application, especially in comparison to Nova Cloud. We’d love for people to try it. We’re planning to launch it on or around the time this article goes live. So, as a call to action, I’d say: we now have this free tool that lets anyone easily do what we’re doing at scale, provide feedback, and build alongside us. That’s something I’d love to highlight if possible.

Conclusion

Stay up to date on the latest with Nova, learn more about them here.

Read our past few Deep Dives below:

If you would like us to ‘Deep Dive’ a founder, team or product launch, please reply to this email ([email protected]) or DM us on Twitter or LinkedIn.