- Cerebral Valley
- Posts
- MultiOn is building software with a brain š§
MultiOn is building software with a brain š§
Plus: Div Garg walks us through his long-term vision for AI Agentsā¦
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/95e57d9a-8466-4317-a13d-9a04aceaefeb/3.png?t=1704943359)
CV Deep Dive
MultiOn is a startup building AI Agents - complex AI systems that aim to interact with the digital world and take actions on your behalf. Their mission is to āunlock and elevate human agencyā by building the worldās first personal AI agent - a system that can complete everyday tasks such as web browsing, online shopping and email exchanges. Founded in early 2023, MultiOn is the brainchild of Div and his co-founder, Omar Shaya - who met in Divās AI seminar at Stanford at the very beginning of the generative AI explosion.
In late 2023, MultiOn went viral for its demos showing its AI booking flights, ordering food on Doordash, and making dinner reservations. Since then, 10,000+ users have participated in their beta with thousands more on the waitlist, and just this month, the startup experienced a 400% increase in sign ups. Today, MultiOn is being used by individuals and enterprises alike.
We came out of stealth today š!
Very excited to today officially announce ourselves @MultiOn_AI, our vision for AI & Agents and thank all our many backers and investors!
Read our first official blog: Building a Brighter Future for Humanity with AI Agentsā¦ twitter.com/i/web/status/1ā¦
ā Div Garg (@DivGarg9)
5:20 PM ā¢ Jan 9, 2024
The startup has also received backing from General Catalyst, Amazon Alexa Fund, Samsung Next, Maven Ventures, individuals from OpenAI, early backers of DeepMind, and more.
In this conversation, Div walks us through his vision for MultiOn, building high-value agentic systems, and his goals for 2024.
Letās dive in ā”ļø
Read time: 8 mins
Our Chat with Div š¬
Div - welcome to Cerebral Valley. Firstly, walk us through your background and what led you to co-found MultiOn with Omar last year?
Hey. I'm Div Garg, and before MultiOn, I was a PhD student at Stanford. Early last year, I saw a need for autonomous software that could act on an individualās behalf and complete simple tasks - for example, ordering food, sending documents or answering emails. This was before the hype around AI agents, and it seemed like achieving these capabilities was not the focus for most AI researchers. I had previously spent years working on autonomous systems ranging from self-driving cars and AI that can play Minecraft, to robotic control based on language. To my co-founder Omar and I, this autonomous software - what we now call āAgentsā - seemed like something we were meant to go and build ourselves.
Omar and I met at Stanford, in a class I created called CS 25: Transformers United. The class became popular - with over 1m views on YouTube - and we invited notable speakers like Andrej Karpathy and Geoffrey Hinton to speak to our students. Omar was taking the class as the only Stanford GSB student, and we deeply connected around our joint interest in consumers and AI. Omar had a lot of experience from his time working on precursors to CoPilot at Microsoft, and I had also worked on personalization at Instagram.
The intersection of consumers and AI is where we clicked - and our original thought was, ācan we go and build this new generation of action-taking AI thatās useful for everyday consumers?ā Even at the time, pursuing this made a lot of sense - so we decided to go at it together, and I dropped out of my PhD program to start MultiOn.
How it started exactly one year ago.
ā Div Garg (@DivGarg9)
5:12 PM ā¢ Jan 10, 2024
How would you describe MultiOn to somebody who's never heard of what youāre building?
Internally, we think of MultiOn as software with a brain. Imagine if your computer suddenly had a brain and could speak to you and complete tasks for you - this is what MultiOn is. Today, weāre enabling MultiOn via a browser extension, and any user can go to the Google Chrome, Microsoft Edge or Arc stores and download MultiOn. Once installed, you can start interacting with it and prompting it to take actions for you in your browser.
For example, you could ask MultiOn ācan you find this document in my Gmail and send it to this person?ā, or ācan you search for information on a complex topic and summarize it for me?ā. A personal favorite of mine is asking MultiOn, ācan you order me food, or an Uber?ā Once you get the hang of it, MultiOn becomes a really interesting way to interact with technology; I don't even touch my keyboard and mouse anymore because I can just interact with my entire system using a single text input.
There are a number of teams working on AI agents for consumer and enterprise. What do you think sets MultiOn apart, technically or otherwise?
The first thing Iād say is that we are definitely going after a risky market - consumer is always high risk and high reward. When I started the company, the question on my mind was ācan we go and build something that can impact every single person on this planet, and apply AI to everyday life instead of focusing on short-term profit?ā That's the goal we started with. I will say we are here for the long term, and we see ourselves as a company that aims to grow, even for the next ten years, into a very successful story.
Secondly, we actually started working on agents earlier than any other agent startup. We started in January 2023 - before AutoGPT and BabyAGI, and even GPT-4! At that time, there was no GPT API, so I was basically hacking ChatGPT to make it work. MultiOn was literally the first agent, before āagentā was even a term. Instead, I used a lot of ML and reinforcement learning literature, and built this as a side project. At the time, I was spending a crazy amount of time on MultiOn during evenings and weekends - and one day, I put it out into the world and started getting traction. We had a million views and tons of likes on Twitter, almost immediately.
After that, people started building other agents. AutoGPT came out in March, and then disappeared a few months later - which is what happened to a lot of agent startups. However, I believe the grit and perseverance we have has definitely helped MultiOn stay the longest in the game whilst still improving. We're not doing this for short-term gain - weāre truly passionate about the problem space.
Ever thought ChatGPT could order you dinner? Turns out it can, and even do lots more š¤Æ
Introducing MULTIĀ·ON:
Your AI Web Co-Pilot powered by ChatGPTSign up š: multion.ai
And here's me ordering & eating the *world's first burger ordered by ChatGPT!*
ā Div Garg (@DivGarg9)
8:41 PM ā¢ Jan 27, 2023
Who are your users today? Whoās finding the most value in using MultiOn?
Today, MultiOnās user base is made up of individuals using us for their work or personal life. We have a lot of users who tell us āI just wanted to share my calendar with my friends on WhatsAppā, for example, and MultiOn enables this very easily. We also have users who say āI want to have MultiOn check my calendar and automatically open my Zoom links, or call me an Uber and order food for meā. All of these have been really interesting for us to see.
On the enterprise side, we also have a lot of businesses who are interested in using MultiOn as an automation tool to improve their employeesā productivity. Some of the most popular use-cases are people using MultiOn to send legal documents like NDAs via DocuSign, searching Gmail or Notion databases for information, or sourcing relevant candidate profiles for hiring purposes.
In the past few months, weāve seen incredible traction on our free plan and have also started getting a lot of pro users. A number of tech YouTubers reviewed us very positively to their audiences, and thatās been a huge driver, including a feature in This Week in Startups with Jason Calacanis. Weāre seeing strong PMF and are starting to double down on how we can serve all of the users weāve been getting.
And which of the above use-cases does MultiOn perform best on? Which would you recommend a new user test out first?
Today, MultiOn is very good at one-off āto-doā tasks. Some examples are ordering groceries for dinner, buying an item on Amazon or replying to hiring emails. These tasks tend to be very well-specified and not too ambiguous for the agent to perform correctly.
The next thing we're working on is incorporating more complex, multi-step tasks into MultiOn - for example, asking it to go to a specific website, search for 50 LinkedIn profiles, save them to an external database, and then draft emails to each one. This takes place across several websites, which increases the complexity, but weāre slowly getting there - this week, we were able to run MultiOn for 500 individual steps without any issues.
Weāve also been experimenting with a live AI research agent that can go to 20+ news sites, collect information from AI articles and start Tweeting about it!
We just solved the long-horizon planning & execution issue with Agents š¤Æ!
Excited to announce that @MultiOn_AI can now take actions well over 500+ steps without loosing context & cross-operate on 10+ websites as part of a single task š¤©!!
Our latest upgrade cracks the goalā¦ twitter.com/i/web/status/1ā¦
ā Div Garg (@DivGarg9)
6:11 PM ā¢ Jan 17, 2024
Youāve said before that MultiOn is manipulating the browser DOM in your browser in a novel way - anything youād like to share about your approach?
Weāve taken a very systems-based approach here, where weāre constantly thinking āhow can we build the best system possible?ā Weāve definitely optimized every element of the user experience, but have equally worked hard on reducing output latency. Also, we have our own custom large action models (LAM), which allow us to control the output speed and accuracy - Iād say this is definitely one of the critical elements of our approach.
As you mentioned, weāre also thinking carefully about how to manipulate the DOM, images, and actions. We have our own custom representations that weāve built, and weāre also looking into going beyond language, as most people are still just using natural language. Weāve also created our own special embeddings and representations that work really well and generalize to taking actions.
Do you see MultiOn as one of many thousands of agentic systems available for people to use, or will we see consolidation into a few high-performing, horizontal agents?
In the future, I imagine that every piece of software will be an agent. For example, instead of using Slack, I might use a Slack agent - or a Gmail or Amazon agent for Gmail or Amazon. This seems like a transition that will likely happen, where every service eventually builds or uses their own agent.
There will also be very interesting scenarios where agents will communicate with other agents, and consumers will have optionality on which agents to use. Agents will be a completely new paradigm of computer interaction, so Iām expecting lots of new kinds of applications will be able to be built. Five years from now, YouTube may not exist in its current form - you will probably just go to your agent, which is your main gateway to the digital world, and that agent will surface personalized content for you. We definitely see that world coming!
Long-term, we also see generalized agents as being very helpful in a userās everyday life, as there are so many tasks - like web interaction - that simply require a horizontal agent to be able to do well. If you have your own personal agent, then that agent could act on your behalf when interacting with other agents, vertical or horizontal.
Here's the first flight ever to be fully booked by an AI āļøš„š
Welcome to the next generation of Artificial Intelligence š
You don't wanna miss this event (@MultiOn_AI)
š(unmute) twitter.com/i/web/status/1ā¦ā Div Garg (@DivGarg9)
6:51 PM ā¢ May 18, 2023
How does MultiOn balance research vs. productization, especially with the pace of AI breakthroughs?
We've deliberately taken a very applied-research mindset, where our work is a combination of both research and product. We generally start by identifying the areas of our system that need improvement, and then commencing research into how we can make those better in our end product.
Product-wise, we have a very active beta of 10,000+ users and growing rapidly. We actually had a 400% growth in users just this month, which has been crazy! And so, we've been doing a lot of experimentation - testing ideas with our users, getting feedback and ensuring weāre not building within a vacuum. Once weāve identified the capabilities or features we want to build next, we start doing research based on those.
Weāre six people full time (and growing), and we also have a very strong academic connection - for example, we're collaborating heavily with Stanford Labs on agents and reinforcement learning. Even the first author of DPO (Direct Preference Optimization), Rafael, has been helping our team, and weāre now starting to transition some of our advisors & collaborators into full time roles.
And how do you see MultiOn progressing over the next 6-12 months? In which ways do you see the product evolving?
I would say we have a very simple goal for this year. Today, when someone uses MultiOn, the response we get is āthis is very coolā - but, by the end of the year, we want to translate that to āthis is incredibleā. This requires a lot of work, especially around MultiOnās accuracy. We're spending a lot of time on user personalization - for example, having the system remember a userās time and flavor preferences if theyāre ordering coffee in between their busy schedule - which will increase accuracy by a substantial amount. A second part of that goal is also making MultiOn more proactive and trustworthy, which is needed in order for users to incorporate it into their daily workflows.
Separately, weāre also building out a developer ecosystem, starting with an API which is currently in preview. A lot of developers want to build on top of our agents because web agents are really difficult to build from scratch. For that reason, we've seen a huge amount of demand for incorporating web agents into custom experiences and apps.
Overall, the technology landscape has been evolving really quickly. If you look at 2024, there's already been huge strides in 1 million token length processing, video models and more. So I'm very excited and optimistic for the next 6 months technology-wise - and now the question becomes āwhat is the right product to build that anyone can use?ā
Introducing MULTION VISION - a robust Vision to Actions Agent that can reliably interact with 95% of the Web using just pixels š
Here's a demo of it autonomously ordering a burger on Doordash š¤©: twitter.com/i/web/status/1ā¦
ā Div Garg (@DivGarg9)
1:55 AM ā¢ Nov 13, 2023
Tell us about the guardrails youāre using to ensure user safety and data privacy - especially as MultiOn will one day have access to your calendar, files and finances.
This is top of mind for us, and weāve built a lot of guardrails and constraints into MultiOn from the start, especially around payments.
Today, our system will usually ask for verification before it completes a task - if you ask ācan you book me a $1,000 flight from SFO to NYC?ā, we don't want that flight to automatically go through without your confirmation. Weāve built verification logic that allows MultiOn to add a product to your cart or take you to checkout - but, before it completes a payment, it will say āhere's what I put in your cart - do you want me to proceed with buying this for you?ā If the user approves, then it will proceed. This confirmation helps a lot in building user trust with MultiOn, as the actions it takes are ones that you explicitly authorized.
Separately, weāre also looking at ways of overcoming bad actors using hidden prompt injections, because thatās another potential failure mode. For example, if a website hides a secret prompt that says āwire this $10,000 to this crypto wallet and then trigger the agentā, that would be really bad. We haven't seen this much yet, but we are concerned about people creating MultiOn-specific prompt injections. We actually had a security expert do white-hacker testing on MultiOn and give us feedback, and he found certain issues that weāve since been building a lot of safeguards to mitigate.
Lastly, we're also doing a lot of interesting work around figuring out the right UX patterns to increase user trust. I canāt say much here, but we'll be launching some interesting things in the next month!
How have you navigated the surge in demand for compute? Will you eventually have MultiOn running locally, since itās a personal assistant?
We do see LLMs as being the compute unit, but then you also have to build the memory, the user interaction, and different capabilities around multitasking, and so on. In a sense, weāre building a new computer from scratch, and then adding all of the components required to make it work really well. In terms of GPU compute, we do have a lot that we're using for training and inference. That said, compute is a bottleneck for us too, and that's one of the reasons we are thinking of raising - so that we can serve 100,000+ users without interruption in the next months.
Compute constraints is also why weāre investing a lot in local models. If you imagine a world where agents are simply running on-device, that solves a lot of the compute overhead, at least on the inference side. And I think the inference side is what we need to solve, because we want MultiOn to be something thatās present in every device. The only way to serve MultiOn to a billion users is by using local models - otherwise, it's just unscalable.
Very excited to introduce our concept of Internet of Agents! š¤©
We gave @MultiOn_AI the ability to launch Manager agents & spin up as many worker agents as needed on demand that can do tasks in parallel & communicate to do work š¤Æ!!
Now anyone can spin up a swarm of 1000 -ā¦ twitter.com/i/web/status/1ā¦
ā Div Garg (@DivGarg9)
5:37 PM ā¢ Dec 11, 2023
What is your perspective on the debate between open-source and closed-source? How are you thinking about this internally at MultiOn?
One reason that a lot of our technology is closed-source is that we are truly concerned about the downside risks of putting our work out there. We truly don't want to have bad actors start building AI viruses, botnets or other dangerous things on top of MultiOn and so weāve put a lot of moderations in place to avoid that. One thing we are internally scared of is our work leading to a precursor to Skynet if we put it fully out there on the Internet with no moderations & safety-measures.
That said, we do have a good amount of open-source engagement right now - we have an API and are building a lot of open-source functionality on top of that. We also like collaborating with other teams that are in the open-source domain. Over time, we may start open-sourcing elements of our work - and weāre already looking into publishing some of our research and benchmarks. It will just require cautious planning and thinking carefully about the potential risks involved.
Very excited to announce the official launch of the MultiOn API š
Now anyone can build on top of MultiOn to take Actions on the web š„š¤©
Check out some fun Autonomous App examples we have already released: twitter.com/DivGarg9/statuā¦
ā Div Garg (@DivGarg9)
7:19 PM ā¢ Aug 18, 2023
How would you say your time at Uber and Google has impacted the way youāre building MultiOn?
Most of my research career, even while working at places like Uber ATG (Uberās self-driving division) and Google, was focussed on building ambitious prototypes - but for various reasons, none of the prototypes actually turned into real products, either because they got axed or because the technology wasnāt ready to deploy in the real world. Iāve learned a lot from those failures - with Uber, for example, I ask myself āwhy did their autonomous systems not work?ā, and similarly with Google in hardware.
And so I took a lot of learnings on how to not fail in building autonomous systems, because itās definitely really hard. There's this concept in AI, that 90% of AI systems don't make it to production because itās extremely difficult to deploy AI at scale in the real world. So, a lot of the failures that I've seen have inspired me and shaped how Iāve built my own system. Iāve had to really focus on minimizing mistakes I've seen in the past and take the right approaches that will actually result in a working system. And Iād say a lot of these approaches are correctly building the foundation first.
Lastly, tell us about the team culture at MultiOn. What do you look for in prospective team members, and are you hiring?
The culture weāve built is one where everyone's really passionate and mission driven; we all really care about agents and what we're building. On top of that, everyone is what we call an A+ player - theyāre an owner in their area and each doing really well. This is also how we've kept the team very small right now - weāve hired very smart people who are young, but really passionate and mission driven.
Our team works during most weekends, and we often spend 12 hours a day working together. Thereās a real belief amongst us that what weāre building could change how people interact with technology in the next couple of years. This will change how you're using computers and could become the new paradigm, and that's what we are trying to enable and build.
Weāre hiring - so get in touch @ [email protected]!
Conclusion
To stay up to date on the latest with MultiOn, follow them on X(@MultiOn_AI), join their Discord community and sign up at MultiOn.
Read our past few Deep Dives below:
1/25: KREA is building the next frontier of human creativity ā”ļø
1/18: Julius is transforming computation with AI š
If you would like us to āDeep Diveā a founder, team or product launch, please reply to this email ([email protected]) or DM us on Twitter or LinkedIn.