Our everyday world looks much as it did twenty years ago. Software learned to read, write and reason, and the street outside stayed the same. This is why—and what changes when machines can see.
The world as it is —
When I look at my surroundings, whether I’m at home, in the office, walking down the street or in a car, I can’t help but notice that the everyday world around me is essentially the same as 20 years ago (apart from my phone being upgraded by several generations).
I still curse to myself when I open the fridge in the morning and realize I forgot to buy milk. When cooking scrambled eggs, I wonder if that extra dollop of butter I added will cost me later. As I open my mailbox and see that stack of letters building up, I wonder for a second whether any of them are important. I’m still annoyed that no one has fixed that pothole on the way to work for weeks, and that I still spend hours in traffic when I need to go to the city.
When I arrive at work, I have to slowly drive through multiple car parks to try and spot an open space. At work, I’m wondering why those dirty coffee mugs from yesterday are still in the sink. During a midday doctor’s appointment, I sit and wait for 45 minutes while a doctor reads my scans, using the same medical equipment they practiced on in residency decades ago. When I get home, I free my robot vacuum from where it’s stuck around mirrors and sliding glass doors, still leaving several rooms in dust.
For my father, a professor of mechanical engineering, he’s still using mostly the same CAD tools he used when I was in high school, with the same familiar interface and similar capabilities. He still needs to spend minutes to create a screw hole on a surface in his design, meticulously sketching the overall plan with pencil and paper, before recreating the design using design software, curve-by-curve.
My friend Amy, who is a civil engineer, still needs to read through 250 pages of dense building code documents, and then carefully cross-check code requirements against the relevant CAD designs, to ensure that the stairway to the roof hatch she's designing has a handrail that extends at least 34 inches above the roof.
On a recent business trip, I met a roboticist who is designing an industrial robot testing platform. They told me they are spending hundreds of hours designing just their small component in the platform, and that they believe the whole platform would take several thousand hours of design work.
Humanity has made huge leaps in software, the Internet, the virtual world, coding and math, yet our daily lives are pretty much the same.
This is in stark contrast to our day-to-day lives even 40 years ago, when pocket-sized phones, smartphones and the World Wide Web didn’t exist. Going back to fifty years ago, we didn’t have GPS, IVF, reusable spacecraft or laptop computers. (from a timeline of historic inventions).
The way forward —
Innovating and making progress in the real world is much harder than in the virtual world. I worked on an open-source MMO back in high school and loved how easy it was to mold a whole new virtual world, city, industry or marketplace from scratch. Back in the real world, just connecting a cable modem to my computer can be an entire ordeal. I have to think about how I’m going to make the wire travel diagonally across the house, while going around doors and avoiding creating any trip hazards. Then, after all that, I realize I can just pass it through the roof instead. Even buying furniture is messy: I have to plan where it’ll sit, take all the measurements, and compare dimensions against door widths, which turns furniture shopping into a semi-mundane task.
Enabling software to see, understand, and reason about the world can help solve our biggest problems as a society. As my former Google Brain colleague, Dario, points out in his "Machines of Loving Grace" essay, biology is probably the area where scientific progress has the most potential to improve human life. But, as an addendum to that, bringing these breakthroughs to the clinic or operating room is fundamentally bottlenecked by vision. Software that has vision and intelligence could advance robot-assisted surgery to the point where both outcomes and patient wait times (which have been growing steadily even for urgent procedures for decades) are improved dramatically. While there are many scientific and regulatory hurdles that must be overcome first, it is clear that visual intelligence is on the critical path.
Software with vision is vital to building robots that help with elderly care in an aging population and proactively notice visual warning signs, like a change in gait, energy or expression. It could help filter and suppress hidden shocking and violent content on social media, to prevent the exposure of such content to young children. Finally, it could help produce entertainment and gaming for underserved populations by making it easier to leverage existing video libraries, translate into rare languages, and simplify the video editing process.
I’ve always believed in the power of AI to expand the reach of humans. The gap between where we are today and where we wish we were tomorrow can be bridge—and that tomorrow can be vastly expanded through AI.
A close illustration of this belief lives in the short story Escape! in the I, Robot series by Isaac Asimov. In the story, a group of scientists are working on the first interstellar drive and they are trying to use their AI brain to help build it. The AI builds the drive but when the scientists are testing it, they are surprised it is able to transport them to a different star system almost instantly, with no feeling of acceleration. It turns out that the AI invented a whole new kind of technology needed to accomplish humanity’s goals. That technology goes on to become foundational to the rest of Asimov’s stories. We’re still far from those kinds of interstellar leaps in technology, but one day I want AI to have as much direct impact on the physical world as it does the virtual.
Dare we imagine an AI that has the same spatial reasoning, perception and understanding that the average person has, both in terms of speed and complexity?
This type of visual AI would transform my life, the lives of my family, friends and coworkers, and the entire world.
To get to strong visual AI, we need to design models for more than just object recognition, detection or segmentation (basic skills that are essential for all animals). We need models that think natively in the real world (whether with images or video), that can encode video naturally (rather than sampling frames), that can internally visualize any problem as part of their reasoning, that have architectures that are efficient on more than just 1D sequences and that move away from static input/output tokens.
Now picture the same day from the start, some years from now.
The same day, some years from now —
EarlierI still curse to myself when I open the fridge in the morning and realize I forgot to buy milk.
The fridge notices the milk is low and orders a new carton online. It also notices that the expiration date on the butter is today, so it both orders a new stick and texts me a note to throw away the old one. My smart glasses quickly tell me the scrambled eggs look like they’re done, so I can take them out before they get rubbery. My glasses notice some government lettering peeking out of the stack of letters and alert me there’s something important there. My household cleaning robot catches a plate that’s about to fall off the counter, and the robot vacuum perfectly navigates my home.
The autonomous flying taxi I use to get to work has a prenegotiated flight path, and swiftly avoids a flock of geese in the way, as well as a high-flying kite that doesn’t show up on radar. When it gets close to work, it spots an open space at a rooftop parking garage. The pothole in the road is already being fixed by an autonomous road maintenance vehicle that automatically sets up traffic cones and fixes road issues as quickly as its vision system spots one.
Flying by my office, some drones are mapping some nearby open space as a potential site for a new high school.
Projects like this now finish in three months rather than three years, with AI-assisted site surveying, building, planning, and design.
The smart office I work in notices what was left out and texts janitors to handle it. During my doctor’s appointment, my scan is read and issues flagged immediately, giving the doctor more time to hear about my symptoms. Nearby, a smart hospital room alerts a nurse that there’s a new mole worth checking. In the hospital’s lab, visual AI is used to measure the effectiveness of different drugs to a patient’s swab culture in real time.
When I get home, my smart home tells me I left my earbuds by the bathroom sink and that I went out with my jacket but didn’t come back with it. It also tells me a dog peed on my lawn and it recommends that I order a sign.
EarlierHe still needs to spend minutes to create a screw hole on a surface in his design, sketching with pencil and paper, before re-creating the design curve-by-curve.
My father now describes his mechanical designs by voice, adjusting details of a design, by simply pointing and saying a few words to add a screw hole and indentations to allow two pieces to fit together. He now spends more time sketching bigger designs, and the AI perfectly translates his intent into changes, rather than painstakingly editing small parts. His colleagues now design new cars and rockets in months instead of years.
EarlierAmy still needs to read through 250 pages of dense building code documents and then carefully cross-check code requirements by hand.
My friend Amy’s design is automatically checked against the 250 pages of building codes, including handrail height. She now simply reviews the flags the AI made, instead of cross-checking manually. In the next office, her colleague is starting the design of a new offshore wind plant, and expects it to be finished in a few weeks.
EarlierThey are spending hundreds of hours building and debugging the platform before any real testing starts.
The roboticist I met now designs the whole test platform, in weeks instead of months. Finally, I look up at the moon, where I think I can see swarms of robots working completely autonomously while self-navigating around mountains, hills, and depressions, without getting stuck on small rocks.
A new threshold —
Just like airplanes in the 1920s and spaceships in the 1970s, this is a new threshold for physical technology.
Seeing comes before words. A child looks and recognizes before it can speak.
The Internet revolutionized the virtual world. I believe that strong visual AI that works and sees like we do is the missing piece that will revolutionize the physical world. Only then can we bring the promise of our longest-held dreams into the real world and escape into the future.
Our new model, Glimpse, is the start of a new quest to put the eye in AI.
Read about Glimpse Sign up for Glimpse