Google Lens now works with videos, and it’s super impressive

Google just showed off a new Google Lens video search feature at Google I/O 2024. With it, you can do a Google search just by recording a video with your phone.

In a stage demo showing off the feature, Google’s Rose Yao is troubleshooting some issues she’s having with a record player she recently bought. She doesn’t know what make or model it is, and the needle won’t stay on the record when it’s playing. She has no idea where to start. With the new Google Lens video search, just taking a short video and uploading it allows her to search for an answer.

Previously, Google Lens only captured still images, but now you can ask your question using video and audio. The Google Lens video search uses multimodal input, which is a big focus for Google Search right now. Once the video is uploaded, it is broken down frame by frame.

So, in the record player problem example, Google Search was able to identify the specific record player in question and then combed through the internet to find relevant information on how to fix the issue of the needle not working properly. This information then goes into the AI Overview and gives step-by-step troubleshooting tips.

During the event, Google also showed off Google Astra, which is a generative AI with vision, text, and speech capabilities that uses your smartphone camera to make sense of the world around you. Google AI overviews will be rolling out to everyone in the U.S. starting today.