AI Pointer: Why Google is Killing the ‘Text Prompt’?
The ‘This and That’ Revolution: Why your cursor is about to get a PhD.
Mixology is UXHack’s Weekly newsletter covering interesting topics in UX, Products and AI, along with Curated Jobs, Events and useful Resources.
Stop typing. Start pointing. For over fifty years, your mouse cursor has been a silent observer, doing little more than clicking ‘Accept Cookies.’ But Google DeepMind’s AI Pointer is about to kill the ‘AI detour’- that annoying dance where you leave your app to copy-paste text into a chatbot.
Powered by Gemini, this research shifts the focus from verbose text prompts to deictic language: the ability to understand ‘this’ and ‘that’ based on where you point. Point at a complex table and say ‘Summarize this,’ and the AI just knows.
For the Indian tech ecosystem, this is a massive UX unlock. Imagine a Meesho seller or a Zomato restaurant partner navigating a complex data dashboard. Instead of wrestling with tech jargon, they can point at a fulfillment error and ask, ‘How do I fix this?’ in their native tongue. It bridges the digital literacy gap by replacing ‘prompt engineering’ with natural human gestures. As we enter the ‘Agentic UX’ era, the challenge for designers shifts.
We aren’t just building click-targets; we are building ‘pointable’ entities that an AI agent can perceive. If your interface becomes a canvas for an AI to ‘see’ and ‘act’ upon, how does your layout change? Are you still designing for human clicks, or for AI eyes?
Sources & Citations:
Shaping the future of AI interaction by reimagining the mouse pointer - Google DeepMind
Google DeepMind Reimagines Mouse Pointer with AI-Powered Context Understanding - The Rift
👉🏻PS: Have an interesting read to recommend to 9900+ readers? You can also recommend your own article/video/case study. Tell us here.
📅 Events @ UXHack:
Weekend Hackathon #112 Results
👉🏻 Product: Notion
Context: Improve the in-app search experience.
💼 Featured Jobs:
Design Jobs 🎨
UX Design Manager, Amazon Music Design at Amazon | Bengaluru
UX Designer at Google | Hyderabad
Product Jobs 🅿️
Senior Product Manager at MongoDB | Gurugram
Lead Product Manager at Mastercard | Pune
⛏️ Tools & Resources
💼Career Resources
UXH CV Review - Get objective AI feedback tailored for Design & Product roles to land more interviews
I have a feeling this will turn out awful. At least with a text poss you have a choice the program may not have in the system. I'm already tortured by yes/no "choices" that don't fit the issue and by fast responses to a random finger move. It may not pertain to mousing today... But eventually you'll end up wasting a lot of time trying to guess the right question in order to get to your issue