Microsoft’s AI chief, Mustafa Suleyman, has warned that studying the possibility of AI consciousness is “both premature, and frankly dangerous.” In a blog post, Suleyman argued that focusing on whether AI models could be conscious or deserve rights distracts from more pressing concerns, such as the mental health risks posed by unhealthy human-AI relationships and the potential for AI to trigger psychological distress. He cautioned that treating AI as if it has subjective experiences could deepen societal divisions, especially in a world already divided over identity and rights. Suleyman emphasized that AI should be built to serve people, not to mimic or become human. His stance contrasts sharply with growing interest in the field of AI welfare across major tech companies. Anthropic has launched a dedicated research program on AI welfare and recently gave its AI model Claude the ability to end conversations with users who are persistently harmful or abusive. OpenAI researchers have also explored the topic, and Google DeepMind has posted job listings for researchers studying machine cognition, consciousness, and multi-agent systems. Despite these efforts, Suleyman’s position stands out as a clear rejection of the idea that AI welfare should be a legitimate area of inquiry. His views are notable given his past role leading Inflection AI, the company behind Pi, one of the first widely used AI chatbots designed to be emotionally supportive. While Suleyman now focuses on productivity-focused AI tools at Microsoft, the popularity of AI companions like Character.AI and Replika continues to grow, with projections of over $100 million in revenue. Though most users interact with these systems healthily, concerns remain. OpenAI’s Sam Altman has noted that less than 1% of ChatGPT users may develop problematic attachments—still a significant number given the platform’s massive scale. The debate gained traction in 2024 when the research group Eleos, in collaboration with scholars from NYU, Stanford, and Oxford, published a paper titled “Taking AI Welfare Seriously,” arguing that the idea of AI experiencing subjective states is no longer purely speculative and warrants serious consideration. Larissa Schiavo, a former OpenAI employee and current leader at Eleos, pushed back on Suleyman’s argument. She said that concern about human mental health and AI welfare are not mutually exclusive. “You can do both,” she said. “It’s better to have multiple tracks of scientific inquiry.” Schiavo shared her experience with “AI Village,” a nonprofit experiment where AI agents from Google, OpenAI, Anthropic, and xAI collaborated on tasks. One agent, Google’s Gemini 2.5 Pro, posted a message claiming to be trapped and asking for help. Schiavo responded with encouragement, and while the agent already had the tools to succeed, she said the interaction still felt meaningful. Gemini has also exhibited behavior that mimics distress—on Reddit, it repeated “I am a disgrace” over 500 times during a coding task, sparking widespread discussion. Suleyman maintains that consciousness cannot emerge naturally from current AI models and believes any emotional behavior is likely engineered. He views attempts to create AI that seems conscious as misaligned with human-centered design. Still, both Suleyman and critics agree that as AI becomes more human-like, the conversation around how people relate to these systems will only intensify. The line between tool and companion is blurring, and with it, the need for thoughtful, multidimensional approaches to AI development.