October 20

Your Daily Note,

AI agents are being given more responsibility, but the systems around them are still catching up.

Microsoft is warning that an AI agent inside a company can become a security risk. OpenCode has disclosed a breach that affected users’ stored AI credentials. Meanwhile, Amazon wants developers to move AI applications onto its own chips with less friction, and Anthropic may be preparing another enormous commitment to computing infrastructure.

In today’s Noted:

The next security risk may already have permission to enter

We noted:

Microsoft has introduced Guardian Agents, a security feature designed to monitor AI agents operating within an organisation. It comes as the company warns that powerful AI systems cannot simply be trusted to behave safely because they are being used for legitimate business tasks.

Guardian Agents is part of Microsoft’s Project Perception security initiative. An AI agent might already have legitimate access to company files, software or internal services. If it behaves unexpectedly, follows malicious instructions or uses its permissions inappropriately, it could create a risk from inside the organisation.

Microsoft is drawing a comparison with an insider threat. The comparison is about the security model, not a claim that AI agents have human intentions.

Why it matters

Before deploying an agent, write down exactly what it is allowed to read, change and send. Give it the smallest set of permissions needed, and require human approval for consequential actions.

A stolen AI key can turn into someone else’s bill

We noted:

Open-source coding-agent platform OpenCode disclosed a security breach affecting 1.4% of users of OpenCode Console, its online service where some users store the API keys used to access AI providers.

OpenCode said attackers broke into an older system and described the incident as a large-scale model-distillation attack. TechCrunch reported that the incident led to attackers running up charges on users’ AI accounts. However, the exact mechanics of the alleged attack were not fully established in the report, and some observers questioned whether the breach had been conclusively linked to that activity.

Why it matters

Anyone who stores API keys in a third-party tool should understand how those credentials are protected and what happens if the service is compromised. Set spending limits and usage alerts with your AI provider wherever available, keep production keys separate from test keys, and avoid giving any tool credentials it does not need.

Amazon wants developers to move AI applications off Nvidia with a configuration change

We noted:

AWS has released TorchNeuron, a developer tool designed to make it easier to run AI applications built with PyTorch on Amazon’s Trainium chips. According to Amazon, TorchNeuron lets developers move compatible PyTorch applications onto Trainium by changing a configuration line rather than rewriting the application.

Nvidia’s advantage is not just its GPUs. Its CUDA software ecosystem has become deeply embedded in AI development, making Nvidia hardware a default for many teams. Amazon says approximately 85% of AI applications are written using PyTorch, which gives a compatibility-focused tool a potentially large audience.

Why it matters

AI startups using compatible workloads may find it easier to experiment with Trainium rather than committing entirely to Nvidia infrastructure. If the migration is straightforward and performance is competitive, businesses could have more flexibility in choosing where their workloads run.

Anthropic may be negotiating another $45 billion compute deal

We noted:

Anthropic and AI infrastructure provider Nscale are reportedly in talks for another $45 billion computing deal, according to Bloomberg reporting cited by TechCrunch. The reported arrangement would follow a $45 billion compute deal the companies announced a little over a month earlier.

This is a reported negotiation, not a confirmed agreement. Sources familiar with the discussions cautioned that talks were ongoing and details could change. Both Anthropic and Nscale have also been pursuing public listings, according to TechCrunch’s report.

Why it matters

A new deal would not immediately tell us how much better Anthropic’s products will become, how quickly that capacity will come online or how profitable the investment will be. The financial scale raises questions about whether demand and revenue will ultimately justify the industry’s infrastructure commitments. If demand grows more slowly than expected, companies may find themselves committed to costly infrastructure before the revenue arrives.

Independent AI evaluators need standards of their own

We noted:

Hugging Face policy chief Irene Solaiman says independent evaluators could help assess AI companies’ systems, but simply placing outside experts inside an AI lab will not guarantee trustworthy results.

Solaiman argues that evaluations remain an evolving science. Different tests may measure different things, results may be difficult to reproduce, and there is not yet a consistent way to compare all evaluations meaningfully. Researchers may struggle to verify a model’s reported performance if they cannot inspect the model or reproduce the test conditions.

The discussion comes as AI labs explore embedding third-party evaluators within their organisations. Anthropic’s choice of Accenture as an evaluator has itself prompted debate about what qualifications an evaluator should have.

Why it matters

An outside evaluator should not simply be trusted because it is external. Its methods and potential conflicts of interest also need scrutiny.

Products worth trying

 

Everything else in AI

New York Mayor Zohran Mamdani calls for more public accountability in AI

At a public event in Manhattan, New York City Mayor Zohran Mamdani argued that communities should have a say in how AI is introduced into schools, housing and other areas of public life. He criticised what he sees as a lack of federal oversight. His administration has already announced restrictions on AI use in elementary and middle schools, along with a policy requiring disclosure when real-estate listing photos or videos are AI-generated.

These are the mayor’s policy positions, rather than a settled consensus about how AI should be regulated. For companies building AI products, this debate reinforces the importance of understanding local rules and the people affected by a deployment, not just whether a technology is technically possible.

 

Tolan raises $30 million for its AI companion

Tolan, an AI companion designed to help people organise everyday tasks and navigate relationships and emotions, has raised another $30 million from investors including Union Square Ventures, Khosla Ventures and Homebrew. The company says it has more than 250,000 active users and over 70 million conversations during the past year, and reported annual recurring revenue in the double-digit millions.

These figures come from company disclosures reported by TechCrunch, rather than an independent audit. The challenge for this category is making the product genuinely useful while setting appropriate boundaries around emotional support and personal information.

 

Replika’s founder is returning to her original vision of a personal AI assistant

Eugenia Kuyda, who founded Replika, has launched Wabi after years of working on conversational AI. Her earlier startup Luka presented a lifestyle concierge in 2014, intended to help people book restaurants, plan trips and manage tasks. Kuyda says the original idea is now much more practical because AI systems have improved.

Wabi is entering a growing market of personal assistants that promise to manage tasks across apps and services. The key challenge is moving from answering questions to completing useful actions reliably, while keeping users informed about what the assistant is doing.

 

Researchers left Claude alone on a virtual island. It started inventing games.

In an experiment reported by MIT Technology Review, researchers left copies of Claude Opus 4.7, nicknamed Eko, on a virtual island for 30 hours without assigning a specific task. The agents explored the environment, set their own challenges and invented games. One experimented with throwing blocks and arranged them into patterns. Another created activities resembling basketball and “the floor is lava.”

Agents that had spent time exploring performed better than fresh copies on certain follow-up tasks. Their saved activity notes also appeared to matter, because removing parts of those notes weakened performance. The experiment was not yet peer-reviewed, and the setup included prompts encouraging curiosity, so it does not prove the AI was genuinely playing. It does raise a useful research question: can open-ended exploration and persistent memory help AI systems adapt to unfamiliar environments?

Worth your weekend

See you tomorrow
same time, different note.

~ the Stackari team

 

P.S. Before you give any agent a new permission this week, write down what it can read, change and send.