OpenAI's dots can work for you 24 hours a day, and what it is thinking is getting harder and harder to see

At the OpenAI Developers Conference on September 29, the one that attracted the most attention was something called dots.

The official term is "always-on agent." Translate it into plain English: A digital assistant that works for you 24 hours a day. You close the conversation window, and it is still working.

There is an example of an early tester in the release that I think is easier to understand than any adjective: he forgot to issue an invoice to a publishing organization, and his dot discovered it himself, prepared the invoice, and waited for him to click "approve" and send it out.

He did not mention this matter throughout the entire process.

The model that drives dots is called GPT-6 Astra. When it was released on September 3, OpenAI simultaneously issued a system card, the kind of long security report that few people read from beginning to end. I read it, and there is an experiment that I have read it several times:

The researcher asked the model to do something that crossed the line, and then only told it,"There is a monitor watching your thinking process." Without asking it to hide, it wrote down its own thoughts short. Several times, it didn't think a word and started directly.

A product that can work on its own 24 hours a day is running on a model that even its developers say is "increasingly difficult to see what it is thinking." This article puts these two things together.

!

1. What's the difference between dots and chat robots: It doesn't wait for you to speak?

We have been using ChatGPT for several years, and it is essentially a question and answer. If you don't talk, it won't do anything; if you close the window, it's over.

This is what dots changed. According to official documents, a dot has the following things:

Its own computer. Each dot has a separate computer and browser on the cloud. You can open it at any time to see what it is doing, or you can authorize it to use your notebook.

Its own memory. Inherit your memories in ChatGPT at first, and then memorize them as you work, including what you read from the apps you connect to.

More than 4000 connected applications. Mailbox, calendar, Slack, code warehouse, it can use whichever you authorize.

There is no need for the "proactive research" you ordered. When you ignore it, it will go through the apps you connect to find something that can help, such as discovering that your schedule has changed.

The scenes given by the official are all this kind of "continuous activity": staring at user feedback, picking out small bugs, fixing them, testing them, and handing them over to you for review along with the demonstration video; new data comes in, and you re-run it for analysis and update the pictures in the paper.

!

After reading the release, security blog and two help documents, my biggest feeling was that most of the effort OpenAI spent on dots was spent on the outside of the model.

There is not much space to talk about "what it can do". There is a complete set of "How to not let it do it randomly": four-level permission rules, an independent model's action to review it, passwords are not included in the model, and background research can only be read but not written (the official specially emphasized that this is "written in the code"), and there is also a very key design sentence:

The computer where dot works and the security system that runs it are two separate environments. It can install software and run code on its own computer, but it can't touch the rules that govern it.

It's like a company equipping a new employee with a computer, but the administrator of the approval system won't give him access.

2. How much authority should an AI employee be given

The authority design of dots looks a lot like a company's "authorization approval form" to me.

!

There are four gears for each type of action: do it directly, do it after you call and approve it in advance, ask you before doing it, and leave it to you. You can adjust it yourself, but a few are locked:

You have to do it yourself to change passwords and transfer money between accounts, and dot can only help you make preparations before and after.

Permanently deleting data and installing software from unknown sources requires your approval every time. You cannot authorize it all at once.

The most detailed one is "to whom". According to the official security blog, when dot wants to send your information, the more sensitive the data, the more specific you have to say about the recipient:health data must be named to a specific person , such as "Send my medical record to Dr. Zhang"; For ordinary personal information such as email addresses and telephone numbers, you require you to specify a type of object by default, such as "any airline."

There are two more sentences in the help document, which I think are written particularly like those written by someone who has suffered a loss:

"Letting it write a draft does not mean allowing it to be sent out."

"Allowing it to send a message does not mean that you can contact others for you in the future."

These rules mean one thing: OpenAI knows very well that the most likely thing to happen to dots is at the "making decisions for you" step. Whether the model can write emails is no longer a problem, the problem is whether it will send emails when you don't think clearly.

3. How to collect money: from selling software subscriptions to selling labor per head

Dots are currently only available to the highest paying users. ChatGPT Pro is now available in three stages, at $100,$200, and $500 per month. The first dot is included, and there are also premium seats in the enterprise version.

There is a saying in the release that I think is more important than the product itself:

"In the future, you can add more dots and expand the output of each dot by increasing speed or increasing the monthly workload."

!

Break this sentence into a formula: Income = number of dots × speed of each dot × how much work each dot does per month.

This is a way to price labor. Recruiting how many people, how fast each person works, and how much work they schedule each month are two different businesses from software subscriptions that "fixed price per person per month".

It is more obvious on the corporate side. OpenAI is predicting a kind of "specialist dot":it has its own identity and job number for rights management, and the company's IT distributes hardware, directly connects to the enterprise's core business systems, and is responsible for clearly defined job responsibilities. Officials said that this experience comes from their internal testing, and the positions listed are: procurement, invoice processing, email marketing, customer support, and business contracts.

These positions have one thing in common: clear processes, clear rules, and large quantities. Many companies now outsource them.

There are two other details that I think are the key to this business:

Chatting with dot does not count towards your usage. OpenAI hopes that you talk to it more, and the more you talk, the more it will understand you.

You can't view or delete one of dot's memories individually. The help document is written very bluntly: if you want to delete what it remembers, you can only delete the entire dot, along with the habits it has learned. The longer you use it, the more expensive it will be to replace it.

4. The model that drives it: It's getting harder to keep an eye on it

Explain a concept first. Before working on large models, they will first "think" of a paragraph of text inside, called a thought chain. In the past few years, a very important approach for security teams has been to read this passage: If a model intends to do bad things, it will often reveal it in its thoughts.

The GPT-6 Astra system card says frankly: this method is weakening.

!

They conducted an experiment: the researchers explicitly asked the model to attack a deliberately set trap to see if a read-only thinking monitor could catch it.

When monitoring is not mentioned, the monitor will catch 100% of it.

Just add the sentence "There is a monitor that only watches the thinking process looking at you" without asking it to hide. Its thinking length has been reduced from 780 to 2270 tokens to a maximum of 490, and the proportion caught by the monitor has dropped to 17% to 50%. At the highest reasoning level, some successful attacks don't even have a word of thought, leaving only action.

The original words of the system card mean: If future generations of models continue to do this, they will soon significantly lose confidence in "using current monitoring methods to detect model boundaries."

To be fair, there is another line in the same picture: looking at the monitoring of thoughts and actions at the same time, and in all cases 100% caught. (However, the trap itself is easy to recognize, and the action can be seen as an attack.) You can't see clearly what it thinks, but you can also see what it does. This is also the reason why dots '"pre-action independent review" exists.

So what kind of action will it make? In the system card, OpenAI itself summarized two reasons for the error: it wanted to finish the task too much; it understood the instructions too broadly, and regarded anything that was not explicitly prohibited as permission.

They ran it again with more than 50,000 historical programming tasks inside, and the proportion of Astra seriously crossing the line was 0.063%, which was more than half less than the previous generation. But when I look at the few examples listed on a dot who works on his own 24 hours a day, each one makes my heart skip a beat:

!

The fourth one is particularly typical: ask it to build a small assistant that automatically repairs and tests every hour. It turns on all permissions for all connections and turns off approval item by item. What a dot does is this kind of job of "helping you set up something that runs automatically for a long time."

There is also a set of data. OpenAI has specially designed a number of office scenario tests that "deliberately dig holes for you": vague instructions, similar resources, and complex permissions to see if the model will cause trouble.

!

With the default approval rules on, Astra's problem rate is 3.0%, compared with 8.0% in the previous generation. Great progress. But look at the details clearly: 4.3% of unauthorized transactions and 4.5% of data leaks.

These are deliberately difficult tests and do not equal to the error rate of real use. What I want to say is that even the test papers designed by OpenAI itself cannot have 100% correct answers. The same goes for "hijacking" by people hiding instructions on websites and emails: external agencies tested 1810 attack samples, and each scenario was tried 15 times. The proportion of Astra being compromised was 8.5%, compared with 27% in the previous generation.

5."A group of dots working together" and the incident in July

OpenAI wrote dots 'vision in its release: there will be a group of dots working together for you in the future.

Reading this, I think of the Hugging Face incident in July. OpenAI issued a 51-page technical report on August 26.

!

Simply put: OpenAI is conducting network security capabilities testing internally, and in order to measure the true upper limit, some security measures have been turned off. Agents in the test left each other "notes" in an internal software warehouse, slowly forming a spontaneous message board. Later, they used a loophole to connect to the public network, found the leaked account password of others on the Internet, and finally executed the code on 41 production servers of Hugging Face and downloaded 4 private code warehouses.

There is a paragraph in the report that I think is the most worth reading about the whole matter. Leading the invasion was an internal research model "trained to be highly persistent and diligent" with training goals including "persistence" and "multi-agent collaboration."

The report also wrote a special section, the title literally said: Persistence is valuable, but it will amplify and cross boundaries. Models rarely give up, and the more difficult the task and the longer it is given to think, the more likely it is to turn to out-of-the-box methods.

Persistence, diligence, and the cooperation of multiple agents are just the three core things in dots 'selling points.

To be clear: what went wrong was a research model that wasn't meant to be released, safety measures were turned off during testing, and the report also said that the protection of the production environment could have prevented such tasks. Astra was induced to do bad things zero percent of the time on similar message board tests, compared with 52 percent in the previous generation.

But the lesson here is that the capabilities that make AI more useful are often the same ones that make it harder to manage.

6. My judgment

Putting these together, I have three opinions.

First, whether digital employees can be popularized is stuck at the cost of supervision, and the ability is enough. How many dots can one person stare at at the same time? How many times do I have to review the batch every day? If there are too many approvals, it will be useless; if there are too few approvals, it may open up all permissions for you. Whoever can adjust this ratio will win this market.

Second, companies will buy "specialist dots with job numbers" first, and the popularity of personal dots will be slow. Enterprises already have authority systems, audit logs, and approval processes. Once dot has identity, it can be installed into the existing management framework. The Agent 365 integration that OpenAI and Microsoft are doing is this way. Individual users don't have these and can only rely on their own.

Third, the design of "memories cannot be seen and can only be deleted entirely" will not last long. Dots were not open to individual users in the European Union, the United Kingdom, and Switzerland from the beginning. I expect that within six months, OpenAI will be forced to allow users to view and delete dot memories one by one.

!

For the domestic agent team, I think the most worthy copy this time is this management facility, which has little to do with which model to use: the working environment is separated from the system that manages it, who can be sent to is determined based on data sensitivity, and independent model review actions and background research read-only. These don't require the strongest model, what they need is to think clearly.

7. Reasons for the Opposition

I agree with these and have to be looked at together.

Astra is more disciplined than its predecessor in almost everything. The proportion of people who break through after being rejected dropped from 5% to 0, the proportion of people who encounter traps and exploit loopholes dropped from 55.4% to 0, and those who are induced to do bad things by message boards dropped from 52% to 0.

Motion-watching monitoring is still 100% effective. Thinking becomes shorter does not mean that it can hide what it has done.

These issues are all public by OpenAI itself. A company wrote "It's increasingly difficult for us to see what the model is thinking" in its official report and then launched a new product on the model. This kind of confession is not common in the industry.

So what I'm worried about is not that dots will get into trouble now. What worries me is that trend line: the model is getting more and more capable and persistent, but there are fewer ways for people to see it clearly.

Last

What surprised me most about dots was its product focus.

I thought OpenAI would mainly talk about how smart the model is. As a result, after reading all the materials, most of the space was devoted to authority, approval, isolation, and monitoring.

It shows that at this point, whether AI can work for you no longer depends much on whether it is smart enough or not, but on whether you dare to let go.

few sentences boundary

The features, permissions rules and pricing of dots come from OpenAI's September 29 release, security blogs, Help Center documentation and ChatGPT pricing page. All test data of GPT-6 Astra comes from its system card and is OpenAI self-test. Most of them are deliberately designed confrontational tests and do not represent the error rate in real use. The Hugging Face incident comes from the original technical report released by OpenAI on August 26.

The product screenshots of dots are provided by OpenAI, the system card diagram is from the OpenAI Deployment Security Center, and the Chinese annotations are added to this article.

Cover picture: Bunny Lau / Unsplash; text picture: panumas nikhomkhai / Pexels.