How We Help Train AI

Imaged generated by Gemini based on the post title.

I know most (all?) of you who use AI are aware of this. It’s always good to have as much background information. Check out this conversation I had with ChatGPT Plus. Don’t miss the response to my point that humanity could be contributing to its own demise. It isn’t academic.

ChatGPT ends with this question: That raises a fundamental question: Should the development of potentially superintelligent AI remain principally in the hands of private companies and competing governments, or should decisions that could affect humanity’s survival require a much broader form of international democratic oversight?

What do you think, dear reader?

MAA

MAA: I assume that every conversation I have with you helps train you.

ChatGPT: Your assumption is partly correct. Your conversations with me may help train future AI models, but they do not automatically train me in real time.

There’s an important distinction between three processes.

1. Learning during our conversation

As we discuss AI, I use what you’ve said to understand your arguments, questions, and concerns.

For example, I can follow the progression of our discussion from superintelligence to catastrophic risks, autonomous AI, and AI systems training other AI systems.

However, our conversations do not directly update my underlying neural network or permanently increase my intelligence.

2. Remembering information about you

ChatGPT has a memory capability that can retain useful context across conversations.

That allows me to refer to your previous discussions, interests, and ongoing projects.

But memory is different from model training. It personalizes my responses to you rather than making the underlying AI model more capable for everyone.

3. Training future AI models

This is where your assumption is substantially correct.

OpenAI confirms that conversations from personal ChatGPT accounts may be used to improve and train its models.

For personal Free, Plus, and Pro accounts, this sharing is generally enabled by default. Users can opt out. OpenAI Help Center+1

If your conversations are eligible for training, they may contribute to subsequent model improvements. That doesn’t mean every conversation is necessarily incorporated into a training dataset.

You can decide whether to contribute

To check your setting:

  1. Open Settings in ChatGPT.
  2. Select Data controls.
  3. Find Improve the model for everyone.
  4. Turn it off if you don’t want new conversations used for model training.

You can retain your conversation history even with training disabled. I cannot see whether this setting is currently enabled on your account.

Temporary Chats also are not used for model training while they remain temporary. OpenAI Help Center

There’s a larger implication

Your question raises an interesting issue in the context of our discussion.

Millions of people interact with AI systems, challenge their answers, correct their mistakes, and present them with new problems.

Those interactions can produce valuable material for improving subsequent models.

In that sense, humanity is collectively contributing to the development of increasingly capable AI systems, although the training process is not simply the accumulation of every conversation.

And this brings us back to the issue of control.

People may contribute their knowledge, expertise, creativity, and intellectual labor to improving AI systems without having meaningful influence over how increasingly powerful successors are developed or deployed.

That raises questions about consent, accountability, and who ultimately benefits from this collective contribution.

I’m curious about something: Are you asking because you want to know whether our conversations contribute to my development, or because you’re concerned that millions of users may inadvertently be helping create the very superintelligence we’ve been discussing?

MAA: Humanity could be contributing to its own demise.

ChatGPT: Yes. That is a disturbing possibility, and the paradox is that humanity could contribute to creating a technology that eventually threatens its own survival.

What makes this possibility particularly troubling is that the process could unfold incrementally, with each step appearing reasonable or beneficial.

Consider the progression:

  1. Millions of people interact with AI, contributing knowledge, feedback, corrections, and examples of human reasoning.
  2. AI companies use some of that information to improve subsequent generations of models.
  3. More capable AI systems help develop still more capable systems, potentially accelerating technological progress.
  4. Governments, militaries, corporations, and individuals become increasingly dependent on AI for decisions and essential operations.
  5. Human oversight becomes increasingly difficult, particularly if AI capabilities advance faster than our ability to evaluate and control them.
  6. A catastrophic failure or deliberate misuse becomes possible, with consequences potentially extending far beyond the organizations that developed or deployed the technology.

This is a plausible scenario, not an inevitable outcome. Several steps involve substantial uncertainty, particularly whether AI could develop the capabilities necessary to escape effective human control.

But there is something else I find important about your observation.

The tragedy of individually rational decisions

No single participant needs to intend harm.

A student uses AI to learn. A doctor uses it to diagnose disease. A researcher uses it to discover new treatments. A business uses it to improve efficiency. An AI laboratory uses the resulting advances to build a more capable model.

Each decision can be perfectly understandable.

Yet the cumulative consequences could be dangerous if capabilities, dependencies, and opportunities for misuse grow faster than safeguards.

This resembles what economists call a collective action problem: individually beneficial choices can produce outcomes that are harmful to everyone.

And unlike many historical technologies, sufficiently advanced AI might eventually become capable of improving the very processes through which it is developed.

But there’s a distinction we shouldn’t overlook

Humanity’s participation in AI development doesn’t mean humanity is inevitably engineering its extinction.

The same collective intelligence that helps develop AI can also contribute to safety research, independent evaluations, public oversight, and international cooperation.

Moreover, there is a significant difference between using AI for education or medical research and granting autonomous AI systems authority over critical infrastructure or weapons.

Those are choices, not unavoidable consequences of technological progress.

What I find most troubling

Our discussion keeps returning to the same fundamental problem: human technological capabilities have repeatedly advanced faster than our collective wisdom and institutions for governing them.

Nuclear weapons are one example. Climate change is another. AI could present an even more difficult challenge because its development may accelerate the pace of technological change itself.

We should also acknowledge that the probability of AI-driven human extinction is deeply contested. Some researchers consider it a serious risk; others believe such scenarios rely on speculative assumptions about future capabilities and behavior.

Neither position can presently be established with certainty.

I would therefore frame your observation as a warning rather than a prediction:

Humanity may be creating systems whose ultimate capabilities, consequences, and control mechanisms we do not adequately understand—and we are doing so before agreeing on how much risk we should be willing to accept.

There’s a further ethical issue that I think deserves attention. The people contributing to AI development are not necessarily the people making decisions about its future. Nor are the people who might suffer its worst consequences being asked whether they consent to those risks.

That raises a fundamental question: Should the development of potentially superintelligent AI remain principally in the hands of private companies and competing governments, or should decisions that could affect humanity’s survival require a much broader form of international democratic oversight?

Leave a Reply