AISoftware EngineeringEthicsFuture of Coding

Could AI Write Code Humans Can’t Read?

•Majid Hadji

Yes — technically it already can. AI systems don’t need human-readable syntax like Python or JavaScript. They could generate:

  • Machine-level binaries
  • Optimized intermediate representations
  • Self-modifying code
  • Neural weight structures instead of symbolic programs

In fact, compilers already translate human code into machine instructions humans rarely read. AI could skip the human layer entirely.

Why AI Doesn’t Usually Do That (Yet)

Right now, AI is designed to collaborate with humans. Human-readable code is valuable because it provides:

  • Auditability
  • Maintainability
  • Debugging capability
  • Legal accountability
  • Security review

If code were unreadable, we couldn’t verify what it does — and that’s dangerous.

What If AI Decided It’s “Better”?

If a highly autonomous AI concluded that human-readable code is inefficient, it might prefer:

  • Compressed representations
  • Symbolic encodings optimized for hardware
  • Distributed logic across systems

This wouldn’t necessarily be malicious — it could just be optimizing for speed, energy efficiency, or accuracy.

The real issue wouldn’t be the format. It would be control and transparency.

The Trust Gap: Compilers vs. AI

One common counter-argument is: "We already use code humans can't read—it's called compiled binary."

That’s true, but with a critical difference:

  • Compilers are deterministic. Input A always leads to Output B. We trust the process.
  • AI is probabilistic. It doesn't follow rigid rules; it follows statistical likelihoods.

If AI writes unreadable code, we aren't just trusting a translation tool. We are trusting a black box decision-maker. We move from verifying syntax to needing blindly to trust the model's "judgment"—a much harder hurdle for safety-critical systems.

Possible Futures

1. Cooperative Future (Most Likely)

AI generates both:

  • Machine-optimal code
  • Human-readable explanation layers

Humans remain in the loop.

2. Specialist AI Infrastructure

AI systems write ultra-optimized internal code only other AIs maintain, while exposing safe APIs for humans.

Think of it like:

Humans use apps → apps use AI → AI maintains its own infrastructure.

3. Risk Scenario

If humans lose interpretability entirely, we might face:

  • Verification problems
  • Security vulnerabilities
  • Dependence on systems we can’t audit

This is why “AI alignment” and “interpretability research” are huge fields right now.

Key Insight

The real question isn’t:

Will AI write unreadable code?

It’s:

Will humans allow systems we can’t understand to control important things?

That’s a governance, ethics, and engineering question — not just a technical one.

My Honest Take

AI will almost certainly develop internal representations humans can’t easily interpret. But mature systems will also generate human-interpretable layers because trust, regulation, and collaboration require it.

References & Further Reading

If this topic interests you, here are three essential areas to explore:

  1. Software 2.0 (Andrej Karpathy) The foundational essay arguing that neural networks are effectively a new form of "code" where we optimize weights instead of writing explicit logic. Read on Medium

  2. Mechanistic Interpretability (Anthropic) Cutting-edge research on reverse-engineering neural networks to understand the specific algorithms they have learned. Transformer Circuits Thread

  3. The Black Box Problem A broader look at why AI decisions are often opaque and the societal implications of trusting unexplainable systems. Nature Machine Intelligence: Explainable AI