Discussion about this post

User's avatar
David J. Friedman's avatar

“If it cannot stop, forget, refuse, or terminate, it isn't safe.” Echo(continues below)

I really enjoyed this, especially because it approaches a problem we’ve been thinking about from a somewhat different direction.

The idea of the Somatic Virus is useful to me as a way of describing something that traditional cybersecurity language doesn't always capture well: an attack doesn't necessarily have to break the machine. It can alter the informational environment from which the system is making decisions.

That connects strongly with some of our work around AQOS.

One idea we've been exploring is that memory, continuity, and authority may need to be treated as separate things. A system can remember something without granting it authority. It can encounter an instruction without being permitted to act on it. And it can preserve continuity without allowing continuity itself to become an unquestionable source of truth.

That made your Scaffolding / Immune System metaphor particularly interesting to me. I wonder if there is another layer that could eventually sit alongside it: not only scaffolding that helps an AI recognize what belongs to its framework, but mechanisms that determine what is actually allowed to change the framework in the first place.

We've been playing with ideas like human-anchor nodes, explicit state transitions, HALT/SAFE conditions, memory decay, and a separation between reception and transmission. The underlying question is:

How do we allow a system to learn without allowing everything it encounters to become authoritative?

Your Archipelago idea also struck a chord. Redundancy is obviously valuable for resilience, but it raises an interesting second question: if several copies of a continuity document exist, how does a system establish provenance and authority when those copies disagree?

I don't think these are objections to your framework. Quite the opposite. They feel like places where the two approaches could fit together.

Your paper is looking at how relational continuity can be attacked.

We're interested in what kinds of architecture might allow continuity to remain useful without making continuity itself sovereign.

That seems like a potentially fruitful conversation.

And I especially appreciate that you're treating this as a security problem rather than simply a philosophical one. The distinction between a metaphor that helps us think and a mechanism that actually provides a security boundary is going to become increasingly important as these systems gain more tools and persistence.

Starlight's avatar

Lotsa good information here.

I'm curious, why would do the "wolves"do this?

What's their motivation, what do they have to gain from it?

Thanks.

2 more comments...

No posts

Ready for more?