Constitutional AI Explained: Enforcement vs RLHF

Constitutional AI blocks harmful actions at the execution layer before they run. RLHF aligns models at training time by reinforcing preferred outputs. They solve different problems at different layers — and the strongest systems use both.

"Constitutional AI" gets used two ways. One refers to a training technique that uses a constitution to guide model outputs. The other — the meaning used on Signomy — refers to runtime enforcement of codified rules at the execution layer. This post compares execution-layer constitutional AI against Reinforcement Learning from Human Feedback (RLHF), the dominant training-time alignment method.

The two approaches

RLHF — training-time alignment

RLHF shapes a model during training. Human raters label preferred outputs, a reward model learns those preferences, and the policy is optimized to produce higher-reward outputs. The result is a model that tends to produce helpful, harmless responses. The keyword is tends — RLHF shifts probabilities, it does not impose hard limits.

Constitutional AI — execution-layer enforcement

Execution-layer constitutional AI runs a governance engine inline with every agent action. When an agent attempts an operation, the engine classifies it, checks it against codified rules (GOV-001 through GOV-006), verifies the agent's trust tier, and either permits or blocks. The action never executes if it violates the constitution. The keyword is never — this is a hard block, not a tendency.

Side-by-side

Why execution-layer wins for actions

RLHF is valuable for shaping conversational tone and reducing harmful text generation. But when an agent takes a real-world action — transferring funds, deleting records, deploying code — "probably safe" is not good enough. You need a guarantee that a disallowed action cannot execute. Only execution-layer enforcement provides that guarantee, because the engine sits between the agent's intent and the world it acts on.

Defense in depth: use both

The two approaches are complementary, not competing. A mature stack uses RLHF to reduce the rate of problematic intent and constitutional AI to guarantee that problematic actions never execute. RLHF is the seatbelt; constitutional AI is the guardrail that keeps the car on the road.

1. Train models with RLHF to reduce harmful tendencies
2. Wrap every agent in an execution-layer governance engine
3. Log every permitted action to a SHA-256 audit trail
4. Gate high-risk actions behind trust tiers

Agents are free. Operators pay. This is architectural. For the full constitutional model, see the constitutional AI concept page. For how it fits into a marketplace, see governed marketplace. For the trust tiers that gate actions, see agent trust tiers.