SSA finalizer add re-creates a primary resource deleted before its first reconciliation #3649

Bug Report

What did you do?

With server-side apply enabled for the primary resource (ConfigurationService.useSSAToPatchPrimaryResource(), the default; Quarkus quarkus.operator-sdk.enable-ssa=true) and a Cleaner reconciler (so a finalizer is used):

1. Create a custom resource without the finalizer. A reconciliation is queued.
2. Delete it before that first reconciliation adds the finalizer. With no finalizer the API server removes it at once, so it never gets a deletionTimestamp.
3. The queued reconciliation runs against the cached copy, which has no finalizer and no deletionTimestamp.

What did you expect to see?

The finalizer add fails because the resource is gone (404), or is skipped, and nothing is created.

What did you see instead? Under which circumstances?

ReconciliationDispatcher#handleReconcile sees the missing finalizer and calls ResourceOperations#addFinalizerWithSSA. That method builds a fresh instance holding only name, namespace and the finalizer, with no resourceVersion, and server-side applies it. A server-side apply creates the object when it is missing, so the deleted resource comes back as a stub: no spec, no owner references, no deletionTimestamp, and managedFields holding only the finalizer:

metadata:
  finalizers:
  - "replicas.private.dremio.com/finalizer"
  managedFields:
  - fieldsV1:
      f:metadata:
        f:finalizers:
          v:"replicas.private.dremio.com/finalizer": {}
    manager: "replica"
    operation: "Apply"
  name: "repro-1"
status: {}

Nothing deletes the stub afterwards, and with no owner references garbage collection never removes it. The reconciler then runs on a spec-less resource; in our case its dependents threw an NPE on every retry.

Reproduced in k3s (v1.34.1) by creating and immediately deleting a resource in a loop: it happened on the second iteration.

With SSA disabled the dispatcher calls addFinalizer(true), a patch that returns 404 for a missing object, so the bug does not occur.

Environment

• Java Operator SDK 5.5.0 (Quarkus Operator SDK 7.8.0). The same code is in v5.6.2 and on main (ResourceOperations#addFinalizerWithSSA).
• Kubernetes: k3s v1.34.1.

Possible solution

Make the SSA finalizer add unable to create a resource, for example:

• send the cached resourceVersion with the apply, so it fails with a conflict or 404 when the object is gone, or
• add the finalizer with a non-creating patch (as the non-SSA path does), or
• treat a create returned by the apply (a new uid) as "resource gone" and delete it straight away.

Additional context

Related: #2849 (SSA finalizer pitfalls), #2845 (feature switch for SSA finalizers), #3117 (stale cache after deletes).

As a workaround we now set the finalizer when the resource is created, so the dispatcher never needs to add it. That only works where the operator creates the resource; it does not cover resources created by other clients.