[Bug]: VirtualServer status.state set to empty string

Projects NGINX Ingress Controller
1

Please use this template for troubleshooting questions.

My issue:

Summary

When a new leader is elected, VirtualServer and VirtualServerRoute resources can have their status.state field overwritten with an empty string, causing the status to become invalid. Additionally, the status.reason and status.message fields get polluted with messages from other controllers, making it appear as though the ingress controller reported those errors.

Root Cause

During leader election, the controller calls updateVirtualServersStatusFromEvents() to restore status from Kubernetes events. The function finds the latest event for each VirtualServer and derives the status from the event’s Reason field using getStatusFromEventTitle().

The bug occurs because:

  1. The function iterated over all events for a VirtualServer, not just events created by the ingress controller
  2. Third-party controllers (e.g., Kyverno) create events on VirtualServer resources with their own Reason values (e.g., PolicyViolation)
  3. getStatusFromEventTitle() returns an empty string for unrecognized event reasons
  4. If a Kyverno event is the most recent event, the status is updated with state: ""

Affected Code

internal/k8s/controller.go:

  • updateVirtualServersStatusFromEvents() (line ~2461)
  • updateVirtualServerRoutesStatusFromEvents() (line ~2505)
  • getStatusFromEventTitle() (line ~2448) - returns "" for unrecognized reasons

Trigger Scenario

  1. VirtualServer exists with valid status.state (e.g., “Valid”)
  2. Kyverno evaluates the VirtualServer and creates a PolicyViolation event
  3. Leader election occurs (pod restart, scaling, etc.)
  4. New leader calls updateVirtualServersStatusFromEvents()
  5. Kyverno’s PolicyViolation event is the most recent event
  6. getStatusFromEventTitle("PolicyViolation") returns ""
  7. Status is updated with empty state, overwriting the valid state

Fix

Filter events to only consider those created by the ingress controller by checking event.ReportingController == "nginx-ingress-controller". Skip the status update entirely if no matching events are found.

I have a PR for that: filter by only ingress controller events when updating status by fcrespofastly · Pull Request #10230 · nginx/kubernetes-ingress · GitHub

Affected Versions

At least 5.5.0

How I encountered the problem:

**- During a rollout of the controller with ArgoCD, we noticed one of the VirtualServer got degraded as the status.state field got empty and the message got polluted with the message event from a kyverno admission that was validating the VirtualServer via a new policy we deployed.

Solutions I’ve tried:**

**- Deleted the policy that was validating the VirtualServer and that made it work.

Version of NIC and/or NGINX: 5.5.0/1.31**

**Deployment environment:

  • GKE with kubernetes 1.33 or 1.34
  • Kops AWS with 1.33 or 1.34**
2

This issue is currently being tracked - VirtualServer status.state set to empty string · Issue #10229 · nginx/kubernetes-ingress · GitHub

Please keep an eye on the issue for further development.