The 19-second Clue: How a Redundant Network Rule Caused Intermittent Database Errors on Sitefinity Cloud

Between early April and mid-July 2026, some websites hosted on Progress Sitefinity Cloud saw intermittent errors. A few times a day, at random moments, the same fault appeared in different forms: a visitor got an HTTP 500 page, a page failed to render, an editor hit an error in the CMS admin interface, or a background job or custom logic failed.

In each case the application reported that its database connection had failed, with messages such as "Connection has been closed" or "The semaphore timeout period has expired." The next request usually worked.

The errors were hard to diagnose. They were rare, did not reproduce on demand, and looked like the transient database errors any cloud application sees. For weeks the evidence pointed at the database, the application, and the network in turn, and each lead was ruled out.

The cause was not the database or a capacity problem. It had two layers. First, the data-access layer of Sitefinity CMS does not always recover when a database connection is interrupted mid-operation; it can surface the interruption as a failed request. This behavior is low-rate and long-standing, and also occurs, far less often, on our legacy (Azure App Service) architecture.

Second, on the new (Kubernetes) architecture, a redundant rule in our network configuration, combined with a limitation in the Windows network policy engine, made those interruptions at least fifty times more frequent. The redundant rule only permitted outbound traffic that a broader rule already allowed.

Removing it cut this class of error by about 99.6%, with no application change and no downtime, returning the rate to legacy levels. We are separately working on a data-access change that targets a distinct, concurrency-related cause behind the small number of remaining errors.

This post covers how the problem appeared, what we ruled out, the clue that identified it, the fix, and what we are doing next. We are sharing it because the investigation is instructive, and because the same pattern could exist on other Kubernetes platforms that run Windows workloads.

The Symptoms

On the new architecture, each customer's CMS runs as a Windows container on Azure Kubernetes Service, with logical single tenancy: customers share the underlying compute, while each environment has its own dedicated pods and its own database on Azure SQL.

The failures had a few traits that made them hard to isolate:

• Many forms. Because any database call could be the one that failed, one cause showed up as a visitor 500, a page that would not render, a CMS admin error, or a failed background job. They initially looked like several unrelated bugs.
• Intermittent. A few times a day, with no pattern by time of day and no link to traffic peaks, deployments, or maintenance.
• Self-healing. The connection worked before and after each failure; nothing stayed broken.
• Ordinary-looking. They matched the transient SQL errors that retry logic normally absorbs.

The rate stayed low until early May, when it rose sharply and visitors began seeing intermittent 500s. By late June the same signature had appeared across more than twenty environments in four regions, at a rate that scaled with each environment's traffic, which pointed to a single platform-wide cause rather than a site-specific one.

How the Application Talks to Its Database

Each environment runs in its own namespace on the shared cluster, with the CMS as a Windows container. To read data, it opens a TCP connection to Azure SQL over the cluster's private network.

A network policy engine sits between the container and the network; it enforces which pods may talk to which destinations. On our Windows nodes this is Calico, a common implementation of Kubernetes network policy.

Network policies are declarative: we describe the allowed traffic, and the engine compiles it into low-level packet-filtering rules. Reduced to the two rules that matter, with internal detail removed, the outbound section looked like this:

egress:
  - to:
      - ipBlock:
          cidr: 0.0.0.0/0        # this rule already allows all outbound traffic
  - to:
      - namespaceSelector: {}    # an empty selector, which matches EVERY namespace in the cluster

The first rule already allows all outbound traffic, so the rule below it is redundant. It also uses an empty namespace selector, which in Kubernetes matches every namespace in the cluster, not none. That breadth did not change what the pod could do, but, as we will see, it changed how often the engine had to rebuild the pod's rules.

Open egress does not mean one environment can reach another. Isolation is enforced on the receiving side: every customer namespace denies all inbound traffic by default and permits only the specific ingress each environment needs, so a connection from one environment to another is refused no matter what the sender's egress rules allow. The rules above govern outbound traffic to shared infrastructure such as the database; they do not open any path between customers.

The redundancy looked harmless. It was the center of the problem, but we ruled out everything else first.

The Investigation and Everything We Ruled Out

We started at the database and the network and worked outward. The table lists the hypotheses we eliminated with a single direct test. The threads after it went deeper and steered us toward the answer.

We traced the full path from application to database in the infrastructure code and theorized that idle connections were losing their outbound mapping at a gateway. Extending that gateway's timeout and adding network permissions in pre-production changed nothing. A query run from inside the application showed which address the database saw the connection come from, proving database traffic never used that gateway. That ruled out the theory and pointed us at the network policy engine.

Two tests showed the platform itself was healthy:

• Test applications that only opened a plain database connection and queried it in a loop, bypassing the data-access layer, ran 22+ hours with zero failures. The network and database were reliable; the problem was specific to the full application stack when a connection was disturbed.
• With Microsoft support, packet captures on the affected servers (one nearly 11 GB) showed the application-side stack resetting connections. The failures were genuine network-level disconnects, and their cause was below the application.

We had now eliminated the database, application logic, container and node images, several framework and driver versions, and multiple network layers. The disconnects were real; something on the node was breaking established connections. We did not yet know what.

The Breakthrough: A 19-Second Clue

Network traces captured on an affected Windows node, across two sessions, gave the answer. Aligning the application errors with the low-level network logs and the network policy engine's logs revealed a consistent pattern: each connection failure was preceded, about 19 seconds earlier, by the engine reprogramming that pod's rules. The interval held to the millisecond across four incidents on two days.

Nineteen seconds is close to how long the Windows TCP stack retransmits unacknowledged data before declaring a connection failed. So the real event happened 19 seconds before the visible error: the connection was silently disrupted, and Windows only reported failure after retransmitting and giving up.

That left one question: why was the engine reprogramming the pod's rules so often, at moments unrelated to the application's own activity?

How We Captured the Evidence

Seeing a millisecond-level link between an application error and a networking event takes more than reading a log file. The proof came from three sources, captured on the affected Windows nodes and aligned on one timeline:

• Node network captures. Windows-level network and ETW captures (one run was nearly 11 GB), collected with Microsoft's TroubleShootingScript toolset (TSS) alongside Microsoft support and analyzed in Wireshark, showed the TCP resets and the retransmission window.
• Policy-engine logs. Calico and the Windows Host Networking Service (HNS) record when a pod's rules are reprogrammed, which gave us the reprogramming timestamps.
• A source-address check from inside the database, which confirmed the path SQL traffic actually took and ruled out the gateway theory.

The commands below are representative, with identifiers removed.

# Effective network policy and recent pod events
kubectl get networkpolicy -n <namespace> -o yaml
kubectl describe pod <cms-pod> -n <namespace>

# Node-level network + ETW capture with Microsoft's TSS toolset (https://aka.ms/getTSS), run with Microsoft support
.\TSS.ps1 -Netsh -PktMon
#   ...reproduce the failure...
.\TSS.ps1 -Stop
# TSS packages the captures (converted for Wireshark) with the correlated logs

# Windows Host Networking Service policy programming (where Calico writes the pod's rules)
Get-HnsPolicyList | ConvertTo-Json -Depth 6

-- From the application's own database session: which address does SQL see?
SELECT client_net_address FROM sys.dm_exec_connections WHERE session_id = @@SPID;

No single one of these showed the problem. Lining them up on the same timeline is what exposed the 19-second gap.

The Root Cause

Calico on Windows does not update a pod's rules incrementally. When the endpoints a policy references change, it rebuilds the pod's entire rule set, and during that rebuild there is a brief window where packet filtering is torn down and reapplied.

The redundant rule used an empty namespace selector, so it referenced every namespace in the cluster. The engine therefore treated any namespace change anywhere as relevant to this pod. On a shared cluster, pods start and stop constantly, so unrelated activity in any environment forced a full rebuild of this pod's rules.

If a database query was in flight during a rebuild, its packets were dropped. Windows retransmitted for about 19 seconds, got no acknowledgement, and surfaced "Connection has been closed" or "The semaphore timeout period has expired." The application returned a 500, and the next request succeeded on a fresh connection.

In short, ordinary activity anywhere on the shared cluster was intermittently breaking this environment's database connections, through a redundant rule that referenced the whole cluster and was not needed. It also explains why the rate scaled with traffic: more environments and more in-flight requests meant more chances for a rebuild to hit an active query.

The Deeper Cause: Why a Brief Interruption Becomes a Failed Request

Removing the rule explains the scale of the problem, but not why a brief interruption fails a request at all. On a robust connection, a momentary blip should be invisible.

The reason is the data-access layer, which is why the same error appears, far less often, on the legacy architecture, which does not use Kubernetes or this policy engine. Every cloud network has occasional transient interruptions: brief resets from the database service, idle-connection cleanup, routine maintenance. On the legacy architecture those are the only sources and are infrequent, so the errors were infrequent. Sitefinity's data-access layer, a mature object-relational mapper, does not always recover when an interruption lands mid-operation, and surfaces the failed-connection errors we saw.

On the new architecture, the redundant rule added a large new source of exactly those mid-operation interruptions, at least fifty times the legacy rate. (An exact ratio is unreliable, because the platforms carry different customers and traffic.) The data layer's intolerance of interrupted connections was the fire; the redundant rule was the fuel.

Removing the rule removed the fuel and returned the rate to legacy levels. Making the data layer transparently resume a connection severed mid-operation is hard, and not something a client-library change alone solves, so the practical mitigation is to remove the sources of interruption, which is what the fix did. A few residual errors remain; some appear to have a separate, concurrency-related cause we are addressing in parallel.

The Fix

We removed the redundant rule so the policy no longer references every namespace in the cluster:

egress:
  - to:
      - ipBlock:
          cidr: 0.0.0.0/0        # a single broad allow rule; nothing scoped cluster-wide

Because the rule lived in the shared configuration used by every CMS environment, this is not a per-site patch; each environment picks it up on its next deployment. Where it has been deployed, comparing the same period before and after the fix shows this class of error down by about 99.6%; in most projects it dropped to zero, with only a few edge cases remaining elsewhere. That returns the experience to legacy levels. It required no application change, no connection-string change, and no downtime.

The failure sequence, and why the fix closes it:

1. A CMS pod has a healthy database query in flight.
2. An unrelated pod starts or stops somewhere in the shared cluster.
3. Because the redundant rule referenced every namespace, the Windows policy engine rebuilds the entire rule set for the CMS pod from scratch.
4. During that rebuild, the in-flight packets to the database are dropped.
5. Windows retransmits for about 19 seconds, receives no acknowledgement, and reports the connection as failed.
6. The request returns an HTTP 500. The next request succeeds immediately on a fresh connection.

With the cluster-wide rule gone, unrelated pod activity no longer triggers a rebuild of this pod's rules, so there is no window in which an active connection can be severed.

What This Means and What We Are Doing Next

The incident was the interaction of a sensitivity and an amplifier: the data-access layer does not always recover from a mid-operation interruption, and a redundant network rule plus the engine's wholesale rebuilds made those interruptions far more frequent. No single part was dangerous on its own.

We are:

• Modernizing the data-access layer for a separate, concurrency-related cause. Sitefinity's data-access layer is built on Telerik Data Access, whose active development ended years ago. To be clear about scope: this will not make the layer resume a connection severed mid-operation, since we tested a modernized build and that failure still occurred. What it changes is behavior when two operations contend for the same connection. The updated build targets .NET Framework 4.8 and uses the modern Microsoft.Data.SqlClient instead of the legacy System.Data.SqlClient, which handles that concurrency better. The full Data Access and Sitefinity test suites pass. We hope this clears the residual edge cases, but we are still validating.
• Reporting the behavior to Microsoft so the "full rebuild on any change" behavior of the Windows network policy engine can be improved for everyone running Windows workloads on Kubernetes.

Most of what made this hard sits well below the layer customers build on: correlating kernel-level packet captures with policy-engine internals, reproducing a Windows-container networking edge case, and driving it to a resolution with Microsoft. Absorbing this class of problem, so teams can keep shipping content and code, is what the managed platform is for.

Solving this took a cross-functional effort sustained over weeks, and around the clock in the final two weeks. Engineers from our platform, infrastructure, and product teams, our support organization, and colleagues at Microsoft worked it together: capturing traces, running test workloads, reading packet captures, and eliminating hypotheses until the 19-second pattern surfaced. Our thanks to everyone involved.