Dynamic Routing in Envoy with a Custom Go Filter: A Practical Guide

Introduction

While developing a custom Go filter for Envoy Proxy, I faced a challenge: after updating request headers in the filter, Envoy's routing logic did not consider the new header values. This post explains the issue, the solution, and the code changes—helping engineers avoid similar problems when extending Envoy with Go.

The Problem

Suppose you want to route requests dynamically based on a custom header (X-Project-Id). The Go filter should:

• Inspect the incoming request for X-Project-Id
• Determine a target environment based on the project ID
• Set a new header (x-target-env) to influence routing

However, after updating the header, Envoy still routed based on the original request. The new header value was ignored in route selection.

The Root Cause

Envoy determines the route early in the request lifecycle and caches it. If you modify headers in a filter, you must clear the route cache for Envoy to re-evaluate routing with the updated headers.

The Solution

Call f.callbacks.ClearRouteCache() after updating the headers. This forces Envoy to re-calculate the route using the latest header values.

Updated Go Filter Code

Here's the relevant section of the Go filter, with new header names and improved comments:

// DecodeHeaders is called during the request path.
// endStream is true if the request has no body.
func (f *filter) DecodeHeaders(header api.RequestHeaderMap, endStream bool) api.StatusType {
        f.path, _ = header.Get(":path")
        api.LogDebugf("Received path: %s", f.path)

        // Run header manipulation in a goroutine to avoid blocking Envoy.
        go func() {
                defer f.callbacks.DecoderFilterCallbacks().RecoverPanic()

                // Check for the custom project header
                projectID, ok := header.Get("X-Project-Id")
                if ok {
                        targetEnv := f.determineTargetEnv(projectID)
                        f.updateHostHeader(header, targetEnv)

                        // Set the routing header
                        header.Set("x-target-env", targetEnv)
                        api.LogDebugf("Set x-target-env header to %s", targetEnv)

                        // Clear the route cache so Envoy re-evaluates routing
                        f.callbacks.ClearRouteCache()
                        api.LogDebugf("Cleared route cache after header update")

                        // Resume filter processing
                        f.callbacks.DecoderFilterCallbacks().Continue(api.Continue)
                }
        }()

        // Suspend filter until goroutine resumes it
        return api.LocalReply
}

Key Takeaways

• Always clear the route cache (ClearRouteCache()) after modifying headers that affect routing.
• Use goroutines for non-blocking operations in Envoy Go filters.
• Log key actions for easier debugging and observability.

Example Envoy Route Configuration

Here's a sample route config that matches on the dynamically set x-target-env header:

route_config:
    name: project_route
    virtual_hosts:
        - name: project_host
          domains: ["*"]
          routes:
              - match:
                    prefix: "/"
                    headers:
                        - name: "x-target-env"
                          string_match:
                              safe_regex:
                                  regex: "^prod|staging|dev$"

Conclusion

This approach enables dynamic routing in Envoy based on runtime header manipulation from a Go filter. By understanding Envoy's route caching and using ClearRouteCache(), you can build flexible, high-performance proxies tailored to your application's needs.