Vary: Accept-Encoding

Specify a Vary: Accept-Encoding header

bootstrapcdn-vary-accept-encoding-2

Bugs in some public proxies may lead to compressed versions of your resources being served to users that don’t support compression. Specifying the Vary: Accept-Encoding header instructs the proxy to store both a compressed and uncompressed version of the resource.

  • The cache-control header is the primary mechanism for an HTTP server to tell a caching proxy the “freshness” of a response. (i.e., how/if long to store the response in the cache)
  • In some situations, cache-control directives are insufficient. A discussion from the HTTP working group is archived here, describing a page that changes only with language. This is notthe correct use case for the vary header, but the context is valuable for our discussion. (Although I believe the Vary header would solve the problem in that case, there is a Better Way.) From that page:

Vary is strictly for those cases where it’s hopeless or excessively complicated for a proxy to replicate what the server would do.

  • This page describes the header usage from the server perspective, this one from a caching proxy perspective. It’s intended to specify a set of HTTP request headers that determine uniqueness of a request.

A contrived example:

Your HTTP server has a large landing page. You have two slightly different pages with the same URL, depending if the user has been there before. You distinguish between requests and a user’s “visit count” based on Cookies. But — since your server’s landing page is so large, you want intermediary proxies to cache the response if possible.

The URL, Last-Modified and Cache-Control headers are insufficient to give this insight to a caching proxy, but if you add Vary: Cookie, the cache engine will add the Cookie header to its caching decisions.

Finally, for small traffic, dynamic web sites — I have always found the simple Cache-Control: no-cache, no-store and Pragma: no-cache sufficient.

Edit — to more precisely answer your question: the HTTP request header ‘Accept’ defines the Content-Types a client can process. If you have two copies of the same content at the same URL, differing only in Content-Type, then using Vary: Accept could be appropriate.

Update 11 Sep 12:

I’m including a couple links that have appeared in the comments since this comment was originally posted. They’re both excellent resources for real-world examples (and problems) with Vary: Accept; Iif you’re reading this answer you need to read those links as well.

The first, from the outstanding EricLaw, on Internet Explorer’s behavior with the Vary header and some of the challenges it presents to developers: Vary Header Prevents Caching in IE. In short, IE (pre IE9) does not cache any content that uses the Vary header because the request cache does not include HTTP Request headers. EricLaw (Eric Lawrence in the real world) is a Program Manager on the IE team.

The second is from Eran Medan, and is an on-going discussion of Vary-related unexpected behavior in Chrome: Backing doesn’t handle Vary header correctly. It’s related to IE’s behavior, except the Chrome devs took a different approach — though it doesn’t appear to have been a deliberate choice.


 

That’s great for direct access, but modern networks use intermediate caches and CDNs. And there’s the problem: how does the cache use headers to decide what to send back? How can it replicate the server’s decision-making logic?

Vary to the rescue. The Vary header describes what information “uniquely” identifies a request — caches should only be used if the incoming request matches the Vary information in the cache.

For example, if a server sends the Vary: User-Agent header, intermediate caches will store a separate cache entry for each User-Agent they see (every OS + browser combination, yikes). This behavior was an issue for me in support (we’re hiring!), because we saw origin servers getting hammered as each user-agent requested new content and sidestepped the cache. After some research, I figured out why this happened (turn off Vary: User-Agent), but the header left a bad taste in my mouth.

 

The fix is for the origin server to send back Vary: Accept-Encoding. Now the intermediate CDNs will keep separate cache entries (one for Accept-encoding: gzip, another if you didn’t send the header). These days you’re unlikely to have clients without compression, but why risk cache mixups?

Origin servers should include Vary: Accept-Encoding, and here’s how:

Apache/.htaccess

<IfModule mod_headers.c>
  <FilesMatch ".(js|css|xml|gz|html)$">
    Header append Vary: Accept-Encoding
  </FilesMatch>
</IfModule>

Nginx

gzip_vary on

IIS

<system.webServer>
  <httpProtocol>
    <customHeaders>
    <remove name="Vary"></remove>
    <add name="Vary" value="Accept-Encoding"></add>
    </customHeaders>
  </httpProtocol>
</system.webServer>