There are a lot of URLs out there on the Web; and a pretty big number of those URLs are either alternative names for something, or old locations that have been superseded. So “redirects” from one URL to another are a common feature of the web, and have been for many years. But recently, the way these redirects behave has been changing, because performance-conscious browser developers have started caching redirects, rather than re-requesting them from the server every time.<\/p>\n
In theory, this makes perfect sense, but in practice, it causes web developers like me a lot of pain, because nothing “permanent” is actually that<\/em>\u00a0permanent. I’m not saying no browser should ever cache a redirect, but I do have a few suggestions of ways they could be a little more helpful about it. Let’s be clear what we’re talking about – the HTTP\/1.1 specification defines a class of status codes grouped as “Redirection” statuses<\/a>, in the range 3xx. The main candidate for browser caching is status code 301, which the specification labels “Moved Permanently”. (( On a pedantic note, while that is the heading in the RFC, and the suggested human-readable “Reason Phrase”<\/a>, it is not actually the definition of the status, as is sometimes implied when discussing it. ))\u00a0The specification specifically states that “this response is cacheable unless indicated otherwise” – that is, unless the response also includes headers specifically relating to cache control, a User Agent can assume that it’s OK to cache this response and not re-request the original URL. It doesn’t go into any more detail, and neither this nor the next sentence (about\u00a0“clients with link editing capabilities”)\u00a0is couched in the standard RFC form of “User Agents MAY …” (let alone “SHOULD”), but the clear message is “this old URI is irrelevant, just use the new one”.<\/p>\n So, newer versions of Firefox, Chrome, and Internet Explorer<\/a> have all started “obeying” this part of the standard, and opting to cache all HTTP 301 responses if not told otherwise.<\/p>\n So, what’s the big problem? Well, the fact that a “resource has been assigned a new permanent URI” doesn’t mean that there won’t be some point in time where someone wants to use the old URI for something else. Worse, people have an unhelpful habit of changing their decisions later.<\/p>\n Consider this scenario:<\/p>\n The biggest problem in all this is that developers seem to have taken the word “permanent” rather too literally. The Random House dictionary (( by which I mean dictionary.com<\/a>, because I couldn’t be bothered to go downstairs and find the OED )) lists 4 definitions of permanent; the first is:<\/p>\n existing\u00a0perpetually;\u00a0everlasting,\u00a0especially\u00a0without significant\u00a0change<\/p><\/blockquote>\n In theory, this seems a reasonable definition, but in reality nothing lasts forever<\/strong>. Nothing physical, nothing electronic, and certainly not the structure of a website. So clearly, when the HTTP specification says “a new permanent URI”, it doesn’t actually expect you to guarantee its perpetual existence. Let’s try the second definition:<\/p>\n intended\u00a0to\u00a0exist\u00a0or\u00a0function\u00a0for\u00a0a\u00a0long,\u00a0indefinite\u00a0period without\u00a0regard\u00a0to\u00a0unforeseeable\u00a0conditions<\/p><\/blockquote>\n Ah, now that’s more like it – by that definition, a 301 status indicates a redirect which will remain valid for “a long, indefinite period”, unless there are “unforeseeable\u00a0conditions”.<\/p>\n I did a quick test earlier, using Firefox 7 to visit a URL which returned a 301 status and no cache instructions, then inspecting the resulting cache entry using the built-in “about:cache page”. Among the details is this – “expires: No expiration time”. That’s pretty permanent, by the first definition.<\/p>\n Effectively, having once seen that 301 response, it is not going to request the original URL ever<\/strong>, simply because the web developer didn’t mention an expiry date, because they didn’t foresee<\/em>\u00a0any conditions in which they would want to change the decision. (And I thought 24-hour TTL<\/abbr>s on DNS<\/abbr> entries were annoying…)<\/p>\n → Suggestion 1: Cache entries for an unadorned 301 response should have a reasonable default life time, not last forever.<\/strong><\/span><\/p>\n → Filed as Mozilla Bugzilla Bug 696595<\/a><\/span><\/p>\n If you’re foolish enough to have created an immortal redirect, or unlucky enough to have inherited one, you might find yourself wanting to put a new redirect pointing back the other way, as in my example above. But if you do, any browser which saw (and cached) the old redirect will simply see the new one as well<\/em>, and follow both back and forth, back and forth, back and forth … until eventually it decides there’s an infinite loop and throws the user an error.<\/p>\n So instead, you have to resort to all sorts of confusing workarounds to keep everything working<\/a>.<\/p>\n But the browser knows<\/em>\u00a0something is wrong, and it’s not the user’s fault, it’s bad data in the browser cache (depending how you look at it, that’s the web developers fault, but a browser that doesn’t cut developers some slack won’t get very far rendering real-world HTML…)<\/p>\n → Suggestion\u00a02: When an infinite redirect is detected, try skipping the cache.<\/span><\/strong><\/p>\n → Filed as Mozilla Bugzilla Bug 696646<\/a><\/span><\/p>\n<\/div>\n The first hurdle for developers is that it’s not obvious\u00a0what the hell is going on, even when they’re just testing out a few different URL schemes in their development copy. I think\u00a0whatever unacceptable language this developer used on MozillaZine<\/a>\u00a0probably sums up the feeling most of us had when first encountering this invisible magic.<\/p>\n But even once you’ve figured it out, it’s not that obvious what to do about it – do you do a deep clean of the browser’s cache every time something’s not quite right?<\/a>\u00a0I see a sledgehammer approaching a nut. And if you’ve released your mistake to a non-technical user, perhaps on a preview version of the site, you’re going to have to talk them through this process as well.<\/p>\n Caching is a pain sometimes, but it’s been around for a long time, and we have UIs for dealing with it. The most common of these is the “hard refresh”<\/a> – hold down Ctrl, or Shift, and hit the reload button or keyboard shortcut, and the cache is by-passed completely and content is reloaded. Brilliant. Oh, but it only reloads the page you’re looking at<\/em>, not the URI you originally requested, so it’s useless for cached redirects.<\/p>\n → Suggestion\u00a03: Make a “hard refresh” request the original URI navigated to, not the one currently being viewed.<\/strong><\/span><\/p>\n → Filed as Mozilla Bugzilla RFE<\/abbr> 696650<\/a><\/span><\/p>\n Apparently, what we’re all doing wrong, as developers, is not sending\u00a0appropriate cache headers along with the 301 status code. If you’re writing, say, a PHP script, and using header('Location: foo')<\/tt>, you should probably be doing it in some kind of wrapper function, so you can make up your own default expiry, and make sure you send a whole bunch of control headers whenever you redirect.<\/p>\n But a lot of redirects are not written in a rich server-side scripting language, they use specific tools built into the web server, like Apache’s incredibly powerful mod_rewrite. I just checked, and there is no function in the latest version of mod_rewrite<\/a> that lets you control caching, or send arbitrary HTTP headers, when it generates a 301 response. I’m sure there are ways of stringing together a whole bunch of Apache directives to achieve the desired effect, but it would take me a while to come up with the right combination, and it would look a mess.<\/p>\n → Suggestion\u00a04: Web server redirect functions, such as mod_rewrite, should build in control over caching headers.<\/strong><\/span><\/p>\n → Posted to Apache HTTPD dev list<\/a><\/span><\/p>\n Finally, let’s assume we’re using a tool for our redirects that lets us control the cache headers, and we don’t have any old immortal redirects to avoid infinite loops with. All is fine, and the browsers will do the right thing and save us some network traffic, right?<\/p>\n
\n<\/p>\nThe Technology<\/h2>\n
The Problem<\/h2>\n
\n
How Permanent is “Permanently”?<\/h2>\n
Chasing your own Tail<\/h2>\n
Escaping the Trap<\/h2>\n
Doing the Right Thing<\/h2>\n
Inconsistent Behaviour<\/h2>\n