In theory, this makes perfect sense, but in practice, it causes web developers like me a lot of pain, because nothing "permanent" is actually that permanent. I'm not saying no browser should ever cache a redirect, but I do have a few suggestions of ways they could be a little more helpful about it.

The Technology

Let's be clear what we're talking about - the HTTP/1.1 specification defines a class of status codes grouped as "Redirection" statuses, in the range 3xx. The main candidate for browser caching is status code 301, which the specification labels "Moved Permanently". The specification specifically states that "this response is cacheable unless indicated otherwise" - that is, unless the response also includes headers specifically relating to cache control, a User Agent can assume that it's OK to cache this response and not re-request the original URL. It doesn't go into any more detail, and neither this nor the next sentence (about "clients with link editing capabilities") is couched in the standard RFC form of "User Agents MAY ..." (let alone "SHOULD"), but the clear message is "this old URI is irrelevant, just use the new one".

So, newer versions of Firefox, Chrome, and Internet Explorer have all started "obeying" this part of the standard, and opting to cache all HTTP 301 responses if not told otherwise.

The Problem

So, what's the big problem? Well, the fact that a "resource has been assigned a new permanent URI" doesn't mean that there won't be some point in time where someone wants to use the old URI for something else. Worse, people have an unhelpful habit of changing their decisions later.

Consider this scenario:

1. A  company has a product called a "Thingummy™", with a description at http://example.com/thingummy/
2. They decide that the name is too unwieldy, so re-brand it as "Widget™". Knowing that cool URIs don't change, the developers permanently redirect the old URL to the new page about the same product, at http://example.com/widget/ Old links remain valid, and direct customers to the information they were looking for.
3. A couple of years down the line, the company decides to boost sales by releasing a "classic" version of the Widget™ under the old Thingummy™ brand. They put up a new page at http://example.com/thingummy/ Sadly, some customers continue being redirected to http://example.com/widget/ by their browsers' cache.
4. With Thingummies™ massively out-selling Widgets™, the company comes full circle, and abandons the new brand. The developers put in a redirect that permanently points http://example.com/widget/ back to http://example.com/thingummy/. At this point, all hell breaks loose. Well, maybe not, but customers trying to access the product page find mysterious messages on their screen about "infinite redirects". Management are not impressed.

Escaping the Trap

The first hurdle for developers is that it's not obvious what the hell is going on, even when they're just testing out a few different URL schemes in their development copy. I think whatever unacceptable language this developer used on MozillaZine probably sums up the feeling most of us had when first encountering this invisible magic.

But even once you've figured it out, it's not that obvious what to do about it - do you do a deep clean of the browser's cache every time something's not quite right? I see a sledgehammer approaching a nut. And if you've released your mistake to a non-technical user, perhaps on a preview version of the site, you're going to have to talk them through this process as well.

Caching is a pain sometimes, but it's been around for a long time, and we have UIs for dealing with it. The most common of these is the "hard refresh" - hold down Ctrl, or Shift, and hit the reload button or keyboard shortcut, and the cache is by-passed completely and content is reloaded. Brilliant. Oh, but it only reloads the page you're looking at, not the URI you originally requested, so it's useless for cached redirects.

→ Suggestion 3: Make a "hard refresh" request the original URI navigated to, not the one currently being viewed.

→ Filed as Mozilla Bugzilla RFE 696650

Doing the Right Thing

Apparently, what we're all doing wrong, as developers, is not sending appropriate cache headers along with the 301 status code. If you're writing, say, a PHP script, and using header('Location: foo'), you should probably be doing it in some kind of wrapper function, so you can make up your own default expiry, and make sure you send a whole bunch of control headers whenever you redirect.

But a lot of redirects are not written in a rich server-side scripting language, they use specific tools built into the web server, like Apache's incredibly powerful mod_rewrite. I just checked, and there is no function in the latest version of mod_rewrite that lets you control caching, or send arbitrary HTTP headers, when it generates a 301 response. I'm sure there are ways of stringing together a whole bunch of Apache directives to achieve the desired effect, but it would take me a while to come up with the right combination, and it would look a mess.

→ Suggestion 4: Web server redirect functions, such as mod_rewrite, should build in control over caching headers.

→ Posted to Apache HTTPD dev list

Inconsistent Behaviour

Finally, let's assume we're using a tool for our redirects that lets us control the cache headers, and we don't have any old immortal redirects to avoid infinite loops with. All is fine, and the browsers will do the right thing and save us some network traffic, right?

Well, according to this test result table from someone who is in favour of redirect caching, probably not. Like everything out there in web land, you have to deal with every browser handling things just a little bit differently. Combinations which should work might or might not actually be reliable across browsers.

Even client-side caching of standard resources is, as one comment I just came upon puts it "a bit of a black art"

→ Suggestion 5: Somebody needs to work out what the most common scenarios for redirect caching actually are, and how to achieve them.

Conclusion

My gut instinct the first, second, and twelfth times I ran up against redirect caching was that it was stupid, and the browser developers should take it out immediately. But I do see that it makes sense, and could save a lot of unnecessary network traffic and server load. So instead, all I ask is that people don't brush the problems off with "oh, well, you should have known what permanent means", and look at how to make redirect caching work in the real world.

And yet, some people seem to do their utmost to make all the hyperlinks in their documents entirely useless. Here are my Top 10 Things Not To Do with Links...

But the vision tends to overlook the nature of both computers and humans. On the one hand, humans have a limited memory, and a flawed ability to apply consistent logic; on the other hand, we have abilities at creatively interpreting knowledge and ideas that are far beyond the capacity of any computer so far designed.

Big Calculators

Artificial Intelligence is a fascinating area of research - it's what I studied at University - but it is one that has had something of a reality check over the years. At first, it seemed like only a matter of time before computers of some sort or another would exceed all human abilities, ushering in a technological utopia - or perhaps an apocalypse. But it turns out there are things computers are really good at and things they're really bad at.

The classic example is chess computers: when researchers started building them, the reasoning was that since it takes an intelligent human to play chess, a computer that could play chess would be intelligent too. Eventually, they built a computer that could consistently beat the best human chess players; unfortunately, they discovered that that's all they'd built - they hadn't broken through to a new level of computing, they'd just broken down chess into a very long list of sums. And computers are good at sums.

Another example is neural networks - the wonderfully futuristic sounding "Multi-Layer Perceptron" is a learning system based on simplified mathematical models of neurons, the building blocks of the human brain. You can "teach" such a network to recognise things, and it starts seeming very clever. But researchers realised that you don't really need the clever model at all, because it's all just maths - and indeed you can make a so-called "N-tuple Neural Network" that consists entirely of lookup tables, with no real maths involved, and certainly no "brain". And while there are now very handy machines that can recognise faces, it's still no use trying to have a conversation with one.

Whose Markup is it Anyway?

The "Semantic Web", at the moment, is like that early AI research - it all seems just around the corner, if only we all try a little bit harder. In fact, the Web has already come a long way, and while the basic language of HTML remains the same, its usage has changed dramatically over the years.

At first, HyperText Markup Language was a very simple way of "marking up" documents in the big text repository that was the World Wide Web - separating out paragraphs, marking headlines and important bits, and, of course, turning text into "anchors" which "hyperlinked" documents together. As it grew, the Web demanded more, prettier, pages, and HTML grew into a rich, presentational language - with colours, tables, images, even simple animations. Then, the needs to automate complex and interactive systems meant HTML had to become a structural language as well - one that could identify blocks of a page, take them apart, and join them back together in a different order.

At this stage, a question arises, which is just how much structure does a document need? Two technologies readily associated with structural HTML are XHTML - HTML with the added rigidity and tools of XML - and the DOM - a way of look at a document as a big tree of "nodes". This is an extremely useful way of looking at a page with lots going on - you can work dynamically with the navigation controls, or the interactive comment system, without touching the main content area. But how useful is it inside the content? In the sentence "I really like structure", the word "really" is wrapped in its own HTML element, but not because of any structural significance - it's just there for presentation.

Now, we're told, HTML needs to evolve further, into a semantic markup, which doesn't just separate out the blocks, it labels them all meaningfully. I've used italics as an example deliberately, because a common misconception is that the <i> tag is bad - it's presentational - and <em> is better - it means emphasise this, rather than draw this in italics: much more semantic, surely? And so the Rich-Text control I'm typing in right now has a button labelled "I" - the recognised icon for italics - which inserts an <em> tag; it even replaces <i> with <em> if you edit the HTML! This is, obviously, a complete misunderstanding - a straight find-and-replace can't magically gain us meaning - but the fact is, it's what people are used to, and it's people that write the content.

So this is the second conflict facing the Web - authors don't want to add all this information. The real reason <i> is discouraged is that when it's not being used for emphasis, the machine has no way of knowing why it is being used: it might be to indicate a foreign phrase, like in situ, in which case we could indicate what language it is; or it might be the name of a publication, like The Times, in which case we could label it with some appropriate global identifier. But the fact is, a human reader won't gain anything from this, and content is written by humans, for humans.

Playing to our Strengths

In the end, the Semantic Web will not be built by insisting that I tell WordPress that in situ is Latin - who would it help if I did? A blog entry is not, and never will be, a database - it's a big block of text, or perhaps a few medium-sized blocks of text, with some decoration and links thrown in. The structural markup can help us divide up those blocks, and mark off the bits of the page which aren't part of the blog entry at all; the semantic markup will improve how those blocks get labelled.

If you want to find pages about a particular topic, what you need is a really big index, and that's a good job to give a computer; if you want to find "the right page" about a particular topic, it's up to you to decide what "right" means. We can help the computers to help us - by making articles that explicitly label their own topics, for instance (as long as you can trust them, but that's another issue). We can even decide that sometimes we do want to embed little bits of data in our content - if we know an address might be useful to someone, we could label it with an appropriate microformat, so they can more easily feed it to another piece of software. But if they want to look up what in situ or Perceptron actually means, we can leave it to a human to look it up in an appropriate index.

This may be a disappointing prognosis if you were hoping the web could tell you the answers without you having to read the articles, but think of it as a division of labour: the computers can get better at categorising and filtering the content, we can carry on being good at understanding it.