The Refresh Rate of the Real World [Digital Density]

(Times Square Photo from flickr)

What if all we had was photographs? What would future people guess our world was like? And if they had millions of photographs to search through? Billions? What would they come to understand about 2010 if they had just a photographic record?

Times Square, New York City is not the most photographed place in the world, according to a study of all 35 million geotagged photographs on Flickr, but it is in the top hundred. Does that make it an important, historical place?

Digital cameras have outsold film cameras for nearly a decade, and every year they get lighter, smaller, and able to hold more photographs. People take more pictures with them, and more of those pictures get shared with the general public on Flickr, Facebook, and other social platforms. Times Square, then, will only become more photographed as time goes on, and as time goes on more and more of that photography will enter the near-public record, on Facebook, Flickr, and other social sites (not to mention whatever ends up part of Homeland Security databases, etc). What sort of world will all that data show off?

We already have such a photographic record, and I think we can start answering those questions.

This photographic record is getting more metadata-rich too. Metadata is data about data- in the case of photographs, things like the time and day the photo was taken, the latitude and longitude of the camera, who took the picture, what the light was like that day, and a host of other things.

As cellular phones and other devices incorporate better and better digital cameras, and as stand-alone cameras add better and better features, digital capture devices can add more and more metadata to those photographs, from Coordinated Universal Time (UTC) to specific GPS geo-tagging.

As recently as two years ago, Microsoft and the University of Washington began taking advantage of all this tagged and coded photo-data to build Photosynth, a software package that generates three dimensional models of heavily-photographed areas, as well as stitched-together continuous images of that object. This is their demo video, from 2008, showing their model of St. Peter's Basilica:



It's a pretty neat demo, but it is also a demo of a highly photographed space, on the order of Times Square or the Eiffel Tower. Sitka, Alaska, on the other hand, probably doesn't get photographed nearly as often, and would not have made the most interesting of demos.

It may sound obvious, but there are a couple of reasons for this. Sitka is remote: it's a major fishing port, and a stopover for Alaskan tourism, but not an international trade center nor an airline hub. Not to be cruel, but Sitka is less important to the vast majority of travelers than the commercial spectacle of Times Square.

One may note from the video above that so photographed is St. Peter's Basilica that the video-authors can actually show the progression of an event registration just by browsing deeper into the future-end of the time-scale of the photos taken and seeing volunteers arrive, chairs unfold, and up spring tables.

If the photographs taken of a place can be imagined as a hyper-rectangular volume populated with photograph points set in space, (as they are in the above video) and perhaps colored by time (red is the past, blue is the future, a feature not in the video), but limited to an hour or a day or so, then one can see that Times Square would be a virtual sea of reference-frames, with thousands and thousands of data-points, many of them overlapping spatially, but splayed across time (implying that there are certain iconic shots that everyone wants to capture). But not all places in the world could boast such a crowded ocean of data.

The photograph of a woman standing in front of Sitka National Forest has no such overlapping photography. Instead, its volume would show something more like a wandering line through space and time, with just a few photo-points demarking its passage. From the photostream the picture is part of, one can see that it is part of a series of vacation photos, a single point in a long, narrow tunnel of personal experience, nowhere near as data-rich as an intersection like Times Square or a landmark like the Basilica.

If one extended the time-frame of the Sitka National Forest volume to a week or so, one might come upon a few dozen more such tunnels-through-space-time, whose various photo-points might reveal what people in general find interesting at the site (the totem pole, apparently).

Comparing these three space-time volumes (Sitka, Times Square, and St. Peter's Basilica), one notes that, beyond the data conveyed by the photographs, there is meta-data that can be derived from photographic density. Density, in this case, is almost shorthand for interestingness or famousness. The thought might be I was here at this important thing, and that makes me a little important too. Let me record this temporal-spacial meeting.

If one was mining this photographic data for, say, an idea of the number of people who were at a rally, or what might have happened during a riot, then density of photodata around that event is a key factor.  Places that are heavily photographed, in effect, can be more accurately surveilled -- the gaps in the record are correspondingly smaller.

So when historically important events happen at famous or crowded locations, this meta-data composed out of many different photographic references becomes magnified, sometimes overwhelming the photographic data itself. An act can appear innocent or incredibly guilty, depending on exactly how it was captured on film. If a man spends five minutes tying his shoes, is he really clumsy, or is he watching someone? The photographic record can save or damn him.

The Kennedy Assassination, for example, has been so scrutinized that every single frame of the famous Zapruder Film has been carefully scanned and preserved, and various conspiracy theories have arisen not based on what was in that series of 18.3 photographs/second, but on what might have happened between frames and whether or not the between-frame events make sense. The questions arise not because of the data itself, but because it is not dense enough.To use a more computer-era phrase, the Zapruder Film needed a higher capture and refresh rate. In general, history would be better understood if data on Dealey Plaza refreshed faster for those key seconds (The Watchmen movie used this data paucity to great effect (at 2:43)).

In a way, one can think of photo-density as bandwidth: data-transfer between the real world and the nebulous digital record that shadows it. The Zapruder film is like a bad dial-up connection: there is a lot of noise mixed in with all that signal. Times Square, on the other hand, is like broadband- a veritable river of data. Sitka would be like a satellite phone: often off, but data-heavy when in use.

But does photo-density actually improve surveillance? I would argue that it doesn't. If photo-dense places are people-dense places, then there are a vast number of crimes, from murder to rape to robbery, that simply would not be captured in the public photo record, as they are not crimes one commits around big crowds of people.

My guess is that the photo-density in the real world is only going to increase as time goes on, and that the higher it goes, the more and more metadata we will figure out how to draw from it. The number of places data-rich enough to be accurately and usefully modeled in both space and time will only go up. The algorithms for enmeshing all this photo-data and recognizing real-world objects within it will only get better. The willingness of people to add their own photographic data to the general data will go up (though whether this is out of a newfound cultural sense of the commons or because a form online somewhere is set up as an opt-out rather than opt-in remains to be seen).

In the future, Times Square may still be highly photographed in the public record. But the rest of the tail of that graph will have seriously caught up.

I doubt that this spread of photo-density to more places will do much to improve security, capture more criminals, or make huge discoveries about human interaction.

No- I think that its ultimate contributions will arrive in the same ad-hoc way that its data does- small, interesting, strange patterns that careful, persistent data-mining will make quite clear.