Valley of the Shadow and the Digital Database

Since its inception as a website in the early 1990s, the digital history project Valley of the Shadow has received awards from the American Historical Association, been profiled in Wired Magazine, and termed a “milestone in American historiography” in Reviews in American History. The project is also widely regarded as one of the principal pioneers within the rough-and-tumble wilderness of early digital history.1 Conceived at the University of Virginia as the brainchild of Edward Ayers (historian of the American South and now president of University of Richmond), the project examines two communities, one Northern and one Southern, in the Shenandoah Valley during the American Civil War. The initiative documented and digitized thousands upon thousands of primary source materials from Franklin County, Pennsylvania and Augusta County, Virginia, including letters, diaries, newspapers, speeches, census and government records, maps, images, and church records.

By any measure, Valley of the Shadow has been a phenomenal success. Over the course of a decade and a half, it has provided the catalyst for a host of books, essays, CD-ROM’s, teaching aids, and articles – not to mention more than a few careers. At times it seems that everyone and their mother in the digital history world has some kind of connection to Valley of the Shadow. The impact the project has had, both within and outside of the academy, is a bit overwhelming. In this light, I decided to revisit Valley of the Shadow with a more critical lens and examine how it has held up over the years.

At the bottom of the Valley‘s portal, it reads “Copyright 1993-2007.” There aren’t many academic sites that can claim that kind of longevity, but this also carries a price. In short, the website already feels a bit dated. The structure of the website is linear, vertical, and tree-like. The parent portal opens up into a choice between three separated sections: The Eve of War (Fall 1859 – Spring 1861), The War Years (Spring 1861 – Spring 1865), and The Aftermath (Spring 1865 – Spring 1870). Each of these are divided into different repositories of source material, from church records to tax and census data to battle maps. Clicking on a repository leads to different links (for instance, two links leading to the two counties’ letters). A few more clicks can lead to, say, a letter from Benjamin Franklin Cochran to his mother in which he leads off with the delicious detail of lived experience that historians love: “I am now writing on a bucket turned wrong side up.”

In this sense, the database is geared towards a vertical experience, in which users “drill down” (largely through hyperlinks) to reach a fine-grained level of detail: Portal -> Time Period -> Source Material Type -> County -> Letter. What this approach lacks is the kind of flexible, horizontal experience that has become a hallmark of today’s online user experience. If one wanted to jump from Cochran’s letter to see, for instance, battle maps of the skirmishes he was referencing or if local newspapers described any of the events he wrote about, the process is disjointed, requiring the user to “drill up” to the appropriate level and then “drill down” again to find battle maps or newspapers. This emphasis on verticality is largely due to the partitioned nature of the website, divided as it is into so many boxed categories. This makes finding a specific source a bit easier, but restricts the exploratory ability of a user to cross boundaries between the sites’ different eras, geography, and source types.

If different sections of the website are partitioned from one another, what kind of options exist for opening the database itself beyond the websites own walls? In October of 2009, NiCHE held a conference on Application Programming Interfaces (APIs) for the Digital Humanities, with the problem it was tackling outlined as follows:

To date, however, most of these resources have been developed with human-friendly web interfaces. This makes it easy for individual researchers to access material from one site at a time, while hindering the kind of machine-to-machine exchange that is required for record linkage across repositories, text and data mining initiatives, geospatial analysis, advanced visualization, or social computing.

This description highlights the major weakness of Valley of the Shadow: its (relative) lack of interactiveness and interoperability. A human researcher can access specific information from the website, but it remains a major challenge to employ more advanced digital research techniques on that information. Every database is inherently incomplete. But one way to mitigate this problem is to open up the contents of a database beyond the confines of the database itself. The following scenario might fall under the “pipe-dream” category, but it illustrates the potential for an online database: a researcher writes a programming script to pull out every letter in Valley of the Shadow written by John Taggart, search both the Valley‘s database and national census records in order to identify the letters’s recipients, capture each household’s location and income level, and use that data to plot Taggart’s social world on a geo-referenced historical map or in a noded social network visualization. Again, this might be a pipe-dream, but it does highlight the possibilities for opening up Valley of the Shadow‘s phenomenally rich historical content into a more interactive and interoperable database.

At the end of the day, Valley of the Shadow deserves every ounce of acclaim it has received. Beyond making a staggering array of primary sources available and accessible to researchers, educators, and students, it helped pave the way for the current generation of digital humanists. Valley of the Shadow embodies many of the tenets of this kind of scholarship: multi-modal, innovative, and most importantly, collaborative. Its longevity and success speaks to the potential of digital history projects, and should continue to serve as a resource and model moving forward.


1 I, for one, imagine the early days of digital history to be a rough-and-tumble wilderness, resplendent with modem-wrangling Mosaic cowboys and Usenet bandits.

11 thoughts on “Valley of the Shadow and the Digital Database

  1. Hmm, the increasingly dated feel is a fair point. But the UI here was something that grew out of the sources themselves as we got each of them (or part of them) up. [Speaking of dated, on the list of people who have worked on the project you link to, I’m listed as a grad student, a status I haven’t had for 6+ years.]

    Still, as you point out, the project was path-breaking, both for digital humanities and digital humanists, many of whom cut their teeth on the Valley coding.

    [As for the early days of digital history, don’t forget Unix servers with command line prompts, hand coding HTML of self-transcribed census pages, “taggin” newspaper articles, and riding X-Windows stations. It wasn’t easy like you youngsters have it today.]

  2. Jeff and Cameron — as I mentioned (in condensed format) on Twitter, the Valley site is one of UVA Library’s “Sustaining Digital Scholarship” projects. That means we’re committed to preserving its content and as much of its original structure and interface as possible. So it’s pretty much frozen in time, since all of the project staff have moved on (although the Library made a huge investment this past year in reworking the text encoding, migrating databases, doing image Q&A, etc. to make it more sustainable in its present form). I agree that Valley is full of treasures just begging for re-engineering and opening up to new inquiry! I’d love to see a dedicated team of scholars return to it one day.

    1. Bethany,
      I think the Valley project probably speaks just as much to the challenges of maintaining a site/database in the long-term. I kept being impressed at how it is still fully functional and vibrant fifteen years later – that in and of itself is pretty amazing given the insansely transitive nature of the internet (a point Larry makes quite well in a recent blog post).

  3. Great post.

    I was discussing Valley of the Shadow just this summer with a very bright colleague who is not a digital historian but who has been surveying the terrain. I started to sing the praises of the site when she cut me off, impatiently.

    “Why are there rooms?” She asked.

    “Huh?” I replied, perceptively.

    “Why are there rooms. You come to the front page and there are all these rooms and you have to pick one. It is all segmented and disjointed and doesn’t give you any idea of what the site is about. Why isn’t the front page a clickable map of the two communities? Or even two soldiers, one in blue and one in gray? Why are there rooms?”

    She had me there. Why are there rooms–23 rooms!–on the portal to the site? I guess the idea was to replicate the archival research experience where one has to travel to multiple locations to research a topic? That might be a great way to teach about doing research in archives, but I am not sure it is the best way to teach history. And to do anything like actual research you have to–as you said–drill up and down, up and down. You enter and leave each virtual room through the same door, thank you very much.

    I have tried to use the site in my classroom but it has never gone very well. I direct students to the “How to Use the Valley of the Shadow” guide but they grow impatient. I think the site might work very well if you were teaching a course on the Civil War and kept coming back to it over the course of a semester, but the learning curve is fairly steep for anything less.

    We should all praise Valley of the Shadow for its important contribution to the early field of digital history. It was a true pioneer, and all of us who followed afterward owe a debt to the project, if only for the credibility it gave to the digital humanities. But the thing about being a pioneer is that one day you are leading settlers over the mountain and George Caleb Bingham is painting your portrait, then later you are just another farmer in the Ohio Valley, respected but unexceptional, and finally you sit by your children’s fire and eat your porridge and recount the brave deeds of your youth to an audience polite enough to hide their impatience.

    1. Larry,
      I’m thinking of adding a line at the top of my post that just says “Skip down to Larry Cebula’s comment – it’s a lot more fun to read.” Also, maybe you’ve come across an anciliary use for the Valley project as a gateway drug for the digital generation to learn how to use analog archives.

  4. I was present at a graduate class in which public history grad students were doing presentations in which they evaluated digital history sites like these; almost all of them commented on the dated feel of the interfaces — and these were students who wouldn’t know an API from an apple. It will indeed be interesting to see if scholars wind up doing new “editions” of classic digital history projects — I believe there was a grad student who did some kind of reenvisioning of the Blake Archive, for instance. He presented at Digital Humanities 2009, but I only caught the end of it.

    1. Amanda, you’re thinking of Jon Saklofske, who is an assistant professor at Acadia University. That was the best paper I heard at DH ’09. I think Jon is contributing it to a special edition of Poetess Archive that I’m supposed to be writing for, too, if I can get my act together…

      Jon’s project is called “NewRadial,” and he released it on SourceForge, I believe largely in response to the conversation that happened at DH: http://sourceforge.net/projects/newradial/

  5. Great post and comments. I’ve been wondering why the Valley of the Shadow still looks about what it looked like when I was using it in 2003-4. I actually like the idea of keeping a preservation version of the site alive and I hope that if major changes are made to the interface in the future, some amount of the original version can be preserved. It’s both a collection of primary sources and a primary source itself.

    My experience as a TA in a couple of 19th century survey courses that used the Valley collections as a primary source for papers was that it worked out all right. Students were asked to base their papers on one of the substantial diary/memoir/correspondence collections – a large list was pre-selected by the instructor to choose from – so they had a lot to go on even if they didn’t really dig into the rest of the site. I don’t think that many did go much further than the main source they selected; that’s probably the consequence of the learning curve Larry talks about. Had I gone on to become a history professor, I probably would have tried out the site for courses myself. Some students really loved it.

    One aspect that I think is worth exploring – and perhaps you’re planning another post on this – is the relationship between the Valley site and the scholarship produced from it. The Ayers and Thomas article was experimental and still unusual in the realm of scholarly articles. (I think Darnton’s electronic article might still be the only similar thing out there, but I haven’t been keeping up.) I have to confess that I was very excited to see it come out, but never really explored it as much as I thought I would.

    On the other hand, Ayers’ In the Presence of Mine Enemies is a very traditionally written history book. As I remember it – it’s been a few years – it’s more on the tell-a-story narrative than a state-and-support-a-thesis book, so it’s still not quite like most monographs, but it’s not as experimental as Ayers “open narrative” Promise of the New South. I believe Ayers has said he intended it that way because part of his goal was to show that hosted-online sources could be just as legitimate – and be used to produce just as legitimate – scholarship as more traditional ink sources. The big difference being that you can check online for just about every footnote to a primary source for his book, but can’t do that with just about any other book.

    I suspect this is related to the way the site is set up. It strikes me as the kind of structure you’d give something when looking up footnotes and writing history is a natural part of your way of thinking. Someone could come along and take completely different approaches to the source material, of course, but the site has the kind of periodization and spatial divisions you’d expect to find in a finished work already built into it. That is, while you can potentially ask any question of the site’s sources, the site is also a reflection of the particular questions its designers asked, or were planning to ask, of it as they built it.

Leave a Reply to nowviskie Cancel reply

Your email address will not be published. Required fields are marked *