Text Analysis of Martha Ballard’s Diary (Part 3)

One of the most basic applications of text mining is simply counting words. I began by stripping out punctuation (in order to avoid differentiating mend and mend. as two separate words), put every word into lowercase, and then ignored a list of stop words (the, and, for, etc.). By writing a program to count occurrences of the 500 most common words, I could get a general (and more quantitative) sense for what general topics Martha Ballard wrote about in her diary. Unsurprisingly, her vocabulary usage followed a standard path of exponential decay: like most people, she utilized a relatively small number of words with extreme frequency. For example, the most common word (mr) occurred 10,050 times, while her 500th most common word (relief) occurred 67 times:

Top500Words

Because each word has information attached to it – specifically what date it was written – we can look at long-term patterns for a particular word’s usage. However, looking at only raw word frequencies can be problematic. For example, if Ballard wrote the word yarn twice as often in 1801 as 1791, it could mean that she was doing a lot more knitting in her old age. But it could also mean that she was writing a lot more words in her diary overall. In order to address this issue, for any word I was examining I made sure to normalize its frequency – first by dividing it by the total word count for that year, then by dividing it by the average usage of the word over the entire diary. This allowed me to visualize how a word’s relative frequency changed from year to year.

In order to visualize the information, I settled on trying out sparklines: “small, intense, simple datawords” advocated by infographics guru Edward Tufte and meant to give a quick, somewhat qualitative snapshot of information. To test my method, I used a theme that Laurel Ulrich describes in A Midwife’s Tale: land surveying. In particular, during the late 1790s Martha’s husband Ephraim became heavily involved in surveying property. In the raw word count list, both survey and surveying appear in the top 500 words, so I combined the two and looked at how Martha’s use of them in her diary changed over the years (1785-1812):

survey_surveying survey(ing)

Looking at the sparkline, we get a visual sense for when surveying played a larger role in Martha’s diary – around the middle third, or roughly 1795-1805, which corresponds relatively well to Ulrich’s description of Ephraim’s surveying adventures. As a basis for comparison, the word clear appeared with numbing regularity (almost always in reference to the weather):

clear clear

Using word frequencies and sparklines, I could investigate and visualize other themes in the diary as well.

Religion

Out of the 500 most frequent words in the diary, only three of them relate directly to religion: meeting (#28), worship (#143), and god (#220).

meeting meeting

worship worship

god god

Meeting, which was used largely in a religious context (going to a church meeting), but also in a socio-political context (attending town meetings), had a relatively consistent rate of use, although it trended slightly upwards over time. Worship (which Martha largely used in the sense of “went to publick worship”), meanwhile, was more erratic and trended slightly downwards. Finally, and perhaps most interestingly, was Martha’s use of the word god. Almost non-existent in the first third of her diary, it then occurred much more frequently, but also more erratically over the final two-thirds of the diary. Not only was it a relatively infrequent word overall (flax, horse, and apples occur more often), but its usage pattern suggests that Martha Ballard did not directly invoke a higher power on a personal level with any kind of regularity (at least in her diary). Instead, she was much more comfortable referring to the more socially and community-based activity of attending a religious service. While a qualitative close reading of the text would give a richer impression of Martha’s spirituality, a quantitative approach demonstrates how little “real estate” she dedicates to religious themes in her diary.

Death

death death

dead dead

funeral funeral

expired expired

interd interd

Most of the words related to death show an erratic pattern. There are peaks and valleys across the years without much correlation between the different words, and the only word that appears with any kind of consistency is interd (interred). In this case, word frequency and sparklines are relatively weak as an analytical tool. They don’t speak to any kind of coherent pattern, and at most they vaguely point towards additional questions for study – what causes the various extreme peaks in usage? Is there a common context with which Martha uses each of the words? Why was interd so much flatter than the others?

Family

In this final section, I’ll offer up a small taste of how analyzing word frequency can reveal interpersonal relationships. I used the particular example of Dolly (Martha’s youngest daughter):

dolly dolly

The sparkline does a phenomenal job of driving home a drastic change in how Martha refers to her daughter. In a matter of a year or two in the mid 1790s, she goes from writing about Dolly frequently to almost never mentioning her. Why? Some quick detective work (or reading page 145 in A Midwife’s Tale) shows that the plummet coincides almost perfectly with Dolly’s marriage to a man named Barnabas Lambart in 1795. But why on earth would Martha go from mentioning Dolly all the time in her diary to going entire years without writing her name? Did Martha disapprove of her daughter’s marriage? Was it a shotgun wedding?

The answer, while not so scandalous, is an interesting one nonetheless that text analysis and visualization helps to elucidate. In short, Martha still writes about her daughter after 1795, but instead of referring to her as Dolly, she begins to refer to her as Dagt Lambd (Daughter Lambert). This is a fascinating shift, and one whose full significance might get lost by a traditional reading. A human poring over these detailed entries might get a vague impression that Martha has started calling her daughter something different, but the sparkline above drives home just how abrupt and dramatic that transformation really was. Martha, by and large, stopped calling her youngest daughter by her first name and instead adopted the new husband’s proper name. Such a vivid symbolic shift opens up a window into an array of broader issues, including marriage patterns, familial relationships, and gender dynamics.

Conclusions

Counting word frequency is a somewhat blunt instrument that, if used carefully, can certainly yield meaningful results. In particular, utilizing sparklines to visualize individual word frequencies offers up two advantages for historical inquiry:

  1. Coherently display general trends
  2. Reveal outliers and anomalies

First, sparklines are a great way to get a quick impression of how a word’s use changes over time. For example, we can see above that the frequency of the word expired steadily increases throughout the diary. While this can often simply reiterate suspected trends, it can ground these hunches in refreshingly hard data. By the end of the diary, a reader might have a general sense for how certain themes appear, but a text analysis can visualize meaningful patterns and augment a close reading of the text.

Second, sparklines can vividly reveal outliers. In the course of reading hundreds of thousands of words over the course of nearly 10,000 entries, it’s quite easy to lose sight of the forest for the trees (to use a tired metaphor). Visualizing word frequencies allows historians to gain a broader perspective on a piece of the text, and they also act as signposts pointing the viewer towards a specific area for further investigation (such the red-flag-raising rupture in how frequently Dolly appears). Relatively basic word frequency by itself (such as what I’ve done here) does not necessarily explain anomalies, but it can do an impressive job of highlighting important ones.

Text Analysis of Martha Ballard’s Diary (Part 2)

Given Martha Ballard’s profession as a midwife, it is no surprise that she carefully recorded the 814 births she attended between 1785 and 1812. These events were given precedence over more mundane occurrences by noting them in a separate column from the main entry. Doing so allowed her to keep track not only of the births, but also record payments and restitution for her work. These hundreds of births constituted one of the bedrocks of Ballard’s experience as a skilled and prolific midwife, and this is reflected in her diary.

As births were such a consistent and methodically recorded theme in Ballard’s life, I decided to begin my programming with a basic examination of the deliveries she attended. This examination would take the form of counting the number of deliveries throughout the course of the diary and grouping them by various time-related characteristics, namely: year, month, and day of the week.

Process and Results

The first basic step for performing a more detailed text analysis of Martha Ballard’s diary was to begin cleaning up the data. One step was to take all the words and (temporarily) turn every uppercase letter into a lowercase letter. This kept Python from seeing “Birth” and “birth” as two separate words. For the purposes of this particular program, it was more important to distill words into a basic unit rather than maintain the complexity of capitalized characters.

Once the data was scrubbed, we could turn to writing a program that would count the number of deliveries recorded in the diary. The program we wrote does the following:

  1. Checks to see if Ballard wrote anything in the “birth” column (the first column of the entries that she also used to keep track of deliveries)
  2. If she did write anything in that column, check to see if it contains any of the words: “birth”, “brt”, or “born”.
  3. I then printed the remainder of the entries that contained text in the “birth” column but did not contain one of the above words. From this short list I manually added an additional seven entries into the program, in which she appeared to have attended a delivery but did not record it using the above words.

Using these parameters, the program could iterate through the text and recognize the occurrence of a delivery. Now we could begin to organize these births.

First, we returned the birth counts for each year of the diary, which were then inserted into a table and charted in Excel:

Year Deliveries

At the risk of turning my analysis into a John Henry-esque woman vs. machine, I compared my figures to the chart that Laurel Ulrich created in A Midwife’s Tale that tallied the births Ballard attended (on page 232 of the soft-cover edition). The two charts follow the same broad pattern:

YearDeliveriesCompare

Note: I reverse-built her chart by creating a table from the printed chart, then making my own bar graph. Somewhere in the translation I seem to have misplaced one of the deliveries (Ulrich lists 814 total, whereas I keep counting 813 on her graph). Sorry!

However, a closer look reveals small discrepancies in the numbers for each individual year. I calculated each year’s discrepancy as follows, using Ulrich’s numbers as the “true” figures (she is the acting President of the AHA, after all) from which my own figures deviated, and found that the average deviation for a given year was 4.86%. Apologies for the poor formatting, I had trouble inserting tables into WordPress:

Year Deliveries Count Difference Deviation (from Ulrich)
Manual (Ulrich) Computer Program
1785 28 24 4 14.29%
1786 33 35 2 6.06%
1787 33 33 0 0.00%
1788 27 28 1 3.70%
1789 40 43 3 7.50%
1790 34 35 1 2.94%
1791 39 39 0 0.00%
1792 41 43 2 4.88%
1793 53 50 3 5.66%
1794 48 48 0 0.00%
1795 50 55 5 10.00%
1796 59 56 3 5.08%
1797 54 55 1 1.85%
1798 38 38 0 0.00%
1799 50 51 1 2.00%
1800 27 23 4 14.81%
1801 18 14 4 22.22%
1802 11 12 1 9.09%
1803 19 18 1 5.26%
1804 11 11 0 0.00%
1805 8 8 0 0.00%
1806 10 11 1 10.00%
1807 13 13 0 0.00%
1808 3 3 0 0.00%
1809 21 22 1 4.76%
1810 17 18 1 5.88%
1811 14 14 0 0.00%
1812 14 14 0 0.00%

Keeping the knowledge in the back of my mind that my birth analysis differed slightly from Ulrich’s, I went on to compare my figures with other factors, including the frequency of deliveries by month over the course of the diary.

MonthDeliveries

If we extend the results of this chart and assume a standard nine-month pregnancy, we can also determine roughly which months that Ballard’s neighbors were most likely to be having sex. Unsurprisingly, the warmer period between May and August appears to be a particularly fertile time:

Conceptions

Finally, I looked at how often births occurred on different days of the week. There wasn’t a strong pattern, beyond the fact that Sunday and Thursday seemed to be abnormally common days for deliveries. I’m not sure why that was the case, but would love to hear speculation from any readers.

DeliveriesDayWeek

Analysis

The discrepancies between the program’s tally of deliveries and Ulrich’s delivery count speak to broader issues in “digital” text mining versus “manual” text mining:

Data Quality

Ulrich’s analysis is a result of countless hours spent eye-to-page with the original text. And as every history teacher drills into their students when conducting research, looking directly at the primary documents minimizes the degrees of interpretation that can alter the original documents.  In comparison, my analysis is the result of the original text going through several levels of transformation, like a game of telephone:

Original text -> Typed transcription -> HTML tables -> Python list -> Text file -> Excel table/chart

Each level increases the chance of a mistake.  For instance, a quick manual examination using the online version of the diary for 1785 finds an instance of a delivery (marked by ‘Birth’) showing up in the online HTML, but which does not appear in the “raw” HTML files our program is processing and analyzing.

On the other hand, a machine doesn’t get tired and miscount a word tally or accidently skip an entry.

Context

Ulrich brings to bear on the her textual analysis years of historical training and experience along with a deeply intimate understanding of Ballard’s diary. This allows her to take into account one of the most important aspects of reading a document: context. Meanwhile, our program’s ability to understand context is limited quite specifically to the criteria we use to build it. If Ballard attended a delivery but did not mark it in the standard “birth” column like the others, she might mention it more subtly in the main body of the entry. Whereas Ulrich could recognize this and count it as a delivery, our program cannot (at least with the current criteria).

Where the “traditional” skills of a historian come into play with data mining is in the arena of defining these criteria. Using her understanding of the text on a traditional level, Ulrich could create far, far superior criteria than I could for counting the number of deliveries Martha Ballard attends. The trick comes in translating a historian’s instinctual eye into a carefully spelled-out list of criteria for the program.

Revision

One area that is advantageous for digital text mining is that of revising the program. Hypothetically, if I realized at a later point that Ballard was also tallying births using another method (maybe a different abbreviated word), it’s fairly simple to add this to the program’s criteria, hit the “Run” button, and immediately see the updated figures for the number of deliveries. In contrast, it would be much, much more difficult to do so manually, especially if the realization came at, say, entry number 7,819. The prospect of re-skimming thousands of entries to update your totals would be fairly daunting.

Text Analysis of Martha Ballard’s Diary (Part 1)

“mr Ballard left home bound for Oxford. I had been Sick with the Collic. mrs Savage went home. mrs foster Came at Evening. it snowd a little.”

This is the first entry in the diary of Martha Ballard. Martha Ballard was a rural Maine midwife who kept an extensive diary between 1785 and 1812 and whose life was immortalized in 1990 by the historian Laurel Thatcher Ulrich‘s award-winning A Midwife’s Tale. Over the course of three decades, Ballard kept a meticulous, near-daily accounting of her life spanning over 10,000 entries.

When reading A Midwife’s Tale, I was struck by how readily the text would seem to lend itself to digital analysis. In an interview, Ulrich noted, “The very thing that had attracted me to the diary in the first place was also the thing that made it difficult to work with. I mean there’s just so much.” To ground herself, she began by simply counting things: “And I would go day by day for every other year of the diary, and I would tick off what was in each entry: baking or brewing, spinning or washing, or trading, sewing, mending, deliveries, general medical accounts, going to church, visitors, people coming for meals, etc.” Because of the sprawling scope, she took this quantitative approach only for the even-numbered years in the diary. The fact that she was working in the late eighties without a computer makes her work even more impressive.

After poking around online I came across DoHistory.org, a website developed and maintained by the Film Study Center at Harvard University and hosted by (who else, really) George Mason’s CHNM. The website presents the diary to the public in two formats: the viewer can either browse through photographed pages of the diary or read the transcript of the pages (transcribed through a monumental effort by Robert R. McCausland and Cynthia MacAlman McCausland):

ballardpage1 ballardpage1text

When I realized the entire diary was online, it got me thinking about possibilities for text mining. As an aspiring digital humanist with little “hard” skills beyond basic GIS, I had been meaning to learn how to program for quite some time. In Martha Ballard’s diary, I had an intriguing source of data with which to learn how to do so. Now I just had to learn how to program. With the patient help of several programming-savvy family members, I gradually learned the basics of Python and how to apply it to Martha Ballard’s diary. What follows are the first steps we took to process the diary’s raw data into an accessible digital format.

Process

At first, I briefly considered learning how to scrape the text of the diary off the website. After some investigation, I decided that was a little beyond my abilities, so I copped out to the much easier route of sending an email to Kelly Schrum at CHNM, who kindly forwarded my request to Ammon Shepherd, who emailed me a zip file containing 1,431 html documents, one for each page of the diary. The html files of the transcribed diary are a basic, 3-column table that look this. My first step was to find a way to strip out the html tags and organize the text into a systematic database of individual entries. Fortunately, Ballard’s meticulousness and consistency lent itself well to such an approach.

The diary’s format translates quite nicely into creating a list of lists – the “main” diary being a list of all the entries, and each entry being a list in and of itself. The first program we wrote was to open each html file and begin extracting the different sections of text (which were conveniently marked by html tags). Iterating through each entry allowed us to separate the different columns in her diary into different items in the list. Here is the breakdown of our “list of lists”:

  1. Diary
    1. Entry
      1. Date
        1. Month
        2. Day
        3. Year
      2. Day of the Week
      3. Main Text of Entry
      4. Day Summaries (Column 3 of actual diary entry)
      5. Birth(s) (Recorded in Column 1 of actual diary entry)

In creating the list, we had to separate out the raw data from the html tags that formatted it. Fortunately, the folks who built the html files originally used an extremely systematic formatting process that actually made the job of distilling one from the other quite straightforward. A Python module called Pickle allowed us to export the list of entries as a manageable single file that we could then easily import into future programs to manipulate.

For example, the third entry in the diary would translate a bit into something like this:

  1. Diary
    1. Entry (3)

      1. Date
        1. 1 (January)
        2. 3
        3. 1785
    2. 3 (Tuesday – Ballard numbered the weekdays, beginning with Sunday as 1)
    3. “Tuesday. mrs. Foster went home. I had threats of thee Collic; by takein peper found releif.”
    4. Empty
    5. Empty

The list allows us to access pieces of information by “calling” their position. It helped me to think of the entire diary list as a warehouse containing almost 10,000 boxes (entries) inside it, with each box containing five compartments, with the first of those compartments divided into three sub-compartments. If you were to open any of the boxes (entries) and look inside the first compartment, then inside sub-compartment number two, you would always find a number that represented the month of that particular entry. If you were to look inside the third compartment of the entry/box, you would always find the main text for that day’s entry.

The advantages of setting up the data in a list structure is the ability to access these specific pieces of information easily and to compare them across entries. In many ways, processing the text to make it readable and programmable is one of the biggest challenges to text mining. Deciding on the most logical way to organize and break down over 1,400 files will lay the groundwork for the fun part: writing programs to actually analyze the diary of Martha Ballard.

***Special-edition sneak preview of future posts in this series***

A simple counting program reveals that the main text of Martha Ballard’s diary alone contains 377,315 words, spanning I-couldn’t-make-this-number-up 9,999 entries. That is a lot of data to play with.

Separating from the Pack

Almost two years ago, I made the decision to go to graduate school. At the time I was basking in what could only be called a history nerd’s dream summer break, spending my workdays as an intern at the New-York Historical Society‘s public programs department doing background research on professors and authors we could invite to give talks. While learning about traditional history occupied my days, learning about digital history began to occupy my evenings (at least when I wasn’t occupied with being a 21 year-old enjoying NYC). I had used GIS extensively the previous summer through a research project and had caught passing glimpses of the broader digital history universe, but I hadn’t fully explored it as a possibility for future study or (gasp) a future career. By the end of the summer, I had realized with a crystal-eyed clarity that digital history is what I wanted to “do” – in the airplane conversation sense of, “So, what do you do?”

By the time I began the actual application process, things had become even clearer, but certainly not easier. I loved history – I loved reading it, researching it, writing it, speaking it, teaching it. The idea that I could potentially spend my life doing these things made me embarrassingly giddy. At the same time, I was endlessly fascinated by the potential that lay in digital scholarship as an exciting frontier with seemingly limitless possibilities. When I sat down to my computer to start looking at schools, I began to feel the intense tug-of-war between these two impulses that would become a constant throughout the next nine months.

On the one side of the spectrum lay the traditional academic program, ivy-wrapped in prestige and brimming with names that fly off the jackets of some of my favorite books. On the other side lay the digital history program, sleekly packaged in technology and humming with voices that build the blogs and websites I trawl. On the one side, my college professors and the academic job market counseling me to apply to the very best schools I could. On the other side, my own geeky impulses were urging me to take a chance and apply somewhere new and different and exciting. I did my best to split the difference, and in the end, I was lucky enough to be accepted to a school with a fantastic combination of these two sides.

My experience led me to the conclusion that just as digital methodology is shifting the scholarly landscape of historical study, it is also altering the competitive landscape of the field. Schools are rapidly carving out digital niches for themselves, and this will prove increasingly attractive to successive waves of graduate applicants and job candidates. Most of these individuals will be reliant on accessing databases and articles online, many will be familiar with new forms of media and technology, and some will be interested in areas of visual design, data mining, or spatial analysis. However, if any of them were to ask their advisor or mentor for suggestions on programs that are strong in digital history, they’d likely hear a one-name (if any) reply: George Mason. Most advisors wouldn’t be able to point towards Nebraska-Lincoln or University of Virginia as digital history strongholds, the same way they would be able to point towards Duke and North Carolina for their strength in African-American history.

This will change. At first glance, the general structure will remain the same. “Top-tier” schools aren’t likely to start hemorrhaging applicants to less-established programs immediately. Innovative schools like UNL will continue to fight the persistent prestige-and-name-recognition battle. Nonetheless, subtler transformations will occur. Even five years ago, it would have been inconceivable that a school like George Mason (whose doctoral program has only been in existence for eight years) would be able to compete on an even footing for history applicants with a school like Stanford. Now, their success in establishing themselves as the dominant industry leader gives them an unparalleled advantage for anyone interested in digital history.

Even for schools on traditionally similar footing, an established track record of integrating new media can easily tip the scales in their favor. Much like special collections and on-campus archives, showing off a sparkling digital infrastructure will emerge as a “sexy” way to pull in both applicants and candidates. Wealthier schools may begin to invest in humanities computing centers and new kinds of software, even if for the simple goal of keeping pace with their competitors. The day is not far off when the mainstream academic history-verse buzzes with the news that an ivy-league school has “poached” a leading history professor in media and technology. Grant proposals for more applicable digital initiatives will cut out bigger slices from an expanding NEH pie.

The schools that will truly separate themselves from the pack, however, will be the ones that demonstrate their support for digital scholarship on an ideological level.  Those programs that establish a sustained committment to encourage, guide, and reward the members of their department (both faculty and students) for digital methodological inquiry will be the ones that will emerge in the best position to attract and train historians eager to tackle the technological opportunities inherent in today’s world.

The Mobile Historian

The rocketing ascent of mobile technology was one of the fundamental shifts of 2008, and many market analysts predict it will only continue throughout 2009. Its rise seems to be following a two-tracked progression: individuals in developing countries are latching onto increasingly affordable mobile phones as a way to log in to a wider network, while wealthier consumers fascinated by the ability to take their online experience on-the-go are snatching up smartphones at a shocking rate (to the point where the smartphone industry appears to be recession resistant). This environment creates an intriguing medium for historians to refine and improve their craft, and the time is ripe for innovation.

Some historians have been leading the charge in utilizing this technology. Bill Turkel has been a pioneer in applying new methods in place-based computing to the field of history. Meanwhile, the majority of similar efforts fall under the sphere of public history. Some museums have long been experimenting with “electronic curators,” or hand-held audio devices that emit information about an aspect of the exhibit depending on where its carrier is standing. Cultural heritage sites, particularly battlefields and/or national parks, have quickly recognized the potential for GPS-enabled devices that guide visitors through a site. Finally, some history educators are experimenting with ways to engage their students using portable technology, including fieldwork and visitations.

Dave Lester, of George Mason University’s CHNM, presented “Mobile Historical Landscapes: Exposing and Crowdsourcing Historical Landmarks” in early April at the American Association for History and Computing conference. Dave’s is currently working on a project called HistoryPlot to encourage user participation in exploring and contributing to a knowledge bank of historical places. The idea is that roving bands of history enthusiasts could visit sites, pull out their iPhone, learn about some of its history, and possibly add both information and multimedia to the site by snapping pictures and/or uploading content – creating a kind of Yelp for the historically-minded. Dave’s project draws upon two specific advantages: 1) the participatory culture of crowdsourcing, and 2) the increasing ubiquitousness of mobile technology

Dan Cohen recently explored the advantage of crowdsourcing when he posted a historical puzzle on his blog at the start of a presentation, which asked people to identify the following picture using minimal clues:

He simultaneously sent out the puzzle via Twitter by asking his 1,600 followers to try to solve it in the next hour. The speed with which Dan got answers was impressive, with an initial correct answer coming in 9 minutes. Although he admits he should have made the puzzle a bit more difficult, the process was successful in highlighting the immense advantages of crowdsourcing historical problems using a fluid and mobile platform such as Twitter.

The growth of a mobile culture in which users are constantly connected magnifies the power of crowdsourcing. Dan’s experiment rested on the assumption that a certain number of his followers would be online and checking their tweets, and enough of them would then be able to use the internet to access his blog, read the clue, and search for the answer online. Two or three years ago, the chances of receiving an answer in 9 minutes would be much, much slimmer. A mobile culture removes barriers to accessing information, and simultaneously increases users’ expectations for accessing that information, many of whom no longer tolerate being shackled by outlets, ethernet cords, or wireless signals.

Consequently, mobile technology is redefining our social conception of space and place, and this has corresponding ramifications for historians. It revisits the fundamental relationship between a physical location and what happened in the past within that space, a relationship with which spatial and geographic historians continuously grapple. This shift is opening up a two-way street for historical researchers. On the one hand, a mobile culture allows efforts such as Dave Lester’s to shed light on previously inaccessible areas. Suddenly, a historian researching a far-away site might be able to “travel” there by looking at uploaded pictures and documents, trading emails or tweets with other researchers who have visited the place, or watching the video of a history enthusiast on vacation at the site.

On the other hand, those shifting expectations that accompany a mobile culture can also turn themselves on historical researchers. A mobile society might question the reliability of a solitary historian writing abstractly about a place they have never actually been to. A constantly connected audience will start to expect the kind of intimate access and exploration that can only be gained from hands-on visitation. A readership conditioned to read reviews on Amazon or tourists’ travel blogs will increasingly dismiss the authority of a specialist who has never visited a location they describe, even if they are describing its past. Audiences will continue to tolerate a historian’s inability to time-travel; they will not continue to tolerate an inability to place-travel.

Fortunately, mobile technology can also create a mobile historian. Imagine a historian writing about shifting gender roles on the Oklahoma Chickasaw reservation during the Dust Bowl. Armed with a laptop, digital camera, and smartphone, the historian can travel to Oklahoma and go to the reservation itself. Once there, traditional archival research is greatly enhanced by technology. Instead of lugging around 3×5 index cards, Zotero can speed up and digitize the note-taking process. The digital camera can capture documents for later perusal, allowing them to find more sources in a shorter amount of time. Is the researcher suddenly curious about gender demographics for a particular town near the reservation, or wants to understand the background to a religious ceremony referenced in a court record? They can use their smartphone to look up census data or send out queries to colleagues likely receive a rapid answer to their question.

Leaving the archives, the historian can dip into oral history by interviewing locals and recording their memories on the smartphone or digital recorder. The smartphone’s GPS capabilities allow him or her to not only locate the homes of the interviewees, but to flag and mark locations to look for spatial patterns at a later date – what if all the traditional “male” venues on a reservation were located on a specific street, while “female” venues were spread over a greater area? The GPS ability of a smartphone can capture these on-the-ground patterns. Finally, the mobile historian can quickly send out updates on their progress, receiving feedback and suggestions from a remote crowd of like-minded researchers, students, assistants, or colleagues.

Mobile technology (like all technology) is not a magic pill that will suddenly transform the historical profession. There are certainly drawbacks. First and foremost exists a strong economic barrier to entry. Already struggling for travel stipends and fellowship money, many historians won’t be able to afford a brand-new iPhone or high-quality digital camera. Those who aren’t already comfortable with mobile technology will often feel overwhelmed or at an unfair disadvantage. On a more abstract level, technology and its inherent distractions can sometimes construct blinders to one of the most important advantages to visiting a place in person: the ability to feel the sense of place, to listen to the wind and hear the accents and taste the food, a decidedly fuzzy process that adds crucial depth and richness to the historian’s understanding of their subject.

As technology itself becomes more refined and more sophisticated, the possibilities for innovation and exploration will continue to expand. As with any new methodology, the traditional skills and strengths of a historian will not fade into obsolescence. Instead, they’ll be ever more critical to the process of responsibly incorporating new techniques and approaches into the broader historical fold. If this process is even moderately successful, the future of the mobile historian appears bright.

AAHC Recap (Afternoon)

The first session of the afternoon at the AAHC conference was by Patrick Murray-John as he discussed “Interlinking Who, What, Where, When, Why, and How We Teach into a Giant EduGraph.” He gave an entertaining presentation that described the benefits and challenges of a semantic web within education. One story stood out in particular as an example for the need for better efforts in the field. Two courses might use Frankenstein as an assigned text: a class in the chemistry department that examined the portrayal of scientists in literature and an introductory English literature course. Unfortunately, the only people who know that both classes are using the same text is the school book store. If the course syllabi were located within a strong semantic web, there could be tremendous potential for interdisciplinary discovery and interaction within a university.

The second session of the afternoon was “Mapping Our Archives,” by Tim Sherratt of the National Archives of Australia. Sherratt gave an incredible presentation, as he demonstrated his work with the records of Australian servicemembers during the First World War in Mapping Our Anzacas. First up, he used Google Earth to show how he mapped where each of the 300,000 plus soldiers were from. It was remarkably powerful, as he used the platform dynamically in order to visualize the origins and density of this segment of the population. In particular, I was struck at how effectively it took advantage of the interface of Google Earth, and its re-population of point data when you zoom in, so even as you get more and more detailed, it seems like the dots just keep multiplying. The map was one point of entry for users of the database, with another critical point of entry being the opportunity for users to upload multimedia to a “scrapbook” of individual servicemembers profiles. Although some curators expressed (standard) anxiety about losing credibility and content control, they were overwhelmed at the number and quality of contributions from “laypeople,” such as this one of Donald Addison:

Addison

Tim also shared his crafting and tweaking of the database for the servicemembers’ records. Each record was scanned, and many of them included photographic sets of the individuals along with vitals and descriptions of their service. He then used CoolIris (which I discussed in one of my first blog posts) to create a virtual gallery of these archival documents, as you could move along a 3-D wall with rather haunting profile photographs and corresponding information (almost like an early 20th-century version of Facebook). He also showed a similar application of CoolIris for other archival records, including those of immigrants and laborers. I’ve always been somewhat skeptical of CoolIris as a visualization technique, always thinking it did little more than create a kind of cool way to zoom around and look at pictures. But Tim substantially changed my mind by demonstrating its power as an interface – without a doubt, being able to see an array of documents displayed in front of you, seeing the faces of Chinese immigrant children looking out at you, was a singularly powerful use of archival material that significantly enhanced the viewer’s interaction with them.

Tim’s presentation got rave reviews from its audience members, and I think I speak for all of them when I say that his use of dynamic technical interfaces (which included not only Google Earth and CoolIris, but also Tumblr and Greasemonkey scripts) was one of the most elegant, powerful, and accessible examples of bringing archival material to life that I’ve seen yet. Tim offered a fantastic glimpse into what is possible when you combine the power of digital applications with the subtlety and thoughtfulness of historical exploration.

The final session of the afternoon was a presentation from Susan Garfinkel, Judy Graves, and Jurretta Heckscher from the Library of Congress. Although I was a little fried from seven hours of absorbing and processing a host of new thoughts and ideas, I always love supporting our friends at the LOC, who have put a lot of time, thought, and effort into advancing and experimenting with digital history. Their presentations brought up several interesting points. First, they stressed that a series of surveys have shown that faculty members are less than enthusiastic about actually utilizing digitization tools and resources that their libraries provide. Additionally, the LOC has decided to expand their earlier initiative to post photographic content on Flickr Commons by posting audio and video content to iTunes and YouTube. The presentation reinforced my optimism that our national library has the courage to pursue such a variety of digital initiatives.

I drove out of George Mason’s campus feeling exhausted and inspired. The conference did a superb job of bringing together and enhancing a community of digital humanists (made even stronger by the constant Twittering activity during the day). This community isn’t necessarily representative of either the general public or even the wider historical community – in particular, I was struck at the rather overwhelming whiteness of the conference participants. I think more efforts could be made to encourage a wider diversity of perspectives and backgrounds within the field of digital history (a subject for another blog post). Nevertheless, the range, depth, and passion with which so many people are pursuing their projects gives me a strong sense of hope. I know that over the next several years I will undoubtedly struggle to balance the demands of training in a traditional graduate history program with my own passion for digital methodology and exporation. But events such as the AAHC conference bolster my faith in the incredible support system of fellow digital humanists that I can lean on as I walk down that path.

AAHC Recap (Morning)

Today I attended the American Association for Historical Computing‘s 2009 annual conference, hosted by the Center for History and New Media at George Mason University. For someone interested in the field of digital history, it was a phenomenal opportunity to meet fellow enthusiasts and explore a variety of topics within the field.

The first session, a presentation by Amanda French of NYU on “Basic Digital History Skills for Historians,” came from her experience in teaching courses in digital history, many geared towards archival and library studies. Of particular interest was a survey she administered to 25 students that measured their comfort and ability in a wide variety of digital skills – everything from using social media to knowledge of metadata systems. She spoke about the fact that there was a gap between the skills being taught to public historians and archivists, and those being taught to traditional historians. Namely, those in the former group usually gain a stronger digital literacy. One of the major action points she drew from the survey was the need to teach students in the following fields: website creation, metadata, and multimedia.

Besides being the first conference presentation that I’ve live-tweeted, it brought up some interesting questions. The biggest one (that recurred throughout the day) was the question of teaching students what I’ll term hard vs. soft skills in gaining digital literacy. Should teachers expect their college students to have a basic skill set (uploading videos onto YouTube, using RSS feed readers, etc.) already? Should you spend the majority of your time teaching the skills and habits that they can then adapt to specific platforms? Is it possible to impart broader concepts of digital history without a concrete base in technical proficiency? My first instinct was to come down on the side of a liberal-artsy instruction of soft skills and underlying “big-picture” principles. But the more I thought about the issue, the more I realized that for many people, the best way of learning these soft skills is by putting on your work gloves and diving into starting a blog, using Zotero, or generating a KML file.

The second session was Dave Lester‘s “Mobile Historical Landscapes: Exposing and Crowdsourcing Historical Landmarks.” Dave explained his ongoing project (History Plot) to create a means for people to contribute to a geolocated database. He compared it to Yelp, in that he dreams of a centralized platform through which people can look up historical landmarks and their metadata (primarily their location). In order to start seeding History Plot, Dave turned to 80,000 historic sites listed in the National Registry of Historic Places. Other resources could include Wikipedia, Flickr, and partnerships with local historical societies.

Dave’s enthusiasm was downright infectious, as he spoke about being able to walk down a city street, use your iPhone to locate a nearby historical building, look up information about it, then take a photograph of it and immediately upload it to the database. Possibly the most exciting aspect, for me, was his idea of leveraging community-based history volunteers (he calls them “street teams”) to crowdsource the project. I think this has tremendous potential. History remains one of the foremost fields for armchair enthusiasts, as legions of geneologists and Civil War re-enactors would provide an incredible resource for this kind of geo-based crowd sourcing. It’s easy to imagine groups of history buffs meeting up on the weekends to explore cities and sites, snapping pictures and contributing research tidbits. I’d love for this to get off the ground, and would jump at the chance to found a local chapter.

Dan Cohen brought up a good point at the end of Dave’s talk: that the issue is finding an incentive structure so that people will actually participate in the project. In particular, there’s a gap between the (usually) younger tech-savvy crowd that lacks a strong interest in local history, and the (usually) older, less tech-savvy crowd that could potentially be the strongest source for knowledge seeding. I think it’s a manageable problem, but one that increases the need for highly accessible mobile technology and platforms that makes the barriers to entry as low as possible, even if it has to come at the cost of losing some technical robustness.

The last session of the morning was “Teaching, History, and Digital Tools: A Roundtable Discussion,” by Jeremy Boggs, Jeff McClurken, and Josh Sternfeld. All of them brought different perspectives to the topic, although each of them came from the similar experience of having taught a digital history course. It was a similar presentation to the one given by Jeremy and Jeff at the AHA convention, and it reinforced a lot of the lessons they had previously given (chief among these is Jeff’s great refrain about trying to make students “uncomfortable but not paralyzed”). One point that it made me think about was the issue of how to value historical scholarship. I’ve been thinking a lot more about not only how the historical academy values research in digital history, but how it values teaching in digital history as well. Does listing “Creating History in New Media” on your C.V. as a course you taught carry more weight than listing an American history survey? Would a tenure review board be impressed with your tech-savvy literacy, or put off because they don’t understand it?

Geeking Out with History

I’ve been meaning to blog about this for awhile, but last month the Digital Youth Project released the results of their three-year study: “Hanging Out, Messing Around, and Geeking Out.” The project,  through funding from the John D. and Catherine T. MacArthur Foundation and collaborative scholarship between researchers in the UC system, looked to examine young people’s use and interaction with new media in three realms: communication, learning and play. The overall results are both fascinating and encouraging, and I’d recommend at least reading the two-page summary of their findings.

The title stems from the three modes of use the researchers identify. Hanging out is the primarily social interaction between friends and peers, exemplified by social networking sites, instant messaging, or text messaging.  The second mode, messing around, is a form of digital exploration and expression, exemplified by uploading videos or photos, trying out different online applications, or passing along discoveries (think Elf Yourself or LOLCats, for two admittedly trivial examples). The final mode, geeking out, is diving into a specific topic, finding a community of like-minded enthusiasts, and working towards a degree of expertise in the area.

For anyone interested in youth participation in new media, reading the white paper is a must. K-12 educators with even a passing interest in what their students are doing should take a glance at it. Upon first skimming it, I thought it did a great job of refuting several commonly-held perceptions about young people’s activity online. First, under the hanging out topic, of particular note is the refutation of a still-pervasive myth that kids  go online and end up primarily interacting with strangers. Instead, the researchers write, “With these ‘friendship-driven’ practices, youth are almost always associating with people they already know in their offline lives.” For the majority of young people, the idea of going into online chatrooms and striking up friendships with complete strangers is largely a relic of the past. With messing around, the researchers stress the fact that young people are not passive recipients of media, but they are increasingly participatory members of a community. There is a critical element of trial-and-error, as kids explore and incorporate (or reject) new activities. Finally, my favorite mode: geeking out. The authors highlight the important point that “one can geek out on topics that are not culturally marked as ‘geeky’.”

For some reason, this relatively innocuous assertion provoked a lot of thought on my end. “Geeking out” still carries strong cultural connotations, bringing to mind images of traditional nerd culture – see Timothy Burke’s recent post on Batman comics, in which he offers a disclaimer: “This entry is going to be the maximally geeky one.” But  “geeking out” as a verb can increasingly apply to non-“geeky” subjects: sports-obsessed fantasy football participants, any and every kind of music enthusiast, political gossip and speculation, etc. This has been one of the true hallmarks of the internet, by breaking down barriers of entry into extremely specialized fields of interest.

Which leads me to the title of this post: historians need to take advantage of the digital landscape to geek out with history. Without any amount of exaggeration, I can confidently say that my own geeking out with history has contributed just as much to my identity as a “historian” as my semesters of traditional scholarly training. Subscribing to blogs or listening to podcasts will not replace formal instruction. But it can certainly enhance the learning process, and in my mind, offers a higher ceiling for immediate participation and access. If a student writes an essay for a college course, most of the time the only reader will be the professor, and possibly some fellow students if it is a seminar. Meanwhile, if a student takes that same energy and enthusiasm to their subject online, they read related thoughts from scholars around the world, exchange comments and dialogue with some of those scholars, or post that same essay as a blog and receive feedback from a much greater number of readers.

On the research side, I think many academics are awakening to the vast potential for vertical exploration of historical source material. Fifteen years ago, doing research on a historical subject meant countless trips to archives and libraries, excursions that were largely hindered by geographical and financial considerations. Today, digitization projects have greatly streamlined the process of finding and accessing this material. In doing so, it is opening up the door for anyone to geek out with history. Genealogists and armchair historians have always greatly contributed to the field, whether or not academics like to admit it. But in the years to come, the ability of non-professionals to do professional work will grow and grow. This is a double-edged sword, as greater participation enhances the possibilities for collective intelligence and collaboration, while also running the risk of suffering from a “Barnes and Noble Syndrome,” of an environment dominated by cream-puff analysis and a lack of vigorous interpretative context.

Academic historians need to get their hands dirty online. Read (and write) blogs, mine some data, listen to podcasts, enter a virtual world, upload media, explore databases, leave comments, and share your research. Take some chances and make mistakes. In short, geek out.

Methodologies and the (Digital) History Major

Stanley N. Katz and James Grossman recently led a working group backed by the National History Center and the Teagle Foundation, and drafted a thought-provoking report titled, The History Major and Undergraduate Liberal Education. The paper got some decent play on the history and education blogosphere, and with good reason. It brought up a variety of interesting issues, but chief among, from my perspective, is one of methodology.

In the report, Katz and Grossman point out that the history academy tends to be moving away from traditional methodological categories – “political history, economic history, social history, intellectual history” – and towards categories of people and places. I would tend to agree, although the line between these two methodological approaches tends to be rather blurry and fluid (and I’m guessing the authors would not imply a distinctive break between them). It makes me wonder – are historians truly engaging in a large-scale shift in methodologies? Or is the academy coming up with new phrases to describe pre-existing approaches? A work such as Erskine Clark’s Dwelling Place: A Plantation Epic, could be read as a traditional work of “social history,” or it could be read as (obviously) African-American history, family history, rural history, or some combination of all three. Do “traditional” methodologies simply imply broader, umbrella categories?

Instead, I would argue (with freely admitted bias) that an equally important shift will take, and is currently taking, place within the academy: the transformation of analog to digital scholarship on a methodological level. Tom Scheinfeldt wrote a particularly incisive blog post on this topic provocatively titled, “Sunset for Ideology, Sunrise for Methodology?”: “I believe we are at a similar moment of change right now, that we are entering a new phase of scholarship that will be dominated not by ideas, but once again by organizing activities, both in terms of organizing knowledge and organizing ourselves and our work.”

Unfortunately, many in the academic blogosphere took the post as an attack on the validity of cherished theoretical “-isms” in the field. Too much focus rested on this aspect of the post, which Tom admitted in its comments was not the aim or intention. Instead, what gets lost is the bold assertion that the next big change in historical scholarship will come from the nuts-and-bolts of how we “do” history.

Katz and Grossman touch upon this change: “Liberal learning in the twenty-first century must include an emphasis on information sifting, the ability to work through massive quantities of data and references to identify what is useful and reliable.” While they offer a few other references to this new paradigm, they don’t spell out exactly how the skills of a history major relate to a liberal education in a specifically digital context (this is not the point of their paper).

I’d like to look at Katz and Grossman’s conclusions through a digital lens, and spell out specifically how I believe some of their observations and suggestions can be specifically linked to Tom’s “sunrise of methodology”:

– “History is thus inherently (though not necessarily for any individual historian) a multidisciplinary field and one in which inquiry begins with the problem and the historical context, not the discipline or dominant theory.”

Digital historians are necessarily engaging in interdisciplinary (or multidisciplinary) studies, as they not only need to know technical skills (programming, statistics, GIS, etc.) but also the broader issues prevalent in these fields. When creating maps for my history thesis in GIS, I not only had to learn how to import shapefiles, but also the background of coordinate projections, issues with small-scale vs. large-scale mapping, and basic tenets of cartographic design and layout. When utilizing a wide range of tools and techniques, a digital humanist is forced to learn not only “hard” skills, but their accompanying “soft” skills as well.

– “History places a premium on the capacity of synthesis.”

I couldn’t agree more. I feel that this will truly be one of the distinct advantages a history major might have over other scholars: the ability to efficiently and effectively sift through mountains of source material in order to extract content, recognize broader patterns, and evaluate their metadata (both traditional and digital). These skills form the basis of historical inquiry, and as our collections of digitized sources grow ever larger the proper utilization of these skills will be placed at a higher and higher premium, especially when paired with new media tools and techniques.

– “The single most important contribution that training in history can make to the liberal learning of undergraduates is to help students to contextualize knowledge, offering an antidote to naive presentism.”

One hallmark of the digital age is the ephemeral nature of information. Lacking the inherent stability and traditional gatekeeping of the analog era, it becomes more and more difficult to “pin down” knowledge. Without assurance that a website will exist tomorrow or next week or next year, knowledge and authority become much more fluid, and users will be even more inclinated towards presentism (whether naive or not). Historians will need to offer their skills in contextualizing and framing a constantly shifting corpus of information, at the very least in order to provide a sense of temporal perspective.

-“We need to be more thoughtful in locating history in relation to other disciplines, and in relating to the ‘historical turn’ in other humanities and social science disciplines.”

History has a lot to learn from other disciplines, and vice-versa. Just as digital humanists use a multidisciplinary toolbox, their utilization of these tools also tends to blur the traditional lines between disciplines. When a historian engages in complex statistical analysis using computer software to examine tax records, where does the line fall between economics and history? There needs to be a dialogue about how to most effectively employ and engage history within these other disciplines. In industry terms, the academy needs to figure out a “value-add” system of mutual benefit. And one key to this process (which Katz and Grossman describe) is that of cross-departmental collaboration, both in research and in teaching.

All in all, this is an excellent report that brings into focus far more important issues than I touched upon here. I would highly recommend it to anyone with an active interest in the current state and possible future of the field.

Towards a “History This” Command Line

Mozilla Labs recently released the 0.1 version of Ubiquity, a Firefox extension that allows the user to interact with and direct their browser through intuitive, written commands. Ubiquity has met with largely positive and excited reviews from the tech community, from folks at Lifehacker to Hackaday to Tools for Thought. The extension currently allows for a variety of commands. The common example that everyone likes to point to is the “map these” command, where you select text, hit the keystroke to bring up ubiquity and type “map these,” which brings up something like the following:

From there, you can do a variety of things with the map itself, including navigating and moving, or inserting it into a separate page. And of course you can also highlight text, and in Ubiquity type “email this to _____,” which then searches through your Gmail contacts and sends the highlighted text to them. The most common example I’ve read is if you are looking for a restaurant at which to eat with a friend. You can highlight or type in the restaurant name, map it, look for reviews on Yelp, check your calendar for conflicts, and email an invitation to your friend with all of this information included.

Ubiquity interacts with a wide variety of sites through APIs, including Youtube, Weather, Yelp, Twitter, and Flickr. In addition, you can translate and define words, run calculations, export events to your calendar, count words in an article, or convert units. In many ways, it seems to blur the earlier function of Hyperwords (which I covered in a previous post) with the intuitive command line structure of Quicksilver (for Mac users) or Launchy (for Windows).

I immediately thought of interesting commands someone could write for engaging in historical research. Developing a Ubiquity command set for historians would go a long way towards encouraging traditionalists to finally break into digital history. Instead of reading scary words like Python or machine learning, a researcher with little technological background could hit a couple of keystrokes and be off in running with relatively in-depth analysis of digitized archival material. In many ways, Ubiquity could potentially act as a “gateway drug” for digital history. Of course, this all hinges on at least two things:

1) Quality, standardized digitization of source materials combined with quality, standardized open API’s. Dan Cohen has great arguments for the importance of a digitized collection like Google Books not only having an API, but having a good one.

2) Someone in the digital humanities would have to develop these tailored commands for different archives (Bill Turkel, you know you’re interested…) There’s already a Mozilla Labs wiki for creating new commands that looks relatively straightforward, but would probably be above most members of the history community. I’m intrigued by the idea, but unfortunately my own forays into digital history programming have presently taken a backseat to applying to grad schools. Please let me know if anyone in the digital humanities is interested in this…

I feel that Ubiquity takes a substantial next step in the evolution of online interactivity. It’s admittedly buggy (although given its 0.1 version status, this will certainly get better), but it embodies so much of what is positive in today’s digital environment: namely open-source collaboration. Mozilla Labs actively encourages anyone and everyone to develop their own commands and to share them with others. This openness combines with an intuitive simplicity that makes it truly remarkable. As of right now, Ubiquity is a fantastic timesaver and cool trick, but it lacks depth. Almost anything you do in Ubiquity could be done before – just slower and with much less efficiency or ease of use. I have absolutely no doubt that as the open-source developer community jumps on board, this will change.

But for right now what Ubiquity does best is to begin to break down the barriers between computer geeks and laypeople. Some people are writing about the irony of returning to the infant state of the computer interface: the command line.  While interesting, these two instances are fundamentally different: not many people would know how to write even a simple program when faced with earlier command lines, but just about anyone I know can type “Map this” into Ubiquity and get far more complex results. Even as programmers find new ways to write more and more advanced commands, ordinary Firefox users will adopt the basics of Ubiquity in greater and greater numbers. What I foresee in Ubiquity is part of a broader movement that shifts common computing further down the Web 2.0-blazed path of heightened and evolving user participation, control, and access. Instead of having the website developer determine how and where you can go, suddenly you are at the controls of an increasingly powerful and easy-to-use command center for accessing and manipulating data. And I can only dream of the day a grad student will be able highlight some archival text, type “history this” into their command line, and have a fully-compiled dissertation written before their eyes.