Names Clean-up in the People Database

The most recent task assigned to me to complete as we prepare to launch the Adams Papers Digital catalog was to review instances were names we assigned as attributes (e.g. adams-john1735 for John Adams (a.k.a. JA) appeared in the XML code, but not in the People database. This is part one of two, the second part being instances were people appear in the database but do not appear in the attributes in the XML.

This is another one of those combination's of human and computer errors, some of which were unavoidable (computer) and some of which happened because of how mundane some of the work was (human).

The attributes that needed seeing to was 1200 strong, and in printed form stretched to 110 pages. (Before you think we're total idiots, this represents approximately 5-6% of the database which means we got 94-95% right. On a ten point grading scale that is a solid A! Sheepishly we realize mistakes happen but anticipating negative criticism about the work we did we wanted to try to spin it positively.)

We did two queries to produce the data. The first query produced the faulty attributes and the slip ID number in which the attribute was contained. For the results that came about in the second run, we included within double-quotes the name associated with the faulty attribute. It gave a little insight into what we could expect before searching the database and editing the code.

Some typical instances looked like this:

a. [extract]-monroe] 160017 - "to Sec. of State [James Monroe] [extract]" | 160043 "to Sec. of State [James Monroe] [extract]"
b. bourne-sylvanus? 072781 - ""to Mr. Sylvanus? Bourne"
c. henry-laurens 031911 - "to Henry Laurens"
d. nicolay-albery-h 341282 - "to Albert H Nicolay"
e. palfrey-john 340735 - "John G. Palfrey" | 341083 - "John G. Palfrey"


There were a number of consistencies as to the errors that were made that were determined quickly when cleaning up the data. In some instances the attribute was valid, but had not been entered into the database. While frustrating, this was an easy fix and was most likely the product of human error. A second reason was due to a typographical error in the XML attribute that was not present in the attribute for that person in the people database. And the reverse, where there was a typo in the people database but the attribute was actually correct. Typographical errors includes transpositions of letters (examples c and d). A third possibility was that in the transformation process which took place in Level 1, the attribute was inaccurately reviewed (examples a and b above). And in the above it should be evident that punctuation and other marks like [ ) ? . were not allowed in the attribute; . Another kind of thing we saw was were attributes in the XML were not as complete as in the database, and the opposite (example e).

The fixes for the above examples should be relatively easy to make yourselves:

a. monroe-james (remove [], add james)
b. bourne-sylvanus (remove ?)
c. laurens-henry (flip first and last names)
d. nicolay-albert-h (typo in albert corrected)
e. palfrey-john-gorham (added middle name)

In addition to checking the slip, we also had to check the database for each name (unless it was a prominent figure). The back-end view for the Adams Papers editors (and us), allows for tabbed browsing of the digital control file. I have not really seen the public interface but imagine this might be a similar feature.

We did not keep count of the number of names added but there were quite a few. In the process we were able to clean up a number of bad attributes that were in the database and in some cases merge or separate people based on a close inspection of names, dates, etc. For example in correcting some of the slips for William Cunningham Jr. I discovered that his attribute should be "cunningham-william0" where as the majority of them were just "cunningham-william", which is the attribute for his father, William Sr. These have all been fixed now so that the users of the digital control file will get the respect they expect when searching for these people.

We will re-run the query in a few days to ensure that every instance was seen to accurately. Let's pretend this was the case, for if any were missed, I won't tell you about it!

The other side to this clean-up, whereby we'll run a query to determine in the database where attributes exist that do not appear in the XML, is more straightforward...those attributes will be deleted from the database.

Also going on has been Beta testing of the public interface. Susan and I were invited to sit in on a meeting yesterday about how that testing went, what some of the feedback was, etc. So I'll post a bit about the public interface next time. The project comes to a close on 30 June 2011 so blogging might slow down - if it doesn't stop outright - after that date. I don't want to turn this into a Brokeback Mountain "I can't quit you" kind of moment, but the reality is is that once the funds stops, so does the blog!

Almost there...

The end of the Adams slip file conversion project is fast approaching, and the digital catalog will be launched shortly. For some of us here at the Massachusetts Historical Society, it's been all Adams all the time.

The MHS, of course, already offers a variety of online resources related to the Adams family, from digitized letters with transcriptions to digital editions of print volumes. The Adams Electronic Archive contains images and transcriptions of over 1,000 letters between John and Abigail Adams, as well as John's diaries and autobiography. The voluminous diaries of John Quincy Adams have also been digitized, and his line-a-day diaries are the subject of our very popular JQA Twitter project. The Adams Papers Digital Editions reproduce 32 of the print volumes published by the Adams Papers Editorial Project, complete with footnotes and intertextual links.

Navigating through all of these pages can be confusing, and we're hoping the digital catalog will help to mitigate that problem and function as a kind of "clearing house" for Adams researchers. Because the catalog contains item-level descriptive data for every known Adams family manuscript, it's the perfect vehicle for linking to individual digitized documents and online transcriptions, wherever they currently "live." Public researchers, as well as Adams editors, will be able to retrieve a specific record, click on a link, and--voila!--read the document itself.

So, to that end, I spent the last several days adding links to hundreds of individual records using web forms in the dynamic interface. Still to be tackled are links to all the diaries of John and John Quincy Adams. Of course, only a fraction of the items described in the catalog are available in digital format, so it will be important to make that clear. Web developer Bill Beck will design the public interface so that users can limit their search to only those items available online.

Saltonstall as "Adams Lite": Tastes Great, Less Filling

Mr. [John Quincy] Adams is now the rising sun, and of course finds many idolaters. You can hardly conceive the strange appearance he makes--so cold--so unbending rigid muscles amidst such smiles, such good humor, gaiety--and among his own guests too. It seems a miracle that he has ever been chosen President of the U. States. Is it an invincible proof of his eminent merit, or the result of a singular concurrence of fortunate circumstances? Mrs. A. is the very antipode--if you will allow the term to be applied to a lady....And I fear that she has a Courtier's hart [sic]--or like him is heartless.

(Letter from Leverett Saltonstall to his wife Mary, 24 Feb. 1825)

This little teaser is just my way of introducing the Saltonstall family papers slip file project, another grant-funded paper-to-digital conversion modeled on the Adams family catalog. We here at the Massachusetts Historical Society have come to call the Saltonstall slip file "Adams Lite" because it is both much smaller (only 3,000 slips) and much more straightforward than its unwieldy counterpart. Both catalogs consist of a series of paper slips describing individual documents: author, recipient, date, place, length, etc. But unlike Adams, this slip file describes papers in a single collection at the MHS: the Saltonstall family papers, Ms. N-2232. The entire data set fits in a single xml file, and all of the information has been entered, controlled, and verified by one person. And lastly, because publication of the Saltonstall papers was completed years ago, this database requires only one static interface--basically a searchable item-level collection guide. The Adams Papers Editorial Project, on the other hand, is ongoing, so that database requires two interfaces: one static, for public use; the other dynamic, to be edited by Adams Papers staff.

The Saltonstall database fulfills a requirement of the original Adams slip file grant, awarded in the fall of 2008, which specified that that project would serve as a prototype for similar projects. The Saltonstall conversion has been in the works since the beginning of 2010, and many MHS staff members have contributed to the project. Laura Lowell processed the Saltonstall family collection, and our digital team of Nancy Heywood, Laura Wulf, and Peter Steinberg have digitized, transcribed, and marked up many individual items for presentation on the web. I was responsible for building the database, using the Adams slip file as a model.

Mary Claffey's work on the Adams slip file laid most of the groundwork for me. Rather than reinvent the wheel, I literally copied and pasted her schema and revised her tags to suit the needs of the Saltonstall family papers, scaling it down by deleting unnecessary elements and adding or repurposing others. Our web developer, Bill Beck, is designing an attractive and user-friendly interface, also modeled on the Adams slip file, and Laura Lowell's collection guide to the Saltonstall family papers will link to the database. It was a lot of fun to work with so many other members of the department; everyone brought their own strengths to the project.

The Saltonstall family, like the Adams family, is chock full of prominent and interesting people, spanning several generations. Leverett, Sr. (quoted above) was a member of the Massachusetts House of Representatives, the Massachusetts Senate, and the U.S. House of Representatives, and he served as the first mayor of Salem, Mass. His wife, Mary Elizabeth (Sanders) Saltonstall, was the daughter of Thomas Sanders, a well-known Salem merchant. Leverett's great-grandson, also named Leverett, was the governor of Massachusetts during World War II and a U.S. senator for over 20 years.

Keep an eye on the MHS website for further information about both projects.

The Goldenrod Affair

In the Adams Paper slip file, the chronological run is considered the master and it is that which was microfilmed and used to create the digital control file. In adding document types, which was based on an equally impressive alphabetical run (which includes, for the prominent Adamses, a select list of subjects) it was determined that some slips were present in the alpha run and not in the chronological run. These were goldenrod slips, which are used in house as cross-reference slips to connect people, places, subjects, correspondence, etc. Probably a lot more. The current task to which I have been assigned is going through the slips in the Alphabetical run and identifying those slips of goldenrod color that were not also present in the chronological run. The process includes making copies of those slips not found and then adding them to the Digital Control File.

Before this, when creating a new slip one would find a slip of similar color, copy it, and then edit to suite. Now, though, we have a new option in the interface that enables us to create a new slip from scratch. A most helpful part of the tool.

Another project that is ongoing at the time - thought not being completed by the contributing members of this blog - is that updates and changes to existing slips is taking place. There is a big three-ringed binder of these changes. A process that took heaps of time in its analog format is now being completed quite a bit more swiftly. This is not always easy and has involved some collaboration between the Adams staff and the control file gurus, some instruction, and the detection & correction of bugs.

Fun with searching

When editing individual records in the Adams slip file, it helps to use a unique keyword or phrase to limit your search results and pinpoint the exact slip you're looking for. Case in point: I was adding document types to records and came across a slip containing the phrase "bill for manure." I typed that phrase into my search box, confident I would retrieve only the specific record I was looking for. Imagine my surprise when I retrieved four! All four bills date from the year 1880 and belong to Charles Francis Adams (1807-1886). Expanding my search to just the word "manure," I found another bill, which I hadn't initially retrieved because of the variant phrasing on the slip. A sixth result was a letter from Charles' grandfather John Adams to Cotton Tufts, dated 80 years earlier, about a funeral oration for George Washington. John then goes on: "The transition to be sure from such a subject to my uncultivated business is very abrupt. But I must say a few words....I hope Porter has carted or shedded all the manure on the Hill....I wish you would contract with Carter to bring me up an hundred Loads."

Image of Charles Francis Adams from the Portraits of American Abolitionists photograph collection at the MHS.