Dissertation Prospectus: Digital Linguistics

Tablets, Tables, and Trees: Data Structures and Documentation

Large, Windless Apartments

Swimming as we all are in a sea of information, it is difficult for modern linguists to extract themselves from the present and imagine a time when their work was not mediated in some way by computers. But a pair of quotes from early practitioners of language documentation bring the sea change from the age of paper to the digital era crashing home. Kathryn Klar (2002) tracked down a revealing insight into the working methods of J.P. Harrington, a linguist whose vast and ramshackle output needs no introduction. Klar recovered an outline by Harrington for a 1914 lecture on fieldwork methodology, perhaps the only explicit record of Harrington’s working methods. In the following quote, he describes how would-be field linguists should sort their fileslips (emphasis added):

The sorting of slips should be done in a large, windless appartments [sic] where the work will be undisturbed. Many Large tables are most convenient, although it often becomes necessary to resort also to placing slips on the floor. The sorting of many thousands of slips is a most tedious and laborious task. To many it is mere drudgery. It cannot well be done by anyone other than the collector. It is easy and interesting to record words on these slips by the thousand. The sorting however takes many times as long as the recording.

We imagine Harrington in a bleak rented room somewhere in California, surrounded by his innumerable slips of paper, deathly afraid of a breeze blowing away a year’s worth of careful organization of precious notes on Chumash or Ohlone.
Harrington was not alone in suffering the curse of “sorting”: it also plagued Leonard Bloomfield, whose complaints of the ailment were recounted by Hockett in Bloomfield (1967), (again, emphasis added):

Bloomfield was speaking of the tremendous difficulty of obtaining a really adequate account of any language, and suggested, half humorously, that linguists dedicated to this task should not get married, nor teach: instead they should take a vow of celibacy, spend as long a summer as feasible each year in the field, and spend the winter collating and filing the material. With this degree of intensiveness, Bloomfield suggested, a linguist could perhaps produce good accounts of three languages in his lifetime.

It turns out that Bloomfield did not live up to his own expectations: he did not live to see the publication of his one full-scale grammar, The Menomini Language, and it was Hockett who would edit and see through the posthumous publication of that work (Bloomfield 1969). While Bloomfield seems to have been speaking somewhat tongue-in-cheek, Harrington’s true evaluation of his own claim can of course be measured in (mostly unpublished) tons. But whatever the correct numerical interpretation of these admonitions, the import is clear enough: two well-known linguists agree in advising aspiring linguists to expect to spend a significant portion of their careers — perhaps one half of their careers — engaged in ‘sorting’ data.

It is humbling to imagine what such productive and prolific linguists could have achieved had they not been constrained in this way. One is reminded of the 18th-century lexicographer Samuel Johnson’s self definition as a ‘a harmless drudge’. Surely our easy modern access to relatively cheap and powerful computers obligate us to view hand-sorting in the era of computers to be drudgery of the “harmful” sort?

And yet, some modern linguists seem to exhibit a curious reluctance to let go of the old paper-based ways of documentation, or at least a lingering nostalgia for those methods:

The first dictionary I did (a preliminary version of my Mojave dictionary) was compiled in three-inch by five-inch slips (some linguists, I know, prefer four-inch by six-inch slips!) - not cards (too thick!), but slips of ordinary paper, which were arranged alphabetically in a file box (one hundred slips take up only a little more than half an inch). Reluctantly, I have stopped introducing field methods classes to the joys of using file slips, which I still feel are unparalleled for their ability to be freely manipulated and arranged in different ways. But I don’t use paper slips much myself any more, so it doesn’t seem right to require students to make a slip file, as I once did. (Munro 2001)

In a more explicit example of reluctance to embrace technology in fieldwork, hear R.M.W. Dixon:

In pre-computer days I’d fill in a 5 inch by 3 inch (12.5 cm by 7.5 cm) card for each lexeme. I continue with the same procedure today, as do many other linguists. Electricity supply is non-existent in many field locations, or else it is intermittent and unreliable. Reliance on a computer leads to frustration and a feeling of impotence when it ceases to work properly. This is why many linguists, working in difficult field situations, prefer to leave their computer back at the university. (Dixon 2009:297)

Both of these voices accept some role for computers, but resignation is not far from the surface: here we see experienced linguists looking dimly at the shift to the digital milieu: there is a loss of “joy,” replaced with “frustration,” and “a feeling of impotence.” (Note also the almost fetishistic specificity in the descriptions of the physical details of the paper medium.) Why is this? Would not Harrington and Bloomfield have given a king’s ransom for one of today’s mid-range, off-the-shelf laptops?

Those seeking support for carrying out fieldwork today are not permitted to vacillate on the question of whether to work digitally. Funding sources stipulate that corpora be deposited in digital form — fileslips and notebooks, nostalgia notwithstanding, are hardly acceptable academic output today. As the field of documentary linguistics has exploded — there are now many conferences focusing on or foregrounding language documentation — we have seen considerable progress toward the principles of “portable documentation” articulated in Bird and Simons’ seminal 2003 paper. In particular, significant progress has been made in archival practice. (For a useful history of these developments, see (Henke and Berez-Kroeker, 2016).)

But despite this intense activity in broadening the scope and improving the methodology of language documentation, the feelings of “impotence” and “frustration” seem to persist. Why do so many documentary linguists remain vexed by the difficulties of using computers to improve their research? Why is so much of our effort dedicated to troubleshooting software for doing linguistics, as opposed to simply doing linguistics? Is there something more behind Munro and Dixon’s complaints that the way we use technology in documentary linguistics is “joyless”?

There are standing answers to these questions: essentially, small groups of developers in research groups of diverse origin and motivations (from Max Planck to SIL) develop software which is intended to meet some of the needs of documentary linguists. These approaches have seen considerable success: ELAN, Toolbox, Praat, and other tools have made great contributions to the field: we are indebted to their creators.

For an overview of the capabilities of most server-based and desktop-based tools, see Dunham (2006). The software landscape described in there, which still obtains today, is a large suite of tools, each with a distinct set of file formats, covering distinct areas of the whole topical domain of documentary linguistics. Some cover the analysis of words (FieldWorks Language Explorer (FLEx), Toolbox), others the alignment of transcriptions and recordings (ELAN, Audacity), others building web interfaces to lexical information (WeSay), and so forth. (Dunham’s own Online Linguistic Database, described in Dunham (2006), is a very capable server-based tool with advanced morphological parsing and collaboration capabilities.)

But even in the face of this large ecosystem of tools, there is still much room to improve the ways in which we build up, interact with, and use data in language documentation. It is easy to enumerate desiderata for documentary software, but practical questions pile up even more quickly: What, exactly, will the software do? How will it be designed? Who will implement it? Who will maintain it? Who will train newcomers to use it and improve it?

This question is at the heart of this dissertation: is the solution to improving the difficulties that documentarists face in producing, using, and re-using their data to create yet another single, monolithic software tool, which will be maintained by a single, small group of developers? I answer this question in the negative. Rather than try to concentrate our efforts on training linguists and other language workers as users, we should shift our basic orientation to one in which we foster a community of development, within which the understanding necessary to define, build, use, and improve software is widely distributed: we need a sort of Cambrian explosion in the design of interfaces and applications which are small, well-defined, and composable.

Such an approach, as far as I can see, is only possible now, and only possible because the Open Web Platform can “multi-task”: it is simultaneously a platform which enables the development and use of applications for language documentation. It is one thing to train linguists to use a piece of software for documentation, it very much another to train them in the tools necessary to understand how that software works.

In order to launch this project, we must first develop a set of well-defined, flexible abstractions which correspond to the conceptual units underlying language documentation. A small set of such abstractions (the basic abstractions being Word, Sentence, Text, Corpus and Lexicon) is sufficient to underlie a vast array of applications. Rather than allowing the implementation of particular software tools (be they desktop- or server-based) to impose conceptual categories on the user, a system where all applications are built to manipulate the same basic abstractions (and which are also capable of defining complex abstractions such as ComparativeLexicon) will enable experimentation and iterative development of new, henceforth unseen tools for doing and using documentation.

The pressures from language endangerment on this endeavor are real, and ever-increasing. As linguists, we owe it to the communities we work with to make our data as reusable as possible for varied ends, and this means that we will have to build tools quickly and efficiently. It is a group effort. We, as linguists, must learn to develop software to meet our own needs, as a group. We will do this by incorporating software development methodologies into documentary linguistics directly. Spreading out the effort and understanding required to build linguistic software among many documentarians is the best way to ensure that not only our software, but also the data produced with it, will be resistent to the vicissitudes of funding, bureaucracy, technological churn, and other maddening realities.

This is a long-term goal. It will require that a significant number of linguists commit to the the task of learning to develop software directly. Obviously, the demands of academia will limit how much time and resources any given linguist can dedicate to this task, but even those who are unable to dedicate the time to learn to develop software in the vein described below independently can benefit from the availability of a system which is widely understood.

Two equally important components are required to make this possible:

The first is a core application suite. This will consist of a a specific, coherent, functioning system of interoperable applications (in the form of Single-page Open Web Platform applications) which handle the core data types in common use in digital documentation. This may seem like an impossibly tall order. Fortunately, we are far from starting from scratch. I describe below how we can build on the incredibly robust infrastructure of the Open Web Platform to build the tools we need.

An accompanying pedagogy. The second component — again, equally important to the first — is essentially a pedagogical one. Simply releasing another software package to the documentation community without making a simultaneous effort to educate users on how the software actually functions is not the most future-proof approach. This dissertation is intended to be, to a significant extent, a how-to, a do-it-yourself guide, a vade mecum.

Electrons, Not Atoms: Using digital documentary data

Thus far, I have largely assumed a set of “linguistic types” without committing to a list of such types or to a description of their internal structure. While extensibility in the data model underlying DLx is certainly a goal, a small inventory of simple, generic linguistic types, together with an enumeration of their required attributes and hierarchical relationships, is essential to enabling simple implementation of the core application suite. For example, an application for transcribing wordlists should not build on a definition of the Word which is at odds with the definition used to implement some other application, such as a interlinear text editor. I will refer to this small set of fundamental data types as the main data types for DLx. These data types will be built up in terms of Objects and Arrays, the two highly generic data structures which constitute JSON.

Metadata and Tags

Before addressing the main data types of DLx, let us consider the important topic of how metadata is to be represented in the system. A constant challenge for documentary linguists is the ubiquitous difficulty of keeping metadata associated with the right database materials or files. One approach that has been advocated for documentary linguists is to develop a strict file-naming scheme which encodes crucial metadata such as topic, speaker, date, and so forth directly into a filename. In the experience of this author, this turns out to be a horrible idea.

Instead, I advocate an approach to handling metadata wherein metadata is itself represented as a JSON object, alongside the data which it describes. In this case, an object called metadata is at the top level of a JSON structure representing the Text instance.

{
  "language_code": "hil",
  "identifier": "001",
  "speaker": "JDL",
  "recording_date": "2017-03-09",
  "creator": "JDL",
  "description": "catching fish in Santa Barbara",
  "format": ".wav"
}

Such objects are seen as values for the metadata key on the example objects in the next section.

Main data types

Language

Basic metadata about a language (names, language codes, phonological inventory, orthographies).

{
  "metadata": {
    "language": "Kashaya Pomo",
    "iso": "kju",
    "glottolog": "kash1280",
  },
  "orthographies": [
    {
      "ipa": "p",
      "oswalt": "p"
    },
    {
      "ipa": "b",
      "oswalt": "b"
    },
    {
      "ipa": "t̪",
      "oswalt": "t̪"
    }
  ],
  "inventory": [
    {
      "name": "voiceless bilabial plosive",
      "voicing": "voiceless",
      "place": "bilabial",
      "manner": "plosive",
      "letter": "p",
      "cv": "consonant"
    },
    {
      "name": "voiced bilabial plosive",
      "voicing": "voiced",
      "place": "bilabial",
      "manner": "plosive",
      "letter": "b",
      "cv": "consonant"
    },
    {
      "name": "voiceless dental plosive",
      "voicing": "voiceless",
      "place": "dental",
      "manner": "plosive",
      "letter": "t̪",
      "cv": "consonant"
    }
  ]
}

Note that this structure includes featural information about the phonetic inventory of the language (which could be collected via an application such as the one in §5.5, below). This information may be used in a range of analytical tasks.

Word

Minimally, a word requires a token and a gloss. Note that there is a key distinction between the occurrence of Word within a Sentence, and a Word within a Lexicon. The former is a token, and the latter is a type. While there is much ink to be spilled about the theoretical status of these concepts, in practice and in developing prototype applications, this minimalist definition of a ‘word’ has proven sufficient. Returning to a Sumerian example, this time with an English-speaking linguist replacing our Akkadian-speaking scribe, a simple instance of a Word might be:

  {
    "token": "udu.gír.ak.a",
    "gloss": "shorn sheep"
  }

Text

From the Word we move two levels “up”, to a Text. Below is an extract of two sentences from the remarkable Ubykh texts of the famed last speaker of Ubykh, Tevfik Saniç. The content is from an excellent web archive at LACITO-CNRS.

This structure, which is admittedly rather difficult to read as JSON, encodes two sentences from an Ubykh text, plus metadata about that text. This Text object has a sentences key whose value is an array, and within that array there are two objects, each representing a Sentence object. The Sentences in turn contain an array of Words, with a structure as given in the previous section.

{
  "metadata": {
    "title": "Eating fish makes you clever",
    "linguist": "Dumézil, Georges",
    "language": "Ubykh",
    "speaker": "Saniç, Tevfik",
    "archive": "LACITO-CNRS",
    "notes": ["This is only a sample of the Ubykh text",
      "Contains only the first 2 sentences"],
    "media": "http://cocoon.huma-num.fr/data/archi/6537_POISSON_22km.wav"
  },
  "sentences": [
    {
      "transcription": "fạ́xʹa tˀq˚ˀá-k˚ábǯʹa kʹˀáɣə.n 
        aza.xʹa.šʹə.na.n á-mɣʹa.n gʹə.kʹa.qˀá.n.",
      "translation": "Two men set out together on the road.",
      "words": [
        { "token": "fạ́xʹa", "gloss": "un jour" },
        { "token": "tˀq˚ˀá-k˚ábǯʹa", "gloss": "deux hommes" },
        { "token": "kʹˀáɣə.n", "gloss": "en compagnons" },
        { "token": "aza.xʹa.šʹə.na.n", "gloss": "étant devenus l'un pour l'autre" },
        { "token": "á-mɣʹa.n", "gloss": "en chemin" },
        { "token": "gʹə.kʹa.qˀá.n", "gloss": "entrèrent sur" }
      ]
    },
    {
      "transcription": "a.fawtə́.nə mɣʹáwəf a.x˚ada.wtən 
          a.kʹˀá.na.n á-za.n faċˀ.ạ́la s˚ə́ḇ.ạla x˚ada.qˀá.",
      "translation": "They bought some provisions for the 
           journey. One bought cheese and bread,",
      "words": [
        { "token": "a.fawtə́.nə", "gloss": "pour eux manger" },
        { "token": "mɣʹáwəf", "gloss": "provision de route" },
        { "token": "a.x˚ada.wtən", "gloss": "eux pour acheter" },
        { "token": "a.kʹˀá.na.n", "gloss": "étant allés" },
        { "token": "á-za.n", "gloss": "l'un" },
        { "token": "faċˀ.ạ́la", "gloss": "et fromage" },
        { "token": "s˚ə́ḇ.ạla", "gloss": "et pain" },
        { "token": "x˚ada.qˀá", "gloss": "il acheta" }
      ]
    }
  ]
}

In addition to Language, Word, Sentence, and Text, the two remaining main data types are Corpus and Lexicon. The Corpus contains a collection of Texts, and the Lexicon contains a collection of words. There is of course much more to be said about the structure and use of all of these data types.

Functions, Sorting, Searching, and Analysis

To this point the dissertation will have advocated the use of structured data and its relationship to traditional conceptualizations of linguistic units as employed in language documentation. But assuming that a corpus of non-trivial size can be assembled in this manner, what benefits will that structured corpus accrue to linguists (and linguistics), both in terms of expediting fieldwork methodology, and in terms of making data more amenable to analysis?

While the main introduction to programming basics in Javascript will be introduced in Chapter 4, at this point I will introduce the notion of a function, as it is indispensable for discussing the sorting, searching and analysis of data.

Functions

A function is a procedure which which takes input and either returns something else, or modifies something else. Here is a simple Javascript function that determines whether its input is one of a, e, i, o, or u:

let isVowel = letter => {
  return [ "a", "e", "i", "o", "u" ].includes(letter)
}

Let us ignore the details of how this is implemented for the moment, and consider only how it is used:

isVowel("a")

This “returns” the boolean value of true, while:

isVowel("x")

…returns false. This sort of named procedure is key to implementing a whole range of analytical operations over data.

Searching

The simplest “search engine” in Javascript is simply to “filter” an array. Considering five Akkadian glosses from chapter 2, we may use our previously defined isVowel function to create another function which determines whether a word starts with a vowel.

let startsWithVowel = word => {
  let firstLetter = word[0];
  if(isVowel(firstLetter)){
    return true
  } else {
    return false
  } 
}

Note that isVowel is called within the definition of this function.

And now we can pass this criterion directly to an array filter, as follows:

[
  "im-me-ru",
  "MIN ma-ru-ú",
  "MIN MIN dam-qa",
  "ar-ri",
  "kaṣ-ṣa"
].filter(startsWithVowel) 

The return value of running this filter will be another, containing only vowel-initial strings:

[
  "im-me-ru",
  "ar-ri"
] 

While there is a fair amount of detail to understand here, once understood, this pattern is tremendously powerful. Far more complex analytical options can be built up from simple component functions combined into complex filters.

Sorting

Sorting arrays of data is a surprisingly powerful task in data analysis. Unlike a shelf of clay tablets or a box full of file slips, abstract Arrays of Objects may be sorted by any criterion which can be defined in terms of the set of attributes on each arrayed object.

In this section I will explain the conceptual background to sorting. In most modern programming languages, the task of sorting is represented in a very simple way: the programmer defines a procedure which answers the question “Are these two items in order, out of order, or equal in order?”. If the programming language is provided with (1) an array to be sorted and (2) such a procedure (often called a comparator function), then all the details of how to carry out the actual reorganization of data in the computer’s memory is abstracted away. Like many modern programming languages, Javascript comes with a set of built-in algorithms for carrying out a sorting routine, one of which is chosen as appropriate to the characteristics of the data to be sorted. The details of what heuristics are used to select the particular sorting algorithm are effectively (and thankfully) opaque to the web application developer. The task of sorting is abstracted to only the task of writing a comparator function which returns -1 (out of order), 0 (equal order), or 1 (in order).

Analysis

The example of filtering a list of strings into a list containing only vowel-initial words is just the beginning of the sorts of analysis that can be carried out on structured ling

uistic data.

Morphosyntax

• Find all sentences which contain the morpheme labeled ERG (ergative).
• Which verbs have the largest number of morphemes?
• What percentage of pronouns refer to the first-person?

Sociophonetics

• Find the speakers in a Mixtec corpus who pronounce the word meaning “burro” with a trill as opposed to a flapped rhotic.

Variation

• Which texts in a corpus contain codeswitching at the intra-sentential level? Which contain it at the intra-word level?

Phonology

• Which lexemes begin with a velar?
• Which words exhibit vowel harmony?
• Are there any words with more than one ejective consonant?
• Find all words containg diphthongs.

Computers don’t have eyes: Implementing user interfaces

This section’s title is a reference to the fact that data which are presented in a layout which implies structural relationships between pieces of information — that is, it implies data structures — may nonetheless fail to encode those data structures explicitly in any persisted format (such as a saved file or a database). To a computer, all data is ultimately just a sequence of characters, and if that sequence is not encoded in such a way that hierarchical relationships can be recovered from it, then those relationships are essentially lost. It is in this sense that one might say that “computers don’t have eyes”: while a human reader may understand data structures that are implicit in visual structure, a computer cannot.

Thus, although it is clear enough that it is possible to conceptualize documentary data as a data structure (as was discussed above for the bilingual clay tablet and Manchu pentaglot dictionary), it is a very different task to implement an actual user interface which can be used to view or modify such data structures in an efficient and intuitive way.

This chapter begins with an introduction to the “Open Web Platform”. Briefly, it is a standard which defines four distinct syntaxes, each of which captures a particular aspect of web applications and data management:

JSON, HTML, CSS, Javascript

JSON (Java Script Object Notation)

At this point, the syntax of JSON will already be familiar from the introduction in Chapter 3.

HTML (Hypertext Markup Language)

This syntax, identifiable by the presence of tags contained within angled brackets, is the most fundamental syntax for the web. It is partially comparable to a normal word processing file, except that it is a markup language rather than a so-called “WYSIWYG” file format. In a word processing file, if a user wants to set a certain stretch of text in a bold typeface (like this), then the user selects the text and presses a button that applies bold formatting. The text then immediately “turns” bold. This is what is meant by the abbreviation “WYSIWYG”: “what you see is what you get.”

In HTML, by contrast, all information related to the structure of the document is itself explicitly encoded into the document as text. The following snippet of HTML also “bolds” a single word:

The sequence <strong>this</strong> contains both a single tag — in this case, the tag strong — and some tagged content, in this simple example, the word this.

In order to “see” the bold text, one must view the file through a program called a web browser. There are many such programs, but the primary browsers in use today on the desktop are Mozilla’s Firefox, Microsoft’s Edge, Google’s Chrome, and Apple’s Safari.

The single line above, then, would be rendered by a browser as follows:

At first blush, this more explicit encoding of document structure may seem to be a step backwards from the WYSIWYG approach, but as we shall see, the opaque way in which document structure is captured by word processing programs is nowhere near as powerful a basis for building interactive user interfaces as HTML and its associated syntaxes.

The HTML standard defines a vocabulary of basic tags for “marking up” many useful sorts of structures within a document: lists, paragraphs, document sections, buttons, etc.

CSS (Cascading Style Sheets)

Note that the “bolded” content in the previous example was not tagged with a tag called “bold,” but rather “strong.” This reflects a key principle of the Open Web Platform (a principle which itself has evolved over the course of development of web standards): the separation of content and presentation. The presentational characteristics of any tag in an HTML document can be customized through the use of style rules. For instance, a CSS author might decide for some reason to express the “semantic” idea of ‘strong’ text by overriding the default styling for the <strong> tag to red text rather than bold text.

A CSS rule to carry this out would look like this:

strong {
  font-weight: normal;
  color: red;
}

Here, the font-weight property is reset from the default value of bold to normal, and the color property is set to red (overriding the default black).

In this section I will explain enough of the workings of CSS for the reader to understand how to lay out basic application interfaces, and to style text according to standard conventions in documentary linguistics (including Leipzig glossing notation).

Javascript programming language

Another important distinction between the OWP and other data-management applications is that browsers have a complete programming language “baked-in.” This means that the tagged elements in an HTML document may be programmed to interact with user input: clicking, moving the mouse, typing, or some combination of these.

This fact has tremendous consequences for working with data of all sorts in documentary linguistics: it means that it is possible to design custom applications with user interfaces designed specifically to make particular annotative or analytical tasks more efficient. A primary design goal of the DLx codebase is simplicity: the entire library for the primary data structures is currently at less than one thousand lines of code (fewer than ten printed), and while some remain to be elaborated, it is my hope that it will remain within that scale.

The most advanced programming technique introduced in the dissertation is the the notion of a class. In programming, the term has a specific meaning: classes a sort of “blueprint” for a particular object which binds data and behavior (in the form of functions) together. For each of the basic linguistic data types (Word, Phrase, Sentence, Text, Corpus, Lexicon), a class will be defined which associates appropriate methods with the data which constitutes an instance of that class. Thus, for example, we might define a Phrase as containing at least 1) a string representing a transcription, 2) a string representing a free translation, and 3) an array of Word objects, each with their own internal structure. While this data is the “raw material” of an object which can be stored in JSON, in order to add “behavior” to a Phrase class, we include functions which can operate on that data. So a particular phrase (in programming, we refer to an “instance” of the Phrase class) will not only serve to bind a particular transcription, translation, and word array together (JSON objects already do this, in essence), it will also have methods such as .tokenize() or .transliterate() which give that instance of the class the ability to carry out particular linguistic operations.

Thus, the specific implementation of Digital Linguistics developed in this chapter is a library of classes which not only encapsulate linguistic data into a useful hierarchy, it “builds in” familiar conceptual to instances of those classes. To take a simple example, assuming the class definition for a Phrase object to be developed in the chapter is already complete, it might be instantiated and used in the following way:

let sampleSentence = new Sentence({
  "transcription": "kómo te βa",
  "translation": "how’s it going?",
  "words": []
}

Notice that our sample sentence was instantiated with an empty words array: that is to say, this sentence has been neither tokenized nor glossed. Because it is an instance of a Phrase class, however, the instance has methods:

sampleSentence.tokenize()

Now, if we were to inspect the value of the .words array (by referring to sampleSentence.words at the Javascript console or in some rendered context), we would see:

[
  { "token": "kómo", "gloss": "" },
  { "token": "te", "gloss": "" },
  { "token": "βa", "gloss": "" }
]

Here we see an array of words with tokens specified, but empty gloss attributes. Assuming again the existence of a Lexicon instance (let’s call it spanishLexicon) which contains Spanish words and their glosses, then the .gloss method of the Phrase instance can be called with said lexicon as a parameter, thusly:

sampleSentence.gloss(spanishLexicon)

Once this method has been executed on this phrase instance, we end up with a data structure like the following:

{
  "transcription": "kómo te βa",
  "translation": "how’s it going?",
  "words": [
    { "token": "kómo", "gloss": "how" },
    { "token": "te", "gloss": "2S.ACC" },
    { "token": "βa", "gloss": "go.3S.PRES" }
  ]
}

Now we have a reasonably complete representation of this phrase, with sufficient information for a basic interlinear gloss representation of this Spanish sentence. There is much more to be said about the design and especially interactions of these classes and their functionality, but it is hoped that this brief introduction is sufficient to give an idea of the sort of work involved in developing the full library of classes for collecting and manipulating documentary data structures.

Having explained the characteristics and functionality of the classes, I turn to the topic of how to build user interfaces (in this case, web applications) which give various interactive forms, or views, to these classes.
Views are responsible for “drawing” or rendering the content of a class instance to the browser screen in some meaningful way — with the vague word “meaningful” being used advisedly: one of the most powerful aspects of the open web platform is the manner in which it enables an almost endless range of presentations.

In general, then, the whole process of designing an application which enables an end-user to view, manipulate, analyze or create documentation is as represented in the following diagram:

Each of three stages of application design — definition of data, presentation of information in an interface, and the behaviors which make that interface interactive — is in turn associated especially with one or more of the four technologies of JSON, HTML, CSS, and Javascript. While in practice these are all mixed in varying degrees in every stage, this is a useful pedagogical approach.

Note also the key fact that both the beginning and end of this work flows takes the form of data which is stored in the JSON format. Summarizing and simplifying, one might think of a DLx application a tool which takes input in the JSON format, and either transforms it or rearranges it in some useful fashion, and then produces either an unchanged JSON structure (in the case of applications which do nothing more than format the data in some useful visual fashion) or a changed JSON structure. For instance, an application whose purpose is to add part-of-speech tags to a Lexicon instance might take in a Lexicon instance with words containing only token and gloss values, but output a Lexicon where each word has a wordClass attribute with particular value.

The actual behavior of such a part-of-speech-tagging application could be defined in many ways, while the input and output JSON might have exactly the same formats in each of those applications’ workflows.

Fundamental DLx applications

At this point we will have considered the longstanding utility of key data structures in linguistics (Chapter 2), a model of digitized versions of such data structures and how to use them in linguistic analysis (Chapter 3), and an introduction to how to build user interfaces for manipulating such data structures (Chapter 4). In this chapter (almost certainly the longest in the dissertation), I will describe in detail the design and implementation of a minimal but powerful set of basic applications for Digital Linguistics.

Texts

There are many ways to present and interact with the most fundamental unit of language documentation, the Text.

Time-aligned parallel text playback interface

This is the simplest text interface: no word-level morphological analysis is rendered. The sample below is currently in use in a pilot project for the diasporic Mixtec community in California. Note that for the purposes of this project, the community is not interested in morphological analysis. Hence, glosses are not rendered.

Data Structures: DLx Text

Presentation: Simple parallel text view with basic metadata, a “Play all” button, and a views of each phrase, with transcription, translation, and a phrase-level audio player.

Behavior: User can play any individual line to hear it, or play the whole text.

Example: http://westcoastmixtecs.com/health/knee_pain/

Sample data:

{
  "metadata": {
    "title": "Dialog on difficulties with the knee",
    "source": "MedicalSpanish.com",
    "speaker": [
      "Rogelia"
    ]
  },
  "phrases": [
    {
      "media": "audio/2016_04_30_Rogelia_knee_pain_01.mp3",
      "transcription": "Na̠xa ki̠í kú'vi̠ ko̠ndo̠-un?",
      "translation": "How long has your knee been hurting you?"
    },
    {
      "media": "audio/2016_04_30_Rogelia_knee_pain_02.mp3",
      "transcription": "Xa̠ u̠vi̠ semána kú'vi̠-ña.",
      "translation": "It's been a couple of weeks."
    },
    {
      "media": "audio/2016_04_30_Rogelia_knee_pain_03.mp3",
      "transcription": "Á ta̠kuě'e̠ ko̠ndo̠-un xina'á?",
      "translation": "Have you ever injured your knee?"
    }
  ]
}

Note that it is equally feasible to render a text which does have morphological glosses, as in this presentation of a text in Ubykh:

This interface serves to load a JSON file in the DLx Text format in a readable format.

Fieldwork interface

This interface is designed to be used in a context where fieldwork is already ongoing on a language, and both an orthography and the beginnings of a lexicon have been produced.

Data Structures: DLx Text, DLx Lexicon, DLx Language

Presentation: Standard Interlinear text format with media playback buttons

Behavior: User can play any individual line to hear it, or play the whole text.

Example: http://westcoastmixtecs.com/health/knee_pain/

Sample data:

In this somewhat articial example, a linguist is using standard Spanish orthography to transcribe and morphologically annotate a few sentences in Spanish, presumably in the presence of a speaker. Note that a (small) lexicon is rendered in the left sidebar

Sample Interface

Note that in combination the parallel text interface and the fieldwork editor interface serve to demonstrate that the very same data structure (a DLx text) can be rendered in static or editable forms.

Transliteration

Revitalization

Because the data for DLx applications is contained in distinct JSON data files, the same data may be used for purposes other than traditional linguistic analysis. Many such uses are applicable to the process of language revitalization: the screenshot below displays a set of printable, educational playing cards which were designed for the use of the Mixtec community in Oxnard, California, using data produced by the 2015-2016 field methods class at UCSB. Note that this presentation has exactly the same input as a simple Lexicon viewer — only the rendering code is distinct.

Managing phonological inventories

Dealing with large phonological inventories such as that of a language like Kashaya Pomo can be challenging. An interface I have been working on makes the enumeration of that phonological system a matter of point-and-click, and the application edits an instance of the Language data type.

Returning to the notion of the 3-stage workflow in the previous chapter, the goal of this behavior may be defined as taking as input an array of objects, each of which corresponds to a “phone” as represented by the IPA, and producing as an output an array which is a subset of that initial array, containing only those phones which are known to exist in the language under study.

Tagging

This in-development application could serve as the basis for an introduction to the utility of the notion of “tagging” across a hierarchical corpus.

The Cavalry Isn’t Coming

The project described by this dissertation is a broad one, with many moving parts. It may help to review what Digital Linguistics is and is not.

Digital linguistics:

…IS a software system with a specific implementation. DLx is an approach to digital language documentation, and it’s an opinionated approach, insofar as it is based on a particular software platform. However, that platform is the platform on which browsers are based, and therefore it is the most reliable, tested, extensible platform in the history of technology.

…IS both technology and pedagogy. Education on the technological underpinnings of DLx is part and parcel of its design: if we are to create a truly sustainable ecosystem for building and using interoperable language documentation, then we must learn together. Participants in DLx must be willing to try to push their own envelopes of technical understanding. DLx is free, but it’s not a freebie. Pedagogy will be provided, but participants must participate.

…Is NOT a single website. DLx is not a .com or a .org or a dot anything. There is a small library of HTML, CSS, and Javascript code which is available as a zipped folder, and which serves as a useful introduction to the principles of DLx. This code is useful without an internet connection — it works offline. We aim to output many formats which are shareable on the internet in a variety of ways, but this does not mean that there will be a single website where you are “expected” to upload anything. Individuals and individual communities must remain in control of where, which, and whether they share any of the work they produce with DLx. Issues of sharing language and “data” are ethical questions, not technological questions, and they cannot be solved by technology.

…IS a set of core applications, but is NOT a single application.

…Is NOT designed to enforce the adoption of some particular theory of language on its users DLx is a system for doing documentation. As such, it encodes the data structures which together comprise the primary sorts of data in language documentation. Arbitrary anntoations can be added to any level of the analytical hierarchy—up to and including parallel structures for more complex systems such as constituency or syntactic analysis—but such additional structures are expansions to the core of DLx.

_…Is NOT a single application. While DLx has as a goal the implementation of basic interfaces for the key documentary tasks of interlinear transcription, morphological analysis, lexicography, and time alignment, it does not follow that there is one “DLx App.” The community will be free to determine how multi-functional a given DLx application is, but DLx will never be equatable with a single application.

…Is NOT the property of anyone. DLx is a software system. It is not an archive. The licensing of documentation which is produced, modified, or analyzed with DLx tools is completely separate from DLx tools themselves. I will strive to include functionality which expedites the proper citation and citability of documentation produced in DLx.

…IS meant to be fun. This document has been written in a rather legalistic, small-print fashion, because these ideas are crucial — indeed, criterial — aspects of DLx. But I hope that bringing DLx to life will be a fun, participatory project which foregrounds the best aspects of documentary linguistics: respect and fascination for the full diversity of human language; satisfaction in collaboration between participants carrying out a variety of roles, and, above all, the valorization of the voices of real people who choose to share their languages with the world.

Timeline

I think this is doable in one year. I’m thinking of the project, ideally, as a three stage project: chapters 1-3 are the “setup” and hopefully faster part of the process; chapters 4-5 are the real work, and chapter 6 is a hopefully brief wrap-up.

1. I feel like the beginning of this prospectus is a fairly good start on the topics of chapters 1-3, so that should perhaps take another month per chapter to flesh out completely.
2. Chapters 4 and 5 constitute the most challenging part of the dissertation, and I suspect they will require the lion’s share of the time and effort. This is because they both involve the development of code, which, like writing in general, has a way of shooting out surprising complications.
3. Chapter 6 will be mostly prose related to my own positionality on the issue of how to advocate for adoption of DLx, and to describe a sort of first-pass syllabus for newcomers to the methodology. I’m sure it will be hastily cranked out in a caffeine-laden haze whilst sleeplessly staring out a window in Providence for the two weeks before my submission deadline.*

References