Recently, I started to explore using Git to maintain and organize my data. This includes both the primary data I work from as a linguist and the various kinds of data I produce in the form of written material (books, papers, etc..). As a field linguist, I primarily work with text that is transcribed from audio and video recordings. Early on I developed an archiving system to preserve the original files in their raw form as much as possible, but I didn't develop a similar system for maintaining my working files, and ended up with lots of duplicates of files that I had copied various places for various purposes.

In this fifth post on Linguistic tools (the overview of which is here), I plan to describe the way I am trying to overcome this issue and become more organized by adopting a version control system. Mostly, though, I'll just try to explain what Git is and what I use it for. This is by no means a complete tutorial, since other people have done that better, but hopefully it will provide some direction for others who are interested in streamlining and organizing their workflow.

When I started learning to program in Python and was introduced to Git, I saw the benefits for coding, but began to wonder if it was possible/useful to use it for other organizational tasks in relation to other kinds of writing and data structures.  There are a few blog posts about using git for paper writing here and here and here, but they are written primarily for coders who also write papers (I'm looking at you, academic comp-sci folks!) and don't translate super well for my use case.

What is Git?
  First, what is Git? Git is a version control system, which means it tracks changes in a repository (or 'repo'), basically serving as a series of backups for your work. It also has tools that allow you to make a copy (or 'branch'), work on it separately, and then when you're ready, you can see the changes made to individual files and 'merge' them together into a new version. For a simple tutorial, see here, and here is a great visual walkthrough.

Most explanations of Git use the example of a tree or a river, which do help to understand the process, but for total newbies (and possibly to address the difficulty of finding a good metaphor) I came up with the following metaphor, and find it to be a bit more useful.

The Git Metaphor
  Imagine you have a drawing project. You have an empty piece of paper - this is your initial blank Git repository. Now you draw a picture on the piece of paper and decide it's an ok version - this is a snapshot that is your first 'commit'. You then decide you want to make some adjustments to the drawing, but you want to preserve the initial drawing, so you take out another piece of paper, lay it on top of the first, trace out the shape in pencil, and then begin to modify the edges of the drawing or the background by erasing or adding new bits. When you're happy with that you 'commit' it, and then every time you modify it, you repeat the process of first tracing it out on a piece of paper. Each of these pieces of paper stack, so if you want to go back to an earlier drawing, you can, without losing any of the other drawings.

  Now let's say your friend wants to work on the same drawing project. You can give him your latest drawing, he can copy it and work on it in the same way. Then when he's done he can give it back to you and you can choose which bits of his version you want to incorporate into your current version. This is super useful for projects with lots of collaborators, especially if you're writing code, which are basically text files and relatively easy to merge based on differences in lines.

[Note that this isn't exactly what the Git system does - it doesn't actually create a NEW version of the repository, like the tracing paper, but instead uses the most recently committed version as the new base/foundation for future modifications.]

The most important thing I've had to keep in mind is to write myself detailed commit messages, and to figure out a good protocol or format for these messages. This is because once you've committed something, you can go back and search through your commits (save states), but this is only easy IF YOU KNOW WHAT THE COMMIT CONTAINS!!! Using the full filename of the committed file, for example, is much better than writing out 'New version of Paper'. Giving more information, such as about what you changed in the file, is even better.

My Use Case
My use case, however (as opposed to collaborating on code with a large team), is for a single repository of data in which I will occasionally share a single paper with collaborators. What I want is a bit more like a backup system (think Apple's Time Machine or the versioning system in Mac OS) but with a lot more control. Git, for example, works on the command line and requires you to provide messages for each commit you make, which also requires you, the user, to be clear about what changes are and why they were made. This (ideally) means less clutter and assists with organization.

In thinking about my needs and doing research, I discovered a drawback of version control systems like Git. If I have a single repository but am working on multiple papers at once (like I often do), guess what happens when I save changes on document A and then checkout a previous version of paper B? Yup, everything reverts to the state it was before, and I 'lose' all the changes I just made to A. I put 'lose' in quotes here because I don't actually lose anything, as long as I had committed it properly, but I'd then have to checkout the commit, copy the file, and then revert... anyway it's annoying.

What I need
So what I REALLY need is a single Git repository with the ability to track individual folders/files separately with their own version histories (like embedding repositories). Git can do this in several ways, none of which are particularly intuitive: submodules, subtrees, subrepo, and simple embedding. The first two are designed for somewhat different use cases than mine, namely using (part of) someone else's code in your own project. They're not really meant to create sub-repositories in your local repo to track separate histories. The third, subrepo, looks quite promising but requires that you run everything from the root of your main repo, and since I have lots of organizational subfolders, typing the names every time just gets tedious. So, I use the strategy of simple embedding.

Simple Embedding
By simple embedding, I refer to navigating to a subfolder within my main repo that I want to track and running the command 'git init'. This does two things:

1. it allows me to track changes within this folder separately from the main git repo. 
2. it makes the changes in this folder somewhat opaque to the main repo.

This means that changes in the subfolder will propagate only at the intervals where I commit ALL to the main repo. Changes committed in the subfolder, however, will retain the history of the subfolder, before the most recent commit of the main repo. Metaphorically, making a separate repo of the subfolder allows me to return to the basement of a particular room in the structure and rummage around down there while retaining the fully built house around me.

This is probably a hack, and not the way Git was intended to be used, but it works for me. The only problem I foresee happening is making too many Git repositories, and not being able to keep track of the multiplicity. So I plan to use this sparingly. For the most part, I'm ok with committing changes across the whole repository, since one of the real benefits of using a VCS is the freedom to throw things away.

Some other great posts on version control systems include this one on the benefits of version control, this one on several different version control systems for writers, this post on the limitations of Github for writers - the last of a 6-part series worth reading if you are debating whether to start on the Git/version control journey. And finally, this one is about one person's suggestions for academic writing and Git.

In any case, Git or another VCS is definitely worth checking out. I also recently discovered that Git repos are supported 'out of the box' by LyX, my go-to LaTeX editor! More on that to come... I also recently discovered SRC (Simple Revision Control), which focuses on tracking revisions on single files. This might actually be what I need. I'm still exploring this, and it might not be necessary since my LaTeX editor supports Git. We'll see.

This is the fourth installment in a series of blog posts where I discuss some of the tools I use for linguistic analysis. In a previous post I described the tools I use for processing and converting video and audio. This is partly important for using the audio with Praat for acoustic analysis, but also because ultimately I want to transcribe the recordings and analyze the grammatical structure (using the transcriptions) while remaining linked to the recordings. Ideally, such a link would allow me to play back the recording from within the program that I use to analyze the grammatical structure of the language, so I'm not constantly having to open the sound files and scrub to the correct part of the WAV.

Fortunately, this IS possibile, and it currently involves using two larger programs, Transcriber and Toolbox, along with a small tool/script that I wrote for conversion (you can read more details about the script HERE). The original workflow, which my supervisor Alec Coupe taught me, depended on another script developed by Andrew Margetts, which was very useful but required internet access - possibly problematic if one was on fieldwork. Then a few years ago the site went down and conversion became a multi-step process involving emails. So I wrote my own script. But I'm getting ahead of myself.

The workflow:
The first step in this process is getting your audio file transcribed. It is possible to do this in Transcriber (thus the name), and the newly repackaged version for Mac works pretty well (as does 1.5.1 on Windows or in a Windows VirtualBox). You can also use another program to play back the audio (i.e. Audacity, which can slow down the audio for you and let you boost the signal, etc) while transcribing in a text editor. Then you copy/paste the text to Transcriber for time-alignment.

I use Transcriber for time-alignment because it's simple, it allows for extremely short time-stamped windows, and it produces text files (basically) as output. There are other tools such as SayMore (Windows only) that also allow you to transcribe and directly output files for use with Toolbox, but in using it I found that SayMore was a bit of a memory hog and wouldn't let me create time-stamped sections shorter than half a second, which could be problematic for short interjections. And it kept freezing, so that it took a really long time to transcribe a short example.

So with Transcriber I simply open the audio file that I want to time-align, copy the block of text that I've transcribed, and then listen to the audio file, follow along in the text, and insert timestamps ('enter' or 'return' key) at the correct points. Once this is done for the whole file (time-consuming, but not as time-consuming as the transcription) I save the file (a '.trs' format).

Once you have the 'TRS' file, you get to use this converter, which you download to your computer. The converter runs as either a Python script or a Windows executable file that you run in the folder that has your 'TRS' file. When you run it for the first time it asks you for the field names in your Toolbox settings, and creates a configuration file in the folder to store the settings. If it doesn't find the settings file the next time, it will ask again. When it has the settings, it outputs a 'TXT' file for each 'TRS' file in the folder.

Once you've got the 'TXT' file, you can open it up to view it. If you already work with Toolbox, you can see that it is already (mostly) in the correct format. If you don't work with Toolbox, try installing it and go through the tutorials with a new project. Then open the Toolbox-created text file and compare it to the one created by my script from the Transcriber file. You should be able to copy/paste the content of the newly-created text file into your Toolbox project text file with no issues. Now when you open your Toolbox project file in Toolbox, you should be able to see the timestamps and play the sound file (and portions of it) from within Toolbox, provided you put the audio file in the correct location.

Why Toolbox?
This may seem a bit cumbersome, begging the question "Why use Toolbox at all?"

• "Because Toolbox files are text files"
  This is the short answer. Text files are easy to store (archivable) and extract information from. They are easy to work with in Python when I want to search through the corpus or convert them into LaTeX for display as interlinearized glossed text (IGT) for other linguists.
• Toolbox is fast
  This program is quite small and not resource-heavy, making it extremely useful for small laptops. I have also used the more resource-heavy Fieldworks Language Explorer (FLEx), but there are some drawbacks.
• It doesn't make assumptions
  Toolbox treats clitics, affixes, and all other sorts of word types and word formations according to how the user defines them, rather than according to a pre-selected set of (theoretical) assumptions. Other programs such as FLEx seem to make unwarranted assumptions about a language and therefore actually get in the way of exploring how the language works. FLEx is great for wordlists and shared projects, but I find that Toolbox preserves the underlying format better and allows me more manual control. If I decide to change the way I represent things in Toolbox, I do have to manually go back and interlinearize things again, but that is arguably beneficial, as it forces me to get to know the language data better.

A final note (ELAN)
As I end this post, I should note that many linguists use ELAN for transcription and annotation of video/audio. In fact, that the suggestions made by the 'trs2txt' script are for start and end codes labeled ELANBegin and ELANEnd, respectively, and the code for participants is labeled ELANParticipant, which should facilitate importing into ELAN. Unfortunately, I haven't found ELAN to otherwise be particularly useful for my purposes, primarily because it doesn't have the dictionary linking functions that Toolbox has, at least the last time I looked at it. It does have a Toolbox import function, so I have seen people import a Toolbox file, re-link their video, and output an html web page with embedded video for display on a website (thus the code labels in the script). Maybe you could also transcribe in ELAN, export to Toolbox, do your analysis, and import back to ELAN. For now, though, I think I'll stick with the Transcriber-Toolbox workflow.